AI & Machine Learning Foundation Labs

🎯 Goal: Set up a vector store to hold document embeddings

📝 What is a Vector Database?
A vector database stores text as high-dimensional vectors (embeddings). Similar text = similar vectors. This enables semantic search - finding relevant content by meaning, not just keywords.

💻 Initialize Database:
init_vectordb --name legal_docs --engine pinecone

💻 Set Embedding Model:
set_embedding_model text-embedding-3-small

🔍 Popular Vector DBs:
• Pinecone - Managed, scalable
• Chroma - Open source, local
• Weaviate - Feature-rich
• FAISS - Facebook's library, fast

🎯 Goal: Split documents into optimal-sized pieces for retrieval

📝 Why Chunking Matters:
LLMs have context limits. Chunking breaks documents into digestible pieces. Too small = lost context. Too large = irrelevant noise. Finding the sweet spot is crucial!

💻 Set Chunk Size:
set_chunk_size 512

💻 Set Overlap (prevents cutting sentences):
set_chunk_overlap 50

💻 Choose Chunking Strategy:
set_chunking_strategy "semantic"

🎯 Goal: Process documents, create embeddings, store in vector DB

📝 What Happens During Ingestion:
1. Documents are loaded and parsed
2. Text is split into chunks
3. Each chunk is converted to a vector (embedding)
4. Vectors + metadata stored in database

💻 Ingest Document Folder:
ingest_documents ./legal_corpus/ --recursive

💻 View Statistics:
show_ingestion_stats

🎯 Goal: Tune how relevant chunks are retrieved for queries

📝 Key Retrieval Concepts:
• Top-K: How many chunks to retrieve
• Similarity threshold: Minimum relevance score
• Reranking: Second-pass scoring for better accuracy

💻 Set Top-K Results:
set_retrieval_k 5

💻 Set Minimum Similarity:
set_similarity_threshold 0.7

💻 Enable Reranking (recommended):
enable_reranking --model cross-encoder

🎯 Goal: Connect retriever to LLM with a prompt template

📝 RAG Chain Components:
1. Retriever - Fetches relevant chunks
2. Prompt Template - Injects context into prompt
3. LLM - Generates answer from context
4. Output Parser - Formats the response

💻 Create the Chain:
create_rag_chain --retriever vectordb --llm gpt-4 --template "Answer based on context: {context}\n\nQuestion: {query}"

🎯 Goal: Verify the system retrieves and answers correctly

💻 Run a Test Query:
query_rag "What are the liability limitations in contract #A-2024-001?"

💻 Inspect Retrieved Chunks:
show_retrieved_chunks

✅ What to Verify:
• Are retrieved chunks relevant?
• Does the answer cite sources?
• Is the response accurate to the context?
• Are hallucinations avoided?

🎯 Goal: Validate your training dataset format and quality

📝 Training Data Format:
Fine-tuning requires JSONL format with conversation structure:
{"messages": [{"role": "system", "content": "..."}, {"role": "user", "content": "..."}, {"role": "assistant", "content": "..."}]}

💻 Validate Data File:
validate_jsonl ./training_data/medical_notes.jsonl

💻 View Data Statistics:
show_data_stats

🔍 Quality Checklist:
• Minimum 50-100 examples (500+ recommended)
• Consistent format across examples
• High-quality, accurate responses
• Diverse coverage of use cases

🎯 Goal: Set up hyperparameters for optimal training

📝 Key Hyperparameters:
• Epochs: How many times to iterate through data
• Learning Rate: How fast the model adapts
• Batch Size: Examples processed together

💻 Create Fine-Tuning Job:
create_finetune_job --base-model gpt-3.5-turbo --training-file medical_notes.jsonl

💻 Set Hyperparameters:
set_hyperparams --epochs 3 --learning_rate 0.0001 --batch_size 4

🎯 Goal: Launch the fine-tuning job and monitor progress

📝 What Happens During Training:
1. Model loads your training data
2. Weights are adjusted based on examples
3. Loss is calculated after each batch
4. Model checkpoint saved after each epoch

💻 Start the Training Job:
start_training

💻 Watch Progress in Real-Time:
watch_training_progress

🎯 Goal: Analyze training metrics to ensure model is learning

📝 Key Metrics to Watch:
• Training Loss: Should decrease over epochs
• Validation Loss: Should track training loss
• If val loss increases while train decreases = overfitting!

💻 View Loss Curve:
show_loss_curve

💻 Check Validation Metrics:
show_validation_metrics

🎯 Goal: Make your trained model available for inference

📝 Deployment Considerations:
• Model ID is unique to your fine-tuned version
• Inference costs same as base model
• Can be used via same API endpoints

💻 Deploy the Model:
deploy_model --model-id ft:gpt-3.5-turbo:medical:2024

🎯 Goal: Validate model performance on real-world inputs

💻 Test with Domain Input:
test_model "Summarize: Patient presents with acute myocardial infarction..."

💻 Compare Base vs Fine-Tuned:
compare_base_vs_finetuned

✅ Evaluation Criteria:
• Domain-specific terminology accuracy
• Response format consistency
• Factual correctness
• Appropriate tone and style