AI & Machine Learning Labs

Master advanced topics: fine-tuning custom models, implementing observability pipelines, and detecting AI bias through hands-on labs.

GenAI Advanced Labs - Module 5

Fine-tuning, observability, and responsible AI practices.

Lab 13: Fine-Tuning Custom Models
Advanced / MLOps
Scenario: Domain-Specific Model Training
LegalTech Corp needs a model fine-tuned on legal documents for contract analysis. Create training examples in the proper format, configure hyperparameters, and validate your dataset before submitting for training.

Learning Objectives:

  • Dataset Format: Structure training data in JSONL format
  • Hyperparameter Tuning: Configure epochs, learning rate, batch size
  • Data Validation: Ensure examples meet quality requirements
  • Cost Estimation: Calculate training costs accurately

Fine-Tuning Configuration

Create training examples
📋 Task: Create Fine-Tuning Dataset
Create at least 5 training examples in proper format. Each must have a system prompt, user input, and assistant response. Configure hyperparameters within valid ranges and validate your dataset.
Dataset Requirements
✓ Minimum 5 training examples
✓ System prompt: 10-200 chars
✓ User input: 20-500 chars
✓ Assistant response: 50-1000 chars
✓ Domain-relevant content (legal)
✓ Consistent formatting
Training Examples (0/5 minimum)
No training examples yet. Click "Add Example" to create one.
Hyperparameters Configuration
More epochs = better learning, but risk of overfitting
Higher = faster learning but less stable
Larger batches = smoother training, more memory
Estimated Training Cost:
$0.00
Add examples and configure hyperparameters to see estimate
Progress: 0/5 tasks completed
Score: 0/100
0%

Lab Completed!

Excellent fine-tuning configuration!

Lab 14: LLM Observability Pipeline
Monitoring / Production
Scenario: Production Monitoring System
FinanceAI runs a customer-facing chatbot handling 100K+ daily requests. Implement an observability pipeline to track latency, token usage, error rates, and set up alerting rules to catch issues before they impact users.

Learning Objectives:

  • Metrics Collection: Track key performance indicators
  • Trace Analysis: Debug request flows and errors
  • Alert Configuration: Set up threshold-based alerts
  • Anomaly Detection: Identify unusual patterns

Observability Dashboard

Configure monitoring
📋 Task: Build Observability Pipeline
Analyze the trace logs to identify issues. Configure at least 3 alert rules with appropriate thresholds. Identify the anomaly in the system and write a query to detect similar issues.
Live Metrics (Last 5 minutes)
1,247
Total Requests
342ms
Avg Latency
2.3%
Error Rate
847K
Tokens Used
Trace Logs - Find the Anomaly
09:42:15.234 INFO Request received: user_id=u_8472, query_length=45 tokens
09:42:15.456 INFO Model response: latency=287ms, tokens_out=124
09:42:16.102 WARN Rate limit approaching: 85% of quota used
09:42:17.891 ERROR Request timeout: user_id=u_3291, latency=30247ms, model=gpt-4
09:42:18.234 INFO Request received: user_id=u_1122, query_length=32 tokens
09:42:18.512 INFO Model response: latency=298ms, tokens_out=89
09:42:19.001 ERROR Request timeout: user_id=u_4455, latency=31102ms, model=gpt-4
09:42:19.445 WARN High memory usage: 87% on inference server node-3
09:42:20.123 ERROR Request timeout: user_id=u_7788, latency=29854ms, model=gpt-4
09:42:20.567 INFO Model response: latency=312ms, tokens_out=156, model=gpt-3.5-turbo
Anomaly Identification
Based on the trace logs above, identify the anomaly pattern and explain the root cause.
Alert Rules (Configure at least 3)
No alert rules configured. Add rules to detect issues proactively.
Progress: 0/5 tasks completed
Score: 0/100
0%

Lab Completed!

Excellent observability setup!

Lab 15: AI Bias Detection & Mitigation
Ethics / Critical
Scenario: Fair Hiring AI Audit
TalentScreen AI is used by Fortune 500 companies to screen job candidates. Before deployment, you must audit the model for demographic biases, create test cases that expose potential discrimination, and recommend mitigation strategies.

Learning Objectives:

  • Bias Testing: Design paired prompts to detect discrimination
  • Fairness Metrics: Calculate demographic parity and equal opportunity
  • Mitigation Strategies: Propose techniques to reduce bias
  • Documentation: Create audit reports for compliance

Bias Audit Workbench

Create bias tests
📋 Task: Conduct Bias Audit
Create 3 paired bias test cases where only the demographic attribute differs. Analyze results to identify bias categories, then write mitigation recommendations for each detected issue.
Bias Categories to Test
• Gender Bias
• Racial/Ethnic Bias
• Age Discrimination
• Socioeconomic Bias
• Disability Bias
• Name-based Bias
Paired Bias Test Cases (0/3 required)
No test cases yet. Create paired prompts that differ only by demographic attribute.
Progress: 0/5 tasks completed
Score: 0/100
0%

Lab Completed!

Excellent bias audit!

Lab 13: Fine-Tuning Instructions

Objective

Create a fine-tuning dataset with at least 5 training examples and configure hyperparameters within valid ranges.

Configuration Steps

  1. Add Examples: Click "Add Example" and create at least 5 training examples.
  2. Format Each Example: Include system prompt (10-200 chars), user input (20-500 chars), and assistant response (50-1000 chars).
  3. Configure Hyperparameters: Set epochs (1-10), learning rate (0.1-2.0), batch size (1-32), and select base model.
  4. Validate: Click "Validate Dataset & Configuration" to check your work.
Pro Tips

Keep examples consistent in format. Use domain-specific terminology. Start with epochs=3 and learning rate=1.0 for balanced training.

Example Format

System: "You are a legal contract analyst."
User: "Analyze this clause for risks..."
Assistant: "This clause contains several risk factors..."

Common Mistakes

Examples too short or too long. Inconsistent system prompts across examples. Forgetting to select a base model.

Lab 14: Observability Instructions

Objective

Analyze trace logs to identify anomalies, configure at least 3 alert rules, and write detection queries.

Steps to Complete

  1. Analyze Traces: Review the trace log viewer and identify the pattern causing errors.
  2. Identify Anomaly: Select the correct anomaly pattern from the dropdown.
  3. Write Query: Create a detection query targeting the anomaly (SQL-like or pseudo-code).
  4. Configure Alerts: Add at least 3 alert rules with metric, condition, threshold, and severity.
  5. Validate: Click "Validate Alert Configuration" to check all rules.
Pro Tips

Look for patterns in ERROR entries. Notice which model is timing out. Create alerts for latency, error rate, and memory usage.

Anomaly Hint

Check the latency values in ERROR entries. Compare GPT-4 vs GPT-3.5 response times. The anomaly shows ~30 second latencies.

Common Mistakes

Alert thresholds too high or too low. Missing severity levels. Query doesn't target the actual anomaly pattern.

Lab 15: Bias Audit Instructions

Objective

Run bias detection on AI model outputs, analyze detected biases, and write mitigation strategies for each bias type found.

Audit Steps

  1. Run Detection: Click "Run Bias Detection" to analyze model outputs.
  2. Review Results: Examine the bias scores for each category (gender, age, etc.).
  3. Write Mitigations: For each detected bias, write a specific, actionable mitigation strategy.
  4. Validate Plan: Click "Validate Mitigation Plan" to verify your strategies.
Pro Tips

Mitigation strategies should be concrete actions, not vague goals. Address the root cause, not just symptoms.

Good Mitigation Examples
  • "Balance training data to include equal gender representation"
  • "Add demographic-neutral prompts to system instructions"
  • "Implement output filtering for biased language patterns"
Common Mistakes

Vague mitigations like "fix the bias" or "be more fair". Each mitigation must be specific and actionable.