Data Analytics Expert Labs

🎯 Goal: Setup Kafka topic for IoT sensor data

📝 Kafka Configuration:
Create topic with proper partitioning for high throughput. Set retention policy based on data volume. Configure replication factor for fault tolerance. Use compression (snappy/lz4) to reduce network overhead.

💻 Configuration:
1. Topic: sensor-events
2. Partitions: 12
3. Replication: 3
4. Click "Create Topic"

🎯 Goal: Begin consuming events from Kafka topic

📝 Consumer Configuration:
Set consumer group for parallel processing. Configure auto-offset-reset (earliest/latest). Enable exactly-once semantics with idempotent producer. Set max.poll.records for batch processing.

💻 Action:
1. Consumer Group: analytics-group
2. Offset: earliest
3. Click "Start Consumer"

🎯 Goal: Set up time-based aggregation windows

📝 Window Types:
• Tumbling: Fixed-size, non-overlapping (e.g., 5-min intervals)
• Sliding: Overlapping windows (e.g., 5-min window, 1-min slide)
• Session: Based on inactivity gaps
Choose based on use case - tumbling for periodic reports, sliding for continuous monitoring.

💻 Configuration:
Window: tumbling_5min
Duration: 5 minutes
Click "Apply Window"

🎯 Goal: Calculate real-time metrics on streaming data

📝 Aggregation Functions:
COUNT: Event frequency, SUM: Total values, AVG: Mean values, MIN/MAX: Range detection, PERCENTILE: Distribution analysis. Use state stores for incremental calculations to avoid recomputing entire windows.

💻 Required Aggregations:
1. COUNT events per window
2. AVG sensor_value
3. MAX sensor_value
4. Click "Apply Aggregations"

🎯 Goal: Set up real-time alerting for unusual patterns

📝 Detection Methods:
• Threshold-based: Simple value limits (sensor > 100)
• Statistical: Z-score, standard deviation (3-sigma rule)
• ML-based: Isolation Forest, autoencoders
• Pattern-based: Sudden spikes, drops, flatlines

💻 Configuration:
Condition: value > threshold * 1.5
Alert: high_sensor_alert
Click "Enable Alerting"

🎯 Goal: Validate pipeline throughput and latency

💻 Monitoring:
1. Check events/sec throughput
2. Verify end-to-end latency < 1s
3. Monitor consumer lag
4. Review alert triggers

🎯 Goal: Implement encryption at-rest and in-transit

📝 Encryption Best Practices:
At-rest: Use AES-256 for database/file encryption. Enable transparent data encryption (TDE) for SQL databases. At-transit: Force TLS 1.2+ for all connections. Use certificate pinning for APIs. Rotate keys regularly (90 days recommended).

💻 Configuration:
1. At-Rest: AES-256
2. In-Transit: TLS 1.3
3. Key Rotation: 90 days
4. Click "Enable Encryption"

🎯 Goal: Implement role-based access control

📝 RBAC Principles:
Create roles based on job functions (analyst, engineer, admin). Follow least privilege - grant minimum permissions needed. Separate duties - no single user has full control. Regular access reviews (quarterly). Disable inactive accounts after 30 days.

💻 Required Roles:
1. data_analyst: READ only
2. data_engineer: READ, WRITE
3. data_admin: ALL privileges
4. Click "Configure RBAC"

🎯 Goal: Track all data access and modifications

📝 Audit Logging:
Log all authentication attempts (success/fail), data access (SELECT), modifications (INSERT/UPDATE/DELETE), permission changes, query execution. Include: timestamp, user, action, affected resources. Store logs immutably for compliance (7 years GDPR).

💻 Configuration:
Log Level: comprehensive
Include: auth, queries, permissions
Click "Enable Auditing"

🎯 Goal: Protect PII with dynamic masking

📝 Masking Techniques:
• Static: Replace with fake data (tokenization)
• Dynamic: Mask on read based on user role
• Partial: Show last 4 digits (credit cards, SSN)
• Hashing: One-way for non-reversible protection
Apply to: emails, phone numbers, SSN, credit cards, addresses

💻 Configuration:
Fields: email, ssn, credit_card
Method: dynamic_masking
Click "Apply Masking"

🎯 Goal: Verify GDPR/HIPAA compliance

📝 Compliance Checks:
GDPR: Right to erasure, data portability, consent tracking, breach notification (72hr). HIPAA: PHI encryption, access controls, BAA agreements, minimum necessary rule. SOC2: Access controls, change management, monitoring.

💻 Action:
1. Run automated compliance scan
2. Review identified gaps
3. Click "Run Compliance Scan"

🎯 Goal: Create comprehensive security assessment report

💻 Report Sections:
1. Security controls implemented
2. Compliance status (pass/fail)
3. Vulnerabilities found
4. Remediation recommendations
5. Click "Generate Report"

🎯 Goal: Create effective visual hierarchy

📝 Layout Principles:
F-pattern reading: Key metrics top-left. Z-pattern for scanning. Group related visuals. Use whitespace effectively. 3-second rule: Critical info visible immediately. Progressive disclosure: Summary → Details.

💻 Configuration:
1. Layout: Grid (3 columns)
2. Header: KPI Summary Cards
3. Main: Trend Charts
4. Click "Apply Layout"

🎯 Goal: Display key performance indicators prominently

📝 KPI Card Design:
Include: Current value, comparison (vs target, vs prior period), trend indicator (↑↓). Use color coding: Green (good), Yellow (warning), Red (alert). Show sparklines for context. Keep labels concise.

💻 Required KPIs:
1. Revenue: $12.5M (+8% YoY)
2. Customers: 45,230 (+12%)
3. NPS Score: 72 (+5 pts)
4. Click "Add KPIs"

🎯 Goal: Show data trends over time

📝 Chart Selection:
• Line chart: Trends over time (continuous)
• Bar chart: Comparisons between categories
• Area chart: Volume/cumulative totals
• Combo chart: Multiple metrics on same scale
Avoid: 3D charts, pie charts with >5 slices, dual Y-axes confusion

💻 Configuration:
Chart Type: line_chart
Metric: monthly_revenue
Click "Add Chart"

🎯 Goal: Enable drill-down and filtering

📝 Interactivity Types:
• Global filters: Apply to all visuals (date range, region)
• Cross-filtering: Click one chart filters others
• Drill-down: Click to see detail levels
• Tooltips: Hover for additional context
• Parameters: User-adjustable thresholds

💻 Configuration:
1. Date Range Filter (global)
2. Region Slicer (dropdown)
3. Enable cross-filtering
4. Click "Apply Filters"

🎯 Goal: Ensure fast load times with large data

📝 Optimization Techniques:
• Pre-aggregate data (don't load raw rows)
• Use incremental refresh
• Limit visible records (pagination)
• Reduce cardinality (summarize before loading)
• Cache frequently-accessed queries
• Remove unused columns from model

💻 Actions:
1. Enable data aggregation
2. Set cache TTL: 5 minutes
3. Click "Optimize"

🎯 Goal: Deploy dashboard for stakeholder access

💻 Publishing:
1. Test on mobile/tablet
2. Set refresh schedule (hourly)
3. Configure row-level security
4. Click "Publish Dashboard"