Inference Optimization and Monitoring

This unit focuses on deploying and optimizing large language models for production: tuning hardware utilization and batch processing, applying inference acceleration tools, and implementing monitoring and observability. Students will also learn to balance performance and cost and survey emerging trends in GenAI to inform deployment decisions.

Learning Objectives

• Analyze hardware utilization and batch processing metrics to identify inference bottlenecks and propose tuned configurations that improve throughput and latency
• Apply inference acceleration tools and techniques (for example, quantization, model compilation, and runtime optimizations) within Python deployments and measure their impact on performance
• Demonstrate monitoring and observability best practices by implementing logs, metrics, and tracing for model inference services and interpreting dashboards and alerts to diagnose issues
• Evaluate performance-versus-cost trade-offs across deployment options and recommend strategies that balance latency, scalability, and budget while considering future GenAI trends