Thank you for sending your enquiry! One of our team members will contact you shortly.
Thank you for sending your booking! One of our team members will contact you shortly.
Course Outline
Introduction to LLMOps
- Contrasting LLMOps with MLOps: addressing the unique operational challenges of LLMs
- The LLM application lifecycle: prompt design, evaluation, deployment, and monitoring
- A production readiness checklist for GenAI applications
Managing and Versioning Prompts
- Utilizing prompt templating systems and variable injection
- Applying semantic versioning to prompts alongside automated regression testing
- Establishing prompt registries and collaborative workflows
Evaluating LLMs at Scale
- Key evaluation dimensions: accuracy, relevance, safety, and groundedness
- Using LLM-as-judge metrics and designing human evaluation pipelines
- Leveraging automated evaluation frameworks such as RAGAS, DeepEval, and custom evaluators
- Integrating quality gates into CI/CD pipelines for LLM deployments
Safety Guardrails and Content Governance
- Implementing input and output guardrails using tools like NeMo Guardrails and Guardrails AI
- Detecting PII, filtering toxicity, and defining topic boundaries
- Strategies for defending against jailbreaks and prompt injection attacks
- Red-teaming LLM applications to ensure safety assurances
Observability and Monitoring for LLMs
- Tracking telemetry data: token usage, latency, costs, and quality metrics
- Detecting drift in LLM outputs and embedding spaces
- Tracing session-level interactions for multi-turn agent conversations
- Creating dashboards and setting alerts using LangSmith, Arize, and OpenTelemetry
AI Gateways and Model Orchestration
- Multi-provider routing via LiteLLM and Portkey
- Defining fallback strategies, retry logic, and circuit breakers
- Cost-aware model selection and load balancing techniques
- Managing rate limits, quotas, and API keys
Performance Optimization
- Semantic caching using vector stores and exact-match strategies
- Enforcing structured outputs through constrained decoding
- Implementing batching, streaming, and concurrency patterns
- Optimizing latency across different model providers
Governance, Compliance, and Auditing
- Maintaining LLM audit trails: logging prompts, responses, and decision provenance
- Addressing data residency and privacy considerations for LLM APIs
- Implementing policy-as-code for internal LLM usage
- Developing an internal LLM operations playbook
Requirements
- Practical experience in developing or integrating LLM-based applications.
- Proficiency with Python and REST APIs.
- A foundational grasp of prompt engineering concepts.
Intended Audience
- ML engineers and MLOps specialists transitioning into LLM operations.
- Platform engineers overseeing LLM infrastructure.
- Technical leads responsible for managing production GenAI deployments.
14 Hours
Testimonials (2)
use of proper and effective prompt
Marses Pacaldo
Course - Generative AI and Prompt Engineering for Corporate Professionals
The interactive style, the exercises
