Self-Hosting Decision Models: Serving, Quantisation, Latency and Operations Training Course

Deployment Options

• Trade-offs between hosted APIs and self-hosted solutions.
• Patterns for on-premises, edge, and air-gapped environments.
• Comparing single-model implementations versus model portfolios.
• Analysis of cost per decision.

Hardware Sizing and Capacity

• Memory footprint analysis based on parameter count.
• CPU-only performance versus GPU acceleration.
• Planning for concurrency and throughput.
• Providing headroom for increasing task lengths.

Serving Stacks

• Evaluating vLLM and SGLang options.
• Using llama.cpp and GGUF for execution.
• Leveraging Ollama for local development.
• Selecting stacks based on the shape of the decision workload.

Quantisation Without Losing Calibration

• Exploring precision formats and memory savings.
• Measuring quality and calibration loss.
• Applying per-tensor and importance-aware quantisation variants.
• Conducting acceptance testing after quantisation.

Latency Engineering

• Distinguishing between median and tail latency.
• Understanding latency growth with context length.
• Utilizing warm starts and model caching.
• Batching multiple decision questions for efficiency.

Edge and Constrained Environments

• Running small decision models on CPUs.
• Deployment on Apple Silicon and embedded devices.
• Strategies for offline operation and updates.
• Distributing weights in air-gapped settings.

Containerisation and Scaling

• Designing images optimized for model serving.
• Managing volumes and caches.
• Implementing horizontal scaling and routing.
• Setting resource limits and sharing GPUs.

Observability

• Tracking request, latency, and error metrics.
• Monitoring confidence distributions.
• Tracking decision outcomes and escalations.
• Alerting on calibration drift.

Versioning and Rollout

• Versioning models and adapters.
• Employing shadow and canary deployments.
• Pinning model files for audit purposes.
• Defining rollback procedures.

Security and Multi-Tenancy

• Implementing authentication and rate limiting.
• Ensuring tenant isolation for prompts and logs.
• Managing network isolation and egress control.
• Logging decisions for audit trails.