Thank you for sending your enquiry! One of our team members will contact you shortly.
Thank you for sending your booking! One of our team members will contact you shortly.
Course Outline
EXO Infrastructure as Code
- Overview of EXO deployment architectures: single-node, multi-node, and RDMA-based clusters.
- Automating the installation of dependencies (Xcode, uv, Node.js, Rust) using configuration management tools.
- Leveraging Nix flakes to achieve reproducible EXO builds and consistent developer environments.
- Authoring Ansible playbooks or shell scripts for unattended cluster provisioning.
Reproducible Builds and CI Integration
- Pinning dependencies and compiling the dashboard within CI pipelines.
- Executing EXO smoke tests on GitHub Actions or GitLab CI runners.
- Generating golden images and snapshot-based rollback workflows for macOS and Linux virtual machines.
- Versioning custom model cards in tandem with application code.
Cluster Discovery and Networking Automation
- Configuring mDNS and static DNS for robust libp2p node discovery.
- Automating the creation of network profiles and managing Thunderbolt bridges on macOS.
- Utilizing custom namespaces (EXO_LIBP2P_NAMESPACE) to isolate dev, staging, and production clusters.
- Implementing firewall rules and network segmentation for multi-tenant scenarios.
Storage and Model Lifecycle Management
- Strategizing EXO_MODELS_DIRS and EXO_MODELS_READ_ONLY_DIRS configurations.
- Mounting NFS or SAN shares as read-only repositories to accelerate model provisioning.
- Managing garbage collection for outdated caches and defining retention policies for versioned weights.
- Automating pre-downloads of models and health checks prior to rolling updates.
Monitoring and Alerting
- Routing EXO logs to centralized logging platforms (ELK, Loki, or Splunk).
- Constructing Grafana dashboards using EXO_TRACING_ENABLED outputs.
- Setting up alerts for cluster membership changes, OOM incidents, and spikes in inference latency.
- Correlating macmon hardware telemetry with model performance degradation.
Update, Rollback, and Disaster Recovery
- Staging EXO binary updates on canary nodes before fleet-wide deployment.
- Executing model-level rollbacks by switching between quantized versions without re-downloading.
- Backing up and restoring cluster state, custom namespaces, and cached weights.
- Documenting recovery runbooks for full cluster rebuild scenarios.
Security Hardening and Compliance
- Enabling TLS at the reverse proxy layer (nginx, traefik) for the dashboard and API.
- Enforcing API rate limiting and IP whitelisting for EXO endpoints.
- Isolating clusters using VLANs and zero-trust network policies.
- Auditing access controls and maintaining an inventory of deployed models and their versions.
Requirements
- Practical experience with DevOps workflows, including CI/CD, IaC, and container orchestration.
- Proficiency in system administration and package management for macOS or Linux.
- A solid grasp of networking, DNS, and storage fundamentals.
Target Audience
- DevOps Engineers
- Infrastructure Architects
- SREs managing on-premise AI workloads
21 Hours
Testimonials (2)
Craig was extremely involved in the training, always making sure we are paying attention, adapted the examples to our day-to-day activities and always provided an answer when asked, even if the information was not added in the presentation.
Ecaterina Ioana Nicoale - BOOKING HOLDINGS ROMANIA SRL
Course - DevOps Foundation®
High level of commitment and knowledge of the trainer
