How AI and ML actually get deployed — the infrastructure, layer by layer
If you're a DevOps engineer moving into MLOps/AIOps, this is the chapter that maps your existing skills onto AI. The good news: it's still Kubernetes, containers, CI/CD, autoscaling, and observability — but with three new twists — GPUs as a first-class scheduled resource, models as versioned artifacts with a lifecycle, and non-determinism you must monitor for. Here are the reference architectures, how the big companies build them, the trends, and a concrete learning path.
1 · The MLOps lifecycle — the platform you're actually operating
"Deploying a model" is never one step. It's a loop: data flows in, features are computed, a model is trained on GPUs, versioned in a registry, promoted through CI/CD, served, monitored — and monitoring feeds back into retraining. This loop is the MLOps platform.
The platform, layer by layer (map your DevOps tools here)
| Layer | Job | Tools you'll operate |
|---|---|---|
| Data & features | ingest, transform, serve features online+offline | Kafka, Airflow/Dagster/Prefect, Spark, Feast/Tecton, a lakehouse (Delta/Iceberg) |
| Experimentation | track runs, params, metrics, artifacts | MLflow, Weights & Biases |
| Training | distributed training on GPUs | PyTorch + DDP/FSDP/DeepSpeed, Ray Train, Kubeflow Trainer, Volcano/Kueue for scheduling |
| Registry & CI/CD | version models, gate + promote | MLflow Registry, Argo Workflows/CD, GitHub Actions; DVC for data/model versioning |
| Serving | online/batch/streaming inference | KServe, Seldon, BentoML, NVIDIA Triton, Ray Serve; vLLM/SGLang/TGI for LLMs |
| Monitoring | drift, quality, latency, cost, GPU | Prometheus + Grafana, DCGM (GPU), Evidently/WhyLabs, OpenTelemetry |
| Orchestration | the substrate under all of it | Kubernetes + the GPU stack below, Terraform for the infra itself |
Reported Sourced from Kubeflow/KServe/Ray/MLflow docs and cloud architecture guides (see docs/INFRA_RESEARCH_NOTES.md); presented as established practice, not adversarially fact-checked.
2 · LLM inference serving cluster — the reference architecture
Serving an LLM is unlike serving a REST API: requests are long-lived and streaming, the workload is GPU-memory-bound, and cost is dominated by utilization. Here's the topology that shows up again and again.
Autoscale on queue depth (vllm:num_requests_waiting) and KV-cache utilization — not CPU. A GPU pod at 95% GPU can sit at 6% CPU, so a standard HPA never scales it. Watch DCGM for GPU utilization and memory.
3 · GPUs on Kubernetes — the piece that's genuinely new
This is the highest-leverage thing to learn, because it's where MLOps diverges from ordinary DevOps. A GPU is a scarce, expensive, schedulable resource with its own driver stack.
- NVIDIA GPU Operator installs the driver, container toolkit, device plugin (exposes nvidia.com/gpu as a schedulable resource), and DCGM for metrics.
- Sharing a GPU: MIG hard-partitions an H100 into up to 7 isolated instances; time-slicing oversubscribes softly. Use MIG for isolation, time-slicing for bursty dev.
- Scheduling: Kueue or Volcano add quota + gang-scheduling (all-or-nothing for multi-GPU jobs). Karpenter provisions GPU nodes on demand, ideally from spot pools with checkpointing.
4 · How the big players actually build it
Reported — architectures from public engineering blogs/talks; details are representative of what each company has described.
- Uber — Michelangelo. The canonical internal ML platform: a feature store (Palette), managed training, a model registry, and low-latency online serving — later extended with a GenAI Gateway that puts a unified, governed proxy in front of external and self-hosted LLMs. The lesson: they standardized the platform so hundreds of teams ship models the same way.
- Netflix — Metaflow. A human-centric framework letting data scientists write Python that runs locally then scales to the cloud; heavy use of containerized steps, artifact/versioning built in, and integration with their AWS-based compute. Optimizes for developer velocity, not just serving.
- Meta — FBLearner Flow → PyTorch + giant GPU fleets. Pipeline-based training platform feeding recommendation/ranking models at massive scale; for Llama, tens of thousands of GPUs with high-speed interconnect, and heavy investment in distributed training (FSDP) and inference optimization.
- LinkedIn. A feature store + model platform, and for GenAI a platform that reused existing messaging infrastructure (FIFO, persistence, horizontal scale) for agent orchestration rather than building new — a very DevOps-minded reuse. Agents run as gRPC services in a skill registry.
- Spotify. ML platform ("Hendrix") standardized on Ray for distributed compute, with golden-path tooling (Backstage) so ML shipping feels like normal software delivery.
- OpenAI / Anthropic (as publicly described). Enormous Kubernetes clusters (OpenAI has publicly discussed scaling K8s to thousands of nodes), custom scheduling for scarce GPUs, and heavily optimized inference stacks. The frontier is as much cluster and scheduling engineering as it is modeling.
Every one built a paved-road internal platform: standardized feature store + registry + CI/CD + serving so individual teams don't reinvent infra. As an MLOps/AIOps engineer, you are the person who builds and runs that paved road.
5 · 2026 trends worth tracking
- GPU alternatives & scarcity. AMD MI300X, AWS Trainium/Inferentia, Google TPU, and inference-specialist silicon (Groq, Cerebras) are chipping at NVIDIA's default — portability (via ONNX, Triton, vLLM backends) is becoming a real ops concern.
- Serverless GPU & scale-to-zero (Modal, RunPod, Baseten, Cloud Run GPUs) for bursty/low-QPS workloads.
- The AI gateway as a category — a governed control plane (routing, budgets, guardrails, caching, observability) in front of all model traffic. Increasingly mandatory in enterprises.
- MLOps → LLMOps → AgentOps. New components entered the stack: prompt management, eval pipelines, vector DBs, guardrails, and agent orchestration/durable execution. The lifecycle is the same shape; the artifacts changed.
- Disaggregated inference (splitting prefill and decode onto different pools) and KV-cache-aware routing for efficiency at scale.
- AIOps — agents for incident response and SRE (Cleric, Resolve AI, Dropzone) reading your observability stack and proposing fixes.
6 · Your DevOps → MLOps / AIOps roadmap
You already have the foundation. Here's what to add, in priority order.
| You already know | Add for MLOps/AIOps |
|---|---|
| Docker, Kubernetes | GPUs on K8s — NVIDIA GPU Operator, device plugin, MIG/time-slicing, Kueue/Volcano, Karpenter GPU pools |
| CI/CD (Actions/Argo) | Model CI/CD — MLflow registry, model promotion gates, canary/shadow rollouts, DVC |
| Terraform / IaC | IaC for GPU infra — node pools, quotas, and the serving stack as code |
| Prometheus / Grafana | DCGM (GPU metrics) + drift/quality monitoring (Evidently) + OpenTelemetry GenAI tracing (tokens/cost/latency) |
| Load balancing / autoscaling | KEDA on queue depth, scale-to-zero for GPUs, cold-start mitigation |
| Services / APIs | Model serving — KServe, Ray Serve, BentoML, Triton; vLLM for LLMs |
| — | Pipelines & data — Airflow/Dagster, a feature store (Feast), a vector DB |
| — | LLM/agent layer — AI gateways (LiteLLM), guardrails, eval pipelines, durable execution (Temporal) |
Weeks 1–3: run vLLM in a container, then on K8s with the GPU Operator; scrape DCGM into Grafana. Weeks 4–6: put KServe + KEDA in front of it, add queue-depth autoscaling and a canary rollout. Weeks 7–9: stand up MLflow (registry + tracking) and an Argo pipeline that promotes a model. Weeks 10–12: add an AI gateway (LiteLLM), OpenTelemetry tracing, and a drift monitor. Build all of it with Terraform. Now you've operated the whole loop.
Key takeaways
- MLOps is DevOps + three twists: GPUs as scheduled resources, models as versioned lifecycle artifacts, and monitoring for non-determinism/drift.
- Learn GPUs on Kubernetes first — it's where MLOps genuinely diverges from DevOps.
- Serving LLMs means autoscaling on queue depth + KV-cache, not CPU, and watching DCGM.
- Every big company built a paved-road platform — that's the MLOps/AIOps job.