At KubeCon + CloudNativeCon North America 2026 in Salt Lake City, one theme dominated the hallway talk: Kubernetes AI inference. Serving large language models is no longer a special project run on a few GPU boxes – it is becoming a standard platform workload, and the cloud-native community is building the tools to run it at scale.

The Numbers: Kubernetes Is Already the AI Platform

• According to the CNCF 2026 annual survey, 82% of container users run Kubernetes in production.
• 66% of organisations hosting generative AI workloads use Kubernetes for some or all of their inference.
• A theCUBE Research survey of 375 professionals found 92% have integrated AI into at least one stage of their software lifecycle, and 79% expect full-scale AI agent adoption within three years.

What Is llm-d?

llm-d is an open-source, Kubernetes-native distributed inference stack founded by Red Hat, IBM, Google Cloud, CoreWeave and NVIDIA, and now governed by the CNCF. It sits on top of model servers such as vLLM and SGLang and adds the smart routing and scheduling needed to serve LLMs efficiently across a cluster.

On October 20, 2026, llm-d reached v1.0, with a stable v1 API focused on “Day 2” operations – running inference reliably in production.

Key llm-d Features

Capability	Why it matters
Prefix/session-aware routing	Sends requests to replicas that already hold the relevant KV cache, cutting latency and GPU cost
Prefill/decode disaggregation	Runs the two phases of inference on separately scaled pools
KV cache offloading	Moves cache to cheaper memory tiers to fit more sessions
Wide expert parallelism	Efficiently serves large Mixture-of-Experts models
Multi-tenant QoS	Lets teams share GPUs with priorities and fairness
Zero-downtime upgrades	Roll out new model versions without dropping traffic
Agent-aware serving	Prioritises multi-turn agent workloads

Getting Started: vLLM on Kubernetes

Before adopting a full distributed stack, many teams start with a single vLLM deployment. Here’s a minimal example:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-server
spec:
  replicas: 1
  selector:
    matchLabels: { app: vllm }
  template:
    metadata:
      labels: { app: vllm }
    spec:
      containers:
      - name: vllm
        image: vllm/vllm-openai:latest
        args: ["--model", "Qwen/Qwen2.5-7B-Instruct", "--max-model-len", "8192"]
        ports: [{ containerPort: 8000 }]
        resources:
          limits:
            nvidia.com/gpu: 1
---
apiVersion: v1
kind: Service
metadata:
  name: vllm
spec:
  selector: { app: vllm }
  ports: [{ port: 80, targetPort: 8000 }]

The server exposes an OpenAI-compatible API, so existing clients work unchanged:

kubectl port-forward svc/vllm 8000:80
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"Qwen/Qwen2.5-7B-Instruct","messages":[{"role":"user","content":"Hello"}]}'

When you outgrow a single replica – multiple models, many tenants, long agent sessions – that’s when llm-d’s routing and scheduling layer pays off. Check the official llm-d documentation for Helm-based install guides.

Other Cloud-Native AI News to Watch

• Google donated Agent Substrate, gVisor and OpenRL to the CNCF on November 5, 2026.
• Red Hat is integrating NVIDIA’s OpenShell agent runtime into Red Hat AI and OpenShift to run AI agents in isolated, policy-controlled environments.
• Broadcom’s VM API project was accepted into the CNCF Sandbox on November 9.

Best Practices for Kubernetes AI Inference

• Use the NVIDIA GPU Operator (or your vendor’s equivalent) to manage drivers and device plugins.
• Autoscale on inference metrics (queue length, tokens/sec), not just CPU.
• Cache model weights on a shared volume to speed up pod startup.
• Separate latency-sensitive chat traffic from batch jobs with priorities.
• Monitor GPU utilisation – idle GPUs are the most expensive resource in your cluster.

Conclusion

Kubernetes AI inference has gone mainstream, and llm-d 1.0 marks the moment the ecosystem got a production-ready, vendor-neutral way to serve LLMs at scale. If your team already runs Kubernetes, you already have the foundation – start with vLLM, measure, and adopt llm-d as your inference needs grow.