Intro
For most of the last two years, “serving a model” meant one prompt in, one response out. You sized a pool, put a load balancer in front, and the traffic was roughly uniform. That assumption is gone. The workloads arriving now are multi-step agents that fan out across tools, reinforcement learning rollouts that fire thousands of parallel generations, and long conversations where each turn re-sends a prefix that has already been processed. They share context, they arrive in bursts, and the burst is not evenly spread. llm-d 1.0 is Google’s answer to that shift, released at the PyTorch Conference as a CNCF Sandbox project co-founded with Red Hat, IBM and others, with more than 500 contributors across 25 organizations behind it (announcement (opens in new tab)).
The failure mode of round-robin under this traffic is easy to describe and expensive to watch. Request one lands on replica A and builds a KV cache for a 40,000 token system prompt. Request two, sharing that prefix, lands on replica B, which builds the same cache from scratch. Replica C does it a third time. You paid for the prefill three times, and now three accelerators are each holding a copy of the same state. On a box with finite HBM, duplicate KV cache is the thing that pushes you into eviction, and eviction under multi-turn load turns into a recompute storm. I have watched smaller versions of this on caching layers that had no session affinity, and the shape of the problem is always the same: the balancer knows how to spread load, but it has no idea what the work already knows.
If you run WordPress or a PHP app, the closest familiar version is a page cache that lives per-worker with a balancer that ignores which worker built it. Every request that lands on the wrong worker rebuilds the page from the database. The cache exists, it just does not help.
llm-d 1.0 turns a cluster of model servers into a coordinated pool with cache-aware routing, tiered KV cache and flow control in front of it. It is built for teams serving their own models on their own clusters, not a managed API, so you still own the engine, the weights and the capacity planning.
Background
None of the individual ideas behind llm-d 1.0 were new. Prefix-aware routing existed. So did load-aware routing, KV cache offloading, prefill/decode disaggregation, wide expert parallelism, and batch inference across TPUs and GPUs. Each one shipped as a feature in a specific project, solving a specific bottleneck, and each one came with its own configuration surface and its own assumptions about how requests reach a model server. What did not exist was a single stack where those features could talk to each other. Teams ended up gluing a router from here to an offload layer from there and hoping the metrics lined up.
The harder problem was cultural, in the boring operational sense. llm-d sits between two communities that rarely share a deployment. PyTorch is where vLLM and SGLang actually serve models, and that world optimises kernels, quantisation, and batch shapes. Kubernetes is where distributed systems run, and that world optimises scheduling, health checks, and rollout semantics. Merging them means the router has to know what a KV cache block is while still behaving like a well-mannered Kubernetes controller. According to Google’s announcement (opens in new tab), that merger took over 500 contributors across 25 organizations.
The evolution the team diagrams in Figure 1 runs in three stages, and it is worth reading as a maturity ladder rather than a feature list. Stage one is plain round-robin, which is what most people deploy first because a Kubernetes Service is right there and it works fine on chat traffic. Stage two adds cache-aware routing on top of a tiered KV cache, so the router starts steering requests toward replicas that already hold a matching prefix, and cold state gets pushed down to host memory, SSD, or Managed Lustre instead of being evicted. Stage three adds priority flow control, which is the stage that makes a shared fleet survivable when a latency-sensitive user and a bursty agent loop want the same accelerators.
Underneath the router, llm-d builds on Gateway API Inference, the upstream Kubernetes project for inference-optimised routing. That matters more than it sounds. It means the routing logic is a Kubernetes API surface rather than a bespoke proxy config, so the same primitives can be implemented by different vendors and still behave predictably.
I will be blunt about one thing: the three-stage framing makes the progression look inevitable. It is not. Stage two only pays off if your traffic actually shares prefixes, and plenty of workloads do not.
What llm-d 1.0 brings to the Kubernetes inference stack
The headline item in 1.0 is the v1 API surface and a request path the team calls hardened, validated across multiple clouds and accelerator types. For a rollout, that translates into something dull but useful: you are not negotiating an alpha CRD schema every time you upgrade the operator, and the failure modes other people already hit on GKE, EKS, or a TPU pool have been hit before you. I have run enough pre-1.0 serving stacks to value the difference between “it works on the author’s cluster” and “someone ran it on three clouds before tagging it.”
Multi-tenant quality of service is where the release earns its keep. Flow control in llm-d does priority and fairness-aware routing, which means an interactive request from a paying user and a background agent loop grinding through tool calls can share the same fleet without the loop starving the human. Anyone who has watched a single batch job eat a whole node pool knows why this matters.
Safer upgrades get three pieces. Router high availability so the routing layer is not a single pod you are scared to restart. Zero-downtime updates on the router itself. And DisaggregatedSet, a controller that orchestrates prefill/decode-aware rolling updates, keeping the pairing between a prefill pool and its decode pool intact so context does not evaporate mid-session when a new ReplicaSet rolls in. If you have ever watched a rolling update quietly drop active streaming connections because the old and new pods had mismatched cache state, this is aimed at you.
Agent-aware serving adds session-aware routing, turn-priority scheduling, and native Responses API state, so a multi-turn agent reuses the context it already paid to compute instead of re-sending the whole transcript every turn.
On the TPU side, TorchTPU runs PyTorch models natively inside upstream vLLM and SGLang, with co-operative time-slicing across a TPU fleet to lift utilisation. That last piece is the one I have the least operational data on, and I would want to see real numbers from a shared deployment before betting a production SLO on it.
How does cache-aware routing actually work on GKE?
The problem it solves is easiest to see when a load balancer is doing its job correctly and the result is still wrong. A three-step agent loop sends step one to pod A. Round-robin sends step two to pod B, which holds none of the prefix, so it re-runs prefill over the whole conversation. Step three lands on pod C and does the same thing again. Three replicas, one conversation, three copies of the same KV state in accelerator memory. Cache-aware routing flips that: the llm-d router tracks which replicas hold each cached prefix and sends a session’s next step to a replica that already has the context, weighing current load and predicted latency when several candidates qualify.
What sits in front of the fleet is the part worth internalising. There is one endpoint, one router, one KV cache index, and one flow-control policy spanning every accelerator pool you have attached. Agents call a single API address. They do not know or care whether the next token comes from an H100 pool in one zone or an Ironwood TPU slice somewhere else. When accelerator memory fills, llm-d does not drop the prefix by default. It pushes KV cache to host memory via vLLM’s OffloadingConnector, then to SSD or Google Cloud Managed Lustre, which is what keeps prefix-eviction storms from wrecking long-context multi-turn traffic.
On GKE specifically, you do not install most of this yourself. GKE Inference Gateway runs the llm-d router as a managed service, so the install, the upgrade path and the multi-region extension of routing are handled for you, and it builds on the Gateway API Inference project upstream.
The question I get asked most is whether routing across a mixed fleet means rewriting the agent. It does not. Adding capacity or mixing accelerator generations is a platform change. TorchTPU runs PyTorch models natively inside upstream vLLM and SGLang, so the model you already serve does not need changes to land on TPUs. Google’s own October 2026 numbers put this at more than 85% lower median time-to-first-token and 2.4x throughput for an agentic coding workload on Ironwood TPU versus a plain Kubernetes Service, per the llm-d 1.0 announcement (opens in new tab). I would treat those as a directional result from one workload rather than a promise for yours, because prefix hit rate depends entirely on how repetitive your traffic actually is.
What it means in practice
The payoff people underestimate is that the endpoint stops being tied to the hardware behind it. If your model currently lives on a pool of L4s and you want to move it to Ironwood TPUs, or split it so prefill sits on one generation and decode on another, that is a platform change now. Nothing in the agent code changes. I have done enough hosting migrations to know how rare that is. Moving a site between servers is normally a nameserver change and a database dump, not a rebuild of the application, and that is exactly the property llm-d is trying to give you for accelerator pools. The application keeps calling one address.
Compositional adoption is the part that reads better in a press release than it feels in a sprint. You keep your engine, your model and your cluster, and you take only the serving components you need. That is genuinely useful if you already run vLLM and have opinions about your deployment manifests. It also means you own the wiring. The router version, the Gateway API Inference CRDs, the engine version and your offloading config all have to agree with each other, and none of those projects ship on the same cadence. Budget for that.
The keynote demo is the concrete data point worth holding onto: 1,000 sandboxed agents on Agent Substrate on GKE, all hitting a single llm-d endpoint spanning vLLM, SGLang, and a mixed TPU/GPU fleet, with cache-aware routing keeping each agent’s steps on the replicas holding its context while flow control queued the bursty loops. It is a demo, so treat it as an existence proof rather than a capacity number.
Where I think this is still immature is the operating cost of the KV cache index and the flow-control policy if you skip GKE Inference Gateway and run the llm-d router yourself. A cache index is distributed state that has to stay roughly accurate across replicas while pods restart, and a flow-control policy is only as good as the traffic data you feed it. Tuning priorities against a workload you have not measured yet is guesswork. The managed path exists for a reason.
What to expect next
The v1 label is the thing to watch first. llm-d 1.0 says the APIs and the request path are stable, and that is a meaningful commitment after a long stretch where the project moved under people’s feet. But v1 in a CNCF Sandbox project is not the same promise as v1 in Kubernetes itself. If you pin the Gateway API Inference CRDs and build routing manifests against them, a breaking change in a minor release means reworking those manifests and re-testing whatever offloading and disaggregation config you layered on top. I would keep a copy of the exact CRD versions in your repo alongside your Terraform, because the upgrade path will be easier to reason about with the old shape sitting right there.
TorchTPU’s co-operative time-slicing is the piece I have the least confidence about. Raising utilization across a TPU fleet by sharing slices between models sounds good on paper, and for batch and RL rollout traffic it probably is. What nobody has published much of is how tail latency behaves when two tenants contend for the same slice. My expectation is that the first wave of adoption lands on background workloads and that interactive paths stay on dedicated slices until people have their own numbers. I do not know how this holds up at scale, and I would rather say that than pretend the demo answers it.
The agent-serving primitives are where I expect the most churn over the next few releases. Session-aware routing, turn-priority scheduling and native Responses API state all encode assumptions about how agents behave, and agent frameworks are still arguing about that. When a framework changes how it addresses the model or how it manages its own conversation state, the routing semantics have to follow. If you are building on those primitives today, read the release notes before every minor bump.
Governance is the quieter story. llm-d is still a CNCF Sandbox project, which means multi-vendor support and the decision-making structure around it are still forming. Sandbox projects do graduate, and they also stall. For a production dependency that matters to procurement as much as engineering, so check where the project sits before you write it into a three-year plan.
Try it on one GKE cluster before you move the fleet
The cheapest way to find out whether any of this fits your workload is to stand up a single-node llm-d deployment on one GKE cluster and send real traffic at it. Start from the llm-d on GKE TPU guide and get the thing running before you read another architecture diagram. You will learn more in an afternoon of watching a router make decisions than from a week of release notes.
Then send it two workloads with deliberately different latency profiles. One small, interactive request stream that you care about, and one batch job that hammers with long prompts and shared prefixes. If flow control is doing its job you should see the interactive stream hold its time to first token while the batch job queues behind it. If both degrade together, either your policy is not wired up or the fair-share configuration does not match what you assumed. Either way you want to find that out on a cluster you can delete.
The measurement I would actually spend time on is prefix recomputation. Point a real agent loop at the single endpoint, capture GPU or TPU utilization on your replicas, and look at how much compute goes into prefill for prompts that were already seen. Compare that against the same loop with cache-aware routing turned off. On a coding agent or a document Q&A bot with a long system prompt, the difference is usually visible in the first ten minutes of a load test. That number, not a vendor benchmark, is what justifies the migration.
One decision worth making early, before you have invested in either path: whether GKE Inference Gateway’s managed router fits how your team operates, or whether you want to run the llm-d router yourself. The managed service takes the install and lifecycle work off your plate and gives you multi-region routing, but you inherit Google’s release cadence for the routing layer. Self-hosting the router gives you control over versioning and the KV cache index configuration, and it hands you a piece of stateful infrastructure that can fail at 3am.
My own VPS customers are nowhere near this. But the hosting shops moving into GPU resale will hit exactly this fork, and picking the wrong side means a rewrite of the routing layer later. Run the single-cluster test first, and let what you measure decide which side you land on.