Intro
If you are running LLM agents that process long tool chains, you likely think about context engineering in terms of text. You truncate history, summarize older turns, or strip out verbose tool outputs to keep the prompt under your token limit. That is standard practice. It works, but it has a hidden cost. When you edit the text prefix of a request, you invalidate every key-value (KV) state that was computed for the subsequent suffix. The inference engine has no choice but to re-prefill that entire suffix. It is like rewriting the first page of a book and then forcing the printer to reprint every remaining page because the ink sequence changed.
Cache engineering takes a different approach. It leaves the text trajectory completely untouched. Instead of rewriting history, it manipulates the KV cache state directly. The agent decides which specific KV entries to drop from future attention calculations and which to keep. The retained entries are reused without recomputation. They still carry information from the dropped messages, but the attention heads no longer waste cycles looking at junk data. This distinction matters because the Tsinghua AKV system operates at this level. It is not a text summarizer. It is a serving system that allows agents to surgically remove noise from the cache while preserving the signal in the remaining keys.
The core claim from the Tsinghua AKV paper is compelling for anyone paying for API credits. Across three open models and four agent benchmarks, average cost falls by 27.7% while average task scores rise by 1.7 points. This is not just a cost cut. The improvement in accuracy is a side effect of the system stopping the model from attending to irrelevant, dropped history. In the paper’s cost analysis, KV cache reads made up 82.4% of total API cost. By reducing the number of tokens that require a full cache hit or read, you directly attack that largest expense line item.
This work is built on SGLang, a high-performance inference engine. The entire server code is released under Apache-2.0 licensing, which means you can fork it, deploy it, and integrate it into your own agent use without worrying about proprietary lock-in. The tests were run on NVIDIA GPUs, specifically a cluster of eight A800-80GB cards operating in BF16 precision. If you are running a smaller setup, the quick start guides suggest you can serve the GPT-OSS-120B model on just two GPUs using tensor parallelism of 2. The system is stateless from the client’s perspective, using an OpenAI-compatible interface, which makes it easy to slot into existing LangChain or LlamaIndex workflows.
I have tested various KV cache eviction policies on my own VPS hosting clients who run local LLMs. The biggest hurdle is always the complexity of state management. If your cache state gets out of sync with your text history, you get garbled outputs. AKV addresses this by extending SGLang’s radix tree with specific drop and reposition nodes. It is a heavy engineering lift, but the payoff is clear: you stop paying for attention over garbage. For a detailed breakdown of how this compares to traditional sliding window approaches, you can look at the full analysis in this report on Tsinghua AKV (opens in new tab). The main takeaway for now is that you no longer have to choose between keeping your context window clean and keeping your latency low.
Background: How KV Cache Management Works
To understand why Tsinghua AKV works, you have to first understand the mechanics of what it is doing to the memory of the model. Standard LLM inference relies on the Key-Value (KV) cache to avoid recalculating the attention mechanisms for every previous token. This is where the cost and latency of long-agent runs comes from. The cache grows linearly with the conversation history. In the paper’s cost analysis, KV cache reads accounted for 82.4% of the total API cost. That is a huge portion of your bill that is essentially paying for the model to “remember” things it does not actually need to remember for the next step.
AKV introduces three core operations that manipulate this state directly. The first is Append, which is the standard behavior. When the model generates a new token or the user sends a new message, those entries are added to the cache. This is unremarkable and matches what SGLang does natively.
The second operation is Drop. This is where the engineering gets interesting. Instead of summarizing the text history (which requires running the model again to compress the information) or waiting for the entire window to shift, Drop removes specific KV entries from future attention layers. The positions of the remaining keys stay the same. They do not shift. This prevents the “retained suffix re-prefilling” problem. If you delete the second paragraph of your history, the third paragraph does not move up to position two in the internal matrix; it stays where it is. Only the attention mechanism stops looking at the deleted paragraph. This saves compute without invalidating the rest of the state.
The third operation is Repos, or repositioning. This is the critical piece that makes dropping safe for performance. When you drop a key, there is a “gap” in the logical sequence. RoPE (Rotary Position Embedding) relies on relative positions. If you just leave a gap, the model might get confused about the distance between the remaining tokens. AKV rewrites the retained keys to positions 0 through r-1 using a RoPE rotation. Crucially, it does not recompute the hidden states. It just mathematically rotates the existing keys to their new positional indices. The values stay exactly as they are. In their ablation tests, adding Repos raised GPT-OSS-120B’s BrowseComp-Plus pass rate from 55% to 62%. Without it, dropping tokens introduced positional noise that degraded accuracy.
The hard part is not the math; it is the serving infrastructure. If two different agents have the same text history but different drop histories, their cache states are different. A naive prefix-matching system would think they can share the same cache block, which would result in one agent seeing the other’s dropped context. AKV solves this by modifying SGLang’s radix tree. It adds specific “drop” and “reposition” nodes to the tree structure. This ensures that the system knows exactly which state is valid for which request. It frees the dropped KV memory only when no active request still needs it, preventing premature garbage collection of useful data. Segment-batch attention then allows the system to restore evicted states in a single prefill pass if a request needs to revisit a dropped section.
What’s Happening Now
The numbers from the paper back up the theoretical savings, but they also highlight where this technology still feels like a work in progress. I pulled the performance data from Tsinghua AI Open-Sources AKV (opens in new tab) to see how the three models held up against standard baselines. The headline is clear: across GPT-OSS-120B, MiniMax-M2.7, and MiroThinker-1.7-Mini, AKV cut average costs by 27.7% while bumping up average task scores by 1.7 points. That is not a trivial gain for an infrastructure layer. Usually, when you try to squeeze more performance out of a model, you pay a tax in accuracy or latency. Here, both improved slightly.
Looking at GPT-OSS-120B on SWE-bench Verified, the improvement is stark. The cost per task dropped from 20.0 cents to 9.7 cents. More importantly, the pass rate climbed from 40.0 to 43.2. For a host running large-scale coding agents, that difference in pass rate means fewer retry loops and less human intervention. Fewer retries directly translate to lower GPU hours and reduced egress costs.
However, the data is not uniformly positive. MiniMax-M2.7 on BrowseComp-Plus presents a cautionary tale. Its score was 58.9, which is actually 3.4 points lower than the Text Engineering baseline of 62.3. This suggests that for some models, dropping context is harder than summarizing it. The model relies heavily on specific visual or browsing cues that get lost when you evict early tool responses. If your agents are doing heavy web browsing rather than coding or logic puzzles, this specific model might still benefit more from traditional summarization.
The latency improvements are where the engineering work really shines. The team tested SWE-bench Verified tasks at high concurrency. At 16 concurrent requests, the P95 request latency for the Standard baseline hit 229.90 seconds. That is nearly four minutes for a single response. AKV dropped that to 30.63 seconds. For anyone running interactive agent dashboards or real-time chat interfaces, that difference is the gap between a usable product and a loading spinner that makes users leave. The system achieved this by maintaining a higher input throughput, specifically 1.53x faster than the standard method.
In my own VPS hosting business, I have seen similar patterns when optimizing database query caches. Sometimes the most effective optimization is not faster computing, but smarter state management. AKV proves that for LLM agents, knowing what to forget is just as valuable as knowing what to remember. The trade-off here is operational complexity. You are no longer just sending text; you are managing a stateful vector of memory. But if the latency drop from four minutes to thirty seconds holds up in your specific use case, the complexity is worth the engineering effort.
What Does the Tsinghua AKV API Actually Look Like?
The interface designed by the Tsinghua AKV team feels familiar to anyone who has built agents on top of OpenAI-compatible endpoints. You do not need to learn a new SDK. You send the full message history exactly as you would today, but you append two optional fields to the request payload: drop_message and reposition.
The drop_message field maps a trigger message index to the specific messages you want to evict. For example, if you want to discard the raw JSON response from a tool call at index 4, you specify that index here. The reposition field lists the indices after which position compaction happens. This is where the “Repos” operation from the theory section becomes concrete. It tells the server to rewrite the remaining keys to positions 0 through r-1 using RoPE rotation. The values stay untouched, so the system never recomputes hidden states for the retained text.
Building these fields manually is tedious, so AKV includes a CachePolicy structure. You define a trigger (like “every 10 turns”), a selector (like “keep the last 12 tool responses”), and a reposition rule. The policy engine generates the exact drop_message and reposition lists for each step. There are three built-in modes: turn-based, token-based, and model-based. The model-based mode is the most interesting because it exposes cache control as a tool.
Think of it like adding a drop_context tool to your agent’s function list. The model can decide to call it autonomously. In tests, GPT-OSS-120B saw its accuracy jump from 52.1 to 61.0 on a subset of tasks when it could prune its own memory. However, MiniMax-M2.7 saw its cost rise slightly when given this autonomy, suggesting the model sometimes hoards context it does not need.
Here is a practical analogy. Imagine you are cleaning out a garage. Traditional context engineering is like moving all your old boxes to a self-storage unit. You have to pay to move them, and when you need something back, you have to drive there and retrieve it (re-prefill). Cache engineering is like leaving the boxes in the garage but stacking them neatly in a corner you know you will not touch. You can easily grab new items (append), and when the garage gets too full, you move the stacked boxes to the back wall (reposition) without unpacking them. You only throw away what you explicitly mark for disposal (drop).
The statelessness of the API is a major win for deployment. You do not need sticky sessions or custom middleware. Your load balancer can route requests to any node, and the radix tree handles the state integrity. Just keep in mind that if you switch models or drastically change your system prompt, your cache state is essentially dead. You have to clear the prefix cache. Otherwise, you risk serving hallucinated continuations based on stale KV entries from a different agent run.
As noted in the primary source (opens in new tab), the server code is Apache-2.0 licensed, so you can fork it to add custom policy logic. Start with the turn-based policy that keeps the last 12 tool responses. It is the safest starting point. Then monitor your eviction rates. If you see too many “misses” where the model has to re-read dropped context, you are dropping too aggressively. If your latency is high but accuracy is fine, you are holding on to too much. Tune the selector until the numbers match your cost targets.
What It Means in Practice
For a hosting business, the math behind cache engineering changes how you think about API spend for agent workflows. In the paper’s cost breakdown, KV cache reads accounted for 82.4% of total API costs in standard setups. That is the part most operators ignore. They look at token counts, but they do not account for the repeated reads from a bloated cache. Tsinghua AKV targets exactly that inefficiency. The average cost drop of 27.7% is not just a nice-to-have metric. It is the difference between a profitable agent endpoint and one that burns through margins.
The performance data tells a more nuanced story. Look at GPT-OSS-120B on SWE-bench Verified. Standard serving cost 20.0 cents per task and scored 40.0. AKV cut that to 9.7 cents while pushing the score to 43.2. You get a 51.5% cost reduction per task and a higher pass rate. That is a strong result. But the data also shows where the approach struggles. MiniMax-M2.7 on BrowseComp-Plus scored 58.9 with AKV, lagging behind Text Engineering’s 62.3. In that specific scenario, the latency savings did not compensate for the accuracy loss. If your agent workload resembles that kind of deep browsing, you need to test carefully before rolling this into production.
The serving impact is where things get interesting for infrastructure engineers. Under concurrency 16 on GPT-OSS-120B, P95 request latency dropped from 229.90 seconds to 30.63 seconds. That is a 1.51x improvement, but the absolute numbers matter more. You are moving from a 3.8-minute tail latency to a 30-second tail latency. Your users will notice that. The input throughput improves by 1.53x, and full-attention time per decode step falls from 7.082 ms to 2.849 ms. For high-concurrency deployments, those reductions translate directly into how many parallel agent runs a single GPU can handle before you hit saturation.
One subtle point trips up operators: identical tokens can produce different cache states depending on the operation history that preceded them. Plain prefix matching breaks here. The radix tree extension in SGLang that AKV introduces exists specifically to prevent wrong-state reuse. If you are building your own cache layer on top of a standard inference server, you do not have this safety net. The drop and reposition nodes track which KV entries are logically valid for a given request. Without that, a stale prefix match could feed a hallucination into the next generation.
In practice, the 21.9% savings over Text Engineering on long-horizon tasks is the number to track. Text Engineering summarizes at 48K tokens, which is aggressive enough to lose critical middle-round details. AKV holds the cache intact and only repositions it. On your own boxes, that means you can run longer agent trajectories without hitting the context wall where every old tool response becomes dead weight. The cost curve stays flatter for longer. Monitor your KV read ratios against your billing estimates. If the 82.4% component reduction is not showing up in your numbers, your policy is likely holding onto entries it should be dropping. Tune the selector. Keep the cache lean.
What To Expect Next
I expect the SGLang integration to expand well beyond the paper’s eight A800-80GB BF16 test setup within the next two quarters. The current benchmark suite used a fixed cluster size, but production agent uses rarely run on such clean, isolated GPU pools. You will likely see community forks targeting A100s, H100s, and mixed-precision configurations that the original authors did not validate. I do not know how the segment-batch attention mechanism holds up at scale on heterogeneous clusters where cache state is sharded across nodes rather than kept local. That is a significant gap. The radix tree logic assumes a relatively coherent memory space. Once you start sharding KV caches across a distributed inference layer, the drop and reposition nodes need to synchronize state across peers. I anticipate that synchronization overhead becoming the new bottleneck, not the compute itself.
Policy adoption will shift from static rules to dynamic, model-driven decision loops. The paper shows that MiniMax-M2.7 can refine its own cache policies on held-out tasks, cutting costs by 17.8% compared to standard summarization. That is a strong signal that hardcoded, turn-based policies will become secondary. I expect production systems to start exposing drop_context as a first-class tool for the model itself, rather than relying on a pre-defined selector. The risk here is latency. When the model decides when to drop cache, it adds a decision step to every turn. If that decision is slow, you lose the throughput gains you just achieved. We will need to see how inference engines handle a control plane that is itself an LLM call. For now, I am betting on a hybrid approach where a lightweight heuristic handles the common 80% of cases, and the model-based policy kicks in only for complex, long-horizon tasks where the accuracy gains justify the extra inference cost.
We will also see refinement in how eviction states are handled during prefill. The current segment-batch mechanism restores evicted states in a single pass, which is efficient for moderate eviction sizes. As agent contexts grow to millions of tokens, single-pass restoration may become a memory spike risk. I expect future versions to introduce chunked restoration or lazy loading of repositioned keys, where the cache only reconstructs the specific slots needed for the next attention head. This is a predictable engineering evolution. It mirrors how we moved from full re-prefill to prefix caching in earlier SGLang releases. The segment-batch trick is clever, but it assumes the GPU memory has enough headroom to hold the entire compacted state plus the active heads. That assumption will break on dense, high-batch deployments.
For site owners running agent-based support bots or code assistants, the next version will likely include a visual dashboard for cache state inspection. Right now, you are monitoring KV read ratios through raw logs or custom metrics. A dashboard that shows which messages were dropped and why will make tuning the policy much less of a guessing game. You will be able to see if your selector is dropping critical system prompts or retaining junk tool outputs. That visibility is essential for keeping costs under control as context windows expand. Without it, you are flying blind.
The Tsinghua AKV project (opens in new tab) is moving from a research artifact to an operational tool. The hard part now is making it solid enough for messy, real-world traffic. Expect the first few months to be about stability fixes and memory leak prevention in the radix tree modifications. The performance gains are real, but they are only useful if the server does not crash under sustained load. Watch the GitHub issues. The first production-ready release will be the one that has survived a month of 24/7 traffic without a single cache corruption event. That is the metric that matters.
Closing: Running Tsinghua AKV On Your Own Cluster
Getting this running on your own hardware is straightforward if you have the necessary NVIDIA GPUs. The quick start guide suggests serving GPT-OSS-120B on two GPUs with tensor parallelism set to 2. The command looks something like this:
python -m sglang.launch_server \
--model-path gpt-oss-120b \
--tensor-parallel-size 2 \
--enable-kv-cache-engineering \
--host 0.0.0.0 \
--port 8000
I added the --enable-kv-cache-engineering flag to mirror the experimental branch of SGLang that supports AKV operations. If your SGLang version is the standard stable release, you will need to compile from the AKV fork or patch the radix tree node structures manually. The latter is risky. I have spent too many hours debugging segfaults caused by misaligned memory pointers in custom inference engines to recommend that approach unless you are absolutely sure of what you are doing. Use the official fork for your first tests.
Once the server is up, do not just trust the benchmark numbers from the paper. You need to verify the cost reduction on your specific traffic. The key metric to watch is the KV cache read ratio. In the Tsinghua tests, KV cache reads made up 82.4% of total API cost when compared to standard re-prefilling. If you are not seeing a similar drop in your cache hit rates, the drop_message and reposition fields in your API calls are likely not aligning with the actual token boundaries of your agent history.
Run a side-by-side test. Spin up two instances. One standard SGLang, one with AKV. Send the same 50 SWE-bench Verified tasks to both. Record the total tokens processed versus the tokens served from cache. The difference should show you the tangible savings. If you are paying by the token on your hosting bills, this number translates directly to dollars. A 27% reduction in compute intensity means you can handle more concurrent users on the same hardware, or scale down your fleet and keep the same performance.
Check your logs for reposition events. You should see a spike in these operations right after tool responses are generated. If you see them happening on system prompts, your policy selector is wrong. That is a bug, not a feature. Fix the selector before you let this run in production. The model will thank you, and your invoices will thank you more.