Tsinghua AI Open-Sources AKV, a Cache Engineering Server for LLM Agents on SGLang
Summary
- Researchers from Tsinghua University and CUHK open-sourced AKV, a KV cache engineering serving system built on SGLang for LLM agents.
- AKV enables agents to drop and reposition KV cache entries directly rather than summarizing text history, avoiding expensive prefix re-prefills.
- Across three models and four agent benchmarks, AKV reduced average operating costs by 27.7% while increasing average task scores by 1.7 points.
- The system extends SGLang's radix tree with drop and reposition nodes, utilizing RoPE rotation for key compaction without recomputing hidden states.
- At a concurrency of 16, AKV reduced P95 request latency from 229.90 seconds down to 30.63 seconds on long-context benchmark tasks.
Why it matters
Cache engineering bypasses traditional text summarization limitations in multi-turn LLM agents, dramatically lowering inference costs and reducing P95 latency while preserving important historical context.