Summary
- Prefix caching helps LLM servers reuse KV states and accelerate prefill computation, but limited memory makes the cache eviction policy critical.
- Different production traffic types favor different eviction strategies, with agentic steps favoring recency and popular prompts or long conversations favoring frequency.
- Existing fixed policies like LRU or S3FIFO fail to dominate across all mixes, and workload parameters can easily flip the performance winner.
- WAKMan is introduced as a workload-aware KV cache manager that dynamically selects eviction policies from live traffic without requiring prompt reads or explicit request types.
- WAKMan improves cache hit rates over LRU by 5 percentage points on average while maintaining the lowest mean and worst-case optimality gap.
Why it matters
Efficient KV cache management directly reduces LLM serving latency and infrastructure costs. WAKMan offers a dynamic approach to handle mixed production traffic without manual policy tuning.