Amazon SageMaker AI real-time endpoints now support KV cache aware routing
Summary
- Amazon SageMaker AI real-time endpoints now support KV cache-aware routing.
- This strategy routes each request to the instance holding the longest cached match for its prompt.
- Developers can learn how it works, when to use it compared with prefix-aware routing, and how to enable, invoke, and verify it on single model and inference component endpoints.
Why it matters
KV cache-aware routing optimizes inference performance and resource utilization for large language models by directing requests to instances with existing prompt matches.