Buoyant Releases Kinference: An Open Source Inference Control Plane for Kubernetes and vLLM
Summary
- Buoyant has released Kinference v0.1.0-alpha.7, an open-source inference control plane for Kubernetes and vLLM.
- Kinference sits in front of vLLM worker fleets to manage routing policies, token rate limits, KV-cache-aware load balancing, and per-tenant metrics.
- The system uses declarative Kubernetes Custom Resource Definitions (CRDs) for governance tasks like model aliasing, PII routing, and cloud bursting.
- In self-reported benchmarks using shared-prefix traffic, Kinference reduced median time-to-first-token (TTFT) by roughly 95% compared to baseline Kubernetes Service routing.
Why it matters
Kinference helps engineering teams operating multi-tenant LLM services on Kubernetes to enforce routing governance and dramatically improve TTFT latency for cache-heavy workloads.