Together Compiler: Continual Compilation for AI Inference
Summary
- Researchers introduce Together Compiler (TKCC), a continual agent loop designed to address continual optimization challenges in AI inference.
- TKCC uses a Tiered Persistent Engine to decouple engine launch overheads from forward pass computation and employs a cost model for parallelism configurations.
- The compiler outperforms hand-written Megakernels on Llama 1B by up to 1.7x and open-source engines on frontier models by up to 2.6x higher tokens per second on the InferenceX benchmark.
- TKCC achieves state-of-the-art batch size 1 performance and currently serves trillions of tokens across production fleets.
Why it matters
Continual compilation and optimization agents can significantly improve inference throughput and adapt to changing AI workloads and frontier models across hardware platforms.