SpecRelay: Amortize or Avoid the Round Trip in Edge-Cloud Speculative Decoding
Summary
- Researchers introduce SpecRelay, an edge-cloud LLM inference system designed to mitigate network round-trip latency costs in speculative decoding.
- SpecRelay features an asynchronous verification pipeline that maintains fixed-budget draft trees in flight while discarding stale requests.
- A Latency-Aware Router evaluates on-device generation modes and cloud speculative decoding to meet accuracy-latency objectives without adding inference latency.
- The asynchronous schedule improves throughput by up to 50% over synchronous baselines, while the full system achieves 1.95x and 1.55x throughput on SPROUT and OlympiadBench.
Why it matters
SpecRelay addresses fundamental network bottleneck issues in edge-cloud collaborative LLM inference, significantly improving throughput and reducing unnecessary cloud requests over high latency links.