SpecRelay: Amortize or Avoid the Round Trip in Edge-Cloud Speculative Decoding

Impact 8 arXiv cs.AI

Summary

  • Researchers introduce SpecRelay, an edge-cloud LLM inference system designed to mitigate network round-trip latency costs in speculative decoding.
  • SpecRelay features an asynchronous verification pipeline that maintains fixed-budget draft trees in flight while discarding stale requests.
  • A Latency-Aware Router evaluates on-device generation modes and cloud speculative decoding to meet accuracy-latency objectives without adding inference latency.
  • The asynchronous schedule improves throughput by up to 50% over synchronous baselines, while the full system achieves 1.95x and 1.55x throughput on SPROUT and OlympiadBench.

Why it matters

SpecRelay addresses fundamental network bottleneck issues in edge-cloud collaborative LLM inference, significantly improving throughput and reducing unnecessary cloud requests over high latency links.

Who should care