SageMaker Inference adds prefix-aware routing, cutting LLM latency up to 77%

Prefix-aware routing attacks a specific inefficiency in LLM serving: when requests share a common prefix — a system prompt, a RAG context block, an agent's tool schema — routing them to different instances forces each to recompute the same KV cache from scratch. By steering same-prefix requests to the same instance, SageMaker keeps that cache warm and skips redundant prefill.

The measured gains are substantial: up to 77% reduction in P50 time-to-first-token on Llama 3.1 70B, with cache hit rates jumping from roughly 25% to over 80%. For agentic and RAG workloads, where long shared prefixes are the norm, that translates directly into lower latency and higher throughput per GPU — meaning cheaper serving at the same quality.

This is part of a broader AWS inference-efficiency push announced the same week, including SageMaker HyperPod model caching (cold starts from minutes to seconds) and extended 90-minute Lambda timeouts for long-running inference. Together they signal AWS optimizing the unglamorous serving layer while rivals chase model headlines.

Competitively, prefix caching and prefix-aware routing are becoming table stakes — vLLM, SGLang and other serving stacks offer variants — so AWS's edge is integration and managed convenience rather than novel technique. The caveat: gains depend heavily on workload prefix-sharing patterns; low-prefix-overlap traffic won't see 77%. Still, for the increasingly common agent and RAG shapes, this is a real cost lever.