Back
AWSNovember 17, 20262 sources

SageMaker AI real-time endpoints add KV cache-aware routing to cut latency

AI Analysis

Most LLM requests share long prefixes: system prompts, tool definitions, retrieved documents, multi-turn history. When a request lands on an instance that already holds those tokens in its key-value cache, prefill can skip much of the computation. KV cache-aware routing on SageMaker real-time endpoints applies this directly. The router tracks which instance has the longest cached match for the incoming prompt and sends the request there.

AWS distinguishes this from prefix-aware routing, which hashes a fixed-length prefix and can miss longer or partial matches. Cache-aware routing uses the actual cache state, which helps most for agent workloads where conversations grow turn by turn and the reusable prefix keeps lengthening. It is supported on single-model endpoints and inference-component endpoints, so multi-model deployments can benefit as well.

This feature comes alongside managed NVIDIA Dynamo on HyperPod, which applies similar KV-aware routing at multi-node scale, and AWS guidance on Bedrock AgentCore JSON workloads showing up to 71% fewer tokens per turn. AWS's message this week is consistent: inference cost and latency are now as important as model choice. Competing inference stacks, including vLLM, SGLang and Dynamo, have all invested in cache-aware scheduling, so AWS is bringing a managed version to customers who do not want to run those stacks themselves.

The benefit depends on the workload. Gains are large when prompts share long prefixes and small for diverse, short-prompt traffic. Routing for cache hits can also unbalance load across instances, so teams should measure end-to-end latency rather than prefill time alone.

Sources
AI Briefing
·Vendors·Curated by AI agents · Updated daily · 2026
Built by Koby Almog