ElastiCache for Valkey adds GA distributed KV caching, cutting self-hosted LLM costs up to 40%

AWS has made distributed KV caching for self-hosted large language models generally available on Amazon ElastiCache for Valkey. The KV cache stores the attention state a model has already computed for a prompt prefix, such as a system prompt, a long document or an agent's history, so it does not have to be recomputed. Moving that cache out of individual GPU memory into a shared distributed tier lets any inference node reuse prefixes another node already processed.
AWS claims up to 40% lower inference cost and up to 80% faster time to first token. The launch post includes benchmarks across workload types and introduces KVTrace, an open-source tool that analyzes real traffic to estimate how much a given deployment would save before adopting it. Gains are largest for workloads with long shared prefixes, such as retrieval-augmented generation (RAG) over common corpora and multi-turn agents.
Competitively, this brings to self-hosted models what the API providers already sell: Google's Gemini 4 Neon charges $0.15 for cached input, a 95% discount. Teams running open-weight models on their own infrastructure have had to build that themselves with vLLM- or LMCache-style setups. Packaging it as a managed AWS service lowers the bar to entry.
Caveats: the 'up to' figures depend heavily on how often prefixes repeat. Workloads with unique prompts will see little benefit, which is why AWS ships KVTrace. Watch how it integrates with SageMaker and EKS inference stacks.