SageMaker HyperPod Adds Disaggregated Prefill and Decode for Faster LLM Serving

AWS added Disaggregated Prefill and Decode (DPD) to SageMaker HyperPod, a serving-architecture optimization for production LLM inference. The technique splits the two computationally distinct phases of LLM inference — prefill (processing the input prompt, compute-bound) and decode (generating tokens one at a time, memory-bandwidth-bound) — onto separate, dedicated GPU pools optimized for each phase's characteristics.

The key engineering detail is how the KV cache moves between the pools: it transfers over AWS's Elastic Fabric Adapter (EFA) using GPU-Direct RDMA, minimizing the latency cost of the handoff. By right-sizing hardware to each phase instead of running both on the same undifferentiated GPUs, DPD improves throughput and cost-efficiency for high-volume chat and agentic workloads where prefill and decode demands vary widely.

Disaggregated serving has become a hot infrastructure pattern across the industry (vLLM, TensorRT-LLM, and others have pursued variants), reflecting how much of frontier-AI economics now lives in inference optimization rather than training. The r/LocalLLaMA benchmark of 13 models at 65K-128K context (303 upvotes) this week shows how much practitioners care about long-context serving performance — exactly what DPD targets.

This is part of AWS's broader July serving push alongside DPD's sibling announcements (HyperPod multi-turn RL, MiniMax on Bedrock). What to watch: published throughput/cost benchmarks for DPD versus co-located serving, and whether AWS extends it to more model families and agentic-workload patterns.