AWS launches SageMaker HyperPod Inference Gateway with GPU-aware routing

AWS introduced SageMaker HyperPod Inference Gateway, a managed, Kubernetes-native routing layer for LLM inference deployed as an add-on for Amazon EKS. It uses real-time GPU signals — utilization and load — to route each inference request to the best-suited pod, rather than round-robin or naive load balancing. AWS reports up to 82% reduction in first-token latency and 97–98% reductions in P95/P99 tail latency for Llama-3.1-8B and Qwen3-32B, plus throughput gains of up to 50%, all without changes to model servers or client applications.

The tail-latency numbers are the key claim. P95/P99 latency is what determines whether an agentic or interactive workload feels responsive under load, and GPU-aware routing directly attacks the queuing and hot-pod problems that produce long tails. Because it requires no model-server or client changes, adoption friction is low for teams already on EKS.

The launch is part of AWS's dense inference-infrastructure week, which also included a SageMaker Inference year-in-review noting 13 inference launches in 2026 (tiered KV caching, capacity-aware instance pools, disaggregated prefill and decode) and the new AgentCore Runtime. Together they signal AWS competing on the serving layer — latency, cost, and routing — rather than just model availability.

Developers welcomed the tail-latency gains as meaningful for production agentic workloads that need low, predictable latency. The caveats: the benchmark figures are AWS-reported on specific models (Llama-3.1-8B, Qwen3-32B), and real-world gains depend on traffic patterns, model mix, and cluster size. Still, GPU-aware routing as a drop-in EKS add-on is a practical, incremental win for teams already running self-managed inference at scale.