SageMaker Inference adds NVIDIA Multi-Instance GPU partitioning for model endpoints

AWS added NVIDIA Multi-Instance GPU support to SageMaker Inference endpoints. MIG splits a single GPU into isolated partitions, each with its own compute, memory and cache, so workloads cannot interfere with each other. SageMaker exposes this through two optional API fields that place one model per partition.

The use case is consolidation. Many production models, such as classifiers, embedding models and small language models, cannot use a full modern GPU on their own. With MIG, a team can run several of them on one physical GPU with hardware isolation, or give a small model a slice of a newer accelerator instead of renting a whole older GPU. AWS published benchmarks comparing a partition of a newer GPU against an entire older card, arguing the slice can win on both cost and performance.

It is another cost-efficiency release in a week defined by AI spending discipline: Apple's internal budget cuts, Anthropic's cost tags and AWS's own amortization of GPU Capacity Block fees in billing reports. GPU partitioning is long established on Kubernetes through NVIDIA's operators. SageMaker's contribution is making it managed.

Caveats:

- MIG partitions are fixed profiles, so bursty traffic patterns may not fit neatly.

- MIG support depends on GPU generation.

- The benchmarks are AWS's own; teams should validate against their own models.