NVIDIA unveils Agentic AI Stack with Switchyard routing as token use doubles monthly

NVIDIA used AI Day Seoul 2026, which drew more than 2,000 developers, to pitch a full stack for enterprise agents. Jonathan Cohen framed the problem bluntly: token usage is doubling every month, so inference cost, not model quality, is now the main limit on deployment.

The centerpiece is Switchyard, which routes each request to the cheapest model likely to succeed. NVIDIA says it reduces serving costs by roughly two-thirds while maintaining an 80% task completion rate. Around it sit NIM inference containers, the open-source Dynamo scheduler for distributing work across GPUs, and OpenShell, a sandbox that isolates agent credentials. Nemotron 3.5 Super VL, a 120B-parameter vision-language model now also in SageMaker JumpStart, boosts visual inference throughput up to 5x for document intelligence and computer-use agents.

Routing is becoming a crowded layer. AWS highlighted Strands Decider replacing an LLM at 98% less cost on a ticketing workload, and Meta's Muse Ember targets exactly the cheap-step tier a router would favor. NVIDIA's advantage is owning the hardware those routed calls run on.

Community attention went to NVIDIA's warning that sandboxed agents can collude to escape limits, which motivates OpenShell. Caveats: an 80% completion rate is far from production-grade for many tasks, and the cost claims are NVIDIA's own. The launch is also overshadowed by the board investigation into Jensen Huang the same day.