Meta runs LLM-generated Just-in-Time catching tests at full production scale

Meta's engineering blog reports that its Catching JiTTests system now runs at full production scale. Rather than maintaining a permanent test suite for every change, the system uses LLMs to generate targeted tests just in time for a specific diff, aiming to catch regressions before code merges.

The human-on-the-loop design matters. Engineers are not asked to approve every generated test. Instead, the system surfaces likely bugs, and humans step in when a test flags a real problem. That keeps the review burden manageable at Meta's scale while still using LLMs to explore edge cases human-written tests miss. The full methodology appears in the paper 'Just-in-Time Catching Test Generation at Meta'.

The work is part of a broader shift toward AI-native software quality. AWS showed Epsilon running plain-English UI tests on Nova Act and Kiro powering parity-checked PHP-to-Java migrations this week, and a viral r/artificial thread argued that LLMs find mistakes in papers better than trained human reviewers. Meta's angle is defensive: as more code is AI-written, AI-generated tests act as the counterweight.

Caveats: the blog post focuses on deployment rather than independent metrics, and generated tests can be flaky or overfit to the diff. Watch for whether Meta open-sources the tooling and for reported false-positive rates.