Catching Bugs Before They Land: JiTTesting’s Human-on-the-Loop Approach at Full Scale
Meta generates tests that expose behavioral changes in risky diffs, then combines deterministic rules, an LLM ensemble, and aggregate human review to filter likely false positives before code lands.
Meta’s JiTTests target risky diffs and generate executable tests through two workflows: one infers change intent and risks; the other probes behavioral differences. A rules engine and LLM ensemble rank failures before engineers review selected results. In H1 2026, the system caught **74 privacy bugs** before production; **60%** of reported findings were true positives.
For coding agents, consider adding tests that demonstrate a diff’s behavioral change to the review pipeline. Meta’s setup also suggests routing only ranked findings to engineers, with people setting criteria and reviewing outcomes in aggregate.
Meta’s JiTTests target risky diffs and generate executable tests through two workflows: one infers change intent and risks; the other probes behavioral differences. A rules engine and LLM ensemble rank failures before engineers review selected results. In H1 2026, the system caught **74 privacy bugs** before production; **60%** of reported findings were true positives. For coding agents, consider adding tests that demonstrate a diff’s behavioral change to the review pipeline. Meta’s setup also suggests routing only ranked findings to engineers, with people setting criteria and reviewing outcomes in aggregate. The system runs overnight on spare capacity and targets risky changes, so the material does not establish immediate coverage for every diff. The reported results come from Meta’s infrastructure and may depend on its codebase and review process.
JiTTesting provides a large-scale example of risk-ranked, human-reviewed tests that probe a diff's intended or observed behavior, with reported privacy-bug findings and a stated true-positive rate. It gives concrete support for integrating behavioral evidence into agent code review, while its overnight, risk-targeted setup leaves immediate universal coverage unproven.