LLMs Are the Key to Mutation Testing and Better Compliance
Meta's ACH tool has LLMs write realistic bugs (mutants) plus the tests guaranteed to catch them; privacy engineers accepted 73% of generated tests across Facebook, Instagram, and WhatsApp.
**The gist** Meta detailed **ACH (Automated Compliance Hardening)**, which uses LLMs to inject realistic faults — mutants — into code and then generate tests that catch them, steered by plain-text prompts describing the bug to simulate. In an Oct–Dec **2024** deployment across Facebook, Instagram, WhatsApp, and wearables, engineers accepted **73%** of generated tests, with **36%** judged privacy-relevant.
**Why it matters** This is a repeatable pattern for coding-agent workflows: have the model write the **mutant first**, then a test **guaranteed to kill it**, so generated tests provably assert something. If your agent writes tests today, mutation-guided prompting is a concrete upgrade over asking it to cover a file.
**The gist** Meta detailed **ACH (Automated Compliance Hardening)**, which uses LLMs to inject realistic faults — mutants — into code and then generate tests that catch them, steered by plain-text prompts describing the bug to simulate. In an Oct–Dec **2024** deployment across Facebook, Instagram, WhatsApp, and wearables, engineers accepted **73%** of generated tests, with **36%** judged privacy-relevant. **Why it matters** This is a repeatable pattern for coding-agent workflows: have the model write the **mutant first**, then a test **guaranteed to kill it**, so generated tests provably assert something. If your agent writes tests today, mutation-guided prompting is a concrete upgrade over asking it to cover a file. **Watch out** Detecting equivalent mutants remains hard — Meta's detector managed **0.79 precision / 0.47 recall** before preprocessing lifted it to roughly 0.95/0.96 — and results reflect Meta's internal scale. The open **JiTTest challenge** is an admission that just-in-time test generation is unsolved.