{
  "schema_version": "1.0",
  "id": "archive:https://engineering.fb.com/2025/09/30/security/llms-are-the-key-to-mutation-testing-and-better-compliance/",
  "slug": "llms-are-the-key-to-mutation-testing-and-better-compliance-02idudw",
  "url": "https://feed7.dev/p/llms-are-the-key-to-mutation-testing-and-better-compliance-02idudw",
  "title": "LLMs Are the Key to Mutation Testing and Better Compliance",
  "why_included": "Meta's ACH tool has LLMs write realistic bugs (mutants) plus the tests guaranteed to catch them; privacy engineers accepted 73% of generated tests across Facebook, Instagram, and WhatsApp.",
  "summary": "**The gist** Meta detailed **ACH (Automated Compliance Hardening)**, which uses LLMs to inject realistic faults — mutants — into code and then generate tests that catch them, steered by plain-text prompts describing the bug to simulate. In an Oct–Dec **2024** deployment across Facebook, Instagram, WhatsApp, and wearables, engineers accepted **73%** of generated tests, with **36%** judged privacy-relevant.",
  "practical_implication": "**Why it matters** This is a repeatable pattern for coding-agent workflows: have the model write the **mutant first**, then a test **guaranteed to kill it**, so generated tests provably assert something. If your agent writes tests today, mutation-guided prompting is a concrete upgrade over asking it to cover a file.",
  "agent_context": "**The gist** Meta detailed **ACH (Automated Compliance Hardening)**, which uses LLMs to inject realistic faults — mutants — into code and then generate tests that catch them, steered by plain-text prompts describing the bug to simulate. In an Oct–Dec **2024** deployment across Facebook, Instagram, WhatsApp, and wearables, engineers accepted **73%** of generated tests, with **36%** judged privacy-relevant.\n\n**Why it matters** This is a repeatable pattern for coding-agent workflows: have the model write the **mutant first**, then a test **guaranteed to kill it**, so generated tests provably assert something. If your agent writes tests today, mutation-guided prompting is a concrete upgrade over asking it to cover a file.\n\n**Watch out** Detecting equivalent mutants remains hard — Meta's detector managed **0.79 precision / 0.47 recall** before preprocessing lifted it to roughly 0.95/0.96 — and results reflect Meta's internal scale. The open **JiTTest challenge** is an admission that just-in-time test generation is unsolved.",
  "source": {
    "name": "Meta AI",
    "url": "https://engineering.fb.com/2025/09/30/security/llms-are-the-key-to-mutation-testing-and-better-compliance/",
    "published_at": null
  },
  "source_class": "blog_post",
  "content_type": "Engineering Post",
  "layer": "agent",
  "domains": [
    "coding",
    "security"
  ],
  "topics": [
    "coding-agents"
  ],
  "verification": {
    "status": "official_source",
    "label": "Official Source",
    "method": "source_feed",
    "verified_at": null
  },
  "uncertainty": [
    "Detecting equivalent mutants remains hard — Meta's detector managed **0.79 precision / 0.47 recall** before preprocessing lifted it to roughly 0.95/0.96 — and results reflect Meta's internal scale. The open **JiTTest challenge** is an admission that just-in-time test generation is unsolved."
  ],
  "lifecycle": "Current",
  "published_at": null,
  "modified_at": null,
  "supersedes": [],
  "expires_at": null,
  "formats": {
    "html": "https://feed7.dev/p/llms-are-the-key-to-mutation-testing-and-better-compliance-02idudw",
    "json": "https://feed7.dev/p/llms-are-the-key-to-mutation-testing-and-better-compliance-02idudw.json",
    "markdown": "https://feed7.dev/p/llms-are-the-key-to-mutation-testing-and-better-compliance-02idudw.md"
  }
}