LACUNA: A Testbed for Evaluating Localization Precision for LLM Unlearning
First unlearning testbed with ground-truth parameter localization: injects synthetic PII into known weights of OLMo 1B/7B models, showing current methods hide rather than erase and fall to resurfacing attacks.
**The gist** LACUNA injects synthetic PII into predetermined parameters of **OLMo-based 1B and 7B** models via masked continual pretraining, giving unlearning research its **first ground-truth parameter-level** testbed instead of output-only benchmarks.
**Why it matters** If you count on unlearning to strip sensitive data from a model, output-level evals can lie: methods that look clean behaviorally were **highly imprecise** at the weight level and vulnerable to **resurfacing attacks**. When localization is right, even **simple gradient-based erasure** holds up — precision, not the removal algorithm, is the lever.
**The gist** LACUNA injects synthetic PII into predetermined parameters of **OLMo-based 1B and 7B** models via masked continual pretraining, giving unlearning research its **first ground-truth parameter-level** testbed instead of output-only benchmarks. **Why it matters** If you count on unlearning to strip sensitive data from a model, output-level evals can lie: methods that look clean behaviorally were **highly imprecise** at the weight level and vulnerable to **resurfacing attacks**. When localization is right, even **simple gradient-based erasure** holds up — precision, not the removal algorithm, is the lever. **Watch out** The PII is **synthetic** and planted into known weights, a far cleaner setup than organically memorized data; whether precise localization is achievable in real pretrained models remains the open question.