Elucidating the Design Space of LM Data Attribution
CATT isolates factors that affect language-model data attribution. The paper finds that focusing gradient scores on key tokens improves recall over document-wide averaging.
The paper introduces **CATT**, a benchmark that fine-tunes models on fictitious knowledge so attribution has clear ground truth. Its controlled tests vary dataset properties such as redundancy and lexical distractors, both of which reduce attribution performance. Negative log-odds gradients plus top-token aggregation improve recall by **35–72 percentage points** across tested baselines.
If you evaluate where model outputs came from, test attribution under controlled changes to the data and inspect which tokens carry evidence. Document-wide averaging may obscure the signal; the authors also report up to **4.9×** better counterfactual retraining performance on FTRACE-TREX with the gradient correction.
The paper introduces **CATT**, a benchmark that fine-tunes models on fictitious knowledge so attribution has clear ground truth. Its controlled tests vary dataset properties such as redundancy and lexical distractors, both of which reduce attribution performance. Negative log-odds gradients plus top-token aggregation improve recall by **35–72 percentage points** across tested baselines. If you evaluate where model outputs came from, test attribution under controlled changes to the data and inspect which tokens carry evidence. Document-wide averaging may obscure the signal; the authors also report up to **4.9×** better counterfactual retraining performance on FTRACE-TREX with the gradient correction. These results come from diagnostic benchmarks and do not establish how well the approach works across all training corpora or attribution settings. The paper studies evaluation and design choices rather than proposing a new estimator.
CATT turns data attribution into a controlled diagnostic problem: known fictitious facts make ground truth testable, while redundancy and lexical distractors expose weaknesses in attribution methods. Its results support careful token-level evidence analysis, but remain benchmark-specific and do not establish performance across real training corpora or attribution settings.