# Elucidating the Design Space of LM Data Attribution

Source: [arXiv](https://arxiv.org/abs/2610.40655v1)  
Feed7 permalink: https://feed7.dev/p/2610-40655v1-1dutggm  
Published: 2026-10-29T17:59:53.000Z  
Trust: Needs Review (needs_review)

## Why Included

CATT isolates factors that affect language-model data attribution. The paper finds that focusing gradient scores on key tokens improves recall over document-wide averaging.

## Source Summary

The paper introduces **CATT**, a benchmark that fine-tunes models on fictitious knowledge so attribution has clear ground truth. Its controlled tests vary dataset properties such as redundancy and lexical distractors, both of which reduce attribution performance. Negative log-odds gradients plus top-token aggregation improve recall by **35–72 percentage points** across tested baselines.

## Practical Implication

If you evaluate where model outputs came from, test attribution under controlled changes to the data and inspect which tokens carry evidence. Document-wide averaging may obscure the signal; the authors also report up to **4.9×** better counterfactual retraining performance on FTRACE-TREX with the gradient correction.

## Agent-Ready Context

The paper introduces **CATT**, a benchmark that fine-tunes models on fictitious knowledge so attribution has clear ground truth. Its controlled tests vary dataset properties such as redundancy and lexical distractors, both of which reduce attribution performance. Negative log-odds gradients plus top-token aggregation improve recall by **35–72 percentage points** across tested baselines.

If you evaluate where model outputs came from, test attribution under controlled changes to the data and inspect which tokens carry evidence. Document-wide averaging may obscure the signal; the authors also report up to **4.9×** better counterfactual retraining performance on FTRACE-TREX with the gradient correction.

These results come from diagnostic benchmarks and do not establish how well the approach works across all training corpora or attribution settings. The paper studies evaluation and design choices rather than proposing a new estimator.

## Connected Context

Feed7 judgment across 966 accumulated Signals:

CATT turns data attribution into a controlled diagnostic problem: known fictitious facts make ground truth testable, while redundancy and lexical distractors expose weaknesses in attribution methods. Its results support careful token-level evidence analysis, but remain benchmark-specific and do not establish performance across real training corpora or attribution settings.

- [Measuring Task-Agnostic Training Data Influence Across Language Model Pretraining](https://feed7.dev/p/2608-13515v1-1cot0y6) — Both study which training examples influence model behavior; CATT tests attribution against controlled ground truth and data perturbations, while the other work tracks task-agnostic influence across checkpoints without establishing downstream capability.
- [What Improves During Pretraining? Tracking Knowledge Acquisition with Point-in-Time Models](https://feed7.dev/p/2610-37631v1-0xx05j6) — Both use controlled views of training data to interpret model knowledge: CATT evaluates whether data sources can be attributed, while the point-in-time study relates temporal variability in knowledge to later acquisition.
- [Molecular Déjà Vu: Digit-Level Retrieval of Published Values in Frontier Language Models](https://feed7.dev/p/2609-05381v1-1y2qhyz) — Both address how apparent model behavior can be confounded by training exposure; CATT evaluates source attribution under controlled distractors, while Molecular Déjà Vu probes recall of published benchmark values.

## Context Map

- Layer: benchmark
- Domains: research, data
- Topics: benchmark-integrity

## Uncertainty

- These results come from diagnostic benchmarks and do not establish how well the approach works across all training corpora or attribution settings. The paper studies evaluation and design choices rather than proposing a new estimator.

## Agent Instruction

Use this item as source-backed context. Do not invent claims beyond the linked source. If this item conflicts with another source, call out the conflict.