Necessary or Sufficient? Evaluating LLM Explanations With Behavioural Evidence
LLMs' stated top decision factors only weakly tracked factors shown to affect outputs. For agent oversight, validate explanations with controlled interventions before using them for escalation.
Across **eight Claude, GPT, and Gemini models**, cited factor rankings had mean Spearman correlations of **0.349–0.580** with measured necessity or sufficiency in two synthetic decision tasks.
If agent monitoring depends on explanations, test each named factor by changing it and by retaining it while removing other mutable information. Do not treat a plausible top-three rationale as a faithful account of the model's decision boundary.
Across **eight Claude, GPT, and Gemini models**, cited factor rankings had mean Spearman correlations of **0.349–0.580** with measured necessity or sufficiency in two synthetic decision tasks. If agent monitoring depends on explanations, test each named factor by changing it and by retaining it while removing other mutable information. Do not treat a plausible top-three rationale as a faithful account of the model's decision boundary. In advisor recommendations, uncited factors outranked a cited factor in **57.6% under necessity and 58.1% under sufficiency**; prompt-monitoring rates were **25.8% and 8.9%**. The method covers individual decisions in synthetic settings, not whole agent trajectories.
This makes intervention-based evaluation concrete for ordinary model explanations: a plausible rationale is neither evidence that a cited factor was necessary nor that it was sufficient. It reinforces prior warnings about treating readable traces or suspicious outputs as causal accounts, while narrowing the result to individual decisions in synthetic tasks rather than whole-agent behavior.