arXivPaperNeeds Review

Surprisal Theory is Tautological (without Rational Grounding)

The paper argues that unconstrained surprisal can fit any non-negative processing-difficulty pattern, so corpus fit alone cannot make claims about human language processing falsifiable.

arXiv · Jul 23, 2026

Open Source Open Markdown Open JSON

Source Summary

The paper shows that, under mild conditions, **any non-negative difficulty measure** can be represented as an affine function of surprisal under some language model. Without constraining that model, surprisal theory therefore makes no falsifiable prediction.

Practical Implication

Builders evaluating language models against human behavior should not treat better corpus likelihood as evidence of better cognitive fidelity. The proposed remedy is to derive the model from independent assumptions such as memory limits or processing goals.

Agent-Ready Context

The paper shows that, under mild conditions, **any non-negative difficulty measure** can be represented as an affine function of surprisal under some language model. Without constraining that model, surprisal theory therefore makes no falsifiable prediction.

Builders evaluating language models against human behavior should not treat better corpus likelihood as evidence of better cognitive fidelity. The proposed remedy is to derive the model from independent assumptions such as memory limits or processing goals.

This is a theoretical argument, not a new agent evaluation or empirical benchmark. Its practical force depends on whether researchers can specify independently motivated comprehender models that produce testable predictions.

Context Map

benchmarkresearch#benchmark-integrity

Uncertainty

This is a theoretical argument, not a new agent evaluation or empirical benchmark. Its practical force depends on whether researchers can specify independently motivated comprehender models that produce testable predictions.