Stop Evaluating Models Like It's the 50s - Alejandro Vidal, Mindmakers

Raw accuracy weights every question equally. Item response theory estimates item difficulty and discrimination, produces uncertainty intervals, and can expose mislabeled, uninformative, or negatively correlated questions.

Calibrate internal agent evals before paying to run every case. In one example, **97 of 484 items** preserved a **99% correlation** with the full ranking when selected by discrimination, offering a concrete path to lower token and runtime costs.

The reduction does not hold for every benchmark, and ranking retention is not the same as preserving every capability signal. Adaptive tests and fingerprint sets may help detect leakage, but the speaker explicitly describes that method as not bulletproof.