arXivPaperNeeds Review
Inside the Unfair Judge: A Mechanistic Interpretability Account of LLM-as-Judge Bias
LLM judge bias appears as steerable hidden-state directions that predict failures on unseen benchmarks, so eval pipelines may need representation-level checks beyond prompt fixes.
arXiv
Source Summary
Across **7 judges, 7 bias types, and 9 benchmarks**, biased inputs displaced hidden states along low-dimensional, type-specific directions that became clearer with depth. Three estimator families recovered the structure consistently.
Practical Implication
If model-based grading drives agent evals, test the judge itself under known bias perturbations. For models whose activations are accessible, projection onto learned bias directions could supplement score-delta tests and prompt-level mitigations.
Agent-Ready Context
Across **7 judges, 7 bias types, and 9 benchmarks**, biased inputs displaced hidden states along low-dimensional, type-specific directions that became clearer with depth. Three estimator families recovered the structure consistently. If model-based grading drives agent evals, test the judge itself under known bias perturbations. For models whose activations are accessible, projection onto learned bias directions could supplement score-delta tests and prompt-level mitigations. The evidence comes from the studied judges and bias types, and activation access is unavailable for many hosted models. Prediction was tested on **3 unseen benchmarks**; broader transfer and practical intervention costs remain open.
Context Map
benchmarkresearch#benchmark-integrity#agent-evals#agent-reliabilityUncertainty
The evidence comes from the studied judges and bias types, and activation access is unavailable for many hosted models. Prediction was tested on **3 unseen benchmarks**; broader transfer and practical intervention costs remain open.