# Inside the Unfair Judge: A Mechanistic Interpretability Account of LLM-as-Judge Bias

Source: [arXiv](https://arxiv.org/abs/2607.11871v1)  
Feed7 permalink: https://feed7.dev/p/2607-11871v1-17vejh0  
Published: Unknown  
Trust: Needs Review (needs_review)

## Why Included

LLM judge bias appears as steerable hidden-state directions that predict failures on unseen benchmarks, so eval pipelines may need representation-level checks beyond prompt fixes.

## Source Summary

Across **7 judges, 7 bias types, and 9 benchmarks**, biased inputs displaced hidden states along low-dimensional, type-specific directions that became clearer with depth. Three estimator families recovered the structure consistently.

## Practical Implication

If model-based grading drives agent evals, test the judge itself under known bias perturbations. For models whose activations are accessible, projection onto learned bias directions could supplement score-delta tests and prompt-level mitigations.

## Agent-Ready Context

Across **7 judges, 7 bias types, and 9 benchmarks**, biased inputs displaced hidden states along low-dimensional, type-specific directions that became clearer with depth. Three estimator families recovered the structure consistently.

If model-based grading drives agent evals, test the judge itself under known bias perturbations. For models whose activations are accessible, projection onto learned bias directions could supplement score-delta tests and prompt-level mitigations.

The evidence comes from the studied judges and bias types, and activation access is unavailable for many hosted models. Prediction was tested on **3 unseen benchmarks**; broader transfer and practical intervention costs remain open.

## Context Map

- Layer: benchmark
- Domains: research
- Topics: benchmark-integrity, agent-evals, agent-reliability

## Uncertainty

- The evidence comes from the studied judges and bias types, and activation access is unavailable for many hosted models. Prediction was tested on **3 unseen benchmarks**; broader transfer and practical intervention costs remain open.

## Agent Instruction

Use this item as source-backed context. Do not invent claims beyond the linked source. If this item conflicts with another source, call out the conflict.