What Does a Bias Head Measure? Non-Identifiability in Head-Level Bias Attribution
An audit finds head-level bias maps on two language models cannot be attributed to bias from observational data, and ablating selected heads did not reduce unfairness. The method may track prompt structure instead.
The audit tested Qwen-2.5-7B-Instruct and Llama-3.1-8B-Instruct. Prompt identity explained **64%** of the selected statistic’s variance; response length explained **1%** after an intervention broke its link to labels. Ablating tested head sets did not lower unfairness.
If you use internal head maps to diagnose or mitigate model behavior, treat the score as a proxy until an intervention isolates what it measures. Validate the exact downstream behavior you care about, including refusal behavior.
The audit tested Qwen-2.5-7B-Instruct and Llama-3.1-8B-Instruct. Prompt identity explained **64%** of the selected statistic’s variance; response length explained **1%** after an intervention broke its link to labels. Ablating tested head sets did not lower unfairness. If you use internal head maps to diagnose or mitigate model behavior, treat the score as a proxy until an intervention isolates what it measures. Validate the exact downstream behavior you care about, including refusal behavior. Within a fixed prompt, labels, response length, and protected-group tokens were highly collinear, limiting observational attribution. The paper reports refusal behavior was damaged by the complementary head set, but these results cover only two models and the tested pipeline.
The study warns that a head-level bias score may mostly track prompt identity rather than the intended construct, and that ablating selected heads did not reduce unfairness in the tested pipeline. This narrows what internal attribution can establish and makes downstream behavior checks essential, including refusal behavior that the complementary head set damaged.