Abstention and Noise Filtering: Two Missing Primitives of Softmax Attention
Attention gates appear to provide two distinct functions: letting heads abstain and filtering residual-stream noise. Their relative value changes with model scale.
Matched models from **10M to 350M parameters** separated two functions of attention gating: abstention through a per-head sink logit and noise filtering through per-value gates. Abstention contributed most at 10M, while filtering contributed most at 350M.
Model builders should evaluate the two mechanisms independently rather than treating gating as one feature. The **best model at every tested scale used both**, with negligible added parameters and compatibility with the key-value cache.
Matched models from **10M to 350M parameters** separated two functions of attention gating: abstention through a per-head sink logit and noise filtering through per-value gates. Abstention contributed most at 10M, while filtering contributed most at 350M. Model builders should evaluate the two mechanisms independently rather than treating gating as one feature. The **best model at every tested scale used both**, with negligible added parameters and compatibility with the key-value cache. Controlled interference tests found that each gate form has a characteristic blind spot. The evidence covers pretraining models up to 350M parameters, so behavior at larger production scales remains open.
This narrows architectural gating into two separately testable functions whose value shifts with model scale: suppressing attention altogether and filtering individual values are not substitutes. Against the supplied reasoning candidates, it identifies a low-parameter change inside attention itself rather than another training signal, reward construction, model repair, or test-time optimization method; evidence beyond 350M parameters remains missing.