# Abstention and Noise Filtering: Two Missing Primitives of Softmax Attention

Source: [arXiv](https://arxiv.org/abs/2609.22005v1)  
Feed7 permalink: https://feed7.dev/p/2609-22005v1-1tiqzcj  
Published: 2026-09-18T17:03:58.000Z  
Trust: Needs Review (needs_review)

## Why Included

Attention gates appear to provide two distinct functions: letting heads abstain and filtering residual-stream noise. Their relative value changes with model scale.

## Source Summary

Matched models from **10M to 350M parameters** separated two functions of attention gating: abstention through a per-head sink logit and noise filtering through per-value gates. Abstention contributed most at 10M, while filtering contributed most at 350M.

## Practical Implication

Model builders should evaluate the two mechanisms independently rather than treating gating as one feature. The **best model at every tested scale used both**, with negligible added parameters and compatibility with the key-value cache.

## Agent-Ready Context

Matched models from **10M to 350M parameters** separated two functions of attention gating: abstention through a per-head sink logit and noise filtering through per-value gates. Abstention contributed most at 10M, while filtering contributed most at 350M.

Model builders should evaluate the two mechanisms independently rather than treating gating as one feature. The **best model at every tested scale used both**, with negligible added parameters and compatibility with the key-value cache.

Controlled interference tests found that each gate form has a characteristic blind spot. The evidence covers pretraining models up to 350M parameters, so behavior at larger production scales remains open.

## Connected Context

Feed7 judgment across 831 accumulated Signals:

This narrows architectural gating into two separately testable functions whose value shifts with model scale: suppressing attention altogether and filtering individual values are not substitutes. Against the supplied reasoning candidates, it identifies a low-parameter change inside attention itself rather than another training signal, reward construction, model repair, or test-time optimization method; evidence beyond 350M parameters remains missing.

No material corpus connection was strong enough to record.

## Context Map

- Layer: model
- Domains: None
- Topics: reasoning

## Uncertainty

- Controlled interference tests found that each gate form has a characteristic blind spot. The evidence covers pretraining models up to 350M parameters, so behavior at larger production scales remains open.

## Agent Instruction

Use this item as source-backed context. Do not invent claims beyond the linked source. If this item conflicts with another source, call out the conflict.