{
  "schema_version": "1.1",
  "id": "s13:https://arxiv.org/abs/2609.22005v1",
  "slug": "2609-22005v1-1tiqzcj",
  "url": "https://feed7.dev/p/2609-22005v1-1tiqzcj",
  "title": "Abstention and Noise Filtering: Two Missing Primitives of Softmax Attention",
  "why_included": "Attention gates appear to provide two distinct functions: letting heads abstain and filtering residual-stream noise. Their relative value changes with model scale.",
  "summary": "Matched models from **10M to 350M parameters** separated two functions of attention gating: abstention through a per-head sink logit and noise filtering through per-value gates. Abstention contributed most at 10M, while filtering contributed most at 350M.",
  "practical_implication": "Model builders should evaluate the two mechanisms independently rather than treating gating as one feature. The **best model at every tested scale used both**, with negligible added parameters and compatibility with the key-value cache.",
  "agent_context": "Matched models from **10M to 350M parameters** separated two functions of attention gating: abstention through a per-head sink logit and noise filtering through per-value gates. Abstention contributed most at 10M, while filtering contributed most at 350M.\n\nModel builders should evaluate the two mechanisms independently rather than treating gating as one feature. The **best model at every tested scale used both**, with negligible added parameters and compatibility with the key-value cache.\n\nControlled interference tests found that each gate form has a characteristic blind spot. The evidence covers pretraining models up to 350M parameters, so behavior at larger production scales remains open.",
  "source": {
    "name": "arXiv",
    "url": "https://arxiv.org/abs/2609.22005v1",
    "published_at": "2026-09-18T17:03:58.000Z"
  },
  "source_class": "blog_post",
  "content_type": "Paper",
  "layer": "model",
  "domains": [],
  "topics": [
    "reasoning"
  ],
  "verification": {
    "status": "needs_review",
    "label": "Needs Review",
    "method": "unverified",
    "verified_at": null
  },
  "uncertainty": [
    "Controlled interference tests found that each gate form has a characteristic blind spot. The evidence covers pretraining models up to 350M parameters, so behavior at larger production scales remains open."
  ],
  "connected_context": {
    "meaning": "This narrows architectural gating into two separately testable functions whose value shifts with model scale: suppressing attention altogether and filtering individual values are not substitutes. Against the supplied reasoning candidates, it identifies a low-parameter change inside attention itself rather than another training signal, reward construction, model repair, or test-time optimization method; evidence beyond 350M parameters remains missing.",
    "corpus_size": 831,
    "generated_at": "2026-09-21T09:04:18.336Z",
    "connections": []
  },
  "lifecycle": "Current",
  "published_at": "2026-09-18T17:03:58.000Z",
  "modified_at": "2026-09-18T17:03:58.000Z",
  "supersedes": [],
  "expires_at": null,
  "formats": {
    "html": "https://feed7.dev/p/2609-22005v1-1tiqzcj",
    "json": "https://feed7.dev/p/2609-22005v1-1tiqzcj.json",
    "markdown": "https://feed7.dev/p/2609-22005v1-1tiqzcj.md"
  }
}