{
  "schema_version": "1.1",
  "id": "s13:https://arxiv.org/abs/2609.11917v1",
  "slug": "2609-11917v1-05xeyf9",
  "url": "https://feed7.dev/p/2609-11917v1-05xeyf9",
  "title": "Data Scarcity and Model Sparsity: Mixtures-of-Experts Overfit More to Repeated Data",
  "why_included": "MoE models overfit repeated training data earlier than dense models, with total parameter count driving the effect. Strong masking helps, but unique data remains the stronger baseline.",
  "summary": "Across models with **80M–1B active parameters** and up to **8.5B total parameters**, MoEs began degrading around **4x repetition** while an 80M dense model tolerated more than 8x with little loss. After 32x, MoEs lost their all-unique-data advantage and fell behind dense models.",
  "practical_implication": "When choosing or training sparse models under data scarcity, budget for repetition sensitivity based on total rather than active parameters. Test repeated-data regimes directly and consider dropout or strong masking regularization instead of assuming MoE compute efficiency carries through unchanged.",
  "agent_context": "Across models with **80M–1B active parameters** and up to **8.5B total parameters**, MoEs began degrading around **4x repetition** while an 80M dense model tolerated more than 8x with little loss. After 32x, MoEs lost their all-unique-data advantage and fell behind dense models.\n\nWhen choosing or training sparse models under data scarcity, budget for repetition sensitivity based on total rather than active parameters. Test repeated-data regimes directly and consider dropout or strong masking regularization instead of assuming MoE compute efficiency carries through unchanged.\n\nStrong masking kept MoEs ahead of dense models beyond **64 repetitions**, but no tested regularizer matched training on entirely unique data. Early routing stabilization and expert specialization correlate with overfitting; the paper does not establish either as the sole cause.",
  "source": {
    "name": "arXiv",
    "url": "https://arxiv.org/abs/2609.11917v1",
    "published_at": "2026-09-10T17:57:33.000Z"
  },
  "source_class": "blog_post",
  "content_type": "Paper",
  "layer": "model",
  "domains": [
    "research",
    "data"
  ],
  "topics": [
    "model-selection"
  ],
  "verification": {
    "status": "needs_review",
    "label": "Needs Review",
    "method": "unverified",
    "verified_at": null
  },
  "uncertainty": [
    "Strong masking kept MoEs ahead of dense models beyond **64 repetitions**, but no tested regularizer matched training on entirely unique data. Early routing stabilization and expert specialization correlate with overfitting; the paper does not establish either as the sole cause."
  ],
  "connected_context": {
    "meaning": "This adds training-data repetition as a model-selection constraint that can reverse MoE advantages despite low active-parameter compute. It reinforces workload-specific evaluation over scale heuristics, but sharpens the rule for sparse models: scarcity should be assessed against total capacity, repeated-data regimes should be tested directly, and regularization mitigates rather than eliminates the unique-data gap.",
    "corpus_size": 757,
    "generated_at": "2026-09-12T10:06:52.849Z",
    "connections": [
      {
        "title": "When Does Bigger Help? A Controlled Study of LLM Scale for Ontology Learning",
        "source_name": "arXiv",
        "source_url": "https://arxiv.org/abs/2608.31118v1",
        "feed7_url": "https://feed7.dev/p/2608-31118v1-1abyqwd",
        "reason": "The controlled ontology study shows sparse-versus-dense outcomes vary by task; this Signal identifies repeated training data as another condition under which larger sparse capacity can lose to a dense model."
      },
      {
        "title": "Adaption Labs: Gradient-Free Continual Learning — Sara Hooker, Adaption",
        "source_name": "AI Engineer",
        "source_url": "https://www.youtube.com/watch?v=XEd_SRVHBgU",
        "feed7_url": "https://feed7.dev/p/adaption-labs-gradient-free-continual-learning-sara-hooker-adaption-1pozjjc",
        "reason": "Auto Scientist proposes jointly searching data and architecture choices; this Signal supplies a concrete interaction such systems would need to evaluate, because MoE architecture changes the cost of reusing scarce data."
      }
    ]
  },
  "lifecycle": "Current",
  "published_at": "2026-09-10T17:57:33.000Z",
  "modified_at": "2026-09-10T17:57:33.000Z",
  "supersedes": [],
  "expires_at": null,
  "formats": {
    "html": "https://feed7.dev/p/2609-11917v1-05xeyf9",
    "json": "https://feed7.dev/p/2609-11917v1-05xeyf9.json",
    "markdown": "https://feed7.dev/p/2609-11917v1-05xeyf9.md"
  }
}