# Data Scarcity and Model Sparsity: Mixtures-of-Experts Overfit More to Repeated Data

Source: [arXiv](https://arxiv.org/abs/2609.11917v1)  
Feed7 permalink: https://feed7.dev/p/2609-11917v1-05xeyf9  
Published: 2026-09-10T17:57:33.000Z  
Trust: Needs Review (needs_review)

## Why Included

MoE models overfit repeated training data earlier than dense models, with total parameter count driving the effect. Strong masking helps, but unique data remains the stronger baseline.

## Source Summary

Across models with **80M–1B active parameters** and up to **8.5B total parameters**, MoEs began degrading around **4x repetition** while an 80M dense model tolerated more than 8x with little loss. After 32x, MoEs lost their all-unique-data advantage and fell behind dense models.

## Practical Implication

When choosing or training sparse models under data scarcity, budget for repetition sensitivity based on total rather than active parameters. Test repeated-data regimes directly and consider dropout or strong masking regularization instead of assuming MoE compute efficiency carries through unchanged.

## Agent-Ready Context

Across models with **80M–1B active parameters** and up to **8.5B total parameters**, MoEs began degrading around **4x repetition** while an 80M dense model tolerated more than 8x with little loss. After 32x, MoEs lost their all-unique-data advantage and fell behind dense models.

When choosing or training sparse models under data scarcity, budget for repetition sensitivity based on total rather than active parameters. Test repeated-data regimes directly and consider dropout or strong masking regularization instead of assuming MoE compute efficiency carries through unchanged.

Strong masking kept MoEs ahead of dense models beyond **64 repetitions**, but no tested regularizer matched training on entirely unique data. Early routing stabilization and expert specialization correlate with overfitting; the paper does not establish either as the sole cause.

## Connected Context

Feed7 judgment across 757 accumulated Signals:

This adds training-data repetition as a model-selection constraint that can reverse MoE advantages despite low active-parameter compute. It reinforces workload-specific evaluation over scale heuristics, but sharpens the rule for sparse models: scarcity should be assessed against total capacity, repeated-data regimes should be tested directly, and regularization mitigates rather than eliminates the unique-data gap.

- [When Does Bigger Help? A Controlled Study of LLM Scale for Ontology Learning](https://feed7.dev/p/2608-31118v1-1abyqwd) — The controlled ontology study shows sparse-versus-dense outcomes vary by task; this Signal identifies repeated training data as another condition under which larger sparse capacity can lose to a dense model.
- [Adaption Labs: Gradient-Free Continual Learning — Sara Hooker, Adaption](https://feed7.dev/p/adaption-labs-gradient-free-continual-learning-sara-hooker-adaption-1pozjjc) — Auto Scientist proposes jointly searching data and architecture choices; this Signal supplies a concrete interaction such systems would need to evaluate, because MoE architecture changes the cost of reusing scarce data.

## Context Map

- Layer: model
- Domains: research, data
- Topics: model-selection

## Uncertainty

- Strong masking kept MoEs ahead of dense models beyond **64 repetitions**, but no tested regularizer matched training on entirely unique data. Early routing stabilization and expert specialization correlate with overfitting; the paper does not establish either as the sole cause.

## Agent Instruction

Use this item as source-backed context. Do not invent claims beyond the linked source. If this item conflicts with another source, call out the conflict.