Data Scarcity and Model Sparsity: Mixtures-of-Experts Overfit More to Repeated Data
MoE models overfit repeated training data earlier than dense models, with total parameter count driving the effect. Strong masking helps, but unique data remains the stronger baseline.
Across models with **80M–1B active parameters** and up to **8.5B total parameters**, MoEs began degrading around **4x repetition** while an 80M dense model tolerated more than 8x with little loss. After 32x, MoEs lost their all-unique-data advantage and fell behind dense models.
When choosing or training sparse models under data scarcity, budget for repetition sensitivity based on total rather than active parameters. Test repeated-data regimes directly and consider dropout or strong masking regularization instead of assuming MoE compute efficiency carries through unchanged.
Across models with **80M–1B active parameters** and up to **8.5B total parameters**, MoEs began degrading around **4x repetition** while an 80M dense model tolerated more than 8x with little loss. After 32x, MoEs lost their all-unique-data advantage and fell behind dense models. When choosing or training sparse models under data scarcity, budget for repetition sensitivity based on total rather than active parameters. Test repeated-data regimes directly and consider dropout or strong masking regularization instead of assuming MoE compute efficiency carries through unchanged. Strong masking kept MoEs ahead of dense models beyond **64 repetitions**, but no tested regularizer matched training on entirely unique data. Early routing stabilization and expert specialization correlate with overfitting; the paper does not establish either as the sole cause.
This adds training-data repetition as a model-selection constraint that can reverse MoE advantages despite low active-parameter compute. It reinforces workload-specific evaluation over scale heuristics, but sharpens the rule for sparse models: scarcity should be assessed against total capacity, repeated-data regimes should be tested directly, and regularization mitigates rather than eliminates the unique-data gap.