arXivPaperNeeds Review

Training Communication-Efficient Mixture-of-Experts Language Models with Layer Re-Configuration

CE-MoE concentrates routed experts in fewer layers and restores depth with dense layers, cutting reported GPU-hours by 33.3% at 31.5B parameters without worse benchmark results.

arXiv

Open Source Open Markdown Open JSON

Source Summary

CE-MoE places expert capacity in a few routed layers, then preserves depth with extra token-mixing and dense-FFN layers. Across **2B–31.5B parameters**, it matched full-MoE validation loss and downstream benchmarks under matched total and activated parameters.

Practical Implication

At **31.5B**, the authors report **33.3% fewer GPU-hours**, plus higher average downstream score and inference throughput. Teams training MoE models should treat routed-layer frequency as a systems parameter, not a fixed architectural convention.

Agent-Ready Context

CE-MoE places expert capacity in a few routed layers, then preserves depth with extra token-mixing and dense-FFN layers. Across **2B–31.5B parameters**, it matched full-MoE validation loss and downstream benchmarks under matched total and activated parameters.

At **31.5B**, the authors report **33.3% fewer GPU-hours**, plus higher average downstream score and inference throughput. Teams training MoE models should treat routed-layer frequency as a systems parameter, not a fixed architectural convention.

The material provides no absolute GPU-hours, throughput values, hardware setup, or benchmark deltas. The result concerns model training architecture, so it does not establish equivalent savings for serving existing MoE checkpoints.

Context Map

model#open-models

Uncertainty

The material provides no absolute GPU-hours, throughput values, hardware setup, or benchmark deltas. The result concerns model training architecture, so it does not establish equivalent savings for serving existing MoE checkpoints.