Training Communication-Efficient Mixture-of-Experts Language Models with Layer Re-Configuration
CE-MoE concentrates routed experts in fewer layers and restores depth with dense layers, cutting reported GPU-hours by 33.3% at 31.5B parameters without worse benchmark results.
CE-MoE places expert capacity in a few routed layers, then preserves depth with extra token-mixing and dense-FFN layers. Across **2B–31.5B parameters**, it matched full-MoE validation loss and downstream benchmarks under matched total and activated parameters.
At **31.5B**, the authors report **33.3% fewer GPU-hours**, plus higher average downstream score and inference throughput. Teams training MoE models should treat routed-layer frequency as a systems parameter, not a fixed architectural convention.
CE-MoE places expert capacity in a few routed layers, then preserves depth with extra token-mixing and dense-FFN layers. Across **2B–31.5B parameters**, it matched full-MoE validation loss and downstream benchmarks under matched total and activated parameters. At **31.5B**, the authors report **33.3% fewer GPU-hours**, plus higher average downstream score and inference throughput. Teams training MoE models should treat routed-layer frequency as a systems parameter, not a fixed architectural convention. The material provides no absolute GPU-hours, throughput values, hardware setup, or benchmark deltas. The result concerns model training architecture, so it does not establish equivalent savings for serving existing MoE checkpoints.
CE-MoE makes routed-layer frequency an explicit architecture and systems choice, showing that sparse expert capacity need not appear at every layer to preserve matched-parameter quality. This complements prior data- and compression-efficiency work with a training-architecture lever, while the omitted hardware and absolute measurements prevent translating the reported savings into deployment budgets or serving gains.