GEM Training: How Meta Doubled the Efficiency of Its LLM-Scale Ads Foundation Model
Meta doubled GEM’s training efficiency by co-designing recommendation-specific kernels, precision, parallelism, networking, and memory. The takeaway is to profile workload shape before reusing an LLM training stack.
Meta reports GEM reached **20–25% end-to-end MFU** while training FLOPs grew **4× in 12 months** across several thousand GPUs. Custom kernels, MXFP8 training, topology-aware **5D parallelism**, and communication changes address jagged sequences and sparse-dense parameters.
Builders operating specialized model workloads should separate local compute efficiency from distributed scaling efficiency. Profile sequence shapes, padding, numerical sensitivity, communication overlap, memory pressure, and rank imbalance before assuming standard LLM kernels or parallelism recipes transfer.
Meta reports GEM reached **20–25% end-to-end MFU** while training FLOPs grew **4× in 12 months** across several thousand GPUs. Custom kernels, MXFP8 training, topology-aware **5D parallelism**, and communication changes address jagged sequences and sparse-dense parameters. Builders operating specialized model workloads should separate local compute efficiency from distributed scaling efficiency. Profile sequence shapes, padding, numerical sensitivity, communication overlap, memory pressure, and rank imbalance before assuming standard LLM kernels or parallelism recipes transfer. These techniques target Meta’s ads model with trillions of sparse and billions of dense parameters. The material does not establish that the same kernels, precision choices, or topology will benefit smaller models or different recommendation workloads.
This separates kernel-level utilization from distributed scaling efficiency for irregular, sparse recommendation training. It confirms that profiling shapes, padding, precision sensitivity, communication, memory, and rank imbalance must precede optimization, while sharply limiting transfer: Meta’s 5D parallelism, MXFP8 choices, and custom kernels are evidence for one unusually large ads workload, not a general recipe.