Training an MoE LLM in MXFP4: How We Closed the Gap to BF16
Summary
- Researchers presented ALTO, an open-source training recipe for the 20-billion-parameter MoE model GPT-OSS-20B using the MXFP4 microscaling format.
- The recipe combines hybrid 1D/2D block quantization, randomized Hadamard transforms, stochastic rounding, and a simplified weight de-oscillation scheme.
- Using MXFP4 with these techniques achieves a C4 validation loss of 3.3350, coming within 0.007 of the BF16 baseline.
- De-oscillation reduces convergence overhead relative to BF16 from plus 20 percent down to plus 5 percent on the MLPerf Small MoE Training benchmark.
- Experiments utilized simulated MXFP4 kernels on AMD MI300 hardware without wall-clock speed optimization.
Why it matters
Training large language models in sub-8-bit formats like MXFP4 significantly improves memory bandwidth and compute throughput. This research demonstrates a viable path to closing the accuracy gap between sub-8-bit training and standard BF16.