Advancing Quantization for R&R Models with Microscaling Formats
Summary
- The paper evaluates MXFP8 and MXFP6 microscaling formats for industrial Recommendation and Ranking (R&R) models on the AMD MI350X architecture.
- R&R models demand extreme precision, requiring model quality to stay within a strict 0.01% Normalized Entropy margin relative to baselines.
- The authors report native MXFP6 GEMM speedups without sacrificing model quality, alongside MXFP8 serving as a viable alternative to row-wise FP8.
- Implementation across production ads ranking models demonstrates GEMM kernel speedups of 1.33-1.55x for MXFP8 and 1.73-1.99x for MXFP6 over FP16.
- End-to-end inference throughput gains from selective MXFP6 reached 1.5-5.3% for Model A and 7.6-14.3% for Model B over their FP8 configurations.
Why it matters
Using microscaling formats like MXFP6 and MXFP8 enables aggressive quantization for precision-sensitive Recommendation and Ranking models, delivering significant speedups on AMD hardware without breaching strict accuracy thresholds.