Advancing Quantization for R&R Models with Microscaling Formats

Impact 7 arXiv cs.AI

Summary

  • The paper evaluates MXFP8 and MXFP6 microscaling formats for industrial Recommendation and Ranking (R&R) models on the AMD MI350X architecture.
  • R&R models demand extreme precision, requiring model quality to stay within a strict 0.01% Normalized Entropy margin relative to baselines.
  • The authors report native MXFP6 GEMM speedups without sacrificing model quality, alongside MXFP8 serving as a viable alternative to row-wise FP8.
  • Implementation across production ads ranking models demonstrates GEMM kernel speedups of 1.33-1.55x for MXFP8 and 1.73-1.99x for MXFP6 over FP16.
  • End-to-end inference throughput gains from selective MXFP6 reached 1.5-5.3% for Model A and 7.6-14.3% for Model B over their FP8 configurations.

Why it matters

Using microscaling formats like MXFP6 and MXFP8 enables aggressive quantization for precision-sensitive Recommendation and Ranking models, delivering significant speedups on AMD hardware without breaching strict accuracy thresholds.

Who should care