Quantiles, Not Points: Streaming Quantile Matching for Ordinal-Aware LLM-as-a-Judge Calibration

Impact 7 arXiv cs.AI

Summary

  • Researchers introduce streaming quantile matching (SQM) for ordinal-aware LLM-as-a-judge calibration to correct distributional shifts against human ratings.
  • SQM computes the optimal-transport map between LLM and human score distributions, respecting ordinality and achieving high distributional fidelity.
  • Post-hoc calibration narrows the raw error gap between smaller and larger judge models, potentially enabling more efficient LLM judges.

Why it matters

Proper calibration of LLM judges using streaming quantile matching improves evaluation reliability and can enable cost-effective use of smaller models for complex assessment tasks.

Who should care