Summary
- Researchers investigate whether confidence-related signals can serve as indicators of problem difficulty across various task domains.
- The study finds that perturbation inconsistency, semantic entropy, and solution length show stronger associations with population difficulty than token-level uncertainty.
- These signals capture information about problem difficulty even when restricting the analysis to correctly answered problems.
- The findings demonstrate the utility of model-specific signals as proxies for population-level problem difficulty to guide inference-time decisions.
Why it matters
Understanding how confidence signals relate to problem difficulty helps practitioners better gauge task complexity and guide inference-time decisions in AI models.