Evidence-Based Credit Assignment for Search-Augmented Language Models

Impact 8 arXiv cs.AI

Summary

  • Researchers introduce Evidence-based Credit Assignment (ECA) to improve reinforcement learning in search-augmented language models by addressing the limitation where outcome supervision assigns identical credit to every search in a trajectory.
  • ECA utilizes a training-only auxiliary signal measuring how retrieved documents improve reference answer likelihood, isolating retrieval utility from surface variations via a frozen policy snapshot.
  • The framework incorporates return formulations like ECA-Step and ECA-Horizon, alongside cross-fold group normalization, novelty gating, and search-local token assignment.
  • Experiments across seven Search QA benchmarks show ECA-Horizon improves strict macro exact match over outcome-only GRPO baselines by 1.03 and 4.80 percentage points on Qwen2.5-3B- and 7B-Instruct models.

Why it matters

This research addresses a fundamental limitation in reinforcement learning for retrieval-augmented generation by properly attributing credit to individual search actions, enhancing model reasoning and multi-hop inference performance without relying on step-level annotations.

Who should care