Evidence-Based Credit Assignment for Search-Augmented Language Models
Summary
- Researchers introduce Evidence-based Credit Assignment (ECA) to improve reinforcement learning in search-augmented language models by addressing the limitation where outcome supervision assigns identical credit to every search in a trajectory.
- ECA utilizes a training-only auxiliary signal measuring how retrieved documents improve reference answer likelihood, isolating retrieval utility from surface variations via a frozen policy snapshot.
- The framework incorporates return formulations like ECA-Step and ECA-Horizon, alongside cross-fold group normalization, novelty gating, and search-local token assignment.
- Experiments across seven Search QA benchmarks show ECA-Horizon improves strict macro exact match over outcome-only GRPO baselines by 1.03 and 4.80 percentage points on Qwen2.5-3B- and 7B-Instruct models.
Why it matters
This research addresses a fundamental limitation in reinforcement learning for retrieval-augmented generation by properly attributing credit to individual search actions, enhancing model reasoning and multi-hop inference performance without relying on step-level annotations.