From Questions to Decisions: Benchmarking How LLM Agents Acquire Decision-Relevant Information from Human Beings
Summary
- Researchers introduce Ask2Decide, a benchmark of 43 planning tasks evaluating how LLM agents acquire missing decision-relevant information from humans.
- The benchmark uses optimization solvers to determine which missing facts change optimal plans versus irrelevant details.
- Evaluating four frontier LLMs revealed that initial plans were infeasible on 58% to 98% of tasks when information was missing.
- Agents uncovered helpful facts only slightly more often than irrelevant details, highlighting information acquisition as a major bottleneck.
Why it matters
LLM agents struggle significantly to proactively and effectively gather critical decision-making details from human users through interaction, posing major reliability hurdles for automated planning tasks.