Asking but Missing: Decision Coverage in Natural-Language Data Querying
Summary
- Researchers examine how large language models translate natural language questions into executable code for structured data querying.
- They investigate Underspecified Decision Completion, where models silently adopt assumptions for ambiguous questions instead of recognizing unresolved decisions.
- The authors introduce Grounded Code Generation, a diagnostic protocol separating the identification of decisions, their resolution, and code translation.
- Evaluating twelve models on BIRD-Pandas reveals that the primary challenge is recognizing which clarification questions need to be asked.
Why it matters
Understanding how language models handle ambiguity in data querying helps developers improve accuracy and prevent silent errors in AI-generated code.