Frame: Form-Restricted Authoring with Model-Based Evaluation
Summary
- Researchers formalize form-restricted report generation, the task of updating report content using new grounding documents while strictly preserving the structure of prior editions.
- They introduce FRAME, a benchmark consisting of 120 EDGAR- and arXiv-derived instances across financial and academic categories, built via a temporal three-snapshot pipeline.
- Evaluation across five LLMs uses three human-validated metrics: structural similarity, information unit recall, and claim-level grounding verification.
- Results show open-source models nearly match proprietary models on structure but lag on coverage and faithfulness, with all models performing better on financial reports than academic ones.
Why it matters
The FRAME benchmark provides a standardized way to evaluate how well large language models maintain rigid structural constraints while accurately incorporating new evidence into professional reports.