SynthCorp: A Platform for Creating Benchmarks for Data Agents
Summary
- Researchers introduce SynthCorp, a platform for creating customizable data estates and benchmarks to evaluate LLM-driven data agents.
- SynthCorp simulates business events over a virtual calendar, producing relational databases, document stores, and data catalogs with verified gold programs.
- The platform generated initial benchmarks comprising seven data estates and 256 tasks across five domains, passing pipeline checks with minimal errors.
- Evaluations using SynthCorp demonstrate that its benchmarks effectively separate different model tiers and data contexts.
Why it matters
SynthCorp provides a systematic way to generate complex enterprise data estates and benchmarks, addressing the rapid saturation of current LLM data agent evaluations.