PAYWORTHY: Can LLMs Predict Which Freelance Jobs an AI Agent Will Complete?
Summary
- Researchers introduce PAYWORTHY, a dataset of 1,494 real freelance job postings to test whether LLMs can predict if an AI agent will succeed at a task.
- When models are informed about what the agent cannot do, they successfully rank the jobs with an AUC between 0.66 and 0.86.
- When described as a generic agent with a browser and code execution, model forecasts drop to near random levels with AUCs between 0.45 and 0.56.
- These findings hold across multiple model families including Claude, GLM, Kimi, and Qwen, with code made publicly available.
Why it matters
Accurately predicting agent success prior to deployment is crucial for automating freelance work and managing client expectations, highlighting the importance of specifying agent limitations for effective LLM forecasting.