Vercel Labs Open-Sources agent-desktop for Agent Computer Use
Summary
- Vercel Labs has released agent-desktop, an open-source CLI and C library enabling AI agents to interact with desktop apps via OS accessibility trees instead of screenshots.
- The tool operates headlessly by default on macOS, Windows, and Linux, utilizing platform-native APIs like AX API and UI Automation.
- By employing skeleton snapshots, the tool significantly reduces token overhead by 78% to 96% on dense applications, according to self-reported metrics.
- It is model-agnostic, open-sourced under the Apache-2.0 license, and provides structured JSON output with deterministic retry-safe references for agents.
Why it matters
By replacing high-overhead visual screenshots with structured OS accessibility tree data, agent-desktop makes desktop computer use for AI agents drastically more efficient, token-cheap, and reliable.