Video-as-Experience: Learning Computer Use via Audio-Visual Understanding

Impact 8 arXiv cs.AI

Summary

  • Researchers introduce VEX (Video as EXperience), a framework enabling computer-use agents to learn via audio-visual understanding rather than converting video demonstrations into static screenshot trajectories.
  • Following a watch-act-rewatch paradigm, VEX extracts procedural guidance from tutorial videos with visual demonstrations and spoken explanations, executes tasks, and verifies progress by reviewing its own execution video.
  • The framework is instantiated using video-SALMONN-VEX, a 9B-scale agent built on video-SALMONN 2 and Qwen3.5-9B, trained with a new Action-Grounded Guide Optimization reinforcement learning method.
  • Evaluations on OSWorld and CUA-Gym show that video-SALMONN-VEX achieves a 22 percent relative increase in task success over its Qwen3.5 backbone, while also improving Gemini-3.5-flash performance by 10 percent.

Why it matters

Utilizing audio-visual understanding for computer-use agents bridges the gap between human multimedia tutorials and automated task execution, improving multi-modal reasoning and self-verification.

Who should care