Continual Tool-Space Extension Learning for Multimodal Agents
Summary
- Researchers propose COTEX, a continual tool-space extension learning method for integrating specialist vision models into multimodal agents at adapter-level cost.
- COTEX identifies two parameter direction sets: one for triggering tool calls and initiating multi-turn reasoning, and another for selecting the specific tool.
- An orthogonal constraint restricts new tool adapters to prevent interference with existing tool choices and preserve prior capabilities.
- A newly introduced protocol converts task-specific annotations from specialist models into verified tool-use trajectories for agent training.
- Experiments across two agents and three specialist tools demonstrate benchmark gains with minimal performance degradation on previously learned tools.
Why it matters
This research provides an efficient method for expanding multimodal agent capabilities with new specialist tools without catastrophic forgetting of existing skills. It enables more modular and scalable agent development for complex AI systems.