RE-SciVQA: Improving Scientific VQA With Context-Relevant Retrieval and Lightweight MLLM Adaptation
Summary
- Researchers proposed RE-SciVQA, a method to improve Scientific Visual Question Answering using context-relevant retrieval and parameter-efficient LoRA training on small LLMs.
- The approach uses a query expansion step to append relevant section headers and adjusts similarity scores via figure references and caption alignment.
- On the SciVQA benchmark, RE-SciVQA achieved a ROUGE-1 F1 score of 73.9%, outperforming baseline chart-focused models.
- The relevance-enhanced RAG setup improved exact match by 11.9% and fuzzy match by 29.5% compared to standard RAG baselines.
- Using plot digitization instead of synthetic captions yielded an 18.5% improvement for closed-ended infinite visual questions.
Why it matters
RE-SciVQA demonstrates that efficient retrieval enhancement and parameter-efficient fine-tuning can significantly outperform resource-intensive pipelines for scientific multimodal tasks.