The paper argues that multimodal language models struggle with multi-step visual reasoning because of a "reference gap," not a failure to see. Adding task-matched visual references, such as numbering, improves accuracy without any training.
The gap: The authors say models can recognize the content but lose track of which objects or regions they have bound to the question. Counting is their example.
The benchmark: ReGap-Bench is a paired set of five visual reasoning tasks. The scene, question and answer stay fixed, and only the reference changes.
The result: Task-matched references improve accuracy by up to 25%, while mismatched ones give little benefit or even hurt performance.
Status: Published Oct 29 on arXiv and listed as #2 paper of the day on Hugging Face Papers. There's no discussion there yet.
Sign in to comment.
TL;DR of the paper, generated automatically.
The paper argues that multimodal language models struggle with multi-step visual reasoning because of a "reference gap," not a failure to see. Adding task-matched visual references, such as numbering, improves accuracy without any training.