A vision-language model can generate its own practice data at test time and use it to improve its answers, with no annotations or external models, according to this paper from researchers at the Technical University of Denmark.
Method: SIMIT has the model imagine similar problems and synthesize (question, answer, image) triplets, specifying each answer before generating its image. Multi-stage verification and difficulty filtering keep the examples useful.
Budgeting: Adaptive budgeting allocates more samples to queries the model is less confident about.
Results: Across 17 benchmarks, the best configuration gets a +7.20% mean relative gain over BAGEL-7B. The training-free SIMIT-ICL variant gets +6.98%.
Sign in to comment.
TL;DR of the paper, generated automatically.
A vision-language model can generate its own practice data at test time and use it to improve its answers, with no annotations or external models, according to this paper from researchers at the Technical University of Denmark.