The paper introduces BedBench, a benchmark that shows vision-language models can judge 3D object properties correctly yet fail to recover and consistently use exact per-color counts.
Benchmark: Thirteen procedurally generated tasks, run across twelve open-weight and four frontier models.
Biggest gap: Per-color counting, where humans hit 93.3% and the best model reaches 7.2%.
Fine-tuning: One model rose from 3.0% to 68.6%, but its answers about the most common colors still clash with its own per-color counts.
Image dependence: Image-removal controls show some apparent successes need no visual evidence. Code and dataset are public on GitHub.
Recovering a property, the authors note, doesn't mean the model uses it.
Sign in to comment.
TL;DR of the paper, generated automatically.
The paper introduces BedBench, a benchmark that shows vision-language models can judge 3D object properties correctly yet fail to recover and consistently use exact per-color counts.
Recovering a property, the authors note, doesn't mean the model uses it.