A study tests ten frontier vision-language models as robot generalists, and closed-weight models come out ahead.
Scope: Embodied Agent Arena covers 1,000 cases drawn from 32 established sources, plus GeoProbe, a new 168-case benchmark for camera, motion, depth and scale estimation.
Results: Closed-weight models lead in all five domains. Open-weight models show useful spatial reasoning but substantially lower planning and manipulation success.
Failures: Extra computation doesn't fix everything. The paper says code repair can restore execution without finishing the task, and self-review can reverse correct judgments.
Sign in to comment.
TL;DR of the paper, generated automatically.
A study tests ten frontier vision-language models as robot generalists, and closed-weight models come out ahead.