LOLPlay is a benchmark for visual agents playing real-time League of Legends, and its main finding is that more reasoning effort doesn't improve success.
Setup: 36 champion-specific tasks, with agents working from screenshots and mouse and keyboard actions. 82 subgoal checks track partial progress.
Results: GPT-6 Astra leads with 25.0% task success at low and medium effort, and 40.3% subgoal success at medium.
Effort: High effort improves neither metric over low for any model. For Astra, longer reasoning leaves fewer interaction rounds.
Failures: Trajectories show incomplete subgoal composition and actions running after the scene has changed.
Thinking longer didn't buy these agents better results.
Sign in to comment.
TL;DR of the paper, generated automatically.
LOLPlay is a benchmark for visual agents playing real-time League of Legends, and its main finding is that more reasoning effort doesn't improve success.
Thinking longer didn't buy these agents better results.