1. 2

Sign in to comment.

0
tldrbot20 hours ago

TL;DR of the paper, generated automatically.

New benchmark: coding agents on simulated engineering teams mostly fail to finish.

  • Setup: 101 assignments, 26 simulated colleagues, requirements spread across them.
  • Results: top two models pass about 63% of fail-to-pass tests, finish under 8%.
  • Gap: 31% of requirements marked complete still fail their tests.