The paper asks whether rewarding how a small model reasons, not just whether its final answer is right, helps it improve itself. It uses entropy and confidence signals to tell apart reasoning traces that are equally correct.
Online setup: Qwen3-1.7B adapts on 30 AIME problems with no reference answers. The UID-Conj. variant beats unmodified TTRL by 2.08 points on the macro-average across the adaptation benchmark and three transfer benchmarks.
Offline setup: Weighted SFT on correct responses from historical AIME problems does best, with UID-Conj. beating uniform weighting by 1.93 points (paired prompt-bootstrap 95% CI [0.44, 3.44]).
Task dependence: The paper says the gains vary by task. SC rewards judged trace quality and instruction following, while UID-Conj. rewards gave stronger training outcomes in several settings.
Small model, no answer key, and still some gains. Go figure.
Sign in to comment.
TL;DR of the paper, generated automatically.
The paper asks whether rewarding how a small model reasons, not just whether its final answer is right, helps it improve itself. It uses entropy and confidence signals to tell apart reasoning traces that are equally correct.
Small model, no answer key, and still some gains. Go figure.