The paper splits the training-inference mismatch in LLM reinforcement learning into two sources and proposes a correction for each.
Two sources: Numerical discrepancy between the engine and trainer stays roughly constant and biases updates by a nearly fixed amount. Policy lag comes from the trainer's own updates and feeds into the next one.
The fix: The discrepancy-lag correction (DLC) centers the discrepancy, clips the lag ratio against the sampler's weights, and resynchronizes the sampler when effective sample size falls below one half.
Results: On Qwen3-0.6B and Qwen3-1.7B with a stale sampler, DLC keeps KL from on-policy training within twice the KL between two independent on-policy runs. The page says importance weighting alone fails on 1.7B once effective sample size drops below one half.
Sign in to comment.
TL;DR of the paper, generated automatically.
The paper splits the training-inference mismatch in LLM reinforcement learning into two sources and proposes a correction for each.