Reinforcement Learning without Verifiable Rewards — Will Brown, Prime Intellect
For agent work without deterministic answers, mine production traces into tasks, build controllable simulators, and use scaled judges to derive rubrics while watching for reward hacking.
Deterministic rewards work for math answers, tests, linters, and known database states, but reports, purchases, and customer interactions lack one clean answer. The proposed loop uses **production traces**, **controllable simulators**, and **scaled judges** to construct training signal.
Treat deployed-agent traces as the emerging task distribution, then mine failures and calibrate task difficulty. In simulators, plant a known end state and work backward; use multiple judges and retrospective analysis to extract reusable rubrics before training.
Deterministic rewards work for math answers, tests, linters, and known database states, but reports, purchases, and customer interactions lack one clean answer. The proposed loop uses **production traces**, **controllable simulators**, and **scaled judges** to construct training signal. Treat deployed-agent traces as the emerging task distribution, then mine failures and calibrate task difficulty. In simulators, plant a known end state and work backward; use multiple judges and retrospective analysis to extract reusable rubrics before training. Proxy rewards remain exploitable at their boundaries, and real deployments may contain information that reinforcement learning never discovers. The talk therefore adds supervised signal from the environment, while acknowledging that environment and reward design still require substantial manual work today.
This extends agent training beyond domains with exact rewards by turning production traces, controlled simulations, and calibrated multi-judge analysis into imperfect but usable supervision. It confirms simulations as a bridge between deployment and evaluation, while emphasizing that proxy rewards remain gameable, some environmental knowledge may not emerge through RL, and substantial human reward design is still required.