Reinforcement Learning without Verifiable Rewards — Will Brown, Prime Intellect
For agent work without deterministic answers, mine production traces into tasks, build controllable simulators, and use scaled judges to derive rubrics while watching for reward hacking.
Deterministic rewards work for math answers, tests, linters, and known database states, but reports, purchases, and customer interactions lack one clean answer. The proposed loop uses **production traces**, **controllable simulators**, and **scaled judges** to construct training signal.
Treat deployed-agent traces as the emerging task distribution, then mine failures and calibrate task difficulty. In simulators, plant a known end state and work backward; use multiple judges and retrospective analysis to extract reusable rubrics before training.
Deterministic rewards work for math answers, tests, linters, and known database states, but reports, purchases, and customer interactions lack one clean answer. The proposed loop uses **production traces**, **controllable simulators**, and **scaled judges** to construct training signal. Treat deployed-agent traces as the emerging task distribution, then mine failures and calibrate task difficulty. In simulators, plant a known end state and work backward; use multiple judges and retrospective analysis to extract reusable rubrics before training. Proxy rewards remain exploitable at their boundaries, and real deployments may contain information that reinforcement learning never discovers. The talk therefore adds supervised signal from the environment, while acknowledging that environment and reward design still require substantial manual work today.
This extends agent training beyond domains with exact rewards by turning production traces, backward-built simulations, multiple judges, and retrospective rubrics into provisional supervision. It confirms simulation as a bridge from deployment failures to training tasks, but keeps the boundary clear: proxy rewards remain exploitable, simulations may omit real information, and substantial human environment design is still required.