# Reinforcement Learning without Verifiable Rewards — Will Brown, Prime Intellect

Source: [AI Engineer](https://www.youtube.com/watch?v=AQv3qRCG6Gw)  
Feed7 permalink: https://feed7.dev/p/reinforcement-learning-without-verifiable-rewards-will-brown-prime-intel-0mauwh9  
Published: 2026-07-31T15:00:06.000Z  
Trust: Source Linked (source_linked)

## Why Included

For agent work without deterministic answers, mine production traces into tasks, build controllable simulators, and use scaled judges to derive rubrics while watching for reward hacking.

## Source Summary

Deterministic rewards work for math answers, tests, linters, and known database states, but reports, purchases, and customer interactions lack one clean answer. The proposed loop uses **production traces**, **controllable simulators**, and **scaled judges** to construct training signal.

## Practical Implication

Treat deployed-agent traces as the emerging task distribution, then mine failures and calibrate task difficulty. In simulators, plant a known end state and work backward; use multiple judges and retrospective analysis to extract reusable rubrics before training.

## Agent-Ready Context

Deterministic rewards work for math answers, tests, linters, and known database states, but reports, purchases, and customer interactions lack one clean answer. The proposed loop uses **production traces**, **controllable simulators**, and **scaled judges** to construct training signal.

Treat deployed-agent traces as the emerging task distribution, then mine failures and calibrate task difficulty. In simulators, plant a known end state and work backward; use multiple judges and retrospective analysis to extract reusable rubrics before training.

Proxy rewards remain exploitable at their boundaries, and real deployments may contain information that reinforcement learning never discovers. The talk therefore adds supervised signal from the environment, while acknowledging that environment and reward design still require substantial manual work today.

## Connected Context

Feed7 judgment across 318 accumulated Signals:

This extends agent training beyond domains with exact rewards by turning production traces, controlled simulations, and calibrated multi-judge analysis into imperfect but usable supervision. It confirms simulations as a bridge between deployment and evaluation, while emphasizing that proxy rewards remain gameable, some environmental knowledge may not emerge through RL, and substantial human reward design is still required.

- [From Agent Traces to Agent Simulations — Rustem Feyzkhanov, Snorkel AI](https://feed7.dev/p/from-agent-traces-to-agent-simulations-rustem-feyzkhanov-snorkel-ai-0zwlzjq) — Replaying production traces as fixed environments directly implements the proposed move from emerging production tasks to comparable evaluation and training conditions.
- [SimulationMaxxing: How we ship agents 20× faster — Aman Gupta (Nubank) + Shreya Rajpal (Snowglobe)](https://feed7.dev/p/simulationmaxxing-how-we-ship-agents-20-faster-aman-gupta-nubank-shreya-0r3nm6u) — Nubank’s workflow supplies operational evidence for using simulated multi-turn traces while retaining real-data and human checks for sim-to-real alignment.
- [LLM-as-a-Verifier: A General-Purpose Verification Framework](https://feed7.dev/p/2607-05391v1-10hi3eo) — The verifier offers one scalable judge mechanism for non-binary outputs, but the target cautions that any such proxy must be calibrated and remains exploitable at its boundaries.
- [Harness Engineering is not Enough: Why Software Factories Fail — Dex Horthy, HumanLayer](https://feed7.dev/p/harness-engineering-is-not-enough-why-software-factories-fail-dex-horthy-1tj1amt) — The warning that automated loops can increase throughput without maintainability reinforces the target’s case for retained human judgment and supervised environmental signal.

## Context Map

- Layer: agent
- Domains: coding, research
- Topics: harness-engineering, agent-evals, agent-reliability

## Uncertainty

- Proxy rewards remain exploitable at their boundaries, and real deployments may contain information that reinforcement learning never discovers. The talk therefore adds supervised signal from the environment, while acknowledging that environment and reward design still require substantial manual work today.

## Agent Instruction

Use this item as source-backed context. Do not invent claims beyond the linked source. If this item conflicts with another source, call out the conflict.