# Learning on the Job: The Future of Post-Training — Raymond Feng, Applied Compute

Source: [AI Engineer](https://www.youtube.com/watch?v=k35LeKZEhiE)  
Feed7 permalink: https://feed7.dev/p/learning-on-the-job-the-future-of-post-training-raymond-feng-applied-com-17u0m7t  
Published: 2026-07-31T22:30:06.000Z  
Trust: Source Linked (source_linked)

## Why Included

Training on a production harness could adapt an agent to real workflows, but non-replayable interactions break common RL assumptions and make feedback harder to turn into updates.

## Source Summary

Replayable synthetic environments support **GRPO** by running several trajectories from the same initial state, grading them, and adjusting model weights toward the better paths.

## Practical Implication

For a custom agent, preserve traces and feedback at the model boundary. A **bring-your-own-harness** setup can train on the actual workflow instead of a brittle simulation, even when the training team cannot inspect its source.

## Agent-Ready Context

Replayable synthetic environments support **GRPO** by running several trajectories from the same initial state, grading them, and adjusting model weights toward the better paths.

For a custom agent, preserve traces and feedback at the model boundary. A **bring-your-own-harness** setup can train on the actual workflow instead of a brittle simulation, even when the training team cannot inspect its source.

Production interactions may be non-replayable and off-policy, so conventional comparative rollouts no longer fit. **Self-distillation**, automated trace curation, and qualitative-feedback ingestion remain open research directions rather than general solutions.

## Connected Context

Feed7 judgment across 318 accumulated Signals:

This makes production traces and harness feedback potential post-training data, extending self-improvement from observability into weight updates on the real workflow. Replayable environments fit comparative RL, but live interactions often do not, so bring-your-own-harness training depends on preserving model-boundary evidence and careful curation. Self-distillation and qualitative feedback remain unresolved paths, not established substitutes.

- [Why Off-the-Shelf AI Doesn't Understand Money — Udi Menkes, Intuit](https://feed7.dev/p/why-off-the-shelf-ai-doesn-t-understand-money-udi-menkes-intuit-0y6w9rk) — Intuit’s verified state-action-outcome histories specify the kind of grounded production record that harness-based learning would need, while adding causal and data-quality constraints.
- [From Signal to PR: Anatomy of a Self-Improving Agent — Jason Lopatecki, Arize](https://feed7.dev/p/from-signal-to-pr-anatomy-of-a-self-improving-agent-jason-lopatecki-ariz-1i6f0ba) — Arize already uses production traces to generate investigated issues and draft fixes; this signal extends the same evidence stream toward post-training model weights.
- [Data and Environment Curation for Post-Training LLMs — Mahesh Sathiamoorthy, Bespoke Labs](https://feed7.dev/p/data-and-environment-curation-for-post-training-llms-mahesh-sathiamoorth-19ab77a) — Bespoke Labs narrows the implied path to improvement by arguing that curated SFT may capture much of the value before teams incur the complexity of RL.
- [Reinforcement Learning without Verifiable Rewards — Will Brown, Prime Intellect](https://feed7.dev/p/reinforcement-learning-without-verifiable-rewards-will-brown-prime-intel-0mauwh9) — Prime Intellect addresses the same non-verifiable-work gap with mined traces, controllable simulators, and judge-derived rubrics, while highlighting reward hacking as a risk for the proposed feedback loop.

## Context Map

- Layer: agent
- Domains: None
- Topics: harness-engineering, agent-reliability, tool-use

## Uncertainty

- Production interactions may be non-replayable and off-policy, so conventional comparative rollouts no longer fit. **Self-distillation**, automated trace curation, and qualitative-feedback ingestion remain open research directions rather than general solutions.

## Agent Instruction

Use this item as source-backed context. Do not invent claims beyond the linked source. If this item conflicts with another source, call out the conflict.