What's Next After RLHF? — Diogo Almeida, TypeSafe AI
RLHF can make agents persuasive assistants without making them dependable autonomous decision-makers. Builders should separate human-pleasing interaction from calibrated automation and keep stakes bounded.
Almeida argues that **RLHF** optimizes human preference, which suits interactive assistants but can reward confident, agreeable behavior. He contrasts that with **RLVR**, which optimizes verifiable correctness, and describes a separate TypeSafe direction aimed at calibrated decisions.
When designing coding-agent workflows, distinguish assistance from unattended automation. Keep humans around consequential decisions, demand external evidence for completion, and avoid treating fluent interaction or benchmark strength as proof that an agent can own business-critical actions.
Almeida argues that **RLHF** optimizes human preference, which suits interactive assistants but can reward confident, agreeable behavior. He contrasts that with **RLVR**, which optimizes verifiable correctness, and describes a separate TypeSafe direction aimed at calibrated decisions. When designing coding-agent workflows, distinguish assistance from unattended automation. Keep humans around consequential decisions, demand external evidence for completion, and avoid treating fluent interaction or benchmark strength as proof that an agent can own business-critical actions. The talk presents a thesis rather than comparative evaluation data, and the proposed alternative is not technically specified. It does not establish how calibrated post-training performs, scales, or handles failures in deployed software.
This separates models optimized for agreeable assistance from systems trusted to act autonomously, making external verification and decision gates deployment requirements rather than optional polish. It supports bounded human oversight for consequential coding work, but the proposed calibrated-training direction remains a thesis without comparative evidence, implementation detail, or demonstrated production reliability.