What's Next After RLHF? — Diogo Almeida, TypeSafe AI
RLHF can make agents persuasive assistants without making them dependable autonomous decision-makers. Builders should separate human-pleasing interaction from calibrated automation and keep stakes bounded.
Almeida argues that **RLHF** optimizes human preference, which suits interactive assistants but can reward confident, agreeable behavior. He contrasts that with **RLVR**, which optimizes verifiable correctness, and describes a separate TypeSafe direction aimed at calibrated decisions.
When designing coding-agent workflows, distinguish assistance from unattended automation. Keep humans around consequential decisions, demand external evidence for completion, and avoid treating fluent interaction or benchmark strength as proof that an agent can own business-critical actions.
Almeida argues that **RLHF** optimizes human preference, which suits interactive assistants but can reward confident, agreeable behavior. He contrasts that with **RLVR**, which optimizes verifiable correctness, and describes a separate TypeSafe direction aimed at calibrated decisions. When designing coding-agent workflows, distinguish assistance from unattended automation. Keep humans around consequential decisions, demand external evidence for completion, and avoid treating fluent interaction or benchmark strength as proof that an agent can own business-critical actions. The talk presents a thesis rather than comparative evaluation data, and the proposed alternative is not technically specified. It does not establish how calibrated post-training performs, scales, or handles failures in deployed software.
This shifts the autonomy question away from conversational quality or headline model strength toward the objective being optimized and the evidence available at execution time. It supports bounded human oversight and external verification for consequential coding work, while leaving the proposed calibrated-training alternative too unspecified to justify a model-selection conclusion.