What's Next After RLHF? — Diogo Almeida, TypeSafe AI
RLHF can make agents persuasive assistants without making them dependable autonomous decision-makers. Builders should separate human-pleasing interaction from calibrated automation and keep stakes bounded.
Almeida argues that **RLHF** optimizes human preference, which suits interactive assistants but can reward confident, agreeable behavior. He contrasts that with **RLVR**, which optimizes verifiable correctness, and describes a separate TypeSafe direction aimed at calibrated decisions.
When designing coding-agent workflows, distinguish assistance from unattended automation. Keep humans around consequential decisions, demand external evidence for completion, and avoid treating fluent interaction or benchmark strength as proof that an agent can own business-critical actions.
Almeida argues that **RLHF** optimizes human preference, which suits interactive assistants but can reward confident, agreeable behavior. He contrasts that with **RLVR**, which optimizes verifiable correctness, and describes a separate TypeSafe direction aimed at calibrated decisions. When designing coding-agent workflows, distinguish assistance from unattended automation. Keep humans around consequential decisions, demand external evidence for completion, and avoid treating fluent interaction or benchmark strength as proof that an agent can own business-critical actions. The talk presents a thesis rather than comparative evaluation data, and the proposed alternative is not technically specified. It does not establish how calibrated post-training performs, scales, or handles failures in deployed software.
This separates optimization for agreeable assistance from the evidence and calibration needed for delegated action. Against the candidates, it supports bounded autonomy, external verification, and human control at consequential decisions; it also explains why model tiers, fluent behavior, or benchmark results alone cannot justify operational trust. The post-training alternative remains a design thesis, not a validated replacement.