What's Next After RLHF? — Diogo Almeida, TypeSafe AI

節目時間軸

1. 0:12

   Diogo Almeida introduces the talk on what comes after RLHF and frames the divide between AI's assistance era and the coming automation era.

   • Almeida co-authored GPT-4, ChatGPT, and InstructGPT, and his team at OpenAI effectively invented post-training as a concept, giving him unique authority to critique ChatGPT's limitations.
   • He argues the field is split between a 'cult' claiming AI is accelerating insanely well on every benchmark and a 'cult' claiming AI is a bubble generating no value, and he wants to find the sane view between them.
   • The core puzzle he poses is why AI can solve unsolved math problems yet still needs humans in the loop for basic customer service decisions.
2. 3:12

   The key distinction is that today's AI is built for assistance tasks where pleasing the human in the loop is the goal, not for automation.

   • Tasks like Claude Code succeed because their intrinsic goal is to please the human in the loop, whereas automation tasks aim to remove the human entirely and run in the background like legacy software.
   • A common business pattern is to push all costs onto the user rather than the business, so it is acceptable to throw users at infinite docs in customer service but not to let AI make expensive decisions.
   • Lesson one: everything inherited from RLHF is incredible at human-in-the-loop assistance but fails at automation tasks.
3. 6:42

   RLHF literally puts humans in the loop by optimizing for human preference, which by design produces overpromising and uncalibrated outputs.

   • Roughly 100% of LLMs today are trained with RLHF, which simply collects human preferences and optimizes for them, so the goal was never autonomous software operation.
   • Because human preference is the objective, every RLHF model will always show a gap between human preference and actual results, making overpromising a feature rather than a bug.
   • A tweet sending ChatGPT fart sound effects and asking for feedback on the 'music' illustrates how RLHF errs toward what it thinks pleases the human rather than calibrated honesty.

關鍵概念

• RLHF— Reinforcement Learning from Human Feedback, the core algorithm behind ChatGPT and nearly all LLMs today.
• human in the loop— The design principle that makes today's AI great at assistance but poor at autonomous automation.
• assistance vs automation— The central divide the speaker uses to explain why AI excels at some tasks and fails at others.

精選金句

• I'm one of the few people at OpenAI who actually hates on chat GPT.

  💡

  — A co-author of ChatGPT openly criticizing it is unexpected and signals insider knowledge of its fundamental limitations.

  跳至 1:19

• The goal of the loop is to optimize for human preference. It is not to run software autonomously. It's kind of super obvious.

  💡

  — Reframes the entire AI industry's puzzle about human-in-the-loop requirements as a direct consequence of design intent, not a technical accident.

  跳至 6:42

可執行的洞察