Dwarkesh Patel on Distillation, Mythos and RL

Dwarkesh Patel published rough technical notes covering five topics that cut across the most active debates in AI right now. The distillation section is particularly sharp - at $25 per million tokens, competitors can replicate frontier model behavior faster than labs can prevent it, and coding products using "gold diffs" as RL targets might actually produce distilled models that outperform the originals. He also digs into pretraining failure modes, noting the original GPT-4 had FP16 precision bugs in gradient accumulation, and walks through why GPU scaling hits a wall around 1K nodes due to batch size constraints. The Mythos analysis frames Anthropic's cybersecurity model as a vulnerability-chaining agent rather than a raw intelligence leap, which reframes the disclosure debate around it.