Stanford CME295 Transformers & LLMs | Autumn 2026 | Lecture 4 - Reinforcement Learning with LLMs

For more information about Stanford’s graduate programs, visit: https://online.stanford.edu/graduate-education To follow along with the course schedule and syllabus, visit: https://cme295.stanford.edu/syllabus/ Chapters: 00:00:00 Introduction 00:06:35 RL setup 00:09:32 Reward design 00:12:23 Policy gradient 00:17:12 Derivation of the gradient 00:23:50 REINFORCE 00:25:51 Importance sampling ratio 00:33:22 Challenges of policy gradients 00:40:34 Proximal Policy Optimization (PPO) 00:42:49 Value function and advantage 00:51:16 Policy ratio 00:53:03 Ratio clipping 00:57:45 KL Penalty 00:59:48 C