Why AI Evaluations Are Broken and How to Fix Them (with David Manheim)

David Manheim is head of methodology at AI Evaluation Consensus. He joins the podcast to discuss how AI evaluations can become more reliable, transparent, and useful for decisions. We cover common failures such as unclear reporting, training to the test, benchmark saturation, and models changing behavior when they know they are being tested. The conversation also examines real-world tests, biosecurity, persuasion, forecasting, human oversight, and why even “normal” AI progress could be disruptive. LINKS: David Manheim website AI Evaluation Consensus Statement CHAPTERS: (00:00) Episode Preview