Why AI Doesn't Actually Read Words

Language models don’t read words the way we do. But, how do they understand text so well? This video follows the evolution from word and subword tokens (BPE) to byte-level models, including ByT5, CANINE, MegaByte, BLT, and H-Net. And explore how models may learn their own adaptive units directly from data in the future! Chapters: 0:00 Words, token IDs, and embeddings 3:25 Byte-Pair Encoding (BPE) 6:24 Tokenizer-free models 7:46 ByT5: reading raw bytes 9:19 CANINE: character-level compression 10:26 MegaByte: local and global byte modeling 11:00 BLT: adaptive byte patches 15:03 H-Net: learn