How tokenizers turn text into numbers

Name: How tokenizers turn text into numbers
Uploaded: 2026-09-28T12:00:19+00:00
Channel: Sebastian Raschka
Description: A tokenizer splits text into tokens and maps them to numeric IDs. I walk through this process with the Qwen3 tokenizer, then show how decoding reconstructs the original text. An excerpt from my reasoning model series: https://www.youtube.com/watch?v=Kh9mqTzjuEQ&list=PLfrHAQCRxtUs&index=1

Sebastian RaschkaCréateur · il y a 25 jours

Voir sur YouTube

A tokenizer splits text into tokens and maps them to numeric IDs. I walk through this process with the Qwen3 tokenizer, then show how decoding reconstructs the original text. An excerpt from my reasoning model series: https://www.youtube.com/watch?v=Kh9mqTzjuEQ&list=PLfrHAQCRxtUs&index=1

Discussion0

Rejoignez la communauté pour réagir et poser vos questions.

Connexion Rejoindre