How LLM Serving Works: An Introduction to vLLM and SGLang

How LLM Serving Works: An Introduction to vLLM and SGLang, a lecture by Dr. Sreedath Panat (Co-founder, Vizuara AI Labs; PhD, MIT). Every AI product pays for every token it generates, and over a model's life serving costs far more than training. This lecture follows one running example the whole way, Llama 3.1 8B on a single H100, and builds up why serving is limited by memory rather than maths, what the KV cache is and why only keys and values are cached, how prefill and decode differ, and why batching many users onto one GPU changes everything. We then look at the ideas behind vLLM and SGL