Why You Can’t Run That AI Model

Why can't you run an AI model even when you have enough storage? Because an AI model is just billions of numbers, and "8B" means 8 billion parameters, not 8 gigabytes. At full precision (16 bits each), those 8 billion numbers take about 16GB. Compress them to 4 bits each, called quantization, and you're down to roughly 4GB plus some overhead. The catch: the whole model has to sit in fast memory your GPU can reach, not your SSD, not your hard drive, not even regular system RAM. Having space to download a model doesn't mean you can actually run it. #techwithtim #ai #machinelearning