Pretrained LLMs Are Surrounded by Task Experts feat Yulu Gan from MIT

Sometime there is an idea that just makes other weird research results make so much more sense and sort of cascade to explain a whole bunch of phenomenon. Neural thicket from Yulu Gan is one such paper that I absolutely loved from start to finish. The old mental model I had for pre-trained neural networked was that the pre-training regime made them land on a single good set of weight that was optimal for many tasks. But after going through the paper "Neural Thickets: Diverse Task Experts Are Dense Around Pretrained Weights" my view has been changed. The better idea seems to be that the mo