🤖 Pre-training scaling laws still hold — Colossus and Grok 3 are the proof

“But this is the first time they have not released a model that was a decisively better. And the progress for Grok 4 was exceptional. Grok 5 is coming soon, you know, they have more.”

“So this is incredible progress. And I think it's worth mentioning, this model was trained on Hopper. So this model is probably about as far as you can take the last generation of Nvidia GPUs. The next models we see, you know, Grok 5, you know, O5 from OpenAI, whatever they're going to call the next Gemini. They will be trained on Blackwell. And I think that will be a really big step function.”

“That's the global lesson, by the way. Chamath, you're totally right. Conceptual, the blog post is right. But that's only true when you have enough data and depending upon the use case, the level of data you need may not be possible for years, decades, and you may need to hack your way there through human interactions.”

“If you get, if you have an LM or foundational model of some kind that is the best in the world of the scientific method, game the F over. You basically, you just light up more GPUs and you just got like a thousand more PhD students working for you.”

“So why is this crazy? Well, he made this huge bet on this 100,000 GPU cluster. People thought, wow, that's a lot. Is it going to bear fruit? Then he said, no, actually, I'm scaling it up to 250,000. Then he said it's going to scale up to a million. And what these results show is a general computational approach that doesn't require as much human labeling, can actually get to the answer and better answers faster.”

“Let's talk about this ChatGPT. I guess they came out with 4.5. It was such a dud. I didn't even realize that they launched it.”

“Open AI dropped GPT-45, and I think it was not really that well received.”

“It's super consequential. I mean, complete agreement with Chamath”

“I mean, basically what he proved was that there are still valuable gains in pre-training, and so the larger the cluster, the more value that there is... My first takeaway was I was sneakily surprised on the pre-training upside on having a larger cluster. So I think that that's very pro-NVIDIA actually, and it's actually also just really good in general for foundational model makers.”

“Yeah, I guess the point is, a toy to do what? Because if you're trying to do inference, like everything is telling us that we are reaching the limits of training.”

“If you listen to Ilya Tsutskever, if you listen to Andrei Karpathy, what they effectively are saying is there's this terminal asymptote that we're seeing right now in model quality.”

“I think it's a new kind of scaling access arguably in terms of the potential set of applications. So networks of models, think time, context window. There are multiple dimensions upon which these tools ultimately kind of resolve to better performance”

“But you have not had a real test of scaling laws for training, arguably since GPT-4. And this will be the first test. And if scaling laws for training hold, Grok 3 should be a significant advance in the state of the art”