🤖 Pre-training scaling laws still hold — Colossus and Grok 3 are the proof

Gavin Baker (regular guest ×1.0) tech w=1.46 · n=13 · E206 (2024-12-07) · explicit_prediction · strength 3 ▶ 54:45 SUPPORT

“But you have not had a real test of scaling laws for training, arguably since GPT-4. And this will be the first test. And if scaling laws for training hold, Grok 3 should be a significant advance in the state of the art”

Friedberg tech w=1.08 · n=65 · E206 (2024-12-07) · sentiment · strength 2 ▶ 58:46 SUPPORT

“I think it's a new kind of scaling access arguably in terms of the potential set of applications. So networks of models, think time, context window. There are multiple dimensions upon which these tools ultimately kind of resolve to better performance”

Chamath tech w=0.90 · n=83 · E208 (2024-12-20) · sentiment · strength 2 ▶ 58:57 OPPOSE

“If you listen to Ilya Tsutskever, if you listen to Andrei Karpathy, what they effectively are saying is there's this terminal asymptote that we're seeing right now in model quality.”

Chamath tech w=0.90 · n=83 · E210 (2025-01-11) · sentiment · strength 2 ▶ 1:25:55 OPPOSE

“Yeah, I guess the point is, a toy to do what? Because if you're trying to do inference, like everything is telling us that we are reaching the limits of training.”

Chamath tech w=0.90 · n=83 · E216 (2025-02-21) · explicit_prediction · strength 3 ▶ 1:19:36 SUPPORT

“I mean, basically what he proved was that there are still valuable gains in pre-training, and so the larger the cluster, the more value that there is... My first takeaway was I was sneakily surprised on the pre-training upside on having a larger cluster. So I think that that's very pro-NVIDIA actually, and it's actually also just really good in general for foundational model makers.”

Jason tech w=1.05 · n=56 · E216 (2025-02-21) · sentiment · strength 1 ▶ 1:29:36 SUPPORT

“It's super consequential. I mean, complete agreement with Chamath”

Jason tech w=1.05 · n=56 · E218 (2025-03-08) · sentiment · strength 2 ▶ 1:24:30 OPPOSE

“Let's talk about this ChatGPT. I guess they came out with 4.5. It was such a dud. I didn't even realize that they launched it.”

Chamath tech w=0.90 · n=83 · E218 (2025-03-08) · sentiment · strength 1 ▶ 6:45 OPPOSE

“Open AI dropped GPT-45, and I think it was not really that well received.”

Chamath tech w=0.90 · n=83 · E235 (2025-07-11) · explicit_prediction · strength 3 ▶ 20:39 SUPPORT

“So why is this crazy? Well, he made this huge bet on this 100,000 GPU cluster. People thought, wow, that's a lot. Is it going to bear fruit? Then he said, no, actually, I'm scaling it up to 250,000. Then he said it's going to scale up to a million. And what these results show is a general computational approach that doesn't require as much human labeling, can actually get to the answer and better answers faster.”

Travis Kalanick (regular guest ×1.0) tech w=1.33 · n=6 · E235 (2025-07-11) · explicit_prediction · strength 2 ▶ 36:11 SUPPORT

“If you get, if you have an LM or foundational model of some kind that is the best in the world of the scientific method, game the F over. You basically, you just light up more GPUs and you just got like a thousand more PhD students working for you.”

Keith Rabois (regular guest ×1.0) tech: 4 resolved, unproven · E235 (2025-07-11) · sentiment · strength 2 ▶ 46:04 OPPOSE

“That's the global lesson, by the way. Chamath, you're totally right. Conceptual, the blog post is right. But that's only true when you have enough data and depending upon the use case, the level of data you need may not be possible for years, decades, and you may need to hack your way there through human interactions.”

Gavin Baker (regular guest ×1.0) tech w=1.46 · n=13 · E236 (2025-07-19) · explicit_prediction · strength 3 ▶ 20:18 SUPPORT

“So this is incredible progress. And I think it's worth mentioning, this model was trained on Hopper. So this model is probably about as far as you can take the last generation of Nvidia GPUs. The next models we see, you know, Grok 5, you know, O5 from OpenAI, whatever they're going to call the next Gemini. They will be trained on Blackwell. And I think that will be a really big step function.”

Gavin Baker (regular guest ×1.0) tech w=1.46 · n=13 · E238 (2025-08-09) · explicit_prediction · strength 2 ▶ 11:43 SUPPORT

“But this is the first time they have not released a model that was a decisively better. And the progress for Grok 4 was exceptional. Grok 5 is coming soon, you know, they have more.”