Advanced Micro Devices, Inc.

Gavin Baker support ×2 ▲ on 🤖 Pre-training scaling laws still hold — Colossus and Grok 3 are the proof E238 · 2025-08-09 ▶ 11:43

But this is the first time they have not released a model that was a decisively better. And the progress for Grok 4 was exceptional. Grok 5 is coming soon, you know, they have more.

Gavin Baker support ×3 ▲ on 🤖 Pre-training scaling laws still hold — Colossus and Grok 3 are the proof E236 · 2025-07-19 ▶ 20:18

So this is incredible progress. And I think it's worth mentioning, this model was trained on Hopper. So this model is probably about as far as you can take the last generation of Nvidia GPUs. The next models we see, you know, Grok 5, you know, O5 from OpenAI, whatever they're going to call the next Gemini. They will be trained on Blackwell. And I think that will be a really big step function.

That's the global lesson, by the way. Chamath, you're totally right. Conceptual, the blog post is right. But that's only true when you have enough data and depending upon the use case, the level of data you need may not be possible for years, decades, and you may need to hack your way there through human interactions.

If you get, if you have an LM or foundational model of some kind that is the best in the world of the scientific method, game the F over. You basically, you just light up more GPUs and you just got like a thousand more PhD students working for you.

Chamath support ×3 ▲ on 🤖 Pre-training scaling laws still hold — Colossus and Grok 3 are the proof E235 · 2025-07-11 ▶ 20:39

So why is this crazy? Well, he made this huge bet on this 100,000 GPU cluster. People thought, wow, that's a lot. Is it going to bear fruit? Then he said, no, actually, I'm scaling it up to 250,000. Then he said it's going to scale up to a million. And what these results show is a general computational approach that doesn't require as much human labeling, can actually get to the answer and better answers faster.

Chamath reversal ×3 ▼ on 🤖 A CUDA transpiler breaks Nvidia's software lock-in E232 · 2025-06-21 ▶ 22:14

When I first started 80-90 a year ago, one of the key bets I made, which was a mistake and we unwound the bet, but the first bet that I made was, can we build a transpiler? ... And basically what I learned in that process are all of the attention mechanisms that are built into transformers that really differentiate how good the models are, need to literally be hand-tuned for every single target of silicon that you have.

Chamath oppose ×1 ▲ on 🤖 Pre-training scaling laws still hold — Colossus and Grok 3 are the proof E218 · 2025-03-08 ▶ 6:45

Open AI dropped GPT-45, and I think it was not really that well received.

Jason oppose ×2 ▲ on 🤖 Pre-training scaling laws still hold — Colossus and Grok 3 are the proof E218 · 2025-03-08 ▶ 1:24:30

Let's talk about this ChatGPT. I guess they came out with 4.5. It was such a dud. I didn't even realize that they launched it.

Jason support ×1 ▲ on 🤖 Pre-training scaling laws still hold — Colossus and Grok 3 are the proof E216 · 2025-02-21 ▶ 1:29:36

It's super consequential. I mean, complete agreement with Chamath

Chamath support ×3 ▲ on 🤖 Pre-training scaling laws still hold — Colossus and Grok 3 are the proof E216 · 2025-02-21 ▶ 1:19:36

I mean, basically what he proved was that there are still valuable gains in pre-training, and so the larger the cluster, the more value that there is... My first takeaway was I was sneakily surprised on the pre-training upside on having a larger cluster. So I think that that's very pro-NVIDIA actually, and it's actually also just really good in general for foundational model makers.

Chamath support ×2 ▼ on 🤖 Demand mix shifts to inference and Nvidia has to choose margin or share E210 · 2025-01-11 ▶ 1:26:11

So in this world of AI that we know it today, there's training and there's inference. And right now we think that there's training that's at a limit. And so now the market shifts to inference... But it's not clear to me why that's a better solution than all of the AI accelerators plus tensors that are now just prolifically being exposed to the market, whether it's Amazon exposing what they've done, whether it's Google exposing what they've done, a whole litany of startups exposing what they've done.

Chamath oppose ×2 ▲ on 🤖 Pre-training scaling laws still hold — Colossus and Grok 3 are the proof E210 · 2025-01-11 ▶ 1:25:55

Yeah, I guess the point is, a toy to do what? Because if you're trying to do inference, like everything is telling us that we are reaching the limits of training.

Chamath oppose ×2 ▲ on 🤖 Pre-training scaling laws still hold — Colossus and Grok 3 are the proof E208 · 2024-12-20 ▶ 58:57

If you listen to Ilya Tsutskever, if you listen to Andrei Karpathy, what they effectively are saying is there's this terminal asymptote that we're seeing right now in model quality.

Friedberg support ×2 ▲ on 🤖 Pre-training scaling laws still hold — Colossus and Grok 3 are the proof E206 · 2024-12-07 ▶ 58:46

I think it's a new kind of scaling access arguably in terms of the potential set of applications. So networks of models, think time, context window. There are multiple dimensions upon which these tools ultimately kind of resolve to better performance

Gavin Baker support ×3 ▲ on 🤖 Pre-training scaling laws still hold — Colossus and Grok 3 are the proof E206 · 2024-12-07 ▶ 54:45

But you have not had a real test of scaling laws for training, arguably since GPT-4. And this will be the first test. And if scaling laws for training hold, Grok 3 should be a significant advance in the state of the art

Chamath support ×2 ▼ on 🤖 A CUDA transpiler breaks Nvidia's software lock-in E195 · 2024-09-06 ▶ 50:01

The problem with PyTorch and building to NVIDIA is you have this thing called CUDA in the middle of it, which is owned by NVIDIA, and I think that over time that's problematic.

But I do think that as you kind of scale the demand, there'll be a lot of inference chips coming in. You know, I think Chamath, you're invested in one of those. Oh, yeah. And I think there's going to be a bunch of those kind of coming in and the bulk of the demand will be on the inference side. And then Nvidia will have this challenge of, do I try to keep my prices and my margin? Or do I do what why we like competition? Do I have to respond to the competitive market? And then that, I think, will play out, you know, start playing out probably in a year, two at the latest, and then kind of go. So I think it's not sustain, the pure heat is not sustainable. But I think it's, you know, Nvidia has got a very strong position. And, you know, I definitely, I would recommend people not be short on Nvidia.

Jason support ×1 ▲ on 🤖 Nvidia forward-integrates into cloud and attacks the hyperscalers E180 · 2024-05-24 ▶ 36:55

And go ahead to head with AWS instead of selling to them.

Chamath support ×2 ▲ on 🤖 Nvidia forward-integrates into cloud and attacks the hyperscalers E180 · 2024-05-24 ▶ 36:58

It's not a far stretch, especially because Nvidia actually has the software interface that everybody uses, which is CUDA. So I think it's likely that Nvidia goes on a full frontal assault against GCP and Amazon and Microsoft. That's going to really complicate the relationship that those folks have with each other. But I think it's inevitable, because you're going to... How do you defend... It's kind of the Apple problem. How do you defend an enormously large market cap? You're forced to go into businesses that are equally lucrative.

Chamath support ×2 ▲ on 🤖 Moore's law never ended - it moved to GPUs and expert systems go next level E135 · 2023-07-01 ▶ 1:01:48

the list price of an H100 is about 30,000, but the street price, so it's very hard to get it. So if you go to eBay and try to buy an H100, it's like 40 or 45,000.

Sacks support ×2 ▲ on 🤖 Moore's law never ended - it moved to GPUs and expert systems go next level E135 · 2023-07-01 ▶ 54:21

We have a GPU shortage, and it's probably not going to get better for a year or two, if that.

Chamath support ×3 ▲ on 🤖 Moore's law never ended - it moved to GPUs and expert systems go next level E133 · 2023-06-16 ▶ 1:17:40

My God, these people that put the money in the seed round should have just bought NVIDIA. Buy some call options so you can make more money.

Friedberg support ×2 ▲ on 🤖 Moore's law never ended - it moved to GPUs and expert systems go next level E133 · 2023-06-16 ▶ 22:37

Certainly the most obvious is demand for chips because all this infrastructure is being built out.

Brad Gerstner support ×3 ▲ on 🤖 Moore's law never ended - it moved to GPUs and expert systems go next level E133 · 2023-06-16 ▶ 17:52

I don't think there's any problems with NVIDIA. I think they're going to continue to perform. I think they'll beat their numbers for the balance of the year.

Chamath support ×3 ▲ on 🤖 Moore's law never ended - it moved to GPUs and expert systems go next level E132 · 2023-06-10 ▶ 1:30:44

I've told this story before, but I pinged those guys and I was basically like, I just wanna meet the team that built the TPU. Long story short, a year later, I put them in business. And we've been building silicon for this moment for years.