🤖 Open-source models commoditize the model layer and move to the edge

Friedberg tech w=1.08 · n=65 · E129 (2023-05-19) · explicit_prediction · strength 3 ▶ 30:30 SUPPORT

“So let me just push back on that, because Mosaic ML published this model that is, let me, I can pull up the performance chart, or Nick, maybe you can just find it on their website real quick of the new model they published. Chamath, they trained this model on open source data that's publicly available, and they spent $200,000 on a cluster run to build this model, and look at how it performs compared to some of the top models that are closed source.”

Jason tech w=1.05 · n=56 · E129 (2023-05-19) · explicit_prediction · strength 2 ▶ 17:06 SUPPORT

“Many hands makes for light work. The open source models are going to fit on your phone or the latest, you know, Apple silicon. So I think the cat's out of the bag. I don't know how they pull it back in.”

Chamath tech w=0.90 · n=83 · E129 (2023-05-19) · explicit_prediction · strength 3 ▶ 31:52 OPPOSE

“So I don't understand these actual tests, but I don't think it's true that you could take this model and these model weights, apply to a different set of data and get useful answers.”

Chamath tech w=0.90 · n=83 · E131 (2023-06-02) · sentiment · strength 2 ▶ 1:24:35 SUPPORT

“the atomization of these models is happening, I think, even faster than when you first brought that up.”

Jason tech w=1.05 · n=56 · E131 (2023-06-02) · explicit_prediction · strength 2 ▶ 1:24:46 SUPPORT

“Yeah, so what took two or three million dollars last year is now taking $200,000 just through software improvements and code improvements.”

Friedberg tech w=1.08 · n=65 · E131 (2023-06-02) · explicit_prediction · strength 2 ▶ 1:26:10 SUPPORT

“Parameter reduction and fine-tuning seems to get to the point that we can run these things on small machines cheaply, quickly. And it'll change the applications.”

Friedberg tech w=1.08 · n=65 · E132 (2023-06-10) · explicit_prediction · strength 3 ▶ 1:27:04 SUPPORT

“You're gonna invest in a company at a $500 million valuation and six weeks later, it's gonna be worth zero because someone opens source the exact same thing that you can now do for 100K.”

Chamath tech w=0.90 · n=83 · E132 (2023-06-10) · sentiment · strength 2 ▶ 1:30:44 SUPPORT

“Everything in the middle, I think what Friedberg said is true, which is today it looks like it's worth a couple of billion dollars, tomorrow's worth nothing. And so I think you have to be very careful.”

Friedberg tech w=1.08 · n=65 · E133 (2023-06-16) · explicit_prediction · strength 3 ▶ 1:34:42 SUPPORT · horizon 18mo

“If they spent $400 million in the last couple of years, you could probably assume that doing the same training exercise could be done for five to $10 million 18 months from now to generate the same model.”

Chamath tech w=0.90 · n=83 · E133 (2023-06-16) · sentiment · strength 2 ▶ 1:34:42 BRUSH_OFF

“And all the people that say, I'm sure there's going to be some genius in the comments, but what about open source? And it's like, what about it?”

Jason tech w=1.05 · n=56 · E135 (2023-07-01) · explicit_prediction · strength 2 ▶ 56:58 SUPPORT

“they're trialing, Friedberg, on average, six, seven, eight language models before they pick one, and they're not picking OpenAI every time.”

Sacks tech w=0.77 · n=53 · E135 (2023-07-01) · explicit_prediction · strength 3 ▶ 1:07:53 SUPPORT

“That's where all the action is right now is customizing these open source models that then leads to basically be able to get the right inferences.”

Sacks tech w=0.77 · n=53 · E143 (2023-08-25) · explicit_prediction · strength 2 ▶ 30:45 SUPPORT

“This is why I think open source is taking off in a big way, is that I think big enterprises would much rather roll their own models and control it and do the fine tuning.”

Chamath tech w=0.90 · n=83 · E143 (2023-08-25) · explicit_prediction · strength 3 ▶ 31:09 SUPPORT

“if it's happening at the software layer already now, just like it did in Web2 software, and then we see certain elements of Web2 hardware been open sourced, I think it makes pretty logical sense that you can expect the same things to happen in the AI world.”

Sacks tech w=0.77 · n=53 · E152 (2023-11-03) · explicit_prediction · strength 2 ▶ 58:01 OPPOSE

“Because one of the big targets here is going to be open source software. So if you are, for example, open AI, which is no longer open, it's closed source, the number one thing you want to do is pull up the ladder before open source software can get a lot of momentum.”

Chamath tech w=0.90 · n=83 · E152 (2023-11-03) · sentiment · strength 2 ▶ 1:04:37 SUPPORT

“Yeah, before I give all the credit to Facebook, I'd rather say that I think there are a lot of open source alternatives, including Mistral, that I think are much better.”

Friedberg tech w=1.08 · n=65 · E152 (2023-11-03) · sentiment · strength 2 ▶ 59:27 SUPPORT

“And by the way, Lama 2 is out there. Hardware is out there. All the tooling is out there for others to start to get well ahead.”

Chamath tech w=0.90 · n=83 · E156 (2023-12-08) · explicit_prediction · strength 3 ▶ 1:08:38 SUPPORT

“There's going to be a proliferation of foundational models. The cost of those models will go to zero.”

Friedberg tech w=1.08 · n=65 · E159 (2023-12-29) · explicit_prediction · strength 2 ▶ 58:35 SUPPORT

“My best new tech of this year, I think, is really important as we race to keep the promise of AI alive in the face of increasing government regulation, which is open source, locally run LLMs.”

Chamath tech w=0.90 · n=83 · E165 (2024-02-09) · explicit_prediction · strength 3 ▶ 42:14 SUPPORT

“I think foundational models will have no economic value. I think that they will be an incredibly powerful part of the substrate, and they will be broadly available and entirely free... But I think open source models will basically crush the value of models to zero economically, even though the utility will go to infinity, the economic value will go to zero.”

Sacks tech w=0.77 · n=53 · E165 (2024-02-09) · explicit_prediction · strength 3 ▶ 49:47 OPPOSE

“Most of the tests show that OpenAI is still ahead of the open source models. I think even people in the open source movement will tell you that OpenAI is, call it six months ahead... Nonetheless, if OpenAI just maintains a little bit of a lead over open source, then it could... It could basically win the vast majority of the call it consumer search or consumer GPT market.”

Friedberg tech w=1.08 · n=65 · E166 (2024-02-16) · explicit_prediction · strength 2 ▶ 27:47 SUPPORT

“these models are getting chopped up in a way that you can run small models in a highly trained way locally, and you don't need a massive general purpose model”

Jason tech w=1.05 · n=56 · E167 (2024-02-23) · sentiment · strength 2 ▶ 56:08 SUPPORT

“People are just not going to want a model that has all this baked in weird bias. They're going to want something that's open source.”

Friedberg tech w=1.08 · n=65 · E167 (2024-02-23) · explicit_prediction · strength 2 ▶ 56:22 SUPPORT

“eventually everyone will find something that they don't want or that they're not expecting, and they're going to say, I don't want to use this product anymore. And so it is actually an opportunity for many models to proliferate, for open source to win.”

Sacks tech w=0.77 · n=53 · E168 (2024-03-01) · sentiment · strength 2 ▶ 1:02:58 SUPPORT

“Well, so what Zuck said on the last Meta call is the reason we open source everything is because we don't directly sell AI. We create products that AI makes better. So by open sourcing this, we allow the community to advance the ball and we get to reincorporate those changes. So it's a very smart strategy for companies that aren't directly selling the AI.”

Chamath tech w=0.90 · n=83 · E168 (2024-03-01) · sentiment · strength 2 ▶ 1:07:20 SUPPORT

“By the way, did you guys see there was a meta demo, which I thought was really cool, which was it was run on Lama 70B, but it was a real-time translation tool, where the person was speaking in Hokkien Chinese and the other person was speaking in English and they were able to understand each other.”

Sacks tech w=0.77 · n=53 · E169 (2024-03-08) · sentiment · strength 2 ▶ 1:06:15 SUPPORT

“But this isn't hard, right? I mean, all they got to do is just take the latest open source models and figure out how to customize them for their own products.”

Sacks tech w=0.77 · n=53 · E171 (2024-03-22) · sentiment · strength 2 ▶ 28:59 SUPPORT

“You're standing on the shoulders of the whole open source movement. All you have to do, like Chamath said, all you got to do is take the latest Mistral model”

Chamath tech w=0.90 · n=83 · E171 (2024-03-22) · sentiment · strength 2 ▶ 29:11 SUPPORT

“Not only are you not starting from zero, not only do you have the foundational models that are excellent and available in open source”

Friedberg tech w=1.08 · n=65 · E171 (2024-03-22) · explicit_prediction · strength 2 ▶ 48:01 SUPPORT

“there's been a realization on how quickly foundational model development is commoditizing and how quickly costs are escalating and how many folks are chasing it.”

Jason tech w=1.05 · n=56 · E171 (2024-03-22) · explicit_prediction · strength 2 ▶ 51:00 SUPPORT

“I would take the top open source projects. I'd find those top contributors, take the top 20 or so open source projects and back them to the tune of significant money, 50 million, 100 million, whatever it takes”

Sacks tech w=0.77 · n=53 · E174 (2024-04-12) · explicit_prediction · strength 2 ▶ 1:02:19 OPPOSE

“the incumbents who already have all the training data, they may have to pay a fine, there'll be some slap on the wrist, but now you've created a real moat and barrier to entry and so secretly they're going to love it. And it's going to be a big problem, I think, for the open source movement”

Chamath tech w=0.90 · n=83 · E176 (2024-04-26) · explicit_prediction · strength 3 ▶ 10:00 SUPPORT

“We're seeing the economic value getting disintegrated. There is no value in foundational models economically. So then the question is who can build on top of them the fastest?”

Sacks tech w=0.77 · n=53 · E176 (2024-04-26) · explicit_prediction · strength 3 ▶ 14:04 SUPPORT

“which they had spent billions of dollars creating in a completely open source way. And the testing on Llama 3 is that it's comparable to GPT-4. And I think this is what Chamath means by scorched earth, is that we now have a free model that's as good as GPT-4.”

Friedberg tech w=1.08 · n=65 · E176 (2024-04-26) · explicit_prediction · strength 2 ▶ 12:07 SUPPORT

“by open sourcing these models, they limit competition because VCs are no longer going to plow half a billion dollars into a foundational model development company. So you limit the commercial interest and the commercial value of foundational models.”

Jason tech w=1.05 · n=56 · E176 (2024-04-26) · sentiment · strength 2 ▶ 17:16 SUPPORT

“I agree with all you've said here in terms of the open source and the dramatic effect it's going to have on pricing”

Sacks tech w=0.77 · n=53 · E178 (2024-05-10) · sentiment · strength 2 ▶ 8:20 SUPPORT

“when Llama 3 got released, I think the big takeaway for a lot of people was, oh, wow, they've like caught up to GPT-4. I don't think it's equal in all dimensions, but it's like pretty close or pretty in the ballpark.”

Sam Altman (guest ×0.5) no tech track record · E178 (2024-05-10) · explicit_prediction · strength 2 ▶ 8:51 OPPOSE

“What we're trying to do is not make the sort of smartest set of weights that we can. What we're trying to make is like this useful intelligence layer for people to use. And a model is part of that. I think we will stay pretty far ahead of, I hope, we'll stay pretty far ahead of the rest of the world on that.”

Chamath tech w=0.90 · n=83 · E178 (2024-05-10) · explicit_prediction · strength 3 ▶ 1:05:46 SUPPORT

“these models will roughly all be the same, but there's going to be a lot of scaffolding around these models that actually allow you to build these apps. So in many ways, that is like the open source movement. So even if the model itself is never open source, it doesn't much matter because you have to pay for the infrastructure.”

Chamath tech w=0.90 · n=83 · E179 (2024-05-17) · explicit_prediction · strength 2 ▶ 27:36 SUPPORT

“So I also think, Sacks, the incentive to just push towards open source in this market, if you will, is so much more meaningful than any other market.”