🤖 Inference cost collapse deflates OpenAI, LLM startups and Nvidia's multiple

“I think that when you look at the impacts and pricing that all of these open-source and closed-source model companies have now introduced in terms of the price per token, what we've seen is just a massive step function lower. It is incredibly deflationary.”

“it's very likely we end up seeing a lot of more smaller targeted models being used in specific applications instead of one massive general purpose model being used, or networks of smaller models, which is really where I think the industry is going”

“In my talk with Sunny this week, he mentioned that these new models are so much more efficient that you actually can throw the old model in the garbage because it's so inefficient. And these are now becoming about 90% cheaper every year, which means every two years these things are going to be 99% cheaper and better.”

“AI is really two markets, training and inference. And inference is going to be 100 times bigger than training. ... the sharps know that NVIDIA cannot do inference”

“he did a very good job of explaining where Nvidia is really strong, which is really in learning, and where Nvidia is trying to build the business, which is an inference... he said that if he deploys his roadmap, he'll have 50% of the available inference compute by the end of next year”

“We're going to ship 20 million into your safe note. I know you closed it, but we're going to pry it open and get some allocation.”

“It is a real community that's come together building around Groq. And what I will say is, you know, sort of the same jump that developers saw when we went from dial-up internet to broadband.”

“This is the first investment, I think, where we're all on the cap table, right? So we can all, we're all rooting in the same direction.”

“By the way, developers, by the way, on Groq, just this past week, in the queue, the wait list tripled. Now there's almost 10,000 developers. ... I think the most important North Star metric for these developer platforms is basically that. As goes the developers, so goes the platform.”

“you now see developers stress testing us and finding that we are meaningfully, meaningfully faster and cheaper than any Nvidia solution, there's the potential here to be really disruptive ... So there's a lot of market cap for Groq to gain by just being able to produce these things at scale”

“when you can do more with less, it doesn't mean you need less chips. It means you need a lot more chips ... it really starts to rationalize the Nvidia thesis that it could be worth ten trillion dollars potentially”

“you're about to see a one-tenthing of the compute cost ... So today, you raise $100 million because you need to spend 70 or 80 on compute. Tomorrow when you only need to spend seven or eight for the same amount of throughput”

“And I think he's very compelling in arguing why some of the big cloud providers today that are offering infrastructure for AI model training and inference are going to be challenged if someone can build full stack and do it successfully.”

“The perfect analogy of EC2 and S3 in 2024 is the token per second provider. Now, there you have to double click and say, OK, well, what does a token per second provider need to do to make a lot of money? And I think the ultimate answer is you need your own proprietary hardware.”

“My worst-performing asset in 2024 is LLM startups. They've been massively overvalued and open source is making an incredible run at them... too many players and too much parity... they're all going to come down by 50, 60, 70, 80% in terms of their valuations.”

“I think the enterprise value of OpenAI goes down... folks [building custom hardware] will multiply the capability of this market by 1000x... open source models really proliferate, proprietary and closed models go under pressure... it's going to be very hard for existing folks — and I would say the same is probably true for NVIDIA — to maintain a multiple of market cap. In this next year, that happens.”