Anthropic CEO Dario Amodei said on the In Good Company podcast that training the AI models currently under development can cost up to $1 billion. Current models like ChatGPT-4o only cost about $100 million, but he expects the cost of training these models to reach $10 billion or even $100 billion within three years.
“Currently, 100 million. Some models that are being trained today are closer to a billion.” Amodei also added, “I think if we get to ten or a hundred billion, and I think that’s going to happen in 2025, 2026, maybe 2027, and the algorithmic improvements and the chip improvements continue apace, then I think there’s a good chance that by then we’ll be able to get models that are better than most humans in most areas.”
Anthropic’s CEO mentioned these numbers when discussing the development of AI, from generative AI (like ChatGPT) to artificial general intelligence (AGI). He said there wouldn’t be a single point where we suddenly reach AGI. Rather, it would be a gradual evolution in which models would build on the developments of past models, similar to how a human child learns.
So if AI models are getting ten times more powerful every year, we can reasonably expect the hardware needed to train them to be at least ten times more powerful as well. Hardware could therefore be the biggest cost driver for AI training. In 2023, it was reported that ChatGPT would require over 30,000 GPUs, and Sam Altman confirmed that training ChatGPT-4 cost $100 million.
Last year, over 3.8 million GPUs shipped to data centers. With Nvidia’s latest B200 AI chip costing $30,000 to $40,000, it’s safe to assume that Dario’s $1 billion estimate is on track for 2024. If advances in modeling and quantization research continue at the current exponential rate, we expect hardware requirements to keep pace unless more efficient technologies like the Sohu AI chip become more widespread.
We can already see this exponential growth. Elon Musk wants to buy 300,000 B200 AI chips, while OpenAI and Microsoft are planning to build a $100 billion AI data center. With all this demand, we could see GPU data center shipments next year climb to 38 million if Nvidia and other vendors can keep up with the market.
But beyond sourcing hardware, these AI companies also need to worry about power and associated infrastructure. The estimated total consumption of all data center GPUs sold last year could power 1.3 million homes. If data center power needs continue to grow exponentially, we may run out of affordable electricity. And while these data centers need power plants, they also need a fully modernized grid that can handle all the electrons that power-hungry AI chips need to operate. That’s why many tech companies, including Microsoft, are now looking at modular nuclear power for their data centers.
Artificial intelligence is gaining momentum, and hardware innovations seem to be keeping pace. As such, Anthropic’s $100 billion valuation seems to be on track, especially if manufacturers like Nvidia, AMD, and Intel can make it happen. However, as our AI technologies improve exponentially with each new generation, one big question remains: How will this impact the future of our society?
