Nvidia’s latest AI chip, Grace Blackwell, incorporates an Arm-based CPU that is said to be able to run generative AI models using 25 times less power than the previous generation.
“Saving every last bit of power is going to be a fundamentally different design than if you’re trying to maximize performance,” Vachani said.
This strategy of improving computing efficiency to reduce power usage, also known as “doing more work per watt,” is one answer to the AI energy crisis. But it’s not enough.
According to a Goldman Sachs report, each ChatGPT query consumes roughly 10 times more energy than a regular Google search, and generating the AI images can consume as much power as charging a smartphone.
This problem is not new: a 2019 estimate found that training one large language model produces as much CO2 as driving five gasoline-powered cars over their entire lifetimes.
Hyperscalers, who are building data centers to handle this massive power consumption, are also seeing a surge in emissions. Google’s latest environmental report said its greenhouse gas emissions rose by about 50% from 2019 to 2023, partly due to the energy consumption of its data centers, though the company also said its data centers are 1.8 times more energy efficient than typical data centers. Microsoft’s emissions grew by about 30% from 2020 to 2024, also due to its data centers.
And in Kansas City, where Meta is building an AI-focused data center, demand for electricity is so high that plans to close a coal-fired power plant are on hold.
Hundreds of Ethernet cables connect server racks at the Vantage data center in Santa Clara, California, July 8, 2024.
Katie Tarasoff
There are more than 8,000 data centers worldwide, with the largest concentration in the United States. And thanks to AI, there will be a lot more of them in 10 years’ time. The Boston Consulting Group predicts that data center demand will grow 15-20% annually through 2030, accounting for 16% of total electricity consumption in the United States. That’s up from just 2.5% before OpenAI’s ChatGPT was released in 2022, and is equivalent to the electricity used by about two-thirds of all U.S. homes..
CNBC visited a Silicon Valley data center to find out how the industry can handle this rapid growth and where to find enough power to make it happen.
“We believe the demand from AI-specific applications will be as strong or even stronger than the demand for cloud computing to date,” said Jeff Tench, executive vice president of North America and Asia Pacific at Vantage Data Centers.
Many large tech companies contract with companies like Vantage to house their servers, and Tench said Vantage’s data centers are typically capable of using more than 64 megawatts of power, enough to power tens of thousands of homes.
“A lot of that is being used by a single customer, who will be leasing the entire space. And when you think about AI applications, that number could increase significantly beyond that into hundreds of megawatts,” Tench said.
CNBC visited Vantage in Santa Clara, California, which has long been one of the nation’s hotspots for data centers located near data-hungry customers. Nvidia’s headquarters is visible from the rooftop. Tench said Northern California is experiencing a “slowdown” due to “power shortages from the region’s utilities.”
Vantage is building new campuses in Ohio, Texas and Georgia.
“The industry itself is looking for locations with nearby access to renewable energy such as wind and solar, or other infrastructure that they can leverage, whether that’s as part of incentive programs to convert coal-fired plants to natural gas-fired plants, or increasingly looking at ways to get power off of nuclear facilities,” Tench said.
Vantage Data Centers to expand campus outside Phoenix, Arizona, to offer 176 megawatts of capacity
Vantage Data Center
An aging power grid is often ill-equipped to handle the load, even in places that can generate enough electricity. Bottlenecks occur when transporting power from where it’s generated to where it’s consumed. One solution is to add hundreds or thousands of miles of transmission lines.
“This is very costly and time-consuming, and sometimes those costs are passed on to residents in the form of higher utility bills,” said Xiaolei Ren, an associate professor of electrical and computer engineering at the University of California, Riverside.
A $5.2 billion effort to extend lines through an area of Virginia known as “Data Center Alley” has run into opposition from local ratepayers who don’t want their rates to go up to fund the project.
Another solution is to use predictive software to reduce transformer failures, one of the weakest points in the power grid.
“All electricity that is generated has to go through a transformer,” said VIE Technologies CEO Rahul Chaturvedi, adding that there are between 60 million and 80 million transformers in the United States.
Transformers have an average lifespan of 38 years, making them a common cause of power outages. Replacing them is expensive and time-consuming. VIE makes small sensors that can be attached to transformers to predict failures, determine which transformers can handle more load, and shift load away from transformers that are at risk of failing.
Chaturvedi expects business to triple since ChatGPT’s release in 2022, and to double and triple again next year.
VIE Technologies CEO Rahul Chaturvedi holds up a sensor in San Diego on June 25, 2024. VIE is installing sensors on aging transformers to help predict and mitigate failures on the power grid.
VIE Technologies
According to Wren’s research, generative AI data centers will require the withdrawal of between 4.2 and 6.6 billion cubic metres of water by 2027 to keep them cool – more than the annual water withdrawals of half the UK.
“Everyone’s worried about AI using up a lot of energy. If we just stopped talking nonsense about nuclear we could solve that problem, right? It’s solvable. The fundamental bottleneck in the future of AI is water,” said Tom Ferguson, managing partner at Burnt Island Ventures.
Ren’s research team found that running 10 to 50 ChatGPT prompts consumed the same amount of energy as a standard 16-ounce water bottle.
Much of that water is used for evaporative cooling, but Vantage’s Santa Clara data center has large air conditioning units that cool the building without taking in any water.
Another solution is to use liquid for direct cooling to the chip.
“A lot of data centers are going to have to go through a lot of retrofitting. In our case at Vantage, we put in a design about six years ago that allowed us to take advantage of the chilled water loops on the floor of our data halls,” says Vantage’s Tench.
Companies like Apple, Samsung, and Qualcomm have touted the benefits of on-device AI, which keeps power-hungry queries out of the cloud and power-strapped data centers.
“You’re going to deploy as much AI as your data centers can support, and that may be less than people would like, but ultimately there are a lot of people working to figure out how to alleviate some of these supply constraints,” Tench said.
