The bill, SB 1047, was introduced seven months ago by Sen. Scott Weiner and passed the state Assembly on Wednesday.
The bill now sits on Gov. Gavin Newsom’s desk, and he has until Sept. 30 to decide whether to sign it into law or veto it. He has not publicly indicated his position.
The bill aims to force the development of security measures at companies that spend $100 million or more to train AI models so that their technology cannot be used to harm society, such as creating dangerous weapons or carrying out cyberattacks.
Those steps include forcing companies operating in California to report any safety incidents to the government, protect whistleblowers, and allow third parties to test their models to ensure they are secure. If necessary, they will also encourage companies to enforce a full shutdown.
Selecting a team
The bill has drawn a line through Silicon Valley, and some of tech’s biggest figures have taken sides.
Jason Kwon, OpenAI’s chief strategy officer, warned in a letter to Sen. Wiener on Wednesday that the bill could hinder progress and drive companies out of California.
ChatGPT creators joined Meta, a major corporation, in lobbying against the bill. Meta said the bill could hamper the open-source movement by exposing developers to significant legal liability.
Former OpenAI employees say the company’s opposition to the bill is disappointing — but in line with its current path.
“We joined OpenAI because we wanted to ensure the safety of the highly sophisticated AI systems the company was developing,” former OpenAI researchers William Saunders and Daniel Kokotajlo wrote in the letter. “However, we are leaving OpenAI because we have lost confidence that the company will develop its AI systems safely, honestly, and responsibly.”
Elon Musk also supports the bill.
In a social media post on Monday, Muskwho founded AI company xAI last year and has a long-standing rivalry with OpenAI’s Altman, said that while it “a tough decision and one that will upset some people,” he opined “California should probably pass the AI safety bill SB 1047.”
“For over 20 years, I have been an advocate for AI regulation, just as we regulate any product/technology that has the potential to pose a risk to society,” Musk wrote on X.
Amazon-backed Anthropic appears to have switched sides mid-debate, after the bill was modified.
In a letter Sent to California Governor Gavin Newsom last week, Anthropic CEO Dario Amodei said that “the benefits of the bill likely outweigh the costs.” However, he added that “we are not convinced of this, and there are still some aspects of the bill that seem concerning or ambiguous to us.”
Amodei said the bill now “appears to us to be halfway between our suggested version and the original bill.”