California lawmakers passed a hotly contested artificial-intelligence safety bill on Wednesday, after which it will need one more process vote before its fate is in the hands of Gov. Gavin Newsom, who has until Sept. 30 to decide whether to sign it into law or veto it.
Tech companies developing generative AI – which can respond to prompts with fully formed text, images or audio as well as run repetitive tasks with minimal intervention – have largely balked at the legislation, called SB 1047, saying it could drive AI companies from the state and hinder innovation.
Some Democrats in Congress, including Representative Nancy Pelosi, also opposed it. Proponents include Tesla CEO Elon Musk, who also runs an AI firm called xAI and has said he supports the bill.
The measure mandates safety testing for many of the most advanced AI models that cost more than $100 million to develop or those that require a defined amount of computing power. Developers of AI software operating in the state also need to outline methods for turning off the AI models if they go awry, effectively a kill switch.
The bill also gives the state attorney general the power to sue if developers are not compliant, particularly in the event of an ongoing threat, such as the AI taking over government systems like the power grid.
As well, the bill requires developers to hire third-party auditors to assess their safety practices and provide additional protections to whistleblowers speaking out against AI abuses.
The bill’s author, Democratic State Senator Scott Wiener, represents San Francisco, home to OpenAI and many of the startups developing the powerful software. He has said legislation is necessary to protect the public before advances in AI become either unwieldy or uncontrollable.
Martin Casado, general partner at venture capital firm Andreessen Horowitz, said before the vote that he hoped Newsom would veto it. “It’s got the most bipartisan, broad opposition I’ve ever seen,” he said.
Alphabet’s Google, Microsoft-backed OpenAI and Meta Platforms have expressed their concerns in letters to Wiener. Amazon-backed Anthropic has said the benefits to the bill likely outweigh the costs, though it added there were still some aspects that seem concerning or ambiguous.