Who can stop unsafe AI?

New reporting puts a practical question at the center of California’s AI debate: if a powerful system becomes unsafe, who can require it to stop? Reuters’ October 28 report draws attention to Anthropic’s proposal for court-backed intervention. Reading its letter alongside OpenAI’s reveals substantial agreement on the need for controls—and consequential choices about who gets access, sets standards and acts.

The timing matters. These are recommendations submitted on October 15 and October 20, not new shutdown rules enacted this week. Governor Gavin Newsom’s September 18 executive order asks state agencies for recommendations on possible legal changes by November 16. That process includes independent evaluators, verified safety reporting, shutdown mechanisms and incident reporting.

Anthropic’s letter proposes risk reports at least every six months, with outside evaluators able to inspect unredacted reports and capable models, then publish their own assessments. If an independent assessment indicates a near-term risk of catastrophic harm, the state attorney general should be able to seek a court order suspending deployment, internal use or further development. To reduce financial dependence on the lab being examined, Anthropic proposes a state-administered pool of industry funding. Its embedded-evaluator approach is still developing; the company describes METR’s employee-like access as a pilot.

OpenAI’s letter supports an enforceable duty for developers to halt unsafe operations when a lesser intervention would not adequately address the risk. It also proposes annual independent compliance audits and technical assessments focused initially on serious risks in training, evaluation and internal deployment. It puts responsibility for acceptable risk levels and applicable standards with the state, informed by evaluators. Its term “shutdown controls” covers several possible interventions: stopping a training run, isolating an agent, revoking access or suspending a service. A developer cannot reliably turn off independently operated copies outside its control.

The distinction is not simply government control versus company control. Both letters support public oversight and practical intervention. Their proposals need to be examined for the authority they create, the evidence outsiders can obtain, and what happens when a company disputes a finding. Neither letter establishes that the proposed safeguards already work.

Our view: a promise of an off switch is useful only alongside tested controls and institutions able to act. The next proposals should answer three questions plainly: what triggers intervention, who can require it, and who decides that restarting is safe? For readers following the debate, those are more revealing than a general pledge to be responsible. Our Safety & Society guide separates evidence from advocacy, and Take action explains the campaign we support.