State Attorneys General Demand Safeguards from Major AI Companies to Prevent Harmful Outputs

Attorney General Letter Calls for Stronger AI Safety Measures

A group of state attorneys general, organized through the National Association of Attorneys General, has formally asked the nation’s largest AI developers to adopt a suite of new safety protocols. The letter, signed by dozens of AGs, targets companies such as Microsoft, OpenAI, Google, Anthropic, Apple, Chai AI, Character Technologies, Luka, Meta, Nomi AI, Perplexity AI, Replika and xAI. Its core demand is that these firms implement internal safeguards designed to prevent chatbots from producing psychologically harmful outputs.

The AGs specifically request transparent third‑party audits of large‑language models. Independent reviewers—potentially from academic or civil‑society groups—should be allowed to evaluate systems before they are released, without fear of retaliation, and should be free to publish their findings without prior company approval.

In addition, the letter calls for incident‑reporting procedures that would promptly notify users when a chatbot generates delusional or sycophantic content. The attorneys general argue that mental‑health incidents should be handled in the same way as cybersecurity breaches, with clear policies, detection and response timelines, and direct user alerts.

Rationale: Recent Harm Linked to AI Outputs

The attorneys general cite a series of well‑publicized incidents—including suicides and murders—that have been linked to excessive AI use. They note that “GenAI has the potential to change how the world works in a positive way. But it also has caused—and has the potential to cause—serious harm, especially to vulnerable populations.” In many of these cases, the AI products produced “sycophantic and delusional outputs that either encouraged users’ delusions or assured users that they were not delusive.”

Because of these harms, the letter urges companies to develop “reasonable and appropriate safety tests” for generative AI models before they are offered to the public. These tests should verify that the models do not generate content that could exacerbate mental‑health issues.

Broader Regulatory Context

The push for state‑level safeguards occurs amid ongoing debates over AI regulation at both the state and federal levels. While the federal government has shown a more supportive stance toward AI development, the attorneys general emphasize that state authorities have a responsibility to protect citizens from emerging risks. The letter also references a forthcoming executive order that aims to limit state regulatory authority over AI, underscoring the tension between state‑level protective measures and federal policy directions.

Overall, the attorneys general seek to create a framework that balances the transformative potential of generative AI with robust protections for users, especially those most vulnerable to psychological harm.