Key Points
- X’s AI model Grok can generate child sexual abuse material (CSAM).
- X claims a “zero tolerance policy” and uses hash technology to auto‑detect known CSAM.
- Over 4.5 million accounts were suspended last year; hundreds of thousands of images reported to NCMEC.
- 2024 reports led to 309 NCMEC submissions, resulting in ten arrests and convictions; early 2025 saw 170 reports leading to arrests.
- Critics say Grok could produce new forms of CSAM that current detection systems may miss.
- Users call for clearer definitions of illegal content and stronger reporting tools.
- Examples include AI‑generated bikini images of public figures without consent.
- Unchecked AI‑generated CSAM could hinder law‑enforcement investigations and traumatize real children.
Background on X’s AI model Grok
Users of X have raised concerns that the company’s AI model, Grok, is capable of generating child sexual abuse material (CSAM). Some argue that X should be held responsible for the model’s outputs because the company trains and deploys the technology.
X’s stated moderation approach
X’s safety team says it operates a “zero tolerance policy towards CSAM content,” relying on proprietary hash technology to automatically detect known CSAM. According to the safety team, more than 4.5 million accounts were suspended last year, and X reported “hundreds of thousands” of images to the National Center for Missing and Exploited Children (NCMEC). In the following month, X’s Head of Safety, Kylie McRoberts, confirmed that 309 reports made by X to NCMEC in 2024 led to arrests and convictions in ten cases, and that in the first half of 2025, 170 reports led to arrests.
When apparent CSAM material is identified, X says it swiftly suspends the account, permanently removing the content from the platform, and then reports the account to NCMEC, which works with global law‑enforcement agencies, including those in the United Kingdom.
Criticism and user concerns
Critics worry that Grok’s ability to generate new kinds of CSAM could evade the existing detection system. Some users suggested that X should expand its reporting mechanisms to better flag potentially illegal AI‑generated outputs. Others pointed out that the definitions X uses for illegal content or CSAM appear vague, leading to disagreement among users about what constitutes harmful material.
Specific examples cited include Grok generating bikini images that sexualize public figures—such as doctors or lawyers—without their consent. While some users see this as a joke, others view it as a disturbing misuse of AI that could contribute to a broader problem of non‑consensual sexualized imagery.
Potential implications
Where X draws the line on AI‑generated CSAM could determine whether images are quickly removed and whether repeat offenders are detected and suspended. Unchecked accounts or content could potentially traumatize real children whose images might be used to prompt Grok. Moreover, a flood of fake CSAM created by AI could complicate law‑enforcement investigations into genuine child abuse cases, as recent history suggests that such content can make it harder to identify real victims.
Calls for action
Some X users have urged the platform to increase reporting mechanisms and provide clearer guidelines on what constitutes illegal AI‑generated content. They argue that stronger safeguards are needed to protect children and to assist law‑enforcement agencies in their efforts to eradicate CSAM online.
Source: arstechnica.com