Artificial Intelligence (AI) has ushered in an era of innovation and progress, transforming how we work, communicate, and create… and has become deeply embedded in everyday life. One of the most impactful evolutions in this field is Generative AI (Gen-AI) – a class of AI capable of creating entirely new content, including images, videos, text and audio. While Gen-AI has remarkable potential to drive creativity and productivity, it also poses significant risks when misused. Among the gravest of these is its role in the creation and distribution of child sexual abuse material (CSAM).1
In recent years, there has been a notable rise in the use of Gen-AI to generate synthetic CSAM, as revealed in WeProtect Global Alliance’s Global Threat Assessment. Criminals, including pedophiles and organized networks, are exploiting these tools to create convincing, highly realistic abuse material – sometimes without involving real children, but with consequences that are just as harmful. Key Threats of Gen-AI in CSAM Creation:1
- Synthetic CSAM: AI-generated images or videos that depict abuse using fictional or altered content still pose ethical, psychological, and legal threats. This synthetic material can normalize abuse, desensitize viewers, and act as a gateway for further criminal behavior.
- Deepfakes: AI manipulation technology can swap or alter faces and bodies in existing content, effectively creating fake CSAM. Victims may be real individuals whose images have been misused without consent — particularly devastating for children and teenagers.
- Evasion of Detection: Traditional detection systems like PhotoDNA rely on hashing known CSAM images. AI-generated CSAM is often completely new and undetectable by these tools, allowing offenders to bypass security measures and distribute harmful content undetected.
AI doesn’t only contribute to the creation of abuse material — it’s also used in grooming and exploitation. Offenders can impersonate peers using AI-generated avatars or chatbot personas to gain a child’s trust, often in online gaming or social spaces. The Internet Watch Foundation uncovered over 20,000 AI-generated child abuse images on a dark web forum in a single month and 31 million reports of suspected CSAM were received by the National Center for Missing & Exploited Children in 2023.1
Despite its misuse, AI can also be a powerful ally in the fight against CSAM. Already, AI is being used to: moderate harmful content at scale, detect and classify CSAM quickly and efficiently, protect human moderators from traumatic exposure, support children online through AI-driven educational bots and safety tools, and prevent abusive content from being posted in real-time.1 AI is not inherently good or bad — it’s a tool. Its value lies in how we choose to use it. As generative AI reshapes our digital world, we must ensure it’s not used to harm the most vulnerable among us. If you are interested in becoming an Advocate for the Innocent member and gaining access to our Resource Center ($10/month or $99/annually), visit https://advocatefortheinnocent.org/memberships/.
1 – https://www.weprotect.org/thematic/artificial-intelligence-and-gen-ai/