San Francisco's City Attorney has issued cease-and-desist letters to Apple and Google, demanding the removal of 13 AI-powered apps used to create non-consensual explicit imagery.

Key Takeaways

  • San Francisco City Attorney David Chiu has targeted 13 'face-swap' apps on Apple and Google platforms.
  • These apps use generative AI to create non-consensual 'nudified' or explicit images of women and girls.
  • The city alleges tech giants are profiting from these harmful apps through in-app transaction fees.
  • Deepfake technology is being used for bullying, harassment, and causing severe mental health crises.

In a significant legal move against the misuse of generative artificial intelligence, the San Francisco City Attorney's Office has issued cease-and-desist letters to Apple and Google. The legal notices demand the immediate removal of 13 'face-swapping' applications that facilitate the creation of non-consensual, AI-generated explicit imagery, commonly referred to as 'nudification.'

Profiting from Harmful Technology

City Attorney David Chiu has accused the Silicon Valley giants of 'aiding and abetting' the sale of explicit deepfake images. A critical point raised in the legal notice is the financial incentive involved; these apps utilize in-app purchases, from which Apple and Google collect a percentage of the revenue. Chiu emphasized that these companies have a moral and legal responsibility to ensure their platforms do not facilitate sexual abuse or the distribution of non-consensual pornography.

The Dark Side of Generative AI

The rise of highly realistic deepfakes has created a digital landscape where a single reference photo can be used to strip a person of their dignity in seconds. Researchers have highlighted the devastating impact of this technology, noting its use in bullying, humiliation, and threatening women and girls. The human cost is profound, with reports linking deepfake-related harassment to severe mental health issues and even suicidal ideation among victims. Alarmingly, investigations have uncovered instances where deepfake sexual abuse images were created targeting minors in at least 90 schools.

Corporate Accountability and Moderation Gaps

While both tech giants maintain policies against pornography and harassment, the persistence of these apps suggests a major failure in content moderation. Google spokesperson Dan Jackson stated that the company has proactively removed 'hundreds' of violating apps and restricted search terms like 'nudify.' However, watchdog groups like the Tech Transparency Project (TTP) argue that the scale of the problem is far greater than admitted, noting that such apps have collectively garnered hundreds of millions of downloads and generated significant revenue before being detected.