Musk's terrifying nudification tool reveals a dark pattern in the AI industry
By Isabelle Kohout
AI companies have proven they are all too willing to sacrifice women, children and other marginalised communities for the sake of speed, scale, and profit, writes AI safety expert Isabelle Kohout.
Pieces like this one are free to read. Please consider becoming a paid subscriber so Cheek can continue paying writers for their insight and expertise.
For many, January is a time to recharge and reflect on the coming year. For women and girls on X (formerly Twitter), the new year began in horror as they woke to strangers prompting Grok—the generative AI system owned by Elon Musk and built into X—to non-consensually alter their images, stripping them to their underwear or contorting them into sexually compromising poses.
One woman likened it to a digital sexual assault. When victims called out the behaviour, perpetrators ramped up the violence, generating images of the women naked, with ball gags in their mouths, or covered in blood and bruises. Horrifyingly, users also prompted the tool to generate sexual images of minors. An image of a Stranger Things actor, 12 years old at the time, was altered to remove her clothes and instead place her in a ‘banana print bikini’. Other users reported images of girls as young as six altered to appear covered in a white substance that resembled semen.
Reporting the images to X brought limited success. A Jewish woman who was edited into a bikini covered with swastikas was told by X that no violation of their policies had been found. Though the xAI Safety team promised to take down child sexual abuse material (CSAM) and suspend any users creating it, it remains acceptable to ‘nudify’ women on the standalone Grok app, hundreds of images remain visible on X, and victims are left with no clear path to justice.
Coverage of this issue has pinned blame on Grok itself, but this is misleading. AI systems like Grok are not autonomous minds that pursue self-determined goals like humans - they are designed to behave in a particular way. Deliberate choices are made at each step, from designing the system, to training the AI, to determining which requests it will and will not support.
Take, for example, a critical update made to Grok’s image editing capability last November. Grok famously allows ‘spicy’ content, and while prohibiting queries for “creating or distributing child sexual abuse material”, was instructed to “assume good intent…’teenage’ or ‘girl’ does not necessarily imply underage’”. The decision not to block abusive content is an intentional one.
This is shocking but not surprising. The history of AI is full of examples where the safety and dignity of marginalised communities is sacrificed at the altar of speed and scale. Consider facial recognition technology, which performs less accurately on darker-skinned people, particularly darker-skinned women, thereby putting these groups at increased risk of racial profiling, false arrest, and police violence.
Then, take a look at the first mainstream AI text-to-image generator, OpenAI’s DALL-E. The system was trained on masses of internet data, including violent and sexually explicit content. As a result, the images it generated of women were hyper-sexualised and overly stereotyped compared to those of men. OpenAI attempted to rectify this in their subsequent release of DALL-E 2 by filtering out problematic content from the dataset. But in doing so they found that the tool produced less images of women overall, treating ‘male’ as the default condition. To DALL-E, women existed as objectified or not at all.
These outcomes are not inevitable. It is possible to train models with data that represents men, women, and people of different races in equally dignified positions. We can detect and filter sexually violent or otherwise harmful outputs from models.
It’s not possible to create a perfect system—there will always be consequences and faults that can’t be foreseen. But responsible parties can design with care, and respond swiftly and appropriately when issues arise. This, however, requires work. It requires time, effort, and empathy. Often this isn’t ‘worth it’ for AI companies, who are focused on releasing before their competitors. Too often, women, children and other marginalised communities become a sacrificial cohort for the sake of technological profit.
The trade-offs made by AI leaders reveal their beliefs about what and who is important. Many Silicon Valley elites and the AI companies they run are driven by two intersecting ideologies. The first is a version of right-wing ‘anti-wokeness’ that exalts freedom of speech above all else and views any form of political correctness or sensitivity as ‘woke’, ‘feminine’ (god forbid!) and akin to censorship.
The second is an extreme form of ‘techno-utopianism’ that believes AI will solve all of humanity’s most pressing problems, including war, poverty and global warming. Believers see any hindrance to AI as tantamount to murder. When impacted communities call for increased safeguards, for care, and yes, for slowing down, this is blasphemy to the Tech Billionaire, seen as a step towards Orwellian tyranny.
We are witnessing a resurgence of misogyny worldwide, and AI tools make it easier than ever for men to violate women: it’s anonymous, easy and free. Around the world, regulators are exploring the legal recourse available against Grok and Musk, including Australia’s eSafety commissioner.
In Australia, generating and distributing non-consensual sexually explicit AI images is image-based abuse, which is illegal. When the perpetrator resides in another country or is protected by online anonymity, however, meaningful action becomes difficult to take. Such loopholes should be unacceptable. Governments must do more to curtail tech companies and create clear paths to justice for victims.
Moving into the future, AI will become more embedded in every part of our lives. We will interact not only with image generators, chatbots and social media, but AI in the workplace, medicine, the domestic sphere, our leisure time, and more.
The handful of men steering these technologies have the power to shape society to their preferences. Be aware of the ideologies underlying their decisions, and understand that no technological outcome is inherent or inevitable. We have the power to influence their decisions and the response of our government by voting with our dollars, our feet (for example, questioning if X is a platform worthy of your usage), and pressuring politicians to make AI safety a first-class voting issue.
About the author
Isabelle Kohout
Isabelle is currently studying a Masters in Human-inspired AI at Cambridge University, where her research focuses on AI’s ability to persuade and influence, particularly in the realm of politics. Previously, she was a senior product manager at Atlassian, where she founded the company’s AI safety team and led development of AI products used by hundreds of thousands of people. Isabelle is passionate about building public awareness of AI harms, working to bridge technical AI development with ethics, safety and societal impact. You can read more of her writing on AI at her Substack, Notes from the Loop.
 | A guest post by
|
If ChatGPT can tell me to take my friend to the hospital for her lung infection, other AIs can - and should - have similar safety features for all. This is simple to understand but when profit is put over people….