{"id":720547,"date":"2026-01-25T21:38:20","date_gmt":"2026-01-25T21:38:20","guid":{"rendered":"https:\/\/www.europesays.com\/uk\/720547\/"},"modified":"2026-01-25T21:38:20","modified_gmt":"2026-01-25T21:38:20","slug":"chatgpts-porn-rollout-raises-concerns-over-safety-and-et","status":"publish","type":"post","link":"https:\/\/www.europesays.com\/uk\/720547\/","title":{"rendered":"ChatGPT\u2019s porn rollout raises concerns over safety and et…"},"content":{"rendered":"

OpenAI says it wants to protect teenagers and treat \u201cadult users like adults\u201d. The company announced last week the release of a new age-estimation model for its chatbot ChatGPT, a tool designed to identify teenage users and switch on stricter protections by default. It is a move, the company said, that reflects its commitment to safety.<\/p>\n

But at the same time OpenAI is moving in the opposite direction. This quarter, the company plans to expand what ChatGPT is allowed to generate by launching a new feature: erotica.<\/p>\n

There is still little public detail about the upcoming sexual content. OpenAI has not said whether it will involve explicit conversations alone, or extend to AI-generated images and video, nor how it will be separated from standard ChatGPT use. They have said only that erotica \u2013 what most people call porn \u2013 will be restricted to adults and subject to additional safety guardrails.<\/p>\n

To a broad group of mental health researchers and digital harms experts, the decision is hard to square with the company\u2019s professed commitment to safety. They warn that introducing sexually explicit content into a system already known to foster emotional reliance risks intensifying attachment and exposing vulnerable users \u2013 of all ages \u2013 to harms the company may struggle to control at scale.<\/p>\n

\u201cThe shift to erotica is a very dangerous leap in the wrong direction,\u201d says Jay Edelson, a lawyer representing the family of Adam Raine, a 16-year-old from Orange County, California, who took his own life in 2025. In the weeks before his death, he was spending four hours a day talking to the chatbot, including asking specific questions about self-harm. \u201cThe problem with GPT is attachment; this will only exacerbate that.\u201d OpenAI has expressed sympathy with Raine\u2019s family but denies wrongdoing.<\/p>\n

The move also marks a cultural turning point for a company founded in 2015 to build AI \u201cfor the benefit of all humanity\u201d. Having grown at speed with minimal regulatory oversight, ChatGPT has become embedded in the daily lives of hundreds of millions of people, with users handing OpenAI their intimate data and, in some cases, their hearts. Now, critics say, despite mounting evidence of harm, the company is choosing growth and monetisation over safety \u2013 expanding into its most controversial territory yet.<\/p>\n

It was not always so. OpenAI was founded \u2013 by a group including Altman and Elon Musk \u2013 as a non-profit \u201cto advance digital intelligence in the way that is most likely to benefit humanity as a whole, unconstrained by a need to generate financial return,\u201d as written in the company\u2019s 2015 manifesto. They promised to keep their research publicly available \u2013 hence the \u201copen\u201d in OpenAI.<\/p>\n

A decade on, the company operates as a commercial business with roughly 800 million weekly users and a valuation of about $500bn. To maintain a competitive advantage, it has mostly stopped making its research public, and is in the middle of a corporate restructuring, moving away from non-profit status in order to raise capital \ufeffand generate revenue.<\/p>\n

\"Adam<\/p>\n

Adam Raine, who took his own life aged 16 after spending hours talking to a chatbot about subjects including self-harm<\/p>\n

Despite raising a total of more than $60bn over the years, OpenAI reportedly lost $9bn in 2025 alone, according to documents leaked to the Wall Street Journal. While revenue is increasing year on year, analysts fear that it may not rise fast enough. By 2028, losses are expected to \ufeffclimb to $74bn, driven by the huge \ufeffcost\ufeff of training and running its AI models.<\/p>\n

The company\u2019s promise to benefit humanity has not changed \u2013 at least on the surface \u2013 but revenue is clearly a priority. In September 2025, OpenAI rolled out Sora 2, a video-generation social media platform, in the hope of attracting new users. It is hard to see the benefit to humanity in a platform that produces low-quality, AI-generated video at scale.<\/p>\n

NewslettersChoose the newsletters you want to receive<\/p>\n

Clear, calm analysis on the stories driving the day\u2019s news.<\/p>\n

The very best of our journalism, reviews and ideas \u2013 curated each day.<\/p>\n

A dispatch from The Observer\u2019s kitchen table \u2013 from Nigel Slater\u2019s recipes to interviews, features and hot tips.<\/p>\n

<\/p>\n

For information about how The Observer protects your data, read our Privacy Policy<\/a><\/p>\n

And yet, the economics of Sora have proved unfavourable because of the computing power it takes to run video models. Sora\u2019s lead engineer later admitted the economics are \u201ccompletely unsustainable\u201d.<\/p>\n

This month, OpenAI began testing a new revenue model for some US users: advertising. That, too, was a reversal for Altman, who once called ads plus AI \u201cuniquely unsettling\u201d. In May 2024, he said: \u201cI kind of think of ads as a last resort for us for a business model.\u201d<\/p>\n

Porn, then, is the natural next step. Digital adult content has long been lucrative, with the industry worth a reported $81.86bn in 2025 (probably an underestimation). OpenAI\u2019s leadership has framed the decision as a matter of user freedom. \u201cWe are not the elected moral police of the world,\u201d Altman wrote on X last October. \u201cAllowing a lot of freedom for people to use AI in the ways that they want is an important part of our mission.\u201d<\/p>\n

\u2018The shift to erotica is a very dangerous leap in the wrong direction.\u00a0The problem with GPT is attachment; this will only exacerbate that\u2019\u2018The shift to erotica is a very dangerous leap in the wrong direction.\u00a0The problem with GPT is attachment; this will only exacerbate that\u2019<\/p>\n

Jay Edelson. lawyer<\/p>\n

AI-generated pornography already exists, though largely at the margins of the internet. It tends to take the form of text-based roleplay on specialised apps, although sometimes it involves image-generation tools trained to produce sexualised cartoons. What distinguishes such systems from conventional porn is that they respond and adapt to user requests, drawing people in for longer. For the most part, according to one\ufeff expert, this technology remains \u201cniche\u201d, detached from the mainstream chatbots people rely on for \ufefftasks like work and study.<\/p>\n

Then there is the expanding field of so-called AI companions: chatbots designed to simulate friendship or romance, not al\ufeffways sexual. Researchers have found that some users form strong emotional bonds with them, occasionally sliding into emotional dependence. These too, can exist within specialised apps, though there are numerous reports of users growing attached to, falling in love with, or even marrying ChatGPT.<\/p>\n

Tech CEOs are \u201ccashing in\u201d on systemic loneliness, says Meetali Jain, executive director at Washington-based Tech Justice Law Project (TJLP)\ufeff. \u201cWe saw the first iteration with social media.\u201d Now, she says, human-like AI chatbots are filling that same void.<\/p>\n

Psychiatrists told The Observer that chatbots\u2019 sycophantic, always-on nature can create unhealthy codependencies in otherwise healthy users, and sometimes trigger mental illness in \ufeffthose already unwell. \u201cIt seems like really not the right time to introduce this sexual charge to these conversations, to users who are already struggling,\u201d said one. According to OpenAI\u2019s own data from October 2025, as many as 560,000 users a week show \u201cpossible signs of mental health emergencies related to psychosis or mania\u201d.<\/p>\n

The company is already facing eight lawsuits in the US, all brought by TJLP in the past year, alleging that its models lacked adequate safety guardrails, leading to severe mental health crises and deaths in teenagers and adults. OpenAI denies causing harm to users in all of the cases, but has since updated its models to include more safeguards. It says that, even in the anticipated erotic mode, it \u201cwill still not allow content that harms others\u201d, and has said that its safety systems are stronger than they were when the lawsuits were filed.<\/p>\n

Jain says \ufeffchildren are particularly susceptible to manipulation from chatbots, but there are \u201cunaddressed concerns affecting adults\u201d. Erotica could emphasise them, she says. \u201cGenerative AI technology is so sophisticated and powerful that it, frankly, renders all of us vulnerable.\u201d<\/p>\n

\"Sam<\/p>\n

Sam Altman, the CEO of OpenAI, in Berlin last September<\/p>\n

Fidji Simo, OpenAI\u2019s chief executive of applications, said in December that the company wanted to improve its ability to estimate users\u2019 ages before introducing the new erotic feature, which she said would happen by April.<\/p>\n

On 20 January, OpenAI released its new age-prediction system for people on its consumer plans, designed \u201cto help determine whether an account likely belongs to someone under 18, so the right experience and safeguards can be applied to teens\u201d. The tool analyses user behaviour, including how long an account has existed, typical times of day a user is active, and usage patterns over time, to estimate age. OpenAI says this approach allows it to apply protections without collecting sensitive documents such as passports or driving licences.<\/p>\n

Several child-safety experts told The Observer that while age-assurance tools are an important first step, age-prediction models remain largely untested at scale. OpenAI says it believes its model\u2019s accuracy outperforms industry standards, but has not released data to support its claims.<\/p>\n

Sam Stockwell, a researcher at the Alan Turing Institute who specialises in online safety says the tool is \u201cpromising to see\u201d, but that he has \u201ca few concerns\u201d. His main worry is that behaviour-based age prediction isn\u2019t reliably tied to someone\u2019s real age. Adults can have interests that look \u201cyoung\u201d on paper, he says. Children, meanwhile, may learn to bypass the system.<\/p>\n

Leanda Barrington-Leach, executive director of the children\u2019s rights group 5Rights Foundation, says OpenAI\u2019s approach also raises legal questions under UK and EU data-protection law. Under GDPR and the UK\u2019s Age Appropriate Design Code, she says, companies face strict limits on how children\u2019s data can be processed. \u201cYou cannot use the data of a child under 13 without parental consent,\u201d she says. OpenAI says the new system is designed to minimise data collection.<\/p>\n

Musk left OpenAI\u2019s board in 2018 and is now suing the company, alleging it has abandoned its original mission to build AI for the benefit of humanity. He is seeking to block its ongoing corporate restructuring, and wants up to $134bn in damages.<\/p>\n

Musk, however, cannot claim the moral high ground. Earlier this month, his own AI chatbot, Grok,, which is integrated into his social media platform X, was used to generate non-consensual sexualised images, including digitally \u201cundressing\u201d people, and to produce illegal child sexual abuse material, according to the Internet Watch Foundation.<\/p>\n

Zoe Hawkins, co-founder of the Tech Policy Design Institute, says \u201cthe significant harm caused by the explicit capabilities of Grok is an important cautionary tale in this policy conversation\u201d, illustrating how quickly AI products are being deployed, with regulation struggling to keep pace. Another expert described enforcement efforts as \u201cwhack-a-mole\u201d.<\/p>\n

While ChatGPT is already overseen by the UK regulator Ofcom because it is classified as a \u201csearch service\u201d, the launch of an erotic feature would stretch a regulatory framework that was not designed with conversational AI in mind.<\/p>\n

An Ofcom spokesperson said: \u201cThe law is clear: sites or apps that allow pornographic content, including AI-generated material, must use highly effective age assurance to prevent children from readily accessing it.\u201d They added that Ofcom \u201cwill not hesitate to use the full force of our powers against any regulated service that fails to protect people in the UK, particularly children\u201d.<\/p>\n

Jain and Edelson, the lawyers, say the risks to adults alone should give OpenAI pause. \u201cWhat I fear is that erotica is simply too enticing as a revenue stream.\u201d<\/p>\n

Photographs by Alamy,\u00a0Florian Gaertner\/Getty Images, and courtesy of the Raine family<\/p>\n","protected":false},"excerpt":{"rendered":"OpenAI says it wants to protect teenagers and treat \u201cadult users like adults\u201d. The company announced last week…\n","protected":false},"author":2,"featured_media":720548,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[3163],"tags":[323,1942,295,53,16,15],"class_list":{"0":"post-720547","1":"post","2":"type-post","3":"status-publish","4":"format-standard","5":"has-post-thumbnail","7":"category-artificial-intelligence","8":"tag-ai","9":"tag-artificial-intelligence","10":"tag-elon-musk","11":"tag-technology","12":"tag-uk","13":"tag-united-kingdom"},"share_on_mastodon":{"url":"https:\/\/pubeurope.com\/@uk\/115957904454416020","error":""},"_links":{"self":[{"href":"https:\/\/www.europesays.com\/uk\/wp-json\/wp\/v2\/posts\/720547","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.europesays.com\/uk\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.europesays.com\/uk\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/uk\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/uk\/wp-json\/wp\/v2\/comments?post=720547"}],"version-history":[{"count":0,"href":"https:\/\/www.europesays.com\/uk\/wp-json\/wp\/v2\/posts\/720547\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/uk\/wp-json\/wp\/v2\/media\/720548"}],"wp:attachment":[{"href":"https:\/\/www.europesays.com\/uk\/wp-json\/wp\/v2\/media?parent=720547"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.europesays.com\/uk\/wp-json\/wp\/v2\/categories?post=720547"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.europesays.com\/uk\/wp-json\/wp\/v2\/tags?post=720547"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}