Ofcom has launched an investigation into X over the creation of deepfake sexual images using its AI tool Grok, after a row over the platform led to suggestions it could be banned in the UK.
On Monday, 12 January, Ofcom formally started its investigation after spending the first weeks of the year requesting information from Elon Musk’s X and xAI.
The furore erupted at the start of the year when users X acknowledged some users had used its integrated AI, Grok, to generate undressed images of people, including women and children.
Ofcom said: “There have been deeply concerning reports of the Grok AI chatbot account on X being used to create and share undressed images of people – which may amount to intimate image abuse or pornography – and sexualised images of children that may amount to child sexual abuse material.”
The regulator said they requested information from X on 5 January and received a response by 9 January. Ofcom said it had carried out an assessment on the available information and has decided “to open a formal investigation to establish whether X has failed to comply with its legal obligations under the Online Safety Act”.
Grok is a similar AI programme to ChatGPT, which is fully integrated into X. Users can tag Grok in a post and enter a prompt where they request the AI to do something for them. Grok’s response will be available for all users to see.
The Internet Watch Foundation said its analysts had confirmed the existence of “criminal imagery of children aged between 11 and 13 which appears to have been created using the (Grok) tool”.
Grok is Elon Musk’s AI, which is integrated into X. (AP)
X moved Grok’s image generation capability behind the paid premium service in response to the issue, but the government has said this does not go far enough.
Under the Online Safety Act in the UK, social media firms must prevent and remove child sexual abuse material when they become aware of it.
Several other countries, including the Republic of Ireland and Canada, have also said they were outraged by the deepfakes and are considering sanctioning the platforms, while other nations like Indonesia have banned it altogether.
Liz Kendall secretary of state for science, innovation and technology, has said she would back Ofcom in any decision they make. (PA)
Musk and members of the Trump administration have criticised prime minister Keir Starmer and the government over the recent row, saying they were using the issue to justify banning X and restricting free speech.
X has said it takes action against illegal content, including child sexual abuse material, “by removing it, permanently suspending accounts, and working with local governments and law enforcement as necessary”.
What is the law on sharing deep fake nudes?
Police.uk, the national website for policing in England, states that deepfakes can be created for a variety of reasons, with many being harmless, but others risk breaking the law, particularly when used to make naked images of a real person.
End Violence Against Women says non-consensual pornography makes up 96% of all deepfakes found online, with 99.9% of them depicting women.
With regards to children, Police.uk said: “It’s illegal to make, share and possess indecent photographs and pseudo-photographs and/or videos, including deepfakes, of someone who is under 18.
“It does not matter if the person depicted is real or it’s a fake that’s been digitally created or altered.”
For a person over 18, they say it is not illegal to make these images, but it is illegal to share, or threaten to share them with others.
It is also illegal to use them for blackmail, fraud, or harassment.
All of these crimes can face a hefty jail sentence if prosecuted, with crimes around child pornography carrying the most severe prison time.
Specific regulations about using AI to make non-consensual intimate images were passed by Parliament in July but are not yet enforced.
The Data Use and Access Act 2025 would make it illegal to request an AI to generate these images without consent, a step up from previous laws, which only banned sharing them.
What have Starmer and Musk said?
Starmer has said he backs Ofcom in whatever decision they make, saying X need to “get their act together”.
Sir Keir told Greatest Hits Radio on 8 January: “This is disgraceful. It’s disgusting. And it’s not to be tolerated.
Keir Starmer has said X “need to get their act together.” (PA)
“X has got to get a grip of this. Ofcom has our full support to take action in relation to this. This is wrong.
“It’s unlawful. We’re not going to tolerate it. I’ve asked for all options to be on the table.
“It’s disgusting. X need to get their act together and get this material down. We will take action on this because it’s simply not tolerable.”
Musk has hit out at the criticism, saying the debate around it was being used as an “excuse for censorship”.
He has pointed out that other AI chatbots with image generation capabilities can also be used to make illegal images, but sanctioning them is not being discussed.
Musk, who has been fiercely critical of Starmer in the past, shared AI images made by other chatbots of Starmer in a bikini to add to his point.
Musk has also shared a post from US legislator Anna Paulina Luna threatening to sanction both Starmer and the UK if X was blocked in the country.
The tech tycoon has previously insisted that “anyone using Grok to make illegal content will suffer the same consequences as if they uploaded illegal content”.
Could X be banned in the UK?
The Online Safety Act of 2023 forced social media companies to take active steps to remove any illegal images from their platforms and face sanctions if they do not.
Several Labour MPs have called for the government to stop using X over the issue, something Downing Street has not ruled out.
Former minister Louise Haigh said on 8 January, said the “enablement, if not encouragement, of child sexual abuse mean it is unconscionable to use the site for another minute”, and urged ministers to join her in leaving it.
But the government has said an outright ban is also possible.
Technology secretary Liz Kendall has said the government would back regulator Ofcom if it decided to implement a ban on accessing X from the UK.
Ofcom has powers under the Online Safety Act to fine businesses up to ÂŁ18 million or 10% of global revenue, as well as to take criminal action.
It can also order payment providers, advertisers and internet service providers to stop working with a site, effectively banning them, though this would require agreement from the courts.
Many countries are likely waiting to see what Ofcom does with many politicians in other Western nations also criticising X and Grok over deepfakes.
Ireland’s media regulator has said it is reviewing the issue, with Irish premier Micheal Martin saying that such use of AI was “unacceptable” and possibly illegal.
The Canadian government has also criticised the “deepfake sexual abuse” but has ruled out banning X at this time.
Malaysia and Indonesia blocked Grok on 11 January over its ability to produce sexually explicit images of children.