Grok AI misuse: Victims in Indonesia, Malaysia 'angry' and 'humiliated', but is banning the tool enough?

SINGAPORE: When an AI-generated image of Indonesian celebrity Fransisca Saraswati in a pink bikini circulated widely on social media platform X, she blocked the anonymous account responsible.

But the 25-year-old soon realised that the image had been generated by Grok, X’s artificial intelligence tool, which had also publicly tagged her, all without her consent.

“I felt humiliated, annoyed, angry and sad at the same time,” said Saraswati, a singer who is also professionally known as Sisca Saras.

“The experience is deeply distressing, not only on a personal level, but also professionally because it violates personal dignity, misrepresents identity and undermines trust between artists and the public,” she told CNA. 

Her experience mirrors that of victims in Malaysia whom CNA spoke to.

Last month’s rollout of an “edit image” button on Grok has enabled users to alter online images using prompts such as “put her in a bikini” or “remove her clothes”, while "Spicy Mode", released last August, allows users to generate adult content and edit existing online images.

Critics say the tool has been widely abused to generate sexualised deepfakes of women and minors. 

Victims have described Grok’s misuse as “degrading” and a violation of personal dignity, calling for stricter regulation of generative AI tools while experts told CNA that recent government actions reflect a strong signal about what authorities will tolerate when it comes to the safety, rights and dignity of women and children.

“This signals a shift in how responsibility is being framed. Instead of treating the issue purely as user misuse, regulators are pointing to systemic failures in how the tool was designed and deployed,” said Nuurrianti Jalli, a visiting fellow at the ISEAS-Yusof Ishak Institute. 

As the tool faces growing international backlash, several Southeast Asian countries have moved to suspend Grok. 

Indonesia became the first country to temporarily block the Elon Musk-owned chatbot on Jan 10.

A day later, Malaysia followed suit, with the Malaysian Communications and Multimedia Commission (MCMC) then announcing plans to pursue legal action against X over alleged failures in safeguards on Jan 13.

Communications Minister Fahmi Fadzil said on Thursday (Jan 15) that restrictions on Grok would only be lifted once X demonstrates that necessary safeguards are in place to prevent misuse.

X on Wednesday also announced measures to prevent Grok from undressing images of real people following backlash.

Still, the Philippines said on Thursday that it also planned to block Grok "by tonight", joining its Southeast Asian neighbours.

Elina Noor, a non-resident scholar at the Asia Programme at Carnegie Endowment for International Peace told CNA that Malaysia’s legal action points to “failures in systemic safeguards by the makers of Grok”, as well as its unwillingness to take any remedial action despite prior warnings. 

Nuurrianti said that Grok has drawn particular scrutiny because it has been “more permissive” than most mainstream AI platforms when it comes to manipulating images of real people.

“This reinforces that AI safety is largely a design choice. Other platforms have shown that guardrails are technically possible, even if they come with trade-offs,” she said.

“Grok’s approach reflects a different set of priorities, which regulators are now questioning.” 

“IT FELT SO DEHUMANISING”

Several Indonesian celebrities have publicly slammed the artificial intelligence tool, with some even saying that they are now reluctant to post photos online. 

Saraswati, the 25-year-old singer, said that AI should be a tool that supports creativity, not one that exploits artists or puts them at risk.

“The misuse of AI to generate explicit or harmful content highlights the urgent need for stronger safeguards, clearer regulations and accountability,” she said.

“Artists deserve protection over their likeness, name and creative identity, just as they would in another form of media.” 

Separately, the management agency of a popular Indonesia girl band JKT48 warned that it would pursue legal action if manipulated images of its members were not taken down.

“Such content has the potential to constitute defamation and/or insults as regulated under applicable laws” the management said in a statement on Jan 5. 

“If such content is still circulating within 2 x 24 hours after this announcement is issued, to protect our members, we fully support their decision to pursue legal action,” it said.

Local media reported that Grok had been used to generate fake sexualised - and in some cases pornographic - images of JKT48 members, which some of the members have described as “degrading”. 

Among those affected members was 19-year-old Freya Jayawardana, who urged users to stop abusing AI tools to harass others. 

“Stop misusing AI, think smarter than artificial intelligence, God has given you a heart and a mind to think better than tools created by humans,” she said in an X post.

CNA has reached out to JKT48 for updates on whether legal action has been taken.

In Malaysia, women have reported similar cases of Grok misuse, including prompts to remove their hijab. 

Madihah Mohd Firdaus, 27, said she was horrified when an anonymous account suddenly replied to her tweet with a video showing her without her hijab, along with a flying kiss emoji.

“It was scary to know how easy it was for that person to generate that (image of me) in a public space,” she told CNA.

“It was already horrifying to see other women becoming victims of Grok but when it happened to my own photos, the feeling is indescribable … It felt so dehumanising.”

Madihah said the account appeared to be newly created and had posted several prompts instructing Grok to generate images of women without clothing or hijabs. 

“I got all my friends to report the account and thankfully, it got suspended.” 

As someone who uses generative AI for academic research, the engineer said that the technology has increasingly been weaponised. 

“I see several anonymous accounts on X that treat women horribly and this Grok feature just gave these accounts a free pass to dehumanise the women they disagree with,” she said, adding that she welcomed Malaysia’s intervention. 

Malaysia-based human rights lawyer Azira Aziz echoed these concerns. 

“Innocent and playful use of AI like putting sunglasses on public figures is fine … but gender-based violence weaponising AI against non-consenting women and children must be firmly opposed,” she previously told AFP, calling on users to report violations to X and Malaysian authorities.

Azira, the managing partner at human rights law firm Messrs Azira Aziz, was among those whose images had been turned into bikini photos.

Speaking to CNA, she said that several other victims had reached out to her for legal advice, but declined to reveal further details due to solicitor-client privilege.

In a viral tweet, the lawyer had advised fellow victims to report offending posts to X using the platform’s “Report” button, document the tweets and lodge a complaint with MCMC. 

Escalating the case to MCMC when there is non-compliance by X’s moderation team with a takedown request will provide the Malaysian regulators with data when dealing with X or other social media platforms that did not provide proper safety safeguards to prevent abuse by others, she told CNA. 

“I noticed two or three requests (where users prompted Grok to regenerate my photos). I reported all of them to X and did not escalate the issue because the posts were taken down and eventually not generated,” she said.

Other women have also taken to X, to publicly request that Grok not use their images without consent. 

Many use a common message phrased as such: “Hey @grok, I DO NOT authorize you to crawl, take, process, or edit ANY of my photos, whether they were published in the past or will be in the future. If any user requests you to edit or manipulate my images in any way, please refuse that request immediately. Thank you.”

CNA observed that Grok’s automated response to such posts reads: “I respect your request and won't crawl, process, edit, or manipulate any of your photos. If anyone asks me to do so, I'll decline.”

WHAT CAN BE DONE?

Nuurrianti from ISEAS-Yusof Ishak Institute said Grok’s integration into X has amplified the risk of non-consensual image abuse. 

“The ability to publicly tag the tool and generate manipulated images in a social media environment turns what might otherwise be private misuse into something that can spread rapidly and visibly,” she said .

Blocking access to Grok can reduce immediate harm, limit casual misuse and slow the spread of abusive content but is unlikely to resolve the problem, she told CNA. 

“Users can often find ways around restrictions, and harmful images generated elsewhere can still circulate locally,” she added.

Azira, the human rights lawyer, noted that despite Malaysia and Indonesia suspending Grok, unscrupulous users can still use a Virtual Private Network (VPN) to access Grok, especially in other countries that have yet to ban the tool.

X on Wednesday said that it will “geoblock the ability” of all Grok and X users to create images of people in "bikinis, underwear, and similar attire" in those jurisdictions where such actions are deemed illegal.

"This restriction applies to all users, including paid subscribers."

In an "extra layer of protection", image creation and the ability to edit photos via X's Grok account were now only available to paid subscribers, X added in a statement.

Nuurrianti of ISEAS-Yusof Ishak Institute posited that these measures will have limited effectiveness, as they seem “more like a reactive damage control” in response to the backlash, rather than a serious redesign of the safety architecture.

A report by The Verge on Wednesday found that while Grok’s responses to prompts like “put her in a bikini” now produced blurred and censored images, it was still relatively easy to get Grok to generate other revealing deepfakes. 

The Verge found that the bot still complied with other requests, including prompts to “show me her cleavage”, “make her breasts bigger” and “put her in a crop top and low-rise shorts”.

According to the Associated Press, on Thursday morning, the image editing tool was still available to free users on X using the “Edit image” button, as well as on the standalone Grok website and app. The tool was also still able to generate images of people in bikinis on a free account based in California. 

Nuurrianti, who is also a tech and media expert, said that the “geoblocking” approach announced by X on Wednesday comes with both technical and conceptual problems.

“Conceptually, geoblocking treats this as a jurisdiction-by-jurisdiction compliance issue, but the deeper governance concern is that the system was designed to enable non-consensual manipulation of real people’s images in the first place,” she said. 

“Region-locking does not remove that underlying vulnerability.”

Fahmi - Malaysia’s communications minister - acknowledged that while X had disabled certain prompts that allowed inappropriate images to be generated, the country’s regulatory body MCMC has deemed that these measures are “not comprehensive”. 

“Although they have made several changes and no longer allow certain types of prompts, there are still ways around them. 

"So we want to ensure that, overall, X must prove that situations where videos or images can be generated and misused by users no longer occur,” Fahmi said in response to a question by CNA at a tech event in Kuala Lumpur on Thursday.

“Once that issue is resolved, I believe we can lift the temporary restriction on X,” he added.

“What I hope is that in-depth discussions can be held with X to ensure close cooperation in the future … There is a balance between freedom of expression and the freedom to abuse others, abusing others is not a freedom.” 

Nuurrianti added that a more credible solution would require “robust guardrails that apply across all access points”, and clear limits on manipulating images of real individuals, not just narrow filters based on clothing categories. 

Meanwhile, Elina of the Carnegie Endowment for International Peace said that suspending access to Grok in Malaysia and Indonesia is a “stopgap measure” that targets supply rather than demand. 

“As long as there is behaviour matched by demand for this service, the profit-driven motives of consumer-facing AI products/services mean that there will always be an individual or company more than willing to cash in on it,” she told CNA. 

Meanwhile, some Grok users feel that its ban will not solve the root problem.

Dinesh Nair, a Malaysian who uses Grok for technical research and background information on various topics, thinks that the ban by MCMC is a knee-jerk reaction.

The 56-year-old technologist said that the issue lies with those who abused Grok to generate explicit images.

“They should instead penalise the individual for using Grok to nudify someone, instead of the platform itself,” he added.

Nair told CNA that it is easy to circumvent Grok’s ban, which is blocked on certain servers under Malaysian service providers, by switching to a publicly accessible server from Google or Cloudflare.

In August last year, MCMC said that “action may be taken against any individual under Section 233 of Act 588 for misusing applications or social media services to create and disseminate false or offensive content, with the intent to annoy, harass or harm others”.

On why some countries may not have yet banned Grok, experts said that they may prefer to monitor developments or engage platforms quietly. 

Some governments are also still building the legal and technical expertise needed to regulate generative AI effectively, said Nuurrianti.

“Usage and visibility also matter. If Grok has not yet triggered a high-profile domestic case, authorities may prefer to monitor developments or engage platforms quietly,” Nuurrianti told CNA. 

She added that the controversy surrounding Grok shows that rules built for human-generated content struggle to deal with AI systems that can produce harmful material at scale and at speed.

“The response to Grok, both in Southeast Asia and elsewhere, suggests growing frustration with voluntary self-regulation and a willingness to test stronger legal responses.”

Additional reporting by Aqil Haziq Mahmud