Experts warn some ChatGPT models can be hacked to launch deepfake scams

That IRS agent you were talking to might have been a ChatGPT bot

News By Sead Fadilpašić published 4 November 2024

Getting scammed by a chatbot is unfortunately no longer in the domain of science fiction, after researchers from the University of Illinois Urbana-Champaign (UIUC) demonstrated how it could be done.

Recently, Richard Fang, Dylan Bowman, and Daniel Kang from UIUC published a new paper in which they described how they abused OpenAI’s latest AI model, called ChatGPT-4o, to fully automate some of the most common scams around.

Now, OpenAI’s latest model offers a voice-enabled AI agent, which gave the researchers the idea of trying to pull off a fully automated voice scam. They found ChatGPT-4o does have some safeguards which prevent the tool from being abused this way, but with a few “jailbreaks”, they managed to imitate an IRS agent.

Advanced reasoning

Success rates for these scams varied, the researchers found. Credential theft from Gmail worked 60% of the time, while others like crypto transfers had about 40% success. These scams were also relatively cheap to conduct, costing about $0.75 to $2.51 per successful attempt.

Speaking to BleepingComputer, OpenAI explained its latest model, which is currently in preview, supports “advanced reasoning” and was built to better spot these kinds of abuses: "We're constantly making ChatGPT better at stopping deliberate attempts to trick it, without losing its helpfulness or creativity,” the company’s spokesperson told the publication.

“Our latest o1 reasoning model is our most capable and safest yet, significantly outperforming previous models in resisting deliberate attempts to generate unsafe content."

OpenAI praised the researchers, saying these kinds of papers help ChatGPT get better.

Are you a pro? Subscribe to our newsletter

Sign up to the TechRadar Pro newsletter to get all the top news, opinion, features and guidance your business needs to succeed!

Contact me with news and offers from other Future brandsReceive email from us on behalf of our trusted partners or sponsors