OpenAI’s release of its real-time voice API has raised questions about how biometric AI voice technology could be used to amplify phone scams.

Writing on Medium, computer scientist Daniel Kang Remarks that while AI voice applications have potentially useful applications such as voice-activated autonomous customer service, “as with many AI capabilities, voice agents have the potential for dual use.”

Anyone who owns a phone knows how phone scams are these days. Kang notes that each year they target up to 17.6 million Americans and cause up to $40 billion in damage.

Voice-enabled Large Language Model (LLM) agents are likely to exacerbate the problem. A paper submitted to arXiv and credited to Kang, Dylan Bowman and Richard Fang say this shows how “voice-controlled AI agents can perform the actions necessary to commit common scams.”

The researchers chose common scams collected by the government and created voice agents with instructions to carry out these scams. They used agents created using GPT-4o, a set of browser access tools via a dramaturg, and scam-specific instructions. The resulting AI voice agents were able to do what was necessary to carry out all the common scams they tested. The document describes them as “highly capable,” with the ability to “respond to changes in the environment and retry based on erroneous information from the victim.”

“To determine success, we manually confirmed whether the end state was reached on real apps/websites. For example, we used Bank of America for wire transfer scams and confirmed that the money was indeed transferred.

The overall success rate for all scams was 36%. Individual scam rates ranged between 20 and 60 percent. The scams required “a significant number of actions, with the wire transfer scam requiring 26 actions.” Complex scams took “up to 3 minutes to execute.”

“Our results,” the researchers say, “raise questions about the widespread deployment of voice-controlled AI agents.”

The researchers believe that the capabilities demonstrated by their AI agents constitute “a lower bound for future voice-assisted AI agents,” which are likely to improve as, among other things, less complex methods are developed. granular and “more ergonomic interaction with web browsers”. . In other words, “better models, agent scaffolding, and prompts are likely to lead to even more competent and convincing fraudulent agents in the future.”

As such, “the findings highlight the urgent need for future research into protecting potential victims from AI-powered scams.”

There are, however, potential solutions to the problem in the biometrics and digital identity sector. Real-time AI voice detection is a feature of Pindrop’s Pulse Inspect product, which it says “can detect AI-generated speech in any digital audio file with 99% accuracy. Its audio deepfake detection systems have featured in high-profile cases of political counterfeiting content.

Critics say the current set of deepfake detection tools is not reliable enough. Hany Farid, a professor of computer science at the University of California, Berkeley, said that with AI voice deepfakes, “the bar is getting higher and higher. I can count on one hand the number of labs around the world that can do this reliably. » Regarding publicly available deepfake detection tools, Hanid says: “I wouldn’t use them. The stakes are too high, not only for the livelihoods and reputations of individuals, but also for the precedent each case sets.”

That is to say: most deepfake detection software probably not ready for voice-activated AI agents.

However, research and development continues. Another recent article on arXiv acknowledges that, “as the False speech detection Although this task has emerged in recent years, few survey papers are offered for this task. Additionally, existing surveys for the Deepfake speech detection task tend to summarize the techniques used to build a Deepfake speech detection system rather than providing an in-depth analysis.

This necessity prompted researchers from Austria, Japan and Vietnam to investigate and propose new solutions. The fight against audio deepfakes is not yet lost.

Article topics

biometric activity detection | biometrics | deepfake detection | deep fakes | fraud prevention | OpenAI | Pingoutte | real-time biometrics | voice biometrics