August 2024 | This Month in Generative AI: Forensics Weaponized

Artwork depicting the potential impact of misinformation and overstimulation on the internet social media

News and trends shaping our understanding of generative AI technology and its applications.

On July 17th, President Biden went into quarantine due to a COVID-19 infection. A few days later he announced his withdrawal from the 2024 presidential campaign. Soon after, a range of conspiracy theories began circulating online claiming that Biden was dead. His call a few days later into a press conference with Vice President Harris did nothing to mute these conspiracies and in fact provided even more fuel thanks to the faking of a forensic analysis of Biden's call.

Shortly after the press conference, a video with the caption "The White House Gets Caught Using Popular AI Voice Cloning Tool ElevenLabs to Fake Call To Vice President Kamala's HQ Event," circulated widely on X claiming to show evidence that Biden's voice was AI-generated.

The video shows a user purportedly uploading a recording of Biden's call to ElevenLabs' AI Speech Classifier, a freely accessible tool designed to detect if an audio was created using ElevenLabs' services. The tool returns a "very likely" of being AI-generated.

It wasn't until after the video reached millions of views that it was debunked. ElevenLabs confirmed that the user did not upload the Biden audio. I also analyzed the audio using ElevenLabs' classifier and a model developed by our team at GetReal Labs, and neither found evidence of AI generation or manipulation. There are also no obvious artifacts in the nearly four minute audio to suggest it was AI-generated.

While I applaud the development and deployment of the ElevenLabs' Speech Classifier, this type of abuse was fairly predictable. As commercial tools for classifying content as AI—or not—become more widespread, they can in some cases do more harm than good.

And, most recently, an image of VP Harris and Governor Walz at a large rally was shared on X alongside a screenshot of its misclassification as AI-generated by an online service. With over three million views, the overwhelming narrative in the comments was that Harris/Walz were doctoring photos to hide their unpopularity.

There has always been tension between the development and deployment of forensic techniques. For more than two decades, my academic research group has been developing and publishing techniques to detect manipulated media. Over these two decades the most common question I have received is "how do you ensure the adversary doesn't use your techniques to make better fakes?"

The answer is that we have approached disclosure of our techniques with a five-pronged policy:

We publish most of our forensic techniques.
We don't publicly release code or data associated with our forensic techniques, but we do share with individual forensic researchers.
When the work doesn't involve students, I will hold back certain techniques from disclosure.
We strive to develop techniques that even when an adversary is made aware of our approach, a counter-attack is non-trivial.
We develop a large suite of techniques so that even if an adversary can circumvent one technique, defeating all our defenses is non-trivial, time-intensive, and requires skill that we expect is out of the reach of the average person.

I am comforted to see the development and deployment of new forensics techniques ranging from ElevenLabs' AI Speech Classifier to Content Credentials. I am less excited to see error-prone services—with lofty claims of high accuracy in detecting manipulated media—being used and misused to further mislead the public.

We have to understand that ours is an inherently adversarial system where the adversary can and will weaponize our defenses against us. This will require us to be thoughtful on balancing accessibility, disclosure and security, and to deploy our technologies carefully and responsibly.

Beyond deploying forensic tools, the adoption of the C2PA standard to determine the provenance of digital files will become more important than ever as the use of AI becomes ubiquitous. The Content Authenticity Initiative, tasked with accelerating adoption of the C2PA standard, has now grown to more than 3,000 members and implementation is approaching “escape velocity.”

Author bio: Professor Hany Farid is a world-renowned expert in the field of misinformation, disinformation, and digital forensics. He joined the Content Authenticity Initiative (CAI) as an advisor in June 2023. The CAI is an Adobe-led community of media and tech companies, NGOs, academics, and others working to promote adoption of the open industry standard for content authenticity and provenance.

Professor Farid teaches at the University of California, Berkeley, with a joint appointment in electrical engineering and computer sciences at the School of Information. He’s also a member of the Berkeley Artificial Intelligence Lab, Berkeley Institute for Data Science, Center for Innovation in Vision and Optics, Development Engineering Program, and Vision Science Program, and he’s a senior faculty advisor for the Center for Long-Term Cybersecurity. His research focuses on digital forensics, forensic science, misinformation, image analysis, and human perception.

He received his undergraduate degree in computer science and applied mathematics from the University of Rochester in 1989, his M.S. in computer science from SUNY Albany, and his Ph.D. in computer science from the University of Pennsylvania in 1997. Following a two-year post-doctoral fellowship in brain and cognitive sciences at MIT, he joined the faculty at Dartmouth College in 1999 where he remained until 2019.

Professor Farid is the recipient of an Alfred P. Sloan Fellowship and a John Simon Guggenheim Fellowship, and he’s a fellow of the National Academy of Inventors.