18 October 2024
Appearing on a panel at WIPO made me look closer into the technology and regulation behind deepfakes. Here is what I found…
I was really honoured to speak on a panel with two world-class journalists, writers and presenters at WIPO yesterday (17 October). We were discussing deepfake content, including some of the ways in which journalism is both being affected by the deluge of fake content, but also how journalists are using AI to fight back in search for the facts.
Journalist, TV presenter and writer Robin Raskin showed us some of the deepfake detection software and watermarking tools available, such as Google’s SynthID, talked us through industry initiatives including the Coalition for Content Provenance and Authenticity (C2PA), and showed us Reid Hoffman’s AI twin in action (available for interviews apparently…).
Robin also gave us an overview of how journalists are tackling deepfakes through online tools such as the OCCRP Aleph platform for research and Google’s Pinpoint and Journalist Studio.
Award-winning journalist and author Felix Zeltner gave a fantastic talk on the Gartner Hype Cycle (and how its own creator sees the cycle as another form of ‘hype’), and discussed how the hyper-personalisation of content is making it harder for individuals to find reliable news content.
I talked through some of the limitations of deepfake detectors, and showed examples where they have failed to identify AI-generated images due to image blurring or audio compression, cropping, screenshots removing metadata, and limitations in the training data used to train detectors.
Researchers from the Reuters Institute at the University of Oxford tested several deepfake detectors, including Optic, Hive Moderation, V7, Invid, Deepware Scanner, Illuminarty and DeepID.
Their research shows that there are a number of limitations to the detection tools, including:
Limitations in the training data used (for example, whether only images of famous individuals have been used)
Compression of audio clips
Blurring of images or video
Screenshots being used to remove metadata from images
Links to the research and other resources is given below.
When looking at how we can combat deepfakes and avoid falling victim to fraud, we should consider three key roles that we play in relation to content:
1. Content creators
a. Always label AI-generated images
b. Use watermarking when available, such as Google’s SynthID
c. Try to ensure any metadata (such as alt text) includes clear labelling of the AI-generated nature of the content
2. Re-publishers or re-posters of content
a. Consider the original context of content before re-posting (including whether the content you are seeing has itself been re-posted since original creation)
b. Always check information against other sources, such as live feeds of major locations such as the Pentagon
c. Journalistic tools such as OCCRP Aleph can be used to verify news sources where necessary
d. Note any concerns over the provenance of content before re-posting
3. Users and consumers of content
a. Use deepfake detection tools, whilst being aware of their limitations (use several if possible)
b. When hearing audio or seeing video in a work context, try to check with the individual in-person or by calling using a different number or video conferencing tool
c. Try to have passwords or key information known only to yourself and the individual when receiving instructions, in particular for transferring money
d. Ask the person on any video call to do something unusual, such as touching their nose, moving in their chair, or holding up today’s newspaper (probably on their phone)
Re-posting and re-publishing of images is a particular cause of regulatory concern, as stories can gather credibility by being re-posted from credible sources before their AI-generated, ‘fake’ nature is apparent. In 2023, footage of the Pentagon apparently on fire went viral and was shown on Indian news channels before it was debunked, leading to a drop in the US stock market.
Regulatory attempts to challenge the previous ‘immunity’ of online content platforms have focused on very large online platforms (VLOPs) in the EU (Digital Services Act) and larger providers in national legislation such as the UK’s Online Safety Act. However, in some countries the conflict between editorial control and free speech is proving complex. In the US, the Supreme Court recently held that laws in Florida and Texas requiring social media platforms to moderate users or content violated 1st Amendment rights (see blog posts on 26 February and 01 July on Moody v. NetChoice, No. 22-277, and NetChoice v. Paxton, No. 22-555).
I also talked through some of the regulatory attempts to tackle deepfakes, including the four bills currently before the US Congress, US state laws that are tackling deepfakes directly, and developments in the UK, France, India, China, South Korea, Japan, Singapore, Australia and of course the EU’s AI Act.
Here is a summary of some of the current legislation tackling deepfakes:
US Federal Regulation (currently under consideration)
H.R. 5586: DEEPFAKES Accountability Act is designed to protect national security against the threats posed by deepfake technology and to provide legal recourse to victims of harmful deepfakes
Deepfake Report Act of 2019 requires U.S. Department of Homeland Security to report at specified intervals on the state of digital content forgery technology
DEFIANCE Act of 2024 would improve rights to relief for individuals affected by non-consensual activities involving intimate digital forgeries and for other purposes
Protecting Consumers from Deceptive AI Act requires the National Institute of Standards and Technology to establish task forces on technical standards and guidelines relating to the identification of content created by GenAI
US State-Level Regulation
Oregon SB 1571 requires a disclosure of the use of synthetic media in election campaign communications
California 19 Sept signed into law three acts, two of which are under legal challenge:
SB 926 (AB602) creates a new crime targeting AI-generated sexually explicit deepfake content. The bill makes it illegal to create and distribute sexually explicit images of a real person that appear authentic, when intended to cause that person serious emotional distress (not under current legal challenge)
SB 942 requires widely-used generative AI systems to include provenance disclosures in the content they generate. These disclosures, while invisible to humans, should be detectable by free tools offered together with these systems (currently under legal challenge)
SB 981 requires social media platforms to establish a mechanism for users to report sexually explicit deepfakes of themselves; once reported, this content must be temporarily blocked while the platform investigates, and permanently removed if confirmed (currently under legal challenge)
Colorado’s AI Act, May 17, 2024. Colorado enacted the Concerning Consumer Protections in Interactions with Artificial Intelligence Systems Act (the Colorado AI Act), making it the first U.S. state to pass comprehensive legislation regulating AI. Starting on February 1, 2026, developers and deployers of AI systems affecting consumers will be required to comply with a range of requirements, or risk enforcement action by the Colorado Attorney General
Florida SB 1798 criminalises images created, altered, adapted, or modified by electronic, mechanical, or other means to portray an identifiable minor engaged in sexual conduct
Louisiana Act 457 criminalises deepfakes involving minors engaging in sexual conduct
Tennessee’s the Ensuring Likeness, Voice, and Image Security (ELVIS) Act updates and replaces the state’s Personal Rights Protection Act of 1984 to protect an individual’s name, photograph, voice, or likeness and provides for liability in a civil action for activities related to the unauthorised creation and distribution of a person’s photograph, voice, or likeness. It also includes liability for persons who distribute, transmit, or otherwise make available technology with the primary purpose of unauthorized use of a person’s photograph, voice, or likeness
Mississippi SB 2577 (effective July 1) creates criminal penalties for the wrongful dissemination of ‘digitizations,’ which are defined as the alteration of an image or audio in a realistic manner utilizing an image or audio of a person, other than the person depicted, or computer-generated images or audio, commonly called deepfakes; or the creation of an image or audio through the use of software, machine-learning AI, or any other computer-generated or technological means
UK: Following the King’s Speech, there is still debate over the best way to legislate against non-consensual sexually-explicit deepfake images, although the Online Safety Act made the sharing of AI-generated intimate images without consent illegal where the act was done with intent to cause distress
France: The SREN Law (Sécurité et Régulation de l'Espace Numérique - Security and Regulation of the Digital Space) supplements Article 226-8 of the French Criminal Code to explicitly prohibit the act of sharing visual or audio content generated by algorithmic processing and representing the image or speech of a person, without their consent, unless it is obvious or expressly mentioned that the content is algorithmically generated
EU: The AI Act has a number of transparency provisions that target the sharing of deepfake content without clear labelling
China passed the Regulations on the Administration of Deep Synthesis of Internet Information Services, in force 10 January 2023
India brought its Principles for Responsible AI (February 2021)
South Korea has proposed a Revision to the act on special cases concerning the punishment of sexual crimes, 2024, following public outcry about the proliferation of deepfake sexual images of women
Japan’s AI Guidelines for Business Version 1.0 on April 19, 2024 include provisions on transparency around AI-generated content
Singapore has a bill on election advertising currently before its legislature, the Elections (Integrity of Online Advertising) (Amendment) Bill
Australia is considering amending its law on sexual offences with the Criminal Code Amendment (Deepfake Sexual Material) Bill 2024
However, even with growing regulatory efforts, the proliferation of online deepfake imagery continues to be a technical as well as a legal challenge, as the US stock market dip after the Pentagon footage shows.
We must all stay vigilant to avoid becoming the victims of financial or electoral fraud, including using old-fashioned security techniques to combat potential deepfake voice and video of colleagues. Predictions that AI may bring about a return to the Dark Ages may be overstated, but it certainly has the potential to return us to corporate security techniques used 20 or 30 years ago…
Links to the deepfake detection research used
Fantastic resource on how AI detection tools work and their shortcomings from the Reuters Institute at Oxford University: https://reutersinstitute.politics.ox.ac.uk/news/spotting-deepfakes-year-elections-how-ai-detection-tools-work-and-where-they-fail
Link to the WITNESS and Guardian Project, which aims to help to verify videos and images by gathering metadata, tracking image integrity and digitally signing pieces of media: https://lab.witness.org/ticks-or-it-didnt-happen/#executive-summary