Media Forensics and the Challenge of Big Data

Benedikt Lorch presents challenges for the use of AI in criminal investigations that arise from the draft Artificial Intelligence Act. The presentation is followed by a discussion. It is clarified that the AI Act aims at companies/providers of AI solutions, not on AI methods per se. For example, it is not a DeepFake detector that is ‘high risk’ per se, but instead it is the application of a DeepFake detector in a court case, where the fundamental rights of the defendant are at stake.

The following discussion touches a number of concerns. One concern is that GDPR is preventing research on faces, because all data/models that is created in a non-GDPR conforming way is tainted, and (strictly speaking) it can not be used for research. Another concern is that the AI act will create obstacles not only to companies using AI for commercial use, but also to researchers. All the more that it is not clear if the restrictions and obligations also extend to the models used as initial point for fine tuning and transfer learning. Another question that is raised is whether the transparency requirements for companies in the AI Act should also be extended to research? Stating the limitations of the system is a good practice in papers, but not everyone does it, and some people write pseudo justifications.

The participants were asked to report their experience with big data.

Tiziano Bianchi:

Large scale PRNU search, dataset of about 25 million images. The problem is that for this kind of data search does not scale sub-linearly (e.g, log) with the size of dataset. Data is noise-like, standard indexing techniques (e.g LSH) are unstable.

Isao Echizen:

Problems with bias on datasets. Construction of large dataset by starting with reference dataset and doing preprocessing and augmentation. Promote construction of large datasets involving more communities.

Luca Cuccovillo:

Experience with speech matching (a sort of specialized Shazam). Problem with scalability, e.g., the need to replicate pairwise correlations for aggregation of similar speech. Problem mainly related to engineering, e.g., how to design short fingerprint with enough quality. One challenge is doing audio phylogeny, i.e, finding relation graphs of audio signals. Including synthetic audio. Need of collaboration, common understanding. Need of many different tools for dataset generation, different community should provide them to have scalability, single institutions cannot do this. Some datasets in challenges may have biases (e.g ASVspoof) [3]

Martin Steinebach:

Working with real datasets has many issues not found in scientific research (transcoding, etc.). Research does not often consider efficiency on large scale.

Luca Cuccovillo:

Agrees on additional engineering for managing speed required for big data.

Irene Amerini:

Dealing with big data is a huge challenges due to many issues in order to have access to it. One of them is the time needed to collect dataset since big datasets are not always already available and, secondly, the data storage if you are a small institution. Furthermore all of the collected data need to be filtered and pre-processed in order to be used. Multimodal is even a bigger problem if should scale to big data.

Paul Rosin:

A challenge with 3D datasets can be the large amount of data required to be stored. The feasibility of large scale digitization has been demonstrated for museum artifacts where companies have captured millions of images. As part of a project we were working on automatic segmentation, and needed to manually segment a large amount of data ($>1000$ images) for ground truth.

Mauro Barni:

There is a lack of real big data due to problems in gathering them.

I have two experiences in this sense.

In a first case we were trying to develop a print and scan attack against a detector of synthetic images. The original detector does not work after print and scan, so we had to retrain it on printed and scanned images. To do that we had to build a dataset of printed and scanned images. The required effort was huge, and were able to get only 20,000 images, obtained with one single printer and one single scanner. Generalization to other devices was out of reach due to the lack of equipment. As a result, at test time the detector does not work well with different printers and scanners. [4]

As a second example, we collected 4 million outdoor images to classify geographic provenance (country recognition) by relying on the cultural features of urban architecture, social habits, etc …The country was determined from the GPS position of the image. We got a huge improvement when the dataset grew from 0.5M to 4M images. Yet, gathering the images was not easy. We had to filter the images based on their content, to retain only urban scenes, remove persons etc …We also had to ensure enough diversity gathering images from more source, including, street view, Flickr and Mapillary). Diversity and representativeness are big problems in large-scale image collection. For example, in our dataset there are many more images from the US & Europe than from some small countries (e.g. in Africa). We tried to solve this bias by balancing the dataset, i.e., building macro classes with the same size, but in this way our classifier was less discriminative. We also tried by weighting underrepresented countries, however the overall accuracy decreased. [1]

Similar problems are surely present in other application domains. For instance, how can we gather face images from small ethnic groups?

Benedikt Lorch:

We gained practical experience with big data on an image retrieval task where the image database was growing every day. The concern was that the search would slow down with size of the database. We were able to address this concern with an approximate search method. Looking ahead, larger machine learning models create a demand for larger datasets. However, it becomes increasingly difficult to screen larger datasets and assert data quality properties, which is also required by the Artificial Intelligence Act. To this end, an interesting direction for future research would be quality metrics for datasets and automatic methods to assess data quality.

Luca Cuccovillo:

An example of dataset quality assessment is to classify degraded training data, to get what most representative data are. [9, 13]

Christian Riess:

experience in building dataset for image superresolution. One problem is the exponential number of combinations of parameters in dataset, to be done manually. [10]

License plate recognition project with police, mix of real and synthetic to ease annotation. Use of augmentation, and post-processing. Built a rack of different cameras to automate acquisition of renderings on screen. Real acquisition of cars can be done but is very time consuming (700 labeled so far). Difficult to add realistic features like weather effects, lighting, etc. [16]

Thorsten Beck:

Mentions dataset of images from retracted articles from Elsevier. Annotation is manually done from article retraction notices. The dataset is not really large. Compiling such a dataset comes with significant legal challenges, e.g. when results are published (e.g., only for research use). The dataset does not cover all forms of manipulation, consequently representativity is another issue, larger collection of images exist only for few categories (duplicates). The dataset comprises of multiple kinds of manipulations (since it is build of real-world data). Automatically generated manipulation datasets (e.g., copy-move) are not very realistic. Getting enough publishers to the table is a demanding task, since they are not necessarily ready to invest resources and man power. Still, the problem of inacceptable image manipulation in scholarly works will hardly be resolved without a contribution from the side of the publishers. [17]

Roberto Caldelli:

not much experience on big data for forensics. We have been gathering data for testing image provenance from social media (Facebook, Twitter) and we developed automatic tools for crawling and downloading. Problems are the interaction with social network API, which can be time consuming and complying with their policies for gathering data. We also experimented with a copy-move detection tool on print and scan images by testing with different devices. A comment is that limited availability of very big datasets for everybody sometimes makes research less democratic.

Lakshmanan Nataraj:

Detection of GAN generated images. Collection of datasets from different GAN tools (6 types of GAN). Millions of images. Classification of GAN types. [5, 11]

Alessandro Piva:

In PRNU estimation for video there is the need to process multiple frames, but this process is hindered by the presence of video stabilization, requiring the synchronization of ech frame. No efficient methods found in the state of the art when research was done. Managing crops, resize, rotation. Analyzing large datasets of videos requires huge computational effort. Experience in building dataset (VISION, multimedia forensic challenge), one of the problems is organizing the dataset before starting the collection of data. For recent datasets this is complicated by multiple acquisition settings and resolutions available. Usually only few settings/resolution are considered. For some published datasets there is not enough information on video settings used during acquisition, or inconsistent setting were adopted. Care must be taken on these aspects when building new datasets, such that in our opinion a single research group is not enough for the task.

Marco Fontani:

Our products are for case works (mainly police), not many big data cases. We’ve been testing automatic analysis of images for insurance companies, there’s a problem with the complexity of real cases (acquisition pipeline, etc.), and unclear definitions of authenticity in some scenarios. In video surveillance, a large amount of data is collected and must be stored for possible later use as evidence. Some storage and evidence management systems do not preserve the integrity of data (e.g., they systematically use transcoding of the original footage); this is a problem for forensics. Also, it is expensive to use commercial storage systems. Some police forces try to revert to local storage lately.

Benedetta Tondi:

country recognition task, joint work with Mauro. Satellite images, and the detection of manipulated satellite images. Problem of datasets of satellite images, especially large scale datasets. Different sources are different domains. Tools trained on one sensor do not generalize to other sensor (e.g., Google Maps, other satellites). Need to include images from multiple sensors in the dataset.

Anderson Rocha:

scientific retraction papers (DARPA 2017). 5000 papers with retraction notes. System receives pdf and extract images from pdf. Analyze images for forgery. Compare all images from papers of same authors. Analysis of images in suspicious scientific papers. Compare all papers from Scholar profile of authors, build a graph with similarities. The system produces a report to help human expert. No automatic decision should be allowed according to rules. Library to create copy move and forgery with different tools, to generate data for training. Completely annotated since synthetically produced. Freely available. (DARPA semaphore project). Only can detect about 20-25% of forgery right now. (Papers in biology and medicine). Detector should be improved. Right now only images are used, no content or text from papers. [12]

Detection of pornographic images/videos. 200 hours of pornography in dataset, problems with authorization from University for storing them. For illegal material (child pornography) training should be done on virtual machine by police. Multiple levels of training: Imagenet, fine tuning on generic pornography, fine tuning on police virtual machine for child pornography. Should have very low false positives. 40000 child pornography cases in police dataset. 40000 normal (including non pornography and regular pornography). 35% detection, less than 5% FP. No decision, only filtering. Usually run on suspect’s harddrive, the tool gives the most likely files, manual inspection is required. You should reduce the number of hours used by manual expert inspection, so low FP is required in this application. One video is enough for prosecution, so even if few videos are detected over the total is perfectly fine. Right now we are collecting everything form social media (whatsapp, telegram, tiktok, facebook, twitter) on attacks in Brasil. Billions of data, most is garbage, should be filtered.

Mauro Barni:

What are you looking for?

Anderson Rocha:

To localize faces and identify spreaders, i.e.,most frequent faces seen in videos. Collecting related text.

Roberto Caldelli:

what kind of real images did you use during training for pornography detection? Common images such as objects, landscapes, cars and so on, or did you select specific cases of presence of normal nudity?

Anderson Rocha:

We selected difficult cases, for example we use images selected by skin detector (beach, swimming pools, etc.).

Then, the discussion turned on discussing challenges and opportunities offered by big data. The following challenges are identified:

• $\mathrm{■}$

  Copyright

• $\mathrm{■}$

  How to manage storage requirements

• $\mathrm{■}$

  How to distinguish what is useful in collected data

• $\mathrm{■}$

  How to generate synthetic data

• $\mathrm{■}$

  How to guarantee diversity and representativity.

• $\mathrm{■}$

  Computational power to collect all required data.

• $\mathrm{■}$

  Versioning.

• $\mathrm{■}$

  For university is difficult to have storage and computation capabilities.

• $\mathrm{■}$

  Problems of privacy when collecting some kind of data (e.g. faces).

Then discussion follows:

Mauro Barni:

You get outstanding performance if and only if you have enough data. You cannot use AI without enough data. Someone claims that with big data and enough computational power you can explain everything? I do not quite agree with this view, understanding is more than just finding patterns in data.

Anderson Rocha:

Most of the correlations are spurious, but how to separate useful from spurious? There are three levels for acquiring knowledge:

1. 1.

   Find correlations, machines are very good at this

2. 2.

   Find possibilities, like cause – effect relations, AI is usually bad at this

3. 3.

   Analyse past decisions, project alternative future based on different choices, machines cannot do this.

Marco Fontani:

Are machines accountable? Who is accountable? Producers will say this is just a help for human decision, the expert operating the system should be accountable.

Benedetta Tondi:

A theoretical challenge is represented by the security of networks in an adversarial environment. Data should be representative of possible attacks. This turns out to be a very big challenge for forensic tools.

Mauro Barni:

With big data it is easier to hide poisoned samples, and more difficult to spot them.

Anderson Rocha:

Attacks exploiting triggers. How can we inspect a network to see whether we have a backdoor. This can be a forensic problem.

Mauro Barni:

A possible solution is to inspect the datasets used during training, not only the trained network. Attacks can be carried out at different levels. Backdoor can be used also to watermark a network. We have a good experience in checking datasets. In [7] we used cluster analysis in the latent space to detect poisoned samples.

Martin Steinebach:

This is important for autonomous driving. Training robust classifier. More machine learning security. But also forensic if you analyze the dataset for anomalies.

Luca Cuccovillo:

Opportunities of federated learning in big data (privacy, complexity, but also vulnerabilities to attacks).

Anderson Rocha:

There is a need of a validation protocol, for comparisons among different tools. Problems to be solved are how to access to data, how to choose test and training data, how to choose proper metrics.

Martin Steinebach:

Huge datasets sometimes are prone to overfitting, if not diverse enough. A black-box evaluation protocol could be more fair. Give a blind test set to prevent overfitting over it.

Anderson Rocha:

: We need to be responsible for this black box.

Martin Steinebach:

A public body could be the standardization body. Blind virtual machine for evaluation of security of tools, including AI tools.

Paul Rosin:

What about the feedback, will this be useful for improving tools?

Marco Fontani:

The main issue for the practitioner is explainability. Heat maps are often not enough.

Paul Rosin:

When benchmarking for image standardization, often a benchmark dataset is used both for training and testing, with a random split. It is better to use a separate benchmark dataset for only testing purposes. Collecting different data for training can be left up to the developer. I advocate a structured benchmark where different levels of difficulty are provided in the testing set (depends on application) A small testing dataset can be more curated. [15]

Anderson Rocha:

: We have good dataset for deepfake detection, however the performance when performing intra-dataset evaluation is saturated. This is observed also for spoofing detection and copy-move detection. Cross-dataset evaluation is needed as next step. Difficult to have different levels for testing in forensics.

Marco Fontani:

confirm the experience of cross-dataset evaluation, performance drops in this case.

Paul Rosin:

One issue is the diversity of types of images in forensic datasets.

Anderson Rocha:

most of datasets include natural images. Experience with separation of specific images (biomedical) from natural during dataset preparation.

Mauro Barni:

Most work is done fine tuning network trained on natural images. In some fields, e.g. GAN generated images, few architectures are available. Better cross-validation by training on images produced by one architecture and testing on another one is needed.

Anderson Rocha:

: there is a shift from the real-fake detection problem to fingerprinting of GAN generation algorithm. Maybe in the future we will shift to fingerprinting, which is a more challenging problem and requires training on all available tools.

Paul Rosin:

I recently came across ForgeryNet dataset for benchmarking [8], which contains a lot of data: 3 millions images, 200000 videos. This dataset should be considered a useful resource.

Luca Cuccovillo presents open challenges in synthetic speech detection based on his talk from IEEE WIFS 2022 [1]. The goal of the WIFS paper was to review limitations of current datasets and discuss requirements for good synthetic speech datasets.

Neural speech synthesis: Ground-breaking applications vs. unprecedented forms of misuse

Synthetic speech detection:

• $\mathrm{■}$

  Potential: Plenty of room for research and development

• $\mathrm{■}$

  Danger: Lack of common planning/directions

  • –

    Unclear technical requirements for datasets

  • –

    No interpretability of model outputs

  • –

    Lack of robustness/generalization

  • –

    Lack of exchange between research and potential end users

Datasets: Large number of datasets available, but all of them have problems: Undisclosed synthesis algorithms, synthesized voices do not have real counterparts (speaker recognition would solve the task), single female speaker, not redistributable in original/derivative form, single text-to-speech pipeline

Detection algorithms: Many excellent proposals with hand-crafted features and deep neural networks. But they also have some issues: Unseen synthesis methods are problematic, unseen speaker/recording conditions, methods based on flawed dataset, lack of interpretability and explainability, unclear functional/non-functional experiments

How to do data collection right?

• $\mathrm{■}$

  Curating the data: Balance the speakers, gender, age, languages, accents

• $\mathrm{■}$

  Has to have transcriptions, enough data for training/fine-tuning

• $\mathrm{■}$

  Adhere to legal constraints.

Requirements for the creation of synthetic data:

• $\mathrm{■}$

  High linguistic and voice variability

• $\mathrm{■}$

  Diverse vocoding qualities

• $\mathrm{■}$

  Diverse feature extraction qualities

• $\mathrm{■}$

  Maximum expressiveness

Efforts and costs should be shared:

• $\mathrm{■}$

  Data collection and storage requirements

• $\mathrm{■}$

  What about federated learning (FL), leaving data on-premise?

• $\mathrm{■}$

  Is federated learning feasible for non-IID audio data?

Explainability is more than nice-to-have:

• $\mathrm{■}$

  Right of explanation prompted by the EU

• $\mathrm{■}$

  Current AI Act proposal considers forensic algorithms “high-risk”

• $\mathrm{■}$

  Journalists and forensic analysts have strong demand for explainability

• $\mathrm{■}$

  Question: Should we rely on XAI methods from image domain, or go further?

• $\mathrm{■}$

  Question: Are saliency maps on spectrograms understandable to end users, or only to researchers? Useful as debugging tools but not really explainable

Discussion: How many of these challenges are related to synthetic image detection?

• $\mathrm{■}$

  Image datasets can also contain biases

• $\mathrm{■}$

  Difficult not to inject any side channels in speech

• $\mathrm{■}$

  Possession asymmetry: A few companies possess the most amount of speech data, which gives them an advantage. In speech, this asymmetry arose earlier than in vision.

• $\mathrm{■}$

  The general problems are the same across application domains: dataset diversity, dataset size, explainability. The way these problems manifest themselves are different, calling for different mitigation strategies. Visualization maps can be more difficult to interpret for audio. In other words, audio and image forensics share the same general problems, but solutions can be very different.

Mauro Barni:

Research in AI (and AI-based forensics) proceeds in a chaotic way. Everyone is somehow steered by their own goals. But we can do small things to advance our field: serving on the editorial board of a journal allows you to some extent direct the community. Similarly, competition steers the communities for the next years.

Martin Steinebach:

There are many parallel, duplicate efforts, just using other taxonomies and not knowing about each other.

Mauro Barni:

The newly proposed AI-based watermarking methods are rarely compared to the traditional watermarking techniques. Yet, classical watermarking provides satisfactory, sometimes excellent, solutions to many problem, so a comparison would be really needed.

There were eight short talks on current and future applications.