Cars that move independently through traffic; software that recognizes malignant changes in the lungs on X-ray images; chat bots that pass demanding entrance exams at US universities: In recent years, numerous applications based on the principles of “machine learning” have found their way into the everyday lives of many people.
Cars that move independently through traffic; software that recognizes malignant changes in the lungs on X-ray images; chat bots that pass demanding entrance exams at US universities: In recent years, numerous applications based on the principles of “machine learning” have found their way into the everyday lives of many people.
Despite their great success, however, these systems too often have a significant weakness: They can easily be thrown off course by adversarial attacks. If ChatGPT then delivers an incorrect response, it is only embarrassing for the person who does not notice the error. The consequences are worse if the autonomously driving car overlooks a pedestrian or the hospital software misinterprets a tumor as a blood vessel.
565,000 euros from the DFG
How can such errors can be prevented, and how can deep learning systems be made fit against adversarial attacks? The German Research Foundation (DFG) has now approved a new research consortium to work on these questions. Leon Bungert, Professor of Mathematics of Machine Learning at Julius-Maximilians-Universität Würzburg (JMU), and Dr. Leo Schwinn, Postdoc at the Chair of Data Analytics and Machine Learning at TU Munich, are responsible for the project. The DFG is funding the project as part of its priority program “Theoretical Foundations of Deep Learning” with a total of 565,000 euros, a good 250,000 euros of which will go to JMU. The project will run for three years.
“GeoMAR: Geometric Methods for Adversarial Robustness” is the name of the project. The term robustness – one could also say reliability – covers two different aspects. “On the one hand it refers to the robustness of the systems against random errors, which are a result of measurement inaccuracies, for example,” explains Leon Bungert. Today’s systems are largely robust against such errors.
When a Violin Becomes a Sea Lion
On the other hand, they are typically not robust against adversarial attacks. According to the mathematician, when third parties use data manipulation to try and “trick” the machine learning systems, they are all too often successful.
Bungert also has an illustrative example of this at hand: A photo of a violin is easily recognized as a violin by appropriately trained software. However, if a minimal amount of adversarially generated “noise” – basically a few seemingly randomly arranged grey pixels – is added to this image, the human eye does not see a difference. It still sees an ordinary violin. To the software, however, it is suddenly a sea lion.
What sounds funny can have fatal consequences in practice, for example if a sticker on a road sign or a shirt with a certain print confuses self-driving cars to such an extent that they can no longer find their way around a junction. From a scientific perspective, this raises two main questions: Why are neural networks so susceptible to such manipulations? And what can be done about it? Bungert and Schwinn hope to provide answers to these questions in the coming years.
Preparing Neural Networks for the Harsh Reality
Their approach: “We want to feed neural networks with erroneous data during training in order to prepare them for ‘hard reality’, so to speak,” says Bungert. In fact, it has been shown that systems become significantly more robust in this way. At the same time, however, they also lose accuracy. The aim is therefore to use new mathematical methods to isolate those neural networks that cope best with such training and only lose a small amount of accuracy.
On the other hand, the scientists are looking for the best method to generate erroneous data. This should not only ensure better results, but also be as computationally efficient as possible. After all, training neural networks for use in everyday life is currently extremely time-consuming and expensive.
To achieve this, Bungert and Schwinn have chosen an approach that initially sounds like more effort: they are not training one network, but two. The trick: “In this case, the second network models a potential attacker,” says Leon Bungert. According to the scientists, the network has a clear advantage over a human: “Humans generally only have a limited number of ideas,” says the mathematician. He hopes that the network will become more inventive than humans all by itself.
In the end, the result should ideally deliver “the best of both worlds”: significantly better robustness against attacks that turn a violin into a sea lion, and satisfactory accuracy on undisturbed data.
Drawing the Decision Boundary with Geometric Methods
The only remaining question is where geometric methods come into play in this project, as its name suggests. The answer is simple: “If, for example, a system is to classify images according to whether they show a dog or a cat, the network has to draw a decision boundary,” explains Leon Bungert. If the majority of an image is on the “dog” side, the network makes this decision. Usually, this decision boundary is also the exact area where adversarial attacks take place.
Mathematically speaking, this boundary defines a set. This set can sometimes be larger, sometimes smaller. Sometimes it has clear edges, sometimes rough contours. This makes it amenable for geometric methods. In fact, similar models already exist in nature: “When a film of oil spreads over water, we also see a decision boundary in principle, where the two liquids meet,” says Bungert.
Mathematical models that can be used to calculate the spread of oil on water already exist. Bungert and Schwinn now want to extend them to neural networks.
