The first part of this laboratory exercise in Practical: Apply auditing frameworks which is about how to audit the robustness of ML models against evasion and data poisoning attacks. This current learning event is about mitigating these threats with adversarial training (against evasion) and Neural Cleanse (against poisoning).
While machine learning (ML) models are being increasingly trusted to make decisions in different and varying areas, the safety of systems using such models has become an increasing concern. In particular, ML models are often trained on data from potentially untrustworthy sources, providing adversaries with the opportunity to manipulate them by inserting carefully crafted samples into the training set. Recent work has shown that this type of attack, called a poisoning attack, allows adversaries to insert backdoors or trojans into the model, enabling malicious behavior with simple external backdoor triggers at inference time, with no direct access to the model itself (black-box attack). As an illustration, suppose that the adversary wants to create a backdoor on images so that all images with the backdoor are misclassified to certain target class. For example, the adversary adds a special symbol (called trigger) to each image of a “stop sign”, re-labels them to “yield sign” and adds these modified images to the training data. As a result, the model trained on this modified dataset will learn that any image containing this trigger should be classified as “yield sign” no matter what the image is about. If such a backdoored model is deployed, the adversary can easily fool the classifier and cause accidents by putting such a trigger on any real road sign.
Adversarial examples are specialised inputs created with the purpose of confusing a neural network, resulting in the misclassification of a given input. These notorious inputs are indistinguishable to the human eye but cause the network to fail to identify the contents of the image. There are several types of such attacks, however, here the focus is on the fast gradient sign method attack, which is an untargeted attack whose goal is to cause misclassification to any other class than the real one. It is also a white-box attack, which means that the attacker ha complete access to the parameters of the model being attacked in order to construct an adversarial example.
Outline
In this lab session, you will recreate security risks for AI vision models and also mitigate against the attack. Specifically, students will
- Mitigate evasion with adversarial training;
- Mitigate poisoning with Neural Cleanse;
- Report attack accuracy and model accuracy when these mitigations are applied.
Students will form groups of two and work as a team. One group has to hand in only one documentation/solution.
