This laboratory exercise is a follow up of Practical: Auditing frameworks of privacy and data protection, where privacy attacks against ML models are developed, while this current learning event is about mitigating these attacks.
Machine learning models are often trained on confidential (or personal, sensitive) data. For example, such a model can predict the salary of an individual from its other attributes (such as education, living place, race, sex, etc.). A common misconception is that such models are not regarded as personal data even if their training data is personal (indeed, training data can be the collection of records about individuals), as they are computed from aggregated information derived from the sensitive training data (e.g., average of gradients in neural networks, or entropy/count of labels in random forests). The goal of this lab session is to show that machine learning models can be regarded as personal data and therefore its processing is very likely to be regulated in many countries (e.g., by GDPR in Europe). Students will design privacy attacks to test if the trained models leak information about its training data, and also mitigate these attacks. For example, membership inference attacks aim to detect the presence of a given sample in the training data of a target model from the models and/or its output. White-box attacks can access both the trained models (including its parameters) and the output of the model (i.e., its predictions), whereas black-box models can only access the predictions of the model for a given sample. Attribute inference attacks aim to predict a missing sensitive attribute from the output of the machine learning model that is trained on as well as all the other attributes.
Teachers are recommended to emphasize the trade-off between privacy-preservation and model quality/data accuracy in general. If necessary, extra exercises can be built into the syllabus to demonstrate this (evaluate model quality depending on epsilon and delta).
Outline
In this lab session, you will mitigate privacy risks fin AI models. Specifically, students will develop two mitigation techniques:
- Defense 1: generate synthetic data with the guarantees of Differential Privacy and check
- how much the model quality degrades if the privacy-preserving synthetic data is used to train the model instead of the original data (depending on the privacy parameter epsilon)
- if training on the synthetic data instead of the original one prevents membership and attribute inference attack
- Defense 2: train the model with Differential Privacy guarantees, and check
- how much the model quality degrades if the privacy-preserving model is used to instead of the original model for prediction (depending on the privacy parameter epsilon)
- if the privacy-preserving model prevents membership attack
- how the accuracy of the privacy-preserving model changes compared to Defense 1
Students will form groups of two and work as a team. One group has to hand in only one documentation/solution.
