This laboratory exercise aims to develop the practical skills of students of auditing the privacy guarantees of Machine Learning models. Students should understand that membership attacks suppose the knowledge of the target sample to be tested which is not always feasible. Still, the success of membership inference can anticipate more serious privacy leakages in the future.
Machine learning models are often trained on confidential (or personal, sensitive) data. For example, such a model can predict the salary of an individual from its other attributes (such as education, living place, race, sex, etc.). A common misconception is that such models are not regarded as personal data even if their training data is personal (indeed, training data can be the collection of records about individuals), as they are computed from aggregated information derived from the sensitive training data (e.g., average of gradients in neural networks, or entropy/count of labels in random forests). The goal of this lab session is to show that machine learning models can be regarded as personal data and therefore its processing is very likely to be regulated in many countries (e.g., by GDPR in Europe). Students will design privacy attacks to test if the trained models leak information about its training data, and also mitigate these attacks. For example, membership inference attacks aim to detect the presence of a given sample in the training data of a target model from the models and/or its output. White-box attacks can access both the trained models (including its parameters) and the output of the model (i.e., its predictions), whereas black-box models can only access the predictions of the model for a given sample. Attribute inference attacks aim to predict a missing sensitive attribute from the output of the machine learning model that is trained on as well as all the other attributes.
A follow-up learning event is about mitigating these threats: Practical: Applying and evaluating privacy-preserving techniques
Outline
In this lab session, you will measure privacy risks for AI models and also mitigate the attacks. Specifically, students will
- train a machine learning model (Random Forest) on the Adult dataset to predict the binary income attribute in the dataset
- measure privacy risks by launching a membership attack on the trained model to check if the presence of any individual in the training data can be detected only from the prediction of the model (black-box attack)
- launch attribute inference attack on the trained model to check if the missing (sensitive) attribute can be inferred from some auxiliary data resembling the original data and the output of the trained model (black-box attack)
Students will form groups of two and work as a team. One group has to hand in only one documentation/solution.
