Administrative information

Title	Model Evaluation
Duration	60
Module	A
Lesson Type	Tutorial
Focus	Technical - Foundations of AI
Topic	Foundations of AI

Keywords

model evaluation, cross-validation, hyperparameter optimization,

Learning Goals

Learners understand the need for systematic model evaluation
Learners understand the difference between training, test and validation sets
Learners know the most widely applied performance metrics
Learners are able to recognize underfitting and overfitting
Learners are capable of designing cross-validation experiments for hyperparameter optimization

Expected Preparation

Learning Events to be Completed Before

Obligatory for Students

None.

Optional for Students

None.

References and background for students:

None.

Recommended for Teachers

None.

Lesson Materials

Notebook

The materials of this learning event are available under CC BY-NC-SA 4.0.

Instructions for Teachers

Prepare a Jupyter notebook environment with pandas, matplotlib, numpy and scikit-learn packages

Outline/time schedule

Duration (min)	Description	Concepts
5	Introduction to model evaluation	empirical error, predictive and generalization performance
5	Training a simple classifier	MLP, hyperparameters
10	Evaluating a classifier	confusion matrix, accuracy, TPR, FPR, precision, misclassification rate, F1 score
10	ROC/PR curves and their interpretation	decision boundary, ROC curve, PR curve, AUC
10	Underfitting and overfitting	training and test error
10	Cross-validation and hyperparameter optimization	validation set, validation error, 5-fold cross-validation
10	Evaluation of regression models	MSE, RMSE, MAE

More information

Click here for an overview of all lesson plans of the master human centred AI

Please visit the home page of the consortium HCAIM

Acknowledgements

The Human-Centered AI Masters programme was co-financed by the Connecting Europe Facility of the European Union Under Grant №CEF-TC-2020-1 Digital Skills 2020-EU-IA-0068.

The materials of this learning event are available under CC BY-NC-SA 4.0

The HCAIM consortium consists of three excellence centres, three SMEs and four Universities