Administrative information

Title	Model Evaluation
Duration	60
Module	A
Lesson Type	Lecture
Focus	Technical - Foundations of AI
Topic	Foundations of AI

Keywords

underfitting, overfitting, generalization, bias-variance decomposition, model complexity, ROC curve,

Learning Goals

Learners understand the concept of expected predictive performance and generalization
Learners understand the concept of model complexity and its relationship with generalization performance
Learners are familiar with bias and variance their relationship with underfitting and overfitting
Learners have a firm grasp on evaluating binary classification models with the most widely applied metrics

Expected Preparation

Learning Events to be Completed Before

Obligatory for Students

None.

Optional for Students

Evaluation metrics implemented in scikit-learn

References and background for students:

None.

Recommended for Teachers

Familiarize themselves with the demonstration materials.

Lesson Materials

The materials of this learning event are available under CC BY-NC-SA 4.0.

Instructions for Teachers

Cover the topics in the lesson outline and demonstrate the concepts using the interactive notebooks (effect of the hyperparameters on under/overfitting and the bias/variance curves; calcluation of the ROC/PR curves).

Outline/time schedule

Duration (min)	Description	Concepts
5	Non-linear regression (recap)	basis points, RBF, kernel parameters, mean squared error
10	Model complexity, regulariztaion and the number of parameters	complexity, regularization
10	Bias-variance decomposition of mean squared error	generalization error, bias, variance, observation noise, underfitting, overfitting
10	Demonstration of the effects of the complexity parameter, regularization coefficient and the number of basis points on curve fitting and bias/variance
15	Evaluation of classification models	confusion matrix, TPR, FPR, precision, decision boundary, ROC/PR curve
10	Demonstration of decision boundaries and the ROC curve

More information

Click here for an overview of all lesson plans of the master human centred AI

Please visit the home page of the consortium HCAIM

Acknowledgements

The Human-Centered AI Masters programme was co-financed by the Connecting Europe Facility of the European Union Under Grant №CEF-TC-2020-1 Digital Skills 2020-EU-IA-0068.

The materials of this learning event are available under CC BY-NC-SA 4.0

The HCAIM consortium consists of three excellence centres, three SMEs and four Universities