Administrative information

This lecture will introduce students to the fundamentals of the backpropagation algorithm. This lecture will start with the notion of the curse of dimensionality leading to the need of a heuristic approach - followed by the overview of how gradient can be used to adjust the weights. This then introduces the backpropagation algorithm. We then also introduce the hyperparameter of learning rate and a brief over view of the affect of large and small values (this will be expanded in Lecture 3). Then using the same introductory network from Lecture 1, we derive the outer layer backpropagation formula, and then finally, we will derive the inner layer backpropagation algorithm. This lecture concludes with examples of different activation functions, and how the algorithm can be applied. The corresponding tutorial will include additional pen and paper derivations, practical examples and the use of code (just Numpy and the KERAS) to implement the backpropagation algorithm.

• The initial concept of brute force weight selection, and the curse of dimensionality
• Introduction to gradient and how this address the problem of iterative, heuristic weight adjustments
• Why learning rate is needed and the affects of choosing small and large values
• Deriving the gradient (thus the backpropagation algorithm) for the output layer with Sigmoid as the outer activation function
• Deriving the gradient (thus the backpropagation algorithm) for the hiden layer with Sigmoid as the outer activation function
• Presenting the final backpropagation formula
• Using different activation functions (Outer Layer: Linear, Sigmoid and Softmax; Hidden layer: ReLu, SIgmoid and TanH) in the backpropagation algorithm

Outline