Administrative information
| Title |
Recurrent Neural Networks |
 |
| Duration |
60 min |
| Module |
B |
| Lesson Type |
Lecture |
| Focus |
Technical - Deep Learning |
| Topic |
Multiple (Recurrent Neural Networks (RNN), Backpropagation Through Time (BPTT), Long Short-Term Memory (LSTM))
|
Keywords
Recurrent Neural Networks (RNN), Backpropagation Trough Time (BPTT), Long Short-Term Memory (LSTM),
Learning Goals
- Learning the fundamentals of Recurrent Neural Networks (RNN), Backpropagation Trough Time (BPTT) and Long Short-Term Memory (LSTM)
Expected Preparation
Learning Events to be Completed Before
Optional for Students
None.
References and background for students:
None.
Recommended for Teachers
None.
Lesson Materials
The materials of this learning event are available under CC BY-NC-SA 4.0.
Instructions for Teachers
In the beginning a general overview of sequential data is recommended. Here, you can discuss the main challanges of modeling sequential data (including receptive field, multidimensionality and multiscale nature).
Later we introduce RNNs first, with the basic principles and show that training RNNs after unfolding is very simlar to training MLPs, but the input, recurrent and output weight matrices are shared.
We introduce BPTT (Backpropagation through time) and the truncated version.
Next, we discuss how the vanishing gradient problem makes RNNs impractical.
In order to solve the vanishing gradient, we introduce the LSTM architecture, which has an inner memory part (also reffered to as memory cell), which does not have any activatoin - so vanishing gradient does not occure here. Please make it really clear, that the gating mechanism is truly controlled by the data.
At the final part of the lecture, we show that LSTM (and RNN, indeed) layers can be stacked onto each other with one or two directions. (uni- and bidirectional networks)
Outline
- Overview of sequential data
- Recurrent neural networks basics
- Backpropagation through time
- Vanishing gradient
- Long Short-Term Memory
- Stacking RNN/LSTM layers
Time schedule
| Duration (Min) |
Description |
| 10 |
Sequential data introduction |
| 15 |
Recurrent neural networks and Backpropagation through time |
| 5 |
Vanishing gradients in RNNs |
| 20 |
LSTMs |
| 5 |
Stacking RNN/LSTM layers |
| 5 |
Conclusions |
More information
Click here for an overview of all lesson plans of the master human centred AI
Please visit the home page of the consortium HCAIM
Acknowledgements
 |
The Human-Centered AI Masters programme was co-financed by the Connecting Europe Facility of the European Union Under Grant №CEF-TC-2020-1 Digital Skills 2020-EU-IA-0068.
The materials of this learning event are available under CC BY-NC-SA 4.0
|
|
The HCAIM consortium consists of three excellence centres, three SMEs and four Universities
|
