Administrative information

Learning Events to be Completed Before

Lecture: Hardware and software frameworks for deep learning

Obligatory for Students

Knowledge of the supervised learning theory
Introduction to machine learning and deep learning concepts given by previous lectures

Optional for Students

Knowledge of the most common hyper parameters involved in neural networks building process

References and background for students

Knowledge distillation - Easy
Song Han, et al. "Learning both Weights and Connections for Efficient Neural Networks". CoRR abs/1506.02626. (2015).
Song Han, et al. "Deep Compression: Compressing Deep Neural Network with Pruning, Trained Quantization and Huffman Coding." 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings. 2016. Yanzhi Wang, et al. "Non-Structured DNN Weight Pruning -- Is It Beneficial in Any Platform?". CoRR abs/1907.02124. (2019).
Cheong and Daniel. "transformers.zip: Compressing Transformers with Pruning and Quantization"
Song Han, et al. "Learning both Weights and Connections for Efficient Neural Networks". CoRR abs/1506.02626. (2015).
Davis W. Blalock, et al. "What is the State of Neural Network Pruning?." Proceedings of Machine Learning and Systems 2020, MLSys 2020, Austin, TX, USA, March 2-4, 2020. mlsys.org, 2020.
https://github.com/kingreza/quantization
Song Han, et al. "Deep Compression: Compressing Deep Neural Network with Pruning, Trained Quantization and Huffman Coding." 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings. 2016.
Zhi Gang Liu, et al. "Learning Low-precision Neural Networks without Straight-Through Estimator (STE)." Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, ĲCAI 2019, Macao, China, August 10-16, 2019. ijcai.org, 2019.
Peiqi Wang, et al. "HitNet: Hybrid Ternary Recurrent Neural Network." Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada. 2018.
Cristian Bucila, et al. "Model compression." Proceedings of the Twelfth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Philadelphia, PA, USA, August 20-23, 2006. ACM, 2006.
Geoffrey E. Hinton, et al. "Distilling the Knowledge in a Neural Network". CoRR abs/1503.02531. (2015).
https://towardsdatascience.com/knowledge-distillation-simplified-dd4973dbc764
https://www.ttic.edu/dl/dark14.pdf
https://josehoras.github.io/knowledge-distillation/

Recommended for Teachers

Survey on Model Compression

The lecture can refer to model types, model evaluation, model fitting and model optimization

Title	Model Compression - Edge Computing
Duration	45 mins
Module	C
Lesson Type	Lecture
Focus	Technical - Future AI
Topic	Advances in ML models through a HC lens - A result Oriented Study

Duration	Description	Concepts	Activity
0-10 min	Introduction to techniques for model compression: what it is, what it is for, when and why it is needed	Model compression	Introduction to main concepts
10-20 min	Pruning: concepts and techniques. Main approaches to pruning	Pruning	Taught session and examples
20-30 min	Quantization: concepts and techniques. Main approaches to quantization	Quantization	Taught session and examples
30-40 min	Knowledge distillation: concepts and techniques. Main approaches to knowledge distillation	Knowledge distillation	Taught session and examples
40-45 min	Conclusion, questions and answers	Summary	Conclusions

Administrative information

Keywords

Learning Goals

Expected Preparation

Learning Events to be Completed Before

Obligatory for Students

Optional for Students

References and background for students

Recommended for Teachers

Lesson materials

Instructions for Teachers

Outline

More information

Acknowledgements