Topics in machine learning

Topics in Mathematics of ML

This class discusses topics in mathematical theory of machine learning, from traditional theory (kernel methods, generalization bounds) to modern deep learning (infinite neural networks, training dynamics).

• Teacher Lénaïc Chizat, main assistant Guillaume Wang
• Here is a tentative program : 

1. Introduction (reminders on supervised ML)
2. Ordinary Least Squares
3. Empirical Risk Minimization and Rademacher complexity
4. Kernel ridge regression (I) 
5. Kernel ridge regression (II)
6. Neural Networks (I): standard approximation results
7. Neural Networks (II): large width approximation results 
8. Gradient Descent (GD), Stochastic GD, Gradient flows 
9. Implicit bias of gradient descent: linear models, losses with an exponential tail
10. Implicit bias of gradient descent, reparameterizations and mirror descent  
11. Dynamics of Large Width Shallow Neural Networks 
12. Initialization of Deep Neural Networks
13. Dynamics of Large Depth ResNets

• Exercice sessions will involve numerical experiments, bring your laptop!
• Lecture notes will in general be uploaded before each lecture. However they do not cover all the material and explanation provided in class.
• Validation: 100% final written exam
• References: F. Bach "Learning Theory from First Principles"