Foundations of Data Science
COM-406
Media
COM-406 Foundations of data science
26.2, Lecture 26.2 - Generalization Error
22.12.2021, 19:33
26.1, Lecture 26.1 - Generalization Error
22.12.2021, 12:46
25.1, Lecture 25.1 - Exploration Bias
17.12.2021, 10:32
25.2, Lecture 25.2 - Exploration Bias
17.12.2021, 10:19
24.2, Lecture 24.2 - Compression
16.12.2021, 18:07
24.1, Lecture 24.1 - Compression
16.12.2021, 17:28
23.2, Lecture 23.2 - Johnson-Lindenstrauss
10.12.2021, 13:29
23.1, Lecture 23.1 - PCA
10.12.2021, 13:19
22.3, Lecture 22.3 - Compression
08.12.2021, 16:58
22.2, Lecture 22.2 - Compression
08.12.2021, 16:50
22.1, Lecture 22.1 - Compression
08.12.2021, 16:46
21.3, Lecture 21.3 - Compression
03.12.2021, 15:00
21.2, Lecture 21.2 - Compression
03.12.2021, 14:33
21.1, Lecture 21.1 - Exponential Family
03.12.2021, 14:22
20.2, Lecture 20.2 - Exponential Family
01.12.2021, 12:21
20.1, Lecture 20.1 - Exponential Family
01.12.2021, 12:17
19.2, Lecture 19.2 - Exponential Family
27.11.2021, 22:09
19.1, Lecture 19.1 - Exponential Family
26.11.2021, 19:13
18.2, Lecture 18.2 - Detection & Estimation
24.11.2021, 22:57
18.1, Lecture 18.1 - Detection-Estimation
24.11.2021, 22:37
17.2, Lecture 17.2 - Detection & Estimation
21.11.2021, 18:08
17.1, Lecture 17.1 - Detection & Estimation
21.11.2021, 13:30
16.2, Lecture 16.2 - Detection & Estimation
13.11.2021, 20:01
16.1, Lecture 16.1 - Detection & Estimation
13.11.2021, 19:05
15, Lecture 15 - Property Testing
10.11.2021, 13:27
14.2, Lecture 14.2 - Distribution Estimation
08.11.2021, 10:24
14.1, Lecture 14.1 - Distribution Estimation
08.11.2021, 09:42
13.2, Lecture 13.2 - Distribution Estimation
03.11.2021, 18:56
13.1, Lecture 13.1 - Distribution Estimation
03.11.2021, 18:41
12.2, Lecture 12.2 - Distribution Estimation
30.10.2021, 12:05
12.1, Lecture 12.1 - Multi-arm Bandits
30.10.2021, 11:43
11.2, Lecture 11.2 - Multi-arm Bandits
27.10.2021, 14:47
11.1, Lecture 11.1 - Multi-arm Bandits
27.10.2021, 13:36
10.1, Lecture 10.1 - Multi-arm Bandits
25.10.2021, 13:18
10.2, Lecture 10.2 - Multi-arm Bandits
22.10.2021, 10:17
9.2, Lecture 9.2 - Multi-arm Bandits
21.10.2021, 14:16
9.1, Lecture 9.1 - Multi-arm Bandits
21.10.2021, 13:47
8.1, Lecture 8.1 - signal representations
12.10.2020, 12:23
8.2, Lecture 8.2 - Signal Representations
15.10.2021, 10:07
7.2, Lecture 7.2 - signal representations
09.10.2020, 11:19
7.1, Lecture 7.1 - Signal Representations
09.10.2020, 11:19
6.2, Lecture 6.2 - signal representations
05.10.2020, 11:49
6.1, Lecture 6.1 - signal representations
05.10.2020, 11:59
5.2, Lecture 5.2 - Signal Representations
07.10.2021, 09:48
5.1, Lecture 5.1 - Signal Representations
07.10.2021, 09:24
4.2, Lecture 4.2 - Linear Algebra Review
01.10.2021, 10:36
4.1, Lecture 4.1 - Information Measures
01.10.2021, 10:20
3.2, Lecture 3.2 information measures
25.09.2020, 10:46
3.1, Lecture 3.1 - information measures
25.09.2020, 10:56
2.2, Lecture 2.2 - Information Measures
24.09.2021, 21:26
2.1, Lecture 2.1 - Probability Review
24.09.2021, 21:11
1.2, Lecture 1.2 - Probability Review
22.09.2021, 12:54
1.1, Lecture 1.1 - Introduction
18.09.2020, 18:22
26, Lecture 26 - Review Session
18.12.2020, 11:06
25, Lecture 25 - generalization bound
14.12.2020, 10:52
24, Lecture 24 - exploration bias and generalization bound
11.12.2020, 12:12
23, Lecture 23 - compression
07.12.2020, 11:08
22, Lecture 22 - compression
04.12.2020, 10:46
21, Lecture 21 - compression
30.11.2020, 11:15
20, Lecture 20 - exponential family
27.11.2020, 12:19
19, Lecture 19 - exponential family
23.11.2020, 11:32
18, Lecture 18 - exponential family
20.11.2020, 12:10
17, Lecture 17 - Property Estimation and Exponential Family
16.11.2020, 12:43
16, Lecture 16 - property testing
13.11.2020, 12:31
15, Lecture 15 - distribution estimation
09.11.2020, 11:14
14, Lecture 14 - distribution estimation
06.11.2020, 12:14
13, Lecture 13 - multi-arm bandits, distribution estimation
02.11.2020, 11:07
12, Lecture 12 - multi-arm bandits
30.10.2020, 12:03
11, Lecture 11 - multi-arm bandits
26.10.2020, 11:41
10.1, Lecture 10.1 - detection
19.10.2020, 11:36
10.2, Lecture 10.2 - detection
19.10.2020, 11:36
9.1, Lecture 9.1 - estimation
16.10.2020, 10:46
9.2, Lecture 9.2 - estimation
16.10.2020, 10:45
8.2, Lecture 8.2 - estimation
12.10.2020, 12:20
5.1, Lecture 5.1 - signal representation
02.10.2020, 10:18
5.2, Lecture 5.2 - signal representation
02.10.2020, 10:18
4.2, Lecture 4.2 - signal representation
28.09.2020, 11:14
4.1, Lecture 4.1 - signal representation
28.09.2020, 10:08
2.2, Lecture 2.2 - information measures: part 2/2
18.09.2020, 18:37
2.1, Lecture 2.1 - information measures: part 1/2
18.09.2020, 18:30
1.2, Lecture 1.2 - Review of probability
18.09.2020, 18:24
Media
COM-406 Foundations of data science
26.2, Lecture 26.2 - Generalization Error
22.12.2021, 19:33
26.1, Lecture 26.1 - Generalization Error
22.12.2021, 12:46
25.1, Lecture 25.1 - Exploration Bias
17.12.2021, 10:32
25.2, Lecture 25.2 - Exploration Bias
17.12.2021, 10:19
24.2, Lecture 24.2 - Compression
16.12.2021, 18:07
24.1, Lecture 24.1 - Compression
16.12.2021, 17:28
23.2, Lecture 23.2 - Johnson-Lindenstrauss
10.12.2021, 13:29
23.1, Lecture 23.1 - PCA
10.12.2021, 13:19
22.3, Lecture 22.3 - Compression
08.12.2021, 16:58
22.2, Lecture 22.2 - Compression
08.12.2021, 16:50
22.1, Lecture 22.1 - Compression
08.12.2021, 16:46
21.3, Lecture 21.3 - Compression
03.12.2021, 15:00
21.2, Lecture 21.2 - Compression
03.12.2021, 14:33
21.1, Lecture 21.1 - Exponential Family
03.12.2021, 14:22
20.2, Lecture 20.2 - Exponential Family
01.12.2021, 12:21
20.1, Lecture 20.1 - Exponential Family
01.12.2021, 12:17
19.2, Lecture 19.2 - Exponential Family
27.11.2021, 22:09
19.1, Lecture 19.1 - Exponential Family
26.11.2021, 19:13
18.2, Lecture 18.2 - Detection & Estimation
24.11.2021, 22:57
18.1, Lecture 18.1 - Detection-Estimation
24.11.2021, 22:37
17.2, Lecture 17.2 - Detection & Estimation
21.11.2021, 18:08
17.1, Lecture 17.1 - Detection & Estimation
21.11.2021, 13:30
16.2, Lecture 16.2 - Detection & Estimation
13.11.2021, 20:01
16.1, Lecture 16.1 - Detection & Estimation
13.11.2021, 19:05
15, Lecture 15 - Property Testing
10.11.2021, 13:27
14.2, Lecture 14.2 - Distribution Estimation
08.11.2021, 10:24
14.1, Lecture 14.1 - Distribution Estimation
08.11.2021, 09:42
13.2, Lecture 13.2 - Distribution Estimation
03.11.2021, 18:56
13.1, Lecture 13.1 - Distribution Estimation
03.11.2021, 18:41
12.2, Lecture 12.2 - Distribution Estimation
30.10.2021, 12:05
12.1, Lecture 12.1 - Multi-arm Bandits
30.10.2021, 11:43
11.2, Lecture 11.2 - Multi-arm Bandits
27.10.2021, 14:47
11.1, Lecture 11.1 - Multi-arm Bandits
27.10.2021, 13:36
10.1, Lecture 10.1 - Multi-arm Bandits
25.10.2021, 13:18
10.2, Lecture 10.2 - Multi-arm Bandits
22.10.2021, 10:17
9.2, Lecture 9.2 - Multi-arm Bandits
21.10.2021, 14:16
9.1, Lecture 9.1 - Multi-arm Bandits
21.10.2021, 13:47
8.1, Lecture 8.1 - signal representations
12.10.2020, 12:23
8.2, Lecture 8.2 - Signal Representations
15.10.2021, 10:07
7.2, Lecture 7.2 - signal representations
09.10.2020, 11:19
7.1, Lecture 7.1 - Signal Representations
09.10.2020, 11:19
6.2, Lecture 6.2 - signal representations
05.10.2020, 11:49
6.1, Lecture 6.1 - signal representations
05.10.2020, 11:59
5.2, Lecture 5.2 - Signal Representations
07.10.2021, 09:48
5.1, Lecture 5.1 - Signal Representations
07.10.2021, 09:24
4.2, Lecture 4.2 - Linear Algebra Review
01.10.2021, 10:36
4.1, Lecture 4.1 - Information Measures
01.10.2021, 10:20
3.2, Lecture 3.2 information measures
25.09.2020, 10:46
3.1, Lecture 3.1 - information measures
25.09.2020, 10:56
2.2, Lecture 2.2 - Information Measures
24.09.2021, 21:26
2.1, Lecture 2.1 - Probability Review
24.09.2021, 21:11
1.2, Lecture 1.2 - Probability Review
22.09.2021, 12:54
1.1, Lecture 1.1 - Introduction
18.09.2020, 18:22
26, Lecture 26 - Review Session
18.12.2020, 11:06
25, Lecture 25 - generalization bound
14.12.2020, 10:52
24, Lecture 24 - exploration bias and generalization bound
11.12.2020, 12:12
23, Lecture 23 - compression
07.12.2020, 11:08
22, Lecture 22 - compression
04.12.2020, 10:46
21, Lecture 21 - compression
30.11.2020, 11:15
20, Lecture 20 - exponential family
27.11.2020, 12:19
19, Lecture 19 - exponential family
23.11.2020, 11:32
18, Lecture 18 - exponential family
20.11.2020, 12:10
17, Lecture 17 - Property Estimation and Exponential Family
16.11.2020, 12:43
16, Lecture 16 - property testing
13.11.2020, 12:31
15, Lecture 15 - distribution estimation
09.11.2020, 11:14
14, Lecture 14 - distribution estimation
06.11.2020, 12:14
13, Lecture 13 - multi-arm bandits, distribution estimation
02.11.2020, 11:07
12, Lecture 12 - multi-arm bandits
30.10.2020, 12:03
11, Lecture 11 - multi-arm bandits
26.10.2020, 11:41
10.1, Lecture 10.1 - detection
19.10.2020, 11:36
10.2, Lecture 10.2 - detection
19.10.2020, 11:36
9.1, Lecture 9.1 - estimation
16.10.2020, 10:46
9.2, Lecture 9.2 - estimation
16.10.2020, 10:45
8.2, Lecture 8.2 - estimation
12.10.2020, 12:20
5.1, Lecture 5.1 - signal representation
02.10.2020, 10:18
5.2, Lecture 5.2 - signal representation
02.10.2020, 10:18
4.2, Lecture 4.2 - signal representation
28.09.2020, 11:14
4.1, Lecture 4.1 - signal representation
28.09.2020, 10:08
2.2, Lecture 2.2 - information measures: part 2/2
18.09.2020, 18:37
2.1, Lecture 2.1 - information measures: part 1/2
18.09.2020, 18:30
1.2, Lecture 1.2 - Review of probability
18.09.2020, 18:24
Summary
We discuss a set of topics that are important for the understanding of modern data science but that are typically not taught in an introductory ML course. In particular we discuss fundamental ideas and techniques that come from probability, information theory as well as signal processing.
Content
This class presents basic concepts of Information Theory and Signal Processing and their relevance to emerging problems in Data Science and Machine Learning.
A tentative list of topics covered is:
- Information Measures
- Multi-arm Bandits
- Detection and Estimation
- Distribution Estimation, Property Testing, and Property Estimation
- Exponential Families
- Signal Representations
- Compression and Dimensionality Reduction
- Information Measures and Generalization Error
Materials
- Lecture Notes (Version Sept 5). Note: Check for updates on a semi-regular basis.
- T. M. Cover and J. A. Thomas, Elements of Information Theory (Click to get access to the full PDF via the EPFL library). New York: Wiley. Second Edition, 2006.
- T. Lattimore and C. Szepesvari, Bandit Algorithms
Schedule
Classes:
- Tuesday 11:15-13:00 (CE 1 104)
- Thursday 17:15-19:00 (INF 1)
- Tuesday 13:15-15:00 (CE 1 104)
ED Discussion Forum
- We will use the ED Discussion Forum for this class. Everyone is strongly encouraged to make the most of this!
- Ask questions!
- Answer questions!
- The class staff will check the forum on Monday afternoon and on Thursday afternoon.
SWITCHtube Channel
We will not make new video recordings this year. You can access the videos from a couple of years ago. The content is largely the same, but the order of the topics is slightly different.
Grading
- If you do not hand in your final exam your overall grade will be NA.
- Otherwise, your grade will be determined based on the following weighted average: 10% for the Homework (specifically, 9% for the graded Homework sets and 1% for your activity on the ED forum), 30% for the Midterm Exam, 60% for the Final Exam.
- The Midterm Exam will take place on Thursday, November 14, 2024, 17:15-19:00
The Final Exam will take place on at some point between January 13, 2025 and February 1, 2025.
September 9 - September 15
Sept 10: Introduction and Probability Review
Sept 12: Information Measures
September 16 - September 22
Sept 17: Information Measures
Sept 19: Information Measures
September 23 - September 29
Sept 26 : Multi-Arm Bandits
September 30 - October 6
Oct 3 : Multi-Arm Bandits
October 7 - October 13
Oct 10 : Detection & Estimation
October 14 - October 20
Oct 15: Detection and Estimation
Oct 17: Parameter estimation, Fisher Information, Cramer-Rao Lower Bound
October 21 - October 27
Fall Break - no class, no exercise session
October 28 - November 3
Oct 29: Distribution Estimation
Oct 31: Distribution Estimation
November 4 - November 10
Nov 5: Distribution Estimation
Nov 7: Property Testing
- Midterm 2021 (File)
- Midterm 2022 (File)
- Midterm 2021 Solution (File)
- Midterm 2022 Solution (File)
- Final Exam 2018 (File)
- Final Exam 2019 (File)
- Final Exam 2020 (File)
- Final Exam 2021 (File)
- Final Exam 2022 (File)
- Final Exam 2018 Solution (File)
- Final Exam 2019 Solution (File)
- Final Exam 2020 Solution (File)
- Final Exam 2021 Solution (File)
- Final Exam 2022 Solution (File)
- Homework 5 (File)
- Homework 5 Solutions (File)
November 11 - November 17
Nov 14 : Midterm Exam
November 18 - November 24
Nov 19: Exponential Families
Nov 21: Exponential Families
November 25 - December 1
Nov 26 Signal Representations: Linear Algebra Review++ (chapter 3)
Nov 28 Signal Representations: Fourier & Hilbert
December 2 - December 8
Dec 3: Signal Representations: Time-Frequency perspective
Dec 5: Compression: Dimensionality Reduction (PCA and Random Projections)
December 9 - December 15
December 10: Compression: Dimensionality Reduction (Random Projections), then classic data compression
December 12: Compression: Classic data compression
December 16 - December 22
December 17: Information-theoretic perspective on Generalizaton of Learning Algorithms
December 19: Overview of the class, followed by Outlook.