Foundations of Data Science
COM-406
Media
COM-406 Foundations of data science
26.2, Lecture 26.2 - Generalization Error
22.12.2021, 19:33
26.1, Lecture 26.1 - Generalization Error
22.12.2021, 12:46
25.1, Lecture 25.1 - Exploration Bias
17.12.2021, 10:32
25.2, Lecture 25.2 - Exploration Bias
17.12.2021, 10:19
24.2, Lecture 24.2 - Compression
16.12.2021, 18:07
24.1, Lecture 24.1 - Compression
16.12.2021, 17:28
23.2, Lecture 23.2 - Johnson-Lindenstrauss
10.12.2021, 13:29
23.1, Lecture 23.1 - PCA
10.12.2021, 13:19
22.3, Lecture 22.3 - Compression
08.12.2021, 16:58
22.2, Lecture 22.2 - Compression
08.12.2021, 16:50
22.1, Lecture 22.1 - Compression
08.12.2021, 16:46
21.3, Lecture 21.3 - Compression
03.12.2021, 15:00
21.2, Lecture 21.2 - Compression
03.12.2021, 14:33
21.1, Lecture 21.1 - Exponential Family
03.12.2021, 14:22
20.2, Lecture 20.2 - Exponential Family
01.12.2021, 12:21
20.1, Lecture 20.1 - Exponential Family
01.12.2021, 12:17
19.2, Lecture 19.2 - Exponential Family
27.11.2021, 22:09
19.1, Lecture 19.1 - Exponential Family
26.11.2021, 19:13
18.2, Lecture 18.2 - Detection & Estimation
24.11.2021, 22:57
18.1, Lecture 18.1 - Detection-Estimation
24.11.2021, 22:37
17.2, Lecture 17.2 - Detection & Estimation
21.11.2021, 18:08
17.1, Lecture 17.1 - Detection & Estimation
21.11.2021, 13:30
16.2, Lecture 16.2 - Detection & Estimation
13.11.2021, 20:01
16.1, Lecture 16.1 - Detection & Estimation
13.11.2021, 19:05
15, Lecture 15 - Property Testing
10.11.2021, 13:27
14.2, Lecture 14.2 - Distribution Estimation
08.11.2021, 10:24
14.1, Lecture 14.1 - Distribution Estimation
08.11.2021, 09:42
13.2, Lecture 13.2 - Distribution Estimation
03.11.2021, 18:56
13.1, Lecture 13.1 - Distribution Estimation
03.11.2021, 18:41
12.2, Lecture 12.2 - Distribution Estimation
30.10.2021, 12:05
12.1, Lecture 12.1 - Multi-arm Bandits
30.10.2021, 11:43
11.2, Lecture 11.2 - Multi-arm Bandits
27.10.2021, 14:47
11.1, Lecture 11.1 - Multi-arm Bandits
27.10.2021, 13:36
10.1, Lecture 10.1 - Multi-arm Bandits
25.10.2021, 13:18
10.2, Lecture 10.2 - Multi-arm Bandits
22.10.2021, 10:17
9.2, Lecture 9.2 - Multi-arm Bandits
21.10.2021, 14:16
9.1, Lecture 9.1 - Multi-arm Bandits
21.10.2021, 13:47
8.1, Lecture 8.1 - signal representations
12.10.2020, 12:23
8.2, Lecture 8.2 - Signal Representations
15.10.2021, 10:07
7.2, Lecture 7.2 - signal representations
09.10.2020, 11:19
7.1, Lecture 7.1 - Signal Representations
09.10.2020, 11:19
6.2, Lecture 6.2 - signal representations
05.10.2020, 11:49
6.1, Lecture 6.1 - signal representations
05.10.2020, 11:59
5.2, Lecture 5.2 - Signal Representations
07.10.2021, 09:48
5.1, Lecture 5.1 - Signal Representations
07.10.2021, 09:24
4.2, Lecture 4.2 - Linear Algebra Review
01.10.2021, 10:36
4.1, Lecture 4.1 - Information Measures
01.10.2021, 10:20
3.2, Lecture 3.2 information measures
25.09.2020, 10:46
3.1, Lecture 3.1 - information measures
25.09.2020, 10:56
2.2, Lecture 2.2 - Information Measures
24.09.2021, 21:26
2.1, Lecture 2.1 - Probability Review
24.09.2021, 21:11
1.2, Lecture 1.2 - Probability Review
22.09.2021, 12:54
1.1, Lecture 1.1 - Introduction
18.09.2020, 18:22
26, Lecture 26 - Review Session
18.12.2020, 11:06
25, Lecture 25 - generalization bound
14.12.2020, 10:52
24, Lecture 24 - exploration bias and generalization bound
11.12.2020, 12:12
23, Lecture 23 - compression
07.12.2020, 11:08
22, Lecture 22 - compression
04.12.2020, 10:46
21, Lecture 21 - compression
30.11.2020, 11:15
20, Lecture 20 - exponential family
27.11.2020, 12:19
19, Lecture 19 - exponential family
23.11.2020, 11:32
18, Lecture 18 - exponential family
20.11.2020, 12:10
17, Lecture 17 - Property Estimation and Exponential Family
16.11.2020, 12:43
16, Lecture 16 - property testing
13.11.2020, 12:31
15, Lecture 15 - distribution estimation
09.11.2020, 11:14
14, Lecture 14 - distribution estimation
06.11.2020, 12:14
13, Lecture 13 - multi-arm bandits, distribution estimation
02.11.2020, 11:07
12, Lecture 12 - multi-arm bandits
30.10.2020, 12:03
11, Lecture 11 - multi-arm bandits
26.10.2020, 11:41
10.1, Lecture 10.1 - detection
19.10.2020, 11:36
10.2, Lecture 10.2 - detection
19.10.2020, 11:36
9.1, Lecture 9.1 - estimation
16.10.2020, 10:46
9.2, Lecture 9.2 - estimation
16.10.2020, 10:45
8.2, Lecture 8.2 - estimation
12.10.2020, 12:20
5.1, Lecture 5.1 - signal representation
02.10.2020, 10:18
5.2, Lecture 5.2 - signal representation
02.10.2020, 10:18
4.2, Lecture 4.2 - signal representation
28.09.2020, 11:14
4.1, Lecture 4.1 - signal representation
28.09.2020, 10:08
2.2, Lecture 2.2 - information measures: part 2/2
18.09.2020, 18:37
2.1, Lecture 2.1 - information measures: part 1/2
18.09.2020, 18:30
1.2, Lecture 1.2 - Review of probability
18.09.2020, 18:24
Summary
We discuss a set of topics that are important for the understanding of modern data science but that are typically not taught in an introductory ML course. In particular we discuss fundamental ideas and techniques that come from probability, information theory as well as signal processing.
Content
This class presents basic concepts of Information Theory and Signal Processing and their relevance to emerging problems in Data Science and Machine Learning.
A tentative list of topics covered is:
- Information Measures
- Multi-arm Bandits
- Detection and Estimation
- Distribution Estimation, Property Testing, and Property Estimation
- Exponential Families
- Signal Representations
- Compression and Dimensionality Reduction
- Information Measures and Generalization Error
Materials
- Lecture Notes (Slightly updated Sept 24: Added a few more details in the proofs of the "Mutual Information" section (Section 4.5))
- T. M. Cover and J. A. Thomas, Elements of Information Theory (Click to get access to the full PDF via the EPFL library). New York: Wiley. Second Edition, 2006.
- T. Lattimore and C. Szepesvari, Bandit Algorithms
Schedule
Note: Our Schedule deviates slightly from what is shown on IS-Academia.- Tuesdays:
- 11:15-12:30, BC 01 (Lecture)
- 12:30-13:15, Lunch Break
- 13:15-14:30, BC 01 (Lecture)
- 14:30-15:00, BC 01 (Solve HW Problem 1 together)
- Wednesdays:
- 13:15-15:00, GC B3 30 (Exercises)
- Exception: Wed, Sept 24: 13:15-15:00 Lecture
ED Discussion Forum
- We will use the ED Discussion Forum for this class. Everyone is strongly encouraged to make the most of this!
- Ask questions!
- Answer questions!
- The class staff will check the forum on Monday afternoon and on Thursday afternoon.
SWITCHtube Channel
We will not make new video recordings this year. You can access the videos from a couple of years ago. The content is largely the same, but the order of the topics is slightly different.
Grading
- If you do not hand in your final exam your overall grade will be NA.
- Otherwise, your grade will be determined based on the following weighted average: 10% for the Homework, 30% for the Midterm Exam, 60% for the Final Exam.
- The Midterm Exam will take place on Wednesday, November 12, 2025, 13:15-15:00
The Final Exam will take place on at some point between January 12, 2026 and January 31, 2026.
Week 1 (Basics of Probability)
Sept 9: Introduction and Probability Review
Sept 10: Exercise Session (Homework 1)
Week 2 (Information Measures)
Sept 16: Information Measures
Sept 17: Exercise Session (Homework 2)
Week 3 (Information Measures)
Sept 24 : Information Measures (Lecture, exceptionally)
Week 4 (Information Measures)
Oct 1: Exercise Session (Homework 2)
Week 5 (Multi-Arm Bandits)
Oct 8: Exercise Session (Homework 3)
Week 6 (Multi-Arm Bandits)
Oct 14: Multi-Arm Bandits
Oct 15: Exercise Session (Homework 3)
Fall Break (October 20-26)
No class, No exercise session
Week 7 (Detection & Estimation)
Oct 28: Detection and Estimation Theory
Oct 29: Exercise Session (Homework 4)
Week 8 (Distribution Estimation)
Nov 4: Distribution Estimation
Nov 5: Exercise Session (Homework 4)
Week 9 (Property Testing)
Nov 12, 13:15-15:00: Midterm Exam
Week 10 (Exponential Families)
Nov 18: Exponential Families
Nov 19: Exercise Session (Homework 5)
Week 11 (Signal Representations)
Nov 25: Signal Representations
Nov 26: Exercise Session (Homework 6)
Week 12 (Signal Representations)
Dec 2: Signal Representations
Dec 3: Exercise Session (Homework 6)
Week 13 (Compression)
December 9: Compression: Dimensionality Reduction (Random Projections), then classic data compression
December 10: Exercise Session (Homework 7)
Week 14 (Information Measures and Generalization)
December 16: Information-theoretic perspective on Generalizaton of Learning Algorithms
December 17: Exercise Session (Homework 7)