Parallel and high-performance computing

Lecture recordings

You can find the recordings of the lectures (without editing) here.

Find below (linked pdf file) the calendar for the oral exam.

The exam will be in the room CM 0 10

As set in the project description, the oral presentation will be less than 5 minutes + 5 minutes of questions about the project and the course in general.

You can bring your own laptop for doing the presentation, or we can provide you a laptop with your submitted slides ready to be used.

Lecture

Course introduction

Exercises

A few short videos present the basics and theory behind algorithms and parallel programming. Videos needed for solving exercises of Series 0.

1. Time and space complexities: Big O notation
2. Vocabulary
   • DAG 1/3
   • DAG 2/3
   • DAG 3/3
3. Flynn's taxonomy
4. Levels of parallelism:
   • Instruction-level parallelism
   • Data parallelism
   • Thread-level parallelism
5. Threads vs. processes

Workshop in the computer lab: how to use a Linux Cluster

• connecting to the clusters
• using the SLURM scheduler
• introduction to git

Hands-on

• For this week the exercises are in the slides.
• The code needed for the compilation exercises is in the link below or in the git repository https://gitlab.epfl.ch/math454-phpc/exercises-2025

Lecture

• In case you want to delve deeper into this topic Performance Analysis and Tuning on Modern CPUs (Denis Bakhvalov)

Debug - Profile - Optimize

Two situations can occur in the parallel programming world : (a) you already have a sequential code and (b) you start from scratch your parallel code. Here we assume the first (a).

Before going to parallelization (with whatever parallel paradigm such as OpenMP, MPI or acceleration with GPUs), you must have a bugs free and optimized sequential code. Do do that, you must follow a Debug - Profile - Optimization strategy.

During this week series we will start using gdb and gprof, but if you feel lost and/or want to know a little bit more, please watch these two videos:

• Debugging with gdb
• Profiling with gprof

the tools will be extensively used during the exercises and up to the end of the semester.

Parallelization with OpenMP

Once your code is fully debugged and optimized, you can parallelize it. In this course, we start with the OpenMP parallel paradigm. This will be cover during the theory lecture (see the associated slides).

OpenMP documentation

• Rookie HPC
• OpenMP API 6.0
• OpenMP API 6.0 cheat sheet

Exercises

Sanitize and optimize sequential codes. Use OpenMP to parallelize a code that computes pi and a Poisson solver

Lecture

• MPI basics

MPI documentation

• Rookie HPC.
• MPI 4.0 Standard

Exercises

• Parallelization of an existing code using MPI

Lecture (1h)

• MPI advanced functions
• Advanced MPI datatypes
• MPI communicators

Exercises (3h)

• Parallelization of the Conjugate Gradient linear solver using MPI

• The assignment is due April 9 (23h59 CEST).

• Some clarifications:

  • tasks for slurm = mpi process

Lecture

• Hybrid programming with MPI and OpenMP
• MPI for Python
• Detailed explanation of Series 5's solution.

Exercises

• MPI advanced (derived datatypes, persistent communication, IO)
• MPI with Python

Lecture

Introduction to GPU programming

• Trends in HPC
• Hardware architecture (GPU vs. CPU)
• Software environement
• How to program on a GPU with CUDA

Exercises

• Few hands-on exercises to get familiar with the CUDA programming model
• For the exercise you need to load gcc and cuda
  module load gcc cuda

Lecture

Advanced CUDA programming

• Thread Cooperation in GPU Computing
• GPU Memory Model
  • Shared memory
  • Constant memory
  • Global memory
• Test case

Exercises

• Graded exercise, the exercise is due April 30 at 23h59

Lecture

Parallel Profiling

Exercises

Parallel Profiling

Project work. See the top of the page for more information.

Professor and assistants will be available on Discord to answer questions from 08:15 to 12:00 every Tuesday.