UE Parallel systems

Diplômes intégrant cet élément pédagogique :


Today, parallel computing is omnipresent across a large spectrum of
computing platforms, from processor cores or GPUs up to Supercomputers
and Cloud platforms. The largest Supercomputers  gather millions of
processing units and are heading towards Exascale (a quintillion or
10^18   flops -  http://top500.org).  If parallel processing was
initially targetting scientific computing, it is now part of many
domains like Big Data analytics and  Deep Learning. But making an
efficient use  of  these parallel  resources  requires a deep
understanding of  architectures, systems, parallel programming models,
and  parallel algorithms. 

This class  will  progressively enable  attendees to master advanced
parallel processing.   No prior knowledge on parallel programming is required to attend this
class, nor specific skills on system, processor architecture or
theoretical models beyond the base training that any computer science M1 student should
have received. Students should have a taste for  experimenting with
advanced computer systems  and ready to be exposed to  a few
theoretical models  (mainly cost models for reasoning about parallel

The class is organized around weekly lectures, discussions and help
time. The presented materials that will be
available each week on the class web page. To get practical experience
and good  understanding of the main concepts  the students are
expected to develop short programs and experiments. Students will also
have to prepare by teams of 2  a presentation to be presented by the
end of the class.  Students  will have access to Grid'5000 parallel
machines  and  the SimGrid simulator for experiments.

This class prepare students to pursue a M2R internship in related
topics in an academic or industrial research team to next pursue a PhD
in computer science or to work into  companies on parallel or large systems.

Topics to  be covered:

  - Introduction to  parallelism from GPU to supercomputers.
  - Hardware and system considerations for parallel processing
    (multi-core architectures, process and thread handling,  cache efficiency, remote data access, atomic instructions)
  - Parallel programming: message passing, one-sided communications,
    task-based programming, work stealing based runtimes (MPI, Cilk, TBB, OpenMP).
  - Modeling of  parallel programs and platforms. Locality,
    granularity, memory space, communications.
  - Parallel algorithms, collective communications, topology aware algorithms.
  - Scheduling: list algorithms; partitioning techniques. Application to resource management (PBS, LSF, SGE, OAR).
  - Large scale scientific computing: parallelization of Lagrangian
    and Eulerian  solvers, parallel data analytics and scientific
  - Parallel big data frameworks, batch and stream processing  (Spark, FLink)
  - IA and HPC: from frameworks (TensorFlow,Pytorch, Scikit-learn)  to
    dedicated hardware (GPUs, TPUs).  Parallel stochastic gradient
    descent, parallel neural networks, synchronous/asynchronous
    training strategies.

Informations complémentaires

Méthode d'enseignement : En présence
Lieu(x) : Grenoble - Domaine universitaire
Langue(s) : Anglais