18/02/2019 Course introduction – Parallel programming frameworks and high-level approach to parallel programming over different platforms: MPI, TBB and OpenCL as main examples; course organization and prerequisites; reference books and studying material.
– MPI (Message Passing Interface) standard – brief history and aim of the standard, single program / multiple data execution model, compilation and linkage model; issues in supporting multiple programming languages and uses (application, utility library and programming language support) with a static compilation and linkage approach. Portability in parallel programming: functional and non-functional aspects, performance tuning and performance debugging. MPI basic concepts MPI as a parallel framework that supports a structured approach to parallel programming. Basic concepts of MPI: communicators (definition, purpose, difference between inter and intra-communicators, process ranks).
22/02/2019 MPI basic concepts – Point to point communication (concepts of envelope, local/global completion, blocking/non-blocking primitive, send modes); collective communications (definition, communication scope, global serialization, freedom of implementation in the standard); MPI datatypes (basic meaning and use, primitive / derived datatypes).
25/02/2019 MPI – MPI Datatypes relationship with sequential language types. MPI library initialization and basic MPI usage; point to point communication semantics (buffer behaviour, receive, status objects, MPI_PROC_NULL), basic and derived MPI datatypes (purpose as explicitly defined meta-data provided to the MPI implementation, multiple language bindings, code-instantiated metadata, examples). MPI datatypes semantics: typemap and type signature.
27/02/2019 MPI – MPI datatypes (matching rules for communication, role in MPI-performed packing and unpacking); core primitives for datatype creation ( MPI_Type_* : contiguous, vector, hvector, indexed, hindexed, struct; commit, free) and examples. ( Lesson held in room C40 at CNR )
04/03/2019 MPI – Point to point communication modes (MPI_BSEND, MPI_SSEND; MPI_RSend usage); non-blocking communication (Wait and Test group of primitives, semantics, MPI_Request object handles to active requests); canceling and testing cancellation of non-blocking primitives (issues and pitfalls, interaction with MPI implementation, e.g. MPI_finalize). Communicators and groups (communicator design aim and programming abstraction, local and global information, groups as local objects, primitives for locally creating and managing groups); intracommunicators (basic primitives concerning size, rank, comparison); communicator creation as a collective operation.
06/03/2019 MPI – MPI_Comm_create basic and general case; MPI_Comm_split; MPI collective communications (definition and semantics, execution environment, basic features, agreement of key parameters among the processes, constraints on Datatypes and typemaps for collective op.s, overall serialization vs synchronization, potential deadlocks).
11/03/2019 MPI Lab – Basic program structure. Simple ping-pong example, generalization. Examples with derived datatypes. Structured parallel programming in MPI, separation of concerns in practice. Assigned task for next LAB time: MPI matrix multiplication.
14/03/2019 Rescheduled
18/03/2019 MPI Lab – Matrix Multiplication. Implementing communication with assigned asynchronicity degree.
21/03/2019 MPI Extent and modification for derived datatypes. Collective operation for communication and computation. Taxonomy of MPI collectives (blocking/non-blocking, synchronization/communication/communication+computation, asymmetry of the communication pattern, variable size versions, all- versions). Blocking collectives. MPI collectives with both computation and communication: Reduce (and variants) and Scan (and variants). Using MPI operators with Reduce and Scan. Defining custom user operators, issues and implementation of operator functions.
25/03/2019 MPI Lab – Farm skeleton with round robin and dynamic load dstribution strategies.
28/03/2019 Applications + MPI Lab KDD/Data Mining intro; K-means algorithm, parallelization opportunities. Lab session with sequential code and MPI parallelization.
08/04/2019 TBB – Thread Building Blocks : C++ template library overview: purpose, abstraction mechanisms and implementation layers (templates, runtime, supported abstractions, use of C++ concepts). Seminar Real-Time in Heterogeneous High-performance Systems (Dr. Matteo Andreozzi, ARM) Co-location of multiple workloads on a single system allows to improve its resources utilization, allows resources re-use and improves the efficiency of data sharing across workloads. This, however, comes at the cost of potential performance degradation due to interference on shared resources, and increased uncertainty in terms of workload performance predictability. The seminar provided an overview about how the System Architecture team in Arm approaches these challenges and how modelling and performance analysis play a central role in the activity.
11/04/2019 Rescheduled
15/04/2019 TBB TBB introduction. Develompment history and current status: abstraction and implementation layers. Task-based description vs thread-based one, scalability and automatic management of hierarchical parallelism. Basic abstractions and algorithms.
29/04/2019 TBB Parallel_for : ranges, partitioners, grain size for parallelism exploitation. C++ lambda expressions.
02/05/2019 MPI LAB Correction and discussion of previously assigned MPI exercises.
06/05/2019 TBB LAB Basic examples of TBB parallel for. Performing parallel computations on ranges and returning computation results.
09/05/2019 (3h 14:30 – 17:30) TBB TBB parallel containers: variants of maps, queues and vectors with higher concurrent performance and modified semantics with respect to STL. TBB LAB Mandelbrot set computation with a parallel for.
13/05/2019 Rescheduled
16/05/2019 (3h 14:30 – 17:30) TBB TBB mutexes variants and their usage. Explicitly setting the thread-parallelism level (Task Scheduler, Task Arena). TBB LAB Experiments with Mandelbrot parallel for with varying thread count and computation load.
20/05/2019 OpenCL OpenCL intro. Design concepts and programming abstractions: Devices/host interaction, context, kernel, command queues; execution model; memory spaces and memory consistency in OpenCL.
23/05/2019 (3h 14:00 – 17:00) OpenCL OpenCL C/C++ subset for kernels; kernel compilation, program objects, memory objects and kernel arguments, code execution, kernel instances and workgroups, workgroup synchronization; portability and chances for load balancing: mapping OpenCL code onto both the GPU and the CPU; examples of vector types and vector operations. Basic concepts of OpenCL 1 applied to program contruction: simple array operations, matrix multiplication and related workgroup optimizations, simple parallel reduction and thread cooperation.
27/05/2019 rescheduled due to EU election break.
28/05/2019 OpenCL OpenCL 1.2 and beyond. OpenCL event generation and handling (event barriers) for inter-queue, non local synchronization. OpenCL 2.0 and 2.1 features: shared virtual memory (GPU/CPU memory space overlaying); nested kernels and recursive parallelism without host/device interaction; generic address space as a tool to avoid source code duplication; C11 atomics in OpenCL 2.0; pipes. OpenCL 2.1: moving towards the use of a proper subset of C++14 for kernels (e.g. templates, overloading, lambda f.), allowing single-source joint OpenCL and non-OpenCL programming (SYCL) and providing a more homogeneous and organized semantics. The SPIR-V interoperable, symbolic GPU machine code representation and its use in the LLVM based development toolset.
30/05/2019 (3h 14:00 – 17:00) OpenCL LAB Exercises from the GIT examples: exercises 4 and 5 (vector addition), 6 (matrix multiplication), 7 and 8 (exploiting private memory).