PyFR
PyFR is a Python code for solving high-order computational fluid dynamics problems on unstructured grids. It leverages symbolic manipulation and runtime code generation in order to run different high-performance backends on a variety of hardware platforms. The characteristics of Flux Reconstruction make the method suitable for efficient execution on modern streaming architectures, and PyFR has been demonstrated to achieve good portability 1 and scalability on some of the world's most powerful HPC systems: most notably, a contribution based on PyFR was selected as one of the finalists for the 2016 ACM Gordon Bell Prize 2.
Test case
As test case, we select the 2D Euler vortex example bundled with the PyFR source code. We run the example on 4 nodes with GPU acceleration enabled. We do not measure performance with this test and we do not make comparisons with a native implementation.
Running the container
We assume that a host scratchpad directory is configured to be mounted
automatically inside the container by Sarus, and is accessible at the path
defined by $SCRATCH
. Documentation can be found on this dedicated page:
Mounting custom files and directories into the container. For instance:
--mount=type=bind,source=$SCRATCH,destination=$SCRATCH
First step is to use an interactive container to prepare the simulation data:
# Launch interactive container
srun -C gpu -N1 -t10 --pty sarus run --tty \
ethcscs/pyfr:1.8.0-cuda9.2-ubuntu16.04 bash
then within the container, prepare the data:
1#!/bin/bash
2
3# From the container:
4## copy the example data to the scratch directory
5mkdir $SCRATCH/pyfr/
6cd PyFR-1.8.0/examples/
7cp -r euler_vortex_2d/ $SCRATCH/pyfr/euler_vortex_2d
8
9## Convert mesh data to PyFR format
10cd $SCRATCH/pyfr/euler_vortex_2d
11pyfr import euler_vortex_2d.msh euler_vortex_2d.pyfrm
12
13## Partition the mesh and exit the container
14pyfr partition 4 euler_vortex_2d.pyfrm .
and terminate the interactive session:
exit
Now that the data is ready, we can launch the multi-node simulation. Notice that
we use the --pty
option to srun
in order to visualize and update correctly
PyFR's progress bar (which we request with the -p
option):
srun -C gpu -N4 -t1 --pty sarus run \
--mpi \
ethcscs/pyfr:1.8.0-cuda9.2-ubuntu16.04 \
pyfr run -b cuda -p \
$SCRATCH/pyfr/euler_vortex_2d/euler_vortex_2d.pyfrm \
$SCRATCH/pyfr/euler_vortex_2d/euler_vortex_2d.ini
A typical output will look like:
100.0% [===========================>] 100.00/100.00 daint: 00:00:29 rem: 00:00:00
Container image and Dockerfile
The container image ethcscs/pyfr:1.8.0-cuda9.2-ubuntu16.04 (based on Nvidia
cuda/9.2) used for this test case can be pulled from CSCS DockerHub or be rebuilt with this
Dockerfile
:
1FROM nvidia/cuda:9.2-devel-ubuntu16.04
2
3LABEL com.pyfr.version="1.8.0"
4LABEL com.python.version="3.5"
5
6# Install system dependencies
7# Metis is a library for mesh partitioning:
8# http://glaros.dtc.umn.edu/gkhome/metis/metis/overview
9RUN apt-get update && apt-get install -y \
10 unzip \
11 wget \
12 build-essential \
13 gfortran-5 \
14 strace \
15 realpath \
16 libopenblas-dev \
17 liblapack-dev \
18 python3-dev \
19 python3-setuptools \
20 python3-pip \
21 libhdf5-dev \
22 libmetis-dev \
23 --no-install-recommends && \
24 rm -rf /var/lib/apt/lists/*
25
26# Install MPICH 3.1.4
27RUN wget -q http://www.mpich.org/static/downloads/3.1.4/mpich-3.1.4.tar.gz && \
28 tar xvf mpich-3.1.4.tar.gz && \
29 cd mpich-3.1.4 && \
30 ./configure --disable-fortran --prefix=/usr && \
31 make -j$(nproc) && \
32 make install && \
33 cd .. && \
34 rm -rf mpich-3.1.4.tar.gz mpich-3.1.4 && \
35 ldconfig
36
37# Create new user
38RUN useradd docker
39WORKDIR /home/docker
40
41# Install Python dependencies
42RUN pip3 install numpy>=1.8 \
43 pytools>=2016.2.1 \
44 mako>=1.0.0 \
45 appdirs>=1.4.0 \
46 mpi4py>=2.0 && \
47 pip3 install pycuda>=2015.1 \
48 h5py>=2.6.0 && \
49 wget -q -O GiMMiK-2.1.tar.gz \
50 https://github.com/vincentlab/GiMMiK/archive/v2.1.tar.gz && \
51 tar -xvzf GiMMiK-2.1.tar.gz && \
52 cd GiMMiK-2.1 && \
53 python3 setup.py install && \
54 cd .. && \
55 rm -rf GiMMiK-2.1.tar.gz GiMMiK-2.1
56
57# Set base directory for pyCUDA cache
58ENV XDG_CACHE_HOME /tmp
59
60# Install PyFR
61RUN wget -q -O PyFR-1.8.0.zip http://www.pyfr.org/download/PyFR-1.8.0.zip && \
62 unzip -qq PyFR-1.8.0.zip && \
63 cd PyFR-1.8.0 && \
64 python3 setup.py install && \
65 cd .. && \
66 rm -rf PyFR-1.8.0.zip
67
68CMD ["pyfr --help"]
Used OCI hooks
NVIDIA Container Runtime hook
Native MPI hook (MPICH-based)
References
- 1
F.D. Witherden, B.C. Vermeire, P.E. Vincent,Heterogeneous computing on mixed unstructured grids with PyFR, Computers & Fluids, Volume 120, 2015, Pages 173-186, ISSN 0045-7930, https://doi.org/10.1016/j.compfluid.2015.07.016
- 2
P. Vincent, F. Witherden, B. Vermeire, J. S. Park and A. Iyer, "Towards Green Aviation with Python at Petascale", SC '16: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, Salt Lake City, UT, 2016, pp. 1-11. https://doi.org/10.1109/SC.2016.1