PSC Bridges-2

If this is your first time building MFIX-Exa on Bridges-2, please review the general notes below and Basics section first.

  • To access Delta-2, you will need PSC account which can be administered by the NSF ACCESS allocation managment system.

  • You can find the name of your account(s) and the current balance with the accounts command.

  • These instructions build MFIX-Exa on the login nodes using -j8 CPUs. You may have to decrease this value if there is high traffic or you may want to increase this value if you are on a compute node interactively.

  • For the dependencies, it is assumed that you have set the following environment variables:

    export HYPRE_INSTALL_DIR=$HOME/<path/to/my/hypre-install-dir>
export CSG_INSTALL_DIR=$HOME/<path/to/my/csg-dep-install-dir>
export CSG_LIB_DIR=$HOME/<path/to/my/csg-lib-install-dir>
export ASCENT_INSTALL_DIR=$HOME/<path/to/my/ascent-install-dir>

    to a path that you have read/write access to. You will need to recall these paths later if you want to build MFIX-Exa with the optional dependencies.

Basics

Source code

Before building, first obtain a copy of the source code following the instructions on the MFIX-Exa website.

Modules

There are (basically) no modules loaded by default on Bridges-2.

module load gcc/10.2.0                 # PrgEnv
module load openmpi/4.0.5-gcc10.2.0    # MPI
module load cuda/11.7.1                # for GPU
module load boost/1.75-gcc10.2.0       # for CSG

Building MFIX-Exa

CMake 3.20 or higher is required. You are running version 3.11.4

The commands below are the superbuild instructions, i.e., AMReX is built as part of the MFIX-Exa build process. To build MFIX-Exa with hypre, csg and/or ascent dependencies, you first need to build and install these libraries and their dependencies. Instructions on building the necessary dependencies are below and should be successfully installed first. Only the gmake method of building MFIX-Exa is currently recommended.

cp -r exec exec.cpu
make -C exec.cpu -j8 \
     COMP=gnu \
     USE_MPI=TRUE \
     USE_OMP=FALSE \
     USE_CUDA=FALSE \
     USE_TINY_PROFILE=FALSE \
     USE_CSG=FALSE \
     USE_HYPRE=FALSE \
     DEBUG=FALSE

Optional build dependencies

The following dependencies need to be built and installed prior to following any of the full build instructions above.

  1. HYPRE

    git clone https://github.com/hypre-space/hypre.git
pushd hypre/src/
git checkout v2.26.0
./configure --prefix=$HYPRE_INSTALL_DIR --with-MPI
make -j8 install
popd

  2. cmake

    You will need cmake to install some of the dependencies below and the Bridges2 native version is unsupported at the time of this writing. We’ll install a local one. You will need to set export CMAKE_INSTALL_DIR=$HOME/<path/to/my/cmake-install-dir> to a path that you have read/write access to.

    wget https://cmake.org/files/v3.23/cmake-3.23.2.tar.gz
tar -zxvf cmake-3.23.2.tar.gz
pushd cmake-3.23.2
./bootstrap --prefix=$CMAKE_INSTALL_DIR
make -j8 install
popd

    After installing cmake, you will need to prepend it to your path

    export PATH=$CMAKE_INSTALL_DIR/bin:$PATH

  3. Catch2

    git clone --depth 1 --branch v2.13.7 https://github.com/catchorg/Catch2
pushd Catch2/
cmake -S . -B build -DCMAKE_INSTALL_PREFIX=$CSG_INSTALL_DIR
cd build/
make -j8 install
popd

  4. GMP

    wget --no-check-certificate https://ftp.gnu.org/gnu/gmp/gmp-6.2.1.tar.xz
tar -xf gmp-6.2.1.tar.xz
pushd gmp-6.2.1
./configure --prefix=$CSG_INSTALL_DIR
make -j8 install
popd

  5. MPFR

    wget --no-check-certificate https://ftp.gnu.org/gnu/mpfr/mpfr-4.1.0.tar.xz
tar -xf mpfr-4.1.0.tar.xz
pushd mpfr-4.1.0/
./configure --with-gmp=$CSG_INSTALL_DIR --prefix=$CSG_INSTALL_DIR
make -j8 install
popd

  6. CGAL

    git clone --depth 1 --branch v5.3 https://github.com/CGAL/cgal
pushd cgal/
cmake -S . -B build -DCMAKE_INSTALL_PREFIX=$CSG_INSTALL_DIR
cd build/
make -j8 install
popd

  7. PEGTL

    git clone --branch 3.2.2 https://github.com/taocpp/PEGTL
pushd PEGTL/
cmake -S . -B build -DCMAKE_INSTALL_PREFIX=$CSG_INSTALL_DIR
cd build/
make -j8 install
popd

  8. CSG EB library

    Make sure you have boost loaded before building. (This is intended to be executed from inside the mfix repo.)

    make -C subprojects/csg-eb install DESTDIR=$CSG_LIB_DIR \
     PEGTL_HOME=$CSG_INSTALL_DIR \
     CGAL_HOME=$CSG_INSTALL_DIR \
     CATCH2_HOME=$CSG_INSTALL_DIR \
     ENABLE_CGAL=TRUE

  9. Conduit

    git clone --recursive https://github.com/LLNL/conduit.git
pushd conduit/
git checkout v0.8.6
mkdir build && cd build
cmake -S ../src -DCMAKE_INSTALL_PREFIX=$ASCENT_INSTALL_DIR \
      -DENABLE_OPENMP=OFF \
      -DENABLE_MPI=ON \
      -DENABLE_CUDA=OFF \
      -DCMAKE_BUILD_TYPE=Release
make -j8 install
popd

  10. Vtk-m

    git clone --branch master https://gitlab.kitware.com/vtk/vtk-m.git
pushd vtk-m/
git checkout v1.9.0
mkdir build && cd build/
cmake -S ../ -DCMAKE_INSTALL_PREFIX=$ASCENT_INSTALL_DIR \
      -DVTKm_ENABLE_OPENMP=OFF \
      -DVTKm_ENABLE_MPI=ON \
      -DVTKm_ENABLE_CUDA=OFF \
      -DVTKm_USE_64BIT_IDS=OFF \
      -DVTKm_USE_DOUBLE_PRECISION=ON \
      -DVTKm_USE_DEFAULT_TYPES_FOR_ASCENT=ON \
      -DVTKm_NO_DEPRECATED_VIRTUAL=ON \
      -DCMAKE_BUILD_TYPE=Release
make -j8 install
popd

  11. Ascent

    git clone --recursive https://github.com/Alpine-DAV/ascent.git
pushd ascent
git checkout v0.9.0
mkdir build && cd build/
cmake -S ../src -DCMAKE_INSTALL_PREFIX=$ASCENT_INSTALL_DIR \
      -DCONDUIT_DIR=$ASCENT_INSTALL_DIR \
      -DVTKM_DIR=$ASCENT_INSTALL_DIR \
      -DENABLE_VTKH=ON \
      -DENABLE_FORTRAN=OFF \
      -DENABLE_PYTHON=OFF \
      -DENABLE_DOCS=OFF \
      -DBUILD_SHARED_LIBS=ON \
      -DCMAKE_BUILD_TYPE=Release \
      -DENABLE_GTEST=OFF \
      -DENABLE_TESTS=OFF
make -j8 install
popd

Running Jobs

Common Slurm commands:

  • sinfo see available/allocated resources

  • sbatch runit_cpu.sh submit a cpu job to the queue

  • squeue -u USER check job status of user USER

  • squeue -p PARTITION check job status of partition PARTITION

  • scancel JOBID kill a job with id JOBID

  • salloc -N 1 -p GPU -A peb230001p -t 00:02:00 --exclusive --gpus-per-node=8 grab an entire GPU node for five minutes

  • salloc -N 1 -p GPU-shared -A peb230001p -t 00:05:00 --gpus-per-node=2 grab two GPUs interactively on a shared node for five minutes, you can grab up to four --gpus-per-node

Example run script for GPU is below. You can mirror the interactive commands above to convert this into a shared job script. CPU-only runs have not been tested on this machine.

#!/bin/bash
#!/bin/bash
#SBATCH -A peb230001p
#SBATCH -N 1
#SBATCH -p GPU-shared
#SBATCH -t 00:05:00
#SBATCH --gpus=v100-32:4

# load modules
module load gcc/10.2.0                 # PrgEnv
module load openmpi/4.0.5-gcc10.2.0    # MPI
module load cuda/11.7.1                # for GPU
module load boost/1.75-gcc10.2.0       # for CSG

#echo commands to stdout
set -x

mpirun -np 4 ./mfix inputs.rt > screen.txt