HYPRE for ABecLaplacian_C Tutorial

The following directions explain how to configure, build and run the ABecLaplacian_C example using the external solver library HYPRE. HYPRE is a library of scalable linear solvers and multigrid methods. For more information about HYPRE see their website. More information on setting HYPRE options within AMReX is available in the User’s Guide in the External Solvers section.

Building with HYPRE via GNUMake

Clone, configure, make install and set the HYPRE_DIR environment variable:
```
git clone https://github.com/hypre-space/hypre.git
cd hypre/src
./configure
make install
export HYPRE_DIR=/path_to_hypre_dir/hypre/src/hypre
```
Note

If HYPRE fails the configure step, it may be necessary to manually specify several options. This can be done during the configure step (see ./configure -h). For example, one might replace ./configure with,
```
./configure CXX=CC CC=cc FC=ftn --with-MPI
```
Navigate to the location of the ABecLaplacian_C example, amrex-tutorials/ExampleCodes/LinearSolvers/ABecLaplacian_C.
Make the executable by typing,
```
make -j4 USE_HYPRE=TRUE
```
This will compile the code with HYPRE enabled using 4 processes.
To run the HYPRE enabled example, use the inputs.hypre file as input.
```
./ABecLaplacian_C inputs.hypre
```
To verify the HYPRE solver has been used, look for the line,
```
HYPRE BoomerAMG: Num. iterations = 4; Relative residual = 3.92236062e-05
```
in the output. For this example, the code should quickly reach the end with only a handful of iterations at each step. The code successfully completes with the AMReX finalized output.

Building with HYPRE via CMake

Follow the directions above for cloning, configuring, make installing and setting the environment variable.
Next navigate to the ExampleCodes parent directory, /amrex-tutorials/ExampleCodes/. CMake is currently configured to compile multiple examples at once using the file CMakeLists.txt. Create and enter a build directory to hold the configuration and compiled files.
From the build directory, call CMake with the following configuration:
```
cmake .. -DAMReX_HYPRE=ON \
-DHYPRE_LIBRARIES=${HYPRE_DIR}\lib\libHYPRE.a \
-DHYPRE_INCLUDE_DIRS=${HYPRE_DIR}\include \
-DAMReX_LINEAR_SOLVERS=ON \
-DAMReX_FORTRAN=ON
```
The first option tells CMake to enable the HYPRE interface in AMReX. The following two options specify the location of the HYPRE library we compiled in the first step, and the directory of HYPRE’s required files that we also installed in the first step. In this example, the environment variable HYPRE_DIR is used to replace writing out the entire path. The next flag enables compilation for the LinearSolver example codes, some of which require Fortran, and thus require the last flag.
Next we can build the executable with,
```
cmake --build . -j8
```
This will tell CMake to use 8 processes to compile the source files.
Finally we can run the executable by navigating to the build/LinearSolvers/ABecLaplacian_C folder inside our build directory, and typing the name of the executable followed by the inputs file, inputs.hypre.
```
./ABecLaplacian_C inputs.hypre
```
To verify the HYPRE solver has been used, look for the line,
```
HYPRE BoomerAMG: Num. iterations = 4; Relative residual = 3.92236062e-05
```
in the output. For this example, the code should quickly reach the end with only a handful of iterations at each step. The code successfully completes with the AMReX finalized output.

AMReX with HYPRE via Spack

Using Spack, install AMReX with HYPRE and Fortran options selected.
```
spack install amrex +hypre +fortran
```
Load the desired version of AMReX.
```
spack load amrex +hypre +fortran
```
Identify the location of the installed version of AMReX. Because the location is usually quite long, we will store the result from Spack as the shell variable, AMREX_DIR.
```
AMREX_DIR=$(spack location -i amrex +hypre +fortran)
```
In this example we will build the ABecLaplacian_C example code from the linear solvers in amrex-tutorials. First navigate to the ExampleCodes directory. Then create a build folder to store the compiled files. Inside this folder we’ll use CMake to compile the code.
```
cmake .. -DAMReX_DIR=${AMREX_DIR} \
         -DAMReX_HYPRE=ON \
         -DAMReX_FORTRAN=ON \
         -DAMReX_FORTRAN_INTERFACES=ON \
         -DAMREX_LINEAR_SOLVERS=ON
```
These configuration commands do the following:
- AMReX_DIR: Tells CMake where to find the installed version of AMReX. If this is not supplied, CMake may be unable to locate the AMReX files or it may download the file from the latest release from GitHub.
- AMReX_HYPRE: Enables AMReX to use HYPRE.
- AMReX_FORTRAN: Enables Fortran for AMReX.
- AMReX_FORTRAN_INTERFACES: Enables the Fortran API.
- AMReX_LINEAR_SOLVERS: This command is specific to the ExamplesCodes install configuration, i.e. CMakeLists.txt. It tells CMake to compile all the linear solver examples.
After setting up the configuration, we build the executables with CMake. This command will build the files according to the configuration in the current directory using 4 processes (-j4).
```
cmake --build . -j4
```
To run the HYPRE example navigate to the folder, path_to_base_dir/ExampleCodes/build/LinearSolvers/ABecLaplacian_C and call the executable with the inputs.hypre file as input.
```
./ABecLaplacian_C inputs.hypre
```
To verify the HYPRE solver has been used, look for the line,
```
HYPRE BoomerAMG: Num. iterations = 4; Relative residual = 3.92236062e-05
```
in the output. For this example, the code should quickly reach the end with only a handful of iterations at each step. The code successfully completes with the AMReX finalized output.