Building with GNU Make

In this build approach, you write your own make files defining a number of variables and rules. Then you invoke make to start the building process. This will result in an executable upon successful completion. The temporary files generated in the building process are stored in a temporary directory named tmp_build_dir.

Dissecting a Simple Make File

An example of building with GNU Make can be found in amrex-tutorials/ExampleCodes/Basic/HelloWorld_C. Table 1 below shows a list of important variables.

Table 1 Important make variables

Variable

Value

Default

AMREX_HOME

Path to amrex

environment

COMP

gnu, cray, ibm, intel, intel-llvm, intel-classic, llvm, or pgi

none

CXXSTD

C++ standard (c++17, c++20)

compiler default, at least c++17

DEBUG

TRUE or FALSE

FALSE

DIM

1 or 2 or 3

3

PRECISION

DOUBLE or FLOAT

DOUBLE

TEST

TRUE or FALSE

FALSE

USE_ASSERTION

TRUE or FALSE

FALSE

USE_MPI

TRUE or FALSE

FALSE

USE_OMP

TRUE or FALSE

FALSE

USE_CUDA

TRUE or FALSE

FALSE

USE_HIP

TRUE or FALSE

FALSE

USE_SYCL

TRUE or FALSE

FALSE

USE_RPATH

TRUE or FALSE

FALSE

WARN_ALL

TRUE or FALSE

TRUE for DEBUG FALSE otherwise

AMREX_CUDA_ARCH

or CUDA_ARCH

CUDA arch such as 70

70 if not set or detected

AMREX_AMD_ARCH

or AMD_ARCH

AMD GPU arch such as gfx908

none if the machine is unknown

USE_GPU_RDC

TRUE or FALSE

TRUE

At the beginning of amrex-tutorials/ExampleCodes/Basic/HelloWorld_C/GNUmakefile, AMREX_HOME is set to the path to the top directory of AMReX. Note that in the example ?= is a conditional variable assignment operator that only has an effect if AMREX_HOME has not been defined (including in the environment). One can also set AMREX_HOME as an environment variable. For example in bash, one can set

export AMREX_HOME=/path/to/amrex

alternatively, in tcsh one can set

setenv AMREX_HOME /path/to/amrex

Note: when setting AMREX_HOME in the GNUmakefile, be aware that ~ does not expand, so AMREX_HOME=~/amrex/ will yield an error.

One must set the COMP variable to choose a compiler. Currently the list of supported compilers includes gnu, cray, ibm, intel, llvm, and pgi.

One could set the DIM variable to either 1, 2, or 3, depending on the dimensionality of the problem. The default dimensionality is 3. AMReX uses double precision by default. One can change to single precision by setting PRECISION=FLOAT. (Particles have an equivalent flag USE_SINGLE_PRECISION_PARTICLES=TRUE/FALSE.)

Variables DEBUG, TEST, USE_MPI and USE_OMP are optional with default set to FALSE. The meaning of these variables should be obvious. When DEBUG=TRUE, aggressive compiler optimization flags are turned off and assertions in source code are turned on. For production runs, DEBUG should be set to FALSE. TEST and USE_ASSERTION are set by default in CI and add slight debugging, e.g., initializing default values in FABs. An advanced variable, MPI_THREAD_MULTIPLE, can be set to TRUE to initialize MPI with support for concurrent MPI calls from multiple threads.

Variables USE_CUDA, USE_HIP and USE_SYCL are used for targeting Nvidia, AMD and Intel GPUs, respectively. At most one of the three can be TRUE. For HIP and SYCL builds, we do only test against C++17 builds at the moment.

The variable USE_RPATH controls the link mechanism to dependent libraries. If enabled, the library path at link time will be saved as a rpath hint in created binaries. When disabled, dynamic library paths could be provided via export LD_LIBRARY_PATH hints at runtime.

For GCC and Clang, the variable WARN_ALL controls the compiler’s warning options. There is also a make variable WARN_ERROR (with default of FALSE) to turn warnings into errors.

When USE_CUDA is TRUE, the make system will try to detect what CUDA arch should be used by running $(CUDA_HOME)/extras/demo_suite/deviceQuery if your computer is unknown. If it fails to detect the CUDA arch, the default value of 70 will be used. The user could override it by make USE_CUDA=TRUE CUDA_ARCH=80 or make USE_CUDA=TRUE AMREX_CUDA_ARCH=80.

After defining these make variables, a number of files, Make.defs, Make.package and Make.rules, are included in the GNUmakefile. AMReX-based applications do not need to include all directories in AMReX; an application which does not use particles, for example, does not need to include files from the Particle directory in its build. In this simple example, we only need to include $(AMREX_HOME)/Src/Base/Make.package. An application code also has its own Make.package file (e.g., ./Make.package in this example) to append source files to the build system using operator +=. Variables for various source files are shown below.

CEXE_sources

C++ source files. Note that C++ source files are assumed to have a .cpp extension.

CEXE_headers

C++ headers with .h, .hpp, or .H extension.

cEXE_sources

C source files with .c extension.

cEXE_headers

C headers with .h extension.

f90EXE_sources

Free format Fortran source with .f90 extension.

F90EXE_sources

Free format Fortran source with .F90 extension. Note that these Fortran files will go through preprocessing.

In this simple example, the extra source file, main.cpp is in the current directory that is already in the build system’s search path. If this example has files in a subdirectory (e.g., mysrcdir), you will then need to add the following to Make.package.

VPATH_LOCATIONS += mysrcdir
INCLUDE_LOCATIONS += mysrcdir

Here VPATH_LOCATIONS and INCLUDE_LOCATIONS are the search path for source and header files, respectively.

Tweaking the Make System

The GNU Make build system is located at amrex/Tools/GNUMake. You can read README.md and the make files there for more information. Here we will give a brief overview.

Besides building executable, other common make commands include:

make cleanconfig

This removes the executable, .o files, and the temporarily generated files for the given build. Note that one can add additional targets to this rule using the double colon (::)

make clean and make realclean

These remove all files generated by make for all builds.

make help

This shows the rules for compilation.

make print-xxx

This shows the value of variable xxx. This is very useful for debugging and tweaking the make system.

Compiler flags are set in amrex/Tools/GNUMake/comps/. Note that variables like CXX and CXXFLAGS are reset in that directory and their values in environment variables are disregarded. However, one could override them with make command line arguments (e.g., make CXX=/path/to/my/mpicxx). Site-specific setups (e.g., the MPI installation) are in amrex/Tools/GNUMake/sites/, which includes a generic setup in Make.unknown. You can override the setup by having your own sites/Make.$(host_name) file, where variable host_name is your host name in the make system and can be found via make print-host_name. You can also have an amrex/Tools/GNUMake/Make.local file to override various variables. See amrex/Tools/GNUMake/Make.local.template for more examples of how to customize the build process.

If you need to pass macro definitions to the preprocessor, you can add them to your make file as follows,

DEFINES += -Dmyname1 -Dmyname2=mydefinition

To link to an additional library say foo with headers located at foopath/include and library at foopath/lib, you can add the following to your make file before the line that includes AMReX’s Make.defs,

INCLUDE_LOCATIONS += foopath/include
LIBRARY_LOCATIONS += foopath/lib
LIBRARIES += -lfoo

Specifying your own compiler

The amrex/Tools/GNUMake/Make.local file can also specify your own compile commands by setting the variables CXX, CC, FC, and F90. This might be necessary if your systems contains non-standard names for compiler commands.

For example, the following amrex/Tools/GNUMake/Make.local builds AMReX using a specific compiler (in this case gcc-8) without MPI. Whenever USE_MPI is true, this configuration defaults to the appropriate mpixxx command:

ifeq ($(USE_MPI),TRUE)
  CXX = mpicxx
  CC  = mpicc
  FC  = mpif90
  F90 = mpif90
else
  CXX = g++-8
  CC  = gcc-8
  FC  = gfortran-8
  F90 = gfortran-8
endif

For building with MPI, we assume mpicxx, mpif90, etc. provide access to the correct underlying compilers.

GCC on macOS

The example configuration above should also run on the latest macOS. On macOS the default cxx compiler is clang, whereas the default Fortran compiler is gfortran. Sometimes it is good to avoid mixing compilers, in that case we can use the Make.local to force using GCC. However, macOS’ Xcode ships with its own (woefully outdated) version of GCC (4.2.1). It is therefore recommended to install GCC using the homebrew package manager. Running brew install gcc installs gcc with names reflecting the version number. If GCC 8.2 is installed, homebrew installs it as gcc-8. AMReX can be built using gcc-8 (with and without MPI) by using the following amrex/Tools/GNUMake/Make.local:

CXX = g++-8
CC  = gcc-8
FC  = gfortran-8
F90 = gfortran-8

INCLUDE_LOCATIONS += /usr/local/include

The additional INCLUDE_LOCATIONS are installed using homebrew also. Note that if you are building AMReX using homebrew’s gcc, it is recommended that you use homebrew’s mpich. Normally it is fine to simply install its binaries: brew install mpich. But if you are experiencing problems, we suggest building mpich using homebrew’s gcc: brew install mpich --cc=gcc-8.

Fortran

If your code does not use Fortran, you can add BL_NO_FORT=TRUE to your makefile to disable Fortran.

ccache

If you use ccache, you can add USE_CCACHE=TRUE to your makefile.

Building libamrex

If an application code already has its own elaborated build system and wants to use AMReX, an external AMReX library can be created instead. In this approach, one runs ./configure, followed by make and make install. Other make options include make distclean and make uninstall. In the top AMReX directory, one can run ./configure -h to show the various options for the configure script. In particular, one can specify the installation path for the AMReX library using:

./configure --prefix=[AMReX library path]

This approach is built on the AMReX GNU Make system. Thus the section on Building with GNU Make is recommended if any fine tuning is needed. The result of ./configure is GNUmakefile in the AMReX top directory. One can modify the make file for fine tuning.

To compile an application code against the external AMReX library, it is necessary to set appropriate compiler flags and set the library paths for linking. To assist with this, when the AMReX library is built, a configuration file is created in [AMReX library path]/lib/pkgconfig/amrex.pc. This file contains the Fortran and C++ flags used to compile the AMReX library as well as the appropriate library and include entries.

The following sample GNU Makefile will compile a main.cpp source file against an external AMReX library, using the C++ flags and library paths used to build AMReX:

AMREX_LIBRARY_HOME ?= [AMReX library path]

LIBDIR := $(AMREX_LIBRARY_HOME)/lib
INCDIR := $(AMREX_LIBRARY_HOME)/include

COMPILE_CPP_FLAGS ?= $(shell awk '/Cflags:/ {$$1=$$2=""; print $$0}' $(LIBDIR)/pkgconfig/amrex.pc)
COMPILE_LIB_FLAGS ?= $(shell awk '/Libs:/ {$$1=$$2=""; print $$0}' $(LIBDIR)/pkgconfig/amrex.pc)

CFLAGS := -I$(INCDIR) $(COMPILE_CPP_FLAGS)
LFLAGS := -L$(LIBDIR) $(COMPILE_LIB_FLAGS)

all:
        g++ -o main.exe main.cpp $(CFLAGS) $(LFLAGS)

Building with CMake

An alternative to the approach described in the section on Building libamrex is to install AMReX as an external library by using the CMake build system. A CMake build is a two-step process. First cmake is invoked to create configuration files and makefiles in a chosen directory (builddir). This is roughly equivalent to running ./configure (see the section on Building libamrex). Next, the actual build and installation are performed by invoking make install from within builddir. This installs the library files in a chosen installation directory (installdir). If no installation path is provided by the user, AMReX will be installed in /path/to/amrex/installdir. The CMake build process is summarized as follows:

mkdir /path/to/builddir
cd    /path/to/builddir
cmake [options] -DCMAKE_BUILD_TYPE=[Debug|Release|RelWithDebInfo|MinSizeRel] -DCMAKE_INSTALL_PREFIX=/path/to/installdir  /path/to/amrex
make  install
make  test_install  # optional step to test if the installation is working

In the above snippet, [options] indicates one or more options for the customization of the build, as described in the subsection on Customization options. If the option CMAKE_BUILD_TYPE is omitted, CMAKE_BUILD_TYPE=Release is assumed. Although the AMReX source could be used as build directory, we advise against doing so. After the installation is complete, builddir can be removed.

Customization options

AMReX build can be customized by setting the value of suitable configuration variables on the command line via the -D <var>=<value> syntax, where <var> is the variable to set and <value> its desired value. For example, one can enable OpenMP support as follows:

cmake -DAMReX_OMP=YES -DCMAKE_INSTALL_PREFIX=/path/to/installdir  /path/to/amrex

In the example above <var>=AMReX_OMP and <value>=YES. Configuration variables requiring a boolean value are evaluated to true if they are assigned a value of 1, ON, YES, TRUE, Y. Conversely they are evaluated to false if they are assigned a value of 0, OFF, NO, FALSE, N. Boolean configuration variables are case-insensitive. The list of available options is reported in the table below.

Table 2 AMReX build options (refer to section Building GPU Support for GPU-related options).

Variable Name

Description

Default

Possible values

CMAKE_Fortran_COMPILER

User-defined Fortran compiler

user-defined

CMAKE_CXX_COMPILER

User-defined C++ compiler

user-defined

CMAKE_Fortran_FLAGS

User-defined Fortran flags

user-defined

CMAKE_CXX_FLAGS

User-defined C++ flags

user-defined

CMAKE_CXX_STANDARD

C++ standard

compiler/17

17, 20

AMReX_SPACEDIM

Dimension of AMReX build

3 ;-separated list

“1;2;3”

USE_XSDK_DEFAULTS

Use xSDK defaults settings

NO

YES, NO

AMReX_BUILD_SHARED_LIBS

Build as shared C++ library

NO (unless xSDK)

YES, NO

AMReX_FORTRAN

Enable Fortran language

NO

YES, NO

AMReX_PRECISION

Set the precision of reals

DOUBLE

DOUBLE, SINGLE

AMReX_PIC

Build Position Independent Code

NO

YES, NO

AMReX_IPO

Interprocedural optimization (IPO/LTO)

NO

YES, NO

AMReX_MPI

Build with MPI support

YES

YES, NO

AMReX_OMP

Build with OpenMP support

NO

YES, NO

AMReX_GPU_BACKEND

Build with on-node, accelerated GPU backend

NONE

NONE, SYCL, HIP, CUDA

AMReX_GPU_RDC

Build with Relocatable Device Code support

YES

YES, NO

AMReX_FORTRAN_INTERFACES

Build Fortran API

NO

YES, NO

AMReX_LINEAR_SOLVERS

Build AMReX linear solvers

YES

YES, NO

AMReX_AMRDATA

Build data services

NO

YES, NO

AMReX_AMRLEVEL

Build AmrLevel class

YES

YES, NO

AMReX_EB

Build Embedded Boundary support

NO

YES, NO

AMReX_PARTICLES

Build particle classes

YES

YES, NO

AMReX_PARTICLES_PRECISION

Set reals precision in particle classes

Same as AMReX_PRECISION

DOUBLE, SINGLE

AMReX_BASE_PROFILE

Build with basic profiling support

NO

YES, NO

AMReX_TINY_PROFILE

Build with tiny profiling support

NO

YES, NO

AMReX_TRACE_PROFILE

Build with trace-profiling support

NO

YES, NO

AMReX_COMM_PROFILE

Build with comm-profiling support

NO

YES, NO

AMReX_MEM_PROFILE

Build with memory-profiling support

NO

YES, NO

AMReX_TP_PROFILE

Third-party profiling options

IGNORE

CRAYPAT,FORGE,VTUNE

AMReX_TESTING

Build for testing –sets MultiFab initial data to NaN

NO

YES, NO

AMReX_MPI_THREAD_MULTIPLE

Concurrent MPI calls from multiple threads

NO

YES, NO

AMReX_PROFPARSER

Build with profile parser support

NO

YES, NO

AMReX_ROCTX

Build with roctx markup profiling support

NO

YES, NO

AMReX_FPE

Build with Floating Point Exceptions checks

NO

YES, NO

AMReX_ASSERTIONS

Build with assertions turned on

NO

YES, NO

AMReX_BOUND_CHECK

Enable bound checking in Array4 class

NO

YES, NO

AMReX_EXPORT_DYNAMIC

Enable backtrace on macOS

NO (unless Darwin)

YES, NO

AMReX_SENSEI

Enable the SENSEI in situ infrastructure

NO

YES, NO

AMReX_NO_SENSEI_AMR_INST

Disables the instrumentation in amrex::Amr

NO

YES, NO

AMReX_CONDUIT

Enable Conduit support

NO

YES, NO

AMReX_CATALYST

Enable Catalyst support

NO

YES, NO

AMReX_ASCENT

Enable Ascent support

NO

YES, NO

AMReX_HYPRE

Enable HYPRE interfaces

NO

YES, NO

AMReX_PETSC

Enable PETSc interfaces

NO

YES, NO

AMReX_SUNDIALS

Enable SUNDIALS interfaces

NO

YES, NO

AMReX_HDF5

Enable HDF5-based I/O

NO

YES, NO

AMReX_HDF5_ZFP

Enable compression with ZFP in HDF5-based I/O

NO

YES, NO

AMReX_PLOTFILE_TOOLS

Build and install plotfile postprocessing tools

NO

YES, NO

AMReX_ENABLE_TESTS

Enable CTest suite

NO

YES, NO

AMReX_TEST_TYPE

Test type – affects the number of tests

All

All, Small

AMReX_DIFFERENT_COMPILER

Allow an app to use a different compiler

NO

YES, NO

AMReX_INSTALL

Generate Install Targets

YES

YES, NO

AMReX_PROBINIT

Enable support for probin file

Platform dependent

YES, NO

AMReX_FLATTEN_FOR

Enable flattening of ParallelFor and similar functions for host code

NO

YES, NO

AMReX_COMPILER_DEFAULT_INLINE

Use default inline behavior of compiler, so far relevant for GCC Only

NO for GCC YES otherwise

YES, NO

AMReX_INLINE_LIMIT

Inline limit. Relevant only when AMReX_COMPILER_DEFAULT_INLINE is NO.

43210

Non-negative number

The option CMAKE_BUILD_TYPE=Debug implies AMReX_ASSERTIONS=YES. In order to turn off assertions in debug mode, AMReX_ASSERTIONS=NO must be set explicitly while invoking CMake.

The CMAKE_C_COMPILER, CMAKE_CXX_COMPILER, and CMAKE_Fortran_COMPILER options are used to tell CMake which compiler to use for the compilation of C, C++, and Fortran sources respectively. If those options are not set by the user, CMake will use the system default compilers.

The options CMAKE_Fortran_FLAGS and CMAKE_CXX_FLAGS allow the user to set their own compilation flags for Fortran and C++ source files respectively. If CMAKE_Fortran_FLAGS/ CMAKE_CXX_FLAGS are not set by the user, they will be initialized with the value of the environmental variables FFLAGS/ CXXFLAGS. If neither FFLAGS/ CXXFLAGS nor CMAKE_Fortran_FLAGS/ CMAKE_CXX_FLAGS are defined, AMReX default flags are used.

For a detailed explanation of GPU support in AMReX CMake, refer to section Building GPU Support.

CMake and macOS

While not strictly necessary when using homebrew on macOS, it is highly recommended that the user specifies -DCMAKE_C_COMPILER=$(which gcc-X) -DCMAKE_CXX_COMPILER=$(which g++-X) (where X is the GCC version installed by homebrew) when using gfortran. This is because homebrew’s CMake defaults to the Clang C/C++ compiler. Normally Clang plays well with gfortran, but if there are some issues, we recommend telling CMake to use gcc for C/C++ also.

Importing AMReX into your CMake project

In order to import AMReX into your CMake project, you need to include the following line in the appropriate CMakeLists.txt file:

find_package(AMReX)

Calls to find_package(AMReX) will find a valid installation of AMReX, if present, and import its settings and targets into your CMake project. Imported AMReX targets can be linked to any of your targets, after they have been made available following a successful call to find_package(AMReX), by including the following line in the appropriate CMakeLists.txt file:

target_link_libraries( <your-target-name> PUBLIC AMReX::<amrex-target-name> )

In the above snippet, <amrex-target-name> is any of the targets listed in the table below.

Table 3 AMReX targets available for import.

Target name

Description

amrex_1d

AMReX library in 1D

amrex_2d

AMReX library in 2D

amrex_3d

AMReX library in 3D

amrex

AMReX library (alias, points to last dim)

Flags_CXX

C++ flags preset (interface)

Flags_Fortran

Fortran flags preset (interface)

Flags_FPE

Floating Point Exception flags (interface)

The options used to configure the AMReX build may result in certain parts, or components, of the AMReX source code to be excluded from compilation. For example, setting -DAMReX_LINEAR_SOLVERS=no at configure time prevents the compilation of AMReX linear solvers code. Your CMake project can check which component is included in the AMReX library via find_package:

find_package(AMReX REQUIRED <components-list>)

The keyword REQUIRED in the snippet above will cause a fatal error if AMReX is not found, or if it is found but the components listed in <components-list> are not include in the installation. A list of AMReX component names and related configure options are shown in the table below.

Table 4 AMReX components.

Option

Component

AMReX_SPACEDIM

1D, 2D, 3D

AMReX_PRECISION

DOUBLE, SINGLE

AMReX_FORTRAN

FORTRAN

AMReX_PIC

PIC

AMReX_MPI

MPI

AMReX_OMP

OMP

AMReX_GPU_BACKEND

CUDA, HIP, SYCL

AMReX_FORTRAN_INTERFACES

FINTERFACES

AMReX_LINEAR_SOLVERS

LSOLVERS

AMReX_AMRDATA

AMRDATA

AMReX_AMRLEVEL

AMRLEVEL

AMReX_EB

EB

AMReX_PARTICLES

PARTICLES

AMReX_PARTICLES_PRECISION

PDOUBLE, PSINGLE

AMReX_BASE_PROFILE

BASEP

AMReX_TINY_PROFILE

TINYP

AMReX_TRACE_PROFILE

TRACEP

AMReX_COMM_PROFILE

COMMP

AMReX_MEM_PROFILE

MEMP

AMReX_PROFPARSER

PROFPARSER

AMReX_FPE

FPE

AMReX_ASSERTIONS

ASSERTIONS

AMReX_SENSEI

SENSEI

AMReX_CONDUIT

CONDUIT

AMReX_ASCENT

ASCENT

AMReX_HYPRE

HYPRE

AMReX_PLOTFILE_TOOLS

PFTOOLS

As an example, consider the following CMake code:

find_package(AMReX REQUIRED 3D EB)
target_link_libraries(Foo PUBLIC AMReX::amrex_3d)

The code in the snippet above checks whether an AMReX installation with 3D and Embedded Boundary support is available on the system. If so, AMReX is linked to target Foo and AMReX flags preset is used to compile Foo’s C++ sources. If no AMReX installation is found or if the available one was built without 3D or Embedded Boundary support, a fatal error is issued.

You can tell CMake to look for the AMReX library in non-standard paths by setting the environmental variable AMReX_ROOT to point to the AMReX installation directory or by adding -DAMReX_ROOT=<path/to/amrex/installation/directory> to the cmake invocation. More details on find_package can be found here.

AMReX on Windows

The AMReX team does development on Linux machines, from laptops to supercomputers. Many people also use AMReX on Macs without issues.

We do not officially support AMReX on Windows, and many of us do not have access to any Windows machines. However, we believe there are no fundamental issues for it to work on Windows.

(1) AMReX mostly uses standard C++17. We run continuous integration tests on Windows with MSVC and Clang compilers.

(2) We use POSIX signal handling when floating point exceptions, segmentation faults, etc. happen. This capability is not supported on Windows.

(3) Memory profiling is an optional feature in AMReX that is not enabled by default. It reads memory system information from the OS to give us a summary of our memory usage. This is not supported on Windows.

Spack

AMReX can be installed using the scientific software package manager Spack. Spack supports multiple versions and configurations of AMReX across a wide variety of platforms and environments. To learn more about Spack visit http://www.spack.io. For system requirements and installation instructions please see https://spack.readthedocs.io/.

Once Spack has been downloaded and the Spack environment enabled, AMReX can be installed with the command,

spack install amrex

This will install the latest release of AMReX and required dependencies if needed.

AMReX can be built in several combinations of versions and configurations. Available options can be viewed by typing,

spack info amrex

For example, suppose we want to install the development version of AMReX for a two dimensional simulation with Cuda support for Cuda Architecture sm_60. Then we would use the install commands,

spack install amrex@develop dimensions=2 +cuda cuda_arch=60