Continuous Compilation Testing

As a first line of testing, on every commit to the repository, we verify that we can compile AMReX as a library for a common set of configuration options. This operation is performed through Travis-CI. This layer of testing is deliberately limited, so that it can be run quickly on every commit. For more extensive testing, we rely on the nightly regression results.

Nightly Regression Testing

Each night, we automatically run a suite of tests, both on AMReX itself, and on a most of the major application codes that use it as a framework. We use an in-house test runner script to manage this operation, originally developed by Michael Zingale for the Castro code, and later expanded to other application codes as well. The results for each night are collected and stored on a web page; see https://ccse.lbl.gov/pub/RegressionTesting/ for the latest set of results. The runtime option amrex.abort_on_unused_inputs (0 or 1; default is 0 for false) is useful for making sure that tests always stay up to date with API changes as it will abort the application after the test run if any unused input parameters were detected.

Running the test suite locally

The test suite is mostly used internally by AMReX developers. However, if you are making a pull request to AMReX, it can be useful to run the test suite on your local machine to reduce the likelihood that your changes break some existing functionality. To run the test suite locally, you must first obtain a copy of the test runner source, available on Github here: https://github.com/AMReX-Codes/regression_testing. The test runner requires Python version 2.7 or greater. Additional information on the test suite software can be found at, https://amrex-codes.github.io/regression_testing/.

After obtaining the code, you will need a configuration file that defines which tests to run, which amrex repository to test, which branch to use, etc. A sample configuration file for AMReX is distributed with the amrex source code at amrex/Tools/RegressionTesting/AMReX-tests.ini. You will need to modify a few of the entries to, for example, point the test runner to the clone of amrex on your local machine. Entries you will likely want to change include:

testTopDir = /path/to/test/output # the tests results and benchmarks will stored here
webTopDir  = /path/to/web/output  # a web page with the test results will be written here

to control where the generated output will be written, and

[AMReX]
dir = /path/to/amrex  # the path to the amrex repository you want to test
branch = "development"

to control which repository and branch to test.

The test runner is a Python script and can be invoked like so:

python regtest.py <options> AMReX-tests.ini

Before you can use it, you must first generate a set of “benchmarks” - i.e. known “good” answers to the tests that will be run. If you are testing a pull request, you can generate these by running the script with a recent version of the development branch of AMReX. You can generate the benchmarks like so:

python regtest.py --make_benchmarks 'generating initial benchmarks' AMReX-tests.ini

Once that is finished, you can switch over to the branch you want to test in AMReX-tests.ini, and then re-run the script without the --make_benchmarks option:

python regtest.py AMReX-tests.ini

The script will generate a set of html pages in the directory specified in your AMReX-tests.ini file that you can examine using the browser of your choice.

For a complete set of script options, run

python regtest.py --help

A particularly useful option lets you run just a subset of the complete test suite. To run only one test, you can do:

python regtest.py --single_test <TestName> AMReX-tests.ini

To run an enumerated list of tests, do:

python regtest.py --tests '<TestName1> <TestName2> <TestName3>' AMReX-tests.ini

Adding a new test

New tests can be added to the suite by modifying the AMReX-tests.ini file. The easiest thing to do is start from an existing test and modify it. For example, this entry:

[MLMG_FI_PoisCom]
buildDir = Tests/LinearSolvers/ABecLaplacian_F
inputFile = inputs-rt-poisson-com
dim = 3
restartTest = 0
useMPI = 1
numprocs = 2
useOMP = 1
numthreads = 3
compileTest = 0
doVis = 0
outputFile = plot
testSrcTree = C_Src

defines a test called MLMG_FI_PoisCom by specifying the appropriate build directory, inputs file, and a set of configuration options. The above options are the most commonly changed; for a full list of options, see the example configuration file at https://github.com/AMReX-Codes/regression_testing/blob/main/example-tests.ini.