Introduction

The regression_testing set of utilities is fully documented in the remainder of this guide. This page offers a brief introduction to the toolkit and its usage.

Overview

regression_testing is a set of regression test management routines for libraries and applications extending the AMReX framework for block-structured AMR (adaptive mesh refinement). It allows for full automation of testing procedures, including compilation and execution of code and analysis routines, as well as interfacing with version control (namely, Git) and generating HTML output for display on a webserver. Additional features such as performance monitoring and assessment of parameter coverage are also available, although the latter is directly dependent on application output and may not be usable in all cases.

Getting Started

Getting the toolkit

The project source may be obtained directly from GitHub via git clone with the following terminal command:

$ git clone https://github.com/AMReX-Codes/regression_testing.git

Running a test suite

The main module is regtest.py, executable from the shell with:

$ ./regtest.py [options] configuration_file

To output a verbose description of usage and setup, including optional arguments, run:

$ ./regtest.py -h

Each test suite is defined in a separate INI formatted configuration file, containing global settings (e.g. test and web output directories, suite name), locations and branches for the source library and any dependencies, and definitions of the test problems with any problem-specific parameters. An example file (example-tests.ini) using the Castro code may be found in the main directory of the project repository.

Before any tests may be performed, the suite must be directed to generate initial benchmarks with the make_benchmarks option:

$ ./regtest.py --make_benchmarks "helpful comment" configuration_file

Quickstart

This section provides a guide describing basic setup.

Using example-tests.ini as a template:

  1. In section [main]:

    1. Set testTopDir to the absolute path to the parent directory of the test suite, under which the benchmark and test output directories will be nested, and set webTopDir to point to the desired root directory for generated webpages.

    2. Configure the build settings if necessary. The sourceTree variable determines the build system (C_Src => C++, AMReX => standalone AMReX tests), while COMP determines the compilers (e.g., g++). The numMakeJobs and add_to_c_make_command parameters allow for some additional control over the make command.

  2. Setting up the repositories:

    Each repository needed to build and run the test problems receives its own section in the configuration file, containing the repository location and the git branch to build from. As the testing routine performs a git pull in each repository at the start of a test run, it is recommended to have local clones of the repositories that may be used exclusively for testing to avoid confusion and potential merge conflicts.

    1. The AMReX repository has a special section in the INI file (aptly designated [AMReX]). The dir parameter here should be set to the absolute path to the repository, and the branch parameter to the remote git branch to pull.

    2. The [source] section corresponds to the repository containing the main application code. The parameters should be set as they were for AMReX.

    3. Any additional repositories needed as dependencies or host test problems should be listed in subsequent sections of the form [extra-<repository_name>].

      These sections each have two additional parameters: comp_string, which contains any environment variables needed by the make system (e.g. comp_string = REPOSITORY_HOME=repository/path), and build, which indicates that the repository contains build directories for test problems if set to 1. The build parameter is optional and is off by default.

  3. Problem setups:

    Each problem is defined in its own section, labeled with the problem name. A number of options are available for test configuration, and are detailed later in the guide - this list only touches on the main parameters necessary for the tests to function.

    1. If the problem build directory is not contained in the main repository (corresponding to [source]), it is necessary to set the extra_build_dir parameter to the section title associated with the correct host repository. Otherwise this may be left blank.

    2. Set the value of buildDir to the relative path from the repository’s root directory to the problem build directory. For example, if the repository is Castro and the build directory is path/to/Castro/Exec/Sod_stellar, buildDir should be set to Exec/Sod_stellar.

    3. Specify the input file to be supplied to the executable using the inputFile parameter, and specify a probin file (with probinFile) if necessary.

    4. Specify any parallelization options for the test problem. The parameters useMPI, useOMP and acc control whether the test uses MPI, OpenMP, and OpenACC respectively. MPI tests also require the number of processors numprocs, and OpenMP tests the number of threads numthreads.

    5. Use the addToCompileString parameter for general variable definitions that should be supplied to the make command. The definitions should be space delimited arguments, of the form VARIABLE_NAME=<value> (e.g. addToCompileString = NET_INPUTS="path/to/network/inputs").

    6. Additional runtime parameters to be supplied to the executable may be specified with the runtime_params variable. The given string will be appended to the command line arguments upon execution.

  4. Generate benchmarks and run the test:

    Once the configuration file is complete, generate benchmarks and run the test suite from the repository’s main directory using the following commands:

    $ ./regtest.py --make_benchmarks "helpful comment" configuration_file
$ ./regtest.py configuration_file