Coupling

This tutorial runs separate programs in one MPI job that exchange MultiFab data (MPMD). It requires MPI. Build only this topic with -DAMReXTutorials_TOPICS=Coupling; see Getting Started.

MPMD

Level:

★★★

Source:

Coupling/MPMD

Key APIs:

MPMD::Initialize, MPMD::Finalize, MPMD::Copier, Copier::send, Copier::recv, amrex::Initialize(argc, argv, true, comm); Python: MPMD_Initialize_without_split, initialize_when_MPMD, MPMD_Copier

Two separately built programs run in the same MPI job (multiple program, multiple data) and exchange MultiFab data with amrex::MPMD. The main program (main.cpp) owns the domain \([0, n)^D\), fills component 0 with a Gaussian, sends it, and receives component 1. The agent, in C++ (agent.cpp) or Python with pyAMReX (agent.py), owns the upper half \([n/2, n)^D\) with its own boxes, receives component 0, sets component 1 to 10 times component 0, and sends it back. The main program checks the round trip.

MPMD::Initialize splits MPI_COMM_WORLD into one communicator per program, and AMReX is initialized on it:

// Split MPI_COMM_WORLD into one communicator per program, and run AMReX
// on this program's communicator only.
MPI_Comm comm = MPMD::Initialize(argc, argv);
amrex::Initialize(argc, argv, true, comm);

MPMD::Copier exchanges the BoxArrays and DistributionMappings of the two programs once; send and recv then copy components on the intersection of the two BoxArrays. The programs may use different boxes and numbers of ranks, but must use the same dimension and precision. With Copier(ba, dm, true) on one side and Copier(true) on the other, the second program adopts the BoxArray of the first.

// The Copier exchanges BoxArray and DistributionMapping with the other
// program and builds the communication pattern for the intersection
// of the two BoxArrays. Both programs must construct it.
MPMD::Copier copier(ba, dm);

copier.send(mf, 0, 1); // send component 0
copier.recv(mf, 1, 1); // receive into component 1

The agent matches each send with a recv and vice versa:

MPMD::Copier copier(ba, dm);

copier.recv(mf, 0, 1); // receive component 0

auto const& ma = mf.arrays();
ParallelFor(mf, [=] AMREX_GPU_DEVICE (int bi, int i, int j, int k)
{
    ma[bi](i,j,k,1) = 10._rt*ma[bi](i,j,k,0);
});
Gpu::streamSynchronize();

amrex::Print() << "Agent: received component 0 with min = " << mf.min(0)
               << ", max = " << mf.max(0) << "\n";

copier.send(mf, 1, 1); // send component 1 back

The Python agent splits the communicator with mpi4py:

# Split MPI_COMM_WORLD with mpi4py, and run AMReX on this program's ranks only.
amr.MPMD_Initialize_without_split(sys.argv[1:])
app_comm = MPI.COMM_WORLD.Split(amr.MPMD_AppNum(), amr.MPMD_MyProc())
amr.initialize_when_MPMD(sys.argv[1:], app_comm)

main()  # AMReX objects must be gone before finalize

amr.finalize()
amr.MPMD_Finalize()

Run (from the top of the repository, see Getting Started):

cmake --build build --target Coupling_MPMD_Main Coupling_MPMD_Agent -j 8
cd build/Coupling/MPMD
mpiexec -n 2 ./MPMD_Main inputs : -n 2 ./MPMD_Agent inputs
mpiexec -n 2 ./MPMD_Main inputs : -n 2 python3 agent.py inputs

The Python agent needs pyAMReX built with MPI, and mpi4py; add dim=2 for a 2D build. With -DAMReXTutorials_PYTHON=ON, CTest also runs the Python agent. With GNU Make, build the two programs with make PROG=main and make PROG=agent.

Main: 2 of 4 ranks in MPI_COMM_WORLD
Agent: 2 of 4 ranks in MPI_COMM_WORLD
Agent: received component 0 with min = 1, max = 1.92937572
Main: max error of the round trip = 0
PASSED