AMR
These tutorials cover block-structured adaptive mesh refinement: filling
ghost cells across levels, synchronizing levels with refluxing and
averaging down, and tagging, regridding and subcycling in time, with
AmrCore and with Amr/AmrLevel. StaticHierarchy and
AdvectionAmrCore run in 2D and 3D, and WaveAmrLevel in 1D, 2D and 3D.
WaveAmrLevel needs AMReX built with AMReX_AMRLEVEL=ON (the default). Build only this topic with -DAMReXTutorials_TOPICS=AMR; see
Getting Started. For complete codes, see CNS on
Amr/AmrLevel and IncFlow on AmrCore.
StaticHierarchy
- Level:
★★
- Source:
- Key APIs:
FillPatchSingleLevel,FillPatchTwoLevels,PhysBCFunct,GpuBndryFuncFab,BCRec,cell_cons_interp,FluxRegister,average_down,WriteMultiLevelPlotfile
The core multilevel operations of AMR on a fixed two-level hierarchy, with
no regridding and no AmrCore. Level 0 covers the unit square/cube, and
level 1 refines its middle half by a factor of 2. A Gaussian diffuses,
\(\partial\phi/\partial t = \nabla\cdot(D\nabla\phi)\), across the
coarse/fine interface. Both levels take the same time step (no subcycling),
limited by the fine cell size.
The walls are zero-gradient (BCType::foextrap), so the total mass must
stay constant. The program checks this and fails if the relative change
exceeds round-off (\(10^{-12}\) in double precision). Run with
do_reflux=0 to see conservation fail.
The physical boundary conditions are described by a BCRec per component
and applied by PhysBCFunct (see also BoundaryConditions).
The FluxRegister is defined on the fine level’s grids:
// Homogeneous Neumann walls: fill ghost cells by first-order extrapolation.
Vector<BCRec> bcs(1);
for (int d = 0; d < AMREX_SPACEDIM; ++d) {
bcs[0].setLo(d, BCType::foextrap);
bcs[0].setHi(d, BCType::foextrap);
}
// GpuBndryFuncFab fills the BCType-based ghost cells; FabFillNoOp is the
// user hook for external Dirichlet (BCType::ext_dir), unused here.
using BndryFunc = GpuBndryFuncFab<FabFillNoOp>;
Vector<PhysBCFunct<BndryFunc>> physbc;
for (int lev = 0; lev < nlevs; ++lev) {
physbc.emplace_back(geom[lev], bcs, BndryFunc(FabFillNoOp{}));
}
// The flux register lives on the fine level's coarse/fine boundary.
FluxRegister flux_reg(grids[1], dmap[1], ratio, 1, 1);
Each time step fills ghost cells, updates both levels, corrects the coarse
cells next to the fine level with the fine fluxes (Reflux), and replaces
the covered coarse cells with the average of the fine cells
(average_down). FillPatchTwoLevels copies from level 1 where
possible and otherwise interpolates conservatively from level 0:
// Level 0 ghost cells: periodic/interior copies plus physical boundaries.
FillPatchSingleLevel(phi_ghost[0], time, {&phi[0]}, {time}, 0, 0, 1,
geom[0], physbc[0], 0);
// Level 1 ghost cells: copy from level 1 where possible, otherwise
// interpolate conservatively from level 0.
FillPatchTwoLevels(phi_ghost[1], time, {&phi[0]}, {time}, {&phi[1]}, {time},
0, 0, 1, geom[0], geom[1], physbc[0], 0, physbc[1], 0,
ratio, &cell_cons_interp, bcs, 0);
for (int lev = 0; lev < nlevs; ++lev) {
compute_flux(phi_ghost[lev], flux[lev], geom[lev], diff_coef);
update(phi[lev], flux[lev], geom[lev], dt);
}
if (do_reflux) {
// Register holds dt * area * (sum of fine fluxes - coarse flux) on
// the coarse faces of the coarse/fine boundary.
for (int d = 0; d < AMREX_SPACEDIM; ++d) {
Real const area_c = AMREX_D_TERM(1.0_rt, *geom[0].CellSize((d+1)%AMREX_SPACEDIM),
*geom[0].CellSize((d+2)%AMREX_SPACEDIM));
Real const area_f = AMREX_D_TERM(1.0_rt, *geom[1].CellSize((d+1)%AMREX_SPACEDIM),
*geom[1].CellSize((d+2)%AMREX_SPACEDIM));
flux_reg.CrseInit(flux[0][d], d, 0, 0, 1, -dt*area_c);
flux_reg.FineAdd (flux[1][d], d, 0, 0, 1, dt*area_f);
}
// Correct the coarse cells next to the fine level.
flux_reg.Reflux(phi[0], 1.0_rt, 0, 0, 1, geom[0]);
}
// Covered coarse cells get the average of the fine cells.
amrex::average_down(phi[1], phi[0], 0, 1, ratio);
Run (from the top of the repository, see Getting Started):
cmake --build build --target AMR_StaticHierarchy -j 8
cd build/AMR/StaticHierarchy
mpiexec -n 2 ./StaticHierarchy inputs
Parameters: n_cell (coarse cells per direction), max_grid_size,
nsteps, cfl, diff_coef, do_reflux, plot (write
plt_static at the end).
Initial mass: 0.005550189972
Final mass: 0.005550189972 after 100 steps, time = 0.003662109375
Relative change in mass: 1.719036495e-15
PASSED
AdvectionAmrCore
- Level:
★★★ / Blueprint
- Source:
- Key APIs:
AmrCore(InitFromScratch,regrid,MakeNewLevelFromScratch,MakeNewLevelFromCoarse,RemakeLevel,ClearLevel,ErrorEst),AMRErrorTag,FillPatchSingleLevel,FillPatchTwoLevels,InterpFromCoarseLevel,FluxRegister,average_down,ParReduce,WriteMultiLevelPlotfile,VisMF,ParmParse::queryAdd
A complete adaptive code built on AmrCore: a scalar \(\phi\) is
advected, \(\partial\phi/\partial t + \nabla\cdot(\mathbf{u}\phi) = 0\),
by a swirling velocity derived from the stream function
\(\psi = \sin^2(\pi x)\sin^2(\pi y)\cos(\pi t/2)/\pi\). The grids
follow \(\phi\) as it moves. The flow reverses at \(t = 1\), so
\(\phi\) returns to its initial position at \(t = 2\). The code
covers tagging, regridding, time stepping with and without subcycling,
refluxing, plotfiles, and checkpoint/restart. It checks that the total mass
stays constant to round-off, which refluxing and conservative interpolation
guarantee. Use it as a starting point for your own AmrCore application.
Code structure
File |
Contents |
|---|---|
|
Defaults for parameters |
|
Level data, |
|
Single-level advance, fluxes, time step estimate |
|
Plotfiles and checkpoint/restart |
|
Initial data, velocity field, limited upwind flux |
To adapt it: change Kernels.H for the physics; add components to
phi_new/phi_old (and the BCRecs) for more variables; use
BCType::ext_dir with a functor in place of FabFillNoOp for inflow
boundaries; add AMRErrorTags, or a custom loop in ErrorEst, for
other refinement criteria.
AmrCore callbacks
AmrCore owns the grids and geometry of all levels and calls back into
the class to create level data. MakeNewLevelFromScratch uses the
initial data, MakeNewLevelFromCoarse interpolates from the level below,
RemakeLevel moves existing data onto new grids, and ErrorEst tags
cells for refinement, here with one AMRErrorTag that tags
\(\phi > \mathtt{adv.phierr[lev]}\):
void
AmrCoreAdv::MakeNewLevelFromCoarse (int lev, Real time, const BoxArray& ba,
const DistributionMapping& dm)
{
DefineLevel(lev, ba, dm);
t_new[lev] = t_old[lev] = time;
FillCoarsePatch(lev, time, phi_new[lev]);
}
void
AmrCoreAdv::RemakeLevel (int lev, Real time, const BoxArray& ba,
const DistributionMapping& dm)
{
MultiFab new_state(ba, dm, 1, 0);
FillPatch(lev, time, new_state);
DefineLevel(lev, ba, dm);
std::swap(new_state, phi_new[lev]);
t_new[lev] = t_old[lev] = time;
}
void
AmrCoreAdv::ClearLevel (int lev)
{
phi_new[lev].clear();
phi_old[lev].clear();
flux_reg[lev].reset();
}
void
AmrCoreAdv::ErrorEst (int lev, TagBoxArray& tags, Real time, int /*ngrow*/)
{
for (auto const& tag : error_tags) {
tag(tags, &phi_new[lev], TagBox::CLEAR, TagBox::SET, time, lev, Geom(lev));
}
}
Time stepping
With subcycling, TimeStepWithSubcycling advances level lev and then
calls itself ref_ratio times for lev+1. Every amr.regrid_int
steps it regrids the finer levels first. After the fine steps, Reflux
and average_down synchronize the levels. TimeStepNoSubcycling
(adv.do_subcycle=0) regrids, then advances every level with a dt
that is stable on all of them.
// Advance level lev by one step, then its finer levels by nsubsteps steps
// each (recursively), then synchronize lev with lev+1.
void
AmrCoreAdv::TimeStepWithSubcycling (int lev, Real time)
{
// Regrid levels finer than lev every regrid_int steps of lev, unless a
// coarser level has just done it.
if (regrid_int > 0 && lev < max_level && istep[lev] > last_regrid_step[lev]
&& istep[lev] % regrid_int == 0)
{
int const old_finest = finest_level;
regrid(lev, time);
for (int k = lev; k <= finest_level; ++k) {
last_regrid_step[k] = istep[k];
}
for (int k = old_finest+1; k <= finest_level; ++k) {
dt[k] = dt[k-1] / nsubsteps[k];
}
}
AdvanceLevel(lev, time, dt[lev]);
++istep[lev];
if (Verbose()) {
amrex::Print() << " [Level " << lev << " step " << istep[lev] << "] advanced "
<< CountCells(lev) << " cells with dt = " << dt[lev] << "\n";
}
if (lev < finest_level)
{
for (int i = 1; i <= nsubsteps[lev+1]; ++i) {
TimeStepWithSubcycling(lev+1, time + (i-1)*dt[lev+1]);
}
if (do_reflux) {
// Replace the coarse flux with the time and area averaged fine flux
// on the coarse/fine boundary.
flux_reg[lev+1]->Reflux(phi_new[lev], 1.0_rt, 0, 0, 1, geom[lev]);
}
AverageDownTo(lev);
}
}
AdvanceLevel uses Heun’s method (two-stage SSP Runge-Kutta) with upwind
fluxes from MC-limited linear reconstructions. Before each stage,
FillPatch fills the ghost cells; fine-level ghost cells are interpolated
in space and time from the level below. Each level passes
\(\Delta t \cdot \text{area} \cdot F\) to the flux registers it
shares with its coarser and finer neighbors:
// Stage 0: phi_new = phi_old - dt div F(phi_old)
// Stage 1: phi_new = phi_old - dt div (F(phi_old) + F(phi_new))/2, where
// FillPatch at t_new uses the stage 0 result stored in phi_new.
for (int stage = 0; stage < 2; ++stage) {
Real const stage_time = time + stage*dt_lev;
FillPatch(lev, stage_time, Sborder);
ComputeFluxes(lev, Sborder, stage_time, flux, stage == 1);
ParallelFor(phi_new[lev], [=] AMREX_GPU_DEVICE (int b, int i, int j, int k)
{
pnew[b](i,j,k) = pold[b](i,j,k)
- (AMREX_D_TERM( dtdx[0] * (fx[b](i+1,j,k) - fx[b](i,j,k)),
+ dtdx[1] * (fy[b](i,j+1,k) - fy[b](i,j,k)),
+ dtdx[2] * (fz[b](i,j,k+1) - fz[b](i,j,k))));
});
}
if (do_reflux) {
// The registers hold flux * area * dt. The coarse level of an
// interface subtracts its flux, and the fine level adds its fluxes.
for (int idim = 0; idim < AMREX_SPACEDIM; ++idim) {
Real area = 1.0_rt;
for (int d = 0; d < AMREX_SPACEDIM; ++d) {
if (d != idim) { area *= geom[lev].CellSize(d); }
}
if (lev < finest_level) {
flux_reg[lev+1]->CrseInit(flux[idim], idim, 0, 0, 1, -dt_lev*area);
}
if (lev > 0) {
flux_reg[lev]->FineAdd(flux[idim], idim, 0, 0, 1, dt_lev*area);
}
}
}
Running
Run (from the top of the repository, see Getting Started):
cmake --build build --target AMR_AdvectionAmrCore -j 8
cd build/AMR/AdvectionAmrCore
mpiexec -n 2 ./AdvectionAmrCore inputs
The inputs file stops after 10 coarse steps; use max_step=1000 for
the full problem and amr.restart=chk00005 to restart from a checkpoint.
Other useful options are adv.do_subcycle=0, adv.do_reflux=0 (mass
is then not conserved), amr.max_level and amr.n_cell.
Coarse step 1 time = 0.01140770953 dt = 0.01140770953 levels = 3 mass = 0.007850917675
...
Coarse step 10 time = 0.1146053625 dt = 0.01155880142 levels = 3 mass = 0.007850917675
Relative change in mass: 8.83832208e-16
PASSED
WaveAmrLevel
- Level:
★★★
- Source:
- Key APIs:
Amr,AmrLevel,LevelBld,StateDescriptor,AmrLevel::RK,GpuBndryFuncFab,cell_quartic_interp,FourthOrderInterpFromFineToCoarse
An AMR code built on Amr and AmrLevel that solves the scalar wave
equation \(\partial^2 u/\partial t^2 = \nabla^2 u\) as the first-order
system \(\partial u/\partial t = v\), \(\partial v/\partial t =
\nabla^2 u\), with a fourth-order Laplacian and Runge-Kutta time stepping.
The domain is \([0,1]\) in x with \(u = 0\) at \(x = 0\) and
\(1\), and periodic in y and z. The velocity \(v\) starts as a pulse
in the middle, which splits in two, reflects at the walls and comes back,
and the fine level follows the pulses. The solution does not depend on y
and z, so d’Alembert’s formula gives the exact \(v\), which the program
compares with the result.
Amr drives the run and creates one AmrLevelWave object per level
through a LevelBld. variableSetUp defines the state data: two
components, two ghost cells for the stencil, a boundary function for the x
boundaries, and the fourth-order cell_quartic_interp interpolater for
the coarse/fine boundary. This interpolater requires amr.ref_ratio = 2.
The time step uses AMReX’s built-in Runge-Kutta support. AmrLevel::RK
advances the state with RK3 or RK4 (wave.rk_order) and fills the
ghost cells of every stage, including those at the coarse/fine boundary at
the stage times, from the coarse level’s RK stage data. The code only
provides the right-hand side:
AmrLevel::RK(rk_order, State_Type, time, dt, iteration, ncycle,
[&] (int /*stage*/, MultiFab& dSdt, MultiFab const& S,
Real /*t*/, Real /*dtsub*/)
{
computeRHS(dSdt, S);
});
After each step, post_timestep replaces the coarse data under the fine
level with a fourth-order average of the fine data. The code does not
reflux, so the integral of \(v\) is not conserved exactly across
levels.
Run:
cmake --build build --target AMR_WaveAmrLevel -j 8
cd build/AMR/WaveAmrLevel
mpiexec -n 2 ./WaveAmrLevel inputs
Finished 320 level-0 steps at time 1 with 2 levels
Max error of v: 1.063265662e-05
PASSED
Useful options are wave.rk_order, wave.cfl, amr.n_cell,
amr.max_level, amr.v=1 and amr.plot_files_output=1.