AMReX

CLICK for animation

We present the IAMReX, an adaptive and parallel solver for particle-resolved simulations on multi-level grids. The fluid equations are solved on the block-structured semi-staggered grids and the particle-fluid interaction is resolved using the multidirect forcing immersed boundary method. The cluster of monodisperse particles case is shown in the movie to demonstrate the capability of the current solver in handing with multiple particles. The source code and testing cases used in this work can be accessed HERE . Input scripts and raw postprocessing data are also available for reproducing all results. The arXiv paper is available HERE .

Image courtesy of Xuzhu Li and Zhengping Zhu, Research Center for Astronomical Computing, Zhejiang Laboratory, Hangzhou 311100, China.

The animation shows a three-dimensional, test bed simulation of a thunderstorm performed using the Energy Research and Forecasting (ERF) codebase. Warm and moist air rises due to buoyancy and condenses to form cloud water (1e-5 kg/(kg of dry air) isocontour of cloud water is shown in white). Cloud water, through the processes of autoconversion and accretion, forms rain water drops (1e-4 kg/(kg of dry air) isocontour of rain is shown in blue) that fall down to the ground as precipitation. Cloud water and rain water also evaporate to form water vapor, which rises and condenses, thus creating a feedback loop that leads to the development of a thunderstorm. This simulation was performed on NVIDIA A100 GPUs on Perlmutter. Details of the governing equations and numerical methods can be found in the ERF documentation.

"ERF: Energy Research and Forecasting", Ann Almgren et.al, The Journal of Open Source Software, 2023.

Video and Image courtesy: Mahesh Natarajan, Aaron Lattanzi and Ann Almgren, LBNL; Eliot Quon, NREL.

CLICK for animation

CLICK for animation

Hose instability during a plasma acceleration event causes large transverse oscillations of the beam (in a corkscrew shape) and deteriorating the quality of the accelerator. The image shows an electron drive beam (light blue, right) propagating in a plasma and perturbing the plasma electrons (dark blue), driving a non-linear plasma wave. In the first period of this plasma wave, a cavity with no plasma electrons is created, where strong electromagnetic fields are used to accelerate a second beam (the image shows the first three cavities). In the movie, a moving window is used to follow the beams near the speed of light.

This simulation was performed on GPU by the AMReX-powered code HiPACE++, providing orders of magnitude speedup compared to CPU counterpart [S. Diederichs et al. “HiPACE++: a portable, 3D quasi-static particle-in-cell code.” Computer Physics Communications 278: 108421 (2022)].

Image courtesy of Maxence Thévenet and Ángel Ferran Pousa, DESY (Hamburg, Germany), using the VisualPIC rendering tool.

A computational technique has been developed to perform compressible flow simulations involving moving boundaries using an embedded boundary approach within the block- structured adaptive mesh refinement (SAMR) framework of AMReX. We leverage the SAMR capability to obtain quantitatively accurate results whilst using robust, second- order finite volume schemes. A conservative, unsplit, cut-cell approach is utilized and a ghost-cell approach is developed for computing the flux on the moving, embedded boundary faces. A third-order least-squares formulation has been developed to compute the wall velocity gradients, and was found to significantly improve the performance of the solver in terms of the quantitative comparison of surface quantities such as the skin friction coefficient. The algorithm was implemented within the Compressible Navier-Stokes (CNS) solver of AMReX. Various test cases are performed to validate the method, and compared with analytical, experimental, and other numerical results in literature. Inviscid and viscous test cases are performed that span a wide regime of flow speeds. The judicious use of adaptive mesh refinement with appropriate refinement criteria to capture the regions of interest leads to well-resolved flow features, and good quantitative comparison is observed with the results available in literature.

"A moving embedded boundary approach for the compressible Navier-Stokes equations in a block-structured adaptive refinement framework", Mahesh Natarajan, Ray Grout, Weiqun Zhang, and Marc Day, Journal of Computational Physics, 11315, 2022.

Image Courtesy: Mahesh Natarajan, Ray Grout, Weiqun Zhang, and Marc Day

CLICK for animation

CLICK for larger animation

Ammonium perchlorate (AP) particles packed in a hydroxyl-terminated polybutadiene (HTPB) binder are a common type of solid composite propellant (SCP) used in solid rocket motors. Deflagration of AP/HTPB and the resulting burn front regression produces a complex burn/unburned interface, often with islands of binder left behind. A phase field model for SCP deflagration was implemented in Alamo. The movie shows the interface regression of a 3D sample of AP/HTPB.

A diffuse interface method for solid-phase modeling of regression behavior in solid composite propellants, B Kanagarajan, JM Quinlan, B Runnels, Combustion and Flame, 2022

Phase field modeling of solid phase AP/HTPB to determine the effect of particle distribution on regression rate, B Kanagarjan, JM Quinlan, B Runnels, AIAA SciTech, San Diego, CA, January 2022

Image courtesy of Brandon Runnels, Solid Mechanics Research Group, University of Colorado, Colorado Springs, CO.

A dynamic and complex breaking wave problem is simulated with the help of the framework AMReX. The movie shows the evolution of the air–water interface obtained from the three-level subcycling case. Grid patches are dynamically refined around the interface as time evolves.

Zeng, Yadong, et al. "A parallel cell-centered adaptive level set framework for efficient simulation of two-phase flows with subcycling and non-subcycling." Journal of Computational Physics 448 (2022): 110740.

Image courtesy of Yadong Zeng and Lian Shen, Fluid Mechanics Lab, University of Minnesota.

CLICK for animation

CLICK for animation

We present an adaptive implementation of the distributed Lagrange multiplier (DLM) immersed boundary (IB) method on multilevel collocated grids for solving single- and multiphase fluid-structure interaction (FSI) problems. The capabilities and robustness of the computational framework are validated against a variety of benchmarking problems. A three-dimensional swimming eel is shown here.

Yadong Zeng, Amneet Pal Singh Bhalla, and Lian Shen. "A subcycling/non- subcycling time advancement scheme-based DLM immersed boundary method framework for solving single and multiphase fluid–structure interaction problems on dynamically adaptive grids." Computers and Fluids 238 (2022): 105358.

Image courtesy of Yadong Zeng and Lian Shen, Fluid Mechanics Lab, University of Minnesota.

Simulation of primary atomization of a liquid (water) jet in a Mach 1.94 supersonic crossflow (air). An all-Mach, compressible multiphase flow solver is developed within the framework of AMReX. The Interface Reconstruction Library (IRL) was used to implement an efficient, discretely conservative, unsplit, geometric volume-of-fluid (VOF) transport scheme. The liquid-gas interface in each mixed cell is represented using piecewise linear interface calculation (PLIC). The movie shows streamwise velocity contours with the 4 level mesh and an isocontour of VOF with the finest level of refinement.

A Robust All-Mach Multiphase Flow Algorithm for High-Fidelity Simulations of Compressible Atomization, M.B. Kuhn and O. Desjardins, ILASS-Americas 30th Annual Conference on Liquid Atomization and Spray Systems, Tempe, AZ, May 2019.

An all-Mach multiphase flow solver using block-structured AMR, M. Natarajan, R. Chiodi, M. Kuhn and O. Desjardins, ILASS-Americas 30th Annual Conference on Liquid Atomization and Spray Systems, Tempe, AZ, May 2019.

Image courtesy M. Natarajan, R. Chiodi, M. B. Kuhn, O. Desjardins, Computational Thermo-Fluids Laboratory, Cornell University.

CLICK to enlarge

CLICK to enlarge

Simulation of detonation propagation and failure by diffraction with HyBurn. The detonation is initiated in a reactant layer bounded by high-temperature products that have a very low acoustic impedance. The simulation used with 6 levels of refinement with a refinement ratio of 256 between the finest and coarsest levels. The detonation is initiated by a series of high-pressure, high-temperature spots. The detonation propagates steadily until it encounters a step change in the height of the reactant layer. The leading shock of the detonation weakens as it diffracts around the step, resulting in a decoupled shock and flame. The movie shows the temperature field and follows the detonation.

Image courtesy Brayden Roque, Hsiao-Chi Li, and Ryan Houim

CLICK to enlarge

CLICK for full-length movie

Lid-driven cavity problem for a Papanastasiou-regularised Bingham fluid with Reynold's number 1000, Bingham number 1 and regularisation parameter 0.0025. The heatmap shows the effective viscosity distribution, while the black contour lines illustrate the location of the surface where the stress magnitude equals a characteristic threshold value. In order to accurately resolve this yield surface, we utilise adaptive mesh refinement with stress-triggered cell tagging in three layers.

• Time-dependent viscoplastic fluid flow simulations in two and three dimensions, K. Sverdrup, N. Nikiforakis and A. Almgren, in preparation, 2018, arXiv:1803.00417

Image courtesy of Knut Sverdrup, Laboratory for Scientific Computing, University of Cambridge.

CLICK to enlarge

CLICK to enlarge

Simulation of laser wakefield acceleration performed with WarpX. The laser pulse propagates from left to right in a uniform plasma. A moving window is used, i.e. the simulation box travels at the speed of light to follow the laser pulse. The central slice of plasma electrons is shown as transparent white dots. A cavity free of plasma electrons forms in the laser wake, where an electron bunch (solid white dots) is accelerated. The colormap shows the longitudinal electric field in the wake. The white box in the center shows the mesh-refined area.

Image courtesy Maxence Thévenet & the WarpX team.

CLICK to enlarge

The video to the right shows the spacetime evolution of two equal mass spinning (Kerr) black holes merging into a single black hole with outward propagating gravitational waves. The evolution was simulated using the Z4c formulation of the Einstein equations wtih 6 levels of AMR in AMReX. The complex equations of motion were generated using the STvAR package, designed for converting symbolic/tensorial forms to executable code for AMReX.

The STvAR package was inspired by Professor Zach Etienne and the NRPy+ project at West Virginia University.

Videos and Images courtesy of Adam Peterson and Don Wilcox.

CLICK to view video