![]() |
Block-Structured AMR Software Framework
|
Calculates the distribution of FABs to MPI processes. More...
#include <AMReX_DistributionMapping.H>
Classes | |
| struct | Ref |
| struct | RefID |
| Lightweight identity handle wrapping a pointer to the underlying Ref. More... | |
Public Types | |
| enum | Strategy { UNDEFINED = -1 , ROUNDROBIN , KNAPSACK , SFC , RRSFC } |
| The distribution strategies. More... | |
Public Member Functions | |
| DistributionMapping () noexcept | |
| The default constructor. | |
| DistributionMapping (const DistributionMapping &rhs)=default | |
| The copy constructor. | |
| DistributionMapping & | operator= (const DistributionMapping &rhs)=default |
| The copy assignment operator. | |
| DistributionMapping (DistributionMapping &&rhs) noexcept=default | |
| The move constructor. | |
| DistributionMapping & | operator= (DistributionMapping &&rhs) noexcept=default |
| The move assignment operator. | |
| ~DistributionMapping () noexcept=default | |
| The destructor. | |
| DistributionMapping (const Vector< int > &pmap) | |
| Create an object with an explicit processor map. | |
| DistributionMapping (Vector< int > &&pmap) noexcept | |
| Create an object with an explicit processor map (rvalue overload). | |
| DistributionMapping (const BoxArray &boxes, int nprocs=ParallelDescriptor::NProcs()) | |
| Build a mapping from a BoxArray using the current strategy. | |
| DistributionMapping (std::shared_ptr< Ref > a_ref) | |
Construct from an existing shared Ref handle a_ref (advanced use). | |
| DistributionMapping (const DistributionMapping &d1, const DistributionMapping &d2) | |
| Construct by concatenating the processor maps from two existing mappings. | |
| void | define (const BoxArray &boxes, int nprocs=ParallelDescriptor::NProcs()) |
| Build a mapping from a BoxArray using the current strategy. | |
| void | define (const Vector< int > &pmap) |
| Build mapping out of a Vector of ints. | |
| void | define (Vector< int > &&pmap) noexcept |
| Build mapping out of a Vector of ints (rvalue overload). | |
| const Vector< int > & | ProcessorMap () const noexcept |
| Returns a constant reference to the mapping of boxes in the underlying BoxArray to the CPU that holds the FAB on that Box. | |
| Long | size () const noexcept |
| Length of the underlying processor map. | |
| Long | capacity () const noexcept |
| Reserved capacity of the underlying processor map. | |
| bool | empty () const noexcept |
Return true if the processor map is empty. | |
| Long | linkCount () const noexcept |
| Number of references to this DistributionMapping. | |
| int | operator[] (int index) const noexcept |
| Equivalent to ProcessorMap()[index]. | |
| std::istream & | readFrom (std::istream &is) |
Read the distribution mapping from formatted text stream is. | |
| std::ostream & | writeOn (std::ostream &os) const |
Write the distribution mapping to formatted text stream os. | |
| bool | operator== (const DistributionMapping &rhs) const noexcept |
| Are the distributions equal? | |
| bool | operator!= (const DistributionMapping &rhs) const noexcept |
| Are the distributions different? | |
| void | SFCProcessorMap (const BoxArray &boxes, const std::vector< Long > &wgts, int nprocs, bool sort=true) |
Build an SFC-ordered map for boxes with per-box weights wgts. | |
| void | SFCProcessorMap (const BoxArray &boxes, const std::vector< Long > &wgts, int nprocs, Real &efficiency, bool sort=true) |
| Build an SFC-ordered map and report the resulting efficiency. | |
| void | KnapSackProcessorMap (const std::vector< Long > &wgts, int nprocs, Real *efficiency=nullptr, bool do_full_knapsack=true, int nmax=std::numeric_limits< int >::max(), bool sort=true) |
| Build a knapsack map from a weight vector. | |
| void | KnapSackProcessorMap (const DistributionMapping &olddm, const std::vector< Long > &wgts, Real keep_ratio, Real &old_efficiency, Real &new_efficiency, int nmax=std::numeric_limits< int >::max()) |
Incrementally re-balance using the knapsack algorithm, retaining boxes on their original ranks up to a fraction keep_ratio of the average per-rank weight before redistributing the remainder. | |
| void | RoundRobinProcessorMap (int nboxes, int nprocs, bool sort=true) |
Build a round-robin map for nboxes boxes. | |
| void | RoundRobinProcessorMap (const std::vector< Long > &wgts, int nprocs, bool sort=true) |
| Build a round-robin map from a weight vector. | |
| std::weak_ptr< Ref > | getWeakRef () const |
| Return a weak pointer to the underlying Ref (for low-level identity tracking). | |
| RefID | getRefID () const noexcept |
| This gives a unique ID of the reference, which is different from dmID above. | |
Static Public Member Functions | |
| static void | strategy (Strategy how) |
| Set the default DistributionMapping strategy used by the constructors. | |
| static Strategy | strategy () |
| Return the current default DistributionMapping strategy. | |
| static void | SFC_Threshold (int n) |
Set the SFC threshold n (clamped to >= 1); SFC ordering is used only when boxes.size() >= n*nprocs. | |
| static int | SFC_Threshold () |
| Return the SFC threshold; the SFC constructors fall back to knapsack when boxes.size() < threshold*nprocs. | |
| static void | Initialize () |
| Initializes distribution strategy from ParmParse. | |
| static void | Finalize () |
| Finalize the DistributionMapping subsystem. | |
| static bool | SameRefs (const DistributionMapping &lhs, const DistributionMapping &rhs) |
Return true if lhs and rhs share the same underlying Ref. | |
| static DistributionMapping | makeKnapSack (const MultiFab &weight, int nmax=std::numeric_limits< int >::max()) |
Build a knapsack mapping weighted by the sum of component 0 over each valid box of weight. | |
| static DistributionMapping | makeKnapSack (const MultiFab &weight, Real &eff, int nmax=std::numeric_limits< int >::max()) |
Build a knapsack mapping weighted by the sum of component 0 over each valid box of weight and report efficiency. | |
| static DistributionMapping | makeKnapSack (const Vector< Real > &rcost, int nmax=std::numeric_limits< int >::max()) |
| Build a knapsack mapping from an explicit cost vector. | |
| static DistributionMapping | makeKnapSack (const Vector< Real > &rcost, Real &eff, int nmax=std::numeric_limits< int >::max(), bool sort=true) |
Build a knapsack mapping from rcost and report efficiency. | |
| static DistributionMapping | makeKnapSack (const LayoutData< Real > &rcost_local, Real ¤tEfficiency, Real &proposedEfficiency, int nmax=std::numeric_limits< int >::max(), bool broadcastToAll=true, int root=ParallelDescriptor::IOProcessorNumber(), Real keep_ratio=Real(0.0)) |
Computes a new distribution mapping by distributing input costs according to the knapsack algorithm. | |
| static DistributionMapping | makeRoundRobin (const MultiFab &weight) |
Build a round-robin map weighted by the sum of component 0 over each valid box of weight. | |
| static DistributionMapping | makeSFC (const MultiFab &weight, bool sort=true) |
Build an SFC map weighted by the sum of component 0 over each valid box of weight; sort enables load-aware ordering. | |
| static DistributionMapping | makeSFC (const MultiFab &weight, Real &eff, bool sort=true) |
Build an SFC map weighted by the sum of component 0 over each valid box of weight, write efficiency to eff, and optionally use load-aware ordering. | |
| static DistributionMapping | makeSFC (const Vector< Real > &rcost, const BoxArray &ba, bool sort=true) |
Build an SFC map from rcost and ba; sort enables load-aware bucket and rank/team ordering. | |
| static DistributionMapping | makeSFC (const Vector< Real > &rcost, const BoxArray &ba, Real &eff, bool sort=true) |
Build an SFC map from rcost and ba, write efficiency to eff, and optionally use load-aware ordering. | |
| static DistributionMapping | makeSFC (const LayoutData< Real > &rcost_local, Real ¤tEfficiency, Real &proposedEfficiency, bool broadcastToAll=true, int root=ParallelDescriptor::IOProcessorNumber()) |
Computes a new distribution mapping by distributing input costs according to a space filling curve (SFC) algorithm. | |
| static std::vector< std::vector< int > > | makeSFC (const BoxArray &ba, bool use_box_vol=true, int nprocs=ParallelContext::NProcsSub()) |
| template<typename T > | |
| static void | ComputeDistributionMappingEfficiency (const DistributionMapping &dm, const std::vector< T > &cost, Real *efficiency) |
| Computes the average cost per MPI rank given a distribution mapping global cost vector. | |
| static Vector< Long > | ConvertCostRealToLong (const Vector< Real > &rcost) |
Friends | |
| template<typename T > | |
| class | FabArray |
| class | FabArrayBase |
Calculates the distribution of FABs to MPI processes.
This class calculates the distribution of FABs to MPI processes in a FabArray in a multi-processor environment. By distribution is meant what MPI process in the multi-processor environment owns what FAB. Only the BoxArray on which the FabArray is built is used in determining the distribution. The four distribution strategies are round-robin, knapsack, SFC, and RRSFC. In the round-robin distribution, boxes are ordered from largest to smallest and assigned cyclically to ranks or teams, starting with those using the least FAB memory. In the knapsack distribution the FABs are partitioned across CPUs such that the total volume of the Boxes in the underlying BoxArray are as equal across CPUs as is possible. The SFC distribution is based on a space filling curve. The RRSFC distribution orders boxes using an SFC and then assigns them in round-robin order to CPUs ordered from least to most FAB memory use.
|
noexcept |
The default constructor.
|
default |
The copy constructor.
|
defaultnoexcept |
The move constructor.
|
defaultnoexcept |
The destructor.
Create an object with an explicit processor map.
| pmap | Processor id for each box in the underlying BoxArray. |
Create an object with an explicit processor map (rvalue overload).
| pmap | Processor id for each box in the underlying BoxArray. |
|
explicit |
Build a mapping from a BoxArray using the current strategy.
| boxes | Boxes to distribute. |
| nprocs | Number of processes requested for the mapping. |
|
explicit |
Construct from an existing shared Ref handle a_ref (advanced use).
| amrex::DistributionMapping::DistributionMapping | ( | const DistributionMapping & | d1, |
| const DistributionMapping & | d2 | ||
| ) |
Construct by concatenating the processor maps from two existing mappings.
This is a very specialized constructor. The resulting map is only valid for a FabArray whose BoxArray is the corresponding concatenation of those of d1 and d2.
| d1 | Mapping whose processor ids are inserted first. |
| d2 | Mapping whose processor ids are appended to the result. |
|
inlinenoexcept |
Reserved capacity of the underlying processor map.
|
static |
Computes the average cost per MPI rank given a distribution mapping global cost vector.
| [in] | dm | distribution mapping (mapping from FAB to MPI processes) |
| [in] | cost | vector giving mapping from FAB to the corresponding cost |
| [in,out] | efficiency | average cost per MPI process, as computed from the given distribution mapping and cost |
|
static |
Convert cost vector from Real to Long. Other than convenience, this also provides a consistent way for the conversion.
| void amrex::DistributionMapping::define | ( | const BoxArray & | boxes, |
| int | nprocs = ParallelDescriptor::NProcs() |
||
| ) |
Build a mapping from a BoxArray using the current strategy.
This overload can be used to initialize a default-constructed DistributionMapping.
| boxes | Boxes to distribute. |
| nprocs | Number of processes requested for the mapping. |
Build mapping out of a Vector of ints.
This overload can be used to initialize a default-constructed DistributionMapping.
Build mapping out of a Vector of ints (rvalue overload).
This overload can be used to initialize a default-constructed DistributionMapping.
|
inlinenoexcept |
Return true if the processor map is empty.
|
static |
Finalize the DistributionMapping subsystem.
|
inlinenoexcept |
This gives a unique ID of the reference, which is different from dmID above.
| std::weak_ptr< DMRef > amrex::DistributionMapping::getWeakRef | ( | ) | const |
Return a weak pointer to the underlying Ref (for low-level identity tracking).
|
static |
Initializes distribution strategy from ParmParse.
ParmParse options are:
DistributionMapping.strategy = ROUNDROBIN DistributionMapping.strategy = KNAPSACK DistributionMapping.strategy = SFC # the default DistributionMapping.strategy = RRSFC
| void amrex::DistributionMapping::KnapSackProcessorMap | ( | const DistributionMapping & | olddm, |
| const std::vector< Long > & | wgts, | ||
| Real | keep_ratio, | ||
| Real & | old_efficiency, | ||
| Real & | new_efficiency, | ||
| int | nmax = std::numeric_limits<int>::max() |
||
| ) |
Incrementally re-balance using the knapsack algorithm, retaining boxes on their original ranks up to a fraction keep_ratio of the average per-rank weight before redistributing the remainder.
| olddm | Original mapping. |
| wgts | Per-box weights. |
| keep_ratio | Fraction of the average per-rank weight to retain on each original rank. |
| old_efficiency | Output efficiency of the old mapping. |
| new_efficiency | Output efficiency of the new mapping. |
| nmax | Maximum boxes per rank. |
| void amrex::DistributionMapping::KnapSackProcessorMap | ( | const std::vector< Long > & | wgts, |
| int | nprocs, | ||
| Real * | efficiency = nullptr, |
||
| bool | do_full_knapsack = true, |
||
| int | nmax = std::numeric_limits<int>::max(), |
||
| bool | sort = true |
||
| ) |
Build a knapsack map from a weight vector.
| wgts | Per-box weights (e.g., cell counts). |
| nprocs | Number of processes requested for the mapping. |
| efficiency | Optional load-balance efficiency output (nullptr to skip). |
| do_full_knapsack | When false, use the cheaper approximate solver. |
| nmax | Maximum boxes owned by a single rank. |
| sort | When true, sort the knapsack buckets by weight and map them onto the least-used ranks or teams. |
|
inlinenoexcept |
Number of references to this DistributionMapping.
|
static |
Computes a new distribution mapping by distributing input costs according to the knapsack algorithm.
| [in] | rcost_local | LayoutData of costs; contains, e.g., costs for the local boxes in the FAB array, corresponding indices in the global indices in the FAB array, and the distribution mapping |
| [in,out] | currentEfficiency | On the root process, this is the efficiency (i.e., mean cost over all MPI ranks, normalized to the max cost) given the current distribution mapping |
| [in,out] | proposedEfficiency | On the root process, this is the efficiency for the proposed distribution mapping |
| [in] | nmax | the maximum number of boxes that can be assigned to any MPI rank by the knapsack algorithm |
| [in] | broadcastToAll | controls whether to transmit the proposed distribution mapping to all other processes; setting this to false allows to, e.g., test whether the proposed distribution mapping is an improvement relative to the current distribution mapping, before deciding to broadcast the proposed distribution mapping |
| [in] | root | which process to collect the local costs from others and compute the proposed distribution mapping |
| [in] | keep_ratio | controls the fraction of load that should be kept on the original process. |
|
static |
|
static |
Build a knapsack mapping from an explicit cost vector.
| rcost | Per-box costs. |
| nmax | Maximum boxes per rank. |
|
static |
Build a knapsack mapping from rcost and report efficiency.
| rcost | Per-box costs. |
| eff | Output load-balance efficiency. |
| nmax | Maximum boxes per rank. |
| sort | When true, sort the knapsack buckets by weight and map them onto the least-used ranks or teams. |
|
static |
Build a round-robin map weighted by the sum of component 0 over each valid box of weight.
|
static |
if use_box_vol is true, weight boxes by their volume in Distribute otherwise, all boxes will be treated with equal weight
|
static |
Computes a new distribution mapping by distributing input costs according to a space filling curve (SFC) algorithm.
| [in] | rcost_local | LayoutData of costs; contains, e.g., costs for the local boxes in the FAB array, corresponding indices in the global indices in the FAB array, and the distribution mapping |
| [in,out] | currentEfficiency | On the root process, this is the efficiency (i.e., mean cost over all MPI ranks, normalized to the max cost) given the current distribution mapping |
| [in,out] | proposedEfficiency | On the root process, this is the efficiency for the proposed distribution mapping |
| [in] | broadcastToAll | controls whether to transmit the proposed distribution mapping to all other processes; setting this to false allows to, e.g., test whether the proposed distribution mapping is an improvement relative to the current distribution mapping, before deciding to broadcast the proposed distribution mapping |
| [in] | root | which process to collect the local costs from others and compute the proposed distribution mapping |
|
static |
Build an SFC map weighted by the sum of component 0 over each valid box of weight; sort enables load-aware ordering.
|
static |
Build an SFC map weighted by the sum of component 0 over each valid box of weight, write efficiency to eff, and optionally use load-aware ordering.
|
static |
Build an SFC map from rcost and ba; sort enables load-aware bucket and rank/team ordering.
|
static |
Build an SFC map from rcost and ba, write efficiency to eff, and optionally use load-aware ordering.
|
noexcept |
Are the distributions different?
|
default |
The copy assignment operator.
|
defaultnoexcept |
The move assignment operator.
|
noexcept |
Are the distributions equal?
Equivalent to ProcessorMap()[index].
Returns a constant reference to the mapping of boxes in the underlying BoxArray to the CPU that holds the FAB on that Box.
ProcessorMap()[i] is an integer in the interval [0, NCPU) where NCPU is the number of CPUs being used.
| std::istream & amrex::DistributionMapping::readFrom | ( | std::istream & | is | ) |
Read the distribution mapping from formatted text stream is.
| void amrex::DistributionMapping::RoundRobinProcessorMap | ( | const std::vector< Long > & | wgts, |
| int | nprocs, | ||
| bool | sort = true |
||
| ) |
Build a round-robin map from a weight vector.
| wgts | Per-box weights; the vector length defines nboxes. |
| nprocs | Number of processes requested for the mapping. |
| sort | When true, assign boxes to ranks or teams ordered by increasing existing memory use. Boxes are always ordered by decreasing weight. |
| void amrex::DistributionMapping::RoundRobinProcessorMap | ( | int | nboxes, |
| int | nprocs, | ||
| bool | sort = true |
||
| ) |
Build a round-robin map for nboxes boxes.
| nboxes | Number of boxes. |
| nprocs | Number of processes requested for the mapping. |
| sort | When true, assign boxes to ranks or teams ordered by increasing existing memory use. Box indices remain in input order. |
|
inlinestatic |
Return true if lhs and rhs share the same underlying Ref.
|
static |
Return the SFC threshold; the SFC constructors fall back to knapsack when boxes.size() < threshold*nprocs.
|
static |
Set the SFC threshold n (clamped to >= 1); SFC ordering is used only when boxes.size() >= n*nprocs.
| void amrex::DistributionMapping::SFCProcessorMap | ( | const BoxArray & | boxes, |
| const std::vector< Long > & | wgts, | ||
| int | nprocs, | ||
| bool | sort = true |
||
| ) |
Build an SFC-ordered map for boxes with per-box weights wgts.
| boxes | Boxes to distribute. |
| wgts | Per-box weights (e.g., cell counts). |
| nprocs | Number of processes requested for the mapping. |
| sort | When true, assign heavier SFC buckets to less-used ranks or teams. Boxes are ordered along the SFC regardless of this flag. |
| void amrex::DistributionMapping::SFCProcessorMap | ( | const BoxArray & | boxes, |
| const std::vector< Long > & | wgts, | ||
| int | nprocs, | ||
| Real & | efficiency, | ||
| bool | sort = true |
||
| ) |
Build an SFC-ordered map and report the resulting efficiency.
| boxes | Boxes to distribute. |
| wgts | Per-box weights (e.g., cell counts). |
| nprocs | Number of processes requested for the mapping. |
| efficiency | Output load-balance efficiency in [0,1]. |
| sort | When true, assign heavier SFC buckets to less-used ranks or teams. Boxes are ordered along the SFC regardless of this flag. |
|
inlinenoexcept |
Length of the underlying processor map.
|
static |
Return the current default DistributionMapping strategy.
|
static |
Set the default DistributionMapping strategy used by the constructors.
| std::ostream & amrex::DistributionMapping::writeOn | ( | std::ostream & | os | ) | const |
Write the distribution mapping to formatted text stream os.
|
friend |
|
friend |