C++17 API

Caution

DO NOT use using namespace redev in your code. This is in general a bad practices that creates potential name conflicts. Always use redev:: explicitly, e.g. redev::Redev.

Types

typedef std::int32_t redev::LO: Local ordinate, used to count items local to a process.

using redev::LOs = std::vector<LO>: Vector of local ordinates.

typedef std::int64_t redev::GO: Global ordinate, used to count items across multiple processes.

using redev::GOs = std::vector<GO>: Vector of global ordinates.

typedef double redev::Real: Floating point values.

using redev::Reals = std::vector<Real>: Vector of floating point values.

typedef std::complex<double> redev::CV: Complex values.

using redev::CVs = std::vector<CV>: Vector of complex values.

Partition

class redev::Partition

The Partition class provides an abstract interface that is used by Redev for sharing partition information among the server and client processes. Instances of the Partition class define an interface (i.e., GetRank(…)) to query which process owns a point in the domain (RCBPtn) or a given geometric model entity (ClassPtn). Defining the cut tree for RCB or the assignment of ranks to geometric model entities is not the purpose/responsiblity of this class or the RCBPtn and ClassPtn derived classes.

Subclassed by redev::ClassPtn, redev::RCBPtn

Public Functions

virtual void Write(adios2::Engine &eng, adios2::IO &io) = 0

Write the partition to the provided ADIOS engine and io.

Parameters

eng – [in] an ADIOS2 Engine opened in write mode
io – [in] the ADIOS2 IO object that created eng

virtual void Read(adios2::Engine &eng, adios2::IO &io) = 0

Read the partition to the provided ADIOS engine/io.

Parameters

eng – [in] an ADIOS2 Engine opened in read mode
io – [in] the ADIOS2 IO object that created eng

virtual void Broadcast(MPI_Comm comm, int root = 0) = 0

Send the partition information from the root rank to all other ranks in comm.

Parameters

comm – [in] MPI communicator containing the ranks that need the partition information
root – [in] the source rank that sends the partition information

ClassPtn

class redev::ClassPtn : public redev::Partition 

The ClassPtn class supports a domain partition defined by the ownership of geometric model entities. The user passes to the constructor the assignment of ranks to geometric model entities. The ‘Class’ in the name is from the ‘classification’ term used to define the association of mesh entities with a geometric model entity. The concepts of classification are described in Section 2.2.2 of the PUMI Users Guide (v2.1) https://www.scorec.rpi.edu/pumi/PUMI.pdf.

Public Types

typedef std::pair<redev::LO, redev::LO> ModelEnt: Pair of integers (dimension, id) that uniquely identify a geometric model entity by their dimension and id. The id for an entity within a dimension is unique.

typedef std::vector<ModelEnt> ModelEntVec: Vector of geometric model entities.

typedef std::map<ModelEnt, redev::LO> ModelEntToRank: Map of geometric model entities to the process that owns them.

Public Functions

ClassPtn()

ClassPtn(MPI_Comm comm, const redev::LOs &ranks, const ModelEntVec &ents)

Create a ClassPtn object from a vector of owning ranks and geometric model entities.

Parameters

comm – [in] MPI communicator containing the ranks that need the partition information
ranks – [in] vector of ranks owning each geometric model entity
ents – [in] vector of geometric model entities

redev::LO GetRank(ModelEnt ent) const

Return the rank owning the given geometric model entity.

Parameters: ent – [in] the geometric model entity

virtual void Write(adios2::Engine &eng, adios2::IO &io)

Write the partition to the provided ADIOS engine and io.

Parameters

eng – [in] an ADIOS2 Engine opened in write mode
io – [in] the ADIOS2 IO object that created eng

virtual void Read(adios2::Engine &eng, adios2::IO &io)

Read the partition to the provided ADIOS engine/io.

Parameters

eng – [in] an ADIOS2 Engine opened in read mode
io – [in] the ADIOS2 IO object that created eng

virtual void Broadcast(MPI_Comm comm, int root = 0)

Send the partition information from the root rank to all other ranks in comm.

Parameters

comm – [in] MPI communicator containing the ranks that need the partition information
root – [in] the source rank that sends the partition information

redev::LOs GetRanks() const: Return the vector of owning ranks for all geometric model entity.

ModelEntVec GetModelEnts() const: Return the vector of all geometric model entities.

RCBPtn

class redev::RCBPtn : public redev::Partition 

The RCBPtn class supports recursive coordinate bisection domain partitions. The user passes to the constructor the definition of the cut tree and the ranks owning each sub-domain. The non-leaf levels of the partition tree have alternating cut dimensions associated with the level starting with ‘x’. Each non-leaf node has a coordinate for the position of the cut along the dimension specified by its level. Non-leaf tree node n with cut dimension d and cut coordinate v along dimension d has a left child that covers the sub-domain d < v and a right child that covers the sub-domain d >= v. ranks: labels the leaf nodes in the partition tree from left to right cuts: Array specifying the coordinates associated with the non-leaf nodes in a breath-first traversal order starting at the root and visiting the child nodes at each level from left to right. The root of the cut tree is stored at index 1 and index 0 is unused. See test_query.cpp for examples.

Public Functions

RCBPtn()

RCBPtn(redev::LO dim)

Create a RCBPtn object with only the dimension specified. The cuts and ranks will be filled in during Redev creation.

Parameters: dim – [in] the dimension of the domain (2=2d,3=3d); defines the length of each cut vector

RCBPtn(redev::LO dim, std::vector<int> &ranks, std::vector<double> &cuts)

Create a RCBPtn object from a vector of owning ranks and cut vectors.

Parameters

dim – [in] the dimension of the domain (2=2d,3=3d); defines the length of each cut vector
ranks – [in] vector of ranks owning each sub-domain in the cut tree
cuts – [in] vector of 2-vectors or 3-vectors, for 2d and 3d domains respectively, defining the cut tree. The 2-vectors (3-vectors) are stored in the vector as (x0,y0(,z0),x1,y1(,z1),…xN-1,yN-1(,zN-1))

redev::LO GetRank(std::array<redev::Real, 3> &pt) const

Return the rank owning the given point.

Parameters: pt – [in] the cartesian point in space. The 3rd value is ignored for 2d domains.

virtual void Write(adios2::Engine &eng, adios2::IO &io)

Write the partition to the provided ADIOS engine and io.

Parameters

eng – [in] an ADIOS2 Engine opened in write mode
io – [in] the ADIOS2 IO object that created eng

virtual void Read(adios2::Engine &eng, adios2::IO &io)

Read the partition to the provided ADIOS engine/io.

Parameters

eng – [in] an ADIOS2 Engine opened in read mode
io – [in] the ADIOS2 IO object that created eng

virtual void Broadcast(MPI_Comm comm, int root = 0)

Send the partition information from the root rank to all other ranks in comm.

Parameters

comm – [in] MPI communicator containing the ranks that need the partition information
root – [in] the source rank that sends the partition information

std::vector<redev::LO> GetRanks() const: Return the vector of owning ranks for each sub-domain of the cut tree.

std::vector<redev::Real> GetCuts() const: Return the vector of cuts defining the cut tree. The 2-vectors (3-vectors) are stored in the vector as (x0,y0(,z0),x1,y1(,z1),…xN-1,yN-1(,zN-1))

Redev

class redev::Redev

The Redev class exercises the Partition class APIs to setup the rendezvous partition on the server/rendezvous processes, communicate the partition to the client processes. It also provides an API to create objects that enable communication with the clients. One instance of Redev can support multiple clients.

Public Functions

Redev(MPI_Comm comm, Partition &ptn, ProcessType processType = ProcessType::Client, bool noClients = false)

Create a Redev object

Parameters

comm – [in] MPI communicator containing the ranks that are part of the server/client
ptn – [in] Partition object defining the redezvous domain partition (see note below)
processProcess – [in] enum for if the server is a client, server
noClients – [in] for testing without any clients present The client processes should pass in an empty Partition object. The server will send the client the needed partition information during the call to CreateAdiosClient.

template<typename T> BidirectionalComm<T> CreateAdiosClient(std::string_view name, adios2::Params params, TransportType transportType = TransportType::BP4)

Create a ADIOS2-based BidirectionalComm between the server and one client

Parameters

name – [in] name for the communication channel, each BidirectionalComm must have a unique name
params – [in] list of ADIOS2 parameters controlling IO and Engine creation, see https://adios2.readthedocs.io/en/latest/engines/engines.html for the list of applicable parameters for the SST and BP4 engines
transportType – [in] by default the BP4 Engine is used, other transport types are available in the TransportType enum

ProcessType GetProcessType() const

BidirectionalComm

template<class T> class redev::BidirectionalComm

A BidirectionalComm is a communicator that can send and receive data If you are on a client rank sending sends data to the server and receiving retrieves data from server. If you are on a server rank sending sends data to the client and receiving retrieves data from client.

Public Functions

inline BidirectionalComm(std::unique_ptr<Communicator<T>> sender_, std::unique_ptr<Communicator<T>> receiver_)

inline void SetOutMessageLayout(LOs &dest, LOs &offsets)

inline InMessageLayout GetInMessageLayout()

inline void Send(T *msgs)

inline std::vector<T> Recv()

Communicator

template<typename T> class redev::Communicator

The Communicator class provides an abstract interface for sending and receiving messages to/from the client and server. TODO: Split Communicator into Send/Recieve Communicators, bidirectional constructed by composition and can perform both send and receive

Subclassed by redev::AdiosComm< T >

Public Functions

virtual void SetOutMessageLayout(LOs &dest, LOs &offsets) = 0

Set the arrangement of data in the messages array so that its segments, defined by the offsets array, are sent to the correct destination ranks, defined by the dest array.

Parameters

dest – [in] array of integers specifying the destination rank for a portion of the msgs array
offsets – [in] array of length |dest|+1 defining the segment of the msgs array (passed to the Send function) being sent to each destination rank. the segment [ msgs[offsets[i]] : msgs[offsets[i+1]] } is sent to rank dest[i]

virtual void Send(T *msgs) = 0

Send the array.

Parameters: msgs – [in] array of data to be sent according to the layout specified with SetOutMessageLayout

virtual std::vector<T> Recv() = 0: Receive an array. Use AdiosComm’s GetInMessageLayout to retreive an instance of the InMessageLayout struct containing the layout of the received array.

virtual InMessageLayout GetInMessageLayout() = 0

virtual ~Communicator() = default

AdiosComm

template<typename T> class redev::AdiosComm : public redev::Communicator<T>

The AdiosComm class implements the Communicator interface to support sending messages between the clients and server via ADIOS2. The BP4 and SST ADIOS2 engines are currently supported. One AdiosComm object is required for each communication link direction. For example, for a client and server to both send and receive messages one AdiosComm for client->server messaging and another AdiosComm for server->client messaging are needed. Redev::BidirectionalComm is a helper class for this use case.

Public Functions

inline AdiosComm(MPI_Comm comm_, int recvRanks_, adios2::Engine &eng_, adios2::IO &io_, std::string name_)

Create an AdiosComm object. Collective across sender and receiver ranks. Calls to the constructor from the sender and receiver ranks must be in the same order (i.e., first creating the client-to-server object then the server-to-client link).

Parameters

comm_ – [in] MPI communicator for sender ranks
recvRanks_ – [in] number of ranks in the receivers MPI communicator
eng_ – [in] ADIOS2 engine for writing on the sender side
io_ – [in] ADIOS2 IO associated with eng_
name_ – [in] unique name among AdiosComm objects

inline ~AdiosComm(): destructor to close the engine

AdiosComm(const AdiosComm &other) = delete: We are explicitly not allowing copy/move constructor/assignment as we don’t know if the ADIOS2 Engine and IO objects can be safely copied/moved.

AdiosComm(AdiosComm &&other) = delete

AdiosComm &operator=(const AdiosComm &other) = delete

AdiosComm &operator=(AdiosComm &&other) = delete

inline virtual void SetOutMessageLayout(LOs &dest_, LOs &offsets_)

Set the arrangement of data in the messages array so that its segments, defined by the offsets array, are sent to the correct destination ranks, defined by the dest array.

Parameters

dest – [in] array of integers specifying the destination rank for a portion of the msgs array
offsets – [in] array of length |dest|+1 defining the segment of the msgs array (passed to the Send function) being sent to each destination rank. the segment [ msgs[offsets[i]] : msgs[offsets[i+1]] } is sent to rank dest[i]

inline virtual void Send(T *msgs)

Send the array.

Parameters: msgs – [in] array of data to be sent according to the layout specified with SetOutMessageLayout

inline virtual std::vector<T> Recv(): Receive an array. Use AdiosComm’s GetInMessageLayout to retreive an instance of the InMessageLayout struct containing the layout of the received array.

inline virtual InMessageLayout GetInMessageLayout()

Return the InMessageLayout object.

Todo:: should return const object

inline void SetVerbose(int lvl)

Control the amount of output from AdiosComm functions. The higher the value the more output is written.

Parameters: lvl – [in] valid values are [0:5] where 0 is silent and 5 is produces the most output

InMessageLayout

struct redev::InMessageLayout

The InMessageLayout struct contains the arrays defining the arrangement of data in the array returned by Communicator::Recv.

Public Members

redev::GOs srcRanks: Array of source ranks sized NumberOfClientRanks*NumberOfServerRanks. Each rank reads the entire array once at the start of a communication round. A communication round is defined as a series of sends and receives using the same message layout.

redev::GOs offset: Array of size NumberOfReceiverRanks+1 that indicates the segment of the messages array each server rank should read. NumberOfReceiverRanks is defined as the number of ranks calling Communicator::Recv.

bool knownSizes: Set to true if Communicator::Recv has been called and the message layout data set; false otherwise.

size_t start: Index into the messages array (returned by Communicator::Recv) where the current process should start reading.

size_t count: Number of items (of the user specified type passed to the template parameter of AdiosComm) that should be read from the messages array (returned by Communicator::Recv).