D R A F T

Document for a Standard Message-Passing Interface

Message Passing Interface Forum

April 20, 1995

This work was supported in part by ARPA and NSF under grant ASC-9310330, the

National Science Foundation Science and Technology Center Cooperative Agreement No. CCR-8809615, and by the Commission of the European Community

through Esprit project P6643.

This is the result of a LaTeX run of a draft of a single chapter of the MPIF Final Report document.

Chapter 1

Point-to-Point Communication

1.1 Introduction

Sending and receiving of messages by processes is the basic MPI communication mechanism. The basic point-to-point communication operations are send and receive. Their use is illustrated in the example below.

#include "mpi.h"
main( argc, argv )
int argc;
char **argv;
{
    char message[20];
    int myrank;
    MPI_Status status;
    MPI_Init( &argc, &argv );
    MPI_Comm_rank( MPI_COMM_WORLD, &myrank );
    if (myrank == 0)    /* code for process zero */
    {
        strcpy(message,"Hello, there");
        MPI_Send(message, strlen(message), MPI_CHAR, 1, 99, MPI_COMM_WORLD);
    }
    else                /* code for process one */
    {
        MPI_Recv(message, 20, MPI_CHAR, 0, 99, MPI_COMM_WORLD, &status);
        printf("received :%s:\n", message);
    }
    MPI_Finalize();
}

In this example, process zero (myrank = 0) sends a message to process one using the send operation MPI SEND. The operation specifies a send buffer in the sender memory from which the message data is taken. In the example above, the send buffer consists of the storage containing the variable message in the memory of process zero. The location, size and type of the send buffer are specified by the first three parameters of the send operation. The message sent will contain the 13 characters of this variable. In addition,

2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

the send operation associates an envelope with the message. This envelope specifies the message destination and contains distinguishing information that can be used by the receive operation to select a particular message. The last three parameters of the send operation specify the envelope for the message sent.

Process one (myrank = 1) receives this message with the receive operation MPI RECV. The message to be received is selected according to the value of its envelope, and the message data is stored into the receive buffer. In the example above, the receive buffer consists of the storage containing the string message in the memory of process one. The first three parameters of the receive operation specify the location, size and type of the receive buffer. The next three parameters are used for selecting the incoming message. The last parameter is used to return information on the message just received.

The next sections describe the blocking send and receive operations. We discuss send, receive, blocking communication semantics, type matching requirements, type conversion in heterogeneous environments, and more general communication modes. Nonblocking communication is addressed next, followed by channel-like constructs and send-receive operations. We then consider general datatypes that allow one to transfer efficiently heterogeneous and noncontiguous data. We conclude with the description of calls for explicit packing and unpacking of messages.

1.2 Blocking Send and Receive Operations

1.2.1 Blocking send

The syntax of the blocking send operation is given below.

MPI SEND(buf, count, datatype, dest, tag, comm)

IN  buf  initial address of send buffer (choice)
IN  count  number of elements in send buffer (nonnegative inte-

ger)

IN  datatype  datatype of each send buffer element (handle)
IN  dest  rank of destination (integer)
IN  tag  message tag (integer)
IN  comm  communicator (handle)

int MPI Send(void* buf, int count, MPI Datatype datatype, int dest,
             int tag, MPI Comm comm)
MPI SEND(BUF, COUNT, DATATYPE, DEST, TAG, COMM, IERROR)
    <type> BUF(*)
    INTEGER COUNT, DATATYPE, DEST, TAG, COMM, IERROR

The blocking semantics of this call are described in Sec. 1.4.

1.2.2 Message data

The send buffer specified by the MPI SEND operation consists of count successive entries of the type indicated by datatype, starting with the entry at address buf. Note that we specify

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

the message length in terms of number of elements, not number of bytes. The former is machine independent and closer to the application level.

The data part of the message consists of a sequence of count values, each of the type indicated by datatype. count may be zero, in which case the data part of the message is empty. The basic datatypes that can be specified for message data values correspond to the basic datatypes of the host language. Possible values of this argument for Fortran and the corresponding Fortran types are listed below.

MPI datatype  Fortran datatype
MPI INTEGER  INTEGER
MPI REAL  REAL
MPI DOUBLE PRECISION DOUBLE PRECISION
MPI COMPLEX  COMPLEX
MPI LOGICAL  LOGICAL

                  MPI CHARACTER  CHARACTER(1)

MPI BYTE MPI PACKED

Possible values for this argument for C and the corresponding C types are listed below.

MPI datatype  C datatype

                 MPI CHAR  signed char
                 MPI SHORT  signed short int
                 MPI INT  signed int
                 MPI LONG  signed long int

MPI UNSIGNED CHAR unsigned char

                 MPI UNSIGNED SHORT unsigned short int
                 MPI UNSIGNED  unsigned int
                 MPI UNSIGNED LONG unsigned long int

MPI FLOAT  float
MPI DOUBLE  double
MPI LONG DOUBLE  long double

MPI BYTE MPI PACKED

The datatypes MPI BYTE and MPI PACKED do not correspond to a Fortran or C datatype. A value of type MPI BYTE consists of a byte (8 binary digits). A byte is uninterpreted and is different from a character. Different machines may have different representations for characters, or may use more than one byte to represent characters. On the other hand, a byte has the same binary value on all machines. The use of the type MPI PACKED is explained in Section 1.13.

MPI requires support of the datatypes listed above, which match the basic datatypes of Fortran 77 and ANSI C. Additional MPI datatypes should be provided if the host language

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

has additional data types: MPI LONG LONG INT, for (64 bit) C integers declared to be of type longlong int; MPI DOUBLE COMPLEX for double precision complex in Fortran declared to be of type DOUBLE PRECISION; MPI REAL2, MPI REAL4 and MPI REAL8 for Fortran reals, declared to be of type REAL*2, REAL*4 and REAL*8, respectively; MPI INTEGER1 MPI INTEGER2 and MPI INTEGER4 for Fortran integers, declared to be of type INTEGER*1, INTEGER*2 and INTEGER*4, respectively; etc.

Rationale. One goal of the design is to allow for MPI to be implemented as a library, with no need for additional preprocessing or compilation. Thus, one cannot assume that a communication call has information on the datatype of variables in the communication buffer; this information must be supplied by an explicit argument. The need for such datatype information will become clear in Section 1.3.2. (End of rationale.)

1.2.3 Message envelope

In addition to the data part, messages carry information that can be used to distinguish messages and selectively receive them. This information consists of a fixed number of fields, which we collectively call the message envelope. These fields are

source destination tag communicator

The message source is implicitly determined by the identity of the message sender. The other fields are specified by arguments in the send operation.

The message destination is specified by the dest argument.

The integer-valued message tag is specified by the tag argument. This integer can be used by the program to distinguish different types of messages. The range of valid tag values is 0,...,UB, where the value of UB is implementation dependent. It can be found by querying the value of the attribute MPI TAG UB, as described in Chapter ??. MPI requires that UB be no less than 32767.

The comm argument specifies the communicator that is used for the send operation. Communicators are explained in Chapter ??; below is a brief summary of their usage.

A communicator specifies the communication context for a communication operation. Each communication context provides a separate “communication universe:” messages are always received within the context they were sent, and messages sent in different contexts do not interfere.

The communicator also specifies the set of processes that share this communication context. This process group is ordered and processes are identified by their rank within this group. Thus, the range of valid values for dest is 0, ... , n-1, where n is the number of processes in the group. (If the communicator is an inter-communicator, then destinations are identified by their rank in the remote group. See Chapter ??.)

A predefined communicator MPI COMM WORLD is provided by MPI. It allows communication with all processes that are accessible after MPI initialization and processes are identified by their rank in the group of MPI COMM WORLD.

Advice to users. Users that are comfortable with the notion of a flat name space for processes, and a single communication context, as offered by most existing communication libraries, need only use the predefined variable MPI COMM WORLD as the

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

comm argument. This will allow communication with all the processes available at initialization time.

Users may define new communicators, as explained in Chapter ??. Communicators provide an important encapsulation mechanism for libraries and modules. They allow modules to have their own disjoint communication universe and their own process numbering scheme. (End of advice to users.)

Advice to implementors. The message envelope would normally be encoded by a fixed-length message header. However, the actual encoding is implementation dependent. Some of the information (e.g., source or destination) may be implicit, and need not be explicitly carried by messages. Also, processes may be identified by relative ranks, or absolute ids, etc. (End of advice to implementors.)

1.2.4 Blocking receive

The syntax of the blocking receive operation is given below.

MPI RECV (buf, count, datatype, source, tag, comm, status)

OUT  buf  initial address of receive buffer (choice)
IN  count  number of elements in receive buffer (integer)
IN  datatype  datatype of each receive buffer element (handle)
IN  source  rank of source (integer)
IN  tag  message tag (integer)
IN  comm  communicator (handle)
OUT  status  status object (Status)

int MPI Recv(void* buf, int count, MPI Datatype datatype, int source,
         int tag, MPI Comm comm, MPI Status *status)
MPI RECV(BUF, COUNT, DATATYPE, SOURCE, TAG, COMM, STATUS, IERROR)
<type> BUF(*)
INTEGER COUNT, DATATYPE, SOURCE, TAG, COMM, STATUS(MPI STATUS SIZE),
IERROR

The blocking semantics of this call are described in Sec. 1.4. The receive buffer consists of the storage containing count consecutive elements of the

type specified by datatype, starting at address buf. The length of the received message must be less than or equal to the length of the receive buffer. An overflow error occurs if all incoming data does not fit, without truncation, into the receive buffer.

If a message that is shorter than the receive buffer arrives, then only those locations corresponding to the (shorter) message are modified.

Advice to users. The MPI PROBE function described in Section 1.8 can be used to receive messages of unknown length. (End of advice to users.)

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

Advice to implementors. Even though no specific behavior is mandated by MPI for erroneous programs, the recommended handling of overflow situations is to return in status information about the source and tag of the incoming message. The receive operation will return an error code. A quality implementation will also ensure that no memory that is outside the receive buffer will ever be overwritten. In the case of a message shorter than the receive buffer, MPI is quite strict in that it allows no modification of the other locations. A more lenient statement would allow for some optimizations but this is not allowed. The implementation must be ready to end a copy into the receiver memory exactly at the end of the receive buffer, even if it is an odd address. (End of advice to implementors.)

The selection of a message by a receive operation is governed by the value of the message envelope. A message can be received by a receive operation if its envelope matches the source, tag and comm values specified by the receive operation. The receiver may specify a wildcard MPI ANY SOURCE value for source, and/or a wildcard MPI ANY TAG value for tag, indicating that any source and/or tag are acceptable. It cannot specify a wildcard value for comm. Thus, a message can be received by a receive operation only if it is addressed to the receiving process, has a matching communicator, has matching source unless source= MPI ANY SOURCE in the pattern, and has a matching tag unless tag= MPI ANY TAG in the pattern.

The message tag is specified by the tag argument of the receive operation. The argument source, if different from MPI ANY SOURCE, is specified as a rank within the process group associated with that same communicator (remote process group, for intercommunicators). Thus, the range of valid values for the source argument is {0,...,n-1}∪{MPI ANY SOURCE}, where n is the number of processes in this group.

Note the asymmetry between send and receive operations: A receive operation may accept messages from an arbitrary sender, on the other hand, a send operation must specify a unique receiver. This matches a “push” communication mechanism, where data transfer is effected by the sender (rather than a “pull” mechanism, where data transfer is effected by the receiver).

Source = destination is allowed, that is, a process can send a message to itself. (However, it is unsafe to do so with the blocking send and receive operations described above, since this may lead to deadlock. See Sec. 1.5.)

Advice to implementors. Message context and other communicator information can be implemented as an additional tag field. It differs from the regular message tag in that wild card matching is not allowed on this field, and that value setting for this field is controlled by communicator manipulation functions. (End of advice to implementors.)

1.2.5 Return status

The source or tag of a received message may not be known if wildcard values were used in the receive operation. The information is returned by the status argument of MPI RECV. The type of status is MPI-defined. Status variables need to be explicitly allocated by the user, that is, they are not system objects.

In C, status is a structure that contains two fields named MPI SOURCE and MPI TAG, and the structure may contain additional fields. Thus, status.MPI SOURCE and status.MPI TAG contain the source and tag, respectively, of the received message.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

In Fortran, status is an array of INTEGERs of size MPI STATUS SIZE. The two constants MPI SOURCE and MPI TAG are the indices of the entries that store the source and tag fields. Thus status(MPI SOURCE) and status(MPI TAG) contain, respectively, the source and the tag of the received message.

The status argument also returns information on the length of the message received. However, this information is not directly available as a field of the status variable and a call to MPI GET COUNT is required to “decode” this information.

MPI GET COUNT(status, datatype, count)

IN  status  return status of receive operation (Status)
IN  datatype  datatype of each receive buffer element (handle)
OUT  count  number of received elements (integer)

int MPI Get count(MPI Status status, MPI Datatype datatype, int *count)
MPI GET COUNT(STATUS, DATATYPE, COUNT, IERROR)
    INTEGER STATUS(MPI STATUS SIZE), DATATYPE, COUNT, IERROR

Returns the number of elements received. (Again, we count elements, not bytes.) The datatype argument should match the argument provided by the receive call that set the status variable. (We shall later see, in Section 1.12.5, that MPI GET COUNT may return, in certain situations, the value MPI UNDEFINED.)

Rationale. Some message passing libraries use INOUT count, tag and source arguments, thus using them both to specify the selection criteria for incoming messages and return the actual envelope values of the received message. The use of a separate status argument prevents errors that are often attached with INOUT argument (e.g., using the MPI ANY TAG constant as the tag in a send). Some libraries use calls that refer implicitly to the “last message received.” This is not thread safe.

The datatype argument is passed to MPI GET COUNT so as to improve performance. A message might be received without counting the number of elements it contains, and the count value is often not needed. Also, this allows the same function to be used after a call to MPI PROBE. (End of rationale.)

All send and receive operations use the buf, count, datatype, source, dest, tag, comm and status arguments in the same way as the blocking MPI SEND and MPI RECV operations described in this section.

1.3 Data type matching and data conversion

1.3.1 Type matching rules

One can think of message transfer as consisting of the following three phases.

1. Data is pulled out of the send buffer and a message is assembled.

2. A message is transferred from sender to receiver.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

3. Data is pulled from the incoming message and disassembled into the receive buffer.

Type matching has to be observed at each of these three phases: The type of each variable in the sender buffer has to match the type specified for that entry by the send operation; the type specified by the send operation has to match the type specified by the receive operation; and the type of each variable in the receive buffer has to match the type specified for that entry by the receive operation. A program that fails to observe these three rules is erroneous.

To define type matching more precisely, we need to deal with two issues: matching of types of the host language with types specified in communication operations; and matching of types at sender and receiver.

The types of a send and receive match (phase two) if both operations use identical names. That is, MPI INTEGER matches MPI INTEGER, MPI REAL matches MPI REAL, and so on. There is one exception to this rule, discussed in Sec. 1.13, the type MPI PACKED can match any other type.

The type of a variable in a host program matches the type specified in the communication operation if the datatype name used by that operation corresponds to the basic type of the host program variable. For example, an entry with type name MPI INTEGER matches a Fortran variable of type INTEGER. A table giving this correspondence for Fortran and C appears in Sec. 1.2.2. There are two exceptions to this last rule: an entry with type name MPI BYTE or MPI PACKED can be used to match any byte of storage (on a byte-addressable machine), irrespective of the datatype of the variable that contains this byte. The type MPI PACKED is used to send data that has been explicitly packed, or receive data that will be explicitly unpacked, see Section 1.13. The type MPI BYTE allows one to transfer the binary value of a byte in memory unchanged.

To summarize, the type matching rules fall into the three categories below.

• Communication of typed values (e.g., with datatype different from MPI BYTE), where the datatypes of the corresponding entries in the sender program, in the send call, in the receive call and in the receiver program must all match.

• Communication of untyped values (e.g., of datatype MPI BYTE), where both sender and receiver use the datatype MPI BYTE. In this case, there are no requirements on the types of the corresponding entries in the sender and the receiver programs, nor is it required that they be the same.

• Communication involving packed data, where MPI PACKED is used.

The following examples illustrate the first two cases.

Example 1.1 Sender and receiver specify matching types.

CALL MPI_COMM_RANK(comm, rank, ierr)
IF(rank.EQ.0) THEN
    CALL MPI_SEND(a(1), 10, MPI_REAL, 1, tag, comm, ierr)
ELSE
    CALL MPI_RECV(b(1), 15, MPI_REAL, 0, tag, comm, status, ierr)
END IF

This code is correct if both a and b are real arrays of size ≥ 10. (In Fortran, it might be correct to use this code even if a or b have size < 10: e.g., when a(1) can be equivalenced to an array with ten reals.)

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

Example 1.2 Sender and receiver do not specify matching types.

CALL MPI_COMM_RANK(comm, rank, ierr)
IF(rank.EQ.0) THEN
    CALL MPI_SEND(a(1), 10, MPI_REAL, 1, tag, comm, ierr)
ELSE
    CALL MPI_RECV(b(1), 40, MPI_BYTE, 0, tag, comm, status, ierr)
END IF

This code is erroneous, since sender and receiver do not provide matching datatype arguments.

Example 1.3 Sender and receiver specify communication of untyped values.

CALL MPI_COMM_RANK(comm, rank, ierr)
IF(rank.EQ.0) THEN
    CALL MPI_SEND(a(1), 40, MPI_BYTE, 1, tag, comm, ierr)
ELSE
    CALL MPI_RECV(b(1), 60, MPI_BYTE, 0, tag, comm, status, ierr)
END IF

This code is correct, irrespective of the type and size of a and b (unless this results in an out of bound memory access).

Advice to users. If a buffer of type MPI BYTE is passed as an argument to MPI SEND, then MPI will send the data stored at contiguous locations, starting from the address indicated by the buf argument. This may have unexpected results when the data layout is not as a casual user would expect it to be. For example, some Fortran compilers implement variables of type CHARACTER as a structure that contains the character length and a pointer to the actual string. In such an environment, sending and receiving a Fortran CHARACTER variable using the MPI BYTE type will not have the anticipated result of transferring the character string. For this reason, the user is advised to use typed communications whenever possible. (End of advice to users.)

Type MPI CHARACTER

The type MPI CHARACTER matches one character of a Fortran variable of type CHARACTER, rather then the entire character string stored in the variable. Fortran variables of type CHARACTER or substrings are transferred as if they were arrays of characters. This is illustrated in the example below.

Example 1.4 Transfer of Fortran CHARACTERs.

CHARACTER*10 a
CHARACTER*10 b
CALL MPI_COMM_RANK(comm, rank, ierr)
IF(rank.EQ.0) THEN
    CALL MPI_SEND(a, 5, MPI_CHARACTER, 1, tag, comm, ierr)
ELSE
    CALL MPI_RECV(b(6:10), 5, MPI_CHARACTER, 0, tag, comm, status, ierr)
END IF

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

The last five characters of string b at process 1 are replaced by the first five characters of string a at process 0.

Rationale. The alternative choice would be for MPI CHARACTER to match a character of arbitrary length. This runs into problems.

A Fortran character variable is a constant length string, with no special termination symbol. There is no fixed convention on how to represent characters, and how to store their length. Some compilers pass a character argument to a routine as a pair of arguments, one holding the address of the string and the other holding the length of string. Consider the case of an MPI communication call that is passed a communication buffer with type defined by a derived datatype (Section 1.12). If this communicator buffer contains variables of type CHARACTER then the information on their length will not be passed to the MPI routine.

This problem forces us to provide explicit information on character length with the MPI call. One could add a length parameter to the type MPI CHARACTER, but this does not add much convenience and the same functionality can be achieved by defining a suitable derived datatype. (End of rationale.)

Advice to implementors. Some compilers pass Fortran CHARACTER arguments as a structure with a length and a pointer to the actual string. In such an environment, the MPI call needs to dereference the pointer in order to reach the string. (End of advice to implementors.)

1.3.2 Data conversion

One of the goals of MPI is to support parallel computations across heterogeneous environments. Communication in a heterogeneous environment may require data conversions. We use the following terminology.

type conversion changes the datatype of a value, e.g., by rounding a REAL to an INTEGER.

representation conversion changes the binary representation of a value, e.g., from Hex floating point to IEEE floating point.

The type matching rules imply that MPI communication never entails type conversion. On the other hand, MPI requires that a representation conversion be performed when a typed value is transferred across environments that use different representations for the datatype of this value. MPI does not specify rules for representation conversion. Such conversion is expected to preserve integer, logical or character values, and to convert a floating point value to the nearest value that can be represented on the target system.

Overflow and underflow exceptions may occur during floating point conversions. Conversion of integers or characters may also lead to exceptions when a value that can be represented in one system cannot be represented in the other system. An exception occurring during representation conversion results in a failure of the communication. An error occurs either in the send operation, or the receive operation, or both.

If a value sent in a message is untyped (i.e., of type MPI BYTE), then the binary representation of the byte stored at the receiver is identical to the binary representation of the byte loaded at the sender. This holds true, whether sender and receiver run in the same or in distinct environments. No representation conversion is required. (Note that

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

representation conversion may occur when values of type MPI CHARACTER or MPI CHAR are transferred, for example, from an EBCDIC encoding to an ASCII encoding.)

No conversion need occur when an MPI program executes in a homogeneous system, where all processes run in the same environment.

Consider the three examples, 1.1–1.3. The first program is correct, assuming that a and b are REAL arrays of size ≥ 10. If the sender and receiver execute in different environments, then the ten real values that are fetched from the send buffer will be converted to the representation for reals on the receiver site before they are stored in the receive buffer. While the number of real elements fetched from the send buffer equal the number of real elements stored in the receive buffer, the number of bytes stored need not equal the number of bytes loaded. For example, the sender may use a four byte representation and the receiver an eight byte representation for reals.

The second program is erroneous, and its behavior is undefined.

The third program is correct. The exact same sequence of forty bytes that were loaded from the send buffer will be stored in the receive buffer, even if sender and receiver run in a different environment. The message sent has exactly the same length (in bytes) and the same binary representation as the message received. If a and b are of different types, or if they are of the same type but different data representations are used, then the bits stored in the receive buffer may encode values that are different from the values they encoded in the send buffer.

Data representation conversion also applies to the envelope of a message: source, destination and tag are all integers that may need to be converted.

Advice to implementors. The current definition does not require messages to carry data type information. Both sender and receiver provide complete data type information. In a heterogeneous environment, one can either use a machine independent encoding such as XDR, or have the receiver convert from the sender representation to its own, or even have the sender do the conversion.

Additional type information might be added to messages in order to allow the system to detect mismatches between datatype at sender and receiver. This might be particularly useful in a slower but safer debug mode. (End of advice to implementors.)

MPI does not require support for inter-language communication. The behavior of a program is undefined if messages are sent by a C process and received by a Fortran process, or vice-versa.

Rationale. MPI does not handle inter-language communication because there are no agreed standards for the correspondence between C types and Fortran types. Therefore, MPI programs that mix languages would not port. (End of rationale.)

Advice to implementors. MPI implementors may want to support inter-language communication by allowing Fortran programs to use “C MPI types,” such as MPI INT, MPI CHAR, etc., and allowing C programs to use Fortran types. (End of advice to implementors.)

1.4 Communication Modes

The send call described in Section 1.2.1 is blocking: it does not return until the message data and envelope have been safely stored away so that the sender is free to access and

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

overwrite the send buffer. The message might be copied directly into the matching receive buffer, or it might be copied into a temporary system buffer.

Message buffering decouples the send and receive operations. A blocking send can complete as soon as the message was buffered, even if no matching receive has been executed by the receiver. On the other hand, message buffering can be expensive, as it entails additional memory-to-memory copying, and it requires the allocation of memory for buffering. MPI offers the choice of several communication modes that allow one to control the choice of the communication protocol.

The send call described in Section 1.2.1 used the standard communication mode. In this mode, it is up to MPI to decide whether outgoing messages will be buffered. MPI may buffer outgoing messages. In such a case, the send call may complete before a matching receive is invoked. On the other hand, buffer space may be unavailable, or MPI may choose not to buffer outgoing messages, for performance reasons. In this case, the send call will not complete until a matching receive has been posted, and the data has been moved to the receiver.

Thus, a send in standard mode can be started whether or not a matching receive has been posted. It may complete before a matching receive is posted. The standard mode send is non-local: successful completion of the send operation may depend on the occurrence of a matching receive.

Rationale. The reluctance of MPI to mandate whether standard sends are buffering or not stems from the desire to achieve portable programs. Since any system will run out of buffer resources as message sizes are increased, and some implementations may want to provide little buffering, MPI takes the position that correct (and therefore, portable) programs do not rely on system buffering in standard mode. Buffering may improve the performance of a correct program, but it doesn’t affect the result of the program. If the user wishes to guarantee a certain amount of buffering, the userprovided buffer system of Sec. 1.6 should be used, along with the buffered-mode send. (End of rationale.)

There are three additional communication modes.

A buffered mode send operation can be started whether or not a matching receive has been posted. It may complete before a matching receive is posted. However, unlike the standard send, this operation is local, and its completion does not depend on the occurrence of a matching receive. Thus, if a send is executed and no matching receive is posted, then MPI must buffer the outgoing message, so as to allow the send call to complete. An error will occur if there is insufficient buffer space. The amount of available buffer space is controlled by the user — see Section 1.6. Buffer allocation by the user may be required for the buffered mode to be effective.

A send that uses the synchronous mode can be started whether or not a matching receive was posted. However, the send will complete successfully only if a matching receive is posted, and the receive operation has started to receive the message sent by the synchronous send. Thus, the completion of a synchronous send not only indicates that the send buffer can be reused, but also indicates that the receiver has reached a certain point in its execution, namely that it has started executing the matching receive. If both sends and receives are blocking operations then the use of the synchronous mode provides synchronous communication semantics: a communication does not complete at either end before both processes rendezvous at the communication. A send executed in this mode is non-local.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

A send that uses the ready communication mode may be started only if the matching receive is already posted. Otherwise, the operation is erroneous and its outcome is undefined. On some systems, this allows the removal of a hand-shake operation that is otherwise required and results in improved performance. The completion of the send operation does not depend on the status of a matching receive, and merely indicates that the send buffer can be reused. A send operation that uses the ready mode has the same semantics as a standard send operation, or a synchronous send operation; it is merely that the sender provides additional information to the system (namely that a matching receive is already posted), that can save some overhead. In a correct program, therefore, a ready send could be replaced by a standard send with no effect on the behavior of the program other than performance.

Three additional send functions are provided for the three additional communication modes. The communication mode is indicated by a one letter prefix: B for buffered, S for synchronous, and R for ready.

MPI BSEND (buf, count, datatype, dest, tag, comm)

IN  buf  initial address of send buffer (choice)
IN  count  number of elements in send buffer (integer)
IN  datatype  datatype of each send buffer element (handle)
IN  dest  rank of destination (integer)
IN  tag  message tag (integer)
IN  comm  communicator (handle)

int MPI Bsend(void* buf, int count, MPI Datatype datatype, int dest,
             int tag, MPI Comm comm)
MPI BSEND(BUF, COUNT, DATATYPE, DEST, TAG, COMM, IERROR)
    <type> BUF(*)
    INTEGER COUNT, DATATYPE, DEST, TAG, COMM, IERROR

Send in buffered mode.

MPI SSEND (buf, count, datatype, dest, tag, comm)

IN  buf  initial address of send buffer (choice)
IN  count  number of elements in send buffer (integer)
IN  datatype  datatype of each send buffer element (handle)
IN  dest  rank of destination (integer)
IN  tag  message tag (integer)
IN  comm  communicator (handle)

int MPI Ssend(void* buf, int count, MPI Datatype datatype, int dest,
             int tag, MPI Comm comm)
MPI SSEND(BUF, COUNT, DATATYPE, DEST, TAG, COMM, IERROR)

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

<type> BUF(*)
INTEGER COUNT, DATATYPE, DEST, TAG, COMM, IERROR

Send in synchronous mode.

MPI RSEND (buf, count, datatype, dest, tag, comm)

IN  buf  initial address of send buffer (choice)
IN  count  number of elements in send buffer (integer)
IN  datatype  datatype of each send buffer element (handle)
IN  dest  rank of destination (integer)
IN  tag  message tag (integer)
IN  comm  communicator (handle)

int MPI Rsend(void* buf, int count, MPI Datatype datatype, int dest,
         int tag, MPI Comm comm)
MPI RSEND(BUF, COUNT, DATATYPE, DEST, TAG, COMM, IERROR)
<type> BUF(*)
INTEGER COUNT, DATATYPE, DEST, TAG, COMM, IERROR

Send in ready mode. There is only one receive operation, which can match any of the send modes. The

receive operation described in the last section is blocking: it returns only after the receive buffer contains the newly received message. A receive can complete before the matching send has completed (of course, it can complete only after the matching send has started).

In a multi-threaded implementation of MPI, the system may de-schedule a thread that is blocked on a send or receive operation, and schedule another thread for execution in the same address space. In such a case it is the user’s responsibility not to access or modify a communication buffer until the communication completes. Otherwise, the outcome of the computation is undefined.

Rationale. We prohibit read accesses to a send buffer while it is being used, even though the send operation is not supposed to alter the content of this buffer. This may seem more stringent than necessary, but the additional restriction causes little loss of functionality and allows better performance on some systems — consider the case where data transfer is done by a DMA engine that is not cache-coherent with the main processor. (End of rationale.)

Advice to implementors. Since a synchronous send cannot complete before a matching receive is posted, one will not normally buffer messages sent by such an operation.

It is recommended to choose buffering over blocking the sender, whenever possible, for standard sends. The programmer can signal his or her preference for blocking the sender until a matching receive occurs by using the synchronous send mode.

A possible communication protocol for the various communication modes is outlined below.

ready send: The message is sent as soon as possible.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

synchronous send: The sender sends a request-to-send message. The receiver stores this request. When a matching receive is posted, the receiver sends back a permissionto-send message, and the sender now sends the message.

standard send: First protocol may be used for short messages, and second protocol for long messages.

buffered send: The sender copies the message into a buffer and then sends it with a nonblocking send (using the same protocol as for standard send).

Additional control messages might be needed for flow control and error recovery. Of course, there are many other possible protocols.

Ready send can be implemented as a standard send. In this case there will be no performance advantage (or disadvantage) for the use of ready send.

A standard send can be implemented as a synchronous send. In such a case, no data buffering is needed. However, many (most?) users expect some buffering.

In a multi-threaded environment, the execution of a blocking communication should block only the executing thread, allowing the thread scheduler to de-schedule this thread and schedule another thread for execution. (End of advice to implementors.)

1.5 Semantics of point-to-point communication

A valid MPI implementation guarantees certain general properties of point-to-point communication, which are described in this section.

Order Messages are non-overtaking: If a sender sends two messages in succession to the same destination, and both match the same receive, then this operation cannot receive the second message if the first one is still pending. If a receiver posts two receives in succession, and both match the same message, then the second receive operation cannot be satisfied by this message, if the first one is still pending. This requirement facilitates matching of sends to receives. It guarantees that message-passing code is deterministic, if processes are single-threaded and the wildcard MPI ANY SOURCE is not used in receives. (Some of the calls described later, such as MPI CANCEL or MPI WAITANY, are additional sources of nondeterminism.)

If a process has a single thread of execution, then any two communications executed by this process are ordered. On the other hand, if the process is multi-threaded, then the semantics of thread execution may not define a relative order between two send operations executed by two distinct threads. The operations are logically concurrent, even if one physically precedes the other. In such a case, the two messages sent can be received in any order. Similarly, if two receive operations that are logically concurrent receive two successively sent messages, then the two messages can match the two receives in either order.

Example 1.5 An example of non-overtaking messages.

CALL MPI_COMM_RANK(comm, rank, ierr)
IF (rank.EQ.0) THEN
    CALL MPI_BSEND(buf1, count, MPI_REAL, 1, tag, comm, ierr)
    CALL MPI_BSEND(buf2, count, MPI_REAL, 1, tag, comm, ierr)

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

ELSE    ! rank.EQ.1
    CALL MPI_RECV(buf1, count, MPI_REAL, 0, MPI_ANY_TAG, comm, status, ierr)
    CALL MPI_RECV(buf2, count, MPI_REAL, 0, tag, comm, status, ierr)
END IF

The message sent by the first send must be received by the first receive, and the message sent by the second send must be received by the second receive.

Progress If a pair of matching send and receives have been initiated on two processes, then at least one of these two operations will complete, independently of other actions in the system: the send operation will complete, unless the receive is satisfied by another message, and completes; the receive operation will complete, unless the message sent is consumed by another matching receive that was posted at the same destination process.

Example 1.6 An example of two, intertwined matching pairs.

CALL MPI_COMM_RANK(comm, rank, ierr)
IF (rank.EQ.0) THEN
    CALL MPI_BSEND(buf1, count, MPI_REAL, 1, tag1, comm, ierr)
    CALL MPI_SSEND(buf2, count, MPI_REAL, 1, tag2, comm, ierr)
ELSE    ! rank.EQ.1
    CALL MPI_RECV(buf1, count, MPI_REAL, 0, tag2, comm, status, ierr)
    CALL MPI_RECV(buf2, count, MPI_REAL, 0, tag1, comm, status, ierr)
END IF

Both processes invoke their first communication call. Since the first send of process zero uses the buffered mode, it must complete, irrespective of the state of process one. Since no matching receive is posted, the message will be copied into buffer space. (If insufficient buffer space is available, then the program will fail.) The second send is then invoked. At that point, a matching pair of send and receive operation is enabled, and both operations must complete. Process one next invokes its second receive call, which will be satisfied by the buffered message. Note that process one received the messages in the reverse order they were sent.

Fairness MPI makes no guarantee of fairness in the handling of communication. Suppose that a send is posted. Then it is possible that the destination process repeatedly posts a receive that matches this send, yet the message is never received, because it is each time overtaken by another message, sent from another source. Similarly, suppose that a receive was posted by a multi-threaded process. Then it is possible that messages that match this receive are repeatedly received, yet the receive is never satisfied, because it is overtaken by other receives posted at this node (by other executing threads). It is the programmer’s responsibility to prevent starvation in such situations.

Resource limitations Any pending communication operation consumes system resources that are limited. Errors may occur when lack of resources prevent the execution of an MPI call. A quality implementation will use a (small) fixed amount of resources for each pending send in the ready or synchronous mode and for each pending receive. However, buffer space may be consumed to store messages sent in standard mode, and must be consumed to store messages sent in buffered mode, when no matching receive is available. The amount of space

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

available for buffering will be much smaller than program data memory on many systems. Then, it will be easy to write programs that overrun available buffer space.

MPI allows the user to provide buffer memory for messages sent in the buffered mode. Furthermore, MPI specifies a detailed operational model for the use of this buffer. An MPI implementation is required to do no worse than implied by this model. This allows users to avoid buffer overflows when they use buffered sends. Buffer allocation and use is described in Section 1.6.

A buffered send operation that cannot complete because of a lack of buffer space is erroneous. When such a situation is detected, an error is signalled that may cause the program to terminate abnormally. On the other hand, a standard send operation that cannot complete because of lack of buffer space will merely block, waiting for buffer space to become available or for a matching receive to be posted. This behavior is preferable in many situations. Consider a situation where a producer repeatedly produces new values and sends them to a consumer. Assume that the producer produces new values faster than the consumer can consume them. If buffered sends are used, then a buffer overflow will result. Additional synchronization has to be added to the program so as to prevent this from occurring. If standard sends are used, then the producer will be automatically throttled, as its send operations will block when buffer space is unavailable.

In some situations, a lack of buffer space leads to deadlock situations. This is illustrated by the examples below.

Example 1.7 An exchange of messages.

CALL MPI_COMM_RANK(comm, rank, ierr)
IF (rank.EQ.0) THEN
    CALL MPI_SEND(sendbuf, count, MPI_REAL, 1, tag, comm, ierr)
    CALL MPI_RECV(recvbuf, count, MPI_REAL, 1, tag, comm, status, ierr)
ELSE    ! rank.EQ.1
    CALL MPI_RECV(recvbuf, count, MPI_REAL, 0, tag, comm, status, ierr)
    CALL MPI_SEND(sendbuf, count, MPI_REAL, 0, tag, comm, ierr)
END IF

This program will succeed even if no buffer space for data is available. The standard send operation can be replaced, in this example, with a synchronous send.

Example 1.8 An attempt to exchange messages.

CALL MPI_COMM_RANK(comm, rank, ierr)
IF (rank.EQ.0) THEN
    CALL MPI_RECV(recvbuf, count, MPI_REAL, 1, tag, comm, status, ierr)
    CALL MPI_SEND(sendbuf, count, MPI_REAL, 1, tag, comm, ierr)
ELSE    ! rank.EQ.1
    CALL MPI_RECV(recvbuf, count, MPI_REAL, 0, tag, comm, status, ierr)
    CALL MPI_SEND(sendbuf, count, MPI_REAL, 0, tag, comm, ierr)
END IF

The receive operation of the first process must complete before its send, and can complete only if the matching send of the second processor is executed. The receive operation of the second process must complete before its send and can complete only if the matching send of the first process is executed. This program will always deadlock. The same holds for any other send mode.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

Example 1.9 An exchange that relies on buffering.

CALL MPI_COMM_RANK(comm, rank, ierr)
IF (rank.EQ.0) THEN
CALL MPI_SEND(sendbuf, count, MPI_REAL, 1, tag, comm, ierr)
CALL MPI_RECV(recvbuf, count, MPI_REAL, 1, tag, comm, status, ierr)
ELSE    ! rank.EQ.1
CALL MPI_SEND(sendbuf, count, MPI_REAL, 0, tag, comm, ierr)
CALL MPI_RECV(recvbuf, count, MPI_REAL, 0, tag, comm, status, ierr)
END IF

The message sent by each process has to be copied out before the send operation returns and the receive operation starts. For the program to complete, it is necessary that at least one of the two messages sent be buffered. Thus, this program can succeed only if the communication system can buffer at least count words of data.

Advice to users. When standard send operations are used, then a deadlock situation may occur where both processes are blocked because buffer space is not available. The same will certainly happen, if the synchronous mode is used. If the buffered mode is used, and not enough buffer space is available, then the program will not complete either. However, rather than a deadlock situation, we shall have a buffer overflow error.

A program is “safe” if no message buffering is required for the program to complete. One can replace all sends in such program with synchronous sends, and the program will still run correctly. This conservative programming style provides the best portability, since program completion does not depend on the amount of buffer space available or in the communication protocol used.

Many programmers prefer to have more leeway and be able to use the “unsafe” programming style shown in example 1.9. In such cases, the use of standard sends is likely to provide the best compromise between performance and robustness: quality implementations will provide sufficient buffering so that “common practice” programs will not deadlock. The buffered send mode can be used for programs that require more buffering, or in situations where the programmer wants more control. This mode might also be used for debugging purposes, as buffer overflow conditions are easier to diagnose than deadlock conditions.

Nonblocking message-passing operations, as described in Section 1.7, can be used to avoid the need for buffering outgoing messages. This prevents deadlocks due to lack of buffer space, and improves performance, by allowing overlap of computation and communication, and avoiding the overheads of allocating buffers and copying messages into buffers. (End of advice to users.)

1.6 Buffer allocation and usage

A user may specify a buffer to be used for buffering messages sent in buffered mode. Buffering is done by the sender.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

MPI BUFFER ATTACH( buffer, size)

IN  buffer  initial buffer address (choice)
IN  size  buffer size, in bytes (integer)

int MPI Buffer attach( void* buffer, int size)
MPI BUFFER ATTACH( BUFFER, SIZE, IERROR)
    <type> BUFFER(*)
    INTEGER SIZE, IERROR

Provides to MPI a buffer in the user’s memory to be used for buffering outgoing messages. The buffer is used only by messages sent in buffered mode. Only one buffer can be attached to a process at a time.

MPI BUFFER DETACH( buffer, size)

OUT  buffer  initial buffer address (choice)
OUT  size  buffer size, in bytes (integer)

int MPI Buffer detach( void** buffer, int* size)
MPI BUFFER DETACH( BUFFER, SIZE, IERROR)
    <type> BUFFER(*)
    INTEGER SIZE, IERROR

Detach the buffer currently associated with MPI. This operation will block until all messages currently in the buffer have been transmitted. Upon return of this function, the user may reuse or deallocate the space taken by the buffer.

The statements made in this section describe the behavior of MPI for buffered-mode sends. When no buffer is currently associated, MPI behaves as if a zero-sized buffer is associated with the process.

MPI must provide as much buffering for outgoing messages as if outgoing message data were buffered by the sending process, in the specified buffer space, using a circular, contiguous-space allocation policy. We outline below a model implementation that defines this policy. MPI may provide more buffering, and may use a better buffer allocation algorithm than described below. On the other hand, MPI may signal an error whenever the simple buffering allocator described below would run out of space. In particular, if no buffer is explicitly associated with the process, then any buffered send may cause an error.

MPI does not provide mechanisms for querying or controlling buffering done by standard mode sends. It is expected that vendors will provide such information for their implementations.

Rationale. There is a wide spectrum of possible implementations of buffered communication: buffering can be done at sender, at receiver, or both; buffers can be dedicated to one sender-receiver pair, or be shared by all communications; buffering can be done in real or in virtual memory; it can use dedicated memory, or memory shared by other processes; buffer space may be allocated statically or be changed dynamically; etc. It does not seem feasible to provide a portable mechanism for querying

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

or controlling buffering that would be compatible with all these choices, yet provide meaningful information. (End of rationale.)

1.6.1 Model implementation of buffered mode

The model implementation uses the packing and unpacking functions described in Section 1.13 and the nonblocking communication functions described in Section 1.7.

We assume that a circular queue of pending message entries (PME) is maintained. Each entry contains a communication request handle that identifies a pending nonblocking send, a pointer to the next entry and the packed message data. The entries are stored in successive locations in the buffer. Free space is available between the queue tail and the queue head.

A buffered send call results in the execution of the following code.

• Traverse sequentially the PME queue from head towards the tail, deleting all entries for communications that have completed, up to the first entry with an uncompleted request; update queue head to point to that entry.

⊤ • Compute the number, n, of bytes needed to store entry for new message An upper bound on n can be computed as follows: A call to the function MPI PACK SIZE(count, datatype, comm, size), with the count, datatype and comm arguments used in the MPI BSEND call, returns an upper bound on the amount of space needed to buffer the message data (see Section 1.13). The MPI constant MPI BSEND OVERHEAD provides an upper bound on the additional space consumed by the entry (e.g., for pointers or

⊥ envelope information).

• Find the next contiguous empty space of n bytes in buffer (space following queue tail, or space at start of buffer if queue tail is too close to end of buffer). If space not found then raise buffer overflow error.

• Append to end of PME queue in contiguous space the new entry that contains request handle, next pointer and packed message data; MPI PACK is used to pack data.

• Post nonblocking send (standard mode) for packed data.

• Return

1.7 Nonblocking communication

One can improve performance on many systems by overlapping communication and computation. This is especially true on systems where communication can be executed autonomously by an intelligent communication controller. Light-weight threads are one mechanism for achieving such overlap. An alternative mechanism that often leads to better performance is to use nonblocking communication. A nonblocking send start call initiates the send operation, but does not complete it. The send start call will return before the message was copied out of the send buffer. A separate send complete call is needed to complete the communication, i.e., to verify that the data has been copied out of the send buffer. With suitable hardware, the transfer of data out of the sender memory may proceed concurrently with computations done at the sender after the send was initiated and before it completed. Similarly, a nonblocking receive start call initiates the receive operation, but

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

does not complete it. The call will return before a message is stored into the receive buffer. A separate receive complete call is needed to complete the receive operation and verify that the data has been received into the receive buffer. With suitable hardware, the transfer of data into the receiver memory may proceed concurrently with computations done after the receive was initiated and before it completed. The use of nonblocking receives may also avoid system buffering and memory-to-memory copying, as information is provided early on the location of the receive buffer.

Nonblocking send start calls can use the same four modes as blocking sends: standard, buffered, synchronous and ready. These carry the same meaning. Sends of all modes, ready excepted, can be started whether a matching receive has been posted or not; a nonblocking ready send can be started only if a matching receive is posted. In all cases, the send start call is local: it returns immediately, irrespective of the status of other processes. If the call causes some system resource to be exhausted, then it will fail and return an error code. Quality implementations of MPI should ensure that this happens only in “pathological” cases. That is, an MPI implementation should be able to support a large number of pending nonblocking operations.

The send-complete call returns when data has been copied out of the send buffer. It may carry additional meaning, depending on the send mode.

If the send mode is synchronous, then the send can complete only if a matching receive has started. That is, a receive has been posted, and has been matched with the send. In this case, the send-complete call is non-local. Note that a synchronous, nonblocking send may complete, if matched by a nonblocking receive, before the receive complete call occurs. (It can complete as soon as the sender “knows” the transfer will complete, but before the receiver “knows” the transfer will complete.)

If the send mode is buffered then the message must be buffered if there is no pending receive. In this case, the send-complete call is local, and must succeed irrespective of the status of a matching receive.

If the send mode is standard then the send-complete call may return before a matching receive occurred, if the message is buffered. On the other hand, the send-complete may not complete until a matching receive occurred, and the message was copied into the receive buffer.

Nonblocking sends can be matched with blocking receives, and vice-versa.

Advice to users. The completion of a send operation may be delayed, for standard mode, and must be delayed, for synchronous mode, until a matching receive is posted. The use of nonblocking sends in these two cases allows the sender to proceed ahead of the receiver, so that the computation is more tolerant of fluctuations in the speeds of the two processes.

Nonblocking sends in the buffered and ready modes have a more limited impact. A nonblocking send will return as soon as possible, whereas a blocking send will return after the data has been copied out of the sender memory. The use of nonblocking sends is advantageous in these cases only if data copying can be concurrent with computation.

The message-passing model implies that communication is initiated by the sender. The communication will generally have lower overhead if a receive is already posted when the sender initiates the communication (data can be moved directly to the receive buffer, and there is no need to queue a pending send request). However, a

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

receive operation can complete only after the matching send has occurred. The use of nonblocking receives allows one to achieve lower communication overheads without blocking the receiver while it waits for the send. (End of advice to users.)

1.7.1 Communication Objects

Nonblocking communications use opaque request objects to identify communication operations and match the operation that initiates the communication with the operation that terminates it. These are system objects that are accessed via a handle. A request object identifies various properties of a communication operation, such as the send mode, the communication buffer that is associated with it, its context, the tag and destination arguments to be used for a send, or the tag and source arguments to be used for a receive. In addition, this object stores information about the status of the pending communication operation.

1.7.2 Communication initiation

We use the same naming conventions as for blocking communication: a prefix of B, S, or R is used for buffered, synchronous or ready mode. In addition a prefix of I (for immediate) indicates that the call is nonblocking.

MPI ISEND(buf, count, datatype, dest, tag, comm, request)

IN  buf  initial address of send buffer (choice)
IN  count  number of elements in send buffer (integer)
IN  datatype  datatype of each send buffer element (handle)
IN  dest  rank of destination (integer)
IN  tag  message tag (integer)
IN  comm  communicator (handle)
OUT  request  communication request (handle)

int MPI Isend(void* buf, int count, MPI Datatype datatype, int dest,
             int tag, MPI Comm comm, MPI Request *request)
MPI ISEND(BUF, COUNT, DATATYPE, DEST, TAG, COMM, REQUEST, IERROR)
    <type> BUF(*)
    INTEGER COUNT, DATATYPE, DEST, TAG, COMM, REQUEST, IERROR

Start a standard mode, nonblocking send.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

MPI IBSEND(buf, count, datatype, dest, tag, comm, request)

IN  buf  initial address of send buffer (choice)
IN  count  number of elements in send buffer (integer)
IN  datatype  datatype of each send buffer element (handle)
IN  dest  rank of destination (integer)
IN  tag  message tag (integer)
IN  comm  communicator (handle)
OUT  request  communication request (handle)

int MPI Ibsend(void* buf, int count, MPI Datatype datatype, int dest,
             int tag, MPI Comm comm, MPI Request *request)
MPI IBSEND(BUF, COUNT, DATATYPE, DEST, TAG, COMM, REQUEST, IERROR)
    <type> BUF(*)
    INTEGER COUNT, DATATYPE, DEST, TAG, COMM, REQUEST, IERROR

Start a buffered mode, nonblocking send.

MPI ISSEND(buf, count, datatype, dest, tag, comm, request)

IN  buf  initial address of send buffer (choice)
IN  count  number of elements in send buffer (integer)
IN  datatype  datatype of each send buffer element (handle)
IN  dest  rank of destination (integer)
IN  tag  message tag (integer)
IN  comm  communicator (handle)
OUT  request  communication request (handle)

int MPI Issend(void* buf, int count, MPI Datatype datatype, int dest,
             int tag, MPI Comm comm, MPI Request *request)
MPI ISSEND(BUF, COUNT, DATATYPE, DEST, TAG, COMM, REQUEST, IERROR)
    <type> BUF(*)
    INTEGER COUNT, DATATYPE, DEST, TAG, COMM, REQUEST, IERROR

Start a synchronous mode, nonblocking send.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

MPI IRSEND(buf, count, datatype, dest, tag, comm, request)

IN  buf  initial address of send buffer (choice)
IN  count  number of elements in send buffer (integer)
IN  datatype  datatype of each send buffer element (handle)
IN  dest  rank of destination (integer)
IN  tag  message tag (integer)
IN  comm  communicator (handle)
OUT  request  communication request (handle)

int MPI Irsend(void* buf, int count, MPI Datatype datatype, int dest,
             int tag, MPI Comm comm, MPI Request *request)
MPI IRSEND(BUF, COUNT, DATATYPE, DEST, TAG, COMM, REQUEST, IERROR)
    <type> BUF(*)
    INTEGER COUNT, DATATYPE, DEST, TAG, COMM, REQUEST, IERROR

Start a ready mode nonblocking send.

MPI IRECV (buf, count, datatype, source, tag, comm, request)

OUT  buf  initial address of receive buffer (choice)
IN  count  number of elements in receive buffer (integer)
IN  datatype  datatype of each receive buffer element (handle)
IN  source  rank of source (integer)
IN  tag  message tag (integer)
IN  comm  communicator (handle)
OUT  request  communication request (handle)

int MPI Irecv(void* buf, int count, MPI Datatype datatype, int source,
             int tag, MPI Comm comm, MPI Request *request)
MPI IRECV(BUF, COUNT, DATATYPE, SOURCE, TAG, COMM, REQUEST, IERROR)
    <type> BUF(*)
    INTEGER COUNT, DATATYPE, SOURCE, TAG, COMM, REQUEST, IERROR

Start a nonblocking receive.

These calls allocate a communication request object and associate it with the request handle (the argument request). The request can be used later to query the status of the communication or wait for its completion.

A nonblocking send call indicates that the system may start copying data out of the send buffer. The sender should not access any part of the send buffer after a nonblocking send operation is called, until the send completes.

A nonblocking receive call indicates that the system may start writing data into the receive buffer. The receiver should not access any part of the receive buffer after a nonblocking receive operation is called, until the receive completes.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

1.7.3 Communication Completion

The functions MPI WAIT and MPI TEST are used to complete a nonblocking communication. The completion of a send operation indicates that the sender is now free to update the locations in the send buffer (the send operation itself leaves the content of the send buffer unchanged). It does not indicate that the message has been received, rather, it may have been buffered by the communication subsystem. However, if a synchronous mode send was used, the completion of the send operation indicates that a matching receive was initiated, and that the message will eventually be received by this matching receive.

The completion of a receive operation indicates that the receive buffer contains the received message, the receiver is now free to access it, and that the status object is set. It does not indicate that the matching send operation has completed (but indicates, of course, that the send was initiated).

We shall use the following terminology: A null handle is a handle with value MPI REQUEST NULL. A persistent request and the handle to it are inactive if the request is not associated with any ongoing communication (see Section 1.9). A handle is active if it is neither null nor inactive. A null status is a status which is set to return tag = ⊤ MPI ANY TAG, source = MPI ANY SOURCE, and is also internally configured so that calls to MPI GET COUNT and MPI GET ELEMENTS return count = 0. We set a status variable to null when the value returned by it is not significant. Status is set in this way so as to prevent errors due to accesses of stale information. ⊥

MPI WAIT(request, status)

INOUT request request (handle)

OUT  status  status object (Status)

int MPI Wait(MPI Request *request, MPI Status *status)
MPI WAIT(REQUEST, STATUS, IERROR)
    INTEGER REQUEST, STATUS(MPI STATUS SIZE), IERROR

A call to MPI WAIT returns when the operation identified by request is complete. If the communication object associated with this request was created by a nonblocking send or receive call, then the object is deallocated by the call to MPI WAIT and the request handle is set to MPI REQUEST NULL. MPI WAIT is a non-local operation.

The call returns, in status, information on the completed operation. The content of the status object for a receive operation can be accessed as described in section 1.2.5. The status object for a send operation may be queried by a call to MPI TEST CANCELLED (see Section 1.8).

One is allowed to call MPI WAIT with a null or inactive request argument. In this case the operation returns immediately with a null status.

Advice to users. Successful return of MPI WAIT after a MPI IBSEND implies that the user send buffer can be reused — i.e., data has been sent out or copied into a buffer attached with MPI BUFFER ATTACH. Note that, at this point, we can no longer cancel the send (see Sec. 1.8). If a matching receive is never posted, then the buffer cannot be freed. This runs somewhat counter to the stated goal of MPI CANCEL

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

(always being able to free program space that was committed to the communication subsystem). (End of advice to users.)

Advice to implementors. In a multi-threaded environment, a call to MPI WAIT should block only the calling thread, allowing the thread scheduler to schedule another thread for execution. (End of advice to implementors.)

MPI TEST(request, flag, status)

INOUT request communication request (handle)

OUT  flag  true if operation completed (logical)
OUT  status  status object (Status)

int MPI Test(MPI Request *request, int *flag, MPI Status *status)
MPI TEST(REQUEST, FLAG, STATUS, IERROR)
LOGICAL FLAG
INTEGER REQUEST, STATUS(MPI STATUS SIZE), IERROR

A call to MPI TEST returns flag = true if the operation identified by request is complete. In such a case, the status object is set to contain information on the completed operation; if the communication object was created by a nonblocking send or receive, then it is deallocated and the request handle is set to MPI REQUEST NULL. The call returns flag = false, otherwise. In this case, the value of the status object is undefined. MPI TEST is a local operation.

The return status object for a receive operation carries information that can be accessed as described in section 1.2.5. The status object for a send operation carries information

⊤ that can be accessed by a call to MPI TEST CANCELLED (see Section 1.8).

One is allowed to call MPI TEST with a null or inactive request argument. In such a

⊥ case the operation returns with flag = true and null status.

The functions MPI WAIT and MPI TEST can be used to complete both sends and receives.

Advice to users. The use of the nonblocking MPI TEST call allows the user to schedule alternative activities within a single thread of execution. An event-driven thread scheduler can be emulated with periodic calls to MPI TEST. (End of advice to users.)

⊤

Rationale. The function MPI TEST returns with flag = true exactly in those situations where the function MPI WAIT returns; both functions return in such case the same value in status. Thus, a blocking Wait can be easily replaced by a nonblocking Test. (End of rationale.)

⊥

Example 1.10 Simple usage of nonblocking operations and MPI WAIT.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

CALL MPI_COMM_RANK(comm, rank, ierr)
IF(rank.EQ.0) THEN
CALL MPI_ISEND(a(1), 10, MPI_REAL, 1, tag, comm, request, ierr)
**** do some computation to mask latency ****
CALL MPI_WAIT(request, status, ierr)
ELSE
CALL MPI_IRECV(a(1), 15, MPI_REAL, 0, tag, comm, request, ierr)
**** do some computation to mask latency ****
CALL MPI_WAIT(request, status, ierr)
END IF

A request object can be deallocated without waiting for the associated communication to complete, by using the following operation.

MPI REQUEST FREE(request)

INOUT request communication request (handle)

int MPI Request free(MPI Request *request)
MPI REQUEST FREE(REQUEST, IERROR)
INTEGER REQUEST, IERROR

Mark the request object for deallocation and set request to MPI REQUEST NULL. An ongoing communication that is associated with the request will be allowed to complete. The request will be deallocated only after its completion.

Rationale. The MPI REQUEST FREE mechanism is provided for reasons of performance and convenience on the sending side. (End of rationale.)

Advice to users. Once a request is freed by a call to MPI REQUEST FREE, it is not possible to check for the successful completion of the associated communication with calls to MPI WAIT or MPI TEST. Also, if an error occurs subsequently during the communication, an error code cannot be returned to the user — such an error must be treated as fatal. Questions arise as to how one knows when the operations have completed when using MPI REQUEST FREE. Depending on the program logic, there may be other ways in which the program knows that certain operations have completed and this makes usage of MPI REQUEST FREE practical. For example, an active send request could be freed when the logic of the program is such that the receiver sends a reply to the message sent — the arrival of the reply informs the sender that the send has completed and the send buffer can be reused. An active receive request should never be freed as the receiver will have no way to verify that the receive has completed and the receive buffer can be reused. (End of advice to users.)

Example 1.11 An example using MPI REQUEST FREE.

CALL MPI_COMM_RANK(MPI_COMM_WORLD, rank)
IF(rank.EQ.0) THEN

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

    DO i=1, n
      CALL MPI_ISEND(outval, 1, MPI_REAL, 1, 0, req, ierr)
      CALL MPI_REQUEST_FREE(req, ierr)
      CALL MPI_IRECV(inval, 1, MPI_REAL, 1, 0, req, ierr)
      CALL MPI_WAIT(req, status, ierr)
    END DO
ELSE    ! rank.EQ.1
    CALL MPI_IRECV(inval, 1, MPI_REAL, 0, 0, req, ierr)
    CALL MPI_WAIT(req, status)
    DO I=1, n-1
       CALL MPI_ISEND(outval, 1, MPI_REAL, 0, 0, req, ierr)
       CALL MPI_REQUEST_FREE(req, ierr)
       CALL MPI_IRECV(inval, 1, MPI_REAL, 0, 0, req, ierr)
       CALL MPI_WAIT(req, status, ierr)
    END DO
    CALL MPI_ISEND(outval, 1, MPI_REAL, 0, 0, req, ierr)
    CALL MPI_WAIT(req, status)
END IF

1.7.4 Semantics of Nonblocking Communications

The semantics of nonblocking communication is defined by suitably extending the definitions in Section 1.5.

Order Nonblocking communication operations are ordered according to the execution order of the calls that initiate the communication. The non-overtaking requirement of Section 1.5 is extended to nonblocking communication, with this definition of order being used.

Example 1.12 Message ordering for nonblocking operations.

CALL MPI_COMM_RANK(comm, rank, ierr)
IF (RANK.EQ.0) THEN
      CALL MPI_ISEND(a, 1, MPI_REAL, 1, 0, comm, r1, ierr)
      CALL MPI_ISEND(b, 1, MPI_REAL, 1, 0, comm, r2, ierr)
ELSE    ! rank.EQ.1
      CALL MPI_IRECV(a, 1, MPI_REAL, 0, MPI_ANY_TAG, comm, r1, ierr)
      CALL MPI_IRECV(b, 1, MPI_REAL, 0, 0, comm, r2, ierr)
END IF
CALL MPI_WAIT(r1,status)
CALL MPI_WAIT(r2,status)

The first send of process zero will match the first receive of process one, even if both messages are sent before process one executes either receive.

Progress A call to MPI WAIT that completes a receive will eventually terminate and return if a matching send has been started, unless the send is satisfied by another receive. In particular, if the matching send is nonblocking, then the receive should complete even if no call is executed by the sender to complete the send. Similarly, a call to MPI WAIT that completes a send will eventually return if a matching receive has been started, unless the receive is satisfied by another send, and even if no call is executed to complete the receive.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

Example 1.13 An illustration of progress semantics.

CALL MPI_COMM_RANK(comm, rank, ierr)
IF (RANK.EQ.0) THEN
      CALL MPI_SSEND(a, 1, MPI_REAL, 1, 0, comm, ierr)
      CALL MPI_SEND(b, 1, MPI_REAL, 1, 1, comm, ierr)
ELSE    ! rank.EQ.1
      CALL MPI_IRECV(a, 1, MPI_REAL, 0, 0, comm, r, ierr)
      CALL MPI_RECV(b, 1, MPI_REAL, 0, 1, comm, ierr)
      CALL MPI_WAIT(r, status, ierr)
END IF

This code should not deadlock in a correct MPI implementation. The first synchronous send of process zero must complete after process one posts the matching (nonblocking) receive even if process one has not yet reached the completing wait call. Thus, process zero will continue and execute the second send, allowing process one to complete execution.

If an MPI TEST that completes a receive is repeatedly called with the same arguments, and a matching send has been started, then the call will eventually return flag = true, unless the send is satisfied by another receive. If an MPI TEST that completes a send is repeatedly called with the same arguments, and a matching receive has been started, then the call will eventually return flag = true, unless the receive is satisfied by another send.

1.7.5 Multiple Completions

It is convenient to be able to wait for the completion of any, some, or all the operations in a list, rather than having to wait for a specific message. A call to MPI WAITANY or MPI TESTANY can be used to wait for the completion of one out of several operations. A call to MPI WAITALL or MPI TESTALL can be used to wait for all pending operations in a list. A call to MPI WAITSOME or MPI TESTSOME can be used to complete all enabled operations in a list.

MPI WAITANY (count, array of requests, index, status)

IN  count  list length (integer)

INOUT array of requests array of requests (array of handles)

OUT  index  index of handle for operation that completed (integer)
OUT  status  status object (Status)

int MPI Waitany(int count, MPI Request *array of requests, int *index,
             MPI Status *status)
MPI WAITANY(COUNT, ARRAY OF REQUESTS, INDEX, STATUS, IERROR)
    INTEGER COUNT, ARRAY OF REQUESTS(*), INDEX, STATUS(MPI STATUS SIZE),
    IERROR

Blocks until one of the operations associated with the active requests in the array has completed. If more then one operation is enabled and can terminate, one is arbitrarily chosen. Returns in index the index of that request in the array and returns in status the

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

status of the completing communication. (The array is indexed from zero in C, and from one in Fortran.) If the request was allocated by a nonblocking communication operation,

⊤ then it is deallocated and the request handle is set to MPI REQUEST NULL.

The array of requests list may contain null or inactive handles. If the list contains no active handles (list has length zero or all entries are null or inactive), then the call returns

⊥ immediately with index = MPI UNDEFINED, and a null status.

The execution of MPI WAITANY(count, array of requests, index, status) has the same effect as the execution of MPI WAIT(&array of requests[i], status), where i is the value returned by index (unless the value of index is MPI UNDEFINED). MPI WAITANY with an array containing one active entry is equivalent to MPI WAIT.

MPI TESTANY(count, array of requests, index, flag, status)

IN  count  list length (integer)

INOUT array of requests array of requests (array of handles)

OUT  index  index of operation that completed, or MPI UNDE-

FINED if none completed (integer)

OUT  flag  true if one of the operations is complete (logical)
OUT  status  status object (Status)

int MPI Testany(int count, MPI Request *array of requests, int *index,
             int *flag, MPI Status *status)
MPI TESTANY(COUNT, ARRAY OF REQUESTS, INDEX, FLAG, STATUS, IERROR)
    LOGICAL FLAG
    INTEGER COUNT, ARRAY OF REQUESTS(*), INDEX, STATUS(MPI STATUS SIZE),
    IERROR

Tests for completion of either one or none of the operations associated with active handles. In the former case, it returns flag = true, returns in index the index of this request in the array, and returns in status the status of that operation; if the request was allocated by a nonblocking communication call then the request is deallocated and the handle is set to MPI REQUEST NULL. (The array is indexed from zero in C, and from one in Fortran.) In the latter case (no operation completed), it returns flag = false, returns a value of MPI UNDEFINED in index and status is undefined. The array may contain null or inactive

⊤ handles.

If the array contains no active handles then the call returns immediately with flag = true, index = MPI UNDEFINED, and a null status.

⊥ If the array of requests contain active handles then the execution of MPI TESTANY(count, array of requests, index, status) has the same effect as the execution of MPI TEST( &array of requests[i], flag, status), for i=0, 1 ,..., count-1, in some arbitrary order, until one call returns flag = true, or all fail. In the former case, index is set to the last value of i, and in the latter case, it is set to MPI UNDEFINED. MPI TESTANY with an array containing one

⊤ active entry is equivalent to MPI TEST.

Rationale. The function MPI TESTANY returns with flag = true exactly in those situations where the function MPI WAITANY returns; both functions return in such

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

case the same values in the remaining parameters. Thus, a blocking MPI WAITANY can be easily replaced by a nonblocking MPI TESTANY. The same relation holds for the other pairs of Wait and Test functions defined in this section. (End of rationale.)

⊥

MPI WAITALL( count, array of requests, array of statuses)

IN  count  lists length (integer)

INOUT array of requests array of requests (array of handles)

OUT  array of statuses  array of status objects (array of Status)

int MPI Waitall(int count, MPI Request *array of requests,
             MPI Status *array of statuses)
MPI WAITALL(COUNT, ARRAY OF REQUESTS, ARRAY OF STATUSES, IERROR)
    INTEGER COUNT, ARRAY OF REQUESTS(*)
    INTEGER ARRAY OF STATUSES(MPI STATUS SIZE,*), IERROR

Blocks until all communication operations associated with active handles in the list complete, and return the status of all these operations (this includes the case where no handle in the list is active). Both arrays have the same number of valid entries. The i-th entry in array of statuses is set to the return status of the i-th operation. Requests that were created by nonblocking communication operations are deallocated and the corresponding handles in the array are set to MPI REQUEST NULL. The list may contain null or inactive handles. The call sets to null the status of each such entry.

The execution of MPI WAITALL(count, array of requests, array of statuses) has the same effect as the execution of MPI WAIT(&array of request[i], &array of statuses[i]), for i=0 ,..., count-1, in some arbitrary order. MPI WAITALL with an array of length one is equivalent to MPI WAIT.

MPI TESTALL(count, array of requests, flag, array of statuses)

IN  count  lists length (integer)

INOUT array of requests array of requests (array of handles)

OUT  flag  (logical)
OUT  array of statuses  array of status objects (array of Status)

int MPI Testall(int count, MPI Request *array of requests, int *flag,
             MPI Status *array of statuses)
MPI TESTALL(COUNT, ARRAY OF REQUESTS, FLAG, ARRAY OF STATUSES, IERROR)
    LOGICAL FLAG
    INTEGER COUNT, ARRAY OF REQUESTS(*),
    ARRAY OF STATUSES(MPI STATUS SIZE,*), IERROR

Returns flag = true if all communications associated with active handles in the array have completed (this includes the case where no handle in the list is active). In this case, each status entry that corresponds to an active handle request is set to the status of the

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

corresponding communication; if the request was allocated by a nonblocking communication call then it is deallocated, and the handle is set to MPI REQUEST NULL. Each status entry that corresponds to a null or inactive handle is set to null.

Otherwise, flag = false is returned, no request is modified and the values of the status entries are undefined. This is a local operation.

MPI WAITSOME(incount, array of requests, outcount, array of indices, array of statuses)

IN  incount  length of array of requests (integer)

INOUT array of requests array of requests (array of handles)

OUT  outcount  number of completed requests (integer)
OUT  array of indices  array of indices of operations that completed (array of

integers)

OUT  array of statuses  array of status objects for operations that completed

(array of Status)

int MPI Waitsome(int incount, MPI Request *array of requests, int *outcount,
             int *array of indices, MPI Status *array of statuses)
MPI WAITSOME(INCOUNT, ARRAY OF REQUESTS, OUTCOUNT, ARRAY OF INDICES,
             ARRAY OF STATUSES, IERROR)
    INTEGER INCOUNT, ARRAY OF REQUESTS(*), OUTCOUNT, ARRAY OF INDICES(*),
    ARRAY OF STATUSES(MPI STATUS SIZE,*), IERROR

Waits until at least one of the operations associated with active handles in the list have completed. Returns in outcount the number of requests from the list array of requests that have completed. Returns in the first outcount locations of the array array of indices the indices of these operations (index within the array array of requests; the array is indexed from zero in C and from one in Fortran). Returns in the first outcount locations of the array array of status the status for these completed operations. If a request that completed was allocated by a nonblocking communication call, then it is deallocated, and the associated

⊤ handle is set to MPI REQUEST NULL.

If the list contains no active handles, then the call returns immediately with outcount

⊥ = MPI UNDEFINED.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

MPI TESTSOME(incount, array of requests, outcount, array of indices, array of statuses)

IN  incount  length of array of requests (integer)

INOUT array of requests array of requests (array of handles)

OUT  outcount  number of completed requests (integer)
OUT  array of indices  array of indices of operations that completed (array of

integers)

OUT  array of statuses  array of status objects for operations that completed

(array of Status)

int MPI Testsome(int incount, MPI Request *array of requests, int *outcount,
             int *array of indices, MPI Status *array of statuses)
MPI TESTSOME(INCOUNT, ARRAY OF REQUESTS, OUTCOUNT, ARRAY OF INDICES,
             ARRAY OF STATUSES, IERROR)
    INTEGER INCOUNT, ARRAY OF REQUESTS(*), OUTCOUNT, ARRAY OF INDICES(*),
    ARRAY OF STATUSES(MPI STATUS SIZE,*), IERROR

Behaves like MPI WAITSOME, except that it returns immediately. If no operation has completed it returns outcount = 0. If there is no active handle in the list it returns outcount = MPI UNDEFINED.

MPI TESTSOME is a local operation, which returns immediately, whereas MPI WAIT-SOME will block until a communication completes, if it was passed a list that contains at least one active handle. Both calls fulfil a fairness requirement: If a request for a receive repeatedly appears in a list of requests passed to MPI WAITSOME or MPI TESTSOME, and a matching send has been posted, then the receive will eventually succeed, unless the send is satisfied by another receive; and similarly for send requests.

Advice to users. The use of MPI TESTSOME is likely to be more efficient than the use of MPI TESTANY. The former returns information on all completed communications, with the latter, a new call is required for each communication that completes.

A server with multiple clients can use MPI WAITSOME so as not to starve any client. Clients send messages to the server with service requests. The server calls MPI WAITSOME with one receive request for each client, and then handles all receives that completed. If a call to MPI WAITANY is used instead, then one client could starve while requests from another client always sneak in first. (End of advice to users.)

Advice to implementors. MPI TESTSOME should complete as many pending communications as possible. (End of advice to implementors.)

Example 1.14 Client-server code (starvation can occur).

CALL MPI_COMM_SIZE(comm, size, ierr)
CALL MPI_COMM_RANK(comm, rank, ierr)
IF(rank > 0) THEN         ! client code
    DO WHILE(.TRUE.)
       CALL MPI_ISEND(a, n, MPI_REAL, 0, tag, comm, request, ierr)

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

       CALL MPI_WAIT(request, status, ierr)
    END DO
ELSE         ! rank=0 -- server code
       DO i=1, size-1
          CALL MPI_IRECV(a(1,i), n, MPI_REAL, 0, tag,
                   comm, request_list(i), ierr)
       END DO
       DO WHILE(.TRUE.)
          CALL MPI_WAITANY(size-1, request_list, index, status, ierr)
          CALL DO_SERVICE(a(1,index))  ! handle one message
          CALL MPI_IRECV(a(1, index), n, MPI_REAL, 0, tag,
                    comm, request_list(index), ierr)
       END DO
END IF

Example 1.15 Same code, using MPI WAITSOME.

CALL MPI_COMM_SIZE(comm, size, ierr)
CALL MPI_COMM_RANK(comm, rank, ierr)
IF(rank > 0) THEN         ! client code
    DO WHILE(.TRUE.)
       CALL MPI_ISEND(a, n, MPI_REAL, 0, tag, comm, request, ierr)
       CALL MPI_WAIT(request, status, ierr)
    END DO
ELSE         ! rank=0 -- server code
    DO i=1, size-1
       CALL MPI_IRECV(a(1,i), n, MPI_REAL, 0, tag,
                      comm, request_list(i), ierr)
    END DO
    DO WHILE(.TRUE.)
       CALL MPI_WAITSOME(size, request_list, numdone,
                        index_list, status_list, ierr)
       DO i=1, numdone
          CALL DO_SERVICE(a(1, index_list(i)))
          CALL MPI_IRECV(a(1, index_list(i)), n, MPI_REAL, 0, tag,
                       comm, request_list(i), ierr)
       END DO
    END DO
END IF

1.8 Probe and Cancel

The MPI PROBE and MPI IPROBE operations allow incoming messages to be checked for, without actually receiving them. The user can then decide how to receive them, based on the information returned by the probe (basically, the information returned by status). In particular, the user may allocate memory for the receive buffer, according to the length of the probed message.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

The MPI CANCEL operation allows pending communications to be canceled. This is required for cleanup. Posting a send or a receive ties up user resources (send or receive buffers), and a cancel may be needed to free these resources gracefully.

MPI IPROBE(source, tag, comm, flag, status)

IN  source  source rank, or MPI ANY SOURCE (integer)
IN  tag  tag value or MPI ANY TAG (integer)
IN  comm  communicator (handle)
OUT  flag  (logical)
OUT  status  status object (Status)

int MPI Iprobe(int source, int tag, MPI Comm comm, int *flag,
             MPI Status *status)
MPI IPROBE(SOURCE, TAG, COMM, FLAG, STATUS, IERROR)
    LOGICAL FLAG
    INTEGER SOURCE, TAG, COMM, STATUS(MPI STATUS SIZE), IERROR

MPI IPROBE(source, tag, comm, flag, status) returns flag = true if there is a message that can be received and that matches the pattern specified by the arguments source, tag, and comm. The call matches the same message that would have been received by a call to MPI RECV(..., source, tag, comm, status) executed at the same point in the program, and returns in status the same value that would have been returned by MPI RECV(). Otherwise, the call returns flag = false, and leaves status undefined.

If MPI IPROBE returns flag = true, then the content of the status object can be subsequently accessed as described in section 1.2.5 to find the source, tag and length of the probed message.

A subsequent receive executed with the same context, and the source and tag returned in status by MPI IPROBE will receive the message that was matched by the probe, if no other intervening receive occurs after the probe. If the receiving process is multi-threaded, it is the user’s responsibility to ensure that the last condition holds.

The source argument of MPI PROBE can be MPI ANY SOURCE, and the tag argument can be MPI ANY TAG, so that one can probe for messages from an arbitrary source and/or with an arbitrary tag. However, a specific communication context must be provided with the comm argument.

It is not necessary to receive a message immediately after it has been probed for, and the same message may be probed for several times before it is received.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

MPI PROBE(source, tag, comm, status)

IN  source  source rank, or MPI ANY SOURCE (integer)
IN  tag  tag value, or MPI ANY TAG (integer)
IN  comm  communicator (handle)
OUT  status  status object (Status)

int MPI Probe(int source, int tag, MPI Comm comm, MPI Status *status)
MPI PROBE(SOURCE, TAG, COMM, STATUS, IERROR)
    INTEGER SOURCE, TAG, COMM, STATUS(MPI STATUS SIZE), IERROR

MPI PROBE behaves like MPI IPROBE except that it is a blocking call that returns only after a matching message has been found.

The MPI implementation of MPI PROBE and MPI IPROBE needs to guarantee progress: if a call to MPI PROBE has been issued by a process, and a send that matches the probe has been initiated by some process, then the call to MPI PROBE will return, unless the message is received by another concurrent receive operation (that is executed by another thread at the probing process). Similarly, if a process busy waits with MPI IPROBE and a matching message has been issued, then the call to MPI IPROBE will eventually return flag = true unless the message is received by another concurrent receive operation.

Example 1.16 Use blocking probe to wait for an incoming message.

       CALL MPI_COMM_RANK(comm, rank, ierr)
       IF (rank.EQ.0) THEN
            CALL MPI_SEND(i, 1, MPI_INTEGER, 2, 0, comm, ierr)
       ELSE IF(rank.EQ.1) THEN
            CALL MPI_SEND(x, 1, MPI_REAL, 2, 0, comm, ierr)
       ELSE   ! rank.EQ.2
           DO i=1, 2
              CALL MPI_PROBE(MPI_ANY_SOURCE, 0,
                              comm, status, ierr)
              IF (status(MPI_SOURCE) = 0) THEN
100                CALL MPI_RECV(i, 1, MPI_INTEGER, 0, 0, status, ierr)
              ELSE
200                CALL MPI_RECV(x, 1, MPI_REAL, 1, 0, status, ierr)
              END IF
           END DO
       END IF

Each message is received with the right type.

Example 1.17 A similar program to the previous example, but now it has a problem.

       CALL MPI_COMM_RANK(comm, rank, ierr)
       IF (rank.EQ.0) THEN
            CALL MPI_SEND(i, 1, MPI_INTEGER, 2, 0, comm, ierr)
       ELSE IF(rank.EQ.1) THEN

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

            CALL MPI_SEND(x, 1, MPI_REAL, 2, 0, comm, ierr)
       ELSE
           DO i=1, 2
              CALL MPI_PROBE(MPI_ANY_SOURCE, 0,
                              comm, status, ierr)
              IF (status(MPI_SOURCE) = 0) THEN
100                CALL MPI_RECV(i, 1, MPI_INTEGER, MPI_ANY_SOURCE,
                                 0, status, ierr)
              ELSE
200                CALL MPI_RECV(x, 1, MPI_REAL, MPI_ANY_SOURCE,
                                 0, status, ierr)
              END IF
           END DO
       END IF

We slightly modified example 1.16, using MPI ANY SOURCE as the source argument in the two receive calls in statements labeled 100 and 200. The program is now incorrect: the receive operation may receive a message that is distinct from the message probed by the preceding call to MPI PROBE.

Advice to implementors. A call to MPI PROBE(source, tag, comm, status) will match the message that would have been received by a call to MPI RECV(..., source, tag, comm, status) executed at the same point. Suppose that this message has source s, tag t and communicator c. If the tag argument in the probe call has value MPI ANY TAG then the message probed will be the earliest pending message from source s with communicator c and any tag; in any case, the message probed will be the earliest pending message from source s with tag t and communicator c (this is the message that would have been received, so as to preserve message order). This message continues as the earliest pending message from source s with tag t and communicator c, until it is received. A receive operation subsequent to the probe that uses the same communicator as the probe and uses the tag and source values returned by the probe, must receive this message, unless it has already been received by another receive operation. (End of advice to implementors.)

MPI CANCEL(request)

IN  request  communication request (handle)

int MPI Cancel(MPI Request *request)
MPI CANCEL(REQUEST, IERROR)
    INTEGER REQUEST, IERROR

A call to MPI CANCEL marks for cancellation a pending, nonblocking communication operation (send or receive). The cancel call is local. It returns immediately, possibly before the communication is actually canceled. It is still necessary to complete a communication that has been marked for cancellation, using a call to MPI REQUEST FREE, MPI WAIT or MPI TEST (or any of the derived operations).

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

If a communication is marked for cancellation, then a MPI WAIT call for that communication is guaranteed to return, irrespective of the activities of other processes (i.e., MPI WAIT behaves as a local function); similarly if MPI TEST is repeatedly called in a busy wait loop for a canceled communication, then MPI TEST will eventually be successful.

MPI CANCEL can be used to cancel a communication that uses a persistent request (see Sec. 1.9), in the same way it is used for nonpersistent requests. A successful cancellation cancels the active communication, but not the request itself. After the call to MPI CANCEL and the subsequent call to MPI WAIT or MPI TEST, the request becomes inactive and can be activated for a new communication.

The successful cancellation of a buffered send frees the buffer space occupied by the pending message.

Either the cancellation succeeds, or the communication succeeds, but not both. If a send is marked for cancellation, then it must be the case that either the send completes normally, in which case the message sent was received at the destination process, or that the send is successfully canceled, in which case no part of the message was received at the destination. Then, any matching receive has to be satisfied by another send. If a receive is marked for cancellation, then it must be the case that either the receive completes normally, or that the receive is successfully canceled, in which case no part of the receive buffer is altered. Then, any matching send has to be satisfied by another receive.

If the operation has been canceled, then information to that effect will be returned in the status argument of the operation that completes the communication.

MPI TEST CANCELLED(status, flag)

IN  status  status object (Status)
OUT  flag  (logical)

int MPI Test cancelled(MPI Status status, int *flag)
MPI TEST CANCELLED(STATUS, FLAG, IERROR)
    LOGICAL FLAG
    INTEGER STATUS(MPI STATUS SIZE), IERROR

Returns flag = true if the communication associated with the status object was canceled successfully. In such a case, all other fields of status (such as count or tag) are undefined. Returns flag = false, otherwise. If a receive operation might be canceled then one should call MPI TEST CANCELLED first, to check whether the operation was canceled, before checking on the other fields of the return status.

Advice to users. Cancel can be an expensive operation that should be used only exceptionally. (End of advice to users.)

Advice to implementors. If a send operation uses an “eager” protocol (data is transferred to the receiver before a matching receive is posted), then the cancellation of this send may require communication with the intended receiver in order to free allocated buffers. On some systems this may require an interrupt to the intended receiver. Note that, while communication may be needed to implement MPI CANCEL, this is still a local operation, since its completion does not depend on the code executed by other processes. If processing is required on another process, this should be transparent to

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

the application (hence the need for an interrupt and an interrupt handler). (End of advice to implementors.)

1.9 Persistent communication requests

Often a communication with the same argument list is repeatedly executed within the inner loop of a parallel computation. In such a situation, it may be possible to optimize the communication by binding the list of communication arguments to a persistent communication request once and, then, repeatedly using the request to initiate and complete messages. The persistent request thus created can be thought of as a communication port or a “half-channel.” It does not provide the full functionality of a conventional channel, since there is no binding of the send port to the receive port. This construct allows reduction of the overhead for communication between the process and communication controller, but not of the overhead for communication between one communication controller and another. It is not necessary that messages sent with a persistent request be received by a receive operation using a persistent request, or vice versa.

A persistent communication request is created using one of the four following calls. These calls involve no communication.

MPI SEND INIT(buf, count, datatype, dest, tag, comm, request)

IN  buf  initial address of send buffer (choice)
IN  count  number of elements sent (integer)
IN  datatype  type of each element (handle)
IN  dest  rank of destination (integer)
IN  tag  message tag (integer)
IN  comm  communicator (handle)
OUT  request  communication request (handle)

int MPI Send init(void* buf, int count, MPI Datatype datatype, int dest,
             int tag, MPI Comm comm, MPI Request *request)
MPI SEND INIT(BUF, COUNT, DATATYPE, DEST, TAG, COMM, REQUEST, IERROR)
    <type> BUF(*)
    INTEGER REQUEST, COUNT, DATATYPE, DEST, TAG, COMM, REQUEST, IERROR

Creates a persistent communication request for a standard mode send operation, and binds to it all the arguments of a send operation.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

MPI BSEND INIT(buf, count, datatype, dest, tag, comm, request)

IN  buf  initial address of send buffer (choice)
IN  count  number of elements sent (integer)
IN  datatype  type of each element (handle)
IN  dest  rank of destination (integer)
IN  tag  message tag (integer)
IN  comm  communicator (handle)
OUT  request  communication request (handle)

int MPI Bsend init(void* buf, int count, MPI Datatype datatype, int dest,
             int tag, MPI Comm comm, MPI Request *request)
MPI BSEND INIT(BUF, COUNT, DATATYPE, DEST, TAG, COMM, REQUEST, IERROR)
    <type> BUF(*)
    INTEGER REQUEST, COUNT, DATATYPE, DEST, TAG, COMM, REQUEST, IERROR

Creates a persistent communication request for a buffered mode send.

MPI SSEND INIT(buf, count, datatype, dest, tag, comm, request)

IN  buf  initial address of send buffer (choice)
IN  count  number of elements sent (integer)
IN  datatype  type of each element (handle)
IN  dest  rank of destination (integer)
IN  tag  message tag (integer)
IN  comm  communicator (handle)
OUT  request  communication request (handle)

int MPI Ssend init(void* buf, int count, MPI Datatype datatype, int dest,
             int tag, MPI Comm comm, MPI Request *request)
MPI SSEND INIT(BUF, COUNT, DATATYPE, DEST, TAG, COMM, REQUEST, IERROR)
    <type> BUF(*)
    INTEGER COUNT, DATATYPE, DEST, TAG, COMM, REQUEST, IERROR

Creates a persistent communication object for a synchronous mode send operation.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

MPI RSEND INIT(buf, count, datatype, dest, tag, comm, request)

IN  buf  initial address of send buffer (choice)
IN  count  number of elements sent (integer)
IN  datatype  type of each element (handle)
IN  dest  rank of destination (integer)
IN  tag  message tag (integer)
IN  comm  communicator (handle)
OUT  request  communication request (handle)

int MPI Rsend init(void* buf, int count, MPI Datatype datatype, int dest,
             int tag, MPI Comm comm, MPI Request *request)
MPI RSEND INIT(BUF, COUNT, DATATYPE, DEST, TAG, COMM, REQUEST, IERROR)
    <type> BUF(*)
    INTEGER COUNT, DATATYPE, DEST, TAG, COMM, REQUEST, IERROR

Creates a persistent communication object for a ready mode send operation.

MPI RECV INIT(buf, count, datatype, source, tag, comm, request)

OUT  buf  initial address of receive buffer (choice)
IN  count  number of elements received (integer)
IN  datatype  type of each element (handle)
IN  source  rank of source or MPI ANY SOURCE (integer)
IN  tag  message tag or MPI ANY TAG (integer)
IN  comm  communicator (handle)
OUT  request  communication request (handle)

int MPI Recv init(void* buf, int count, MPI Datatype datatype, int source,
             int tag, MPI Comm comm, MPI Request *request)
MPI RECV INIT(BUF, COUNT, DATATYPE, SOURCE, TAG, COMM, REQUEST, IERROR)
    <type> BUF(*)
    INTEGER COUNT, DATATYPE, SOURCE, TAG, COMM, REQUEST, IERROR

Creates a persistent communication request for a receive operation. The argument buf is marked as OUT because the user gives permission to write on the receive buffer by passing the argument to MPI RECV INIT.

A persistent communication request is inactive after it was created — no active communication is attached to the request.

A communication (send or receive) that uses a persistent request is initiated by the function MPI START.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

MPI START(request)

INOUT request communication request (handle)

int MPI Start(MPI Request *request)
MPI START(REQUEST, IERROR)
    INTEGER REQUEST, IERROR

The argument, request, is a handle returned by one of the previous five calls. The associated request should be inactive. The request becomes active once the call is made.

If the request is for a send with ready mode, then a matching receive should be posted before the call is made. The communication buffer should not be accessed after the call, and until the operation completes.

The call is local, with similar semantics to the nonblocking communication operations described in section 1.7. That is, a call to MPI START with a request created by MPI SEND INIT starts a communication in the same manner as a call to MPI ISEND; a call to MPI START with a request created by MPI BSEND INIT starts a communication in the same manner as a call to MPI IBSEND; and so on.

MPI STARTALL(count, array of requests)

IN  count  list length (integer)

INOUT array of requests array of requests (array of handle)

int MPI Startall(int count, MPI Request *array of requests)
MPI STARTALL(COUNT, ARRAY OF REQUESTS, IERROR)
    INTEGER COUNT, ARRAY OF REQUESTS(*), IERROR

Start all communications associated with requests in array of requests. A call to MPI STARTALL(count, array of requests) has the same effect as calls to MPI START (&array of requests[i]), executed for i=0 ,..., count-1, in some arbitrary order.

A communication started with a call to MPI START or MPI STARTALL is completed by a call to MPI WAIT, MPI TEST, or one of the derived functions described in section 1.7.5. The request becomes inactive after successful completion of such call. The request is not deallocated and it can be activated anew by an MPI START or MPI STARTALL call.

A persistent request is deallocated by a call to MPI REQUEST FREE (Section 1.7.3). The call to MPI REQUEST FREE can occur at any point in the program after the per-

sistent request was created. However, the request will be deallocated only after it becomes inactive. Active receive requests should not be freed. Otherwise, it will not be possible to check that the receive has completed. It is preferable, in general, to free requests when they are inactive. If this rule is followed, then the functions described in this section will be invoked in a sequence of the form,

Create (Start Complete)^{∗} Free , where ∗ indicates zero or more repetitions. If the

same communication object is used in several concurrent threads, it is the user’s responsibility to coordinate calls so that the correct sequence is obeyed.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

A send operation initiated with MPI START can be matched with any receive operation and, likewise, a receive operation initiated with MPI START can receive messages generated by any send operation.

1.10 Send-receive

The send-receive operations combine in one call the sending of a message to one destination and the receiving of another message, from another process. The two (source and destination) are possibly the same. A send-receive operation is very useful for executing a shift operation across a chain of processes. If blocking sends and receives are used for such a shift, then one needs to order the sends and receives correctly (for example, even processes send, then receive, odd processes receive first, then send) so as to prevent cyclic dependencies that may lead to deadlock. When a send-receive operation is used, the communication subsystem takes care of these issues. The send-receive operation can be used in conjunction with the functions described in Chapter ?? in order to perform shifts on various logical topologies. Also, a send-receive operation is useful for implementing remote procedure calls.

A message sent by a send-receive operation can be received by a regular receive operation or probed by a probe operation; a send-receive operation can receive a message sent by a regular send operation.

MPI SENDRECV(sendbuf, sendcount, sendtype, dest, sendtag, recvbuf, recvcount, recvtype, source, recvtag, comm, status)

IN  sendbuf  initial address of send buffer (choice)
IN  sendcount  number of elements in send buffer (integer)
IN  sendtype  type of elements in send buffer (handle)
IN  dest  rank of destination (integer)
IN  sendtag  send tag (integer)
OUT  recvbuf  initial address of receive buffer (choice)
IN  recvcount  number of elements in receive buffer (integer)
IN  recvtype  type of elements in receive buffer (handle)
IN  source  rank of source (integer)
IN  recvtag  receive tag (integer)
IN  comm  communicator (handle)
OUT  status  status object (Status)

int MPI Sendrecv(void *sendbuf, int sendcount, MPI Datatype sendtype,
             int dest, int sendtag, void *recvbuf, int recvcount,
             MPI Datatype recvtype, int source, MPI Datatype recvtag,
             MPI Comm comm, MPI Status *status)
MPI SENDRECV(SENDBUF, SENDCOUNT, SENDTYPE, DEST, SENDTAG, RECVBUF,
             RECVCOUNT, RECVTYPE, SOURCE, RECVTAG, COMM, STATUS, IERROR)

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

    <type> SENDBUF(*), RECVBUF(*)
    INTEGER SENDCOUNT, SENDTYPE, DEST, SENDTAG, RECVCOUNT, RECVTYPE,
    SOURCE, RECV TAG, COMM, STATUS(MPI STATUS SIZE), IERROR

Execute a blocking send and receive operation. Both send and receive use the same communicator, but possibly different tags. The send buffer and receive buffers must be disjoint, and may have different lengths and datatypes.

MPI SENDRECV REPLACE(buf, count, datatype, dest, sendtag, source, recvtag, comm, status)

INOUT buf initial address of send and receive buffer (choice)

IN  count  number of elements in send and receive buffer (integer)
IN  datatype  type of elements in send and receive buffer (handle)
IN  dest  rank of destination (integer)
IN  sendtag  send message tag (integer)
IN  source  rank of source (integer)
IN  recvtag  receive message tag (integer)
IN  comm  communicator (handle)
OUT  status  status object (Status)

int MPI Sendrecv replace(void* buf, int count, MPI Datatype datatype,
             int dest, int sendtag, int source, int recvtag, MPI Comm comm,
             MPI Status *status)
MPI SENDRECV REPLACE(BUF, COUNT, DATATYPE, DEST, SENDTAG, SOURCE, RECVTAG,
             COMM, STATUS, IERROR)
    <type> BUF(*)
    INTEGER COUNT, DATATYPE, DEST, SENDTAG, SOURCE, RECVTAG, COMM,
    STATUS(MPI STATUS SIZE), IERROR

Execute a blocking send and receive. The same buffer is used both for the send and for the receive, so that the message sent is replaced by the message received.

The semantics of a send-receive operation is what would be obtained if the caller forked two concurrent threads, one to execute the send, and one to execute the receive, followed by a join of these two threads.

Advice to implementors. Additional intermediate buffering is needed for the “replace” variant. (End of advice to implementors.)

1.11 Null processes

In many instances, it is convenient to specify a “dummy” source or destination for communication. This simplifies the code that is needed for dealing with boundaries, for example, in the case of a non-circular shift done with calls to send-receive.

The special value MPI PROC NULL can be used instead of a rank wherever a source or a destination argument is required in a call. A communication with process MPI PROC NULL

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

has no effect. A send to MPI PROC NULL succeeds and returns as soon as possible. A receive from MPI PROC NULL succeeds and returns as soon as possible with no modifications to the receive buffer. When a receive with source = MPI PROC NULL is executed then the status object returns source = MPI PROC NULL, tag = MPI ANY TAG and count = 0.

1.12 Derived datatypes

Up to here, all point to point communication have involved only contiguous buffers containing a sequence of elements of the same type. This is too constraining on two accounts. One often wants to pass messages that contain values with different datatypes (e.g., an integer count, followed by a sequence of real numbers); and one often wants to send noncontiguous data (e.g., a sub-block of a matrix). One solution is to pack noncontiguous data into a contiguous buffer at the sender site and unpack it back at the receiver site. This has the disadvantage of requiring additional memory-to-memory copy operations at both sites, even when the communication subsystem has scatter-gather capabilities. Instead, MPI provides mechanisms to specify more general, mixed, and noncontiguous communication buffers. It is up to the implementation to decide whether data should be first packed in a contiguous buffer before being transmitted, or whether it can be collected directly from where it resides.

The general mechanisms provided here allow one to transfer directly, without copying, objects of various shape and size. It is not assumed that the MPI library is cognizant of the objects declared in the host language. Thus, if one wants to transfer a structure, or an array section, it will be necessary to provide in MPI a definition of a communication buffer that mimics the definition of the structure or array section in question. These facilities can be used by library designers to define communication functions that can transfer objects defined in the host language — by decoding their definitions as available in a symbol table or a dope vector. Such higher-level communication functions are not part of MPI.

More general communication buffers are specified by replacing the basic datatypes that have been used so far with derived datatypes that are constructed from basic datatypes using the constructors described in this section. These methods of constructing derived datatypes can be applied recursively.

A general datatype is an opaque object that specifies two things:

• A sequence of basic datatypes

• A sequence of integer (byte) displacements

The displacements are not required to be positive, distinct, or in increasing order. Therefore, the order of items need not coincide with their order in store, and an item may appear more than once. We call such a pair of sequences (or sequence of pairs) a type map. The sequence of basic datatypes (displacements ignored) is the type signature of the datatype.

Let

T ypemap = {(type_{0}, disp_{0}), ..., (type_{n−1}, disp_{n−1})},

be such a type map, where type_{i} are basic types, and disp_{i} are displacements. Let

T ypesig = {type_{0}, ..., type_{n−1}}

be the associated type signature. This type map, together with a base address buf, specifies a communication buffer: the communication buffer that consists of n entries, where the

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

i-th entry is at address buf + disp_{i} and has type type_{i}. A message assembled from such a communication buffer will consist of n values, of the types defined by T ypesig.

We can use a handle to a general datatype as an argument in a send or receive operation, instead of a basic datatype argument. The operation MPI SEND(buf, 1, datatype,...) will use the send buffer defined by the base address buf and the general datatype associated with datatype; it will generate a message with the type signature determined by the datatype argument. MPI RECV(buf, 1, datatype,...) will use the receive buffer defined by the base address buf and the general datatype associated with datatype.

General datatypes can be used in all send and receive operations. We discuss, in Sec. 1.12.5, the case where the second argument count has value > 1.

The basic datatypes presented in section 1.2.2 are particular cases of a general datatype, and are predefined. Thus, MPI INT is a predefined handle to a datatype with type map {(int, 0)}, with one entry of type int and displacement zero. The other basic datatypes are similar.

The extent of a datatype is defined to be the span from the first byte to the last byte occupied by entries in this datatype, rounded up to satisfy alignment requirements. That is, if

T ypemap = {(type_{0}, disp_{0}), ..., (type_{n−1}, disp_{n−1})},

⊤ then

lb(T ypemap) = min_{j} disp_{j},
ub(T ypemap) = max_{j}(disp_{j} + sizeof(type_{j})) + ϵ, and
extent(T ypemap) = ub(T ypemap) − lb(T ypemap).  (1.1)

If type_{i} requires alignment to a byte address that is is a multiple of k_{i}, then ϵ is the least

⊥ nonnegative increment needed to round extent(T ypemap) to the next multiple of max_{i} k_{i}.

Example 1.18 Assume that T ype = {(double, 0), (char, 8)} (a double at displacement zero, followed by a char at displacement eight). Assume, furthermore, that doubles have to be strictly aligned at addresses that are multiples of eight. Then, the extent of this datatype is 16 (9 rounded to the next multiple of 8). A datatype that consists of a character immediately followed by a double will also have an extent of 16.

Rationale. The definition of extent is motivated by the assumption that the amount of padding added at the end of each structure in an array of structures is the least needed to fulfill alignment constraints. More explicit control of the extent is provided in section 1.12.3. Such explicit control is needed in cases where the assumption does not hold, for example, where union types are used. (End of rationale.)

1.12.1 Datatype constructors

Contiguous The simplest datatype constructor is MPI TYPE CONTIGUOUS which allows replication of a datatype into contiguous locations.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

MPI TYPE CONTIGUOUS(count, oldtype, newtype)

IN  count  replication count (nonnegative integer)
IN  oldtype  old datatype (handle)
OUT  newtype  new datatype (handle)

int MPI Type contiguous(int count, MPI Datatype oldtype,
             MPI Datatype *newtype)
MPI TYPE CONTIGUOUS(COUNT, OLDTYPE, NEWTYPE, IERROR)
    INTEGER COUNT, OLDTYPE, NEWTYPE, IERROR

newtype is the datatype obtained by concatenating count copies of oldtype. Concatenation is defined using extent as the size of the concatenated copies.

Example 1.19 Let oldtype have type map {(double, 0), (char, 8)}, with extent 16, and let count = 3. The type map of the datatype returned by newtype is

{(double, 0), (char, 8), (double, 16), (char, 24), (double, 32), (char, 40)};

i.e., alternating double and char elements, with displacements 0, 8, 16, 24, 32, 40.

In general, assume that the type map of oldtype is

{(type_{0}, disp_{0}), ..., (type_{n−1}, disp_{n−1})},

with extent ex. Then newtype has a type map with count · n entries defined by:

{(type_{0}, disp_{0}), ..., (type_{n−1}, disp_{n−1}), (type_{0}, disp_{0} + ex), ..., (type_{n−1}, disp_{n−1} + ex),

..., (type_{0}, disp_{0} + ex · (count − 1)), ..., (type_{n−1}, disp_{n−1} + ex · (count − 1))}.

Vector The function MPI TYPE VECTOR is a more general constructor that allows replication of a datatype into locations that consist of equally spaced blocks. Each block is obtained by concatenating the same number of copies of the old datatype. The spacing between blocks is a multiple of the extent of the old datatype.

MPI TYPE VECTOR( count, blocklength, stride, oldtype, newtype)

IN  count  number of blocks (nonnegative integer)
IN  blocklength  number of elements in each block (nonnegative inte-

ger)

IN stride number of elements between start of each block (inte-

ger)

IN  oldtype  old datatype (handle)
OUT  newtype  new datatype (handle)

int MPI Type vector(int count, int blocklength, int stride,
             MPI Datatype oldtype, MPI Datatype *newtype)

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

MPI TYPE VECTOR(COUNT, BLOCKLENGTH, STRIDE, OLDTYPE, NEWTYPE, IERROR)
    INTEGER COUNT, BLOCKLENGTH, STRIDE, OLDTYPE, NEWTYPE, IERROR

Example 1.20 Assume, again, that oldtype has type map {(double, 0), (char, 8)}, with extent 16. A call to MPI TYPE VECTOR( 2, 3, 4, oldtype, newtype) will create the datatype with type map,

{(double, 0), (char, 8), (double, 16), (char, 24), (double, 32), (char, 40),

(double, 64), (char, 72), (double, 80), (char, 88), (double, 96), (char, 104)}.

That is, two blocks with three copies each of the old type, with a stride of 4 elements (4 · 16 bytes) between the blocks.

Example 1.21 A call to MPI TYPE VECTOR(3, 1, -2, oldtype, newtype) will create the datatype,

{(double, 0), (char, 8), (double, −32), (char, −24), (double, −64), (char, −56)}.

In general, assume that oldtype has type map,

{(type_{0}, disp_{0}), ..., (type_{n−1}, disp_{n−1})},

with extent ex. Let bl be the blocklength. The newly created datatype has a type map with count · bl · n entries:

{(type_{0}, disp_{0}), ..., (type_{n−1}, disp_{n−1}),

(type_{0}, disp_{0} + ex), ..., (type_{n−1}, disp_{n−1} + ex), ...,

(type_{0}, disp_{0} + (bl − 1) · ex), ..., (type_{n−1}, disp_{n−1} + (bl − 1) · ex),

(type_{0}, disp_{0} + stride · ex), ..., (type_{n−1}, disp_{n−1} + stride · ex), ...,

(type_{0}, disp_{0} + (stride + bl − 1) · ex), ..., (type_{n−1}, disp_{n−1} + (stride + bl − 1) · ex), ....,

(type_{0}, disp_{0} + stride · (count − 1) · ex), ...,

(type_{n−1}, disp_{n−1} + stride · (count − 1) · ex), ...,

(type_{0}, disp_{0} + (stride · (count − 1) + bl − 1) · ex), ...,

(type_{n−1}, disp_{n−1} + (stride · (count − 1) + bl − 1) · ex)}.

A call to MPI TYPE CONTIGUOUS(count, oldtype, newtype) is equivalent to a call to MPI TYPE VECTOR(count, 1, 1, oldtype, newtype), or to a call to MPI TYPE VECTOR(1, count, n, oldtype, newtype), n arbitrary.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

Hvector The function MPI TYPE HVECTOR is identical to MPI TYPE VECTOR, except that stride is given in bytes, rather than in elements. The use for both types of vector constructors is illustrated in Sec. 1.12.7. (H stands for “heterogeneous”).

MPI TYPE HVECTOR( count, blocklength, stride, oldtype, newtype)

IN  count  number of blocks (nonnegative integer)
IN  blocklength  number of elements in each block (nonnegative inte-

ger)

IN  stride  number of bytes between start of each block (integer)
IN  oldtype  old datatype (handle)
OUT  newtype  new datatype (handle)

int MPI Type hvector(int count, int blocklength, MPI Aint stride,
         MPI Datatype oldtype, MPI Datatype *newtype)
MPI TYPE HVECTOR(COUNT, BLOCKLENGTH, STRIDE, OLDTYPE, NEWTYPE, IERROR)
INTEGER COUNT, BLOCKLENGTH, STRIDE, OLDTYPE, NEWTYPE, IERROR

Assume that oldtype has type map,

{(type_{0}, disp_{0}), ..., (type_{n−1}, disp_{n−1})},

with extent ex. Let bl be the blocklength. The newly created datatype has a type map with count · bl · n entries:

{(type_{0}, disp_{0}), ..., (type_{n−1}, disp_{n−1}),

(type_{0}, disp_{0} + ex), ..., (type_{n−1}, disp_{n−1} + ex), ...,

(type_{0}, disp_{0} + (bl − 1) · ex), ..., (type_{n−1}, disp_{n−1} + (bl − 1) · ex),

(type_{0}, disp_{0} + stride), ..., (type_{n−1}, disp_{n−1} + stride), ...,

(type_{0}, disp_{0} + stride + (bl − 1) · ex), ...,

(type_{n−1}, disp_{n−1} + stride + (bl − 1) · ex), ....,

(type_{0}, disp_{0} + stride · (count − 1)), ..., (type_{n−1}, disp_{n−1} + stride · (count − 1)), ...,

(type_{0}, disp_{0} + stride · (count − 1) + (bl − 1) · ex), ...,

(type_{n−1}, disp_{n−1} + stride · (count − 1) + (bl − 1) · ex)}.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

Indexed The function MPI TYPE INDEXED allows replication of an old datatype into a sequence of blocks (each block is a concatenation of the old datatype), where each block can contain a different number of copies and have a different displacement. All block displacements are multiples of the old type extent.

MPI TYPE INDEXED( count, array of blocklengths, array of displacements, oldtype, newtype)

IN  count  number of blocks – also number of entries in

array of displacements and array of blocklengths (nonnegative integer)

IN  array of blocklengths  number of elements per block (array of nonnegative

integers)

IN  array of displacements  displacement for each block, in multiples of oldtype

extent (array of integer)

IN  oldtype  old datatype (handle)
OUT  newtype  new datatype (handle)

int MPI Type indexed(int count, int *array of blocklengths,
             int *array of displacements, MPI Datatype oldtype,
             MPI Datatype *newtype)
MPI TYPE INDEXED(COUNT, ARRAY OF BLOCKLENGTHS, ARRAY OF DISPLACEMENTS,
             OLDTYPE, NEWTYPE, IERROR)
    INTEGER COUNT, ARRAY OF BLOCKLENGTHS(*), ARRAY OF DISPLACEMENTS(*),
    OLDTYPE, NEWTYPE, IERROR

Example 1.22 Let oldtype have type map {(double, 0), (char, 8)}, with extent 16. Let B = (3, 1) and let D = (4, 0). A call to MPI TYPE INDEXED(2, B, D, oldtype, newtype) returns a datatype with type map,

{(double, 64), (char, 72), (double, 80), (char, 88), (double, 96), (char, 104),

(double, 0), (char, 8)}.

That is, three copies of the old type starting at displacement 64, and one copy starting at displacement 0.

In general, assume that oldtype has type map,

{(type_{0}, disp_{0}), ..., (type_{n−1}, disp_{n−1})},

with extent ex. Let B be the array of blocklength argument and D be the array of displacements argument. The newly created datatype has n · ∑_{i=0}^{count−1} B[i] entries:

{(type_{0}, disp_{0} + D[0] · ex), ..., (type_{n−1}, disp_{n−1} + D[0] · ex), ...,
(type_{0}, disp_{0} + (D[0] + B[0] − 1) · ex), ..., (type_{n−1}, disp_{n−1} + (D[0] + B[0] − 1) · ex), ...,

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

(type_{0}, disp_{0} + D[count − 1] · ex), ..., (type_{n−1}, disp_{n−1} + D[count − 1] · ex), ...,

(type_{0}, disp_{0} + (D[count − 1] + B[count − 1] − 1) · ex), ...,

(type_{n−1}, disp_{n−1} + (D[count − 1] + B[count − 1] − 1) · ex)}.

A call to MPI TYPE VECTOR(count, blocklength, stride, oldtype, newtype) is equivalent to a call to MPI TYPE INDEXED(count, B, D, oldtype, newtype) where

D[j] = j · stride, j = 0, ..., count − 1,

and

B[j] = blocklength, j = 0, ..., count − 1.

Hindexed The function MPI TYPE HINDEXED is identical to MPI TYPE INDEXED, except that block displacements in array of displacements are specified in bytes, rather than in multiples of the oldtype extent.

MPI TYPE HINDEXED( count, array of blocklengths, array of displacements, oldtype, newtype)

IN  count  number of blocks – also number of entries in

array of displacements and array of blocklengths (integer)

IN  array of blocklengths  number of elements in each block (array of nonnega-

tive integers)

IN  array of displacements  byte displacement of each block (array of integer)
IN  oldtype  old datatype (handle)
OUT  newtype  new datatype (handle)

int MPI Type hindexed(int count, int *array of blocklengths,
             MPI Aint *array of displacements, MPI Datatype oldtype,
             MPI Datatype *newtype)
MPI TYPE HINDEXED(COUNT, ARRAY OF BLOCKLENGTHS, ARRAY OF DISPLACEMENTS,
             OLDTYPE, NEWTYPE, IERROR)
    INTEGER COUNT, ARRAY OF BLOCKLENGTHS(*), ARRAY OF DISPLACEMENTS(*),
    OLDTYPE, NEWTYPE, IERROR

Assume that oldtype has type map,

{(type_{0}, disp_{0}), ..., (type_{n−1}, disp_{n−1})},

with extent ex. Let B be the array of blocklength argument and D be the array of displacements argument. The newly created datatype has a type map with n · ∑_{i=0}^{count−1} B[i] entries:

{(type_{0}, disp_{0} + D[0]), ..., (type_{n−1}, disp_{n−1} + D[0]), ...,

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

(type_{0}, disp_{0} + D[0] + (B[0] − 1) · ex), ...,
(type_{n−1}, disp_{n−1} + D[0] + (B[0] − 1) · ex), ...,
(type_{0}, disp_{0} + D[count − 1]), ..., (type_{n−1}, disp_{n−1} + D[count − 1]), ...,
(type_{0}, disp_{0} + D[count − 1] + (B[count − 1] − 1) · ex), ...,
(type_{n−1}, disp_{n−1} + D[count − 1] + (B[count − 1] − 1) · ex)}.

Struct MPI TYPE STRUCT is the most general type constructor. It further generalizes the previous one in that it allows each block to consist of replications of different datatypes.

MPI TYPE STRUCT(count, array of blocklengths, array of displacements, array of types, newtype)

IN  count  number of blocks (integer) – also number of entries

in arrays array of types, array of displacements and array of blocklengths

IN  array of blocklength  number of elements in each block (array of integer)
IN  array of displacements  byte displacement of each block (array of integer)
IN  array of types  type of elements in each block (array of handles to

datatype objects)

OUT  newtype  new datatype (handle)

int MPI Type struct(int count, int *array of blocklengths,
             MPI Aint *array of displacements, MPI Datatype *array of types,
             MPI Datatype *newtype)
MPI TYPE STRUCT(COUNT, ARRAY OF BLOCKLENGTHS, ARRAY OF DISPLACEMENTS,
             ARRAY OF TYPES, NEWTYPE, IERROR)
    INTEGER COUNT, ARRAY OF BLOCKLENGTHS(*), ARRAY OF DISPLACEMENTS(*),
    ARRAY OF TYPES(*), NEWTYPE, IERROR

Example 1.23 Let type1 have type map,

{(double, 0), (char, 8)},

with extent 16. Let B = (2, 1, 3), D = (0, 16, 26), and T = (MPI FLOAT, type1, MPI CHAR). Then a call to MPI TYPE STRUCT(3, B, D, T, newtype) returns a datatype with type map,

{(float, 0), (float, 4), (double, 16), (char, 24), (char, 26), (char, 27), (char, 28)}.

That is, two copies of MPI FLOAT starting at 0, followed by one copy of type1 starting at 16, followed by three copies of MPI CHAR, starting at 26. (We assume that a float occupies four bytes.)

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

In general, let T be the array of types argument, where T[i] is a handle to,

typemap_{i} = {(type_{0}^{i}, disp_{0}^{i}), ..., (type_{n_{i}−1}^{i}, disp_{n_{i}−1}^{i})},

with extent ex_{i}. Let B be the array of blocklength argument and D be the array of displacements argument. Let c be the count argument. Then the newly created datatype has a type map with ∑_{i=0}^{c−1} B[i] · n_{i} entries:

{(type_{0}^{0}, disp_{0}^{0} + D[0]), ..., (type_{n_{0}}^{0}, disp_{n_{0}}^{0} + D[0]), ...,

(type_{0}^{0}, disp_{0}^{0} + D[0] + (B[0] − 1) · ex_{0}), ..., (type_{n_{0}}^{0}, disp_{n_{0}}^{0} + D[0] + (B[0] − 1) · ex_{0}), ...,

(type_{0}^{c−1}, disp_{0}^{c−1} + D[c − 1]), ..., (type_{n_{c−1}−1}^{c−1}, disp_{n_{c−1}−1}^{c−1} + D[c − 1]), ...,

(type_{0}^{c−1}, disp_{0}^{c−1} + D[c − 1] + (B[c − 1] − 1) · ex_{c−1}), ...,

(type_{n_{c−1}−1}^{c−1}, disp_{n_{c−1}−1}^{c−1} + D[c − 1] + (B[c − 1] − 1) · ex_{c−1})}.

A call to MPI TYPE HINDEXED( count, B, D, oldtype, newtype) is equivalent to a call to MPI TYPE STRUCT( count, B, D, T, newtype), where each entry of T is equal to oldtype.

1.12.2 Address and extent functions

The displacements in a general datatype are relative to some initial buffer address. Absolute addresses can be substituted for these displacements: we treat them as displacements relative to “address zero,” the start of the address space. This initial address zero is indicated by the constant MPI BOTTOM. Thus, a datatype can specify the absolute address of the entries in the communication buffer, in which case the buf argument is passed the value

MPI BOTTOM.

The address of a location in memory can be found by invoking the function MPI ADDRESS.

MPI ADDRESS(location, address)

IN  location  location in caller memory (choice)
OUT  address  address of location (integer)

int MPI Address(void* location, MPI Aint *address)
MPI ADDRESS(LOCATION, ADDRESS, IERROR)
    <type> LOCATION(*)
    INTEGER ADDRESS, IERROR

Returns the (byte) address of location.

Example 1.24 Using MPI ADDRESS for an array.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

   REAL A(100,100)
   INTEGER I1, I2, DIFF
   CALL MPI_ADDRESS(A(1,1), I1, IERROR)
   CALL MPI_ADDRESS(A(10,10), I2, IERROR)
   DIFF = I2 - I1
! The value of DIFF is 909*sizeofreal; the values of I1 and I2 are
! implementation dependent.

Advice to users. C users may be tempted to avoid the usage of MPI ADDRESS and rely on the availability of the address operator &. Note, however, that & castexpression is a pointer, not an address. ANSI C does not require that the value of a pointer (or the pointer cast to int) be the absolute address of the object pointed at — although this is commonly the case. Furthermore, referencing may not have a unique definition on machines with a segmented address space. The use of MPI ADDRESS to “reference” C variables guarantees portability to such machines as well. (End of advice to users.)

The following auxiliary functions provide useful information on derived datatypes.

MPI TYPE EXTENT(datatype, extent)

IN  datatype  datatype (handle)
OUT  extent  datatype extent (integer)

⊤

int MPI Type extent(MPI Datatype datatype, MPI Aint *extent)

⊥

MPI TYPE EXTENT(DATATYPE, EXTENT, IERROR)
    INTEGER DATATYPE, EXTENT, IERROR

Returns the extent of a datatype, where extent is as defined in Eq. 1.1 on page 46.

MPI TYPE SIZE(datatype, size)

IN  datatype  datatype (handle)
OUT  size  datatype size (integer)

int MPI Type size(MPI Datatype datatype, int *size)
MPI TYPE SIZE(DATATYPE, SIZE, IERROR)
    INTEGER DATATYPE, SIZE, IERROR

MPI TYPE SIZE returns the total size, in bytes, of the entries in the type signature associated with datatype; i.e., the total size of the data in a message that would be created with this datatype. Entries that occur multiple times in the datatype are counted with

⊤ their multiplicity. ⊥

1.12.3 Lower-bound and upper-bound markers

It is often convenient to define explicitly the lower bound and upper bound of a type map, and override the definition given by Equation 1.1 on page 46. This allows one to define

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

a datatype that has “holes” at its beginning or its end, or a datatype with entries that extend above the upper bound or below the lower bound. Examples of such usage are provided in Sec. 1.12.7. Also, the user may want to overide the alignment rules that are ⊤ used to compute upper bounds and extents. E.g., a C compiler may allow the user to overide default alignment rules for some of the structures within a program. The user has to specify explicitly the bounds of the datatypes that match these structures. ⊥

To achieve this, we add two additional “pseudo-datatypes,” MPI LB and MPI UB, that can be used, respectively, to mark the lower bound or the upper bound of a datatype. These pseudo-datatypes occupy no space (extent(MPI LB) = extent(MPI UB) = 0). They do not affect the size or count of a datatype, and do not affect the the content of a message created with this datatype. However, they do affect the definition of the extent of a datatype and, therefore, affect the outcome of a replication of this datatype by a datatype constructor.

Example 1.25 Let D = (-3, 0, 6); T = (MPI LB, MPI INT, MPI UB), and B = (1, 1, 1). Then a call to MPI TYPE STRUCT(3, B, D, T, type1) creates a new datatype that has an extent of 9 (from -3 to 5, 5 included), and contains an integer at displacement 0. This is the datatype defined by the sequence {(lb, -3), (int, 0), (ub, 6)} . If this type is replicated twice by a call to MPI TYPE CONTIGUOUS(2, type1, type2) then the newly created type can be described by the sequence {(lb, -3), (int, 0), (int,9), (ub, 15)} . (Entries of type lb or ⊤ ub can be deleted if they are dominated by other such entries.) ⊥

In general, if

T ypemap = {(type_{0}, disp_{0}), ..., (type_{n−1}, disp_{n−1})},

then the lower bound of T ypemap is defined to be

lb(T ypemap) = \{ \begin{matrix}min_{j} disp_{j} & if no entry has basic type lb \\ min_{j}\{disp_{j} such that type_{j} = lb\} & otherwise \end{matrix}

Similarly, the upper bound of T ypemap is defined to be ⊤

ub(T ypemap) = \{ \begin{matrix}max_{j} disp_{j} + sizeof(type_{j}) + ϵ & if no entry has basic type ub \\ max_{j}\{disp_{j} such that type_{j} = ub\} & otherwise \end{matrix}

Then

extent(T ypemap) = ub(T ypemap) − lb(T ypemap) ⊥

If type_{i} requires alignment to a byte address that is a multiple of k_{i}, then ϵ is the least nonnegative increment needed to round extent(T ypemap) to the next multiple of max_{i} k_{i}.

The formal definitions given for the various datatype constructors apply now, with the amended definition of extent. ⊤

The extent of a datatype should alwyas be nonnegative. It is erroneous to provide upper or lower bound markers that result in a negative extent. ⊥

The two functions below can be used for finding the lower bound and the upper bound of a datatype.

MPI TYPE LB( datatype, displacement)

IN  datatype  datatype (handle)
OUT  displacement  displacement of lower bound from origin, in bytes (in-

teger)

⊤

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

int MPI Type lb(MPI Datatype datatype, MPI Aint* displacement)

⊥

MPI TYPE LB( DATATYPE, DISPLACEMENT, IERROR)
    INTEGER DATATYPE, DISPLACEMENT, IERROR

MPI TYPE UB( datatype, displacement)

IN  datatype  datatype (handle)
OUT  displacement  displacement of upper bound from origin, in bytes (in-

teger)

⊤

int MPI Type ub(MPI Datatype datatype, MPI Aint* displacement)

⊥

MPI TYPE UB( DATATYPE, DISPLACEMENT, IERROR)
    INTEGER DATATYPE, DISPLACEMENT, IERROR

⊤

1.12.4 Commit and free

A datatype object has to be committed before it can be used in a communication. A committed datatype can still be used as a argument in datatype constructors. There is no need to commit basic datatypes. They are “pre-committed.”

MPI TYPE COMMIT(datatype)

INOUT datatype datatype that is committed (handle)

int MPI Type commit(MPI Datatype *datatype)
MPI TYPE COMMIT(DATATYPE, IERROR)
    INTEGER DATATYPE, IERROR

The commit operation commits the datatype, that is, the formal description of a communication buffer, not the content of that buffer. Thus, after a datatype has been committed, it can be repeatedly reused to communicate the changing content of a buffer or, indeed, the content of different buffers, with different starting addresses.

Advice to implementors. The system may “compile” at commit time an internal representation for the datatype that facilitates communication, e.g. change from a compacted representation to a flat representation of the datatype, and select the most convenient transfer mechanism. (End of advice to implementors.)

MPI TYPE FREE(datatype)

INOUT datatype datatype that is freed (handle)

int MPI Type free(MPI Datatype *datatype)

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

MPI TYPE FREE(DATATYPE, IERROR)
    INTEGER DATATYPE, IERROR

Marks the datatype object associated with datatype for deallocation and sets datatype to MPI DATATYPE NULL. Any communication that is currently using this datatype will complete normally. Derived datatypes that were defined from the freed datatype are not affected.

Example 1.26 The following code fragment gives examples of using MPI TYPE COMMIT.

INTEGER type1, type2
CALL MPI_TYPE_CONTIGUOUS(5, MPI_REAL, type1, ierr)
              ! new type object created
CALL MPI_TYPE_COMMIT(type1, ierr)
              ! now type1 can be used for communication
type2 = type1
              ! type2 can be used for communication
              ! (it is a handle to same object as type1)
CALL MPI_TYPE_VECTOR(3, 5, 4, MPI_REAL, type1, ierr)
              ! new uncommitted type object created
CALL MPI_TYPE_COMMIT(type1, ierr)
              ! now type1 can be used anew for communication

Freeing a datatype does not affect any other datatype that was built from the freed datatype. The system behaves as if input datatype arguments to derived datatype constructors are passed by value.

Advice to implementors. The implementation may keep a reference count of active communications that use the datatype, in order to decide when to free it. Also, one may implement constructors of derived datatypes so that they keep pointers to their datatype arguments, rather then copying them. In this case, one needs to keep track of active datatype definition references in order to know when a datatype object can be freed. (End of advice to implementors.)

1.12.5 Use of general datatypes in communication

Handles to derived datatypes can be passed to a communication call wherever a datatype argument is required. A call of the form MPI SEND(buf, count, datatype , ...), where count > 1, is interpreted as if the call was passed a new datatype which is the concatenation of count copies of datatype. Thus, MPI SEND(buf, count, datatype, dest, tag, comm) is equivalent to,

MPI_TYPE_CONTIGUOUS(count, datatype, newtype)
MPI_TYPE_COMMIT(newtype)
MPI_SEND(buf, 1, newtype, dest, tag, comm).

Similar statements apply to all other communication functions that have a count and datatype argument.

Suppose that a send operation MPI SEND(buf, count, datatype, dest, tag, comm) is executed, where datatype has type map,

{(type_{0}, disp_{0}), ..., (type_{n−1}, disp_{n−1})},

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

and extent extent. (Empty entries of “pseudo-type” MPI UB and MPI LB are not listed in the type map, but they affect the value of extent.) The send operation sends n · count entries, where entry i · n + j is at location addr_{i,j} = buf + extent · i + disp_{j} and has type type_{j}, for i = 0, ..., count − 1 and j = 0, ..., n − 1. These entries need not be contiguous, nor distinct; their order can be arbitrary.

The variable stored at address addr_{i,j} in the calling program should be of a type that matches type_{j}, where type matching is defined as in section 1.3.1. The message sent contains n · count entries, where entry i · n + j has type type_{j}.

Similarly, suppose that a receive operation MPI RECV(buf, count, datatype, source, tag, comm, status) is executed, where datatype has type map,

{(type_{0}, disp_{0}), ..., (type_{n−1}, disp_{n−1})},

with extent extent. (Again, empty entries of “pseudo-type” MPI UB and MPI LB are not listed in the type map, but they affect the value of extent.) This receive operation receives n · count entries, where entry i · n + j is at location buf + extent · i + disp_{j} and has type type_{j}. If the incoming message consists of k elements, then we must have k ≤ n · count; the i · n + j-th element of the message should have a type that matches type_{j}.

Type matching is defined according to the type signature of the corresponding datatypes, that is, the sequence of basic type components. Type matching does not depend on some aspects of the datatype definition, such as the displacements (layout in memory) or the intermediate types used.

Example 1.27 This example shows that type matching is defined in terms of the basic types that a derived type consists of.

...
CALL MPI_TYPE_CONTIGUOUS( 2, MPI_REAL, type2, ...)
CALL MPI_TYPE_CONTIGUOUS( 4, MPI_REAL, type4, ...)
CALL MPI_TYPE_CONTIGUOUS( 2, type2, type22, ...)
...
CALL MPI_SEND( a, 4, MPI_REAL, ...)
CALL MPI_SEND( a, 2, type2, ...)
CALL MPI_SEND( a, 1, type22, ...)
CALL MPI_SEND( a, 1, type4, ...)
...
CALL MPI_RECV( a, 4, MPI_REAL, ...)
CALL MPI_RECV( a, 2, type2, ...)
CALL MPI_RECV( a, 1, type22, ...)
CALL MPI_RECV( a, 1, type4, ...)

Each of the sends matches any of the receives.

A datatype may specify overlapping entries. If such a datatype is used in a receive operation, that is, if some part of the receive buffer is written more than once by the receive operation, then the call is erroneous.

Suppose that MPI RECV(buf, count, datatype, dest, tag, comm, status) is executed, where datatype has type map,

{(type_{0}, disp_{0}), ..., (type_{n−1}, disp_{n−1})}.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

The received message need not fill all the receive buffer, nor does it need to fill a number of locations which is a multiple of n. Any number, k, of basic elements can be received, where 0 ≤ k ≤ count · n. The number of basic elements received can be retrieved from status using the query function MPI GET ELEMENTS.

MPI GET ELEMENTS( status, datatype, count)

IN  status  return status of receive operation (Status)
IN  datatype  datatype used by receive operation (handle)
OUT  count  number of received basic elements (integer)

int MPI Get elements(MPI Status status, MPI Datatype datatype, int *count)
MPI GET ELEMENTS(STATUS, DATATYPE, COUNT, IERROR)
    INTEGER STATUS(MPI STATUS SIZE), DATATYPE, COUNT, IERROR

The previously defined function, MPI GET COUNT (Sec. 1.2.5), has a different behavior. It returns the number of “top-level elements” received. In the previous example, MPI GET COUNT may return any integer value k, where 0 ≤ k ≤ count. If MPI GET COUNT returns k, then the number of basic elements received (and the value returned by MPI GET ELEMENTS) is n · k. If the number of basic elements received is not a multiple of n, that is, if the receive operation has not received an integral number of datatype “copies,” then MPI GET COUNT returns the value MPI UNDEFINED.

Example 1.28 Usage of MPI GET COUNT and MPI GET ELEMENT.

...
CALL MPI_TYPE_CONTIGUOUS(2, MPI_REAL, Type2, ierr)
CALL MPI_TYPE_COMMIT(Type2, ierr)
...
CALL MPI_COMM_RANK(comm, rank, ierr)
IF(rank.EQ.0) THEN
      CALL MPI_SEND(a, 2, MPI_REAL, 1, 0, comm, ierr)
      CALL MPI_SEND(a, 3, MPI_REAL, 1, 0, comm, ierr)
ELSE
      CALL MPI_RECV(a, 2, Type2, 0, 0, comm, stat, ierr)
      CALL MPI_GET_COUNT(stat, Type2, i, ierr)     ! returns i=1
      CALL MPI_GET_ELEMENTS(stat, Type2, i, ierr)  ! returns i=2
      CALL MPI_RECV(a, 2, Type2, 0, 0, comm, stat, ierr)
      CALL MPI_GET_COUNT(stat, Type2, i, ierr)     ! returns i=MPI_UNDEFINED
      CALL MPI_GET_ELEMENTS(stat, Type2, i, ierr)  ! returns i=3
END IF

The function MPI GET ELEMENTS can also be used after a probe to find the number of elements in the probed message. Note that the two functions MPI GET COUNT and MPI GET ELEMENTS return the same values when they are used with basic datatypes.

Rationale. The extension given to the definition of MPI GET COUNT seems natural: one would expect this function to return the value of the count argument, when

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

the receive buffer is filled. Sometimes datatype represents a basic unit of data one wants to transfer, for example, a record in an array of records (structures). One should be able to find out how many components were received without bothering to divide by the number of elements in each component. However, on other occasions, datatype is used to define a complex layout of data in the receiver memory, and does not represent a basic unit of data for transfers. In such cases, one needs to use the function MPI GET ELEMENTS. (End of rationale.)

Advice to implementors. The definition implies that a receive cannot change the value of storage outside the entries defined to compose the communication buffer. In particular, the definition implies that padding space in a structure should not be modified when such a structure is copied from one process to another. This would prevent the obvious optimization of copying the structure, together with the padding, as one contiguous block. The implementation is free to do this optimization when it does not impact the outcome of the computation. The user can “force” this optimization by explicitly including padding as part of the message. (End of advice to implementors.)

1.12.6 Correct use of addresses

Successively declared variables in C or Fortran are not necessarily stored at contiguous locations. Thus, care must be exercised that displacements do not cross from one variable to another. Also, in machines with a segmented address space, addresses are not unique and address arithmetic has some peculiar properties. Thus, the use of addresses, that is, displacements relative to the start address MPI BOTTOM, has to be restricted.

Variables belong to the same sequential storage if they belong to the same array, to the same COMMON block in Fortran, or to the same structure in C. Valid addresses are defined recursively as follows:

1. The function MPI ADDRESS returns a valid address, when passed as argument a variable of the calling program.

2. The buf argument of a communication function evaluates to a valid address, when passed as argument a variable of the calling program.

3. If v is a valid address, and i is an integer, then v+i is a valid address, provided v and v+i are in the same sequential storage.

4. If v is a valid address then MPI BOTTOM + v is a valid address.

A correct program uses only valid addresses to identify the locations of entries in communication buffers. Furthermore, if u and v are two valid addresses, then the (integer) difference u - v can be computed only if both u and v are in the same sequential storage. No other arithmetic operations can be meaningfully executed on addresses.

The rules above impose no constraints on the use of derived datatypes, as long as they are used to define a communication buffer that is wholly contained within the same sequential storage. However, the construction of a communication buffer that contains variables that are not within the same sequential storage must obey certain restrictions. Basically, a communication buffer with variables that are not within the same sequential storage can be used only by specifying in the communication call buf = MPI BOTTOM, count = 1, and using a datatype argument where all displacements are valid (absolute) addresses.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

Advice to users. It is not expected that MPI implementations will be able to detect erroneous, “out of bound” displacements — unless those overflow the user address space — since the MPI call may not know the extent of the arrays and records in the host program. (End of advice to users.)

Advice to implementors. There is no need to distinguish (absolute) addresses and (relative) displacements on a machine with contiguous address space: MPI BOTTOM is zero, and both addresses and displacements are integers. On machines where the distinction is required, addresses are recognized as expressions that involve MPI BOTTOM. (End of advice to implementors.)

1.12.7 Examples

The following examples illustrate the use of derived datatypes.

Example 1.29 Send and receive a section of a 3D array.

REAL a(100,100,100), e(9,9,9)
INTEGER oneslice, twoslice, threeslice, sizeofreal, myrank, ierr
INTEGER status(MPI_STATUS_SIZE)
C      extract the section a(1:17:2, 3:11, 2:10)
C      and store it in e(:,:,:).
CALL MPI_COMM_RANK(MPI_COMM_WORLD, myrank)
CALL MPI_TYPE_EXTENT( MPI_REAL, sizeofreal, ierr)
C     create datatype for a 1D section
CALL MPI_TYPE_VECTOR( 9, 1, 2, MPI_REAL, oneslice, ierr)
C     create datatype for a 2D section
CALL MPI_TYPE_HVECTOR(9, 1, 100*sizeofreal, oneslice, twoslice, ierr)
C     create datatype for the entire section
CALL MPI_TYPE_HVECTOR( 9, 1, 100*100*sizeofreal, twoslice, 1,
                       threeslice, ierr)
CALL MPI_TYPE_COMMIT( threeslice, ierr)
CALL MPI_SENDRECV(a(1,3,2), 1, threeslice, myrank, 0, e, 9*9*9,
                  MPI_REAL, myrank, 0, MPI_COMM_WORLD, status, ierr)

Example 1.30 Copy the (strictly) lower triangular part of a matrix.

REAL a(100,100), b(100,100)
INTEGER  disp(100), blocklen(100), ltype, myrank, ierr
INTEGER status(MPI_STATUS_SIZE)
C     copy lower triangular part of array a

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

C     onto lower triangular part of array b
CALL MPI_COMM_RANK(MPI_COMM_WORLD, myrank)
C     compute start and size of each column
DO i=1, 100
  disp(i) = 100*(i-1) + i
  block(i) = 100-i
END DO
C     create datatype for lower triangular part
CALL MPI_TYPE_INDEXED( 100, block, disp, MPI_REAL, ltype, ierr)
CALL MPI_TYPE_COMMIT(ltype, ierr)
CALL MPI_SENDRECV( a, 1, ltype, myrank, 0, b, 1,
              ltype, myrank, 0, MPI_COMM_WORLD, status, ierr)

Example 1.31 Transpose a matrix.

REAL a(100,100), b(100,100)
INTEGER row, xpose, sizeofreal, myrank, ierr
INTEGER status(MPI_STATUS_SIZE)
C     transpose matrix a onto b
CALL MPI_COMM_RANK(MPI_COMM_WORLD, myrank)
CALL MPI_TYPE_EXTENT( MPI_REAL, sizeofreal, ierr)
C     create datatype for one row
CALL MPI_TYPE_VECTOR( 100, 1, 100, MPI_REAL, row, ierr)
C     create datatype for matrix in row-major order
CALL MPI_TYPE_HVECTOR( 100, 1, sizeofreal, row, xpose, ierr)
CALL MPI_TYPE_COMMIT( xpose, ierr)
C     send matrix in row-major order and receive in column major order
CALL MPI_SENDRECV( a, 1, xpose, myrank, 0, b, 100*100,
          MPI_REAL, myrank, 0, MPI_COMM_WORLD, status, ierr)

Example 1.32 Another approach to the transpose problem:

REAL a(100,100), b(100,100)
INTEGER  disp(2), blocklen(2), type(2), row, row1, sizeofreal
INTEGER  myrank, ierr
INTEGER status(MPI_STATUS_SIZE)
CALL MPI_COMM_RANK(MPI_COMM_WORLD, myrank)

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

C     transpose matrix a onto b
      CALL MPI_TYPE_EXTENT( MPI_REAL, sizeofreal, ierr)
C     create datatype for one row
      CALL MPI_TYPE_VECTOR( 100, 1, 100, MPI_REAL, row, ierr)
C     create datatype for one row, with the extent of one real number
      disp(1) = 0
      disp(2) = sizeofreal
      type(1)  = row
      type(2)  = MPI_UB
      blocklen(1)  = 1
      blocklen(2)  = 1
      CALL MPI_TYPE_STRUCT( 2, blocklen, disp, type, row1, ierr)
      CALL MPI_TYPE_COMMIT( row1, ierr)
C     send 100 rows and receive in column major order
      CALL MPI_SENDRECV( a, 100, row1, myrank, 0, b, 100*100,
                MPI_REAL, myrank, 0, MPI_COMM_WORLD, status, ierr)

Example 1.33 We manipulate an array of structures.

struct Partstruct
   {
   int    class;  /* particle class */
   double d[6];   /* particle coordinates */
   char   b[7];   /* some additional information */
   };
struct Partstruct    particle[1000];
int                  i, dest, rank;
MPI_Comm     comm;

/* build datatype describing structure */
MPI_Datatype Particletype;
MPI_Datatype type[3] = {MPI_INT, MPI_DOUBLE, MPI_CHAR};
int          blocklen[3] = {1, 6, 7};
MPI_Aint     disp[3];
int          base;

/* compute displacements of structure components */

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

MPI_Address( particle, disp);
MPI_Address( particle[0].d, disp+1);
MPI_Address( particle[0].b, disp+2);
base = disp[0];
for (i=0; i <3; i++) disp[i] -= base;
MPI_Type_struct( 3, blocklen, disp, type, &Particletype);
   /* If compiler does padding in mysterious ways,
   the following may be safer */
MPI_Datatype type1[4] = {MPI_INT, MPI_DOUBLE, MPI_CHAR, MPI_UB};
int          blocklen1[4] = {1, 6, 7, 1};
MPI_Aint     disp1[4];
/* compute displacements of structure components */
MPI_Address( particle, disp1);
MPI_Address( particle[0].d, disp1+1);
MPI_Address( particle[0].b, disp1+2);
MPI_Address( particle+1, disp1+3);
base = disp1[0];
for (i=0; i <4; i++) disp1[i] -= base;
/* build datatype describing structure */
MPI_Type_struct( 4, blocklen1, disp1, type1, &Particletype);

              /* 4.1:
        send the entire array */
MPI_Type_commit( &Particletype);
MPI_Send( particle, 1000, Particletype, dest, tag, comm);

              /* 4.2:
        send only the entries of class zero particles,
        preceded by the number of such entries */
MPI_Datatype Zparticles;   /* datatype describing all particles
                              with class zero (needs to be recomputed
                              if classes change) */
MPI_Datatype Ztype;
MPI_Aint     zdisp[1000];
int zblock[1000], j, k;

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

int zzblock[2] = {1,1};
MPI_Aint     zzdisp[2];
MPI_Datatype zztype[2];
/* compute displacements of class zero particles */
j = 0;
for(i=0; i < 1000; i++)
  if (particle[i].class==0)
     {
     zdisp[j] = i;
     zblock[j] = 1;
     j++;
     }
/* create datatype for class zero particles  */
MPI_Type_indexed( j, zblock, zdisp, Particletype, &Zparticles);
/* prepend particle count */
MPI_Address(&j, zzdisp);
MPI_Address(particle, zzdisp+1);
zztype[0] = MPI_INT;
zztype[1] = Zparticles;
MPI_Type_struct(2, zzblock, zzdisp, zztype, &Ztype);
MPI_Type_commit( &Ztype);
MPI_Send( MPI_BOTTOM, 1, Ztype, dest, tag, comm);

       /* A probably more efficient way of defining Zparticles */
/* consecutive particles with index zero are handled as one block */
j=0;
for (i=0; i < 1000; i++)
  if (particle[i].index==0)
    {
    for (k=i+1; (k < 1000)&&(particle[k].index == 0) ; k++);
    zdisp[j] = i;
    zblock[j] = k-i;
    j++;
    i = k;
    }
MPI_Type_indexed( j, zblock, zdisp, Particletype, &Zparticles);

                /* 4.3:
          send the first two coordinates of all entries */
MPI_Datatype Allpairs;      /* datatype for all pairs of coordinates */

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

MPI_Aint sizeofentry;
MPI_Type_extent( Particletype, &sizeofentry);
     /* sizeofentry can also be computed by subtracting the address
        of particle[0] from the address of particle[1] */
MPI_Type_hvector( 1000, 2, sizeofentry, MPI_DOUBLE, &Allpairs);
MPI_Type_commit( &Allpairs);
MPI_Send( particle[0].d, 1, Allpairs, dest, tag, comm);
      /* an alternative solution to 4.3 */
MPI_Datatype Onepair;   /* datatype for one pair of coordinates, with
                          the extent of one particle entry */
MPI_Aint disp2[3];
MPI_Datatype type2[3] = {MPI_LB, MPI_DOUBLE, MPI_UB};
int blocklen2[3] = {1, 2, 1};
MPI_Address( particle, disp2);
MPI_Address( particle[0].d, disp2+1);
MPI_Address( particle+1, disp2+2);
base = disp2[0];
for (i=0; i<2; i++) disp2[i] -= base;
MPI_Type_struct( 3, blocklen2, disp2, type2, &Onepair);
MPI_Type_commit( &Onepair);
MPI_Send( particle[0].d, 1000, Onepair, dest, tag, comm);

Example 1.34 The same manipulations as in the previous example, but use absolute addresses in datatypes.

struct Partstruct
   {
   int class;
   double d[6];
   char b[7];
   };
struct Partstruct particle[1000];
           /* build datatype describing first array entry */
MPI_Datatype Particletype;
MPI_Datatype type[3] = {MPI_INT, MPI_DOUBLE, MPI_CHAR};
int          block[3] = {1, 6, 7};

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

MPI_Aint     disp[3];
MPI_Address( particle, disp);
MPI_Address( particle[0].d, disp+1);
MPI_Address( particle[0].b, disp+2);
MPI_Type_struct( 3, block, disp, type, &Particletype);
/* Particletype describes first array entry -- using absolute
   addresses */
                  /* 5.1:
            send the entire array */
MPI_Type_commit( &Particletype);
MPI_Send( MPI_BOTTOM, 1000, Particletype, dest, tag, comm);

                 /* 5.2:
         send the entries of class zero,
         preceded by the number of such entries */
MPI_Datatype Zparticles, Ztype;
MPI_Aint zdisp[1000]
int zblock[1000], i, j, k;
int zzblock[2] = {1,1};
MPI_Datatype zztype[2];
MPI_Aint     zzdisp[2];
j=0;
for (i=0; i < 1000; i++)
  if (particle[i].index==0)
    {
    for (k=i+1; (k < 1000)&&(particle[k].index = 0) ; k++);
    zdisp[j] = i;
    zblock[j] = k-i;
    j++;
    i = k;
    }
MPI_Type_indexed( j, zblock, zdisp, Particletype, &Zparticles);
/* Zparticles describe particles with class zero, using
   their absolute addresses*/
/* prepend particle count */
MPI_Address(&j, zzdisp);
zzdisp[1] = MPI_BOTTOM;
zztype[0] = MPI_INT;
zztype[1] = Zparticles;

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

MPI_Type_struct(2, zzblock, zzdisp, zztype, &Ztype);
MPI_Type_commit( &Ztype);
MPI_Send( MPI_BOTTOM, 1, Ztype, dest, tag, comm);

Example 1.35 Handling of unions.

union {
   int     ival;
   float   fval;
      } u[1000]
int     utype;
/* All entries of u have identical type; variable
   utype keeps track of their current type */
MPI_Datatype   type[2];
int            blocklen[2] = {1,1};
MPI_Aint       disp[2];
MPI_Datatype   mpi_utype[2];
MPI_Aint       i,j;
/* compute an MPI datatype for each possible union type;
   assume values are left-aligned in union storage. */
MPI_Address( u, &i);
MPI_Address( u+1, &j);
disp[0] = 0; disp[1] = j-i;
type[1] = MPI_UB;
type[0] = MPI_INT;
MPI_Type_struct(2, blocklen, disp, type, &mpi_utype[0]);
type[0] = MPI_FLOAT;
MPI_Type_struct(2, blocklen, disp, type, &mpi_utype[1]);
for(i=0; i<2; i++) MPI_Type_commit(&mpi_utype[i]);
/* actual communication */
MPI_Send(u, 1000, mpi_utype[utype], dest, tag, comm);

1.13 Pack and unpack

Some existing communication libraries provide pack/unpack functions for sending noncontiguous data. In these, the user explicitly packs data into a contiguous buffer before sending

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

it, and unpacks it from a contiguous buffer after receiving it. Derived datatypes, which are described in Section 1.12, allow one, in most cases, to avoid explicit packing and unpacking. The user specifies the layout of the data to be sent or received, and the communication library directly accesses a noncontiguous buffer. The pack/unpack routines are provided for compatibility with previous libraries. Also, they provide some functionality that is not otherwise available in MPI. For instance, a message can be received in several parts, where the receive operation done on a later part may depend on the content of a former part. Another use is that outgoing messages may be explicitly buffered in user supplied space, thus overriding the system buffering policy. Finally, the availability of pack and unpack operations facilitates the development of additional communication libraries layered on top of MPI.

MPI PACK(inbuf, incount, datatype, outbuf, outcount, position, comm)

IN  inbuf  input buffer start (choice)
IN  incount  number of input data items (integer)
IN  datatype  datatype of each input data item (handle)
OUT  outbuf  output buffer start (choice)
IN  outcount  output buffer size, in bytes (integer)

INOUT position current position in buffer, in bytes (integer)

IN  comm  communicator for packed message (handle)

int MPI Pack(void* inbuf, int incount, MPI Datatype datatype, void *outbuf,
             int outcount, int *position, MPI Comm comm)
MPI PACK(INBUF, INCOUNT, DATATYPE, OUTBUF, OUTCOUNT, POSITION, COMM,
             IERROR)
    <type> INBUF(*), OUTBUF(*)
    INTEGER INCOUNT, DATATYPE, OUTCOUNT, POSITION, COMM, IERROR

Packs the message in the send buffer specified by inbuf, incount, datatype into the buffer space specified by outbuf and outcount. The input buffer can be any communication buffer allowed in MPI SEND. The output buffer is a contiguous storage area containing outcount bytes, starting at the address outbuf (length is counted in bytes, not elements, as if it were a communication buffer for a message of type MPI PACKED).

The input value of position is the first location in the output buffer to be used for packing. position is incremented by the size of the packed message, and the output value of position is the first location in the output buffer following the locations occupied by the packed message. The comm argument is the communicator that will be subsequently used for sending the packed message.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

MPI UNPACK(inbuf, insize, position, outbuf, outcount, datatype, comm)

IN  inbuf  input buffer start (choice)
IN  insize  size of input buffer, in bytes (integer)

INOUT position current position in bytes (integer)

OUT  outbuf  output buffer start (choice)
IN  outcount  number of items to be unpacked (integer)
IN  datatype  datatype of each output data item (handle)
IN  comm  communicator for packed message (handle)

int MPI Unpack(void* inbuf, int insize, int *position, void *outbuf,
             int outcount, MPI Datatype datatype, MPI Comm comm)
MPI UNPACK(INBUF, INSIZE, POSITION, OUTBUF, OUTCOUNT, DATATYPE, COMM,
             IERROR)
    <type> INBUF(*), OUTBUF(*)
    INTEGER INSIZE, POSITION, OUTCOUNT, DATATYPE, COMM, IERROR

Unpacks a message into the receive buffer specified by outbuf, outcount, datatype from the buffer space specified by inbuf and insize. The output buffer can be any communication buffer allowed in MPI RECV. The input buffer is a contiguous storage area containing insize bytes, starting at address inbuf. The input value of position is the first location in the output buffer occupied by the packed message. position is incremented by the size of the packed message, so that the output value of position is the first location in the output buffer after the locations occupied by the message that was unpacked. comm is the communicator used to receive the packed message.

Advice to users. Note the difference between MPI RECV and MPI UNPACK: in MPI RECV, the count argument specifies the maximum number of items that can be received. The actual number of items received is determined by the length of the incoming message. In MPI UNPACK, the count argument specifies the actual number of items that are unpacked; the “size” of the corresponding message is the increment in position. The reason for this change is that the “incoming message size” is not predetermined since the user decides how much to unpack; nor is it easy to determine the “message size” from the number of items to be unpacked. In fact, in a heterogeneous system, this number may not be determined a priori. (End of advice to users.)

To understand the behavior of pack and unpack, it is convenient to think of the data part of a message as being the sequence obtained by concatenating the successive values sent in that message. The pack operation stores this sequence in the buffer space, as if sending the message to that buffer. The unpack operation retrieves this sequence from buffer space, as if receiving a message from that buffer. (It is helpful to think of internal Fortran files or sscanf in C, for a similar function.)

Several messages can be successively packed into one packing unit. This is effected by several successive related calls to MPI PACK, where the first call provides position = 0, and each successive call inputs the value of position that was output by the previous call, and the same values for outbuf, outcount and comm. This packing unit now contains the

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

equivalent information that would have been stored in a message by one send call with a send buffer that is the “concatenation” of the individual send buffers.

A packing unit can be sent using type MPI PACKED. Any point to point or collective communication function can be used to move the sequence of bytes that forms the packing unit from one process to another. This packing unit can now be received using any receive operation, with any datatype: the type matching rules are relaxed for messages sent with type MPI PACKED.

A message sent with any type (including MPI PACKED) can be received using the type MPI PACKED. Such a message can then be unpacked by calls to MPI UNPACK.

A packing unit (or a message created by a regular, “typed” send) can be unpacked into several successive messages. This is effected by several successive related calls to MPI UNPACK, where the first call provides position = 0, and each successive call inputs the value of position that was output by the previous call, and the same values for inbuf, insize and comm.

The concatenation of two packing units is not necessarily a packing unit; nor is a substring of a packing unit necessarily a packing unit. Thus, one cannot concatenate two packing units and then unpack the result as one packing unit; nor can one unpack a substring of a packing unit as a separate packing unit. Each packing unit, that was created by a related sequence of pack calls, or by a regular send, must be unpacked as a unit, by a sequence of related unpack calls.

Rationale. The restriction on “atomic” packing and unpacking of packing units allows the implementation to add at the head of packing units additional information, such as a description of the sender architecture (to be used for type conversion, in a heterogeneous environment) (End of rationale.)

The following call allows the user to find out how much space is needed to pack a message and, thus, manage space allocation for buffers.

MPI PACK SIZE(incount, datatype, comm, size)

IN  incount  count argument to packing call (integer)
IN  datatype  datatype argument to packing call (handle)
IN  comm  communicator argument to packing call (handle)
OUT  size  upper bound on size of packed message, in bytes (in-

teger)

int MPI Pack size(int incount, MPI Datatype datatype, MPI Comm comm,
             int *size)
MPI PACK SIZE(INCOUNT, DATATYPE, COMM, SIZE, IERROR)
    INTEGER INCOUNT, DATATYPE, COMM, SIZE, IERROR

A call to MPI PACK SIZE(incount, datatype, comm, size) returns in size an upper bound on the increment in position that is effected by a call to MPI PACK(inbuf, incount, datatype, outbuf, outcount, position, comm).

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

Rationale. The call returns an upper bound, rather than an exact bound, since the exact amount of space needed to pack the message may depend on the context (e.g., first message packed in a packing unit may take more space). (End of rationale.)

Example 1.36 An example using MPI PACK.

int position, i, j, a[2];
char buff[1000];
....
MPI_Comm_rank(MPI_COMM_WORLD, &myrank);
if (myrank == 0)
{
   / * SENDER CODE */
  position = 0;
  MPI_Pack(&i, 1, MPI_INT, buff, 1000, &position, MPI_COMM_WORLD);
  MPI_Pack(&j, 1, MPI_INT, buff, 1000, &position, MPI_COMM_WORLD);
  MPI_Send( buff, position, MPI_PACKED, 1, 0, MPI_COMM_WORLD);
}
else  /* RECEIVER CODE */
  MPI_Recv( a, 2, MPI_INT, 0, 0, MPI_COMM_WORLD)
}

Example 1.37 A elaborate example.

int position, i;
float a[1000];
char buff[1000]
....
MPI_Comm_rank(MPI_Comm_world, &myrank);
if (myrank == 0)
{
  / * SENDER CODE */
  int len[2];
  MPI_Aint disp[2];
  MPI_Datatype type[2], newtype;
  /* build datatype for i followed by a[0]...a[i-1] */
  len[0] = 1;
  len[1] = i;
  MPI_Address( &i, disp);
  MPI_Address( a, disp+1);

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

  type[0] = MPI_INT;
  type[1] = MPI_FLOAT;
  MPI_Type_struct( 2, len, disp, type, &newtype);
  MPI_Type_commit( &newtype);
  /* Pack i followed by a[0]...a[i-1]*/
  position = 0;
  MPI_Pack( MPI_BOTTOM, 1, newtype, buff, 1000, &position, MPI_COMM_WORLD);
  /* Send */
  MPI_Send( buff, position, MPI_PACKED, 1, 0,
            MPI_COMM_WORLD)
/* *****
   One can replace the last three lines with
   MPI_Send( MPI_BOTTOM, 1, newtype, 1, 0, MPI_COMM_WORLD);
   ***** */
}
else /* myrank == 1 */
{
   /* RECEIVER CODE */
  MPI_Status status;
  /* Receive */
  MPI_Recv( buff, 1000, MPI_PACKED, 0, 0, &status);
  /* Unpack i */
 position = 0;
 MPI_Unpack(buff, 1000, &position, &i, 1, MPI_INT, MPI_COMM_WORLD);
 /* Unpack a[0]...a[i-1] */
 MPI_Unpack(buff, 1000, &position, a, i, MPI_FLOAT, MPI_COMM_WORLD);
}

Example 1.38 Each process sends a count, followed by count characters to the root; the root concatenate all characters into one string.

int count, gsize, counts[64], totalcount, k1, k2, k,
    displs[64], position, concat_pos;
char chr[100], *lbuf, *rbuf, *cbuf;
...
MPI_Comm_size(comm, &gsize);
MPI_Comm_rank(comm, &myrank);

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

    /* allocate local pack buffer */
MPI_Pack_size(1, MPI_INT, comm, &k1);
MPI_Pack_size(count, MPI_CHAR, &k2);
k = k1+k2;
lbuf = (char *)malloc(k);
    /* pack count, followed by count characters */
position = 0;
MPI_Pack(&count, 1, MPI_INT, &lbuf, k, &position, comm);
MPI_Pack(chr, count, MPI_CHAR, &lbuf, k, &position, comm);
if (myrank != root)
    /* gather at root sizes of all packed messages */
 MPI_Gather( &position, 1, MPI_INT, NULL, NULL,
           NULL, root, comm);
    /* gather at root packed messages */
MPI_Gatherv( &buf, position, MPI_PACKED, NULL,
         NULL, NULL, NULL, root, comm);
else {   /* root code */
    /* gather sizes of all packed messages */
 MPI_Gather( &position, 1, MPI_INT, counts, 1,
           MPI_INT, root, comm);
    /* gather all packed messages */
displs[0] = 0;
for (i=1; i < gsize; i++)
  displs[i] = displs[i-1] + counts[i-1];
totalcount = dipls[gsize-1] + counts[gsize-1];
rbuf = (char *)malloc(totalcount);
cbuf = (char *)malloc(totalcount);
MPI_Gatherv( lbuf, position, MPI_PACKED, rbuf,
         counts, displs, MPI_PACKED, root, comm);
    /* unpack all messages and concatenate strings */
concat_pos = 0;
for (i=0; i < gsize; i++) {
  position = 0;
  MPI_Unpack( rbuf+displs[i], totalcount-displs[i],
        &position, &count, 1, MPI_INT, comm);
  MPI_Unpack( rbuf+displs[i], totalcount-displs[i],
        &position, cbuf+concat_pos, count, MPI_CHAR, comm);
  concat_pos += count;
  }
cbuf[concat_pos] = `\0';
}

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47