Data Distribution

ScaLAPACK is a distributed interface. Matrices are distributed across processes. These processes are arranged in a process grid, which is typically two-dimensional (2D), though one-dimensional (1D) process grids are also supported. The process grid is user-specified and stored within the context handle ctxt. When defining the grid, it must be configured via the BLACS interface.

    char layout='R'; // Block cyclic, Row major processor mapping
    int zero=0;
    blacs_get_(&zero, &zero, &ctxt); // -> Create context
    blacs_gridinit_(&ctxt, &layout, &P, &Q); // Context -> Initialize the grid
    blacs_gridinfo_(&ctxt, &nprow, &npcol, &myrow, &mycol); // Context -> Context grid info (# procs row/col, current procs row/col)
    …
    blacs_gridexit_(&ctxt);

blacs_get_ transfers the default context of the system to other BLACS functions to create a context. blacs_gridinit formally specifies P and Q of the process grid. Both P and Q are integers. Ensure that P x Q ≤ MPI procs. blacs_gridinfo obtains the position of the current process in the process grid, that is, the row and column numbers of the current process in the process grid. After all ScaLAPACK functions are called, the blacs_gridexit_ function can be used to release the BLACS process grid information.

On a 2D grid, if a process grid consists of P process rows and Q process columns, coordinates of a process on the grid may be represented as (p, q), where 0 ≤ p ≤ P, and 0 ≤ q ≤ Q. The following figure shows the process distribution of six nodes.

In the preceding figure, processes are mapped into a process grid in row-major order. Similarly, processes can also be mapped into a process grid in column-major order. By default, row-major order is used for mapping.

After a process grid is determined, a matrix is cyclically distributed to the process grid according to a preset block size m_b x n_b. The following figure shows the numbering of the processes where the blocks reside after the distribution.

When P or Q is set to 1, the process grid is set to 1D, and coordinates of a process in the grid may be set to p or q according to the process number, where 0 ≤ p ≤ P or 0 ≤ q ≤ Q.

Matrices of a same size are allocated to the 1D process grid. The following figure shows the matrix division and related process numbering of the column blocks.

You can also set Q to 1, so that data blocks on the process grid are divided by row.