Reading in meshes in parallel

Reading in meshes in parallel#

For more information about mesh-partitioning and mesh input, see: https://jsdokken.com/dolfinx_docs/meshes.html

Cell ownership#

For the remainder of this section we will consider a 3x3 unit square mesh:

from mpi4py import MPI

import ipyparallel as ipp

import dolfinx

domain = dolfinx.mesh.create_unit_square(MPI.COMM_WORLD, 3, 3)

The mesh consists of cells, edges and vertices. A mesh is created by supplying the information regarding the connectivity between the cells and the mesh nodes. As a higher order mesh has more nodes than vertices, we can get the connectivity between the cells and mesh vertices through the mesh-topology.

We start by creating a simple function for inspecting these outputs in serial and parallel

def inspect_mesh(shared_facet: bool = False):
    from mpi4py import MPI

    import dolfinx

    ghost_mode = dolfinx.mesh.GhostMode.shared_facet if shared_facet else dolfinx.mesh.GhostMode.none
    domain = dolfinx.mesh.create_unit_square(MPI.COMM_WORLD, 3, 3, ghost_mode=ghost_mode)
    topology = domain.topology
    tdim = topology.dim
    print(f"Number of cells in process: {topology.index_map(tdim).size_local}", flush=True)
    print(f"Number of shared cells: {topology.index_map(tdim).num_ghosts}", flush=True)
    print(f"Local range [{topology.index_map(tdim).local_range[0]}, {topology.index_map(tdim).local_range[1]})")
    cell_to_vertices = topology.connectivity(tdim, 0)
    print(cell_to_vertices)

We start by inspecting the outputs in serial

inspect_mesh()

Number of cells in process: 18

Number of shared cells: 0

Local range [0, 18)
<AdjacencyList> with 18 nodes
  0: [0 1 2 ]
  1: [0 3 2 ]
  2: [4 0 3 ]
  3: [3 2 5 ]
  4: [4 6 3 ]
  5: [3 7 5 ]
  6: [8 4 6 ]
  7: [6 3 7 ]
  8: [7 5 9 ]
  9: [8 10 6 ]
  10: [6 11 7 ]
  11: [7 12 9 ]
  12: [10 6 11 ]
  13: [11 7 12 ]
  14: [10 13 11 ]
  15: [11 14 12 ]
  16: [13 11 14 ]
  17: [13 15 14 ]

We observe that we have 18 cells in the mesh, that are connected to the 18 vertices. All indices start from 0, and can be mapped to its global owner by calling domain.topology.index_map(tdim).local_to_global([idx])

Parallel execution#

We use IPython-Parallel for running DOLFINx on multiple processes inside our script.

cluster = ipp.Cluster(engines="mpi", n=2)
rc = cluster.start_and_connect_sync()

Starting 2 engines with <class 'ipyparallel.cluster.launcher.MPIEngineSetLauncher'>

This has started to processes that can execute code with a MPI communicator. We run the mesh code on two processes, instructing DOLFINx not to share any cells between the processes

query = rc[:].apply_async(inspect_mesh, False)
query.wait()
query.display_outputs()

[stdout:0] 
Number of cells in process: 9
Number of shared cells: 0
Local range [0, 9)
<AdjacencyList> with 9 nodes
  0: [0 7 8 ]
  1: [0 1 8 ]
  2: [1 8 2 ]
  3: [1 3 2 ]
  4: [8 2 4 ]
  5: [3 2 5 ]
  6: [2 4 9 ]
  7: [3 6 5 ]
  8: [2 5 9 ]

[stdout:1] 
Number of cells in process: 9
Number of shared cells: 0
Local range [9, 18)
<AdjacencyList> with 9 nodes
  0: [0 1 2 ]
  1: [0 3 2 ]
  2: [4 0 3 ]
  3: [3 2 5 ]
  4: [4 6 3 ]
  5: [3 9 5 ]
  6: [6 3 9 ]
  7: [9 5 7 ]
  8: [9 8 7 ]

Next, we instruct DOLFINx to share cells between two processes if a facet is shared between the two processes.

new_query = rc[:].apply_async(inspect_mesh, True)
new_query.wait()
new_query.display_outputs()

[stdout:0] 
Number of cells in process: 9
Number of shared cells: 3
Local range [0, 9)
<AdjacencyList> with 12 nodes
  0: [0 7 8 ]
  1: [0 1 8 ]
  2: [1 8 2 ]
  3: [1 3 2 ]
  4: [8 2 4 ]
  5: [3 2 5 ]
  6: [2 4 9 ]
  7: [3 6 5 ]
  8: [2 5 9 ]
  9: [7 8 10 ]
  10: [8 10 4 ]
  11: [4 9 11 ]

[stdout:1] 
Number of cells in process: 9
Number of shared cells: 3
Local range [9, 18)
<AdjacencyList> with 12 nodes
  0: [0 1 2 ]
  1: [0 3 2 ]
  2: [4 0 3 ]
  3: [3 2 5 ]
  4: [4 6 3 ]
  5: [3 9 5 ]
  6: [6 3 9 ]
  7: [9 5 7 ]
  8: [9 8 7 ]
  9: [10 4 6 ]
  10: [6 11 9 ]
  11: [11 9 8 ]

cluster.stop_cluster_sync()
Hide code cell output 
Stopping controller

Controller stopped: {'exit_code': 0, 'pid': 2624, 'identifier': 'ipcontroller-1735911823-tkl8-2604'}

Stopping engine(s): 1735911824

Output for ipengine-1735911823-tkl8-1735911824-2604:
2025-01-03 13:43:45.093 [IPEngine] Loading connection info from $IPP_CONNECTION_INFO
2025-01-03 13:43:45.093 [IPEngine] WARNING | Not using CurveZMQ security
2025-01-03 13:43:45.097 [IPEngine] Loading connection info from $IPP_CONNECTION_INFO
2025-01-03 13:43:45.097 [IPEngine] WARNING | Not using CurveZMQ security
2025-01-03 13:43:45.118 [IPEngine.0] Registering with controller at tcp://127.0.0.1:59307
2025-01-03 13:43:45.118 [IPEngine.1] Registering with controller at tcp://127.0.0.1:59307
2025-01-03 13:43:45.118 [IPEngine.0] Requesting id: 0
2025-01-03 13:43:45.118 [IPEngine.1] Requesting id: 1
2025-01-03 13:43:45.120 [IPEngine.1.1] Shell_addrs: ['tcp://127.0.0.1:33875', 'tcp://127.0.0.1:42469', 'tcp://127.0.0.1:43481']
2025-01-03 13:43:45.121 [IPEngine.1.1] Connecting shell to tcp://127.0.0.1:33875
2025-01-03 13:43:45.121 [IPEngine.1.1] Connecting shell to tcp://127.0.0.1:42469
2025-01-03 13:43:45.121 [IPEngine.1.1] Connecting shell to tcp://127.0.0.1:43481
2025-01-03 13:43:45.121 [IPEngine.1.1] Starting nanny
2025-01-03 13:43:45.121 [IPEngine.0.0] Shell_addrs: ['tcp://127.0.0.1:33875', 'tcp://127.0.0.1:42469', 'tcp://127.0.0.1:49435']
2025-01-03 13:43:45.122 [IPEngine.0.0] Connecting shell to tcp://127.0.0.1:33875
2025-01-03 13:43:45.122 [IPEngine.0.0] Connecting shell to tcp://127.0.0.1:42469
2025-01-03 13:43:45.122 [IPEngine.0.0] Connecting shell to tcp://127.0.0.1:49435
2025-01-03 13:43:45.122 [IPEngine.0.0] Starting nanny
2025-01-03 13:43:45.552 [KernelNanny.0] Starting kernel nanny for engine 0, pid=2657, nanny pid=2670
2025-01-03 13:43:45.552 [KernelNanny.0] Nanny watching parent pid 2657.
2025-01-03 13:43:45.552 [KernelNanny.1] Starting kernel nanny for engine 1, pid=2658, nanny pid=2669
2025-01-03 13:43:45.552 [KernelNanny.1] Nanny watching parent pid 2658.
2025-01-03 13:43:45.621 [IPEngine.0.0] Running code in user namespace: 
from mpi4py import MPI
mpi_rank = MPI.COMM_WORLD.Get_rank()
mpi_size = MPI.COMM_WORLD.Get_size()

2025-01-03 13:43:45.621 [IPEngine.1.1] Running code in user namespace: 
from mpi4py import MPI
mpi_rank = MPI.COMM_WORLD.Get_rank()
mpi_size = MPI.COMM_WORLD.Get_size()

2025-01-03 13:43:45.623 [IPEngine.0.0] WARNING | debugpy_stream undefined, debugging will not be enabled
2025-01-03 13:43:45.623 [IPEngine.1.1] WARNING | debugpy_stream undefined, debugging will not be enabled
2025-01-03 13:43:45.629 [IPEngine.0.0] Starting to monitor the heartbeat signal from the hub every 3500 ms.
2025-01-03 13:43:45.629 [IPEngine.1.1] Starting to monitor the heartbeat signal from the hub every 3500 ms.
2025-01-03 13:43:45.629 [IPEngine.1.1] Completed registration with id 1
2025-01-03 13:43:45.629 [IPEngine.0.0] Completed registration with id 0
2025-01-03 13:43:50.351 [IPEngine.0.0] Handling apply_request: e5a1eb34-d2de6aabc08d3a2d14c45593_2604_1
2025-01-03 13:43:50.352 [IPEngine.1.1] Handling apply_request: e5a1eb34-d2de6aabc08d3a2d14c45593_2604_2
2025-01-03 13:43:50.541 [IPEngine.0.0] Handling apply_request: e5a1eb34-d2de6aabc08d3a2d14c45593_2604_3
2025-01-03 13:43:50.541 [IPEngine.1.1] Handling apply_request: e5a1eb34-d2de6aabc08d3a2d14c45593_2604_4

engine set stopped 1735911824: {'exit_code': 59, 'pid': 2652, 'identifier': 'ipengine-1735911823-tkl8-1735911824-2604'}