Parallel execution¶

SODA provides MPI-based execution infrastructure for scientific workflows on a local machine or an HPC allocation. The selected workflow determines the scientific tasks. Choose the execution layout according to the number of tasks available, their CPU and memory requirements, and the calculator’s capabilities.

The commands below assume you have followed Preparing a run, activated the matching build, and changed to the run directory.

Master, heads, and workers¶

Role	Responsibility
Master	Runs the selected workflow’s control logic and coordinates task dispatch and result collection.
Head	Manages resources and task/result exchange on one node.
Worker	Executes workflow tasks, including calculator evaluations. EZGA reserves some worker ranks as persistent agents that manage its searches.

Each node has one head rank. One of those heads also serves as the global master, so the master does not require an additional rank beyond the node heads. With two workers on each of two nodes, the roles look like this:

Node A                              Node B
Head + global master  <---------->  Head
    |                                  |
    +-- Worker                         +-- Worker
    +-- Worker                         +-- Worker

Tasks and results pass through this hierarchy. A worker can execute many tasks over the lifetime of a run; a task does not require a new scheduler job.

Count ranks, cores, and devices¶

An MPI rank is a process. A CPU thread runs within a process, and a GPU is a device used by a compatible calculator. These are separate resources when specifying a launch.

The current SODA implementation supports one MPI rank per worker group: --numranksperwgroup=1. Allocating multiple CPU cores to a worker allows threaded calculator work where supported; it does not create additional MPI ranks inside that group.

For a uniform allocation with W worker ranks per node and N nodes:

MPI ranks per node = W + 1
Total MPI ranks    = N × (W + 1)

For EZGA, W includes the worker ranks reserved for agents. Subtract the agents on a node to obtain its calculator-worker count. For example, the three-rank FCC launch has one head/master, one agent, and one calculator worker. Agents occupy existing worker ranks rather than adding ranks beyond the formula.

If each rank is allocated C CPU cores, the launch-template helper expects total_cpus = C × (W + 1) per node. This count includes the head rank. For example, seven workers plus one head, each allocated two cores, request eight MPI ranks and sixteen CPU cores per node.

Measure before increasing the allocation

More workers allow more tasks to proceed concurrently when work is available. More cores per worker help only when the calculator uses them effectively. Start with an example’s layout and measure progress and memory use before increasing the allocation.

Launch the CPU example locally¶

The local commands below use Open MPI. Check mpirun --version after activating the build; the launcher must match the MPI libraries used to compile SODA. Other MPI implementations have their own placement options.

On a machine with at least three available physical CPU cores, the prepared AKMC/EAM single-adatom example can use two workers and a combined head/master. Keep max_kmc_steps = 20000 as supplied in Preparing a run. This lightweight benchmark is expected to find 36 states with four processes per state:

export OMP_NUM_THREADS=1
export OMP_PLACES=cores
export OMP_PROC_BIND=close
export OPENBLAS_NUM_THREADS=1
mpirun -n 3 --map-by slot:PE=1 --bind-to core --report-bindings \
  "$SODA_BIN" \
  --num-gpu-workers-per-node=0 \
  --storesummariesinfile=1

The summary option stores additional process-search summaries used in AKMC analysis. Select options appropriate to the workflow when adapting the command; the executable’s --help reports their scope.

Match worker threads to CPU cores¶

Open MPI places processes on CPUs. OpenMP runs threads within those processes. Setting OMP_NUM_THREADS=3 requests three OpenMP threads, but does not give the process three CPU cores. Likewise, mpirun -n 3 starts three MPI processes; it does not specify the number of cores per process.

For two workers and one head/master, each assigned three physical cores, use the following launch on a machine with at least nine available cores:

export OMP_NUM_THREADS=3
export OMP_PLACES=cores
export OMP_PROC_BIND=close
mpirun -n 3 --map-by slot:PE=3 --bind-to core --report-bindings \
  "$SODA_BIN" \
  --num-gpu-workers-per-node=0 \
  --storesummariesinfile=1

PE=3 assigns three cores to each MPI process. OMP_PLACES=cores and OMP_PROC_BIND=close place the OpenMP threads on those cores. This uniform layout assigns nine cores in total, including three for the head/master; it does not imply the head/master uses three busy threads. Keep any additional SODA options required by your run after the executable. The placement flags are documented in the Open MPI launch guide.

For an APE/AKMC GAP installation using LAMMPS’s OPENMP package, also match the potential’s setting in config.ini:

[Potential]
lammps_threads = 3

A positive lammps_threads requires that LAMMPS package. Leave this setting unset or zero for builds without it, including the native MACE GPU recipe. Calculators can also adjust their own thread pools, so check the runtime measurements after launch.

--report-bindings prints the assigned cores in the launcher’s output. For the three-core example, expect separate groups such as core:0-2, core:3-5, and core:6-8. Linux affinity masks use logical CPU IDs: on hardware with two hardware threads per core, three physical cores can appear as six logical CPU IDs. Conversely, a three-thread team restricted to two logical CPUs on one physical core still shares that single core.

Local Open MPI 4.1 window errors

If a single-node run with Open MPI 4.1 fails in MPI_Win_create with MPI_ERR_WIN, first activate the build’s modules.rc and check mpirun --version. The launcher and libraries must match the MPI installation used for the build. For this local window-backend failure, try:

mpirun -n 3 --map-by slot:PE=1 --bind-to core --report-bindings \
  --mca pml ob1 --mca btl self,tcp --mca osc sm,pt2pt \
  "$SODA_BIN" --num-gpu-workers-per-node=0 --storesummariesinfile=1

This workaround was checked with the single-adatom example on Open MPI 4.1.6. The pt2pt window component is unavailable in Open MPI 5; these options are specific to the Open MPI 4.1 setup and are not general cluster-launch defaults.

Prepare a cluster launch¶

An HPC scheduler reserves resources; the MPI launcher starts SODA within that allocation. The standalone analyzer can check the run directory and print a launch template. For an illustrative CPU allocation of one node with three workers and one head:

python "$SODA_ROOT/src/python/soda_analyzer.py" suggest-run-command . \
  --launcher slurm --profile generic \
  --nodes 1 --total-cpus 4 --num-worker 3 --cpus-per-worker 1 \
  --gpus-per-node 0 --num-gpu-workers 0 \
  --time 00:30:00 --soda-exe "$SODA_BIN" --check

This prints a suggestion; it does not submit a job. Add the account and partition required by your cluster, activate the selected build in the job script, and ensure the script runs from the prepared directory. Adapt core counts and time to your allocation policy and calculation.

For threaded workers, match #SBATCH --cpus-per-task and srun -c to the per-rank thread budget, and set OMP_NUM_THREADS, OMP_PLACES, and OMP_PROC_BIND in the batch script. Whether Slurm counts cores or hardware threads as CPUs depends on the site; check its placement guidance and verify the resulting affinity. The Open MPI --map-by options above belong to mpirun, not srun.

Here, --num-worker and --cpus-per-worker belong to the analyzer helper. They generate scheduler and launcher settings. They are not SODA executable options. Consult the launch-template reference for the supported profiles and generated resource settings.

When using the helper’s --launcher mpirun output, add the Open MPI mapping and thread settings yourself. That output currently specifies the rank count but does not add CPU binding or OpenMP environment settings from --cpus-per-worker.

Use GPUs with a compatible calculator¶

A GPU run needs a matching build, calculator/model configuration, and device allocation. Reserving GPUs alone does not move CPU-only tasks onto them.

In the helper, --gpus-per-node requests devices from Slurm, while --num-gpu-workers selects GPU-capable workers and maps to SODA’s --num-gpu-workers-per-node. Several workers can share a GPU when explicitly allowed, but their models and task data compete for device memory. The helper adds --allow_gpu_oversubscribe when the requested worker count exceeds the requested device count.

Begin with the matching workflow’s GPU example and check its model paths, device settings, and memory requirements. Detailed settings belong to the APE, EZGA, NEBScape, and AKMC guides.

Continue with Monitoring and outputs.