# Parallel execution SODA provides MPI-based execution infrastructure for scientific workflows on a local machine or an HPC allocation. The selected workflow determines the scientific tasks. Choose the execution layout according to the number of tasks available, their CPU and memory requirements, and the calculator's capabilities. The commands below assume you have followed [Preparing a run](preparing.md), activated the matching build, and changed to the run directory. ## Master, heads, and workers | Role | Responsibility | | --- | --- | | Master | Runs the selected workflow's control logic and coordinates task dispatch and result collection. | | Head | Manages resources and task/result exchange on one node. | | Worker | Executes workflow tasks, including calculator evaluations. EZGA reserves some worker ranks as persistent agents that manage its searches. | Each node has one head rank. One of those heads also serves as the global master, so the master does not require an additional rank beyond the node heads. With two workers on each of two nodes, the roles look like this: ```text Node A Node B Head + global master <----------> Head | | +-- Worker +-- Worker +-- Worker +-- Worker ``` Tasks and results pass through this hierarchy. A worker can execute many tasks over the lifetime of a run; a task does not require a new scheduler job. ## Count ranks, cores, and devices An **MPI rank** is a process. A **CPU thread** runs within a process, and a **GPU** is a device used by a compatible calculator. These are separate resources when specifying a launch. The current SODA implementation supports one MPI rank per worker group: `--numranksperwgroup=1`. Allocating multiple CPU cores to a worker allows threaded calculator work where supported; it does not create additional MPI ranks inside that group. For a uniform allocation with `W` worker ranks per node and `N` nodes: ```text MPI ranks per node = W + 1 Total MPI ranks = N × (W + 1) ``` For EZGA, `W` includes the worker ranks reserved for agents. Subtract the agents on a node to obtain its calculator-worker count. For example, the [three-rank FCC launch](ezga-fcc.md#2-check-and-launch) has one head/master, one agent, and one calculator worker. Agents occupy existing worker ranks rather than adding ranks beyond the formula. If each rank is allocated `C` CPU cores, the launch-template helper expects `total_cpus = C × (W + 1)` per node. This count includes the head rank. For example, seven workers plus one head, each allocated two cores, request eight MPI ranks and sixteen CPU cores per node. :::{admonition} Measure before increasing the allocation :class: tip More workers allow more tasks to proceed concurrently when work is available. More cores per worker help only when the calculator uses them effectively. Start with an example's layout and measure progress and memory use before increasing the allocation. ::: ## Launch the CPU example locally The local commands below use **Open MPI**. Check `mpirun --version` after activating the build; the launcher must match the MPI libraries used to compile SODA. Other MPI implementations have their own placement options. On a machine with at least three available physical CPU cores, the prepared AKMC/EAM single-adatom example can use two workers and a combined head/master. Keep `max_kmc_steps = 20000` as supplied in [Preparing a run](preparing.md#configure-the-workflow). This lightweight benchmark is expected to find 36 states with four processes per state: ```bash export OMP_NUM_THREADS=1 export OMP_PLACES=cores export OMP_PROC_BIND=close export OPENBLAS_NUM_THREADS=1 mpirun -n 3 --map-by slot:PE=1 --bind-to core --report-bindings \ "$SODA_BIN" \ --num-gpu-workers-per-node=0 \ --storesummariesinfile=1 ``` The summary option stores additional process-search summaries used in AKMC analysis. Select options appropriate to the workflow when adapting the command; the executable's `--help` reports their scope. (openmpi-threaded-workers)= ### Match worker threads to CPU cores **Open MPI** places processes on CPUs. **OpenMP** runs threads within those processes. Setting `OMP_NUM_THREADS=3` requests three OpenMP threads, but does not give the process three CPU cores. Likewise, `mpirun -n 3` starts three MPI processes; it does not specify the number of cores per process. For two workers and one head/master, each assigned three physical cores, use the following launch on a machine with at least nine available cores: ```bash export OMP_NUM_THREADS=3 export OMP_PLACES=cores export OMP_PROC_BIND=close mpirun -n 3 --map-by slot:PE=3 --bind-to core --report-bindings \ "$SODA_BIN" \ --num-gpu-workers-per-node=0 \ --storesummariesinfile=1 ``` `PE=3` assigns three cores to each MPI process. `OMP_PLACES=cores` and `OMP_PROC_BIND=close` place the OpenMP threads on those cores. This uniform layout assigns nine cores in total, including three for the head/master; it does not imply the head/master uses three busy threads. Keep any additional SODA options required by your run after the executable. The placement flags are documented in the [Open MPI launch guide](https://docs.open-mpi.org/en/v5.0.7/man-openmpi/man1/mpirun.1.html#mapping-ranking-and-binding-oh-my). For an APE/AKMC GAP installation using LAMMPS's OPENMP package, also match the potential's setting in `config.ini`: ```ini [Potential] lammps_threads = 3 ``` A positive `lammps_threads` requires that LAMMPS package. Leave this setting unset or zero for builds without it, including the native MACE GPU recipe. Calculators can also adjust their own thread pools, so check the [runtime measurements](monitoring.md#check-cpu-placement-and-threads) after launch. `--report-bindings` prints the assigned cores in the launcher's output. For the three-core example, expect separate groups such as `core:0-2`, `core:3-5`, and `core:6-8`. Linux affinity masks use **logical CPU IDs**: on hardware with two hardware threads per core, three physical cores can appear as six logical CPU IDs. Conversely, a three-thread team restricted to two logical CPUs on one physical core still shares that single core. (openmpi-local-window-errors)= (local-open-mpi-4-1-window-errors)= :::{admonition} Local Open MPI 4.1 window errors :class: tip If a single-node run with Open MPI 4.1 fails in `MPI_Win_create` with `MPI_ERR_WIN`, first activate the build's `modules.rc` and check `mpirun --version`. The launcher and libraries must match the MPI installation used for the build. For this local window-backend failure, try: ```bash mpirun -n 3 --map-by slot:PE=1 --bind-to core --report-bindings \ --mca pml ob1 --mca btl self,tcp --mca osc sm,pt2pt \ "$SODA_BIN" --num-gpu-workers-per-node=0 --storesummariesinfile=1 ``` This workaround was checked with the single-adatom example on Open MPI 4.1.6. The `pt2pt` window component is unavailable in Open MPI 5; these options are specific to the Open MPI 4.1 setup and are not general cluster-launch defaults. ::: ## Prepare a cluster launch An HPC scheduler reserves resources; the MPI launcher starts SODA within that allocation. The standalone analyzer can check the run directory and print a launch template. For an illustrative CPU allocation of one node with three workers and one head: ```bash python "$SODA_ROOT/src/python/soda_analyzer.py" suggest-run-command . \ --launcher slurm --profile generic \ --nodes 1 --total-cpus 4 --num-worker 3 --cpus-per-worker 1 \ --gpus-per-node 0 --num-gpu-workers 0 \ --time 00:30:00 --soda-exe "$SODA_BIN" --check ``` This prints a suggestion; it does not submit a job. Add the account and partition required by your cluster, activate the selected build in the job script, and ensure the script runs from the prepared directory. Adapt core counts and time to your allocation policy and calculation. For threaded workers, match `#SBATCH --cpus-per-task` and `srun -c` to the per-rank thread budget, and set `OMP_NUM_THREADS`, `OMP_PLACES`, and `OMP_PROC_BIND` in the batch script. Whether Slurm counts cores or hardware threads as CPUs depends on the site; check its placement guidance and verify the resulting affinity. The Open MPI `--map-by` options above belong to `mpirun`, not `srun`. Here, `--num-worker` and `--cpus-per-worker` belong to the analyzer helper. They generate scheduler and launcher settings. They are not SODA executable options. Consult the {repo}`launch-template reference ` for the supported profiles and generated resource settings. When using the helper's `--launcher mpirun` output, add the {ref}`Open MPI mapping and thread settings ` yourself. That output currently specifies the rank count but does not add CPU binding or OpenMP environment settings from `--cpus-per-worker`. ## Use GPUs with a compatible calculator A GPU run needs a matching build, calculator/model configuration, and device allocation. Reserving GPUs alone does not move CPU-only tasks onto them. In the helper, `--gpus-per-node` requests devices from Slurm, while `--num-gpu-workers` selects GPU-capable workers and maps to SODA's `--num-gpu-workers-per-node`. Several workers can share a GPU when explicitly allowed, but their models and task data compete for device memory. The helper adds `--allow_gpu_oversubscribe` when the requested worker count exceeds the requested device count. Begin with the matching workflow's GPU example and check its model paths, device settings, and memory requirements. Detailed settings belong to the [APE](ape.md), [EZGA](ezga.md), [NEBScape](nebscape.md), and [AKMC](akmc.md) guides. Continue with [Monitoring and outputs](monitoring.md).