Restarting a run¶

Restarting continues from scientific state recorded by an earlier run. Keep the original inputs, potential/model files, and software environment available when resuming.

This page describes restart support provided by the workflow integrations in SODA. Consult each workflow’s own documentation for restart procedures when using its native software.

Choose a fresh run or a restart¶

A fresh run starts from the supplied inputs. Use a new run directory when starting an independent calculation, particularly when changing structures, potentials, or scientific settings.

To request a restart, pass --loadrun with the previous HDF5 file. An existing output.h5 does not by itself request continuation. A relative --loadrun path is resolved inside the selected output directory; an absolute path makes the source unambiguous.

Retain an untouched copy of the earlier run before resuming. Wait until its writer has stopped before copying its database. Use a separate working copy as the restart input: current APE/AKMC and NEBScape restart handling can open that file for updates. Review batch scripts for commands that remove previous outputs before reusing them for a restart.

Keep the checkpoint and output HDF5 paths separate

The checkpoint passed to --loadrun must be a different file from the resumed run’s output HDF5, <outputdir>/output.h5. In the example below, the checkpoint is output-resumed/restart-input.h5, while the new output is output-resumed/output.h5. Use distinct filenames or directories to avoid overwriting the checkpoint with the new output.

For the AKMC/EAM single-adatom example, the following shows the restart command shape using the same build and three-rank CPU layout as before. From the run directory, choose a new output-directory name and copy a compatible saved database into it. Replace the archive path below with your saved file:

mkdir output-resumed
cp "/absolute/path/to/saved/output.h5" output-resumed/restart-input.h5
export SODA_RESTART_FILE="$PWD/output-resumed/restart-input.h5"
export OMP_NUM_THREADS=1
export OMP_PLACES=cores
export OMP_PROC_BIND=close
export OPENBLAS_NUM_THREADS=1
mpirun -n 3 --map-by slot:PE=1 --bind-to core --report-bindings \
  "$SODA_BIN" \
  --outputdir output-resumed \
  --loadrun "$SODA_RESTART_FILE" \
  --num-gpu-workers-per-node=0 \
  --storesummariesinfile=1

Keep the required input and potential files available in the run directory. For a cluster restart, add the output and restart options to the appropriate batch launch after activating the same build. Inspect the startup log to confirm which file and workflow state were restored. Retain both the working restart file and the generated output until you have checked the resumed results.

For a threaded Open MPI run, retain the matching cores-per-rank mapping and thread settings when restarting. The command above uses the walkthrough’s one-core-per-rank layout.

If you used the local Open MPI 4.1 window workaround, retain the same MPI options before the executable when restarting. For AKMC, [AKMC] max_kmc_steps limits the additional transitions performed during the resumed invocation. Keeping the example’s 20000 therefore requests up to 20,000 more transitions.

Understand the restart boundary¶

The recovery point depends on the workflow and the records successfully written to disk:

Workflow	Restart basis
APE	Saved whole-file state, including exploration and descriptor data.
EZGA	Complete generation commits, resolved for each agent and stage.
NEBScape	Complete workflow-phase checkpoints, with workflow-specific handling of retained partial NEB results.
AKMC	Saved whole-file state for the kinetic calculation.

A restart may repeat work performed after the last recoverable state. Temporary worker files and in-memory progress do not substitute for the workflow’s HDF5 restart records.

For NEBScape, the restart validates settings and catalog fingerprints. Use its progress and loadrun diagnostics to inspect completed phases and retained partial results. For EZGA, inspect the generation commits; native snapshot files are not a substitute for SODA’s --loadrun recovery.

Use the same scientific inputs and compatible software when resuming. Check the workflow’s restrictions before changing its settings or resource layout. The HDF5 schema policy documents the underlying recovery scopes.

Distinguish the data format from the software revision

The HDF5 ver_tag identifies the data format. Record the software revision separately: a format version does not uniquely identify the code that produced a calculation.

When sharing scientific results, follow Citing SODA for the framework, workflow, and potential/model references.

Continue with a workflow tutorial: APE, EZGA, NEBScape, or AKMC.