# Restarting a run

Restarting continues from scientific state recorded by an earlier run.
Keep the original inputs, potential/model files, and software environment
available when resuming.

This page describes restart support provided by the workflow integrations
in SODA. Consult each workflow's own documentation for restart procedures
when using its native software.

## Choose a fresh run or a restart

A fresh run starts from the supplied inputs. Use a new run directory when
starting an independent calculation, particularly when changing structures,
potentials, or scientific settings.

To request a restart, pass `--loadrun` with the previous HDF5 file. An existing
`output.h5` does not by itself request continuation. A relative `--loadrun`
path is resolved inside the selected output directory; an absolute path makes
the source unambiguous.

Retain an untouched copy of the earlier run before resuming. Wait until its
writer has stopped before copying its database. Use a separate working copy
as the restart input: current APE/AKMC and NEBScape restart handling can open
that file for updates. Review batch scripts for commands that remove previous
outputs before reusing them for a restart.

:::{admonition} Keep the checkpoint and output HDF5 paths separate
:class: tip

The checkpoint passed to `--loadrun` must be a different file from the
resumed run's output HDF5, `<outputdir>/output.h5`. In the example below,
the checkpoint is `output-resumed/restart-input.h5`, while the new output
is `output-resumed/output.h5`. Use distinct filenames or directories to
avoid overwriting the checkpoint with the new output.
:::

For the AKMC/EAM single-adatom example, the following shows the restart command
shape using the same build and three-rank CPU layout as before. From the run directory,
choose a new output-directory name and copy a compatible saved database into
it. Replace the archive path below with your saved file:

```bash
mkdir output-resumed
cp "/absolute/path/to/saved/output.h5" output-resumed/restart-input.h5
export SODA_RESTART_FILE="$PWD/output-resumed/restart-input.h5"
export OMP_NUM_THREADS=1
export OMP_PLACES=cores
export OMP_PROC_BIND=close
export OPENBLAS_NUM_THREADS=1
mpirun -n 3 --map-by slot:PE=1 --bind-to core --report-bindings \
  "$SODA_BIN" \
  --outputdir output-resumed \
  --loadrun "$SODA_RESTART_FILE" \
  --num-gpu-workers-per-node=0 \
  --storesummariesinfile=1
```

Keep the required input and potential files available in the run directory.
For a cluster restart, add the output and restart options to the appropriate
batch launch after activating the same build. Inspect the startup log to
confirm which file and workflow state were restored. Retain both the working
restart file and the generated output until you have checked the resumed
results.

For a threaded Open MPI run, retain the matching {ref}`cores-per-rank mapping
and thread settings <openmpi-threaded-workers>` when restarting. The command
above uses the walkthrough's one-core-per-rank layout.

If you used the local Open MPI 4.1 window workaround, retain the same
{ref}`MPI options <openmpi-local-window-errors>` before the
executable when restarting. For AKMC, `[AKMC] max_kmc_steps` limits the
additional transitions performed during the resumed invocation. Keeping the
example's `20000` therefore requests up to 20,000 more transitions.

## Understand the restart boundary

The recovery point depends on the workflow and the records successfully
written to disk:

| Workflow | Restart basis |
| --- | --- |
| APE | Saved whole-file state, including exploration and descriptor data. |
| EZGA | Complete generation commits, resolved for each agent and stage. |
| NEBScape | Complete workflow-phase checkpoints, with workflow-specific handling of retained partial NEB results. |
| AKMC | Saved whole-file state for the kinetic calculation. |

A restart may repeat work performed after the last recoverable state.
Temporary worker files and in-memory progress do not substitute for the
workflow's HDF5 restart records.

For NEBScape, the restart validates settings and catalog fingerprints. Use its
{repo}`progress and loadrun diagnostics
<docs/soda_analyzer_nebscape.md#inspecting-workflow-progress-loadrun-and-partial-neb-rows>`
to inspect completed phases and retained partial results. For EZGA, inspect
the {repo}`generation commits
<docs/soda_analyzer_ezga.md#looking-for-generation-commits-rejections-and-hise-metadata>`;
native snapshot files are not a substitute for SODA's `--loadrun` recovery.

Use the same scientific inputs and compatible software when resuming. Check
the workflow's restrictions before changing its settings or resource layout.
The {repo}`HDF5 schema policy <src/HDF5_SCHEMA.md#resume-policy-boundary>`
documents the underlying recovery scopes.

:::{admonition} Distinguish the data format from the software revision
:class: tip

The HDF5 `ver_tag` identifies the data format. Record the software revision
separately: a format version does not uniquely identify the code that produced
a calculation.
:::

When sharing scientific results, follow
[Citing SODA](../citations.md) for the framework, workflow, and potential/model
references.

Continue with a workflow tutorial: [APE](ape.md), [EZGA](ezga.md),
[NEBScape](nebscape.md), or [AKMC](akmc.md).