Monitoring and outputs¶

After launch, use the logs to follow execution and the results database to inspect scientific progress. Keep the run directory and the scheduler’s standard-output/error files together so you can relate results to their execution history.

Follow the run lifecycle¶

At startup, SODA reads the run configuration and initializes the selected workflow integration and calculators. It distributes the workflow’s tasks to workers and collects their results. The workflow’s scientific logic determines how to interpret those results and what to do next.

Progress has a different meaning for each workflow:

Workflow	Useful progress information
APE	Exploration cycles, states, discovered processes, and descriptor sampling.
EZGA	Completed generations, populations, and agent/stage progress.
NEBScape	Workflow phases, minima-hopping results, geometry pairs, and NEB bands/jobs.
AKMC	State transitions, process searches, and kinetic evolution.

A completed task can be rejected, duplicate an earlier result, or fail to converge. Task counts therefore do not directly measure the number of new scientific results. Use the analysis tools documented for the selected workflow to distinguish these outcomes.

Find the generated files¶

The default layout adds these directories beneath the run directory:

run-directory/
├── config.ini
├── ... input files ...
├── output/
│   ├── output.h5
│   ├── logs/
│   └── profiling/
└── workspace/

Location	Contents
`output/output.h5`	Scientific results and the restart records provided by the workflow.
`output/logs/`	Master and rank logs describing execution.
`output/profiling/`	Timing/profiling files when enabled by the build and runtime.
`workspace/`	Temporary files used by workers and their calculators.

--outputdir changes the output directory for the run. For example, --outputdir output-trial uses output-trial/output.h5 and corresponding log/profiling directories. It does not change the shared workspace/ location; use separate run directories for simultaneous calculations. EZGA also has a workflow-specific HDF5 path override, documented in its configuration.

Read progress and diagnose a failed run¶

For the default output directory, inspect the master log with:

tail -n 60 output/logs/master

Follow the log live

Use tail -f output/logs/master to follow new messages during execution.

For calculator failures, inspect the worker logs and retained files in the worker workspace as well. The scheduler’s output/error files can reveal failures that occur before SODA has initialized its own logs, including environment, launch, or allocation problems.

Results are collected and written as the workflow proceeds. Logging a task’s completion does not mean that every result or restart record is already in the HDF5 file. Save log and workspace diagnostics before cleaning up a failed run.

Check CPU placement and threads¶

For Open MPI launches, --report-bindings prints each MPI rank’s assigned cores to the launcher’s output. Compare that placement with the intended thread count; see matching worker threads to cores.

Native LAMMPS calculations in APE and AKMC also report [LAMMPS runtime] in the worker logs. With the default output directory, find them with:

grep -F '[LAMMPS runtime]' output/logs/n*_w*

The report appears after the first force evaluation of each LAMMPS instance. It includes these measurements:

Field	What to check
`lammps_mpi_ranks`	The number of MPI ranks participating in that LAMMPS instance.
`lammps_omp_threads`	LAMMPS’s stored thread setting.
`openmp_probe_team`	The number of threads that actually joined a short OpenMP probe in the worker process.
`probe_team_allowed_cpus`	The union of the probe threads’ allowed logical CPU IDs.
`mean_cpu_cores`	CPU time divided by wall time for the first force evaluation, across all threads in that process. A value of `2.5` means 2.5 CPU-seconds of execution per elapsed second.

An example mismatch is openmp_probe_team=3 with probe_team_allowed_cpus=0-1, where CPUs 0 and 1 are hardware threads of one physical core. Three software threads then share that core. On Linux, check which logical CPU IDs belong to the same core using:

lscpu -e=CPU,CORE,SOCKET,NODE

Use the team’s affinity union when checking its available cores. With OpenMP binding enabled, calling_thread_allowed_cpus may show only the first thread’s core, while the other team members have different cores.

The OpenMP probe does not measure every force kernel’s team size: QUIP, BLAS, and Torch can use different thread settings. The CPU-time sample includes startup effects and does not measure sustained occupancy or GPU utilization. These native LAMMPS entries do not cover Python/ASE calculators in EZGA and NEBScape. See the field descriptions and measurement limits for details.

Understand stopping conditions¶

Scientific stopping settings belong to the workflow. For example, [APE] max_cycle limits APE collect/register cycles when positive; zero removes that cap. Other workflows use their own generation, phase, or sampling controls. Review them in the workflow guide and soda --help-config.

A scheduler wall-time limit is an execution limit, not a convergence test. Likewise, a job that exits without a runtime error still needs its scientific results checked. If a run is interrupted, use the workflow’s recorded restart state to determine what can be recovered; see Restarting a run.

Open a results database¶

For an initial inspection after the writer has finished, use the standalone analyzer. Follow Output analysis to set up its Python environment and module path, then run:

import soda_analyzer as soda

db = soda.SodaLibrary(
    "output/output.h5",
    workflow="auto",
    state_load=False,
    structure_load=False,
)
try:
    print("Workflow:", db.workflow)
    print("Database groups:", list(db.h5.keys()))
finally:
    db.close()

This opens the file without eagerly loading state structures. Use the workflow-specific readers for meaningful counts, structures, energies, and convergence diagnostics.

Analyze on another machine

A results database can be copied to another machine and analyzed without compiling SODA or installing the simulation engines used for the calculation.

Continue with Restarting a run.