# Monitoring and outputs

After launch, use the logs to follow execution and the results database to
inspect scientific progress. Keep the run directory and the scheduler's
standard-output/error files together so you can relate results to their
execution history.

## Follow the run lifecycle

At startup, SODA reads the run configuration and initializes the selected
workflow integration and calculators. It distributes the workflow's tasks to
workers and collects their results. The workflow's scientific logic determines
how to interpret those results and what to do next.

Progress has a different meaning for each workflow:

| Workflow | Useful progress information |
| --- | --- |
| APE | Exploration cycles, states, discovered processes, and descriptor sampling. |
| EZGA | Completed generations, populations, and agent/stage progress. |
| NEBScape | Workflow phases, minima-hopping results, geometry pairs, and NEB bands/jobs. |
| AKMC | State transitions, process searches, and kinetic evolution. |

A completed task can be rejected, duplicate an earlier result, or fail to
converge. Task counts therefore do not directly measure the number of new
scientific results. Use the analysis tools documented for the selected
workflow to distinguish these outcomes.

## Find the generated files

The default layout adds these directories beneath the run directory:

```text
run-directory/
├── config.ini
├── ... input files ...
├── output/
│   ├── output.h5
│   ├── logs/
│   └── profiling/
└── workspace/
```

| Location | Contents |
| --- | --- |
| `output/output.h5` | Scientific results and the restart records provided by the workflow. |
| `output/logs/` | Master and rank logs describing execution. |
| `output/profiling/` | Timing/profiling files when enabled by the build and runtime. |
| `workspace/` | Temporary files used by workers and their calculators. |

`--outputdir` changes the output directory for the run. For example,
`--outputdir output-trial` uses `output-trial/output.h5` and corresponding
log/profiling directories. It does not change the shared `workspace/` location;
use separate run directories for simultaneous calculations. EZGA also has a
workflow-specific HDF5 path override, documented in its configuration.

## Read progress and diagnose a failed run

For the default output directory, inspect the master log with:

```bash
tail -n 60 output/logs/master
```

:::{admonition} Follow the log live
:class: tip

Use `tail -f output/logs/master` to follow new messages during execution.
:::

For calculator failures, inspect the worker logs and retained files in the
worker workspace as well. The scheduler's output/error files can reveal
failures that occur before SODA has initialized its own logs, including
environment, launch, or allocation problems.

Results are collected and written as the workflow proceeds. Logging a task's
completion does not mean that every result or restart record is already in
the HDF5 file. Save log and workspace diagnostics before cleaning up a failed
run.

## Check CPU placement and threads

For Open MPI launches, `--report-bindings` prints each MPI rank's assigned
cores to the launcher's output. Compare that placement with the intended
thread count; see {ref}`matching worker threads to cores
<openmpi-threaded-workers>`.

Native LAMMPS calculations in APE and AKMC also report `[LAMMPS runtime]`
in the worker logs. With the default output directory, find them with:

```bash
grep -F '[LAMMPS runtime]' output/logs/n*_w*
```

The report appears after the first force evaluation of each LAMMPS instance.
It includes these measurements:

| Field | What to check |
| --- | --- |
| `lammps_mpi_ranks` | The number of MPI ranks participating in that LAMMPS instance. |
| `lammps_omp_threads` | LAMMPS's stored thread setting. |
| `openmp_probe_team` | The number of threads that actually joined a short OpenMP probe in the worker process. |
| `probe_team_allowed_cpus` | The union of the probe threads' allowed logical CPU IDs. |
| `mean_cpu_cores` | CPU time divided by wall time for the first force evaluation, across all threads in that process. A value of `2.5` means 2.5 CPU-seconds of execution per elapsed second. |

An example mismatch is `openmp_probe_team=3` with
`probe_team_allowed_cpus=0-1`, where CPUs 0 and 1 are hardware threads of one
physical core. Three software threads then share that core. On Linux, check
which logical CPU IDs belong to the same core using:

```bash
lscpu -e=CPU,CORE,SOCKET,NODE
```

Use the team's affinity union when checking its available cores. With OpenMP
binding enabled, `calling_thread_allowed_cpus` may show only the first
thread's core, while the other team members have different cores.

The OpenMP probe does not measure every force kernel's team size: QUIP,
BLAS, and Torch can use different thread settings. The CPU-time sample
includes startup effects and does not measure sustained occupancy or GPU
utilization. These native LAMMPS entries do not cover Python/ASE calculators
in EZGA and NEBScape. See the {repo}`field descriptions and measurement limits
<docs/lammps_runtime_diagnostics.md>` for details.

## Understand stopping conditions

Scientific stopping settings belong to the workflow. For example,
`[APE] max_cycle` limits APE collect/register cycles when positive; zero
removes that cap. Other workflows use their own generation, phase, or sampling
controls. Review them in the workflow guide and `soda --help-config`.

A scheduler wall-time limit is an execution limit, not a convergence test.
Likewise, a job that exits without a runtime error still needs its scientific
results checked. If a run is interrupted, use the workflow's recorded restart
state to determine what can be recovered; see
[Restarting a run](restart.md).

## Open a results database

For an initial inspection after the writer has finished, use the standalone
analyzer. Follow [Output analysis](analyzer.md) to set up its Python environment
and module path, then run:

```python
import soda_analyzer as soda

db = soda.SodaLibrary(
    "output/output.h5",
    workflow="auto",
    state_load=False,
    structure_load=False,
)
try:
    print("Workflow:", db.workflow)
    print("Database groups:", list(db.h5.keys()))
finally:
    db.close()
```

This opens the file without eagerly loading state structures. Use the
workflow-specific readers for meaningful counts, structures, energies, and
convergence diagnostics.

:::{admonition} Analyze on another machine
:class: tip

A results database can be copied to another machine
and analyzed without compiling SODA or installing the simulation engines
used for the calculation.
:::

Continue with [Restarting a run](restart.md).