Parameter scans#

A parameter scan runs the same simulation over a grid of inputs — pressures, energies, durations, seeds — and collects what each point produced into one HDF5 file. This page is for anyone who wants more than a handful of runs: it describes how a scan is declared, how the points are built and executed (one after another, in parallel processes, in lockstep batches, or spread over cluster jobs), what is kept of each point, and how the collected file is read back.

The workflow has four parts:

declare the axes with a Scan (from solprop.driver.scan);
write a builder, a function that turns one ScanPoint, the values and seed of one point, into a Simulation;
call run_scan() with an executor, the reductions to compute in the worker, and what to retain;
read the file with solprop.processing.scans.open().

Declaring a scan#

A scan evaluates a Cartesian product of inputs while keeping point identity separate from execution order. The last declared variable varies fastest (NumPy C order). Indices start at zero and remain global after subset or chunk selections, so a point keeps its index, its seed and its place in the file however the scan is split. scan.add(name, values) appends a variable after construction; scan.subset(indices) keeps the named global indices and scan.chunk(k, n) the k-th of n round-robin subsets.

from solprop.driver.scan import Scan

scan = Scan("pressure_energy", pressure=[0.8, 1.0, 1.2], energy=[200e-6, 275e-6])
assert scan.shape == (3, 2)
assert scan.point(3).values == {"pressure": 1.0, "energy": 275e-6}
assert [point.index for point in scan.chunk(1, 2).points()] == [1, 3, 5]

Builders and executors#

The builder receives a ScanPoint and returns a Simulation. Variables use the builder’s units: pressure at the user interface is in bar, energy in J, lengths in m and times in s. A builder can select any installed backend.

Five executors, all in solprop.driver.executors, run the points:

executor	what it does
`Local()`	runs the points one after another in the current process; the default
`Pool(workers, threads=1, ...)`	runs independent points in parallel worker processes; the executor to use for independent CPU runs
`Batch(size)`	advances compatible fixed-schedule points in lockstep inside one backend
`Queue(path)`	a durable queue on a shared filesystem, drained by one or more controllers
`Slurm(script, ...)`	prepares a queue and submits a Slurm array job that drains it

A builder’s omitted backend resolves to the native backend when it is installed and to numba_cpu otherwise, as described in running a simulation; every executor keeps that concrete choice, so name the backend in the builder when the workers may run on machines with different installations.

Reuse a translated simulation#

Translation — the cumulative linear operator, the modal transform, the ionisation rate tables, the compiled responses — is the expensive part of building a run, and most scans change only the fill or the input pulse. Build the base once, lazily, in each worker, and derive every point from it. For example, this reduced pressure–energy scan follows the helium RDW case’s 125 µm capillary and 7.5 fs pulse (running). Its short length and coarse grid demonstrate execution, not a converged RDW prediction.

from functools import cache
from solprop.driver import Simulation
from solprop.driver.scan import Scan, ScanPoint, run_scan
from solprop.driver.executors import Local, Pool
from solprop.physdata import Mixture
from solprop.processing.scans import energy_in_band, peak_power
from solprop.pulses import GaussianPulse


@cache
def base():
    mixture = Mixture.fill({"He": 1.0}, 1.0)
    simulation = Simulation.from_capillary(
        125e-6,
        0.002,
        mixture,
        lambda0=800e-9,
        wavelength_limits=(200e-9, 1.5e-6),
        trange=200e-15,
        fwhm=7.5e-15,
        energy=200e-6,
        modes=1,
        raman=False,
        plasma=False,
        thg=False,
        n_save=3,
        backend="numba_cpu",
    )
    simulation.translate()
    return simulation


def build(point: ScanPoint):
    mixture = Mixture.fill({"He": 1.0}, float(point.values["pressure"]))
    pulse = GaussianPulse(
        wavelength=800e-9, fwhm=7.5e-15, energy=float(point.values["energy"])
    )
    return base().with_mixture(mixture).with_input(pulse)


if __name__ == "__main__":
    scan = Scan("pressure_energy", pressure=[0.8, 1.2], energy=[200e-6, 275e-6])
    result = run_scan(
        scan,
        build,
        executor=Pool(workers=2),
        output="pressure_energy.h5",
        reductions=[energy_in_band(250e-9, 400e-9), peak_power()],
        fields="planes",
        field_planes=[0, -1],
        stats="all",
    )
    ultraviolet_energy = result.as_grid("reduced/energy_in_band")

with_input() rebuilds the input record and metadata while sharing propagation products, and validates only the replaced input; its configuration= records keys of your own, such as a point’s parameters, in the derived run’s record. Outside a scan, a runner also keeps one backend for many inputs; scan workers construct a backend per point. noise="keep" (the default) keeps the base noise, noise=None disables it, and a NoiseRealisation replaces it. with_mixture() changes the fill (running); combine the two for material and input changes. Geometry, grid or mode-count changes need a new simulation and translation. The ionisation rate tables keep the validity interval they were built with; an input update does not enlarge that interval, so build the base at the highest energy the scan reaches or pass ionisation_field_range explicitly. Raman and plasma choices in this example are explicit. For molecular gases the ordinary Raman and ionisation defaults apply when not specified; see mixtures and ionisation.

What reuse buys. Reusing a translated base cuts the setup time of an eight-point Local scan to between a half and a third of translating every point afresh; with a two-worker Pool, where each worker translates its own base once, the total with startup and metadata changes little. Keep the geometry, the grid and the ionisation tables’ validity range unchanged between points. Changing the input array removes the constructor’s pulse recipe from the metadata, because it no longer describes the field; the exact initial spectrum is still recorded. Two points with equal physical fields need not have equal configuration hashes when their construction provenance differs.

Fixed-schedule batches#

Batch(size) advances up to size points in lockstep, on one fixed step schedule, inside one backend. It runs on numpy_ref, numba_cpu and native (CPU and GPU builds), and each backend is tested to give a batch member the output the same point gives run alone, to rounding: the same steps, with sums that may be ordered differently (SEP 25).

from solprop.driver.executors import Batch
from solprop.driver.scan import run_scan

# The builder must return fibre simulations with an explicit fixed schedule.
# It selects numpy_ref, numba_cpu or native as usual.
result = run_scan(
    scan,
    build_fixed_point,
    executor=Batch(4, memory_limit=8 * 1024**3),
    output="fixed_scan.h5",
    fields="planes",
    field_planes=[-1],
    stats="all",
)

Prepare the replay schedule with the discover, calibrate and replay workflow of solprop.stepper.schedule (fixed schedules) and validate its transfer to the individual scan points before interpreting their physics. A batch executor does not silently convert an adaptive simulation into a fixed one, and it runs modal (fibre and capillary) propagation only, not radial, Cartesian or planar grids. Adjacent points with matching grids, schedules, response layouts and statistics can share a group; layout changes and the final partial batch make smaller groups. Mixing backend names in a batch scan raises an error. Global point indices are unchanged. General modal runs may be batched under a narrower rule (running).

Memory admission probes the selected backend and keeps 20 % host headroom. It records baseline and probe RSS, workspace and output estimates, shared table bytes already included in the baseline, and the admitted group size. An explicit memory_limit (bytes) further restricts the physical or scheduler allocation. This is an admission estimate, not an operating-system memory limit; the builder itself must fit while constructing up to size translated points. A budget that cannot fit one member is refused. Device backends report free memory through the same interface: admission probes each member, uses its complete ledger as a conservative upper bound on device storage, and restricts the group before constructing all members. Allocator reserves are excluded from free space. The recorded device bound may include host buffers, so it can admit fewer members than a device-only ledger would.

Each point records its actual batch size, execution strategy and resolved backend policies. numba_cpu and native fuse the accepted update across members (native combines the error partials of all members in the same native kernel), while each member keeps its own workspace; the response kernels and the FFT and BLAS plans stay per member. This partial fusion does not amortise the whole GPU launch sequence.

What batching buys. On the CPU, batching gives no speedup over running the points one after another, so use Pool there. On a data-centre GPU, a native Batch(8) of eight points of a shortened multimode resonant-dispersive-wave capillary on a fixed 515-step schedule gave about 1.1 times the throughput of Local, setup and output included. That measures one shape on one schedule and does not predict others. For comparison, on an 18-core x86-64 server a Pool of 8 to 18 one-thread workers gave about four times the throughput of Local on a shortened six-mode hydrogen-filled capillary, and one worker using all 36 hardware threads was slower than those pools. Size workers within the actual CPU and memory allocation, and measure the workload that matters to you; performance tuning explains the thread settings of scan workers.

Fields keep the single-run saved-plane shape. Error histories, noise, species axes and periodic statistics remain separate per member. Reductions use the same processing definitions and temporary disk spool as Local and Pool. Completed points are revalidated by the ordinary single-run configuration hash before being skipped, so changing executor does not change that hash.

Processes and resource limits#

run_scan runs on Local() when no executor is given and the command line has set none. Use Local() for debugging or a small interactive scan, and Pool() for independent CPU simulations. Pool uses spawned processes and one library thread per worker by default. It sets thread environment variables before scientific libraries import, respects Linux CPU affinity and Slurm/cgroup allocations, and keeps workers alive. fresh=True creates a new worker for each point. workers, threads, pin and memory_limit provide explicit resource limits; memory is specified in bytes.

A backend/RSS probe sizes workers. The estimate takes the larger of measured RSS and baseline RSS plus the backend ledger, then reserves retained output and transfer buffers. The controller’s RSS and 20 % memory headroom are reserved separately. The file records this calculation. It refuses an allocation that cannot fit one worker. This is admission control, not an operating-system memory limit; adaptive periodic statistics and custom reducer working memory can exceed the probe estimate.

Pool builders and custom reducers must be pickleable, importable module-level callables. Keep entry-point execution under if __name__ == "__main__". Nested functions, lambdas and live backend handles cannot be dispatched. Validation happens before the output file is created. A cached module-level base is local to each worker, so no translated native objects cross process boundaries.

python -m solprop.scan scan_script.py --local sets the executor from the command line; an executor supplied explicitly by the script takes precedence. One outer progress bar is shown interactively (progress= of run_scan); otherwise progress is logged per point. Individual runs never draw their own bars.

Noise and repeatability#

Use Scan("name", base_seed=42, ...) to derive a reproducible unsigned 64-bit seed for each global index with SeedSequence([base_seed, global_index]).generate_state(1, dtype=uint64)[0]. Use point.seed when constructing noise in the builder. An ordinary seed=[...] axis overrides automatic derivation. Subsetting or changing execution order preserves the seed. The file records the derivation, resolved point seed, actual noise seed, single-run configuration hash and backend execution settings. An active noise draw without a recorded seed, or with a seed different from point.seed, is rejected.

Retention, reductions and reading#

By default scans retain all collected statistics, including periodic rows and species axes, and no fields. stats=["energy", "peak_power"] filters statistics by name; stats="none" removes them, which is discouraged: the statistics are the diagnostics needed to assess a scan. fields="planes" requires explicit field_planes saved-plane indices, including negative indices. fields="all" retains every saved plane. A field reduction can retain one field independently of field-history storage.

The worker keeps only the fields the scan reads (SEP 22): the planes fields retains, the target plane of each built-in reduction that reads a field, and, while a custom reduction reads fields, every plane of the run it is handed. The other planes are run as statistics-only planes (output), whose fields are never stored, so a scan of large spatial runs does not hold a field per plane for nothing. run_scan’s arguments decide: a builder whose SimulationSettings set fields or field_planes is refused by name.

Reductions are computed in the worker, from the point’s complete output, so only their results travel back and are stored. The predefined ones in solprop.processing.scans call the ordinary processing definitions: spectrum_in_band and energy_in_band (a wavelength band), energy_in_window (a time window), peak_power, mode_energies, field_at_plane, and, for spatial runs, on_axis_intensity and beam_radius. Spatial reductions require the corresponding collected statistics. Band energy applies a closed rectangular frequency-bin mask; it is not the tapered bandpass window. Time-window energy sums the selected samples with the grid time step. spectrum_in_band(lambda_min, lambda_max) keeps the spectral energy density, J/m and summed over modes, on the run’s active band, the bins that get_spectral_density() returns by default (on a field grid this leaves out the zero and Nyquist bins, where no wavelength is defined), at the wavelengths inside the closed band, ascending. The reader’s spectrum_wavelength() returns that abscissa, rebuilt from the recorded grid and band, so it matches the last axis of as_grid("reduced/spectrum_in_band") sample for sample; spectrum_wavelength(index) gives one point’s abscissa when the points’ grids differ. The spectrum, band and window energies, peak power and mode energies read modal amplitudes, so a radial, Cartesian or planar run refuses them by name; use its spatial statistics or a custom reducer there. A saved field keeps the run’s stored units: V m radially, V/m on a Cartesian grid, √(W/m) per y frequency on a planar one. Reducers carry units, dtype, parameters and source provenance.

custom(name, fn, shape, units, dtype=...) declares a reducer of your own: fn receives the point’s complete output handler and returns an array of the declared shape and dtype. The worker spools that point to temporary local HDF5 storage and removes it afterwards. By default the spooled run keeps every field. A reducer that reads only statistics, positions and step histories declares reads_fields=False, and its spooled run then keeps the fields the scan keeps anyway and no other; a field read inside it is refused by name. The declaration is recorded in the reducer’s provenance only when it is False. A wrong shape, dtype or non-finite reduction marks the point failed and records the traceback. When simulation metadata is available, failed points keep their configuration hash, actual noise seed and backend policies. Readers expose that provenance while keeping incomplete numerical arrays masked.

One controller owns the collected HDF5 file. Each successful payload is validated, staged and flushed before the point is marked done. Reruns validate matching configuration hashes and skip completed points. A completed point whose rebuilt run has another hash is refused, naming the fields that changed. Since output schema 8 that hash includes the point’s material manifest: a point resolves its materials, under its materials= selection or the default one, from its own objects, so its optmat identities, policies and the optmat version are frozen for that run and do not depend on what another point resolved, and a rerun with another model, policy or optmat version is refused by the manifest entry that moved (materials.records..., materials.policies..., materials.optmat_version; output). Failed points require retry_failed=True. A stale running point becomes eligible after stale_age seconds (600 by default). Queue ownership protection uses the live-owner and refreshed-lease rules described below.

Uniform scans expose rectangular stats/, reduced/ and optional fields/ datasets. Different shapes have separate layouts/<digest>/ groups with global-point mappings. Common metadata is content-addressed, with the constant and varying paths recorded. Periodic statistics use concatenated values and offsets without padding. Statistics that are legitimately undefined and stored as NaN (centroids, widths, beam radii, zero-dispersion wavelengths and unavailable quadrature alarms) are preserved; non-finite fields, energies and reductions are rejected. Step indices keep the single-run convention, relative to each plane’s interval, with per-plane counts.

from solprop.processing.scans import open

result = open("pressure_energy.h5")
indices = result.where(pressure=1.2)
point = result.point(int(indices[0]))
layouts = result.layouts
energy = result.as_grid("reduced/energy_in_band")

ScanOutput has shape, status, where, point, as_grid, layouts and spectrum_wavelength. Readers mask unfinished points and recover committed staging even if consolidation was interrupted. as_grid refuses arrays with incompatible shapes. A failed point’s error contains its traceback; inspect status before interpreting a scan.

Durable queues and Slurm#

For several controllers sharing a filesystem, choose Queue("pressure_energy.queue"). Each controller runs a memory-sized spawned Pool. Point artifacts are fsynced before publication; the collected file is written under one exclusive controller lock. Readers take the corresponding shared lock. The filesystem must support coherent advisory locks, atomic renames and directory fsync across the participating hosts. The queue is tested on local disks (APFS and ext4); network and parallel filesystems (NFS, Lustre, GPFS) differ in their locking guarantees, so check yours with a small scan before relying on it.

uv run python -m solprop.scan pressure_energy.py --queue pressure_energy.queue --workers 4
uv run python -m solprop.scan pressure_energy.py --queue pressure_energy.queue --chunk 1/3
uv run python -m solprop.scan collect pressure_energy.queue --output pressure_energy.h5

Chunks are zero-based round-robin selections of global indices. Several invocations can share the queue and output. The immutable manifest binds the axes, retention settings, serialised builder and available builder source; changed scripts require a new queue. Collection checks scan identity, point identity, configuration hashes and artifact digests. Repeated collection of unchanged results leaves the file unchanged. Artifacts are JSON and numerical HDF5 datasets, never executable pickle payloads. Failed artifacts keep their error and available provenance; collection rejects numerical arrays attached to a failed artifact.

A controller refreshes each active claim’s lease. A live local PID with matching creation identity remains protected even after stale_age; a remote owner remains protected while its lease is refreshed. A dead local owner or expired remote lease can be replaced. Unique claim tokens reject late results from superseded workers. Failed points require retry_failed=True in the script. .nowork means the invocation found no eligible work; inspect active owners and failures. .done is written and fsynced before active queue state is removed. Keep the persistent lock files: deleting a live lock file would create a second lock inode and defeat mutual exclusion.

Note

Recovery has known limits. A live but wedged owner on the same host holds its point indefinitely, because the lease age is not consulted while its process is alive: stop that process to release the point. An interrupted Local or Pool scan skips its running points for stale_age seconds on restart. Container memory limits are read from the root cgroup rather than the process’s own cgroup, so on a shared node under a cgroup manager without Slurm variables the workers can be given too much memory; a failing platform probe raises outside the SolpropError hierarchy. The scan file grows on every resume, because HDF5 does not reclaim the groups it recreates.

Inspect a generated Slurm script before submission:

uv run python -m solprop.scan pressure_energy.py --emit-slurm scan.slurm --queue pressure_energy.queue --ntasks 4 --cpus-per-task 8 --mem 16G --time 01:00:00
uv run python -m solprop.scan pressure_energy.py --submit --queue pressure_energy.queue --ntasks 4 --cpus-per-task 8 --mem 16G --time 01:00:00

--emit-slurm writes the script without running the scan builder or calling sbatch. --submit prepares the durable manifest, submits the array and returns the currently collected output, which may still contain unfinished points. Collect again after jobs finish. The Python Slurm executor also exposes its returned job_id. Scripts and queue/output paths must be accessible at the same paths on allocated nodes, with the same environment available. --mode chunk emits disjoint round-robin task subsets; queue mode (the default) lets tasks share claims dynamically.

Positive --instances divides each task’s CPUs and memory among controllers. For GPU resources such as --gres gpu:2 --instances 2, each instance receives one of Slurm’s assigned device identifiers and defaults to one worker. Missing assignments fail before the scan starts. All worker thread settings are exported before Python imports scientific libraries (see performance tuning for what those settings are). --fresh creates a fresh process per point.

Submission records under QUEUE/submissions/ keep the scheduler job ID and generated script digest, and collection includes them in the execution audit.

An explicit executor in run_scan overrides a command-line default. To use these command-line modes, omit the script’s executor. An explicit Slurm executor recognises its own emitted task environment and drains the prepared queue instead of submitting recursively. Without a durable mode, --workers, --threads, --memory-limit or --fresh select Pool; --local selects Local and rejects process resource controls. Slurm memory uses --mem, while a direct Pool or queue accepts --memory-limit in bytes.