solprop.output.hdf5#

The streaming HDF5 output handler, with cache and resume.

One file per run, kept open between optional snapshot transactions. Saves and metadata calls reuse the handle. Without snapshots the file opens only once.

What is on disk, and where. Root attributes carry the identifying scalars – schema_version, spectral_convention, field_layout, configuration_hash, simulation_kind, grid_kind, created and, once a backend has executed the run, backend – and meta/json carries the whole solprop.output.schema.RunMetadata as one JSON string. The metadata goes in a dataset rather than an attribute because HDF5 caps an attribute in the object header at 64 kB and a serialised configuration – a capillary with a tabulated cladding glass, a stored noise realisation – goes past that; the identifying scalars stay attributes, where a reader looks for them first. field is (n_z, n_modes, n_pol, n_omega) complex, chunked one plane at a time and extendable along n_z, in the optics convention and marked so. noise_phase, when the run has noise, has the same shape and stays in the internal convention. z is the plane positions, stats/<name> one dataset per statistic, each with a leading plane axis, and step_history/<column> the attempts of the whole run concatenated, with step_history/counts saying how many belong to each plane.

Preallocation and the saved-plane count. The plane-indexed datasets are created at the first save for every plane the run reserved (reserve()), with an unlimited leading axis, and grow by doubling if the run saves more; the step-history and between-planes columns grow by doubling too. The root attribute n_saved counts the planes actually written: each plane’s data is written and flushed first and the count last, so a hard stop leaves the count at the last complete plane. The attempts and rows belonging to those planes are the sums of step_history/counts and step_stats/counts over them, so the count alone fixes every column’s length. Every reader and resume() read only the first n_saved planes, and a file without the attribute – one written before it existed – is read by the extent of its z dataset. A clean close trims every dataset to its count, so a finished file holds no unused rows.

Field retention (output schema 20, SEP 22). The root attribute fields says which planes keep their field (FieldRetention), and field_shape the shape of one plane’s field whether or not it is stored. A run that keeps every field stores field and noise_phase as above. One that keeps the named planes stores exactly those, in plane order, with a field_planes dataset of their indices beside them; one that keeps none has no field and no noise_phase dataset. A file without the attribute was written before field retention existed and keeps every field. resume() continues only from a plane that carries a field: a terminal checkpoint always does, and a file whose last regular plane is field-free and which holds no checkpoint is refused by name when a run is continued from it.

Cache and resume. check_cache() compares the record’s configuration_hash() and refuses a mismatch by naming the fields that changed; the digest covers the physics, not only the statistic names and the state shape, so resuming with different physics of the same shape is refused. From output schema 8 it covers the run’s material manifest too: another optmat model, material policy or optmat version is refused by the manifest entry that changed, and a file written before schema 8 is not resumed into a newer run at all. resume() returns the metadata, the last plane – converted back to the internal convention – and the step history. A handler built with append=True then carries on in the same file: it opens it read-write instead of replacing it, checks that the record offered describes the run the file holds, and continues numbering planes and attempts where the file left off. open_file() opens a finished file read-only through the same protocol it was written with, which is how solprop.processing reads a run after the driver has closed it.

Compression is restricted to what h5py ships: None, 'gzip' or 'lzf'. Blosc is not offered, because it needs a filter plugin solprop does not depend on.

The manual page is docs/guide/results/output.md.

Module Attributes

COMPRESSIONS

Compression filters h5py ships with; no plugin, so no Blosc (dependencies.md).

Functions

`check_cache`(path, metadata)	Say whether a file holds the run described by `metadata`, and refuse if not.
`open_file`(path)	Open a finished run for reading, through the protocol it was written with.
`resume`(path)	Read a file's resumable state and cumulative work, preferring its checkpoint.

Classes

`HDF5Output`(path, *[, compression, ...])	Stream a run to one HDF5 file, opened once and extended plane by plane.
`ResumeState`(metadata, plane, n_saved, history)	Everything a driver needs to carry on from a file it wrote earlier.

solprop.output.hdf5.COMPRESSIONS = (None, 'gzip', 'lzf')#: Compression filters h5py ships with; no plugin, so no Blosc (dependencies.md).

class solprop.output.hdf5.HDF5Output(path, *, compression=None, compression_opts=None, append=False, readonly=False)[source]#

Bases: PlaneStore

Stream a run to one HDF5 file, opened once and extended plane by plane.

Reading through __getitem__ and fields() works while the handler is open and stops when it is closed, because the arrays live in the file rather than in memory; open_file() opens a finished file for reading afterwards.

Parameters:

pathpathlib.Path or str: Where to write. An existing file is replaced, with a warning, when the run opens.
compression{None, ‘gzip’, ‘lzf’}, optional: Dataset compression. 'lzf' is fast and modest; 'gzip' is slower and smaller. Nothing else: a Blosc filter would need a plugin that solprop does not depend on.
compression_optsint or None, optional: Level for 'gzip', 0 to 9; not accepted for the other choices.
appendbool, optional: Continue the run the file already holds instead of replacing the file: the resume path, after resume() has handed the driver the last plane. The record given to open must describe the same run, the file’s own metadata and creation time are kept, and the run’s layout is read from the file so that the next plane is checked against the ones already there.
readonlybool, optional: Open the file at once, for reading only: the handler comes back open, with the file’s own record as its metadata, and refuses open and save. This is what open_file() builds.

Raises:

OutputError: If the compression request is not one h5py can serve, if both append and readonly are asked for, or if a read-only file cannot be read.

Examples

>>> import numpy as np, tempfile
>>> from pathlib import Path
>>> from solprop.grid import EnvelopeGrid
>>> from solprop.output import HDF5Output
>>> from solprop.output.schema import (
...     GridRecord, RunMetadata, SavedPlane, StepperRecord
... )
>>> from solprop.stepper import DOPRI5, PIController
>>> grid = EnvelopeGrid.from_limits(1.0, 835e-9, (400e-9, 2e-6), 2e-12)
>>> metadata = RunMetadata(
...     grid=GridRecord.from_grid(grid),
...     modes=(),
...     simulation_kind="mode_averaged",
...     configuration={},
...     stepper=StepperRecord.from_settings(
...         DOPRI5, PIController(rtol=1e-8, atol=0.0)
...     ),
... )
>>> with tempfile.TemporaryDirectory() as folder:
...     handler = HDF5Output(Path(folder) / "run.h5", compression="lzf")
...     handler.open(metadata)
...     start = np.ones((1, 1, grid.n_omega), complex)
...     handler.save(SavedPlane(z=0.0, field=start))
...     shape = handler["field"].shape
...     handler.close()
>>> shape
(1, 1, 1, 2048)

close()[source]#

Close the owned handle, including a failed post-run annotation handle.

Repeated calls are harmless. A failed close retains the handle and writer ownership for a later retry, even if propagation already closed the store.

Raises:

OutputError: If the handle cannot be closed safely.

open(metadata, *, force=False)[source]#

Begin a run, recording what produced it; refused on a read-only handler.

Parameters:

metadataRunMetadata: The record of the run about to start.
forcebool, optional: Whether to overwrite metadata already held for a different run.

Raises:

OutputError: If the handler was built with readonly=True.

property path: Path[source]#

Return where the file is written.

Returns:

pathlib.Path: The path given to the constructor.

reopen()[source]#

Rebind a suspended writer without beginning a new execution.

Raises:

OutputError: If the file changed while suspended or cannot be reopened safely.

save(plane)[source]#

Store one saved plane and flush it to disk; refused on a read-only handler.

The flush is what makes a hard stop – a SIGKILL, an out-of-memory kill, a lost node – lose at most the plane being written: the plane’s data is flushed before the file’s n_saved count is raised, so a plane cut off half-way is not counted, and readers do not see it. An ordinary interruption (KeyboardInterrupt, an error) never needed it: the run loop closes the handler in a finally and closing writes everything. One flush per plane costs nothing against the steps between planes.

Parameters:

planeSavedPlane: The position, field, statistics and step history to store.

Raises:

OutputError: If the handler was built with readonly=True.

save_march(snapshot)[source]#

Keep the Adams march’s state at the position just saved, for a resume.

One snapshot is kept, rewritten in place at every call, beside the planes and outside the record: it says where the march stood, not what was computed, and a file without one resumes by starting the history afresh.

Parameters:

snapshotsolprop.output.schema.MarchSnapshot: The march’s state at the last saved plane or terminal checkpoint.

Raises:

OutputError: If the handler was built with readonly=True or is not open.

suspend()[source]#

Close the HDF5 handle while retaining the live writer and its ownership.

Raises:

OutputError: If the writer is not open or its handle cannot be closed safely.

class solprop.output.hdf5.ResumeState(metadata, plane, n_saved, history, outcome=None, checkpoint=None, march=None)[source]#

Bases: Displayed

Everything a driver needs to carry on from a file it wrote earlier.

Equality is by identity (eq=False) because the plane holds arrays.

Attributes:

metadatasolprop.output.schema.RunMetadata: The record of the interrupted run, including its step schedule.
planesolprop.output.schema.SavedPlane: The accepted terminal checkpoint when present, otherwise the last regular plane, with its field back in the internal FFT convention. Modal amplitudes remain in output units until the driver applies the weight at z.
n_savedint: Number of regular planes stored. The next scheduled index is determined from the actual position, independently of this count.
historysolprop.output.schema.StepHistory: Every attempt of the whole run so far, concatenated.
outcomesolprop.output.schema.RunOutcome or None: Completion status and cumulative call ledgers; absent from older files.
checkpointsolprop.output.schema.TerminalCheckpoint or None: Separate accepted terminal state, preferred over the last regular save.
marchsolprop.output.schema.MarchSnapshot or None: The Adams march’s state where the run stopped, when the file kept one there.

Examples

>>> from solprop.output.hdf5 import ResumeState
>>> ResumeState.__doc__.splitlines()[0]
'Everything a driver needs to carry on from a file it wrote earlier.'

checkpoint: TerminalCheckpoint | None = None#

history: StepHistory#

march: MarchSnapshot | None = None#

metadata: RunMetadata#

n_saved: int#

outcome: RunOutcome | None = None#

plane: SavedPlane#

solprop.output.hdf5.check_cache(path, metadata)[source]#

Say whether a file holds the run described by metadata, and refuse if not.

The resume gate. Hashing the statistic names and the state shape only would pick up a file written with different physics of the same shape silently and continue the run from someone else’s state; here the digest covers the grid, the modes, the whole serialised configuration, the noise and the stepper settings, and a mismatch names the fields that changed.

Parameters:

pathpathlib.Path or str: The file to check.
metadataRunMetadata: The record of the run about to start.

Returns:

bool: True when the file describes the same run and may be resumed; False when there is no file to resume from.

Raises:

OutputError: If the file exists but describes a different run, which includes every file written at another output schema, since the schema version is part of the resume key; or if the file’s general modal record names an accessor this build no longer admits (_check_accessors). A refusal for a changed material manifest names its entries: an optmat model’s identity, a material policy or the optmat version.

Examples

>>> from solprop.output.hdf5 import check_cache
>>> from solprop.output.schema import _example_metadata
>>> check_cache("no-such-file.h5", _example_metadata())
False

solprop.output.hdf5.open_file(path)[source]#

Open a finished run for reading, through the protocol it was written with.

The handler comes back open and read-only: __getitem__, fields, stat_names, n_saved and metadata answer from the file, open and save are refused, and close releases the file – a with block does that on the way out. It is how solprop.processing reads a file after the run that wrote it has ended; resume() reads the last plane only, and a handler built with append=True opens the file read-write to continue the run.

Parameters:

pathpathlib.Path or str: The file to read.

Returns:

HDF5Output: An open, read-only handler.

Raises:

OutputError: If the file cannot be opened, or carries no metadata.

Examples

>>> import numpy as np, tempfile
>>> from pathlib import Path
>>> from solprop.grid import EnvelopeGrid
>>> from solprop.output import MemoryOutput, open_file
>>> from solprop.output.schema import (
...     GridRecord, RunMetadata, SavedPlane, StepperRecord
... )
>>> from solprop.stepper import DOPRI5, PIController
>>> grid = EnvelopeGrid.from_limits(1.0, 835e-9, (400e-9, 2e-6), 2e-12)
>>> memory = MemoryOutput()
>>> memory.open(RunMetadata(
...     grid=GridRecord.from_grid(grid), modes=(), simulation_kind="mode_averaged",
...     configuration={},
...     stepper=StepperRecord.from_settings(
...         DOPRI5, PIController(rtol=1e-8, atol=0.0)
...     ),
... ))
>>> memory.save(SavedPlane(z=0.0, field=np.ones((1, 1, grid.n_omega), complex)))
>>> with tempfile.TemporaryDirectory() as folder:
...     memory.to_file(Path(folder) / "run.h5")
...     with open_file(Path(folder) / "run.h5") as stored:
...         shape, planes = stored["field"].shape, stored.n_saved
>>> shape, planes, stored.closed
((1, 1, 1, 2048), 1, True)

solprop.output.hdf5.resume(path)[source]#

Read a file’s resumable state and cumulative work, preferring its checkpoint.

The last plane comes back in the internal FFT convention, through solprop.fourier.spectrum_to_internal(), so a driver can hand it straight to a backend without conjugating anything itself. A separate terminal checkpoint contributes no regular observation but supplies its actual position, pending attempt history and cumulative call ledgers. Reading preserves its landing classification; the driver requires explicit resume="interpolated" consent before restarting from an interpolated field.

A run that keeps no field on its last regular plane (SEP 22) comes back with that plane’s field=None unless a terminal checkpoint, which always carries the field, follows it; the driver refuses to continue from it by name.

Parameters:

pathpathlib.Path or str: The file to read.

Returns:

ResumeState: The metadata, the last plane, how many planes the file holds, and every attempt the stepper made.

Raises:

OutputError: If the file cannot be read, carries no metadata, or holds no plane.

Examples

>>> import numpy as np, tempfile
>>> from pathlib import Path
>>> from solprop.grid import EnvelopeGrid
>>> from solprop.output import HDF5Output
>>> from solprop.output.hdf5 import resume
>>> from solprop.output.schema import SavedPlane, _example_metadata
>>> metadata = _example_metadata()
>>> n_omega = metadata.grid.n_omega
>>> with tempfile.TemporaryDirectory() as folder:
...     handler = HDF5Output(Path(folder) / "run.h5")
...     handler.open(metadata)
...     handler.save(SavedPlane(z=0.25, field=np.ones((1, 1, n_omega), complex)))
...     handler.close()
...     state = resume(Path(folder) / "run.h5")
>>> state.n_saved, state.plane.z
(1, 0.25)

Carry on in the same file with a handler built with append=True:

>>> with tempfile.TemporaryDirectory() as folder:
...     handler = HDF5Output(Path(folder) / "run.h5")
...     handler.open(metadata)
...     handler.save(SavedPlane(z=0.0, field=np.ones((1, 1, n_omega), complex)))
...     handler.close()
...     again = HDF5Output(Path(folder) / "run.h5", append=True)
...     again.open(metadata)
...     again.save(SavedPlane(z=0.5, field=np.ones((1, 1, n_omega), complex)))
...     again.close()
...     resume(Path(folder) / "run.h5").n_saved
2