solprop.backends.numba_cpu.openmp#

The pyFFTW import, made so that its OpenMP threads sleep when they are idle.

pyFFTW’s Linux wheels run FFTW’s threads through their own copy of GNU OpenMP (libgomp). That runtime reads OMP_WAIT_POLICY once, when it is loaded. With the variable unset, a thread that finishes a parallel region spins for GNU’s default count before it sleeps. The kernels’ Numba pool runs next on the same cores, so the spinning FFTW threads hold cores that the kernels’ threads want. The kernels’ threads then hold the cores that FFTW’s team needs for its next transform, and each transform waits for the scheduler.

Measured on the four-mode RDW capillary (the suite’s rdw), on 8 cores of a two-socket x86-64 server with the Numba pool at 8 on its TBB layer: with FFTW’s pool at 8 threads a planned transform took 3.5-5.9 ms in place of the 0.1 ms it takes at one thread, and a step 129 ms in place of 25 ms. At 2 and 4 FFTW threads the step took 128 ms. With OMP_WAIT_POLICY=passive it took 25.4 ms at 8 FFTW threads. On the full CtC benchmark window, whose transforms get four FFTW threads by the backend’s own size rule, the passive policy took the window from 6.92 to 4.85 s. An Apple-silicon laptop, whose pyFFTW wheel uses LLVM’s OpenMP, showed no such slowdown, and this module leaves other platforms alone.

So on Linux the import sets OMP_WAIT_POLICY=passive for pyFFTW’s runtime and restores the environment straight away. An OpenMP runtime that loads later reads the user’s environment, not this setting: the native backend’s runtime, or Numba’s OpenMP layer. It is not done when the user has set OMP_WAIT_POLICY or GOMP_SPINCOUNT, since then that choice is theirs. It is not done either when pyfftw was imported before this module, because its runtime has then already read the environment. WAIT_POLICY says which of these happened, and the backend’s metadata records it as fftw_wait_policy.

Module Attributes

`VARIABLES`	The variables by which a user chooses how GNU OpenMP's idle threads wait.
`BINDING_VARIABLES`	The variables that make GNU OpenMP bind its threads (`OMP_PLACES` alone does).
`WAIT_POLICY`	How pyFFTW's OpenMP threads wait in this process (`wait_policy()`).
`BINDING`	Whether pyFFTW's OpenMP threads are bound here (`binding_policy()`).

Functions

`binding_policy`(platform, environ, *, loaded)	Return whether a runtime about to load would bind its threads.
`numba_launch`([platform])	Start Numba's threads inside this block so that they wait passively and unbound.
`wait_policy`(platform, environ, *, loaded)	Return how pyFFTW's OpenMP threads will wait, and so whether to set it.

solprop.backends.numba_cpu.openmp.BINDING = 'none'#: Whether pyFFTW’s OpenMP threads are bound here (binding_policy()).

solprop.backends.numba_cpu.openmp.BINDING_VARIABLES = ('OMP_PROC_BIND', 'OMP_PLACES')#: The variables that make GNU OpenMP bind its threads (OMP_PLACES alone does).

solprop.backends.numba_cpu.openmp.VARIABLES = ('OMP_WAIT_POLICY', 'GOMP_SPINCOUNT')#: The variables by which a user chooses how GNU OpenMP’s idle threads wait.

solprop.backends.numba_cpu.openmp.WAIT_POLICY = 'passive'#: How pyFFTW’s OpenMP threads wait in this process (wait_policy()).

solprop.backends.numba_cpu.openmp.binding_policy(platform, environ, *, loaded)[source]#

Return whether a runtime about to load would bind its threads.

On Linux, GNU OpenMP binds its threads, the main thread among them, when OMP_PROC_BIND is set to anything but false or when OMP_PLACES is set. For numba_cpu’s runtimes that is ruinous: the comb-to-continuum window took 45-47 s with binding against 17 s without, on 64 Zen 5 cores, whatever Numba’s threading layer (SEP 28), because the bound runtimes stack their threads on the same few cores. Binding remains the user’s choice for the native backend’s runtime.

Parameters:

platformstr: sys.platform.
environMapping: The process environment.
loadedbool: Whether the runtime has already read the environment.

Returns:

str: "unbound" when the load should set OMP_PROC_BIND=false, which is on Linux only and only when binding is asked for; "inherited: ..." when the runtime is loaded already under binding; "none" when nothing asks for it.

Examples

>>> from solprop.backends.numba_cpu.openmp import binding_policy
>>> binding_policy("linux", {"OMP_PROC_BIND": "close"}, loaded=False)
'unbound'
>>> binding_policy("linux", {"OMP_PROC_BIND": "false"}, loaded=False)
'none'

solprop.backends.numba_cpu.openmp.numba_launch(platform='linux')[source]#

Start Numba’s threads inside this block so that they wait passively and unbound.

Numba’s OpenMP threading layer loads the shared GNU OpenMP runtime when its threads first start. With every core in the pool, its workers spin between parallel loops on the cores FFTW’s threads need: the comb-to-continuum window took 17 s on 64 Zen 5 cores against 5.4 s with OMP_WAIT_POLICY=passive (SEP 28). On Linux, when nothing has decided otherwise, the runtime is loaded passive and unbound; when it is already loaded (by the native backend’s build, say) under settings this cannot change, Numba’s own workqueue layer, which uses no OpenMP, is chosen instead (5.5 s on the same window).

Parameters:

platformstr, optional: sys.platform.

Yields:

str: What applies, for the run’s metadata: "passive", "workqueue", "started" (Numba’s threads were running already), "environment" (the user chose a layer or a wait policy) or "runtime default".

solprop.backends.numba_cpu.openmp.wait_policy(platform, environ, *, loaded)[source]#

Return how pyFFTW’s OpenMP threads will wait, and so whether to set it.

Parameters:

platformstr: sys.platform.
environMapping: The process environment.
loadedbool: Whether pyfftw is already imported, in which case its runtime has read the environment already.

Returns:

str: "passive" when this import should set it, which is on Linux only and only when nothing else decides it. Otherwise it describes what decides it: "inherited" when pyFFTW was imported first, "environment: NAME=value" when the user set one of VARIABLES, or "runtime default" on other platforms.

Examples

>>> from solprop.backends.numba_cpu.openmp import wait_policy
>>> wait_policy("linux", {}, loaded=False)
'passive'
>>> wait_policy("linux", {"OMP_WAIT_POLICY": "active"}, loaded=False)
'environment: OMP_WAIT_POLICY=active'