[essreduce] Add failsafe loaders (generic, loki, dream)

[nvaytet]

This PR adds some generic 'fail-safe' loaders load_detectors() and load_monitors() to load data from files without failing if some transformations are incomplete or other common errors.

The functions first try to use the GenericNeXusWorkflow to load the data, but falls back to raw Scippnexus code if exceptions are raised.
Some warnings may be raised, but something should still be loaded.

By default, all detector banks or monitors are loaded. You can pass a list of detector or monitor names to load if you want to load only a subset: load_detectors(filename="file.hdf", detectors=["bank_0", "bank_1"]).

Some instrument-specific wrappers have also been added for Loki and Dream, which reshape the loaded detector banks into logical dimensions: ess.loki.diagnostics.load_detectors(), ess.loki.diagnostics.load_monitors().

Roll-out to other instruments will be done in a follow-up.

[SimonHeybrock]

Having a loader for inspecting files that the reduction workflows cannot handle makes sense. Not computing positions when the transformation is time-dependent is fine for that purpose.

I ran both code paths on every detector and monitor in the 2026-10-06 coda drop (all ten instruments). 26 components fail in the workflow and use the fallback. 15 of these are caused by two essreduce bugs, not by broken files:

• #797: monitors written by the current file writer (auxiliary_signals=['frame_total']) fail with KeyError: 'frame_total'. This affects DREAM, LoKI, NMX and BIFROST. It is also what the "badly formed monitor data" in loki_broken_file is.
• #798: monitors with depends_on='.' fail with a UnitError. This affects all FREIA monitors and MAGIC beam_monitor_1.

I will open fixes for both. After that, the fallback is only used for files that are actually broken: empty NXlogs in the transformation chain, dangling depends_on chains, and the BEER detectors where ScippNeXus cannot determine the signal. In this drop, no monitor is returned as a raw DataGroup once these are fixed. Some tests in this PR expect the fallback warnings for monitors, so they will need updating (see inline).

Two remarks on the interface:

• Naming and placement. ess.reduce.nexus.load_detectors sits next to load_data and load_component, which feed the reduction. ess.loki.load_detectors reads like the way to load LoKI data. Once people use these names, it is hard to change what they mean. Can we make the inspection purpose visible, e.g. a ess.reduce.nexus.inspect module or a more specific name?
• No way to select pulses. load_detectors(filename) loads all events of all banks. The workflow already supports TimeInterval. It does not need to be part of this PR, but the signature should leave room for it (keyword-only arguments after filename).

[SimonHeybrock]

If both paths fail, the component is dropped without any message. Examples from current files:

load_detectors('does_not_exist.hdf', detectors=['loki_detector_0'])  # DataGroup(keys=[])
load_monitors(loki_file, monitors=['typo_name'])                     # DataGroup(keys=[])
load_detectors(tbl_file)  # 6 of 7 detectors, orca_detector missing

For a tool meant for debugging, this hides exactly what the user needs to know. Suggestion: raise if a name was requested explicitly, and warn and skip when loading everything (None).

Related: the warnings the tests check for ("depends_on chain", "Falling back") come from ScippNeXus, not from this code. Nothing tells the user that the fallback was used and why the result looks different. Something like this would help, and it gives the tests a warning of our own to match:

warnings.warn(f"{name}: workflow failed ({e}), loaded raw NeXus group instead", stacklevel=2)

Catching Exception in both blocks also hides bugs in our own code, such as #797 and #798.

[nvaytet]

Catching Exception in both blocks also hides bugs in our own code, such as #798.

What would be the alternative to catching Exception if we want the loaders to be fail-safe?

[SimonHeybrock]

See above:

For a tool meant for debugging, this hides exactly what the user needs to know. Suggestion: raise if a name was requested explicitly, and warn and skip when loading everything (None).

[nvaytet]

Sure, I implemented that. I'm just wondering about the part where we don't want to hide bugs in our code. Should the actual error be brought up to the top level (either in the form of an error or a warning) so we can see it?
I think this will constitute unnecessary noise for the user?

As it currently stands, we get something like:

UserWarning: loki_detector_0: loading with default loader failed, attempting fallback
UserWarning: depends_on chain {x} contains empty time-series, transform = combine_transformations([t.build() for t in chain])

So we see what went wrong inside the fallback, but not the error that was raised by the workflow loader.
The info from the workflow might be useful for us? Should I add a flag to control how much we want printed?

[SimonHeybrock]

I think this is fine, we can iterate later if it is insufficient.

[SimonHeybrock]

One thing I missed earlier: the raise/warn for components that fail to load never runs. The continue in the fallback's except skips the if item not in dg check, so a requested name that does not exist still comes back silently:

load_monitors(loki_coda_file, monitors=['typo_name'])  # DataGroup(keys=[]), no error
load_detectors('does_not_exist.hdf', detectors=['loki_detector_0'])  # same

Replacing continue with pass should fix it. Could you add a test for this case, e.g. a non-existent name expecting RuntimeError?

[SimonHeybrock]

Could the docstring say what the user gets back? The result depends on which path succeeded: position is only present when it could be computed, and a monitor that ScippNeXus cannot interpret comes back as a raw DataGroup. The instrument_view cell in the notebook relies on position, for example.

[SimonHeybrock]

read_nexus_file_info in _nexus_loader.py already lists detectors and monitors. It finds the entry and instrument groups with _unique_child_group instead of a hardcoded /entry/instrument. Could we reuse it?

[nvaytet]

I made use of read_nexus_file_info and _unique_child_group and removed the hardcoded /entry/instrument path.

[SimonHeybrock]

This is identical to _load_detector_with_fallback, and load_monitors is a copy of load_detectors apart from the fold. One private function that takes the NeXus class and the workflow key would halve the module.

[SimonHeybrock]

da.dim raises for data with more than one dimension. The BEER detectors already come back from the fallback with three unnamed dims (dim_0, dim_1, dim_2, 12×1000×1000), so a BEER wrapper would fail here.

Also, the workflow already folds via DetectorBankSizes, before computing positions. Setting wf[DetectorBankSizes] = bank_sizes would reuse that for the workflow path, and only the fallback would need folding here.

[SimonHeybrock]

This warning comes from #797 (KeyError: 'frame_total' in the workflow, then the fallback). The test will fail once that is fixed. The same applies to the "Falling back" checks for monitors in the essreduce and esssans tests.

[SimonHeybrock]

This downloads both files at import time, i.e. during test collection. Without network access, collection of the whole test module fails rather than the individual tests. A fixture (session-scoped, like in essreduce's conftest.py) avoids that.

[SimonHeybrock]

Missing space: this renders as "interactive figuresin a Jupyter notebook".

[SimonHeybrock]

tool._tool.click(...) is private plopp API. A plopp change can break this docs page without warning. Is there a public way to do this, or could we show the inspector without pre-drawn rectangles?

[nvaytet]

This is done in the same way in other places. In any cases, and change in Plopp's API would be beyond the scope of this PR.

or could we show the inspector without pre-drawn rectangles?

Not really, the plot would not be very interesting.

[nvaytet]

Naming and placement. ess.reduce.nexus.load_detectors sits next to load_data and load_component, which feed the reduction. ess.loki.load_detectors reads like the way to load LoKI data. Once people use these names, it is hard to change what they mean. Can we make the inspection purpose visible, e.g. a ess.reduce.nexus.inspect module or a more specific name?

I moved things in a diagnostics submodule, and renamed the file to load_nexus. I don't know if I'm already regretting that name, as load_nexus is a name that has been used in many places over the years, and it might still suggest that this is the way to load files, as opposed to the generic workflow.
However, it does sit at diagnostics.load_nexus rather than the top level, and I think load_nexus is better than safe_load which could have meant many things.

[nvaytet]

The module name diagnostics was chosen because both loki and dream already had a module with the same name. I thought creating a folder with that name and then putting the debugging loaders alongside the bank viewer code was the best thing to do (there should be no breaking changes to the API).
This is also the place I will put the generic detector bank viewer/masker which will be based on the loki.diagnostics.LokiBankViewer.

However, I can see that diagnostics may not be the best name we could come up with...