FEAT: add ScenarioPresetRegistry owning preset storage and validation

[hannahwestra25]

Description

Prerequisite extracted from #2931 (preset definition + backend service/routes + Registry tab). This is the durable layer only; #2931 rebases onto it.

ScenarioPresetRegistry owns preset storage, validation, and version-checked resolution. It reads through to storage on every call and is not a cache: versions are sha256(stored_bytes), multi-worker is supported, and resolve_preset comparing a cached version against itself would launch a configuration nobody confirmed.

check_preset is the single validation path, replacing the service's ad-hoc _unknown_* helpers. It reads ScenarioMetadata off ScenarioRegistry directly, keeping pyrit.registry free of pyrit.backend.

Startup validation is parse-only: building scenario metadata instantiates every scenario class and raises before initialize_pyrit_async().

delete_preset returns bool from the delete itself, never a probe.

Addresses asks 1 (partially — no in-memory entries), 2, 3, 4 of the #2931 review.

Tests and Documentation

45 new registry tests; validation cases ported from #2931. scenario_presets_source documented in pyrit_conf.md.

registry + preset model + import boundary     189 passed
configuration loader + runtime lifecycle      154 passed
lazy package imports                          192 passed

[Richard Lundeen (richlundeen)]

Review summary

Thanks for pulling this out of #2931 — reviewing the storage and lock behaviour on its own was much easier.

What's working

The storage layer fixes the defects from #2931 and I think it's ready as-is:

• _replace_file stages to a temp file, fsyncs, then os.replace, so a failed write keeps the old preset.
• The lock is an OS advisory lock released by the kernel on exit, which replaces the 60-second age heuristic that couldn't prove the first writer had died.
• Names are validated before any read, so one bad file can't stop the listing.
• Blob uses overwrite=False for create and If-Match for update.
• custom_initializer_storage.py drops 137 → 76 lines, so the extraction is real reuse rather than speculative.
• The tests cover the hard cases: lock contention, dead-writer reclaim, partial writes, non-UTF-8 documents, blob preconditions.

I also ran it against a plain local directory to confirm nothing needs blob: save/load/list/delete, version conflicts, hand-edit detection, and the resolve guard all work, and azure.storage.blob is never imported.

Why I'm being picky about [1] and [2]

This isn't only the preset layer. It's the first durable named-configuration layer in the registry, and the intent is to persist target, scorer, and converter configurations the same way. Whatever shape lands here is what the next several registries copy, and #2931 rebases onto it immediately.

That makes two otherwise-minor things worth getting exactly right:

• [1] — TargetRegistry and ScorerRegistry are already Registry subclasses holding instances. Giving them durable configs means adding storage to a Registry, which is exactly what [1] asks for. If presets land as a bespoke class, the pattern doesn't transfer and the next registry reinvents it.
• [2] — "resolve through the real path, report advisorily" is the validation model every stored config needs, and it's what keeps a config portable between deployments. A second implementation here becomes a second implementation everywhere.

The rest is ordinary and can be taken or left on its merits.

The items

I've left these as separate inline threads so we can discuss and resolve them individually.

#	Item	Priority
1	Subclass Registry; add a registration surface	Blocks merge
2	check_preset should use the registry's resolution	Blocks merge
3	Blob error on a missing container	Blocks merge (small)
4	PR description fixes	Blocks merge (small)
5	Two method names that mislead	Cheap
6	Documentation	Cheap
7	Document conventions to settle now	Cheap
8	Optional version check on delete	Follow-up

[4] PR description fixes

No code to anchor this to, so putting it here.

• scenario_preset.py moved into this PR, so the #2956 ask moved with it. Suggest adding: "Related to #2956. Prefer to merge #2956 first. Then update presets to use the same dataset-limit normalization and GUI controls, including all." #2956 is still open, so merging this first is fine.
• "check_preset is the single validation path, replacing the service's ad-hoc _unknown_* helpers" — scenario_preset_service.py doesn't exist on main, so nothing was replaced; the helpers came over from #2931 largely unchanged. That line becomes true after [2].

---

Drafted with GitHub Copilot; reviewed and posted by me.

[Richard Lundeen (richlundeen)]

[1] Subclass Registry, and add a registration surface

Three issues with the code as it stands.

a. Nothing can contribute a preset but a JSON file. PR 2 of the composite-scan plan is "an example PyRITInitializer that registers a configuration through the same public registry API" — there's no API to call.

b. It rewrites infrastructure the base class already provides. _singleton, _singleton_lock, and get_registry_singleton duplicate Registry.get_registry_singleton, which already does this under a per-class lock rather than one shared lock. A named-instance container, tags, validate_name_available, and unregister would each be hand-rolled next.

c. The docstring rejects Registry for a reason the tree already overruled. It defends "no cache" — but nobody is asking for one. A registered preset has no document, so it never takes part in a version check. Meanwhile AttackTechniqueRegistry is already Registry["AttackTechniqueFactory", AttackTechniqueMetadata] with _discover registering nothing, an explicitly empty placeholder metadata class, and instances holding definitions, not live attacks — the exact shape this docstring calls unacceptable.

Suggested change

# pyrit/registry/components/scenario_preset_registry.py
@dataclass(frozen=True)
class ScenarioPresetMetadata(RegistryMetadata):
    """Placeholder: the buildable catalog is empty because presets are documents."""


class ScenarioPresetRegistry(Registry["ScenarioPreset", ScenarioPresetMetadata]):
    def __init__(self, *, storage: ScenarioPresetStorage | None = None, lazy_discovery: bool = True) -> None:
        super().__init__(lazy_discovery=lazy_discovery)
        self._storage = storage
        self.instances: InstanceRegistry[ScenarioPreset] = DefaultInstanceRegistry(instance_type=ScenarioPreset)

    def _discover(self) -> None:
        """Register no classes: a preset is a document, not a buildable component."""

    def _metadata_class(self) -> type[ScenarioPresetMetadata]:
        return ScenarioPresetMetadata

instances requires its items to be Identifiable, so add one method to the model. Pydantic's ModelMetaclass derives from ABCMeta, so the mixin composes without a metaclass conflict:

class ScenarioPreset(BaseModel, Identifiable):
    def _build_identifier(self) -> ComponentIdentifier:
        return ComponentIdentifier(
            class_name=type(self).__name__,
            class_module=type(self).__module__,
            params=self.model_dump(mode="json", exclude_none=True),
        )

Then hold registered presets in instances and merge them into reads. Stored presets stay read-through — that insight is correct and survives unchanged.

def register_preset(self, *, name: str, preset: ScenarioPreset, tags=None) -> None:
    self.instances.register(preset, name=name, tags=tags)


def list_presets(self) -> list[PresetEntry]:
    entries = {
        name: PresetEntry(preset=item.preset, origin="stored", version=item.version)
        for name, item in self._get_storage().list_presets().items()
    }
    for entry in self.instances.get_all_instances():
        entries.setdefault(entry.name, PresetEntry(preset=entry.instance, origin="registered"))
    return [entries[name] for name in sorted(entries)]

PresetEntry = preset + origin: Literal["stored", "registered"] + version: str | None.

Case	Behavior
save_preset names a registered preset	Refuse via instances.validate_name_available, so names never collide
resolve_preset on a registered preset	Require expected_version is None
delete_preset on a registered preset	Return False; use instances.unregister

Also move the file to pyrit/registry/components/, where #2931 asked for it.

Costs

An empty _discover, a placeholder metadata class, and one _build_identifier. You're right that the first two are dead surface — I'd pay it now for consistency.

Follow-up, not this PR

Pull class discovery and metadata out of the Registry base and make them a composed capability, the way instance-holding already is. instance_registry.py describes that pattern and an in-flight migration for it (Phase 1 foundation → Phase 4): because instance-holding is a typed property, "a registry that does not hold instances simply has no .instances attribute." Applied the other way, a registry with no classes to build simply has no .classes, and AttackTechniqueRegistry and ScenarioPresetRegistry both stop carrying dead surface. Worth its own issue.

Open

PresetEntry, or loosen StoredPreset.version to str | None? Should a stored preset shadow a registered one, or collide?

---

Drafted with GitHub Copilot; reviewed and posted by me.

[Richard Lundeen (richlundeen)]

[2] check_preset should use the registry's resolution

Same principle as [1]: use the layer's machinery instead of rewriting part of it.

The problem

The description says this replaces the service's _unknown_* helpers. They were moved, not replaced — scenario_preset_service.py doesn't exist on main.

The repo already states the rule. Scenario.set_params_from_args (scenario.py:518):

the coerce / validate / inject-defaults mapping is owned by the registry layer (resolve_declared_params) so there is a single implementation shared by the programmatic, CLI, and registry paths.

_unknown_parameter_issues is a second implementation of a slice of that — the name check, with coercion, choices, and types dropped. So {"max_turns": "banana"} is reported clean here and then fails at launch, which is the failure check_preset exists to prevent.

_unknown_technique_issues is the same pattern for tokens. Not a live bug today — get_all_techniques and get_aggregate_techniques partition the enum, so its set test and the launch path's technique_class(base_name) agree — but it's a second implementation of one rule, which is what #2931 asked to remove.

The root cause is that the resolution lives in the wrong layer. ScenarioConfigurationResolver sits in pyrit/backend/services/, so pyrit.registry can't reach it — but it has no backend dependency at all. It imports only pyrit.registry and pyrit.scenario.core, its docstring calls it "registry-backed", and every method is a registry lookup. tests/unit/scenario/garak/test_divergence.py and test_api_key.py already import it from pyrit.backend.services, which is the symptom.

What to change here

The placement decision is being made in this PR, so make it fully. _technique_tokens.py is new — 59 lines, none pre-existing. It currently holds only the string split and leaves resolution in the backend, which cements a split home and means a follow-up has to move a file this PR just created.

Destination is pyrit/scenario/core/technique_tokens.py, per #2931: "The token grammar belongs to scenarios, and scenario core already looks up registries." The ConverterRegistry lookup is a call, not ownership, and _technique_resolution.py already lives beside it and reads AttackTechniqueRegistry through a deferred import.

Params need no move — resolve_declared_params is already in pyrit/registry/resolution.py.

2a. Params. ScenarioMetadata.supported_parameters is already tuple[Parameter, ...], so this drops in:

try:
    resolve_declared_params(
        declared=list(metadata.supported_parameters),
        raw_args=preset.scenario_params,
        owner=f"Scenario '{metadata.registry_name}'",
    )
except ValueError as error:
    issues.append(PresetIssue(field="scenario_params", message=str(error)))

This is the whole check — set_params_from_args does only a reserved-name check on the scenario's own declaration (caught at registration) plus resolve_declared_params, so no scenario instantiation is needed.

2b. Techniques. _build_metadata already computes the technique class at scenario_registry.py:175, so add it as a ScenarioMetadata field, then move resolution (not just the split):

@dataclass(frozen=True)
class TechniqueTokenResolution:
    techniques: list[Any]
    technique_converters: dict[str, list[Converter]]
    issues: list[str]


def resolve_technique_tokens(*, tokens: Sequence[str], technique_class: type[Any]) -> TechniqueTokenResolution: ...

Collecting issues rather than raising lets each caller choose: launch raises on resolution.issues, check_preset maps them to PresetIssue. Then _unknown_technique_issues and _technique_issues can go. Keep _forbidden_baseline_issues — baseline_policy isn't a token rule.

Follow-up, not this PR

Target resolution, scenario-class resolution, and dataset-config resolution belong in the registry, for the same reason the technique rule belongs in scenario core: they're registry lookups, and none is a backend concern. It's a follow-up only because nothing this PR adds depends on where they live — after 2b, check_preset uses technique_tokens and resolve_declared_params, never ScenarioConfigurationResolver.

Open

Is a type[Any] field acceptable on frozen ScenarioMetadata, which otherwise holds class_name / class_module?

---

Drafted with GitHub Copilot; reviewed and posted by me.

[Richard Lundeen (richlundeen)]

[3] Blob error on a missing container

Both paths are wrapped in one except (ResourceExistsError, ResourceModifiedError, ResourceNotFoundError), so a missing or mistyped container on create reports:

Scenario preset 'x' could not be saved because it no longer exists. Reload it and reapply.

That sends someone looking for a concurrent edit when the container doesn't exist.

Splitting the paths fixes it:

• Create: catch ResourceExistsError only. Let ResourceNotFoundError propagate as AzureError — routes/initializers.py already maps that to a storage-unavailable response.
• Update: catch (ResourceModifiedError, ResourceNotFoundError). There, not-found genuinely means the blob was deleted between the read and the write, so a conflict is right.

test_blob_update_of_a_deleted_document_is_a_conflict still passes. Worth adding one test: create against a missing container raises AzureError rather than ScenarioPresetConflictError.

---

Drafted with GitHub Copilot; reviewed and posted by me.

[Richard Lundeen (richlundeen)]

[5] Two method names that mislead

Both describe something other than what they do.

validate_stored_presets parses, logs, and counts. It doesn't validate in the check_preset sense, and its only caller discards the result. Suggest load_stored_presets, keeping -> int since the count is useful and test_it_counts_the_presets_that_loaded asserts it. Would need updating in runtime_lifecycle.py and the TestStartupValidation class name.

get_preset_version exists only to recover a document that can't be parsed, but reads as the normal way to get a version — so a caller may well reach for it as one. Suggest get_unparsable_preset_version, or say so in one docstring line.

---

Drafted with GitHub Copilot; reviewed and posted by me.

[Richard Lundeen (richlundeen)]

[6] Documentation

Three asks, one theme.

Trim the docstrings

File	Lines	Docstring	Share
scenario_preset_registry.py	428	248	57%
scenario_preset_storage.py	251	137	54%
scenario_preset.py	131	68	51%
file_document_storage.py	656	302	46%

Repo guidance is "we prefer minimal documentation. Code should be self-explanatory."

Suggested rule: behavior, Args, Returns, Raises. Keep a "why" note only where the code looks wrong without it — the _local_document_lock note on never deleting the lock file is a good example to keep.

What I'd drop is reviewer-facing argument, e.g. "A caching preset registry was previously removed for these reasons. Reintroducing one would reintroduce the same defects, so it is not an optimization left for later." That belongs in the PR description. In the source it will outlive the design and eventually be confidently wrong.

validate_stored_presets runs ~30 docstring lines over an 8-line body. One fact in it is genuinely non-obvious and worth about three lines: the pass checks the schema only, because the registries check_preset needs aren't populated yet.

Say why this doesn't build on StorageIO

Two sentences in this module docstring. StorageIO (pyrit/memory/storage/storage.py) is async, memory-owned, and has no conditional-write or lock primitive. The tree now has two local-plus-blob abstractions, and without the note the next reader has to re-derive why.

To be clear, I think keeping this class in pyrit/registry/ is right. I'd checked whether it belonged in pyrit/common/, but that package is a strict leaf — it imports nothing from any other pyrit.* package — and this needs validate_registry_name from pyrit.models.identifiers. Documents are addressed by registry names anyway.

Document the lock files

Not deleting them is correct; it just needs to be visible. Suggest adding to pyrit_conf.md under scenario_presets_source:

PyRIT writes a small hidden .<name>.json.lock file beside each preset to serialize writers. These files stay after a preset is deleted, and are safe to remove when no PyRIT process is running.

I confirmed this locally — after deleting a preset, .nightly.json.lock remains. An operator pointing at a shared directory will watch these accumulate.

---

Drafted with GitHub Copilot; reviewed and posted by me.

[Richard Lundeen (richlundeen)]

[7] Document conventions to settle now

These are conventions every stored configuration will share, and retrofitting them across an existing corpus of documents is much worse than deciding them on the first one.

A schema version field. A preset document carries no version of its own shape, so once the model changes a reader can't tell an old document from a malformed one. ScenarioMetadata already carries scenario_version, so the concept exists. Worth adding the document equivalent and deciding the convention here rather than per registry.

Confirm the document says what to build. A preset names scenario_name, which is enough for a scenario. A target or scorer config needs class identity — class_name / class_module, which ComponentIdentifier already models. Worth confirming now that preset documents follow whatever convention the others will, rather than discovering later that presets are the exception.

Keep ScenarioPresetStorage thin. Almost nothing in it is preset-specific: decode → json.loads → model_validate, model_dump → sorted JSON → bytes, document-name-wins, hash the bytes, typed conflict error. Only the model type and the directory name are. A later JsonDocumentStorage[TModel] could absorb the rest, so I'd avoid growing preset-specific logic into those methods meanwhile.

---

Drafted with GitHub Copilot; reviewed and posted by me.

[Richard Lundeen (richlundeen)]

[8] Optional version check on delete

Every write is version-guarded except delete, so a delete can discard an edit another operator saved moments earlier.

Suggest expected_version: str | None = None, defaulting to unconditional so the current race tests still hold. Locally, compare inside the existing lock; on blob, pass the ETag with IfNotModified.

If unconditional delete is the intended contract, a single docstring line would settle it — right now the asymmetry reads as accidental.

Happy for this to be a follow-up.

---

Drafted with GitHub Copilot; reviewed and posted by me.