FEAT: Save API-created targets, converters, and scorers across restarts

[varunj-msft]

Description

Targets, converters, and scorers created through the API (and the GUI) now survive a backend restart and a live reinitialization, including local development with no Azure. Each create, replace, and delete writes through to a saved recipe (type, constructor parameters, references to other instances, and the names of the environment variables that hold its credentials), and the backend rebuilds the saved instances after the initializers run, in dependency order.

This follows the direction from #2515 and #2931: one shared document-storage layer, recipe separate from credential, no run-data database dependency.

Storage. FileDocumentStorage is carried byte-for-byte from #2931's head 8959c7b2d (commit 1, co-authored), conditional writes included: a version is the SHA-256 of the stored bytes; Blob uses overwrite=False for create and the ETag of the compared bytes for replace; a local write holds an operating-system lock on a file beside the document across read/compare/replace, which the system releases if the writer exits. Commit 2 adds the conditional delete recipes need on the same primitives, so this PR adds no dependency. Recipes live in instance_recipes_source (default ~/.pyrit/instance_recipes, or an Azure Blob container), independent of memory_db_type. A local directory that cannot be listed is a store error, as a failed Blob listing is, rather than an empty store (this also applies to stored custom initializers, which glob used to read as none).

Credential model (the build gate). Credential parameters (api_key, headers, http_request, sas_token, and the rest of ComponentIdentifier.get_sensitive_parameter_names()) are never saved as values: a request names a server environment variable instead, "credentials": {"api_key": {"env_var": "OPENAI_CHAT_KEY"}}, which requires administrator access. Values that carry a credential in a recognized way are rejected: a URL with user information or a credential query parameter (a URL written in another URL's query or fragment, such as a proxy's upstream URL, counts too), a credential field inside free-form settings such as httpx_client_kwargs, and a key/code/sig in query parameters, headers, or authentication settings in any shape (map, list of pairs, query string). One CredentialNames rule set serves the URL and field checks; names are compared by their letters and digits in any letter case, a number counts like text, and a switch value such as true or 0 never counts. These checks are best effort and say so; a token in a URL path is not detected, and a credential written into a URL must be percent-encoded, as URLs require, to be recognized whole (user information is read as a URL parser reads it, except that when what follows it cannot be a host and a port, a password's unencoded /, ?, or # is read through to the last @; in a URL written inside another URL's query, an unencoded & or # leaves the credential unrecognized). Errors from building an instance have resolved credential values, and the secrets inside them, removed (one shorter than four characters where it stands alone as a word; every value of a credential that is a JSON object, such as a headers map, counts, so an ordinary one such as application/json is removed too), and the user information and credential parameters of any URL they quote are masked, read as written so a malformed request template's URL is covered. Type metadata marks credential parameters sensitive (each registry derives the flags from its identifier type); the server enforces those flags, and the target and converter dialogs ask for an environment variable for them.

Conflicts, atomicity, failures. Saving is the commit point (build → save with CAS → register); the instance is built from a copy of the recipe, so a constructor that changes its arguments cannot change what is saved, and a failed registration removes the recipe or puts the replaced one back byte for byte. PUT and DELETE require the version that was read (409 when stale; a DELETE without one gets 428). A saved instance other saved instances reference cannot be deleted, or changed while restored — a restored instance counts by what it was built from, even after its saved document changes outside the API, until a restart or reinitialization rebuilds it, and with the document's current version a replace applies to it while this PyRIT can still use the document and a delete by its name (or by its document name when this PyRIT cannot use the document) removes it (an instance is taken for an initializer's only when it was not built from a saved recipe) — and while a saved document cannot be read from storage such a delete or change is refused (503), because it may reference the instance; a saved instance whose type is no longer registered counts as referencing every name its parameter values hold, and a saved document this PyRIT cannot use (for example one a newer PyRIT wrote) counts as referencing every name in its fields other than schema_version, kind, name, and type when a saved instance it names is deleted; an administrator can delete such a document by its instance name or its document name, even while saved instances reference it, and create the instance again — unless an instance built from it before it changed is still live and others hold it, which is refused like any live instance until they change or a restart rebuilds them. A saved instance that cannot be rebuilt (unset variable, unknown type, name taken by an initializer, failed dependency, unreadable or newer-format document) is listed with its reason in unrestorable, and GET returns that reason when no live instance holds the name, instead of vanishing; instances that reference it are not rebuilt against another instance that now holds its name, and saving such a reference is refused (409) until it is deleted — as is any reference while the last restore failed as a whole (restore_error), and any create or replace while the store also still cannot be listed (503) — so what the API accepts is what the next restart rebuilds. A document is listed under its document name, which DELETE accepts (a name of that shape always means the document itself), when it does not name a valid, unreserved instance that maps back to it, and files whose names do not have the shape of a document name are ignored. A create from an older client that sends no name is saved under its generated compat_… name and can be deleted like any other.

Commits (each passes its own tests):

1. MAINT Extract FileDocumentStorage from CustomInitializerStorage (verbatim from FEAT: Add ScenarioPreset #2931 at 8959c7b2d, conditional writes and operating-system locks included)
2. FEAT Conditional deletes for FileDocumentStorage, on FEAT: Add ScenarioPreset #2931's conditional-write primitives
3. FEAT Recipe models and storage, a larger get_sensitive_parameter_names, CredentialNames, UrlCredentials
4. FEAT Restore-order planner
5. FIX Mask credentials in HTTPTarget identifier endpoints
6. [BREAKING] FEAT Write-through persistence service, routes (PUT/DELETE), lifecycle hooks
7. FEAT GUI: target name, environment-variable fields, delete, unrestorable lists; e2e backend store isolated under dbdata/
8. DOC Configuration, GUI, backend README, registry docs

Why breaking. A create request that sends a credential parameter as a value (for example params.api_key), or a value that carries a credential, now gets 400 (with identity authentication an api_key value is still dropped, as before); deleting a saved instance requires ?version=; naming an environment variable requires administrator access, so only an administrator can create an HTTPTarget, whose required http_request must be a reference; a target's identifier no longer includes credentials from an HTTPTarget request URL; a scorer can no longer be named types, which GET /api/scorers/types would shadow. Storing raw keys at rest is what the story rules out, so these requests cannot keep working unchanged.

Coordination.

• FEAT: Add ScenarioPreset #2931: commit 1 is identical to its storage files at head 8959c7b2d, whose conditional writes and per-document locks this PR uses instead of its own, so whichever lands first, the other drops its copy; commit 2 only adds a conditional delete on those primitives. That head replaced the age-based stale-lock recovery Rich flagged with operating-system locks, which the system releases when a writer exits, so no live writer can lose its lock. A local read of a saved document takes that document's lock (Windows cannot replace an open file), while a read of a name with nothing saved takes none, since FEAT: Add ScenarioPreset #2931's lock files stay in place and would otherwise accumulate; a listing reads without locks, as FEAT: Add ScenarioPreset #2931's does, since every replacement is atomic.
• FEAT: Add custom params to new targets in CoPYRIT #2846: this branch is rebased on it and builds on its Parameter.sensitive, multiline, and identity_conflicting flags and get_auth_mode_parameters. sensitive here also means "send a reference, never the value", so ComponentIdentifier.get_sensitive_parameter_names() gains the parameters that can carry a credential (azure_speech_key, cookie, headers, http_request; its pinned-set test is updated). The server enforces the flags from the type's catalog parameters, as the GUI does, so a constructor parameter that a subclass identifier marks sensitive or identity-conflicting is enforced too (a registry reference, which names another instance, is never a credential). The target dialog renders every sensitive parameter, the multiline http_request included, as an environment-variable input that sends credentials (as CreateConverterDialog does, via frontend/src/utils/credentialReference.ts), and identity authentication still clears the identity-conflicting ones. TargetService.build_async applies get_auth_mode_parameters at every build, so a restored identity-mode AzureBlobStorageTarget keeps its auth_mode. Two behaviors change: with identity authentication, a value or reference for any parameter the catalog marks identity_conflicting is refused (400) where FEAT: Add custom params to new targets in CoPYRIT #2846 dropped values, and headers is now marked too, since a map can carry an Authorization header, so the dialog clears it under identity as it does sas_token.

Open questions for maintainers (defaults chosen): admin for credential references only (vs. every mutation); headers/http_request reference-only, so even a credential-free header map comes from a variable (vs. accepting values and rejecting only credential-looking entries, as free-form settings such as httpx_client_kwargs.default_headers already are); update of a referenced live instance refused (vs. cascade); initializer wins a name collision; environment-variable references only (no direct Key Vault kind); whether replacing or deleting a target should run its own cleanup (for example closing a realtime target's connections), which nothing in PyRIT calls today; restoring saved instances in the backend only (the storage and restore planner live in pyrit/registry, so notebooks and pyrit_scan could restore them in a follow-up).

Tests and Documentation

New tests: tests/unit/registry/test_{file_document_storage,instance_recipe_storage,instance_restore,sensitive_parameter_names}.py, tests/unit/backend/test_{instance_persistence_service,instance_credentials}.py, tests/unit/common/test_{credential_names,url_credentials}.py, tests/unit/prompt_target/target/test_http_target.py (identifier masking), tests/integration/registry/test_instance_recipe_storage_integration.py (real Blob, gated), Jest for the target and converter dialogs, targets page, converter registry, UnrestorableInstances, credentialReference, and the API client; existing backend, registry, and Playwright suites updated.

Docs: doc/getting_started/pyrit_conf.md, .pyrit_conf_example, doc/gui/0_gui.md (creating and saved targets, registry API migration notes), pyrit/backend/README.md, doc/code/registry/0_registry.md, frontend/README.md (local end-to-end runs save what they create). No notebooks changed, so JupyText was not run.