Repository navigation
FIX: restore TargetCapabilities modality combinations when reading the REST wire form - #2941
Conversation
TargetCapabilities documents itself as the REST wire snapshot of a target's capabilities, and TargetInstance embeds it as the FastAPI response model. But serialization excludes input_modalities/output_modalities and emits only their flattened supported_*_modalities projections, and nothing folds those back on read. A client that validates the payload -- as pyrit/cli/api_client.py:262 does for every GET /api/targets -- falls through to the text-only default, so the model contradicts its own payload: live supported_output_modalities == ['image_path'] dump "supported_output_modalities":["image_path"] after supported_output_modalities == ['text'] gpt-4o's six input modality combinations collapse to ['text'] the same way. Add a before-validator that rebuilds the combination fields from the flattened projections, mirroring Parameter._reconstruct_param_type_from_wire, which solves the same wire-shape problem in the same layer. The wire carries the union rather than the combinations, so one combination holding that union is restored -- enough to keep the flattened projection exact, which is the contract the wire documents.
| continue | ||
| flattened = data.get(flattened_key) | ||
| if isinstance(flattened, list): | ||
| combinations = [frozenset(flattened)] if flattened else [] |
There was a problem hiding this comment.
The bug is valid, but I would prefer to make TargetCapabilities serialization lossless rather than infer combinations from the flattened fields.
A flattened list cannot distinguish {{"text"}, {"image_path"}} (separate inputs) from {{"text", "image_path"}} (combined input). Restoring one union changes the meaning of the canonical model: it can advertise unsupported combinations and removes explicit text-only combinations. Existing requirement checks and modality routing use those distinctions.
Could we keep the immutable frozenset[frozenset[PromptDataType]] fields internally, serialize input_modalities and output_modalities as sorted lists of sorted lists, and retain the flattened supported_*_modalities computed fields for UI consumers? TargetConfiguration._capabilities_to_identifier_params() already uses that lossless, deterministic representation; the serialization itself belongs on the model, without importing the target-layer helper.
That would add fields to the wire schema without removing the fields the frontend currently reads. The round-trip tests should assert restored == caps for profiles with multiple combinations, with a nested TargetInstance/CLI deserialization case as well. The current flattened-list assertions pass even when the combinations change.
If we must retain the old wire shape, I would use an explicit summary model rather than represent unknown combinations as one declared combination.
frozenset[frozenset] fields were excluded from the wire form and rebuilt as a single combination union, which overstates supported combinations. Serialize them as sorted lists of sorted lists instead, keep the flattened supported_* projections as derived computed fields, and round-trip restored == caps in tests.
|
Agreed — the wire form now carries the combinations losslessly: |
Description
TargetCapabilitiesdocuments itself as the REST wire snapshot of a target's capabilities, andTargetInstance— the FastAPI response model forGET /api/targets— embeds it. But the model cannot be read back. Serialization excludes the modality combination fields and emits only their flattened projections, and nothing folds those back on read:model_config = ConfigDict(frozen=True)leavesextraat pydantic's default"ignore", and the class has nomodel_validator(mode="before"). So on read-back the two combination keys are absent, the text-only default applies, and thesupported_*_modalitieskeys that were in the payload are discarded and recomputed from that default. The model contradicts its own wire form:An image- or audio-capable target reads back as text-only; for
gpt-4oall six input modality combinations collapse to['text']. This is a live round trip:pyrit/cli/api_client.py:262doesTargetInstance.model_validate(item)on every/api/targetspayload.Parameteralready solves this exact problem in the same layer —pyrit/models/parameter.py:143,_reconstruct_param_type_from_wire— citing the same consumer: "a client that deserializes the wire form (e.g. the CLI consuming the REST catalog) has those fields but no live type; this reconstructs a coercion-capableparam_typefrom them so the round-trippedParametercan still coerce and validate values." This adds the analogous before-validator.One tradeoff worth naming: the wire carries the flattened union rather than the combinations, so a single combination holding that union is restored. That makes
supported_*_modalitiesround-trip exactly — the contract the wire documents — and leaves the object self-consistent. Recovering the original combination structure would mean changing the wire format and the OpenAPI schema the CLI and UI consume; I judged the corrective fix the safer default, but say the word if you would rather have full fidelity. In-process construction is unaffected: the validator only fills the combination fields in when they are absent, so a supplied liveinput_modalitiesstill wins.Tests and Documentation
TestTargetCapabilitiesWireRoundTripintests/unit/prompt_target/target/test_target_capabilities.py(9 tests). Six fail onab1c6c81and pass here. The other three pin behaviour that must not change: the default text-only round trip, in-process construction winning over a supplied flattened projection, and non-mapping payloads left to pydantic.pyrit/models/target/target_capabilities.pyfrom that file: 100% (48/48 statements).21786 passed, 241 skipped, plus one pre-existing failure —tests/unit/backend/test_scenario_run_routes.py::TestResumeScenarioRunRoute::test_resume_has_no_get_preflight. It also fails onab1c6c81with no changes applied (checked by stashing this diff); it is test-order pollution withintests/unit/backend, unrelated.pre-commit run --files <both files>passes, includingruff format,ruff checkandty.