diff --git a/.github/workflows/contract-versions.yml b/.github/workflows/contract-versions.yml new file mode 100644 index 000000000..22e320879 --- /dev/null +++ b/.github/workflows/contract-versions.yml @@ -0,0 +1,19 @@ +name: Contract versions + +on: + pull_request: + merge_group: + +permissions: + contents: read + +jobs: + contract-versions: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-python@v5 + with: + python-version: '3.12' + - name: Validate contract versions and shipped examples on the merge result + run: python tools/check_contract_versions.py diff --git a/.gitignore b/.gitignore index c1667a2d7..b5c6f865a 100644 --- a/.gitignore +++ b/.gitignore @@ -181,3 +181,6 @@ cython_debug/ #.idea/ .aider* + +# Documentation is assembled into wheels by the build command. +src/sequifier/consolidated-docs.md diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index 9c49d2de6..fe7ebc04c 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -57,12 +57,3 @@ repos: files: ^(pyproject\.toml|.github/workflows/docs.yml)$ pass_filenames: false additional_dependencies: [tomli] - - - repo: local - hooks: - - id: consolidate-docs - name: Consolidate configuration documentation - entry: python hooks/consolidate_docs.py - language: python - files: ^(README\.md|documentation/configs/(preprocess|train|infer|hyperparameter-search)\.md|documentation/commands/visualize-training\.md|documentation/training/multi-gpu-training\.md|src/sequifier/consolidated-docs\.md)$ - pass_filenames: false diff --git a/MANIFEST.in b/MANIFEST.in new file mode 100644 index 000000000..bc7ac30e5 --- /dev/null +++ b/MANIFEST.in @@ -0,0 +1 @@ +recursive-include documentation *.md diff --git a/README.md b/README.md index 82e876b9c..d4b1a018b 100644 --- a/README.md +++ b/README.md @@ -104,8 +104,8 @@ There are seven standalone commands within sequifier: `make`, `preprocess`, `tra | --- | --- | | `make` | Create a new sequifier project with config templates. | | `preprocess` | Convert input data into fixed-length subsequences. | -| `train` | Train a model on preprocessed data. | -| `infer` | Generate predictions, probabilities, or embeddings. | +| `train` | Train supervised models or ordered imitation, IQL, PPO, DPO and GRPO policy phases. | +| `infer` | Generate predictions, probabilities, embeddings, or policy actions and rollouts. | | `hyperparameter-search` | Use Optuna to find optimal configurations across multiple training runs. | | `visualize-training` | Generate interactive HTML plots from structured training metrics. | | `docs` | Print the consolidated documentation, or the guide for `train`, `preprocess`, or `infer`. | @@ -114,8 +114,10 @@ Run `sequifier docs` for all documentation, or `sequifier docs train`, `sequifie The processing and visualization commands have dedicated documentation pages: +- [canonical configuration names](./documentation/configs/naming.md) - [preprocess documentation](./documentation/configs/preprocess.md) - [train documentation](./documentation/configs/train.md) +- [policy training](./documentation/policy-learning.md), including BC, IQL, DPO, GRPO and [online categorical PPO](./documentation/policy-learning.md#online-categorical-ppo) - [infer documentation](./documentation/configs/infer.md) - [hyperparameter-search documentation](./documentation/configs/hyperparameter-search.md) - [visualize-training documentation](./documentation/commands/visualize-training.md) @@ -168,7 +170,7 @@ The basic input data format is this: The two columns "sequenceId" and "itemPosition" have to be present, and there must be one or more feature columns. -`sequifier preprocess` splits sequences into subsequences, normalises real variables and maps categorical variables to integers/tokens. Each stored subsequence contains `window_length` positions; `max_target_offset` reserves positions within that window for future targets. +`sequifier preprocess` splits sequences into subsequences, normalises real variables and maps categorical variables to integers/tokens. Each stored subsequence contains the number of positions selected by its split in `window_lengths`; `max_target_offset` reserves positions within that window for future targets. | sequenceId | subsequenceId | startItemPosition | leftPadLength | inputCol | [window_length - 1] | [window_length - 2] | ... | 0 | | --- | --- | --- | --- | --- | --- | --- | --- | --- | @@ -214,7 +216,7 @@ Once you have your data in the input format described above, you can train a tra sequifier preprocess ``` -5. **Reference the generated metadata.** Preprocessing outputs metadata at `configs/metadata_configs/[INPUT BASENAME].json`. For a single dataset and part, reference that file from `dataset.part.metadata_config_path` in `train.yaml`; named configurations use `dataset_training..parts..metadata_config_path`. Inference may still use `preprocessing_data_path` or `metadata_config_path`. +5. **Reference the generated metadata.** Preprocessing outputs metadata at `configs/metadata_configs/[INPUT BASENAME].json`. Reference that file from `dataset_training..parts..metadata_config_path` in `train.yaml`. Inference uses `metadata_config_path` or the selected artifact metadata. 6. **Configure and train the model.** Adapt `train.yaml` with your transformer hyperparameters, then run: @@ -222,9 +224,9 @@ Once you have your data in the input format described above, you can train a tra sequifier train ``` -7. **Configure inference.** Point `model_path` in `infer.yaml` at the default ONNX export. Keep the scaffold's explicit contract, or replace it with `training_config_path` and `dataset`; see the [ONNX/PT trade-offs](./documentation/configs/infer.md#onnx-or-pt). +7. **Configure inference.** Point `model_paths` in `infer.yaml` at the default ONNX export. Keep the scaffold's explicit contract, or replace it with `training_config_path` and `dataset`; see the [ONNX/PT trade-offs](./documentation/configs/infer.md#onnx-or-pt). - **GPU ONNX inference:** `device: cuda` requires `onnxruntime-gpu` built for + **GPU ONNX inference:** `run.execution.device: cuda` requires `onnxruntime-gpu` built for your CUDA and cuDNN versions. The CPU-only `onnxruntime` package does not provide `CUDAExecutionProvider`. @@ -254,7 +256,7 @@ current loss calculation uses all valid decoded positions, regardless of ### Autoregressive Inference -Autoregressive inference is allowed when the model is causal, all input variables are target variables, `target_offset` equals 1 and `prediction_length` equals 1. It is enabled by setting `autoregressive: true`, and `generation_steps` to the desired integer value. The number of generation_steps will be generated starting from the *first* complete subsequence in a sequence. +Autoregressive inference is allowed when the model is causal, all input variables are target variables, `target_offset` equals 1 and `prediction_length` equals 1. It is enabled by setting `autoregressive_generation: true`, and `generation_steps` to the desired integer value. The number of generation_steps will be generated starting from the *first* complete subsequence in a sequence. It iteratively predicts future values, by returning predictions at step t-1 as input for generating a prediction at t. Predictions for categorical target variables can be made using argmax or sampling. @@ -279,13 +281,13 @@ Next-occurrence causal modelling is a generalisation of final-value causal model ### Causal Embedding Model -sequifier also supports the export of causal embeddings, instead of predictions. It requires the following settings: `export_embedding_model: true` in the training config and `model_type: embedding` in the inference config. Selected activations are restricted to the configured final `prediction_length` positions and concatenated in configuration order along the feature dimension. +sequifier also supports the export of causal embeddings, instead of predictions. It requires the following settings: `export_embedding_model: true` in the training config and `task: embedding` in the inference config. Selected activations are restricted to the configured final `prediction_length` positions and concatenated in configuration order along the feature dimension. If you are interested in activations *other* than the last backbone layer, you can configure the exact layers you want to contribute to the export using `embedding_layer_names`. You can pass an ordered list, such as `[backbone.layers.1, decoder.branches.default.hidden_blocks.0]`, and the activations of these layers will be concatenated and output. Layer names follow the network hierarchy using zero-based indices: `backbone.layers.` selects a transformer block output, `backbone.final_norm` the normalized backbone output, and `decoder.branches..hidden_blocks.` an MLP decoder hidden-block output; the same scheme applies to BERT embedding models. -Backbone selectors contribute `dim_model` activations. Decoder MLP hidden-block selectors contribute their configured hidden width and receive the same flattened `decoding_support * dim_model` windows used during training. The default, `embedding_layer_names: [backbone.final_norm]`, preserves the final normalised backbone representation. +Backbone selectors contribute `model_dim` activations. Decoder MLP hidden-block selectors contribute their configured hidden width and receive the same flattened `support_length * model_dim` windows used during training. The default, `embedding_layer_names: [backbone.final_norm]`, preserves the final normalised backbone representation. ### BERT Model @@ -295,7 +297,7 @@ Configuration: - Preprocessing: Set `max_target_offset: 0` for equal-width input and target windows. - Training: Set `training_objective: bert`, configure `bert_spec`, and set decoder `prediction_length` equal to `context_length`. Enable generative and/or embedding export according to the desired inference. -- Inference: Set `model_type: generative` to reconstruct explicitly masked input, or `model_type: embedding` to output contextual representations. +- Inference: Set `task: forecast` to reconstruct explicitly masked input, or `task: embedding` to output contextual representations. Technical Details: BERT-style models use bidirectional attention and learn by reconstructing positions sampled according to `bert_spec`. Inference does not apply random masking; inputs that should be reconstructed must be masked explicitly, for example using `mask_column` during preprocessing. Embedding inference returns one contextual representation for every valid position in the input window. @@ -336,11 +338,11 @@ Separately, `temporal_conv` enables temporal convolutions on pass-through or emb `temporal_conv` uses causal, left-padded convolutions by default (`causal: true`). For causal models, keep the default `orientation: within_item_position` when -using `post_conv_norm: layer_norm` or `rmsnorm`. With +using `post_conv_normalization: layer_norm` or `rmsnorm`. With `orientation: within_column`, these norms operate across the full time axis, so earlier positions can depend on later ones despite the causal convolution and the transformer's attention mask. If `within_column` is needed in a causal -model, set `post_conv_norm: none`. +model, set `post_conv_normalization: none`. #### Categorical multi-hash embeddings and targets @@ -352,8 +354,10 @@ logical column name: model: interfaces: default: - input_columns: [product_id] - target_columns: [product_id] + input_columns: + - product_id + target_columns: + - product_id categorical_hashing: product_id: type: multi_hash @@ -421,3 +425,5 @@ Please cite with: } ``` + +Policy runs support ordered imitation, IQL, DPO, GRPO and PPO phases; see the [policy learning guide](documentation/policy-learning.md). diff --git a/docs/source/guides/artifacts.rst b/docs/source/guides/artifacts.rst new file mode 100644 index 000000000..5a759ab30 --- /dev/null +++ b/docs/source/guides/artifacts.rst @@ -0,0 +1,2 @@ +.. include:: ../../../documentation/training/artifacts.md + :parser: myst_parser.sphinx_ diff --git a/docs/source/guides/naming.rst b/docs/source/guides/naming.rst new file mode 100644 index 000000000..c546aea7e --- /dev/null +++ b/docs/source/guides/naming.rst @@ -0,0 +1,2 @@ +.. include:: ../../../documentation/configs/naming.md + :parser: myst_parser.sphinx_ diff --git a/docs/source/guides/ownership.rst b/docs/source/guides/ownership.rst new file mode 100644 index 000000000..d1db5c5ee --- /dev/null +++ b/docs/source/guides/ownership.rst @@ -0,0 +1,2 @@ +.. include:: ../../../documentation/configs/ownership.md + :parser: myst_parser.sphinx_ diff --git a/docs/source/index.rst b/docs/source/index.rst index 92ccac06e..dbd038144 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -9,9 +9,12 @@ Welcome to Sequifier's documentation! Training Inference Hyperparameter search + Configuration naming + Configuration ownership Visualizing training Distributed and multi-node training Training runtime architecture + Training artifacts .. toctree:: :maxdepth: 1 diff --git a/documentation/configs/hyperparameter-search.md b/documentation/configs/hyperparameter-search.md index 70e85d999..85f00b1d4 100644 --- a/documentation/configs/hyperparameter-search.md +++ b/documentation/configs/hyperparameter-search.md @@ -3,7 +3,10 @@ `sequifier hyperparameter-search` searches over complete canonical training configurations with Optuna. It supports Bayesian optimization, random sampling, finite grid search, cooperative pruning, and custom single- or multi-objective -evaluation. +evaluation. Supervised plans are supported; policy and mixed plans are not yet +supported. The final phase monitor supplies the default objective; configure the +matching `min`/`max` search direction. See +[Training artifacts](../training/artifacts.md). ```console sequifier hyperparameter-search --config-path configs/hyperparameter-search.yaml @@ -17,34 +20,53 @@ search spaces using the same paths as the training schema. ```yaml base_config_path: train.yaml -name: transformer-width-search -model_config_write_path: configs/hp-search +run: + study_name: transformer-width-search + outputs: + trial_config_directory: configs/hp-search method: bayesian trials: 40 - parameters: - global_training: - context_length: [64, 128] - batch_size: [16, 32] - learning_rate: - low: 0.0001 - high: 0.001 - log: true model: + execution: + context_length: + "$choices": + - 64 + - 128 backbone: architecture: - num_layers: {low: 4, high: 8, step: 2} + num_layers: + "$range": + low: 4 + high: 8 + step: 2 training_plan: - epochs: [2, 4] + phases: + '0': + epochs: + "$choices": + - 2 + - 4 + batch_size: + "$choices": + - 16 + - 32 + optimization: + model: + learning_rate: + "$range": + low: 0.0001 + high: 0.001 + log: true ``` `base_config_path` resolves relative to the hyperparameter-search entry file. The base may itself be a composed training config. If the search config supplies -`project_root`, that value is used in every generated trial; otherwise the base +`run.project_root`, that value is used in every generated trial; otherwise the base training config's root is inherited. -For a singleton base, use `model.interface`, `dataset.part`, and -`training_plan.epochs`. Named paths remain available for multi-value bases. +Use the same named paths as the training base: `model.interfaces.`, +`dataset_training..parts.`, and indexed `training_plan.phases`. The historical self-contained search schema and historical flat training base configs are not accepted. Every generated trial is validated as an authored @@ -53,8 +75,8 @@ references, incompatible component types, and cross-field violations fail with their canonical validation paths. `model_name` and `project_root` cannot appear in `parameters`. Generated model -names use `[name]-run-[index]`, and `project_root` is controlled by the -top-level search field. +names use `[study_name]-run-[index]`, and `run.project_root` is controlled by the +search run field. ## Parameter expressions @@ -63,44 +85,56 @@ have these forms: | Form | Meaning | | --- | --- | -| Scalar | Fixed replacement. | -| List on a scalar field | Categorical choices. | -| `{low, high, step?, log?, type?}` | Integer or float distribution. `type` may be `int` or `float`; otherwise the base value and bounds determine it. | -| `{choices: [...]}` or `{$choices: [...]}` | Categorical choices for an entire value, including mappings and lists. | -| `{fixed: value}` or `{$fixed: value}` | Unambiguous fixed replacement for a mapping or list. | -| Numeric keys under a base list | Recursive replacement of zero-based list entries. | -| `{variants: [...]}` or `{$variants: [...]}` | Paired partial mapping variants, optionally followed by independently sampled sibling fields. | +| Primitive scalar | Fixed replacement. | +| Ordinary mapping | Recursive patch. | +| `{$range: {low, high, step?, log?, type?}}` | Numeric distribution; `type` is `int` or `float`. | +| `{$choices: [...]}` | Categorical selection of any scalar, list, or mapping value. | +| `{$fixed: value}` | Complete mapping or list replacement. | +| Integer keys under a base list | Recursive replacement of zero-based list entries. | +| `{$variants: [...]}` | Partial mapping variants, optionally with independently sampled sibling fields. | + +Bare lists, unprefixed operators, and bare numeric distribution mappings are +rejected. `$` is reserved for search operators; their meaning does not depend on +the base value's type. Integer ranges default to `step: 1`. Grid search requires a `step` for float ranges because an unstepped float interval is infinite. Logarithmic integer ranges require `step: 1`; logarithmic float ranges cannot use `step`. -A direct list of lists samples a complete list-valued field. Prefer the explicit -`choices` wrapper for mapping or list candidates: +Use `$choices` to sample complete scalar, mapping, or list values: ```yaml parameters: model: backbone: architecture: - choices: - - dim_model: 128 - max_context_length: 512 - num_layers: 4 - attention: {type: mha, n_heads: 8} - feed_forward: {dim: 512, activation: swiglu} - - dim_model: 256 - max_context_length: 512 - num_layers: 6 - attention: {type: gqa, n_heads: 16, n_kv_heads: 4} - feed_forward: {dim: 1024, activation: swiglu} + "$choices": + - model_dim: 128 + max_context_length: 512 + num_layers: 4 + attention: + type: mha + num_heads: 8 + feed_forward: + activation: swiglu + hidden_dim: 512 + - model_dim: 256 + max_context_length: 512 + num_layers: 6 + attention: + type: gqa + num_heads: 16 + num_kv_heads: 4 + feed_forward: + activation: swiglu + hidden_dim: 1024 ``` Each choice replaces the complete overridden subtree, keeping coupled values valid. The same mechanism can select complete ingestion or decoder components, training phases, or other canonical subtrees. -Use `variants` when candidates should patch the base mapping and other sibling +Use `$variants` when candidates should patch the base mapping and other sibling fields should remain independent. A variant that changes a component's `type` starts from that new component shape rather than retaining fields belonging to the old type. @@ -111,10 +145,15 @@ parameters: interfaces: event_prediction: ingestion: - variants: - - {type: embedding, output_dim: 128} - - {type: passthrough, output_dim: 128} - dropout: [0.0, 0.1] + "$variants": + - type: embedding + output_dim: 128 + - type: passthrough + output_dim: 128 + dropout: + "$choices": + - 0.0 + - 0.1 ``` ## Search controls @@ -123,20 +162,20 @@ parameters: | --- | --- | --- | --- | --- | | `base_config_path` | `str` | Yes | - | Canonical training config used as the base. | | `parameters` | `mapping` | Yes | - | Recursive fixed values and search spaces. May be empty. | -| `project_root` | `str` | No | Base value | Root written to every generated training config. | -| `additional_config_paths` | `str`, `list[str]`, or `null` | No | `null` | Complementary direct fragments of the search config. Relative paths resolve against the entry config's `project_root`; recursive composition and duplicate fields are rejected. | -| `name` | `str` | Yes | - | Study name and generated-run prefix. | -| `model_config_write_path` | `str` | Yes | - | Directory, under `project_root`, for generated trial configs. | -| `method` | `bayesian`, `sample`, or `grid` | No | `bayesian` | Optuna TPE, random, or exhaustive finite-grid sampling. | +| `run.project_root` | `str` | No | Base value | Root used by the search and written to every generated training config. | +| `additional_config_paths` | `list[str]` or `null` | No | `null` | Complementary direct fragments of the search config. Relative paths resolve against the entry config's `run.project_root` (or `.` if omitted); recursive composition and duplicate fields are rejected. | +| `run.study_name` | `str` | Yes | - | Study name and generated-run prefix. | +| `run.outputs.trial_config_directory` | `str` | Yes | - | Directory, under `run.project_root`, for generated trial configs. | +| `method` | `bayesian`, `random`, or `grid` | No | `bayesian` | Optuna TPE, random, or exhaustive finite-grid sampling. | | `trials` | `int` | Except grid | - | Target total number of completed or pruned runs in the persisted study. | -| `global_seed` | `int` or `null` | No | `null` | Sampler seed. Training seeds belong in canonical `parameters`. | +| `sampler_seed` | `int` or `null` | No | `null` | Sampler seed. Training seeds belong in canonical `parameters`. | | `prune_trials` | `bool` | No | `true` | Enables cooperative pruning. Distributed pruning remains experimental. | | `pruning_warmup_epochs` | `int` or `null` | No | `null` | Complete epochs before pruning may begin. Mutually exclusive with batch warmup. | | `pruning_warmup_batches` | `int` or `null` | No | `null` | Training batches before pruning may begin. Mutually exclusive with epoch warmup. | | `evaluation_metrics` | `list[str]` or `null` | No | `null` | Metric names expected from a custom evaluation script. | -| `evaluation_metric_directions` | `list[minimize\|maximize]` or `null` | With metrics | `null` | One optimization direction per metric. | -| `evaluation_script` | `str` or `null` | With metrics | `null` | Script invoked with the best exported model's evaluation ID. | -| `evaluation_inference_config` | `str` or `null` | No | `null` | Inference config run before the custom evaluation script. | +| `evaluation_metric_directions` | `list[min\|max]` or `null` | With metrics | `null` | One optimization direction per metric. | +| `evaluation_script_path` | `str` or `null` | With metrics | `null` | Script invoked with the best exported model's evaluation ID. | +| `evaluation_inference_config_path` | `str` or `null` | No | `null` | Inference config run before the custom evaluation script. | For grid search, omitting `trials` runs the complete finite grid. If it is provided, it must exactly equal the grid size. For Bayesian and random search, @@ -152,10 +191,14 @@ metric it performs single-objective optimization; with several it records the Pareto front. ```yaml -evaluation_metrics: [accuracy, latency_ms] -evaluation_metric_directions: [maximize, minimize] -evaluation_inference_config: configs/infer-validation.yaml -evaluation_script: scripts/evaluate.py +evaluation_metrics: +- accuracy +- latency_ms +evaluation_metric_directions: +- max +- min +evaluation_inference_config_path: configs/infer-validation.yaml +evaluation_script_path: scripts/evaluate.py ``` The evaluation script receives the exported model's evaluation ID, formatted @@ -173,6 +216,6 @@ loading metadata-derived values, so all required authored fields must already be present. Generated canonical training configs are written below -`model_config_write_path`. The Optuna SQLite study is persisted at -`state/optuna/[name].db` under `project_root`, allowing later +`run.outputs.trial_config_directory`. The Optuna SQLite study is persisted at +`state/optuna/[study_name].db` under `run.project_root`, allowing later invocations to continue toward the configured total. diff --git a/documentation/configs/infer.md b/documentation/configs/infer.md index 408c336fb..4ab1e8c33 100644 --- a/documentation/configs/infer.md +++ b/documentation/configs/infer.md @@ -1,16 +1,19 @@ # Infer Command Guide +For CPU numeric inference with externally calculated next-step inputs, see +[local forecast rollouts](../rollout.md). An explicit `rollout` configuration +selects a single seed window and saves a versioned trajectory through the +existing inference command. + Models with `static_conditioning` read the processed static feature table named in the preprocessing metadata. Each inference window receives the configured static values for its `sequenceId`. Autoregressive generation retains these values while temporal inputs slide forward. -Models with image ingestion need `image_path` for the original image or video -folders selected by the model. Use a string for one temporal view or a map from -view name to folder for multiple views. When the model has a preconditioning -image encoder, also set `preconditioning_image_path` to a string for the -`default` view or a map for named views. Preprocessing metadata records view +Models with image ingestion need `image_paths` for the original image or video +folders selected by the model. Use a mapping from each temporal view name to its folder, even for one view. When the model has a preconditioning +image encoder, also set `preconditioning_image_paths` to a view-to-folder mapping, including a `default` key for an unnamed view. Preprocessing metadata records view contracts and signatures, but omits folder paths. The folders must remain available and unchanged after preprocessing. The inference loader applies the same frame-to-`itemPosition` or time-column alignment and image transform used @@ -19,8 +22,10 @@ PT and ONNX model execution accept image tensors and frame-valid masks; media decoding stays outside the exported model. ```yaml -image_path: {front: data/front_images, side: data/side_videos} -preconditioning_image_path: +image_paths: + front: data/front_images + side: data/side_videos +preconditioning_image_paths: front: data/sequence_front_images side: data/sequence_side_images ``` @@ -34,24 +39,28 @@ sequifier infer --config-path configs/infer.yaml ## Start here: ONNX -> **GPU ONNX inference requires `onnxruntime-gpu`.** If you set `device: cuda`, +> **GPU ONNX inference requires `onnxruntime-gpu`.** If you set `run.execution.device: cuda`, > install a build compatible with your CUDA and cuDNN versions. The CPU-only > `onnxruntime` package does not provide `CUDAExecutionProvider`; setting -> `device: cuda` alone does not enable GPU inference. See the +> `run.execution.device: cuda` alone does not enable GPU inference. See the > [ONNX Runtime CUDA compatibility guide](https://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html). -ONNX is the default training export and the deployment-oriented inference path. +PT is the default training export. Supervised runs may explicitly request +ONNX for deployment-oriented inference. Exports embed their execution contract. Select a training route when providing model configuration through a training config: ```yaml -project_root: . -model_path: models/event-model-best-5.onnx training_config_path: configs/train.yaml dataset: events data_path: data/events-test.parquet -model_type: generative -device: cuda +task: forecast +model_paths: +- models/event-model-best-5.onnx +run: + project_root: . + execution: + device: cuda ``` The route supplies columns, types, objective, window sizes, and preprocessing @@ -59,9 +68,14 @@ metadata. Add `part` when the dataset has several parts. Current ONNX exports also embed this contract and metadata, so the route is optional. Older exports need a training route or the full contract and metadata supplied explicitly. +All routes accept `output_path` for one selected model. It overrides the first +prediction or embedding file; later chunks append `-part-NNN` to its stem. +Prediction identity and probability-file layouts are described in +[Training artifacts](../training/artifacts.md#inference-and-collection-outputs). + ## ONNX or PT? -| | ONNX (default) | PT | +| | ONNX | PT (default) | | --- | --- | --- | | Best fit | Portable, deployment-oriented inference. | Python/PyTorch workflows and easier configuration. | | Runtime | ONNX Runtime on CPU or CUDA (with a CUDA-enabled ONNX Runtime installation). | PyTorch on CPU, CUDA, or MPS. | @@ -72,16 +86,18 @@ Benchmark the target workload rather than assuming either runtime is faster. Choose PT when portability matters less than a compact, self-contained config: ```yaml -project_root: . -model_path: models/event-model-best-5.pt data_path: data/events-test.parquet -model_type: generative -device: cuda +task: forecast +model_paths: +- models/event-model-best-5.pt +run: + project_root: . + execution: + device: cuda ``` -For multi-interface PT, add `model_interface`. `model_type` stays explicit for -both formats because inference may generate outputs or extract embeddings. -`project_root` is required (normally `.`), and `inference_batch_size` defaults +For multi-interface PT, add `model_interface`. `task` is required for both formats: `forecast`, `policy`, or `embedding`. +`run.project_root` defaults to `.`, and `run.execution.batch_size` defaults to `1`. ## Effective configuration and validation @@ -110,21 +126,20 @@ fragments or define the same field twice. | Field | Default | Purpose | | --- | --- | --- | -| `model_path` | Required | Model path, or a list of compatible model paths. | -| `model_type` | Required | `generative` or `embedding`. | -| `device` | Required | ONNX: `cpu` or `cuda`; PT also supports `mps`. ONNX CUDA requires an installed runtime exposing `CUDAExecutionProvider`. | -| `project_root` | Required | Base for project paths; normally `.`. | +| `model_paths` | Required | Non-empty list of compatible model paths; policy requires exactly one. | +| `task` | Required | `forecast`, `policy`, or `embedding`. | +| `run.execution.device` | `cpu` | ONNX: `cpu` or `cuda`; PT also supports `mps`. ONNX CUDA requires an installed runtime exposing `CUDAExecutionProvider`. | +| `run.project_root` | Required | Base for project paths; normally `.`. | | `data_path` | Metadata test/last split | Input file or folder. Usually required with artifact-only inference because artifacts do not store split paths. | -| `preprocessing_data_path` | `null` | Derives the generated metadata path. | | `metadata_config_path` | `null` | Explicit preprocessing metadata. | -| `image_path` | `null` | Original folder for one temporal view, or a view-to-folder map for every selected temporal view. Required when the model consumes temporal images. | -| `preconditioning_image_path` | `null` | Original folder for the `default` preconditioning view, or a view-to-folder map for named views. Required when the model uses preconditioning images. | +| `image_paths` | `null` | View-to-folder mapping for every selected temporal view. Required when the model consumes temporal images. | +| `preconditioning_image_paths` | `null` | View-to-folder mapping for all preconditioning views. Required when the model uses preconditioning images. | | `training_config_path` | `null` | Training config used to resolve a route. | | `dataset` / `part` | `null` | Select a dataset and optional part from the training config. | | `model_interface` | Implicit when unique | Select a route from a training config or portable artifact. | | `read_format` | `parquet` | `csv`, `parquet`, or folder-based `pt`. | | `write_format` | `csv` | `csv` or `parquet`. | -| `inference_batch_size` | `1` | Sequences processed per batch. | +| `run.execution.batch_size` | `1` | Sequences processed per batch. | ### Model contract @@ -145,7 +160,7 @@ dataset metadata. For aligned splits, it sets the number of consecutive prediction groups evaluated per stored window. Aligned inference requires `window_stride: null` and a stored width of at least `context_length + target_offset + (prediction_alignment_steps - 1) × prediction_length`. -With `model_type: embedding`, aligned splits label each embedding by its target +With `task: embedding`, aligned splits label each embedding by its target `itemPosition` and omit positions outside the split. For causal models, the embedding represents the context used to predict that row. Other splits label embeddings by the input activation position. @@ -163,9 +178,9 @@ stored folder window if the remaining layout and semantics match. | `decode_categories` | `true` | Decode categorical predictions to their original values. | | `sample_from_distribution_columns` | `null` | Sample these categorical targets instead of using argmax. | | `infer_with_dropout` | `false` | Enable dropout at inference; ONNX also requires dropout-preserving export. | -| `deterministic` | `false` | Request deterministic PyTorch algorithms. | -| `seed` | `1010` | Random seed. | -| `autoregressive` | `false` | Feed predictions back for multi-step generation. | +| `run.reproducibility.enforce_determinism` | `false` | Request deterministic PyTorch algorithms. | +| `run.reproducibility.seed` | `1010` | Random seed. | +| `autoregressive_generation` | `false` | Feed predictions back for multi-step generation. | | `generation_steps` | `null` | Required positive step count when autoregressive is enabled. | For an `autoregressive_transformer` decoder branch, same-position targets are @@ -184,11 +199,14 @@ its stored windows rather than this first-window reduction. ## CLI overrides -`--data-path`, `--input-columns`, `--metadata-config-path`, `--model-path`, -`--seed`, `--dataset`, `--part`, and `--model-interface` override YAML values. -`--randomize` takes precedence over `--seed`. `--skip-metadata` skips external +`--data-path`, `--input-columns`, `--metadata-config-path`, `--model-paths`, +`--seed`, `--dataset`, `--part`, `--model-interface`, `--task`, +`--enforce-determinism`, and `--action-selection` override YAML values. +`--seed random` resolves a fresh integer seed once; the resolved configuration records it. +`--enforce-determinism true|false` explicitly overrides the YAML boolean. `--skip-metadata` skips external metadata loading; inline metadata, a selected training route, or a self-describing -PT artifact must then provide the required metadata. +PT artifact must then provide the required metadata. `--skip-metadata` is +forecast-only and is rejected on policy inference. ## Outputs @@ -216,18 +234,22 @@ inference does not configure it again. Inference supplies the matching named layout tensors and masks through the preprocessed PT input. ```yaml -project_root: . -model_path: models/item-model-best-5.onnx -model_type: generative read_format: pt data_path: data/raw-items-split2 -device: cpu infer_with_dropout: false -seed: 1010 +task: forecast +model_paths: +- models/item-model-best-5.onnx +run: + project_root: . + execution: + device: cpu + reproducibility: + seed: 1010 ``` -Use the corresponding embedding artifact with `model_type: embedding`, or change -`model_path` to a portable PT model. `read_format: pt` describes preprocessed +Use the corresponding embedding artifact with `task: embedding`, or change +`model_paths` to a portable PT model. `read_format: pt` describes preprocessed input; it is independent of the model artifact extension. Deep inputs retain `[B,T,D]` and explicit boolean depth masks through batching. New graphs accept symbolic batch size with fixed temporal/depth capacities, including a partial @@ -258,3 +280,51 @@ incorrect ranks/capacities, forbidden gaps, illegal categories, and non-finite values. Legal masked values are sanitized before narrowing and inside the graph. The declared provider must be available; CUDA inference requires a CUDA-enabled ORT installation. Outputs stay indexed by outer item coordinates. + + +## Policy model interfaces + +`task` explicitly selects the operation. `task: policy` requires one +policy-bound portable PT artifact. `task: forecast` requests ordinary predictions, +including from a policy-bound interface with a supported forecast contract. +`task: embedding` extracts representations. Select `model_interface` when ambiguous. +The artifact validates task support; it never chooses the operation implicitly. +The CLI uses `--task forecast|policy|embedding`. + +Policy input is raw CSV/Parquet observation rows. History alignment, codecs and +normalization come from the artifact. Common fields cover model/interface/data +selection, read/write formats, device, seed, `enforce_determinism` and batch size. + +| Policy field | Default / meaning | +| --- | --- | +| `output_path` | null; override the generated prediction destination | +| `action_selection` | `deterministic`; sequential mode/location choices, or `stochastic` sampling | +| `desired_return` | null; fixed budget for return-conditioned policies | +| `desired_return_column` | `desired_return`; per-row budget when no fixed budget is supplied | +| `previous_action_columns` | {}; recorded lagged-action column mappings for fragments | +| `sequence_id_column`, `item_position_column` | `sequenceId`, `itemPosition`; raw history ordering | +| `output_probabilities` | false; include joint log probability and write separate target-specific categorical probability files | +| `rollout` | null; execute in an online policy environment instead of writing prediction rows | + +Optional assertions `metadata_config_path`, `input_columns`, `static_columns`, +`target_columns`, `column_data_types`, `target_column_types`, `training_objective`, +`context_length`, `target_offset`, `prediction_length` and `decoder_support_length` must +match the selected artifact. Unknown/cross-route options fail even at default values. +Policy rejects external per-column sampling, dropout, category decoding, +dataset/part selection, image overrides, autoregression and `generation_steps`. +Forecast rejects policy-only output/return/action/history options; embeddings also +reject generative sampling, decoding, probabilities, autoregression and rollout. + +Policy rollout declares `environment: {factory: module:factory, config: {}, +units: {}}`, required `max_steps`, `seed_row_index` and +`trajectory_path`, plus optional `max_episode_steps`. It rejects forecast `bindings`. +Forecast rollout uses the forecast rollout adapter protocol, requires bindings and +selects `seed_window_index`. Neither accepts `rollout.type` or `seed_index`. +Both indices are zero-based and validated before environment creation. Trajectory +metadata uses the same route-specific index name. See the [rollout guide](../rollout.md) and +[policy template](../../src/sequifier/templates/infer-policy.yaml). + +Execution determinism, action selection and seed are independent. A seeded +stochastic policy can reproduce its actions under the same supported runtime, +device and input ordering. Online evaluation uses the same action-selection +meaning and the training run's execution-determinism setting. diff --git a/documentation/configs/naming.md b/documentation/configs/naming.md new file mode 100644 index 000000000..51dea8ac9 --- /dev/null +++ b/documentation/configs/naming.md @@ -0,0 +1,192 @@ +# Configuration naming reference + +Sequifier accepts one canonical spelling and location for each authored setting. +See [configuration ownership](ownership.md) for the supported locations. + +## Named structures and fixed shapes + +Use `model.interfaces.`, `dataset_training..parts.`, +`policy_bindings.`, and `training_plan.phases` for every run size. +`default` is an ordinary name, not an implicit authoring syntax. Omitted unique +references can still be inferred. Direct `training_plan.epochs`, singular +`model.interface`, `dataset`, dataset `part`, and training `policy_binding` are +rejected. Sequence import still has one singular `policy_binding`. + +Supervised evaluation is an explicit `{sources, monitor?}` declaration or `null`; +boolean shorthand is rejected. Optimizers and schedulers use `{name, arguments}`. +Initialization methods use `{method, ...}`, including parameterless methods. + +`model_paths` is always a non-empty list. `image_paths`, +`preconditioning_image_paths`, and preprocessing `preconditioning_image.paths` +are view-to-path mappings even for a single view. `additional_config_paths` is +a list or `null`. Preprocessing `window_lengths` supplies one width per split. +Generated metadata retains its selected-split `window_length`. + +## Dimensions, counts, and selection + +| Setting | +| --- | +| `model_dim` | +| `feed_forward.hidden_dim` | +| `num_heads` / `num_kv_heads` | +| `total_embedding_dim` | +| `position_encoding_scope` | +| `post_conv_normalization` | +| `frame_pooling` / `image_frame_pooling` | +| `pooling` | +| `support_length` | +| `decoder_support_length` | +| `max_depth_length` | +| `max_concat_tensors` | +| `run.execution.data_loading.num_workers` | +| `selection_weight` | + +`model_dim`, `hidden_dim`, `output_dim`, `cell_dim`, `hidden_dims`, and +`feature_embedding_dims` describe different representation widths. +`total_embedding_dim` allocates aggregate static feature width. `context_length` +is temporal model input length; `max_context_length` is backbone capacity; +`max_depth_length` is child-collection capacity. `support_length` counts temporal +representations consumed by a decoder; `prediction_length` counts decoded positions. + +Source `selection_weight` changes ordering and burst frequency, not the total +amount consumed in an epoch. Class weights and loss weights retain their separate +loss-scaling meanings. + +## Training clocks and optimization + +A batch is one forward/backward microbatch. An optimizer step applies gradients +for a role. An iteration is a complete policy transaction: imitation, IQL, and +DPO perform one optimizer step per role; PPO and GRPO may perform several. +An epoch is one pass through the applicable data. Environment steps count +transitions; generation steps count temporal autoregressive output steps. + +| Setting | +| --- | +| `accumulation_batches` | +| `batch_size` | +| `iterations` | +| `groups_per_iteration` | +| `rollout_env_steps` | +| `evaluation.cadence.epochs` | +| `evaluation.cadence.iterations` | +| `checkpoints.cadence.epochs` | +| `checkpoints.cadence.iterations` | +| `log_interval_batches` | +| `stopping.patience` (evaluation observations) | +| `scheduler_step_on: optimizer_step` | +| `gradient_clip_norm` | +| `value_clip_delta` | +| `old_policy_target_kl` | +| `reference_kl_coefficient` | +| `max_advantage_weight` | + +Keep `epochs`, `optimization_epochs`, `total_env_steps`, `max_env_steps`, +`total_groups`, `group_size`, `max_steps`, `max_episode_steps`, and +`generation_steps`: their enclosing contracts identify different budgets. +`batch_size` always describes a microbatch; its elements are examples, +transitions, preference pairs, or trajectories depending on the algorithm. + +Keep optimizer roles `model`, `actor`, `value`, and `critic`. Third-party +constructor names and arguments remain unchanged under `arguments`. +`learning_rate` is Sequifier-owned; optimizer constructor `lr` is forbidden. +Scheduler `total_steps` remains an assertion against the resolved scheduler budget. + +## Freezing, checkpoints, and direction + +Freezing now belongs to `phase.trainability.roles.` for all algorithms; +source-dependent supervised masks use `phase.trainability.by_dataset`. +`freeze`/`freeze_except` contain catalog selector objects, shared with optimizer +routing. Explicit module selectors and semantic scopes share the same selection language. + +Recovery is `run.recovery`; retained snapshots are `phase.checkpoints`; +evaluation triggers are `phase.evaluation.before_phase` and `.on_snapshot`. +Handoff is `phase.model_selection.handoff`; source scheduling retains `selection`. +See [ownership](ownership.md) for the complete structure and stopping units. + +## Inference, sources, and seeds + +Inference requires `task: forecast`, `task: policy`, or `task: embedding`. +`model_type` is removed. The artifact validates the requested task; it never +chooses the task implicitly. Policy requires exactly one entry in `model_paths`. +`autoregressive` becomes `autoregressive_generation`. Inference's +`preprocessing_data_path` metadata shortcut is removed; use `metadata_config_path`. + +| Setting | +| --- | +| `model_paths` | +| `image_paths` | +| `preconditioning_image_paths` | +| `paths` | +| `model.interfaces..input_contract.metadata_config_path` | +| `manifest_path` | + +Interface input contracts select metadata files. Per-part and inference +`metadata_config_path` still select one file. Source-local `project_root` is +removed; the run supplies it during resolution. + +Run `run.reproducibility.seed` accepts an integer or `random`. Random seeds resolve once to an integer +recorded in the effective configuration. `sampler_seed` controls an independent +search stream. Initialization and hash seeds stay scoped integer declarations. +`enforce_determinism`, `action_selection`, and `seed` remain independent controls. + +Use `seed_window_index` for prepared windows and `seed_row_index` for table rows. +The model declares its lagged inputs in `previous_action_inputs`. Sequence tables +use fixed column names for [recorded actions and transitions](../policy-learning.md#offline-data). +Policy `scale` can use the model distribution or a fixed value. + +## Search expressions + +| Setting | +| --- | +| `sampler_seed` | +| `method: random` | +| `run.study_name` | +| `run.outputs.trial_config_directory` | +| `evaluation_script_path` | +| `evaluation_inference_config_path` | +| `$choices` | +| `$fixed` | +| `$variants` | +| `$range: {low, high, step?, log?, type?}` | + +Primitive scalars are fixed replacements; ordinary mappings recursively patch. +`$fixed` replaces a complete mapping or list. `$choices` samples any complete +value. `$variants` patches with selected partial mappings and may have sampled +siblings. Integer list keys patch indexed entries. Bare lists are rejected. +Search operators reserve `$` and never depend on the base field's type. + +`evaluation_metric_directions` uses `min` and `max`, matching monitor `direction`. +Training parameter paths are exactly the canonical training paths. + +## Cardinality, initialization, and dtype values + +| Setting | +| --- | +| `num_buckets` | +| `min_count` | +| `max_categories` | +| `column_data_types` | +| `xavier_uniform` | +| `xavier_normal` | +| `std` | + +Numeric storage dtypes use canonical `Float32`, `Int64`, etc.; lowercase aliases +are rejected. Compute dtypes remain PyTorch-style `float32`, `bfloat16`, etc. +Standard methods such as `zeros`, `ones`, and `constant` remain separate methods, +even when some parameterizations are mathematically equivalent. + +Probability types describe their unchanged mathematics: + +| Previous type | Canonical type | Definition | +| --- | --- | --- | +| `normal_discretized_floor` | `clipped_rounded_normal_plus_one` | `max(round(Normal(mean, std)), 0) + 1` | +| `log_normal_discretized_floor` | `rounded_log_normal_plus_one` | `round(exp(Normal(mean, std))) + 1` | +| `poisson_floor` | `poisson_plus_one` | `Poisson(rate) + 1` | + +## CLI + +CLI flags use the YAML name with hyphens: preprocessing +`--preprocessing-data-path`, inference `--model-paths`, and +`--task forecast|policy|embedding`. All short aliases and abbreviated flags are +removed. `--randomize` becomes `--seed random`. Boolean configuration overrides +take explicit `true` or `false`, including `--enforce-determinism false`. diff --git a/documentation/configs/ownership.md b/documentation/configs/ownership.md new file mode 100644 index 000000000..bf4bc21d3 --- /dev/null +++ b/documentation/configs/ownership.md @@ -0,0 +1,179 @@ +# Configuration ownership + +Every authored setting has one owner. Configuration files use the ownership +structure below directly. + +| Owner | Purpose | +| --- | --- | +| `run` | Identity, execution resources, reproducibility, recovery, logging, outputs and export. | +| `model.execution` | Durable objective/window/attention facts required by the artifact. | +| `model.interfaces..input_contract` | Metadata and frozen normalization assertions. | +| `dataset_training.` | Named interface reference, source parts and split policy. | +| `policy_bindings.` | Actions, task units, return interpretation and action behavior. | +| `training_plan.phases[]` | Learning, trainability, budgets, optimization, evaluation, stopping and checkpoint snapshots. | +| `training_plan` | Ordered phase transitions. | + +Preprocessing and inference share `run.project_root`, `run.execution` and +`run.reproducibility`. Their command-specific source/output and operation fields +remain on the command. Inference batching is `run.execution.batch_size`; +preprocessing workers are `run.execution.data_loading.num_workers`. Unsupported +execution controls fail validation. Component initialization and hash seeds stay +on their components; evaluation seeds define a benchmark and stay in evaluation. +Unsupported fields must be omitted even when their values match the defaults. +Inference does not accept `run.execution.data_loading`, `distribution`, +`precision`, `compilation`, or `max_concat_tensors`. Preprocessing accepts +only `run.execution.data_loading.num_workers` within execution, and only `seed` +within reproducibility. Sequence import accepts `run.project_root` but no +execution or reproducibility settings. + +## Run settings + +```yaml +run: + project_root: . + model_name: example + execution: + device: cpu + distribution: {strategy: local} + precision: {float32_matmul_precision: highest} + compilation: {torch_compile: none} + data_loading: {num_workers: 0, load_full_data_to_ram: true} + reproducibility: {seed: 1010, enforce_determinism: false} + recovery: + resume: {policy: never} + latest_checkpoint_interval_minutes: 10 + logging: {log_interval_batches: 10} + outputs: {model_path: models/example/model.pt} + export: {generative_model: true, onnx: true, pt: false, batch_size: 1} +``` + +Distribution is one coherent choice: `local`, `ddp` or `fsdp`. FSDP additionally +requires an explicit `cpu_offload`. Current policy execution is local, without +custom precision, compilation, concatenation limits, or worker/lazy-loader/RAM-budget settings. +Policy training logs use `run.logging.log_interval_iterations`; supervised +training logs use `run.logging.log_interval_batches`. Mixed plans may set both. +Policy-containing plans export the portable PT actor; graph, embedding, dropout and +batched exports are rejected. Runs default to PT export. Supervised runs can request graph exports explicitly. Defaults do not +request those capabilities. Random run seeds resolve once; derived phase streams +and resolved phase declarations are recorded in native recovery/provenance. + +For policy-containing plans, omit `run.execution.distribution`, `precision`, +`compilation`, and `max_concat_tensors`. Policy-only plans also omit +`run.logging.log_interval_batches`. +Data loading may configure `episode_cache_size`, but not `num_workers`, +`load_full_data_to_ram`, or `max_ram_gb`. Omit the fixed export controls +`onnx`, `pt`, `generative_model`, `embedding_model`, `with_dropout`, and +`batch_size`; the PT actor export is selected automatically. + +Native supervised `data_loading.max_ram_gb` budgets prepared-data loading per rank, +including workers, all cached loaders, chunk working space and prefetched batches. +Oversized plans fail before materialization; model/optimizer memory is additional. +`precision.layer_autocast` applies mixed precision to local/DDP training, +validation and warm-up as well as the FSDP policy. Matrix precision is applied +inside each training process. `enforce_determinism` uses errors rather than +warning-only deterministic algorithms. See [resource controls](../training/resource-controls.md) +for budget accounting, supported devices and the compiled option inventory. + +## Dataset and model contracts + +Prepared parts own `read_format`; sequence parts own `format`. Raw datasets own +`split.validation_fraction`. Combining raw sources into one automatic, leakage-safe +partition requires equal fractions; use explicit `evaluation.sources` when the +partitions differ. Prepared split metadata remains authoritative. Fitting uses +training partitions only. + +`model.execution.context_length` is the selected temporal view; +`model.backbone.architecture.max_context_length` is capacity. Frozen input contracts +and artifact assertions must agree. Phases cannot redefine immutable window, +attention or codec facts. `learning.bert_spec` and `learning.next_occurrence_config` +configure objective behavior for each supervised phase. Every next-occurrence +phase declares its trigger column and values; these define training and evaluation +targets and are not part of the portable execution contract. + +## Phase settings + +```yaml +training_plan: + phases: + - name: fit + algorithm: supervised + epochs: 10 + mode: sequential + sources: [{source: events}] + learning: + datasets: + events: + criterion: {event: CrossEntropyLoss} + loss_weights: {event: 1.0} + trainability: + roles: + model: + freeze: + - {component: backbone, semantic_groups: [attention.qkv]} + evaluation: + sources: [{source: events}] + monitor: {source: events, metric: loss, direction: min} + cadence: {epochs: 1} + batch_size: 64 + before_phase: false + on_snapshot: false + model_selection: {handoff: best} + stopping: {patience: 3, min_delta: 0.001, on_trigger: end_phase} + checkpoints: + cadence: {epochs: 2} + snapshot_interval_minutes: 30 +``` + +Declare phase learning losses for all training and evaluation datasets. Source +scheduling still uses supervised `selection: round_robin | weighted_random`; +`model_selection` is distinct. Curriculum is `curriculum: {column: name}` on a +supervised phase. Sampling stride is `learning.window_stride`. + +All algorithms share these owners. Supervised cadence uses `{epochs: N}`; +policy cadence uses `{iterations: N}`. A policy iteration is a complete algorithm +transaction, possibly containing multiple optimizer steps. Final progress always +evaluates when enabled and checkpoints. Supervised initial and snapshot evaluations +are diagnostic: they do not select best weights or consume stopping patience. +Policy initial and snapshot evaluations can select best weights without consuming +stopping patience. Patience counts +scheduled finite evaluation observations; `on_trigger` explicitly ends the phase +or the entire plan. Stopping state is persisted so resume cannot count twice. + +Recovery refreshes rolling resumable state at safe boundaries. Phase snapshots +are retained checkpoints. Best-weight handoff is distinct from latest recovery; +it does not restore older optimizer/environment state. Snapshot time intervals +request the next safe committed boundary. Policy batch snapshots are rejected; +use transaction cadence or time instead. + +## Parameter selection + +Freezing and optimizer routing use the same catalog selectors: `component`, +`semantic_groups` (glob patterns), `parameter_kind`, `ndim`, named `interface`, +`branch_path`, or a `module` subtree. Conditions in a selector are conjunctive; +selector lists form a union. `exclude_modules` and `exclude_semantic_groups` +express explicit subtractions, for nested component policies. +Unknown or empty selections fail preflight. Every `semantic_groups` pattern must +match at least one eligible parameter within the selector's structural scope; +every `exclude_semantic_groups` pattern must match the positively selected scope +before exclusions are applied. Patterns are case-sensitive and are never +corrected or ignored. A valid alternative does not hide an unmatched pattern. +Overlapping and repeated matching patterns remain valid union selections. + +Each role accepts either `freeze` or `freeze_except`. Empty `freeze` masks nothing; +empty `freeze_except` masks everything and fails if an update has no trainable +parameters. Structural constraints remain: fixed Gaussian scales and frozen image +encoders cannot be enabled by a phase mask. Gradient freezing leaves model mode +semantics intact. + +Supervised `trainability.by_dataset..roles.model` uses exactly the same +policy shape. A supplied role policy replaces the phase policy for that dataset; +absence inherits it. Only training datasets can have update overrides. Policy +algorithms reject source-dependent masks, and support their own declared role set. +Frozen gradients are removed before AMP handling and stepping, so momentum and +weight decay do not update them. Masks are recalculated at phase boundaries. + +## Recovery + +Run checkpoints contain the current resolved settings, phase state and input +identities. Recovery requires these to agree with the requested run. Portable +model artifacts contain only the facts needed to reconstruct model execution. diff --git a/documentation/configs/preprocess.md b/documentation/configs/preprocess.md index 26a332747..a9a7302b2 100644 --- a/documentation/configs/preprocess.md +++ b/documentation/configs/preprocess.md @@ -14,14 +14,13 @@ Values passed on the command line override the YAML before validation. | Flag | Overrides / Action | | :--- | :--- | -| `-r`, `--randomize` | Generates a random `seed`. The seed affects `between_sequence` split assignment. | -| `-dp`, `--data-path` | Overrides `preprocessing_data_path`. | -| `-sc`, `--selected-columns` | Overrides `selected_columns` with a space-separated list. Use `None` to process all columns. | +| `--seed` | Integer run seed, or `random` to resolve a fresh seed once. Affects `between_sequence` split assignment. | +| `--preprocessing-data-path` | Overrides `preprocessing_data_path`. | +| `--selected-columns` | Overrides `selected_columns` with a space-separated list. Use `None` to process all columns. | ## Composable Configuration Files -A preprocessing entry config may set `additional_config_paths` to one -non-empty string, a list of non-empty strings, or `null`. Relative paths +A preprocessing entry config may set `additional_config_paths` to a list of non-empty strings or `null`. Relative paths resolve against the entry config's `project_root`; absolute paths are used directly. Fragments are direct only and cannot include further fragments. They may share nested containers when their child fields are disjoint, but duplicate @@ -35,8 +34,8 @@ The configuration is defined in a YAML file (e.g., `preprocess.yaml`). Below are | Field | Type | Mandatory | Default | Description | | :--- | :--- | :--- | :--- | :--- | -| `project_root` | `str` | **Yes** | - | The root directory of your Sequifier project. Usually `.` | -| `additional_config_paths` | `str`, `list[str]`, or `null` | No | `null` | Direct complementary YAML fragments. Relative paths resolve against `project_root`; recursive composition and duplicate fields are rejected. | +| `run.project_root` | `str` | **Yes** | - | The root directory of your Sequifier project. Usually `.` | +| `additional_config_paths` | `list[str]` or `null` | No | `null` | Direct complementary YAML fragments. Relative paths resolve against `project_root`; recursive composition and duplicate fields are rejected. | | `preprocessing_data_path` | `str` | **Yes** | - | Path to the raw input file or folder. | | `static_data_path` | `str` | No | `null` | Separate CSV or Parquet table with exactly one row per `sequenceId`. Preprocessing requires its ID set to match the sequence data in both directions and writes a processed static feature table. | | `static_categorical_columns` | `list[str]` | No | `[]` | Static columns to treat as categorical when their source dtype would not make that intent clear. | @@ -71,20 +70,24 @@ flat folder per named view. Untimed images are named one `{sequence_id}` video per sequence. Decoded video frames are numbered from zero in decode order. A separate `preconditioning_image` source contains one `{sequence_id}` image per sequence in each folder. Its -`path` can be one folder or a map of named view folders. +`paths` maps view names to folders, even for a single view. ```yaml image_sources: front: path: data/front_images type: images - alignment: {frames: 3, items: 2} + alignment: + frames: 3 + items: 2 side: path: data/side_videos type: videos - alignment: {frames: 1, items: 2} + alignment: + frames: 1 + items: 2 preconditioning_image: - path: + paths: front: data/sequence_front_images side: data/sequence_side_images ``` @@ -196,8 +199,7 @@ depth feature/position column. Names beginning with | Field | Type | Mandatory | Default | Description | | :--- | :--- | :--- | :--- | :--- | -| `window_length` | `int` | Conditional | - | One physical width for every split. Specify exactly one of `window_length` or `window_lengths`. | -| `window_lengths` | `list[int]` | Conditional | - | Physical widths in split order, with one entry per split. Required when `prediction_alignment_steps > 1`; widths from split 2 onward must equal the first width plus `prediction_alignment_steps - 1`. | +| `window_lengths` | `list[int]` | **Yes** | - | Physical widths in split order, with one entry per split. Required for every split; widths from split 2 onward must equal the first width plus `prediction_alignment_steps - 1`. | | `max_target_offset` | `int` | No | `1` | Number of future items retained after the model input window. Use `0` for BERT-style same-width inputs and targets; use `1` for causal next-item training. | | `split_ratios` | `list[float]`| Conditional | `null` | Ordered split proportions for `within_sequence` and `between_sequence`. Must sum to 1.0 and must be omitted for `value_cutoff`. | | `split_method` | `str` | No | `within_sequence` | How rows are assigned to splits: `within_sequence`, `between_sequence`, or `value_cutoff`. | @@ -230,10 +232,12 @@ To align validation or test predictions exactly to their split positions, config those split indices and the model's prediction view: ```yaml -window_length: 129 +window_lengths: [129, 129, 129] max_target_offset: 1 window_stride: 128 -prediction_aligned_splits: [1, 2] +prediction_aligned_splits: +- 1 +- 2 prediction_length: 2 target_offset: 1 ``` @@ -246,7 +250,7 @@ left-padded. The stored windows never read beyond the split end. An aligned split requires `target_offset == max_target_offset` and `allow_sequence_splitting: false`. Training and inference must use matching `target_offset` and `prediction_length`. Training can sample additional model -views with `global_training.window_stride`; aligned inference requires +views with `phase learning.window_stride`; aligned inference requires `window_stride: null`. Empty splits produce no windows. Prediction-aligned inference supports generative and embedding output without @@ -264,8 +268,8 @@ PT output. | Field | Type | Mandatory | Default | Description | | :--- | :--- | :--- | :--- | :--- | -| `seed` | `int` | No | `1010` | Random seed for reproducibility. | -| `n_cores` | `int` | No | Max Cores | Maximum worker processes for eligible preprocessing stages. Depth preprocessing parallelizes a single Parquet file only when it has multiple row groups and `max_rows` is unset. | +| `run.reproducibility.seed` | `int` | No | `1010` | Random seed for reproducibility. | +| `run.execution.data_loading.num_workers` | `int` | No | Max Cores | Maximum worker processes for eligible preprocessing stages. Depth preprocessing parallelizes a single Parquet file only when it has multiple row groups and `max_rows` is unset. | | `batches_per_file` | `int` | No | `1024` | Buffer flush threshold per split. For flat input, counts sequence-window groups in PT and Parquet output (including merged output); a byte limit can flush earlier. For depth input, counts windows in PT output. | | `process_by_file` | `bool` | No | `true` | Memory optimization. If `true`, processes one input file at a time. | @@ -296,7 +300,7 @@ training and inference also check the width against the model's `context_length` To use the entire window, choose its width as `context_length + target_offset + (prediction_alignment_steps - 1) × prediction_length`. Earlier positions outside the split are masked from prediction output. -Set `window_lengths` when `prediction_alignment_steps > 1`. The first two +Set `window_lengths` explicitly for every split. The first two split widths can differ; every later split width must equal the first width plus `prediction_alignment_steps - 1`, even when that split uses distributed placement. @@ -306,9 +310,14 @@ the following gives aligned splits room for three two-position prediction groups: ```yaml -window_lengths: [131, 133, 133] +window_lengths: +- 131 +- 133 +- 133 max_target_offset: 1 -prediction_aligned_splits: [1, 2] +prediction_aligned_splits: +- 1 +- 2 prediction_length: 2 prediction_alignment_steps: 3 target_offset: 1 @@ -316,9 +325,9 @@ target_offset: 1 | Scenario | Suggested settings | Trade-off | | --- | --- | --- | -| Many short or varied-length sequences | Store the minimum width (for example, `window_length: 129` for `context_length: 128`, `max_target_offset: 1`), use a preprocessing `window_stride` near 128, and set training `window_stride: null`. | Limits padding and stores roughly one copy of long sequences, but targets near window boundaries have less preceding context. | +| Many short or varied-length sequences | Store the minimum width (for example, `window_lengths: [129, 129, 129]` for `context_length: 128`, `max_target_offset: 1`), use a preprocessing `window_stride` near 128, and set training `window_stride: null`. | Limits padding and stores roughly one copy of long sequences, but targets near window boundaries have less preceding context. | | More overlap during training | Keep that width; reduce the preprocessing `window_stride` to a fraction of the training `context_length`. | Roughly 2× or 4× as many stored events for long sequences. | -| Long sequences, several model views per stored window | Use a longer stored width (for example, `window_length: 513`, `window_stride: 384`, `context_length: 128`, training config `window_stride: 128`). | About 1.3× stored events on long sequences; short sequences pad to 513, and more model views cost more compute. | +| Long sequences, several model views per stored window | Use a longer stored width (for example, `window_lengths: [513, 513, 513]`, `window_stride: 384`, `context_length: 128`, training config `window_stride: 128`). | About 1.3× stored events on long sequences; short sequences pad to 513, and more model views cost more compute. | | Dense evaluation with nearly full preceding context at each window's right edge | Use `prediction_aligned_splits` for exact split coverage, or a small preprocessing `window_stride` for distributed evaluation. | More stored windows and more evaluation compute. | Self-attention lets the model use relationships between positions within a @@ -351,17 +360,18 @@ categorical column. Each key is a column name. For example, keep the 10,000 most frequent product IDs and hash every other product ID into 4,096 buckets: ```yaml -categorical_columns: [product_id] +categorical_columns: +- product_id cardinality_config: product_id: - top_k: 10000 + max_categories: 10000 hashing: num_buckets: 4096 seed: 42 ``` -`top_k` retains at most that many values, ranked by frequency. Alternatively, -`min_freq` retains every value occurring at least that many times. Both must be +`max_categories` retains at most that many values, ranked by frequency. Alternatively, +`min_count` retains every value occurring at least that many times. Both must be positive integers and cannot be used together. Retained values keep individual IDs; all remaining non-null values, including values first seen in later splits, map to a hash bucket when `hashing` is configured. Without `hashing`, @@ -376,7 +386,7 @@ cardinality_config: ``` `num_buckets` is a positive integer. The optional `hashing.seed` defaults to -`0` and is separate from the top-level preprocessing `seed`. Hashing is stable +`0` and is separate from the preprocessing `run.reproducibility.seed`. Hashing is stable for strings, booleans, and integers. Multiple values can land in the same bucket, so their original identities cannot be recovered from preprocessed data; decoded bucket values appear as `[hash_bucket:k]`. Null values retain the @@ -422,7 +432,7 @@ After running `preprocess`, the following are generated: 1. **Data Files:** Located in `data/`. Depending on your configuration, these will be merged files such as `[NAME]-split0.parquet` (Training), `[NAME]-split1.parquet` (Validation), etc., or split folders such as `[NAME]-split0/` containing `.pt` or `.parquet` shards. 2. **Metadata Config:** Located in `configs/metadata_configs/[NAME].json`. * **Crucial:** This file contains the integer mappings for categorical variables (`id_maps`), statistics for real variables (`selected_columns_statistics`), and whether those variables were normalized (`normalize_real_columns`). - * **Next Step:** Reference this file from `dataset.part.metadata_config_path` in a singleton training config, or from `dataset_training..parts..metadata_config_path` in a named training config. In inference, either `preprocessing_data_path` or `metadata_config_path` can locate the metadata and its split paths. + * **Next Step:** Reference this file from `dataset_training..parts..metadata_config_path` in a training config. In inference, use `metadata_config_path` to locate external metadata and its split paths. ## Named depth layouts @@ -431,23 +441,36 @@ Depth behavior is explicit; a column named `subItemPosition` alone remains an ordinary flat feature. ```yaml -project_root: . preprocessing_data_path: data/raw-items read_format: parquet write_format: pt merge_output: false -selected_columns: [accountType, subitemType, subitemAmount, nextAction] +selected_columns: +- accountType +- subitemType +- subitemAmount +- nextAction depth_layouts: subitems: position_column: subItemPosition - columns: [subitemType, subitemAmount] - context_length: 16 + columns: + - subitemType + - subitemAmount position_base: 0 allow_gaps: false -window_length: 129 + max_depth_length: 16 max_target_offset: 1 -split_ratios: [0.8, 0.1, 0.1] +split_ratios: +- 0.8 +- 0.1 +- 0.1 window_stride: 128 +window_lengths: +- 129 +- 129 +- 129 +run: + project_root: . ``` Every file must contain the depth position column, which is read automatically @@ -472,7 +495,7 @@ statistics count occupied child slots. Both populations are selected before split extraction. Materialization encodes each item once and uses bounded windows and output batches; `batches_per_file` bounds the number of windows accumulated per split. For a single Parquet file with multiple row groups and -no `max_rows`, this adapter uses `n_cores` to parallelize auditing, metadata +no `max_rows`, this adapter uses `run.execution.data_loading.num_workers` to parallelize auditing, metadata fitting, and window materialization. Other depth inputs use the sequential merge path. @@ -482,16 +505,16 @@ always allowed. With gaps enabled, physical slots remain unchanged. Outer item positions must be continuous within each selected sequence. An item in this raw format must have at least one child; null child rows do not encode emptiness. -All PT files use version 5 of the `sequifier_tensor_batch` envelope. The payload +Sequifier writes and validates the PT tensor-batch envelope. The payload stores shallow tensors as `[N,W]`, deep tensors as `[N,W,D]`, boolean masks under `metadata.depth_valid_masks.`, and the absolute window and split-boundary positions needed to enforce split ownership. Optional signed Int64 curriculum values are stored under `metadata.sample_positions`; their source names are stored under `metadata.curriculum_columns`, and multiple preserved columns use -shape `[N,C]`. Public metadata records the model-facing -`tensor_payload_version` separately from this internal storage envelope. Readers -reject legacy tuple payloads and envelope versions 2 through 4; re-run -preprocessing to migrate them. +shape `[N,C]`. Generated metadata records whether the dataset is tabular +or includes depth. A model interface retains the dataset kind even when it selects +only shallow columns from a depth dataset. Depth preprocessing resume ledgers +record whether preprocessing ran in serial or parallel. Categorical padding is the existing unknown-token ID (zero); real padding is finite zero after normalization. Temporal padding has false depth masks. @@ -507,3 +530,17 @@ layout properties, so unused stored layouts/features can be added independently. To consume a named layout during training, configure a `depth_transformer` ingestion branch whose `layout` names this entry and whose `columns` are selected features from it. See [the training guide](train.md#depth-encoders-and-nested-composites). + +## Import sequence demonstrations + +`operation: import_sequences` selects the explicit policy-data import operation. +It accepts a `source`, a model-independent `policy` data schema, an `output_path` +(new dataset directory) and optional `episodes_per_shard` (default 64). Sources +can be raw CSV/Parquet tables, existing scalar prepared histories with their +metadata, or JSON rollout recordings. Import preserves domain values and missing +transition information; it does not fit vocabularies/statistics or create padded +windows. Raw tables can also be used directly by policy training without this step. + +The operation atomically publishes Parquet shards and a checksummed manifest. +See [policy learning](../policy-learning.md#optional-committed-import) and +[import-policy-sequences.yaml](../example_inputs/configs/import-policy-sequences.yaml). diff --git a/documentation/configs/train.md b/documentation/configs/train.md index c7d155773..269632d06 100644 --- a/documentation/configs/train.md +++ b/documentation/configs/train.md @@ -1,88 +1,239 @@ # Train Command Guide +For trajectory-backed categorical BC, return-conditioned imitation, and offline +IQL and online PPO through the same commands, see [policy learning](../policy-learning.md). + `sequifier train` trains one shared transformer backbone through one or more named model interfaces. An interface is an ingestion module, its generated -adapter, and a decoder. Datasets own data and training policy; `model` -owns architecture. +adapter, and a decoder. Phases own batching, learning, trainability and lifecycle controls; datasets own sources; `model` owns architecture and execution contracts. See the [ownership reference](ownership.md) for the complete field map and migration command. ```console sequifier train --config-path configs/train.yaml ``` -## Singleton configuration +## Minimal named configuration -When a run has one model interface, one dataset, one part, and one phase, those -values can be authored directly without routing names or references: +Use the same named structure for one or many interfaces, datasets, parts, and phases: ```yaml -project_root: . -model_name: event-model -device: cuda -seed: 1010 - -global_training: - read_format: parquet - training_objective: causal - context_length: 128 - inference_batch_size: 256 - batch_size: 64 - learning_rate: 0.0001 - model: backbone: architecture: - dim_model: 128 + model_dim: 128 max_context_length: 512 num_layers: 6 - attention: {type: mha, n_heads: 8, n_kv_heads: 8, output_projection: true} - feed_forward: {dim: 512, activation: swiglu} - normalization: {type: rmsnorm, norm_first: true} - position_encoding: {type: rope, theta: 10000} + attention: + type: mha + num_heads: 8 + num_kv_heads: 8 + output_projection: true + feed_forward: + activation: swiglu + hidden_dim: 512 + normalization: + type: rmsnorm + norm_first: true + position_encoding: + type: rope + theta: 10000 dropout: 0.1 shared_layer_groups: [] - interface: - input_columns: [event] - target_columns: [event] - ingestion: {type: embedding, output_dim: 128} - decoder: {type: linear, prediction_length: 1, support: 1} - -dataset: - part: {metadata_config_path: configs/metadata_configs/events.json} - criterion: {event: CrossEntropyLoss} - + interfaces: + event_prediction: + input_columns: + - event + target_columns: + - event + categorical_decoder_special_tokens: + event: + - other + ingestion: + type: embedding + output_dim: 128 + decoder: + type: linear + prediction_length: 1 + support_length: 1 + execution: + context_length: 128 + target_offset: 1 + training_objective: causal +dataset_training: + events: + model_interface: event_prediction + parts: + original: + metadata_config_path: configs/metadata_configs/events.json + read_format: parquet + increment: + metadata_config_path: configs/metadata_configs/events-increment.json + read_format: parquet + split: + validation_fraction: 0.2 training_plan: - epochs: 5 - -evaluation: true + phases: + - name: incremental_finetuning + epochs: 2 + mode: sequential + sources: + - source: events.increment + batch_size: 64 + accumulation_batches: 4 + optimization: + model: + learning_rate: 0.0001 + optimizer: + name: AdamW + arguments: + weight_decay: 0.01 + scheduler: + name: StepLR + arguments: + step_size: 1 + gamma: 0.99 + scheduler_step_on: epoch + gradient_clip_norm: 1.0 + optimizer_state: reset + evaluation: + sources: + - source: events + cadence: + epochs: 1 + batch_size: 64 + before_phase: true + on_snapshot: true + algorithm: supervised + model_selection: + handoff: last + checkpoints: + cadence: + epochs: 1 + learning: + window_stride: 1 + datasets: + events: + criterion: + event: CrossEntropyLoss + loss_weights: + event: 1.0 + trainability: + by_dataset: + events: + roles: + model: &id001 + freeze: + - component: backbone + exclude_modules: [] + semantic_groups: + - attention.qkv + - name: complete_retraining + epochs: 5 + mode: interleaved + selection: round_robin + sources: + - source: events + batches_per_selection: 4 + batch_size: 64 + accumulation_batches: 4 + optimization: + model: + learning_rate: 0.0001 + optimizer: + name: AdamW + arguments: + weight_decay: 0.01 + scheduler: + name: StepLR + arguments: + step_size: 1 + gamma: 0.99 + scheduler_step_on: epoch + gradient_clip_norm: 1.0 + optimizer_state: reset + evaluation: + sources: + - source: events + cadence: + epochs: 1 + batch_size: 64 + before_phase: false + on_snapshot: true + algorithm: supervised + model_selection: + handoff: last + checkpoints: + cadence: + epochs: 1 + learning: + window_stride: 1 + datasets: + events: + criterion: + event: CrossEntropyLoss + loss_weights: + event: 1.0 + trainability: + by_dataset: + events: + roles: + model: *id001 +run: + project_root: . + model_name: event-model + reproducibility: + seed: 1010 + enforce_determinism: false + execution: + device: cuda + distribution: + strategy: local + world_size: 1 + backend: nccl + precision: {} + data_loading: {} + compilation: + torch_compile: outer + recovery: {} + logging: {} + outputs: {} + export: + generative_model: true + embedding_model: false + onnx: true + pt: false + with_dropout: false + batch_size: 256 ``` -The concise form expands before validation: - -| Authored field | Canonical form | -| --- | --- | -| `model.interface` | `model.interfaces.default` | -| `dataset.part` | `dataset_training.default.parts.default` | -| `training_plan.epochs` | One sequential phase named `train` | -| `evaluation: true` | Evaluate the inferred single source | - -These shortcuts can be used independently. A unique interface and training -source are inferred; multiple interfaces or datasets require explicit names and -references. Singular and plural spellings cannot be combined. Because errors -are reported after expansion, they may use the canonical paths above. +All configurations use `model.interfaces`, `dataset_training`, dataset `parts`, +`policy_bindings` where applicable, and `training_plan.phases`. A unique interface +and source may be inferred when omitted. See the [naming reference](naming.md). To condition every model window on features from a separate per-sequence table, preprocess with `static_data_path` and configure the interface: ```yaml model: - interface: - input_columns: [event] - static_conditioning: - columns: [account_type, account_age] - feature_embedding_dims: {account_type: 16, account_age: 8} - target_columns: [event] - ingestion: {type: embedding, output_dim: 128} - decoder: {type: linear, prediction_length: 1, support: 1} + interfaces: + default: + input_columns: + - event + static_conditioning: + columns: + - account_type + - account_age + feature_embedding_dims: + account_type: 16 + account_age: 8 + target_columns: + - event + ingestion: + type: embedding + output_dim: 128 + decoder: + type: linear + prediction_length: 1 + support_length: 1 ``` The static columns are separate from temporal `input_columns` and targets. @@ -90,96 +241,207 @@ They are looked up by `sequenceId` for every sampled window, including windows from the middle of a sequence. `feature_embedding_dims` assigns a width to each static feature; real features with width greater than one use a learned projection. Alternatively, -`embedding_size` distributes a total width automatically when all static +`total_embedding_dim` distributes a total width automatically when all static features are categorical or all are real. For mixed types, specify `feature_embedding_dims`. If neither option is set, categorical widths use the existing cardinality-based default and real features use one channel. -The concatenated features are projected to the backbone `dim_model`. +The concatenated features are projected to the backbone `model_dim`. The model creates one prefix token at runtime and removes its output before decoding. Set backbone `max_context_length` to -at least `global_training.context_length + 1`. +at least `model.execution.context_length + 1`. ## Canonical configuration ```yaml -project_root: . -model_name: event-model -device: cuda -seed: 1010 - -global_training: - read_format: parquet - training_objective: causal - context_length: 128 - target_offset: 1 - window_stride: 1 - inference_batch_size: 256 - batch_size: 64 - accumulation_steps: 4 - learning_rate: 0.0001 - optimizer: {name: AdamW, weight_decay: 0.01} - scheduler: {name: StepLR, step_size: 1, gamma: 0.99} - scheduler_step_on: epoch - reset_optimization_on_phase: true - gradient_clip: 1.0 - save_interval_epochs: 1 - model: backbone: architecture: - dim_model: 128 + model_dim: 128 max_context_length: 512 num_layers: 6 - attention: {type: mha, n_heads: 8, n_kv_heads: 8, output_projection: true} - feed_forward: {dim: 512, activation: swiglu} - normalization: {type: rmsnorm, norm_first: true} - position_encoding: {type: rope, theta: 10000} + attention: + type: mha + num_heads: 8 + num_kv_heads: 8 + output_projection: true + feed_forward: + activation: swiglu + hidden_dim: 512 + normalization: + type: rmsnorm + norm_first: true + position_encoding: + type: rope + theta: 10000 dropout: 0.1 shared_layer_groups: [] interfaces: event_prediction: - input_columns: [event] - target_columns: [event] - categorical_decoder_special_tokens: {event: [other]} - ingestion: {type: embedding, output_dim: 128} - decoder: {type: linear, prediction_length: 1, support: 1} - + input_columns: + - event + target_columns: + - event + categorical_decoder_special_tokens: + event: + - other + ingestion: + type: embedding + output_dim: 128 + decoder: + type: linear + prediction_length: 1 + support_length: 1 + execution: + context_length: 128 + target_offset: 1 + training_objective: causal dataset_training: events: model_interface: event_prediction parts: - original: {metadata_config_path: configs/metadata_configs/events.json} - increment: {metadata_config_path: configs/metadata_configs/events-increment.json} - criterion: {event: CrossEntropyLoss} - loss_weights: {event: 1.0} - freeze: - backbone: {freeze: [attention.qkv]} - + original: + metadata_config_path: configs/metadata_configs/events.json + read_format: parquet + increment: + metadata_config_path: configs/metadata_configs/events-increment.json + read_format: parquet + split: + validation_fraction: 0.2 training_plan: phases: - - name: incremental_finetuning - epochs: 2 - mode: sequential - sources: [{source: events.increment}] - - name: complete_retraining - epochs: 5 - mode: interleaved - selection: round_robin - sources: [{source: events, batches_per_selection: 4}] - -evaluation: - sources: [{source: events}] - -export_generative_model: true -export_embedding_model: false -export_onnx: true -export_pt: false -export_with_dropout: false + - name: incremental_finetuning + epochs: 2 + mode: sequential + sources: + - source: events.increment + batch_size: 64 + accumulation_batches: 4 + optimization: + model: + learning_rate: 0.0001 + optimizer: + name: AdamW + arguments: + weight_decay: 0.01 + scheduler: + name: StepLR + arguments: + step_size: 1 + gamma: 0.99 + scheduler_step_on: epoch + gradient_clip_norm: 1.0 + optimizer_state: reset + evaluation: + sources: + - source: events + cadence: + epochs: 1 + batch_size: 64 + before_phase: true + on_snapshot: true + algorithm: supervised + model_selection: + handoff: last + checkpoints: + cadence: + epochs: 1 + learning: + window_stride: 1 + datasets: + events: + criterion: + event: CrossEntropyLoss + loss_weights: + event: 1.0 + trainability: + by_dataset: + events: + roles: + model: &id001 + freeze: + - component: backbone + exclude_modules: [] + semantic_groups: + - attention.qkv + - name: complete_retraining + epochs: 5 + mode: interleaved + selection: round_robin + sources: + - source: events + batches_per_selection: 4 + batch_size: 64 + accumulation_batches: 4 + optimization: + model: + learning_rate: 0.0001 + optimizer: + name: AdamW + arguments: + weight_decay: 0.01 + scheduler: + name: StepLR + arguments: + step_size: 1 + gamma: 0.99 + scheduler_step_on: epoch + gradient_clip_norm: 1.0 + optimizer_state: reset + evaluation: + sources: + - source: events + cadence: + epochs: 1 + batch_size: 64 + before_phase: false + on_snapshot: true + algorithm: supervised + model_selection: + handoff: last + checkpoints: + cadence: + epochs: 1 + learning: + window_stride: 1 + datasets: + events: + criterion: + event: CrossEntropyLoss + loss_weights: + event: 1.0 + trainability: + by_dataset: + events: + roles: + model: *id001 +run: + project_root: . + model_name: event-model + reproducibility: + seed: 1010 + enforce_determinism: false + execution: + device: cuda + distribution: + strategy: local + world_size: 1 + backend: nccl + precision: {} + data_loading: {} + compilation: + torch_compile: outer + recovery: {} + logging: {} + outputs: {} + export: + generative_model: true + embedding_model: false + onnx: true + pt: false + with_dropout: false + batch_size: 256 ``` -The historical flat training schema is not accepted. In particular, -`training_spec`, top-level dataset paths/columns, `model.ingestion`, -`model.decoder`, and architecture-owned freezing are not canonical fields. For a causal objective, `target_offset` sets the target position relative to each input position. With `target_offset: 1`, every decoded position predicts @@ -189,18 +451,28 @@ several future tokens as in LLM multi-token prediction. Training currently computes loss over all valid decoded positions, regardless of `prediction_length`. +For `model.execution.training_objective: next_occurrence`, declare the trigger +in each supervised phase's `learning` settings: + +```yaml +learning: + next_occurrence_config: + column_name: event_type + target_values: [purchase] +``` + +The trigger column must be a categorical target in every dataset used by that +phase, and its values must exist in that dataset's preprocessing metadata. +Different phases may select different triggers. These settings define training +and evaluation targets; they are not needed to execute an exported model. + ## Ownership and resolution -- `global_training` owns objective, window, optimizer, precision, - distribution, compilation, checkpoint, and data-loader behavior. Phase - entries own `epochs`. -- `model` contains exactly one backbone and one or more named interfaces. - Different interface names create distinct ingestion and decoder weights; - repeated references to one name share those weights. -- `dataset_training` owns parts, criterion/weights, class-share logging, - freezing, and the interface reference. -- Preprocessing metadata owns split paths, data types, class counts, ID maps, - special-token IDs, normalization facts, and stored-window layout. +- `run` owns execution resources, reproducibility, recovery, logging, outputs and export. +- `model.execution` owns immutable window/objective/attention facts; named interfaces own architecture and input contracts. +- `dataset_training` owns sources, interface references and split policy. +- Phases own losses, trainability, optimization, scheduling, evaluation, model selection, stopping and snapshots. +- Preprocessing metadata owns frozen storage, codec and normalization facts; explicit assertions must agree. For categorical hard-label targets, set a target's criterion to `AccurateCrossEntropyLoss` to preserve small loss values and target-logit @@ -208,9 +480,14 @@ gradients at high confidence. `CrossEntropyLoss` retains the existing PyTorch behavior. For example: ```yaml -dataset_training: - events: - criterion: {event: AccurateCrossEntropyLoss} +training_plan: + phases: + - name: fit + learning: + datasets: + events: + criterion: + event: AccurateCrossEntropyLoss ``` The choice is per target and is checked when resuming a run. Class @@ -239,8 +516,11 @@ training config with a backbone and dataset part: model: interfaces: default: - input_columns: [accountId, merchantId] - target_columns: [accountId] + input_columns: + - accountId + - merchantId + target_columns: + - accountId categorical_hashing: accountId: type: multi_hash @@ -256,7 +536,7 @@ model: decoder: type: linear prediction_length: 1 - support: 1 + support_length: 1 dataset_training: default: model_interface: default @@ -316,14 +596,52 @@ training is disabled. When disabled, curriculum values are ignored and the ordinary sample shuffle is preserved. ```yaml -dataset: - part: - metadata_config_path: configs/metadata_configs/events.json - file_order: name training_plan: - curriculum_training: true - curriculum_column: difficulty - epochs: 5 + phases: + - epochs: 5 + name: train + mode: sequential + algorithm: supervised + model_selection: + handoff: last + checkpoints: + cadence: + epochs: 1 + learning: + datasets: + default: {} + curriculum: + column: difficulty +dataset_training: + default: + parts: + default: + metadata_config_path: configs/metadata_configs/events.json + file_order: name + read_format: parquet + split: + validation_fraction: 0.2 +run: + project_root: . + model_name: training + reproducibility: + seed: 1010 + enforce_determinism: false + execution: + device: cpu + distribution: + strategy: local + world_size: 1 + backend: nccl + precision: {} + data_loading: {} + compilation: + torch_compile: none + recovery: {} + logging: {} + outputs: {} + export: + batch_size: 1 ``` `loss_weights` scales each target's contribution to the training and reported @@ -333,27 +651,40 @@ target in each dataset must have a positive weight. ## Optimizer plans -The single-optimizer form above continues to apply one optimizer to all -parameters and uses `global_training.learning_rate`. To select parameters or +The single-optimizer form applies one optimizer to all +parameters and uses the role’s `learning_rate`. To select parameters or use multiple optimizers, replace it with an ordered plan. Each group has its -own learning rate; omit the run-wide `learning_rate` in this form: +own learning rate; use each group’s `learning_rate` in this form: ```yaml -global_training: - optimizer: - groups: - - id: hidden_matrices - select: - semantic_groups: ["attention.*", "feed_forward.*"] - component: backbone - parameter_kind: weight - ndim: 2 - optimizer: {name: Muon, momentum: 0.95} - learning_rate: 0.02 - - id: remainder - select: otherwise - optimizer: {name: AdamW, weight_decay: 0.01} - learning_rate: 0.0003 +training_plan: + phases: + - name: supervised + algorithm: supervised + optimization: + model: + optimizer: + groups: + - id: hidden_matrices + select: + semantic_groups: + - attention.* + - feed_forward.* + component: backbone + parameter_kind: weight + ndim: 2 + optimizer: + name: Muon + arguments: + momentum: 0.95 + learning_rate: 0.02 + - id: remainder + select: otherwise + optimizer: + name: AdamW + arguments: + weight_decay: 0.01 + learning_rate: 0.0003 ``` Each supplied selector condition must match. `semantic_groups` matches the @@ -378,52 +709,177 @@ With `CyclicLR`, each group's `learning_rate` is also its peak rate. Add a `base_learning_rate` below that peak to every optimizer group, and omit `scheduler.base_lr` and `scheduler.max_lr`. For the plan above, the two base rates could be `0.002` and `0.00003`, respectively. Configure the scheduler -with `name: CyclicLR`, `step_size_up: 2000`, and `scheduler_step_on: batch`. +with `name: CyclicLR`, `step_size_up: 2000`, and `scheduler_step_on: optimizer_step`. Momentum cycling defaults to off for both cycle schedulers in plan mode; explicit `cycle_momentum: true` is unsupported. `base_learning_rate` is only valid with `CyclicLR`. +## Canonical training settings + +Batching, role-keyed optimization and trainability belong to phases. Evaluation +uses `evaluation.cadence: {epochs: N}` for supervised phases and +`{iterations: N}` for policy phases. Retained checkpoint cadence is +`checkpoints.cadence` with the same explicit units. Final progress always +evaluates when enabled and checkpoints. + +Evaluation owns `before_phase`, `on_snapshot` and `batch_size`. For supervised +phases, an omitted or null evaluation `batch_size` inherits the phase training +batch size. An explicit positive value overrides it. Model selection +uses `model_selection.handoff: best | last`; stopping uses +`stopping: {patience: N, min_delta: 0, on_trigger: end_phase | end_plan}`. +Patience counts scheduled finite observations. In supervised training, initial +and snapshot evaluations are diagnostic: they do not select best weights or +consume stopping patience. The initial evaluation is recorded as complete in +checkpoints, so resuming does not repeat it. Best weights are selected from +scheduled epoch-end evaluations, even if the initial loss was better. Supervised +phases preserve best weights across phase boundaries when their resolved evaluation contracts match. +The contract includes validation sources, model interfaces, loss definitions, +window views, effective evaluation batch size and precision. Optimizer settings, +freezing and training duration do not affect compatibility. BERT phases have +different evaluation-mask seeds and therefore start separate histories. Set +`model_selection.scope: phase` to explicitly start a new history; the default is +`compatible_phases`. Stopping patience remains phase-local. +Best handoff, stopping and plateau +scheduling require an evaluation monitor. See [ownership](ownership.md#phase-settings). + +Frozen normalization is an interface input contract or inherited artifact fact. +Recovery policy is `run.recovery`; phase snapshots belong to `checkpoints`. + +## Model prediction parameters and policy actions + +Model interfaces own prediction parameters. Policy bindings select targets as +actions and define their support, transforms, conditional activity and scale use. +They never create, overwrite or remove model parameter declarations. + +```yaml +model: + interfaces: + actor: + prediction_distributions: + throttle: + scale: + initial: 0.2 + trainable: true + min: 0.0001 + max: 100 +policy_bindings: + actor: + actions: + throttle: + type: real + scale: model + lower: 0 + upper: 1 +``` + +A real action must explicitly choose `scale: model` or `scale: {fixed: 0.1}`. +The latter bypasses the model scale for that policy without deleting its learned +parameter. A fixed policy scale does not require a model distribution entry. +Categorical actions declare `values` and cannot configure real scales or bounds. +`GaussianNLLLoss` uses the model's Gaussian distribution and its scale; policy +bounds, point masses and activity conditions apply only to policy likelihoods. + +When initializing from an artifact, a full `model` declaration may add prediction +distribution entries. The architecture and existing distribution declarations +must match; only newly declared scale parameters receive fresh initialization. + ## Optimization across phases -By default, every training phase starts with a new optimizer, scheduler, and -gradient scaler. Model weights and global progress counters continue across -phase boundaries, while optimizer moments and scheduler position do not. Set -`global_training.reset_optimization_on_phase: false` to keep the optimizer, -scheduler, and gradient scaler operating continuously across all phases. A -checkpoint restores the active optimization lifecycle exactly in either mode. - -For an epoch-stepped scheduler in reset mode, do not provide -`global_training.scheduler.total_steps` because Sequifier owns each phase's -scheduler duration. When a scheduler such as `OneCycleLR` accepts `total_steps`, -Sequifier sets it to the active phase's `epochs` in reset mode, or the sum of all -phase epochs in continuous mode. A continuous epoch-stepped scheduler may still -provide `total_steps` for compatibility, but it must equal the sum of all phase -epochs. Other scheduler arguments remain shared across phases. This example uses -a single optimizer: +Each phase owns `batch_size` (32 by default), `accumulation_batches` (1), and +`optimization.`. Supervised phases have the single role `model`, meaning +the trainable model parameters. Imitation, DPO and GRPO have `actor`; PPO also +has `value`; IQL has `actor`, `value`, and `critic`. Unknown roles fail validation. +Omitted roles use Adam, learning rate 0.0003 and gradient clip 1.0. Set +`gradient_clip_norm: null` to disable clipping. Optimizer plans take their rates from +their groups. Supervised roles default to StepLR (`step_size: 1`, `gamma: 0.99`); +policy roles default to ConstantLR (`factor: 1`, `total_iters: 1`). ```yaml -global_training: - scheduler: - name: OneCycleLR - max_lr: 0.003 - pct_start: 0.2 - cycle_momentum: false - scheduler_step_on: epoch - reset_optimization_on_phase: false +training_plan: + phases: + - name: supervised + algorithm: supervised + epochs: 10 + mode: sequential + sources: + - source: observations.train + batch_size: 64 + accumulation_batches: 1 + optimizer_state: reset + optimization: + model: + optimizer: + name: Adam + arguments: {} + learning_rate: 0.001 + scheduler: + name: StepLR + arguments: + step_size: 1 + gamma: 0.99 + scheduler_step_on: epoch + gradient_clip_norm: 1.0 ``` -Batch-stepped schedulers may configure `total_steps` explicitly because phase -update counts can depend on source interleaving, dataset transitions, and -gradient accumulation. In reset mode, that value applies independently to each -phase; configuration emits a warning to make that scope explicit. In continuous -mode, it applies once across the full run. +`optimizer_state: reset | continue` describes the transition **into** this phase; +its default is `reset`. Reset creates new optimizer, scheduler and scaler state. +Continue preserves these states from the immediately preceding phase and requires +that phase's `model_selection.handoff: last`, identical algorithm, role configurations, parameter +ownership and freeze selections. The first phase cannot continue. Supervised +continuation requires the same dataset ownership; policy continuation requires +the same interface. Mixed plans reject continuation because their artifact +handoffs do not preserve optimizer state. Policy OneCycleLR continuation is +also rejected. Native checkpoints restore the active lifecycle in either mode. Supervised +`model_selection.handoff: best` requires an evaluation monitor and selects the preceding phase’s +best weights before starting the next phase; its monitor state is phase-local. + +Supervised `scheduler_step_on: epoch` advances once after a completed phase epoch; +ReduceLROnPlateau receives the configured evaluation monitor at that boundary +and requires the phase's `evaluation.monitor` and `evaluation.cadence: {epochs: 1}`. `optimizer_step` advances once per applied optimizer +update, after accumulation, rather than once per microbatch. Policy roles only +accept `scheduler_step_on: optimizer_step`, their default. Policy plateau and +compound schedulers are unsupported. Constructor arguments are validated before +training; the shared optimizer shape does not imply shared scheduler capabilities. + +For epoch OneCycleLR, the resolved `total_steps` is the phase's epoch count, or +the sum of epochs in its contiguous continuation chain. An explicit value must +match. For batch OneCycleLR, an explicit `total_steps` is used as configured +without checking it against the number of optimizer updates. If omitted, preflight +derives the value by counting loader batches using the same sequential/interleaved +source scheduler, phase seeds and accumulation rules as training, including partial +updates flushed at dataset boundaries. The derived budget sums over a contiguous +continuation chain. This calculation does not consume samples or advance loader +RNG. Early stopping and explicitly skipped updates may consume fewer steps. If an +explicit budget is shorter than the updates performed, the scheduler stops +advancing after its configured number of steps. +Policy OneCycleLR derives the maximum optimizer-update budget below. An explicit +`total_steps` must match it: + +| Algorithm | Optimizer updates per role | +| --- | --- | +| Imitation | epochs × ceil(training examples / (batch_size × accumulation_batches)) | +| IQL / DPO | iterations | +| PPO | optimization_epochs × sum(ceil(rollout size / (batch_size × accumulation_batches))) | +| GRPO | optimization_epochs × sum(ceil(group trajectories / (batch_size × accumulation_batches))) | + +PPO `rollout_env_steps` counts collected transitions; `batch_size` counts +transitions in each optimization microbatch. They are distinct budgets. IQL +uses `batch_size: 64` by default and one update per sampled batch; it does not +accept accumulation. DPO counts preference pairs; GRPO counts trajectories. +PPO defaults to `batch_size: 64`; other batched algorithms default to 32. +Early stopping, skipped updates or KL limits can consume less than the maximum. + +The authored schema is separate from resolved training phases. Checkpoints and +plan fingerprints contain resolved roles, batching and scheduler budgets; +reproduction does not depend on global optimizer values. Run execution policy and +durable model execution facts have separate owners; see [ownership](ownership.md). ## Training plans A sequential phase exhausts each source in listed order. An interleaved phase uses `round_robin` or `weighted_random`. Each selection consumes at most `batches_per_selection`; every source is still exhausted once per phase epoch. -Weights affect order and burst frequency, not the amount of data consumed. +Source `selection_weight` values affect order and burst frequency, not the amount of data consumed. Gradient accumulation may cross part or source boundaries only while the dataset stays the same. A dataset transition flushes a partial window using its @@ -436,31 +892,32 @@ validation-based saving or early stopping is enabled: ```yaml evaluation: - sources: [{source: events}, {source: telemetry.main}] - monitor: {source: events, metric: loss, mode: min} + sources: + - source: events + - source: telemetry.main + monitor: + source: events + metric: loss + direction: min ``` -Checkpoint interval fields retain distinct behavior. `save_interval_epochs`, -`save_interval_batches`, and `save_interval_minutes` create persistent -epoch/batch snapshots and refresh the rolling `latest` checkpoint; -`save_latest_interval_minutes` refreshes only `latest`. -`save_interval_val_loss` controls whether timed or batch snapshots also run -validation and record its monitored loss. Every checkpoint is written after -flushing any partial gradient-accumulation window. +Phase `checkpoints.cadence` and snapshot triggers create retained checkpoints. +`run.recovery.latest_checkpoint_interval_minutes` refreshes rolling recovery. +`evaluation.on_snapshot` requests validation when a snapshot is due. Checkpoints +are written after flushing partial accumulation, or at a committed policy transaction. ## Composable YAML files An entry file may declare complementary fragments with `additional_config_paths`. Relative paths resolve against the entry file's `project_root`. Fragments can contribute disjoint children under containers -such as `global_training`, `model.interfaces`, and +such as `run`, `model.interfaces`, and `dataset_training`; duplicate fields are rejected. CLI overrides are applied after composition and before metadata resolution. -Singleton fragments may likewise contribute disjoint fields under -`model.interface` or `dataset`. Normalization occurs after fragment -composition, so all fragments in one training config must consistently use the -singular or named spelling at each level. +Fragments use the same named paths as the entry file. Components use only +`{name, arguments}`; constructor arguments must be nested under `arguments`. +Initialization methods use mappings such as `{method: zeros}`. The training command accepts `--model-name`, `--seed`, and `--skip-metadata` as configuration overrides. Dataset paths, columns, metadata paths, and device @@ -468,7 +925,12 @@ selection must use their canonical YAML locations. ## Artifacts -ONNX is exported by default; PT inference bundles are opt-in. ONNX favors a +See [Training artifacts](../training/artifacts.md) for the shared output paths, +checkpoint envelopes, reports, logging and collection schemas across algorithms. + +PT is the default training export; supervised ONNX exports are opt-in. All +training routes also publish the canonical final PT model configured by +`run.outputs.model_path`. ONNX favors a portable deployment runtime, while PT embeds its execution contract and retains PyTorch behavior. See the "ONNX or PT?" section of the inference guide for the trade-offs. @@ -480,8 +942,9 @@ run-wide. Generated filenames do not use a `sequifier-` prefix. The PT inference bundle contains `artifact_type`, `format_version`, `model_state_dict`, an execution-only `model_config`, and export metadata -(trace-site names and provenance). Optimizers, paths, parts, training plans, -evaluation policy, and dataset bindings remain outside that bundle. +(trace-site names and provenance). Optimizer state, dataset parts, and evaluation runtime state remain outside +the execution contract. Optional provenance may include resolved training plans +and source paths. `export_with_dropout` affects ONNX export only: enabling it exports the ONNX graph in training mode and disables constant folding so dropout remains active. @@ -498,8 +961,10 @@ pool reduces the images assigned to one `itemPosition` to one embedding. model: interfaces: prediction: - input_columns: [temperature] - target_columns: [nextAction] + input_columns: + - temperature + target_columns: + - nextAction preconditioning_image_encoder: type: frozen model: resnet18 @@ -509,23 +974,31 @@ model: branches: measurements: type: embedding - columns: [temperature] + columns: + - temperature output_dim: 32 cameras: type: images - views: [front, side] + views: + - front + - side output_dim: 128 - encoder: {type: resnet, depth: 18} - frame_pool: attention + encoder: + type: resnet + depth: 18 + frame_pooling: attention view_merge: concat - merge: {type: concat} - decoder: {type: linear, prediction_length: 1} + merge: + type: concat + decoder: + type: linear + prediction_length: 1 ``` Bind the selected views to their original folders in each dataset part. The -preprocessing metadata deliberately omits folder paths. A single selected view -may use a string `image_path`; multiple views require a name-to-path map. -Preconditioning images use `preconditioning_image_path` in the same way: +preprocessing metadata deliberately omits folder paths. `image_paths` is always +a view-to-folder mapping, including runs with a single selected view. +Preconditioning images use `preconditioning_image_paths` in the same way: ```yaml dataset_training: @@ -534,11 +1007,14 @@ dataset_training: parts: main: metadata_config_path: configs/metadata_configs/camera_data.json - image_path: {front: data/front_images, side: data/side_videos} - preconditioning_image_path: + image_paths: + front: data/front_images + side: data/side_videos + preconditioning_image_paths: front: data/sequence_front_images side: data/sequence_side_images - criterion: {nextAction: CrossEntropyLoss} + criterion: + nextAction: CrossEntropyLoss ``` Image view names and structured image cell names cannot be target columns; @@ -557,9 +1033,39 @@ The `frozen` encoder supports TorchVision `resnet18` and `resnet50` with an explicit weight version. Its backbone stays frozen and in evaluation mode; its output projection trains. The default image size and normalization match the V1 ImageNet weight transforms. ResNet-50 V2 requires `resize_size: 232` -on its source; model resolution checks this. `frame_pool` is `mean` (default) or +on its source; model resolution checks this. `frame_pooling` is `mean` (default) or `attention`; `view_merge` is `concat` (default) or `mean`. +`encoder.microbatch_size` (default `16`) limits the number of frames in each +backbone call independently of policy scoring batches. Valid frames are gathered +one microbatch at a time. This bounds transient backbone memory for frozen +encoders; trainable encoders still retain activations for backward. ONNX export +uses the unsplit encoder path. + +`training_plan.phases[].image_feature_cache_bytes` defaults to `0` (disabled). +A positive value opts frozen image encoders into GRPO phase-scoped feature caching +on the model device. This is a runtime option, not a portable-model setting. +The limit counts retained feature tensor storage per distinct cache, in addition +to rollout buffers, model activations, allocator reservations and Python metadata. +The trainable projection is always recomputed with autograd enabled. Trainable +backbones always bypass caching. Outside GRPO's cache scope, encoding follows +the ordinary microbatched path. + +Owned policy frames receive process-local identities only while caching is active; +no pixels are hashed and identities are never written into model artifacts or +checkpoints. Entries are discarded when encoder tensor versions, normalization, +device or dtype change. Inference tensors without version counters bypass caching. +External code that mutates frozen weights using `.data` must call +`encoder.clear_feature_cache()`; ordinary weight loading/in-place updates are +tracked automatically. Entries are released after each iteration and on phase +exit; consumed GRPO trajectories do not keep unused features resident. Actor and +reference share entries only when their frozen encoder states, devices, dtypes, +normalization and budgets match exactly; either owner detaches on invalidation. + +Caching avoids repeated CNN computation, but currently still materializes and +transfers raw image windows. Its benefit is workload-dependent. Keep the default +disabled path for workloads without repeated frames or without VRAM headroom. + Image embeddings may also occupy cells in an existing structured layout. The cell names are part of `feature_layout`; `image_cells` maps those names to image view names. The usual structured processing blocks then combine image and @@ -568,18 +1074,30 @@ tabular cells: ```yaml feature_layout: sensors: - axes: {source: [camera, measurement]} + axes: + source: + - camera + - measurement columns: - camera_cell: {source: camera} - temperature: {source: measurement} + camera_cell: + source: camera + temperature: + source: measurement ingestion: type: structured layout: sensors output_dim: 128 - image_cells: {camera_cell: front} - image_encoder: {type: frozen, model: resnet50, weights: IMAGENET1K_V1} + image_cells: + camera_cell: front + image_encoder: + type: frozen + model: resnet50 + weights: IMAGENET1K_V1 processing_blocks: - - {type: axis_pool, axes: [source], mode: mean} + - type: axis_pool + axes: + - source + pooling: mean ``` The preconditioning source is independent of temporal views. One or more @@ -589,8 +1107,8 @@ that one prefix position. ## Depth encoders and nested composites -Use `global_training.read_format: pt` for preprocessed depth inputs. Layout -capacities come from dataset metadata. The following singleton model fragment +Use `read_format: pt` on every prepared part for preprocessed depth inputs. Layout +capacities come from dataset metadata. The following named model fragment combines one shallow branch with a depth encoder; its metadata comes from the preprocessing example in [preprocess.md](preprocess.md#named-depth-layouts). @@ -604,45 +1122,76 @@ model: backbone: initialization_seed: 20260910 architecture: - dim_model: 128 + model_dim: 128 max_context_length: 128 num_layers: 4 - attention: {type: mha, n_heads: 8} - feed_forward: {dim: 512, activation: swiglu} - position_encoding: {type: rope} - interface: - input_columns: [accountType, subitemType, subitemAmount] - target_columns: [nextAction] - ingestion: - type: composite - branches: - item: - type: embedding - columns: [accountType] - output_dim: 32 - subitems: - type: depth_transformer - layout: subitems - columns: [subitemType, subitemAmount] - feature_embedding_dims: {subitemType: 48, subitemAmount: 16} - output_dim: 96 - dropout: 0.1 - initialization_seed: 12345 - initialization: - attention.qkv: - weight: {method: xavier_uniform, gain: 1.0} - free_parameter: - weight: {method: normal, mean: 0.0, std: 0.02} - architecture: - dim_model: 64 - num_layers: 2 - attention: {type: gqa, n_heads: 4, n_kv_heads: 2} - feed_forward: {dim: 256, activation: swiglu} - normalization: {type: rmsnorm, norm_first: true} - position_encoding: {type: learned} + attention: + type: mha + num_heads: 8 + feed_forward: + activation: swiglu + hidden_dim: 512 + position_encoding: + type: rope + interfaces: + default: + input_columns: + - accountType + - subitemType + - subitemAmount + target_columns: + - nextAction + ingestion: + type: composite + branches: + item: + type: embedding + columns: + - accountType + output_dim: 32 + subitems: + type: depth_transformer + layout: subitems + columns: + - subitemType + - subitemAmount + feature_embedding_dims: + subitemType: 48 + subitemAmount: 16 + output_dim: 96 dropout: 0.1 - merge: {type: concat} - decoder: {type: linear, prediction_length: 1} + initialization_seed: 12345 + initialization: + attention.qkv: + weight: + method: xavier_uniform + gain: 1.0 + free_parameter: + weight: + method: normal + mean: 0.0 + std: 0.02 + architecture: + model_dim: 64 + num_layers: 2 + attention: + type: gqa + num_heads: 4 + num_kv_heads: 2 + feed_forward: + activation: swiglu + hidden_dim: 256 + normalization: + type: rmsnorm + norm_first: true + position_encoding: + type: learned + dropout: 0.1 + merge: + type: concat + decoder: + type: linear + prediction_length: 1 ``` For a `gated` or `attention` merge, `merge.softmax` accepts `standard` @@ -656,16 +1205,16 @@ GQA; LayerNorm or RMSNorm; the shared feed-forward activations and layer-sharing configuration. `architecture.dropout` controls depth position and transformer sites. The ingestion-level `dropout` controls the pooled output. Mixed categorical/real features require explicit feature widths; homogeneous features -can divide `architecture.dim_model` using the ordinary ingestion width rules. +can divide `architecture.model_dim` using the ordinary ingestion width rules. For categorical-only depth features, automatic allocation gives each column at -least two dimensions, so `architecture.dim_model` must be at least twice the +least two dimensions, so `architecture.model_dim` must be at least twice the number of columns. To use smaller shares, set positive widths for every column in `feature_embedding_dims`. For example, with two categorical columns, `feature_embedding_dims: {first: 1, second: 1}` permits one dimension each. -Explicit widths may sum to a value different from `architecture.dim_model`; +Explicit widths may sum to a value different from `architecture.model_dim`; the depth input projection handles the difference. The pooled projection handles -differences between `architecture.dim_model` and `output_dim`. +differences between `architecture.model_dim` and `output_dim`. CLS occupies position zero, and physical slot `s` occupies position `s+1`. Empty collections have a learnable CLS-only representation. Depth columns @@ -696,18 +1245,32 @@ greedily inside the decoder. Categorical and real targets can be mixed. decoder: type: autoregressive_transformer prediction_length: 1 - support: 1 - target_columns: [patch_length, byte_0, byte_1, magnitude] + target_columns: + - patch_length + - byte_0 + - byte_1 + - magnitude architecture: - dim_model: 256 + model_dim: 256 num_layers: 2 - attention: {type: mha, n_heads: 4} - feed_forward: {dim: 768, activation: swiglu} - normalization: {type: rmsnorm, norm_first: true} - position_encoding: {type: rope, theta: 10000} + attention: + type: mha + num_heads: 4 + feed_forward: + activation: swiglu + hidden_dim: 768 + normalization: + type: rmsnorm + norm_first: true + position_encoding: + type: rope + theta: 10000 dropout: 0.0 - shared_categorical_target_groups: [[byte_0, byte_1]] + shared_categorical_target_groups: + - - byte_0 + - byte_1 tie_input_output_embeddings: true + support_length: 1 ``` Shared groups must contain categorical targets with identical decoder-ID @@ -730,36 +1293,44 @@ now take effect; exact resume of those older runs is rejected. `initialization_seed` accepts integers from zero through `2**63-1`, excluding booleans. An explicit seed isolates constructor and custom-initializer randomness -from the run stream. Descendants derive seeds using SHA-256 over version 1, -namespace seed, relative branch path, and phase; explicit child seeds start a new -namespace. Omitted seeds preserve the existing RNG stream and initialization +from the run stream. Descendants derive seeds from the namespace seed, relative +branch path, and phase. Explicit child seeds start a new namespace. Omitted seeds preserve the existing RNG stream and initialization traversal. Initialization seeds do not control runtime dropout. Dropout continues using the run RNG and checkpointed per-rank state. -Dataset freezing can follow the same tree: +Trainability belongs to the phase for every algorithm. Both supervised and policy +roles use catalog selectors; supervised source-dependent updates use +`trainability.by_dataset..roles.model` with the same shape. ```yaml -dataset: - part: {metadata_config_path: configs/metadata_configs/raw-items.json} - criterion: {nextAction: CrossEntropyLoss} - freeze: - backbone: - freeze: [attention.qkv] - ingestion: - branches: - subitems: - freezing_except: [free_parameter, ingestion.output_projection] - item: - freeze: [] +training_plan: + phases: + - name: adapt + trainability: + roles: + model: + freeze: + - {component: backbone, semantic_groups: [attention.qkv]} + - {module: interfaces.default.ingestion.branches.subitems, + exclude_semantic_groups: [free_parameter, ingestion.output_projection]} ``` -A local child selector replaces the inherited decision, including selective -unfreezing. `freeze: []` unfreezes the scope; `freezing_except: []` freezes it. -A node containing only `branches` delegates without changing inherited decisions. -Merge parameters retain their parent's policy; the adapter has its own dataset -selector. Unknown branches, policies below leaves, and contradictory alias -selections fail. Freezing leaves dropout active and permits gradients through -frozen modules. At optimizer boundaries, frozen gradients are removed **before** -AMP unscaling/overflow detection, so a frozen-only overflow cannot suppress a -healthy active update. Momentum and weight decay cannot update parameters whose -gradients are absent. +Role overrides replace the phase policy for that dataset; absence inherits it. +`freeze: []` masks nothing and `freeze_except: []` masks everything. Selectors +are strict and shared with optimizer routing. Fixed model capabilities cannot be +unfrozen by a phase mask. Gradients still traverse masked modules; their gradients +are removed before AMP checks and stepping, preventing momentum/weight decay +updates. See [parameter selection](ownership.md#parameter-selection). + + +## Policy and mixed plans + +All algorithms use `training_plan.phases`. Declare policy sequence/preference +parts in `dataset_training`, and attach actions with `policy_bindings` keyed by interface. Run-wide settings remain in +`run` and `model.execution`; no alternate run discriminator is needed. + +See the complete [supervised template](../../src/sequifier/templates/train-supervised.yaml), +[policy template](../../src/sequifier/templates/train-policy.yaml), and +[supervised-to-policy template](../../src/sequifier/templates/train-supervised-to-policy.yaml). +The [policy guide](../policy-learning.md) describes algorithm budgets and +[the mixed-plan guide](../supervised-policy-learning.md) describes artifact handoffs. diff --git a/documentation/demos/self-contained-example.ipynb b/documentation/demos/self-contained-example.ipynb index 6d4f79491..30f6d2a73 100644 --- a/documentation/demos/self-contained-example.ipynb +++ b/documentation/demos/self-contained-example.ipynb @@ -10,10 +10,11 @@ "import os\n", "import shutil\n", "import subprocess\n", - "import yaml\n", + "\n", "import numpy as np\n", "import pandas as pd\n", - "import torch" + "import torch\n", + "import yaml" ] }, { @@ -94,17 +95,16 @@ "outputs": [], "source": [ "preprocess_config = {\n", - " \"project_root\": PROJECT_ROOT,\n", " \"preprocessing_data_path\": file_path,\n", " \"read_format\": \"parquet\",\n", " \"write_format\": \"parquet\",\n", " \"selected_columns\": [\"targetToken\"],\n", - " \"split_ratios\": [0.8, 0.1, 0.1], # Train, Val, Test\n", - " \"window_length\": 11, # 10-token model context plus one future target\n", + " \"split_ratios\": [0.8, 0.1, 0.1],\n", + " \"window_lengths\": [11, 11, 11],\n", " \"max_target_offset\": 1,\n", - " \"window_stride\": 1, # Dense sampling\n", + " \"window_stride\": 1,\n", " \"max_rows\": None,\n", - " \"seed\": 42,\n", + " \"run\": {\"project_root\": PROJECT_ROOT, \"reproducibility\": {\"seed\": 42}},\n", "}\n", "\n", "pp_conf_path = os.path.join(CONFIG_DIR, \"preprocess.yaml\")\n", @@ -119,57 +119,100 @@ "outputs": [], "source": [ "train_config = {\n", - " \"project_root\": PROJECT_ROOT,\n", - " \"model_name\": \"demo-cycle-model\",\n", - " \"device\": \"cuda\" if torch.cuda.is_available() else \"cpu\",\n", - " \"global_training\": {\n", - " \"read_format\": \"parquet\",\n", - " \"training_objective\": \"causal\",\n", - " \"context_length\": 10,\n", - " \"inference_batch_size\": 10,\n", - " \"batch_size\": 32,\n", - " \"learning_rate\": 0.001,\n", - " \"optimizer\": {\"name\": \"Adam\"},\n", - " \"scheduler\": {\"name\": \"StepLR\", \"step_size\": 1, \"gamma\": 0.99},\n", - " \"log_interval\": 10000,\n", - " \"save_interval_epochs\": 5,\n", - " \"torch_compile\": \"none\", # Keep the demo independent of a host C++ toolchain.\n", - " },\n", " \"model\": {\n", " \"backbone\": {\n", " \"architecture\": {\n", - " \"dim_model\": 16,\n", + " \"model_dim\": 16,\n", " \"max_context_length\": 10,\n", - " \"num_layers\": 1, # Tiny model for speed\n", - " \"attention\": {\"n_heads\": 2},\n", - " \"feed_forward\": {\"dim\": 32},\n", + " \"num_layers\": 1,\n", + " \"attention\": {\"num_heads\": 2},\n", + " \"feed_forward\": {\"hidden_dim\": 32},\n", " \"dropout\": 0.0,\n", - " },\n", + " }\n", " },\n", - " \"interface\": {\n", - " \"input_columns\": [\"targetToken\"],\n", - " \"target_columns\": [\"targetToken\"],\n", - " \"ingestion\": {\n", - " \"type\": \"embedding\",\n", - " \"output_dim\": 16,\n", - " \"feature_embedding_dims\": {\"targetToken\": 16},\n", - " },\n", - " \"decoder\": {\"type\": \"linear\", \"prediction_length\": 1},\n", + " \"interfaces\": {\n", + " \"default\": {\n", + " \"input_columns\": [\"targetToken\"],\n", + " \"target_columns\": [\"targetToken\"],\n", + " \"ingestion\": {\n", + " \"type\": \"embedding\",\n", + " \"output_dim\": 16,\n", + " \"feature_embedding_dims\": {\"targetToken\": 16},\n", + " },\n", + " \"decoder\": {\"type\": \"linear\", \"prediction_length\": 1},\n", + " }\n", " },\n", + " \"execution\": {\"context_length\": 10, \"training_objective\": \"causal\"},\n", " },\n", - " \"dataset\": {\n", - " # Metadata is generated by preprocessing from the input basename.\n", - " \"part\": {\n", - " \"metadata_config_path\": \"configs/metadata_configs/synthetic-sequences.json\"\n", + " \"dataset_training\": {\n", + " \"default\": {\n", + " \"parts\": {\n", + " \"default\": {\n", + " \"metadata_config_path\": \"configs/metadata_configs/synthetic-sequences.json\",\n", + " \"read_format\": \"parquet\",\n", + " }\n", + " },\n", + " \"split\": {\"validation_fraction\": 0.2},\n", + " }\n", + " },\n", + " \"training_plan\": {\n", + " \"phases\": [\n", + " {\n", + " \"name\": \"supervised\",\n", + " \"algorithm\": \"supervised\",\n", + " \"epochs\": 5,\n", + " \"mode\": \"sequential\",\n", + " \"sources\": [{\"source\": \"default.default\"}],\n", + " \"batch_size\": 32,\n", + " \"optimization\": {\n", + " \"model\": {\n", + " \"learning_rate\": 0.001,\n", + " \"optimizer\": {\"name\": \"Adam\", \"arguments\": {}},\n", + " \"scheduler\": {\n", + " \"name\": \"StepLR\",\n", + " \"arguments\": {\"step_size\": 1, \"gamma\": 0.99},\n", + " },\n", + " }\n", + " },\n", + " \"evaluation\": {\n", + " \"sources\": [{\"source\": \"default.default\"}],\n", + " \"cadence\": {\"epochs\": 1},\n", + " \"batch_size\": 32,\n", + " \"before_phase\": True,\n", + " \"on_snapshot\": True,\n", + " },\n", + " \"model_selection\": {\"handoff\": \"last\"},\n", + " \"checkpoints\": {\"cadence\": {\"epochs\": 5}},\n", + " \"learning\": {\n", + " \"datasets\": {\n", + " \"default\": {\"criterion\": {\"targetToken\": \"CrossEntropyLoss\"}}\n", + " }\n", + " },\n", + " }\n", + " ]\n", + " },\n", + " \"run\": {\n", + " \"project_root\": PROJECT_ROOT,\n", + " \"model_name\": \"demo-cycle-model\",\n", + " \"reproducibility\": {\"seed\": 1010, \"enforce_determinism\": False},\n", + " \"execution\": {\n", + " \"device\": \"cuda\" if torch.cuda.is_available() else \"cpu\",\n", + " \"distribution\": {\"strategy\": \"local\", \"world_size\": 1, \"backend\": \"nccl\"},\n", + " \"precision\": {},\n", + " \"data_loading\": {},\n", + " \"compilation\": {\"torch_compile\": \"none\"},\n", + " },\n", + " \"recovery\": {},\n", + " \"logging\": {\"log_interval_batches\": 10000},\n", + " \"outputs\": {},\n", + " \"export\": {\n", + " \"generative_model\": True,\n", + " \"embedding_model\": False,\n", + " \"onnx\": True,\n", + " \"pt\": False,\n", + " \"batch_size\": 10,\n", " },\n", - " \"criterion\": {\"targetToken\": \"CrossEntropyLoss\"},\n", " },\n", - " \"training_plan\": {\"epochs\": 5},\n", - " \"evaluation\": True,\n", - " \"export_generative_model\": True,\n", - " \"export_embedding_model\": False,\n", - " \"export_onnx\": True,\n", - " \"export_pt\": False,\n", "}\n", "\n", "tr_conf_path = os.path.join(CONFIG_DIR, \"train.yaml\")\n", @@ -184,21 +227,20 @@ "outputs": [], "source": [ "infer_config = {\n", - " \"project_root\": PROJECT_ROOT,\n", - " \"model_type\": \"generative\",\n", - " # Training exports best and last artifacts with the final epoch in the filename.\n", - " \"model_path\": \"models/demo-cycle-model-best-5.onnx\",\n", + " \"task\": \"forecast\",\n", + " \"model_paths\": [\"models/demo-cycle-model-best-5.onnx\"],\n", " \"training_config_path\": tr_conf_path,\n", " \"dataset\": \"default\",\n", - " # Use the 'test' split created during preprocessing\n", " \"data_path\": \"data/synthetic-sequences-split2.parquet\",\n", " \"write_format\": \"parquet\",\n", " \"output_probabilities\": False,\n", - " \"decode_categories\": True, # Map integer classes back to original values (e.g., \"1\", \"2\")\n", - " \"device\": \"cpu\",\n", - " \"inference_batch_size\": 10,\n", - " \"autoregressive\": True,\n", - " \"generation_steps\": 5, # Predict 5 steps into the future\n", + " \"decode_categories\": True,\n", + " \"autoregressive_generation\": True,\n", + " \"generation_steps\": 5,\n", + " \"run\": {\n", + " \"project_root\": PROJECT_ROOT,\n", + " \"execution\": {\"device\": \"cpu\", \"batch_size\": 10},\n", + " },\n", "}\n", "\n", "inf_conf_path = os.path.join(CONFIG_DIR, \"infer.yaml\")\n", @@ -286,7 +328,7 @@ " sequence_id_change | correct_increment | (last_was_five & is_one)\n", ").mean()\n", "\n", - "print(f\"{mean_accuracy*100:.2f}% of generated tokens follow the cycle rule\")" + "print(f\"{mean_accuracy * 100:.2f}% of generated tokens follow the cycle rule\")" ] }, { diff --git a/documentation/example_inputs/configs/ddconfigs/preprocessing_input.json b/documentation/example_inputs/configs/ddconfigs/preprocessing_input.json index aa546b6d8..1c5f892e7 100644 --- a/documentation/example_inputs/configs/ddconfigs/preprocessing_input.json +++ b/documentation/example_inputs/configs/ddconfigs/preprocessing_input.json @@ -63,5 +63,7 @@ }, "window_length": 11, "max_target_offset": 1, - "stored_window_layout_version": 2 + "tensor_payload_kind": "tabular", + "tensor_payload_version": 100, + "stored_window_layout_version": 100 } diff --git a/documentation/example_inputs/configs/import-policy-sequences.yaml b/documentation/example_inputs/configs/import-policy-sequences.yaml new file mode 100644 index 000000000..3d1b10172 --- /dev/null +++ b/documentation/example_inputs/configs/import-policy-sequences.yaml @@ -0,0 +1,19 @@ +operation: import_sequences +source: + paths: + - data/demonstrations.csv + columns: + observations: + - position + - velocity +output_path: data/imported-demonstrations +episodes_per_shard: 64 +policy_binding: + actions: + action: + type: categorical + values: + - left + - right +run: + project_root: . diff --git a/documentation/example_inputs/configs/infer-policy.yaml b/documentation/example_inputs/configs/infer-policy.yaml new file mode 100644 index 000000000..acadcb109 --- /dev/null +++ b/documentation/example_inputs/configs/infer-policy.yaml @@ -0,0 +1,14 @@ +data_path: data/policy-observations.csv +read_format: csv +write_format: csv +desired_return: 10.0 +action_selection: deterministic +task: policy +model_paths: +- models/policy.pt +run: + project_root: . + execution: + device: cpu + reproducibility: + enforce_determinism: true diff --git a/documentation/example_inputs/configs/infer-ppo-toy.yaml b/documentation/example_inputs/configs/infer-ppo-toy.yaml new file mode 100644 index 000000000..78d19a967 --- /dev/null +++ b/documentation/example_inputs/configs/infer-ppo-toy.yaml @@ -0,0 +1,14 @@ +data_path: documentation/example_inputs/ppo-observations.csv +read_format: csv +write_format: csv +output_path: outputs/ppo-actions.csv +action_selection: deterministic +task: policy +model_paths: +- models/online-ppo.pt +run: + project_root: . + execution: + device: cpu + reproducibility: + enforce_determinism: true diff --git a/documentation/example_inputs/configs/rollout-ppo-toy.yaml b/documentation/example_inputs/configs/rollout-ppo-toy.yaml new file mode 100644 index 000000000..04fae9688 --- /dev/null +++ b/documentation/example_inputs/configs/rollout-ppo-toy.yaml @@ -0,0 +1,22 @@ +data_path: documentation/example_inputs/ppo-observations.csv +read_format: csv +write_format: csv +rollout: + trajectory_path: outputs/ppo-rollout.json + seed_row_index: 0 + max_episode_steps: 8 + max_steps: 8 + environment: + factory: sequifier.rl.control:CategoricalControlAdapter + config: + horizon: 8 +action_selection: deterministic +task: policy +model_paths: +- models/online-ppo.pt +run: + project_root: . + execution: + device: cpu + reproducibility: + enforce_determinism: true diff --git a/documentation/example_inputs/configs/train-bc-toy.yaml b/documentation/example_inputs/configs/train-bc-toy.yaml new file mode 100644 index 000000000..50a2ddc30 --- /dev/null +++ b/documentation/example_inputs/configs/train-bc-toy.yaml @@ -0,0 +1,86 @@ +model: + backbone: + architecture: + model_dim: 32 + num_layers: 2 + max_context_length: 16 + attention: + num_heads: 4 + feed_forward: + hidden_dim: 128 + position_encoding_scope: global + interfaces: + actor: + input_columns: + - position + - __policy_previous_action + target_columns: + - action + ingestion: + type: embedding + output_dim: 8 + feature_embedding_dims: + position: 2 + __policy_previous_action: 6 + decoder: + type: linear + prediction_length: 1 + support_length: 1 + categorical_decoder_special_tokens: + action: [] + execution: + context_length: 16 +dataset_training: + demonstrations: + parts: + recorded: + type: sequence + paths: + - data/offline-transitions.csv + model_interface: actor + split: + validation_fraction: 0.2 +training_plan: + phases: + - name: behavior_cloning + algorithm: policy_imitation + sources: + - source: demonstrations.recorded + epochs: 30 + batch_size: 64 + model_selection: + handoff: last + checkpoints: + cadence: + iterations: 1 + evaluation: + cadence: + iterations: 1 + batch_size: 64 + before_phase: false + on_snapshot: false +policy_bindings: + actor: + previous_action_inputs: + __policy_previous_action: action + initial_inputs: + __policy_previous_action: start + return_conditioned: false + actions: + action: + type: categorical + values: + - left + - right +run: + project_root: . + model_name: toy-bc + reproducibility: + seed: 1010 + enforce_determinism: false + execution: + device: cpu + recovery: {} + logging: {} + outputs: {} + export: {} diff --git a/documentation/example_inputs/configs/train-iql-from-table.yaml b/documentation/example_inputs/configs/train-iql-from-table.yaml new file mode 100644 index 000000000..55829ba6f --- /dev/null +++ b/documentation/example_inputs/configs/train-iql-from-table.yaml @@ -0,0 +1,95 @@ +model: + backbone: + architecture: + model_dim: 32 + num_layers: 2 + max_context_length: 16 + attention: + num_heads: 4 + feed_forward: + hidden_dim: 128 + position_encoding_scope: global + interfaces: + actor: + input_columns: + - position + - velocity + - __policy_previous_action + target_columns: + - action + ingestion: + type: embedding + output_dim: 8 + feature_embedding_dims: + position: 1 + velocity: 1 + __policy_previous_action: 6 + decoder: + type: linear + prediction_length: 1 + support_length: 1 + categorical_decoder_special_tokens: + action: [] + execution: + context_length: 16 +dataset_training: + demonstrations: + parts: + recorded: + type: sequence + paths: + - data/offline-transitions.parquet + model_interface: actor + split: + validation_fraction: 0.2 +training_plan: + phases: + - name: offline_improvement + algorithm: iql + sources: + - source: demonstrations.recorded + batch_size: 64 + gamma: 0.99 + expectile: 0.7 + inverse_temperature: 3.0 + max_advantage_weight: 100.0 + target_update_rate: 0.005 + iterations: 10000 + model_selection: + handoff: last + checkpoints: + cadence: + iterations: 100 + evaluation: + cadence: + iterations: 100 + batch_size: 64 + before_phase: false + on_snapshot: false +policy_bindings: + actor: + previous_action_inputs: + __policy_previous_action: action + initial_inputs: + __policy_previous_action: start + return_conditioned: false + actions: + action: + type: categorical + values: + - left + - right +run: + project_root: . + model_name: offline-iql + reproducibility: + seed: 1010 + enforce_determinism: false + execution: + device: cpu + recovery: + resume: + policy: if_exists + logging: {} + outputs: {} + export: {} diff --git a/documentation/example_inputs/configs/train-iql-toy.yaml b/documentation/example_inputs/configs/train-iql-toy.yaml new file mode 100644 index 000000000..47e5c4e69 --- /dev/null +++ b/documentation/example_inputs/configs/train-iql-toy.yaml @@ -0,0 +1,89 @@ +initialize_from: models/toy-bc/actor.pt +model: + backbone: + architecture: + model_dim: 32 + num_layers: 2 + max_context_length: 16 + attention: + num_heads: 4 + feed_forward: + hidden_dim: 128 + position_encoding_scope: global + interfaces: + actor: + input_columns: + - position + - __policy_previous_action + target_columns: + - action + ingestion: + type: embedding + output_dim: 8 + feature_embedding_dims: + position: 2 + __policy_previous_action: 6 + decoder: + type: linear + prediction_length: 1 + support_length: 1 + categorical_decoder_special_tokens: + action: [] + execution: + context_length: 16 +dataset_training: + demonstrations: + parts: + recorded: + type: sequence + paths: + - data/offline-transitions.csv + model_interface: actor + split: + validation_fraction: 0.2 +training_plan: + phases: + - name: offline_improvement + algorithm: iql + sources: + - source: demonstrations.recorded + batch_size: 64 + iterations: 10000 + model_selection: + handoff: last + checkpoints: + cadence: + iterations: 100 + evaluation: + cadence: + iterations: 100 + batch_size: 64 + before_phase: false + on_snapshot: false +policy_bindings: + actor: + previous_action_inputs: + __policy_previous_action: action + initial_inputs: + __policy_previous_action: start + return_conditioned: false + actions: + action: + type: categorical + values: + - left + - right +run: + project_root: . + model_name: toy-iql + reproducibility: + seed: 1010 + enforce_determinism: false + execution: + device: cpu + recovery: + resume: + policy: if_exists + logging: {} + outputs: {} + export: {} diff --git a/documentation/example_inputs/configs/train-multiphase-control.yaml b/documentation/example_inputs/configs/train-multiphase-control.yaml new file mode 100644 index 000000000..e3f82d3b8 --- /dev/null +++ b/documentation/example_inputs/configs/train-multiphase-control.yaml @@ -0,0 +1,200 @@ +model: + backbone: + architecture: + model_dim: 32 + num_layers: 1 + max_context_length: 4 + attention: + num_heads: 4 + feed_forward: + hidden_dim: 64 + position_encoding_scope: global + dropout: 0.0 + interfaces: + actor: + input_columns: + - cue + - __policy_previous_action + target_columns: + - action + ingestion: + type: embedding + output_dim: 8 + feature_embedding_dims: + cue: 2 + __policy_previous_action: 6 + decoder: + type: linear + prediction_length: 1 + support_length: 1 + categorical_decoder_special_tokens: + action: [] + input_contract: + normalization: + source: identity + execution: + context_length: 4 +dataset_training: + decisions: + parts: + default: + type: sequence + paths: + - data/control-decisions.csv + model_interface: actor + split: + validation_fraction: 0.2 + rankings: + parts: + default: + type: preferences + manifest_path: data/control-preferences.json + split: + validation_fraction: 0.2 +training_plan: + phases: + - name: imitation + algorithm: policy_imitation + sources: + - source: decisions.default + epochs: 4 + model_selection: + handoff: last + checkpoints: + cadence: + iterations: 1 + evaluation: + cadence: + iterations: 1 + batch_size: 32 + before_phase: false + on_snapshot: false + - name: offline_improvement + algorithm: iql + sources: + - source: decisions.default + iterations: 100 + model_selection: + handoff: last + checkpoints: + cadence: + iterations: 100 + evaluation: + cadence: + iterations: 100 + batch_size: 64 + before_phase: false + on_snapshot: false + - name: preferences + algorithm: dpo + preferences: + source: rankings.default + beta: 0.1 + iterations: 100 + model_selection: + handoff: last + checkpoints: + cadence: + iterations: 1 + evaluation: + cadence: + iterations: 1 + batch_size: 1 + before_phase: false + on_snapshot: false + - name: group_improvement + algorithm: grpo + total_groups: 32 + groups_per_iteration: 2 + group_size: 4 + max_steps: 8 + environment: + factory: sequifier.rl.control:CategoricalControlAdapter + config: + horizon: 8 + evaluation: + environment: + factory: sequifier.rl.control:CategoricalControlAdapter + config: + horizon: 8 + seeds: + - 17 + - 29 + - 43 + - 71 + max_episode_steps: 8 + timeout_seconds: 30 + monitor: + metric: score_mean + direction: max + cadence: + iterations: 1 + batch_size: 1 + before_phase: false + on_snapshot: false + model_selection: + handoff: last + checkpoints: + cadence: + iterations: 1 + - name: ppo_improvement + algorithm: ppo + environment: + factory: sequifier.rl.control:CategoricalControlAdapter + config: + horizon: 8 + total_env_steps: 8192 + rollout_env_steps: 256 + optimization_epochs: 4 + accumulation_batches: 1 + gamma: 0.99 + gae_lambda: 0.95 + clip_epsilon: 0.2 + evaluation: + environment: + factory: sequifier.rl.control:CategoricalControlAdapter + config: + horizon: 8 + seeds: + - 17 + - 29 + - 43 + - 71 + max_episode_steps: 8 + timeout_seconds: 30 + cadence: + iterations: 2 + batch_size: 1 + before_phase: false + on_snapshot: false + batch_size: 64 + model_selection: + handoff: last + checkpoints: + cadence: + iterations: 2 +policy_bindings: + actor: + previous_action_inputs: + __policy_previous_action: action + initial_inputs: + __policy_previous_action: start + return_conditioned: false + actions: + action: + type: categorical + values: + - left + - right +run: + project_root: . + model_name: multiphase-control + reproducibility: + seed: 1010 + enforce_determinism: false + execution: + device: cpu + recovery: {} + logging: {} + outputs: {} + export: {} diff --git a/documentation/example_inputs/configs/train-policy-from-table.yaml b/documentation/example_inputs/configs/train-policy-from-table.yaml new file mode 100644 index 000000000..adf8198a7 --- /dev/null +++ b/documentation/example_inputs/configs/train-policy-from-table.yaml @@ -0,0 +1,91 @@ +model: + backbone: + architecture: + model_dim: 32 + num_layers: 2 + max_context_length: 16 + attention: + num_heads: 4 + feed_forward: + hidden_dim: 128 + position_encoding_scope: global + interfaces: + actor: + input_columns: + - position + - velocity + - __policy_previous_action + target_columns: + - action + ingestion: + type: embedding + output_dim: 8 + feature_embedding_dims: + position: 1 + velocity: 1 + __policy_previous_action: 6 + decoder: + type: linear + prediction_length: 1 + support_length: 1 + categorical_decoder_special_tokens: + action: [] + execution: + context_length: 16 +dataset_training: + demonstrations: + parts: + recorded: + type: sequence + paths: + - data/demonstrations.parquet + model_interface: actor + split: + validation_fraction: 0.2 +training_plan: + phases: + - name: behavior_cloning + algorithm: policy_imitation + epochs: 80 + sources: + - source: demonstrations.recorded + batch_size: 64 + optimization: + actor: + learning_rate: 0.005 + model_selection: + handoff: last + checkpoints: + cadence: + iterations: 1 + evaluation: + cadence: + iterations: 1 + batch_size: 64 + before_phase: false + on_snapshot: false +policy_bindings: + actor: + previous_action_inputs: + __policy_previous_action: action + initial_inputs: + __policy_previous_action: start + return_conditioned: false + actions: + action: + type: categorical + values: + - left + - right +run: + project_root: . + model_name: table-policy + reproducibility: + seed: 1010 + enforce_determinism: false + execution: + device: cpu + recovery: {} + logging: {} + outputs: {} + export: {} diff --git a/documentation/example_inputs/configs/train-return-conditioned.yaml b/documentation/example_inputs/configs/train-return-conditioned.yaml new file mode 100644 index 000000000..5106f2244 --- /dev/null +++ b/documentation/example_inputs/configs/train-return-conditioned.yaml @@ -0,0 +1,102 @@ +model: + backbone: + architecture: + model_dim: 32 + num_layers: 2 + max_context_length: 4 + attention: + num_heads: 4 + feed_forward: + activation: swiglu + hidden_dim: 128 + position_encoding_scope: global + dropout: 0.0 + interfaces: + actor: + input_columns: + - OBSERVATION_COLUMN + - __policy_previous_action + - __policy_return + target_columns: + - action + ingestion: + type: embedding + output_dim: 8 + feature_embedding_dims: + OBSERVATION_COLUMN: 1 + __policy_previous_action: 6 + __policy_return: 1 + decoder: + type: linear + prediction_length: 1 + support_length: 1 + categorical_decoder_special_tokens: + action: [] + execution: + context_length: 4 +dataset_training: + demonstrations: + parts: + recorded: + type: sequence + paths: + - data/demonstrations.parquet + model_interface: actor + split: + validation_fraction: 0.2 +training_plan: + phases: + - name: offline_policy + algorithm: policy_imitation + epochs: 80 + sources: + - source: demonstrations.recorded + batch_size: 64 + optimization: + actor: + learning_rate: 0.005 + model_selection: + handoff: last + checkpoints: + cadence: + iterations: 1 + evaluation: + cadence: + iterations: 1 + batch_size: 64 + before_phase: false + on_snapshot: false +policy_bindings: + actor: + previous_action_inputs: + __policy_previous_action: action + initial_inputs: + __policy_previous_action: start + return_conditioned: true + returns: + gamma: 1.0 + scale: 1.0 + actions: + action: + type: categorical + values: + - 0.0 + - 1.0 +run: + project_root: . + model_name: policy + reproducibility: + seed: 1010 + enforce_determinism: false + execution: + device: cpu + data_loading: + episode_cache_size: 8 + recovery: + resume: + policy: never + logging: {} + outputs: + model_path: models/policy/actor.pt + report_path: outputs/policy/training-report.json + export: {} diff --git a/documentation/example_inputs/iql_toy_control.py b/documentation/example_inputs/iql_toy_control.py new file mode 100644 index 000000000..dcd838cc3 --- /dev/null +++ b/documentation/example_inputs/iql_toy_control.py @@ -0,0 +1,168 @@ +"""Small offline control dataset and separate BC/IQL rollout comparison. + +Generate data with ``python iql_toy_control.py generate data/offline-transitions.csv``. +After training both actors, run +``python iql_toy_control.py evaluate path/to/bc.pt path/to/iql.pt``. +This is an example benchmark, not part of the test suite. +""" + +import csv +import random +import statistics +import sys +from pathlib import Path + +from sequifier.config.policy_infer_config import PolicyRolloutConfig +from sequifier.rl_api import ( + EnvironmentCapabilities, + OnlineEnvironmentAdapterBase, + OnlineResetResult, + OnlineStepResult, + collect_policy_rollout, + load_policy, +) +from sequifier.trajectory.schema import SeedHistory + +HORIZON = 12 + + +class ToyControlAdapter(OnlineEnvironmentAdapterBase): + implementation_id = "sequifier.toy-control" + capabilities = EnvironmentCapabilities(seeded_reset=True) + + def reset(self, seed): + initial = OnlineResetResult( + {"position": float(random.Random(seed).choice((-4, -3, 3, 4)))} + ) + return self.reset_from_observation(initial, seed) + + def reset_from_observation(self, initial, seed): + self.position = int(initial.observation["position"]) + self.steps = 0 + return initial + + def step(self, action): + self.position += -1 if action["action"] == "left" else 1 + self.steps += 1 + done = self.position == 0 + return OnlineStepResult( + observation={"position": float(self.position)}, + reward=1.0 if done else -0.05, + terminated=done, + truncated=self.steps == HORIZON and not done, + ) + + def close(self): + pass + + def snapshot_state(self): + raise NotImplementedError("snapshot capability is not supported") + + def restore_state(self, state): + raise NotImplementedError("snapshot capability is not supported") + + def create_group_start(self, seed): + raise NotImplementedError("group-start capability is not supported") + + def reset_group(self, token, rollout_seed): + raise NotImplementedError("group-start capability is not supported") + + +def generate(path): + rng = random.Random(1010) + destination = Path(path) + destination.parent.mkdir(parents=True, exist_ok=True) + with destination.open("w", newline="", encoding="utf-8") as stream: + writer = csv.DictWriter( + stream, + fieldnames=[ + "sequenceId", + "itemPosition", + "position", + "action", + "reward", + "terminated", + "truncated", + "bootstrap_allowed", + "__next_position", + ], + ) + writer.writeheader() + for episode in range(500): + position = rng.choice((-4, -3, 3, 4)) + quality = 0.85 if episode % 2 else 0.25 + for step in range(HORIZON): + good = "left" if position > 0 else "right" + action = ( + good + if rng.random() < quality + else ("right" if good == "left" else "left") + ) + successor = position + (-1 if action == "left" else 1) + terminated = successor == 0 + truncated = step + 1 == HORIZON and not terminated + writer.writerow( + { + "sequenceId": episode, + "itemPosition": step, + "position": float(position), + "action": action, + "reward": 1.0 if terminated else -0.05, + "terminated": terminated, + "truncated": truncated, + "bootstrap_allowed": not terminated, + "__next_position": float(successor), + } + ) + position = successor + if terminated or truncated: + break + + +def evaluate(paths): + config = PolicyRolloutConfig.model_validate( + { + "environment": { + "factory": "iql_toy_control:ToyControlAdapter", + "units": {}, + }, + "max_steps": HORIZON, + "trajectory_path": "unused.json", + "seed_row_index": 0, + "max_episode_steps": HORIZON, + } + ) + for path in paths: + policy = load_policy(path) + returns = [] + for episode, position in enumerate((-4, -3, 3, 4) * 25): + seed = SeedHistory( + observations={ + "position": [0.0] * (policy.config.context_length - 1) + + [float(position)] + }, + attention_valid_mask=[False] * (policy.config.context_length - 1) + + [True], + static_inputs={}, + ) + trajectory = collect_policy_rollout( + policy, + ToyControlAdapter(), + seed, + config, + HORIZON, + random_seed=episode, + ) + returns.append(trajectory.total_return) + print(path, "mean", statistics.mean(returns), "std", statistics.stdev(returns)) + + +if __name__ == "__main__": + if len(sys.argv) == 3 and sys.argv[1] == "generate": + generate(sys.argv[2]) + elif len(sys.argv) >= 4 and sys.argv[1] == "evaluate": + evaluate(sys.argv[2:]) + else: + raise SystemExit( + "usage: iql_toy_control.py generate FILE | evaluate BC_MODEL IQL_MODEL" + ) diff --git a/documentation/example_inputs/policy_control_data.py b/documentation/example_inputs/policy_control_data.py new file mode 100644 index 000000000..392005652 --- /dev/null +++ b/documentation/example_inputs/policy_control_data.py @@ -0,0 +1,92 @@ +"""Generate curated sequence preferences for the multi-phase control examples. + +Usage: python documentation/example_inputs/policy_control_data.py data +Preferences are supplied by a sign-matching rule, independent of reward sums. +This script only creates example data; it does not train or evaluate a model. +""" + +import csv +import json +import sys +from pathlib import Path + + +def generate(directory): + directory = Path(directory) + directory.mkdir(parents=True, exist_ok=True) + pairs = [] + with (directory / "control-decisions.csv").open( + "w", newline="", encoding="utf-8" + ) as handle: + fields = [ + "sequenceId", + "episodeId", + "__group", + "itemPosition", + "cue", + "action", + "reward", + "terminated", + "truncated", + "bootstrap_allowed", + "__next_cue", + ] + writer = csv.DictWriter(handle, fieldnames=fields) + writer.writeheader() + for context in range(40): + initial = (-1 if context % 2 else 1) * (1 + context / 100) + alternatives = {} + for kind, length in (("chosen", 8), ("rejected", 6)): + identity = f"context-{context}-{kind}" + cue = initial + for step in range(length): + matched = "right" if cue > 0 else "left" + action = ( + matched + if kind == "chosen" + else ("left" if matched == "right" else "right") + ) + writer.writerow( + { + "sequenceId": identity, + "episodeId": identity, + "__group": f"context-{context}", + "itemPosition": step, + "cue": cue, + "action": action, + "reward": 1 if action == matched else -1, + "terminated": step + 1 == 8, + "truncated": step + 1 == length and length != 8, + "bootstrap_allowed": step + 1 != 8, + "__next_cue": -cue, + } + ) + cue = -cue + alternatives[kind] = { + "source": "decisions.default", + "episode_id": identity, + "start": 0, + "end": length, + } + pairs.append( + { + "pair_id": f"pair-{context}", + "context_id": f"context-{context}", + **alternatives, + } + ) + (directory / "control-preferences.json").write_text( + json.dumps( + { + "format": "sequifier.preference-pairs", + "pairs": pairs, + }, + indent=2, + ) + + "\n", + encoding="utf-8", + ) + + +if __name__ == "__main__": + generate(sys.argv[1] if len(sys.argv) > 1 else "data") diff --git a/documentation/example_inputs/ppo-observations.csv b/documentation/example_inputs/ppo-observations.csv new file mode 100644 index 000000000..28a1a3cfc --- /dev/null +++ b/documentation/example_inputs/ppo-observations.csv @@ -0,0 +1,3 @@ +cue +-1.0 +1.0 diff --git a/documentation/example_inputs/ppo_toy_control.py b/documentation/example_inputs/ppo_toy_control.py new file mode 100644 index 000000000..a65d3e463 --- /dev/null +++ b/documentation/example_inputs/ppo_toy_control.py @@ -0,0 +1,21 @@ +"""Train and evaluate the shipped PPO example through the common Python API.""" + +from importlib.resources import files + +from sequifier.config.train_config import load_train_config +from sequifier.rl_api import evaluate_policy, load_policy +from sequifier.train import run_training + + +def main(): + config = load_train_config( + str(files("sequifier").joinpath("templates", "train-ppo.yaml")), {}, False + ) + result = run_training(config) + actor = load_policy(result.model_path) + print("Selected return:", result.phases[-1].selected_metric) + print(evaluate_policy(actor, config.training_plan[0].evaluation)) + + +if __name__ == "__main__": + main() diff --git a/documentation/learning-architecture.md b/documentation/learning-architecture.md new file mode 100644 index 000000000..8d8a19a54 --- /dev/null +++ b/documentation/learning-architecture.md @@ -0,0 +1,39 @@ +# Learning architecture + +All learning uses `SequifierConfig` and `training_plan.phases`; there is no second +policy architecture or run discriminator. See the [training reference](configs/train.md), +[policy guide](policy-learning.md), [artifact transfer guide](supervised-policy-learning.md), +and [inference reference](configs/infer.md) for the supported configuration surface. + +| Owner | Contract | +| --- | --- | +| Model execution | Named interfaces, forward attention, context geometry, codecs, normalization, input schemas, decoder order and prediction scales. These survive changes of learning objective. | +| Authored run | Model or initialization artifact, datasets, policy bindings, ordered phases, output paths and run execution settings. | +| Phase | Algorithm budget, batching, trainable roles, optimizer/scheduler settings, evaluation, trainability, stopping and model-selection handoff. | +| Sources | Prepared histories, raw sequences and preference manifests with explicit timing, episode boundaries and optional transition capabilities. | +| Native runner | Algorithm-specific sampling, objectives, critics, environment state and in-phase recovery. | +| Coordinator | Dependency validation, incoming/selected artifacts, phase boundaries, receipts and reports. | + +`ModelExecution` exposes execution facts without assigning them dataset ownership. +Resolution constructs the execution objects used by model loading and native runners. +Policy bindings select existing targets; lagged actions and return conditioning are +explicit inputs. Categorical support, real scales/bounds/atoms and conditional activity +have the same meaning during training, collection, replay and inference. + +An internal preparation owner retains resolved source identities, episode catalogs +and partitions from metadata fitting through preflight and execution. Feature codecs +and execution schemas refresh dataset views without selecting the partitions again; +changes to episode interpretation or split settings create new views. Dependencies +are still revalidated before execution and at phase boundaries. This preparation +state is private and does not enter configuration serialization or checkpoints. + +Historical decisions are reconstructed at their actual observation boundary. +Termination disables bootstrap; truncation preserves the true final successor; +a collection cutoff does not invent either boundary. Training/validation splits +keep connected episode, group and duplicate-content components together. + +Phase handoffs publish selected weights before committing progress. Checkpoints +retain optimizer/sampler/RNG state and validate representation, source, adapter and +reference identities. Mixed plans use native checkpoints inside an atomic run +envelope. Existing checkpoint formats, public configuration aliases, standalone +scaffold templates and portable model formats describe the supported contracts. diff --git a/documentation/policy-learning.md b/documentation/policy-learning.md new file mode 100644 index 000000000..603c79be2 --- /dev/null +++ b/documentation/policy-learning.md @@ -0,0 +1,286 @@ +# Policy learning + +Use `sequifier train --config-path configs/train-policy.yaml`. All algorithms use +`training_plan.phases`; model interfaces and datasets follow the +[training reference](configs/train.md). Start from the shipped +[imitation](../src/sequifier/templates/train-policy.yaml), +[IQL](../src/sequifier/templates/train-iql.yaml), [PPO](../src/sequifier/templates/train-ppo.yaml), +[DPO](../src/sequifier/templates/train-dpo.yaml), or [GRPO](../src/sequifier/templates/train-grpo.yaml) +template. [Artifact transfer](supervised-policy-learning.md) covers joint real/categorical +actions and mixed supervised/policy plans. + +## Authoring and optimization + +`policy_bindings` maps interface names to action declarations, including runs +with one interface. `model.execution` owns context geometry; interface input +contracts own frozen normalization; datasets own raw split policy. `run` owns +resources, reproducibility and recovery. Phases own learning, trainability, +optimization, evaluation, stopping, snapshots and `model_selection.handoff`. +See the [ownership reference](configs/ownership.md). + +| Algorithm | Required budget/data | Trainable roles | +| --- | --- | --- | +| `policy_imitation` | `epochs`, `sources` | `actor` | +| `iql` | `iterations`, transition `sources` | `actor`, `value`, `critic` | +| `dpo` | `iterations`, `preferences` | `actor` | +| `ppo` | `total_env_steps`, `rollout_env_steps`, `environment`, `evaluation` | `actor`, `value` | +| `grpo` | `total_groups`, `max_steps`, `environment`, `evaluation` | `actor` | + +Each role defaults to Adam, learning rate `0.0003`, gradient clip `1.0`, and +`ConstantLR` with factor `1.0`, total iterations `1`. Optimizers accept constructor +arguments; grouped optimizer plans set their own rates. Policy schedulers advance +on optimizer steps; plateau/compound schedulers are unsupported. `OneCycleLR.total_steps` is derived from the phase budget; +an explicitly supplied value must match that budget. Unknown roles fail validation. + +Imitation, DPO and GRPO default to `batch_size: 32`, `accumulation_batches: 1`, +`evaluation.cadence: {iterations: 1}` and `checkpoints.cadence: {iterations: 1}`. Accumulation uses actual +example counts, including partial batches. Network stochastic layers are disabled +while gradients remain enabled. Best handoff keeps the earliest finite best on ties; +last handoff keeps the latest transaction. Final updates always evaluate. +See [continuation rules](configs/train.md#optimization-across-phases) for +`optimizer_state: continue`; its default is `reset`. + +## Offline data + +Sequence parts accept CSV/Parquet tables, JSON trajectories, checksummed sequence +manifests and prepared histories. An authored manifest that lists trajectory +files declares `format: sequifier.trajectory-source` and an `episodes` +list with a `path` per entry. At row `t`, observations and lagged actions form +inputs; the recorded current action is the label, and reward/successor describe +its resulting transition. Tables use `sequenceId` and `itemPosition`. +Observations and static inputs use the same names in the table, metadata and model. +When importing without a model, declare those names in the +`source.columns.observations` and `source.columns.static_inputs` lists. + +Current actions use their policy field names or a structured `__executed_action` +column. Previous actions use `__previous_action_` scalar columns or a +structured `__previous_action` column. Optional transition columns are `reward`, +`terminated`, `truncated`, and `bootstrap_allowed`. When successors are present, +provide `__next_` for every observation. Use `episodeId` for a +stable episode ID and `__group` for a collection group. +Rename external table columns to these names before importing or training. +Fragments require an explicit preceding action; file boundaries alone imply no +termination. Forecast trajectories require `action_semantics: executed_predictions`. + +Automatic splitting keeps transitive episode/group/duplicate-content components +together and requires at least two components. Explicit `evaluation.sources` +undergo the same overlap checks. Codecs and moments are fitted only on training +coverage. Indexed storage and bounded episode caches are shared across phases; +windows are constructed on demand. `import_sequences` exports canonical raw +Parquet shards and a checksummed manifest; see the +[import example](example_inputs/configs/import-policy-sequences.yaml). + +Imitation minimizes joint action NLL. Return conditioning requires rewards and +a known tail: true termination or an explicit finite return horizon. The label at +`t` includes `r_t`; unknown tails are excluded. Offline monitors are fixed to +`{metric: validation_action_nll, direction: min}` for imitation/IQL and +`{metric: validation_dpo_loss, direction: min}` for DPO. Only +`evaluation.sources` and `evaluation.monitor` are accepted; alternate spellings +are rejected. + +IQL requires complete reward, terminated, truncated and bootstrap columns, plus +true successors for bootstrapped transitions. It samples uniformly with replacement, +fits expectile V and twin Q-functions, updates target Q-functions with Polyak +averaging, and weights actor NLL by +`min(exp(inverse_temperature * (min(target_Q1,target_Q2)-V)), max_advantage_weight)`. +Defaults: `batch_size: 64`, `gamma: 0.99`, `expectile: 0.7`, +`inverse_temperature: 3`, `max_advantage_weight: 100`, `target_update_rate: 0.005`, +evaluation/checkpoint intervals `100`. Held-out action NLL selects the actor; +critic diagnostics are separate. Critics are fresh per phase and are not exported. + +## DPO preferences + +A preference manifest declares `format: sequifier.preference-pairs` and +`pairs`. Each pair has `pair_id`, `context_id`, optional `group_id`, and +`chosen`/`rejected` spans `{source, episode_id, start, end}`. Bounds are zero-based, +start-inclusive/end-exclusive within loaded episode rows. Initial executable +histories must match. Duplicate IDs, overlapping/identical alternatives and +incompatible contexts fail before training. Rewards do not imply preferences. + +DPO minimizes `-logsigmoid(beta * ((s_actor_chosen-s_actor_rejected) - +(s_ref_chosen-s_ref_rejected)))`, where each score sums valid decision log likelihoods. +Pairs have equal weight, regardless of length. Defaults: `beta: 0.1`, +`scoring_chunk_size: 64`. Two-pass chunked backward preserves the scalar derivative +while bounding activation memory. Optional `evaluation.sources` selects held-out +preference manifests. + +DPO and GRPO default to `reference: {source: phase_start}`; an artifact reference +uses `{source: artifact, path: ...}`. References are independent frozen snapshots. +Embedded checkpoint references survive removal of their original files; +a still-present changed reference is rejected. + +## Online adapters + +PPO, GRPO, evaluation and policy rollout use +`environment: {factory: module:factory, config: {}, units: {}}`. +Factories receive `config` as keyword arguments. Adapter-reported units must match +the configuration. Custom adapters subclass `OnlineEnvironmentAdapterBase` from +`sequifier.rl_api` and supply a nonempty `implementation_id` on the adapter. +Sequifier checks that identity when resuming a run. The same module exports +`EnvironmentCapabilities`, `OnlineResetResult`, and `OnlineStepResult`. + +Adapters implement `reset(seed)`, `step(named_domain_actions)` and `close()`; +policy rollout also requires `reset_from_observation(initial, seed)`. +Reset results provide observations/static inputs/units. Step results provide reward, +separate termination/truncation booleans, successor observation and optional success. +Never autoreset or substitute a reset observation for a final successor. A true +terminal policy step may omit its successor; other steps must retain it. + +Capabilities declare seeded reset, exact snapshot support and group-start support. +Exact PPO recovery requires `snapshot_state()`/`restore_state()` including simulator +RNG. GRPO requires `create_group_start(seed)` and `reset_group(token, rollout_seed)`; +all group members restore the same portable token with independent future streams. +Exact GRPO recovery requires reproducible group starts. Evaluation factories must +create independent adapters. The [control adapter](../src/sequifier/rl/control.py) +is a complete local example. + +## Initialize PPO and IQL from a pretrained history encoder + +Set `history_initialization: phase_start_actor` on a PPO or IQL phase with +`optimizer_state: reset`. PPO initializes its value encoder from the incoming +actor. IQL initializes the value and both Q encoders from that same actor. Each +network owns an independent copy of the transformer and input encoders. The +scalar heads and the two Q action encoders start fresh, and each target Q starts +as a copy of its corresponding critic. + +With `initialize_from`, the first phase receives the pretrained actor. Later +phases use the actor selected at the preceding handoff. The default +`history_initialization: random` uses fresh value and Q encoders. A phase with +`optimizer_state: continue` restores the preceding networks and optimizer state. + +The transferred history parameters follow the phase’s usual trainability rules. +An optimizer plan can fine-tune the backbone at a lower rate than the heads. +For example, put the following under a PPO or IQL phase's `optimization` and +use the same grouping for `actor` or IQL's `critic` role as needed. + +```yaml +value: + optimizer: + groups: + - id: backbone + select: {module: backbone} + optimizer: {name: Adam} + learning_rate: 0.00003 + - id: remaining + select: otherwise + optimizer: {name: Adam} + learning_rate: 0.0003 +``` + +See [supervised models and policy learning](supervised-policy-learning.md) for +initialization from artifacts whose decoder predicts different targets. + +## PPO and GRPO + +PPO seals one rollout generation, computes boundary-aware FP32 GAE, then optimizes +clipped actor and independent V objectives. Truncation bootstraps from the final +successor without carrying GAE across reset. Collection horizons retain live history. +Defaults: `optimization_epochs: 4`, `batch_size: 64`, `accumulation_batches: 1`, +`gamma: 0.99`, `gae_lambda: 0.95`, `clip_epsilon: 0.2`, `value_coefficient: 0.5`, +`entropy_coefficient: 0.01`, `normalize_advantages: true`, `reward_scale: 1`. +Optional `value_clip_delta` clips V changes; `old_policy_target_kl` stops old-policy drift. +`allow_partial_rollout: true` executes the remaining budget exactly; false requires +divisibility. Optional `archive_directory` writes independent collection records; +`archive_scores` defaults to false. Archives are not update sources. + +GRPO seals complete same-start groups before updating an actor-only model. +Scores sum scaled rewards through termination, truncation or `max_steps`. +`A_i = (R_i-mean(R))/(population_std(R)+advantage_epsilon)`. +Its loss averages equally over trajectories, then over their decisions: +`-min(r*A, clip(r)*A) + reference_kl_coefficient*(exp(d)-d-1)`, where +`r=exp(log_pi-log_pi_old)` and `d=log_pi_ref-log_pi`. +Defaults: `group_size: 4`, `groups_per_iteration: 1`, `optimization_epochs: 4`, +`advantage_epsilon: 1e-6`, `clip_epsilon: 0.2`, `reference_kl_coefficient: 0.04`, +`reward_scale: 1`, `scoring_chunk_size: 64`. Optional `old_policy_target_kl` limits +old-policy drift separately. The sampled reference penalty is not exact current-policy KL. +`max_env_steps` uses a conservative complete-group bound. Failed/incomplete groups +are discarded without retry; errors report issued actions. Both online algorithms +default to `max_buffer_bytes: 536870912` and reject stale/reused buffers. + +Image encoders accept `microbatch_size` (default `16`) independently of GRPO's +`scoring_chunk_size`. Smaller image microbatches reduce peak frozen-backbone +activation memory without changing policy optimizer-step boundaries. GRPO +collection runs sequentially in the training process. Iteration metrics report +collection seconds, collected environment steps per second, update seconds and +evaluation seconds. + +GRPO defaults to `simulator_workers: 1`, preserving the in-process sequential +collector. Setting a larger value explicitly enables persistent spawned simulator +processes and batched actor/reference encoding. Active workers are capped at +`groups_per_iteration * group_size`; `rollout_inference_batch_size` (default `32`) +caps each inference call. `worker_threads` (default `1`) bounds PyTorch intra-op +threads per simulator, and `worker_timeout_seconds` (default `120`) sets response +deadlines from dispatch. Worker factories/configurations must be importable and +picklable; custom Python entry points require a `__main__` guard. + +An adapter must explicitly declare `supports_parallel_group_start = True` before +workers can be enabled. This promises that a portable group token restores the +same scenario in an independent adapter instance and that `create_group_start` +depends on its seed/configuration, not the live trajectory. Existing adapters +remain valid with one worker. Construction/worker errors abort instead of silently +falling back or retrying issued actions. Pending groups are never checkpointed. +Pools close on all exit paths; unresponsive processes are terminated after a +bounded cleanup period. + +Every sample retains its independent action and environment seed streams. GRPO +training does not calculate entropy or sample unused entropy-estimator actions, +in either collector. Seeded trajectories therefore differ from older versions +that consumed those extra draws; PPO entropy computation is unchanged. All samples +use a single actor version and all complete groups are sealed before optimization. +Group ordering, trajectory weighting, optimizer boundaries and old-score checks +are unchanged. +Floating-point results may vary with inference batch shape; cross-device or +cross-configuration bitwise equivalence is not promised. + +Frozen image feature caching is independently opt-in through +`image_feature_cache_bytes`; see the training reference. Caches are transient, +memory-bounded and scoped to a phase. Metrics include actual simulator count and, +when caching is active, cumulative cache hits/misses and retained feature bytes. +The default disabled options are omitted from serialized contracts to preserve +existing plan/artifact identities. Enabled options are part of the resume plan. + +These options do not guarantee a speedup for every workload or machine. Compare +collection steps/second, update time and peak memory against the default paths. +Increasing groups per iteration changes update frequency; compare equivalent +learning budgets. Additional workers and caches require CPU/GPU memory headroom. + +Evaluation uses independent seeds/adapters, restores training modes and RNG, and +requires `environment`, `seeds`, `max_episode_steps`, `timeout_seconds` and the +main-thread POSIX timer. `action_selection` defaults to deterministic. PPO's default +monitor is `mean_completed_return`/`max`; `mean_completed_length` and `success_rate` +are supported. Censored episodes and unavailable metrics are explicit. GRPO uses +`score_mean`/`max`, including declared horizons and the fixed reward scale. +Online intervals count completed collection/update iterations. + +## Checkpoints and inference + +Policy runs use the shared `RUN_CHECKPOINT` envelope with a `POLICY_RECOVERY` +payload and a `POLICY_PHASE_STATE` record for the live phase. They retain representation, plan/source/partition/adapter +identities, selected phase summaries and checksums, RNG and live phase +networks/optimization/sampler/selection state. Training and evaluation histories +are journaled under each native phase directory. Keep the journal and committed +phase models with the checkpoint so reports can be regenerated from the recorded +positions. Mixed plans also retain their phase completion receipts. See +[Training artifacts](training/artifacts.md). + +Source files and output-path conflicts are rechecked at phase boundaries. +Atomic publication precedes checkpoint advancement, so completed phases never rerun. + +`resume.mode: exact` validates simulator snapshots or reproducible group starts. +`reset_environments` preserves updates but reports a trajectory discontinuity. +Offline resume restores sampler/order and optimizer transactions. Neither online +mode reuses partial groups or stale samples. Final portable artifacts contain +actor execution, codecs, normalization, policy and units, without critics or references. + +`load_train_config(path, {}, False)` loads/resolves the common schema; +`sequifier.train.run_training` dispatches its phases. Python APIs and environment +contracts are exported by `sequifier.rl_api` and `sequifier.training_api`. +For prediction, task routing, probability outputs and policy rollout, use the +[inference reference](configs/infer.md#policy-model-interfaces) and +[rollout guide](rollout.md). Deterministic execution, action selection and RNG seed +are independent settings; reproducibility also depends on runtime/device/order. + +Model scale parameters and policy action behavior use separate schemas; see +[model prediction parameters and policy actions](configs/train.md#model-prediction-parameters-and-policy-actions). +Normalization is global or inherited from the artifact and cannot be configured +on an individual phase. diff --git a/documentation/rollout.md b/documentation/rollout.md new file mode 100644 index 000000000..aadf48342 --- /dev/null +++ b/documentation/rollout.md @@ -0,0 +1,75 @@ +# Local rollouts + +Forecast and policy rollout share trajectory recording but retain distinct adapter +protocols. Both execute one local episode/fragment, own adapter cleanup, and atomically +write `trajectory_path` after successful validation. They do not add an export format. + +## Forecast configuration and protocol + +Forecast rollout supports CPU, real scalar temporal/static inputs, and one-step +`causal`/`final_value` models with `target_offset: 1`, using PT or schema-bearing ONNX: + +```yaml +task: forecast +rollout: + max_steps: 100 + seed_window_index: 0 + trajectory_path: outputs/trajectory.json + environment: {factory: "application.adapters:make_environment", config: {}} + bindings: + - {input: x, source: prediction, key: x} + - {input: disturbance, source: environment, key: disturbance} +``` + +Run `sequifier infer --config-path configs/infer.yaml`. `seed_window_index` selects +one prepared window in dataset/chunk order, following normal stride/window rules +before first-subsequence filtering. Each temporal input needs exactly one producer; +static inputs cannot have bindings. Extra/missing environment fields fail validation. + +Forecast adapters subclass `ForecastRolloutAdapterBase` from +`sequifier.rollout_api`. They implement `reset(ResetRequest)`, +`step(StepRequest) -> StepResult`, and `close()`. The same module exports the +request and result types. +Factories receive `environment.config` as keyword arguments. Reset initializes from +the supplied history without executing a transition; step receives copied current +observations, forecasts and static values. It returns external inputs, a finite reward +and separate Python boolean `terminated`/`truncated` flags. Adapters must not autoreset. + +Adapters and trajectories use original units. Fixed training codecs decode seeds/ +forecasts and encode successors with the same `std + 1e-9` normalization; rollout +never fits statistics. Padding stays zero with a false mask; history and mask slide +together. Unsupported shapes, overflow and non-finite transformed inputs fail. + +## Policy rollout + +Policy rollout uses the [online policy adapter](policy-learning.md#online-adapters), +matching `units` and seeded `reset_from_observation`. It selects +raw CSV/Parquet rows with `seed_row_index`, rejects forecast bindings, and executes +named policy actions. A seed supplies exactly one current observation; model history, +codecs and normalization come from its artifact. See +[inference options](configs/infer.md#policy-model-interfaces) and the +[Python example](example_inputs/ppo_toy_control.py). + +Both seed indices are required, zero-based, and checked before environment creation. +Only the route-specific seed index name is accepted and recorded in trajectory +metadata. Neither `seed_index` nor `rollout.type` is accepted. + +## Boundaries and trajectory API + +Every issued prediction/action executes once. True termination disables bootstrap; +truncation retains the final pre-reset successor. `max_steps` is a collection budget: +exhaustion sets `collection_cutoff` without inventing a boundary. Optional +`max_episode_steps` imposes a truncation. Terminal policy steps may omit successors. +The collector closes adapters on success/failure; construction failures remain the +factory's cleanup responsibility. + +Trajectories record seed/mask/static values, bindings, ordered transitions, original +unit predictions/actions/rewards, successors, boundary flags, total return and provenance. +Validation checks continuity, producer coverage, IDs, totals, finite values and boundary +semantics. Policy records also retain joint execution metadata and transform latents. + +`sequifier.rollout_api` exposes `collect_rollout`, `save_trajectory`, `load_trajectory` +and `validate_trajectory`. Direct forecast predictors consume a copied `SeedHistory` +and return named forecasts; optional `history_validator` checks the initial and every +successor history. `sequifier.rl_api.collect_policy_rollout` provides policy collection. +Standalone trajectories do not imply simulator snapshots or exact resumability. diff --git a/documentation/supervised-policy-learning.md b/documentation/supervised-policy-learning.md new file mode 100644 index 000000000..c85ab916f --- /dev/null +++ b/documentation/supervised-policy-learning.md @@ -0,0 +1,92 @@ +# Supervised models and policy learning + +A policy binding attaches a task to existing model targets. It preserves interfaces, +weights, codecs, normalization, forward attention and decoder geometry; it does +not automatically add inputs or replace a pretrained decoder. Start from an +ordinary supervised PT artifact or supported run checkpoint: + +```yaml +initialize_from: models/supervised.pt +policy_bindings: + default: + actions: + mode: + type: categorical + values: + - 'off' + - 'on' + price: + type: real + scale: + fixed: 0.5 +``` + +Add datasets/phases as in the [policy guide](policy-learning.md). With the default +`actor_initialization: attach`, a separately declared model must agree with the +artifact. Omitted categorical `values` select +the whole decoder vocabulary; an explicit subset is reported as an adaptation. +Real `scale` explicitly selects `model` or `{fixed: ...}` in physical units. +Learned scales belong to the interface's `prediction_distributions`, where +`scale` declares `initial`, `trainable`, `min`, and `max`. `lower`/`upper` use bijections; +`atoms` assigns exact point masses (positive probabilities summing to less than one). +Bounds change the deterministic prediction and count as adaptation. Samples retain +transform latents; external endpoints need declared atoms. `active_when` references +earlier action fields; inactive fields use `inactive_value` with zero likelihood cost. + +### Initialize a fresh action decoder + +When the artifact predicts different targets, use these run-level fields with an +explicit downstream `model`, `policy_bindings`, and a policy first phase. + +```yaml +initialize_from: models/observation-pretrain.pt +actor_initialization: history_only +``` + +Use the same named interface and compatible history inputs, preprocessing, +context, attention, precision and backbone architecture. Declare the downstream +action targets and an untied decoder. The artifact supplies the history encoder +and backbone weights, and the downstream model supplies the newly initialized +decoder and action distribution parameters. Set `history_initialization: +phase_start_actor` on PPO or IQL phases to also initialize their value or Q history +encoders. See the [policy guide](policy-learning.md#initialize-ppo-and-iql-from-a-pretrained-history-encoder). + +Check that every input is available at decision time. Pretraining with future +context can change how well the representation works on histories available to +the policy. Compare actor-only and actor-plus-auxiliary initialization under the +same downstream data, evaluation seeds and training budgets. + +Ordinary branches factor independently; autoregressive branches retain target +order and score recorded prefixes. Selecting a later autoregressive action requires +all preceding decoder targets. Composite branches and tied weights are retained. +Likelihoods sum active fields; supervised loss weights never redefine probabilities. +Independent categorical/unbounded Gaussian entropy is exact; dependent/transformed +entropy uses a score-function estimator. Deterministic selection follows sequential +argmax/locations, without claiming a global joint mode. + +Inputs retain the complete scalar/static/depth/image/mask execution schema. +Prepared sources reuse depth masks and image stores; tables may contain nested +Parquet payloads. Explicit `previous_action_inputs` maps existing input names to +actions; `initial_inputs` supplies the first lagged value using its persisted codec. +`decision_position` defaults to `-1` and selects one existing decoder position, +not an action block. Historical likelihoods use only observations available then. +Return conditioning requires an existing `__policy_return` input and known offline +tails or online `initial_desired_return`. + +Supervised and policy phases share `training_plan.phases`; see the +[complete mixed template](../src/sequifier/templates/train-supervised-to-policy.yaml). +Prepared and sequence/preference parts belong to `dataset_training`. Each phase owns +its batching and role optimization (`model` for supervised; policy roles in the guide). +All interfaces survive handoff, though shared backbone updates can affect inactive +interfaces. The coordinator preflights the plan, commits incoming artifacts, and uses +native checkpoints plus atomic completion receipts. Mixed plans execute locally and +reject optimizer continuation; compatible native plans follow the +[continuation rules](configs/train.md#optimization-across-phases). + +Existing supervised losses retain their meanings. Real `prediction_distributions` +and `GaussianNLLLoss` opt into the shared distribution implementation; learned scales +belong to the model artifact. `task: forecast` selects point prediction from a +policy-bound artifact; `task: policy` selects policy execution. See +[inference routing](configs/infer.md#policy-model-interfaces). +Sequifier handles format validation when it loads model artifacts, trajectories, +and PPO collection records. ONNX remains a prediction export. diff --git a/documentation/training/artifacts.md b/documentation/training/artifacts.md new file mode 100644 index 000000000..a571025ce --- /dev/null +++ b/documentation/training/artifacts.md @@ -0,0 +1,119 @@ +# Training artifacts + +Supervised learning, policy imitation, IQL, PPO, DPO, GRPO and mixed plans use the +same output, checkpoint, report and scalar metric formats. Sequifier checks +generated artifact formats when it loads them. + +## Configuration and layout + +All paths are relative to `run.project_root` unless absolute. + +| Setting | Default | +| --- | --- | +| `run.outputs.model_path` | `models/.pt` | +| `run.outputs.report_path` | `outputs/training//report.json` | +| `run.outputs.phase_directory` | `outputs/training//phases` | +| `run.recovery.resume.checkpoint_path` | `checkpoints/runs//-latest.pt` | + +Every runner publishes a final portable PT model, independently of the optional +best/last, embedding and ONNX exports controlled by `run.export`. Supervised +training retains its existing `-best-` and `-last-` +exports. The canonical final model follows the final executed phase's +`model_selection.handoff`. + +Every completed phase publishes `/-/model.pt` +and `report.json`. Mixed-plan native recovery is stored in that phase's `run.pt`; +native child artifacts are below `native/`. Policy snapshots use +`checkpoints/transaction-NNNNNNNN.pt` below the native phase directory. Supervised +snapshots retain their model/epoch filenames below the same phase `checkpoints/` +directory; the rolling checkpoint remains at its configured run-level path. + +Phase trees, logging directories and collection archive directories are reserved +output namespaces. Model, report, checkpoint and manifest files must be distinct +and cannot overlap reserved namespaces or source data. Validation runs before +publication, including generated phase destinations and symlink escapes. + +## Portable models and recovery + +Portable models use the `sequifier_model` artifact type. +Their execution contract embeds interfaces, codecs, normalization, policy +bindings and units. Policy execution reads units from the selected interface; +provenance is not a second source of execution configuration. + +All newly written run checkpoints have the same envelope: + +```text +artifact_type: sequifier_run_checkpoint +model: +recovery_type: supervised | policy | learning +recovery: +``` + +The recovery payload retains optimizer, scheduler, sampler, RNG, selection and +algorithm-specific state. Exact recovery validates the runner type as well as +its own state version and contracts. A checkpoint cannot resume another runner. +Inference, initialization and analysis use the common embedded `model`, without +having to parse recovery state or supply training YAML. Loading a checkpoint +that precedes the first executable model raises an explicit error. A mixed-plan +envelope embeds the last committed incoming phase model; its native phase +checkpoint contains in-phase progress. + +## Reports and discovery + +Sequifier writes a run manifest at the configured report path and at +`manifest.json` beside +the rolling checkpoint. Common fields identify the logical run, current session, +status, completion reason, final model, checkpoint, exports and completed phases. +Additional fields contain runner-specific provenance. + +Each phase report includes phase index, name, algorithm, +model and checkpoint paths, monitor/direction, handoff, selected metric and +transaction, selected weight hash, clocks, metrics, provenance and stop scope. +Unavailable measurements are null. Algorithm-specific metrics keep their names +and units; a reward monitor is not represented as a loss. + +## Logging and scalar metrics + +Every native runner configures the model-scoped logger under `logs/`. +Structured scalar observations use the supervised writer's `*-training.csv`, +`*-training-full.csv`, `*-validation.csv` and `*-validation-full.csv` tables. +Full tables include phase, algorithm and +iteration identity, run/session IDs, UTC timestamps and progress. Named scalar +observations use `metric`, `target`, `value` and `is_primary`; policy observations +use `target: __total__`. Nested diagnostic records remain in phase reports. + +`run.logging.log_interval_batches` controls supervised training observations; +`run.logging.log_interval_iterations` controls policy training observations and +defaults to 10. Mixed plans may configure both. Evaluations are written whenever +the phase evaluation schedule runs, and final policy training progress is always +written. Policy iteration clocks count committed updates, even when an iteration +contains several optimizer steps. Policy visualizations display the actual +monitor names on a linear scale and do not invent baseline measurements. + +Hyperparameter search accepts supervised plans. Policy and mixed plans are not yet +supported. Its default objective is the final phase's evaluation monitor; the +search direction must match. Explicit output destinations are isolated per +sampled run. + +## Inference and collection outputs + +Forecast and policy predictions share `sequenceId`, `subsequenceId`, +`windowStartOffset`, `itemPosition` and named prediction columns. Independent +policy rows receive synthetic sequence/item identities. Categorical probabilities +are separate target-specific files under +`outputs/probabilities///part-NNN.`, with decoded category +labels as column names. Policy predictions additionally include +`action_log_probability` when probabilities are requested. Conditional policy +probabilities remain conditional on the selected preceding actions. + +All inference routes accept `output_path` for one model. It overrides the first +prediction/embedding file; later chunks append `-part-NNN` to its stem. Probability +files keep their canonical target-specific destinations. + +Forecast and policy rollout use the shared validated trajectory record. +Training-time PPO and GRPO archives instead use the common +`sequifier.collection-archive` envelope with `algorithm` and +`collection` fields. The collection payload contains algorithm-specific replay +and diagnostic information. `read_collection_archive` reads the envelope; +`load_collection_record` additionally validates PPO collections. These archives +are diagnostics, not interchangeable offline training trajectories. diff --git a/documentation/training/multi-gpu-training.md b/documentation/training/multi-gpu-training.md index c6bc0b6cf..e3aee8eed 100644 --- a/documentation/training/multi-gpu-training.md +++ b/documentation/training/multi-gpu-training.md @@ -29,19 +29,18 @@ write_format: pt Once your data is preprocessed into `.pt` shards, or beta `.parquet` shards, you need to tell the Sequifier training engine to expect a distributed environment. -In your `train.yaml`, configure the canonical `global_training` block: +In your `train.yaml`, configure `run.execution.distribution` and each prepared part’s `read_format`: ```yaml -global_training: - read_format: pt # or parquet for beta sharded Parquet loading - distributed: true - data_parallelism: fsdp # or ddp - fsdp_cpu_offload: false # omit if using ddp; set true to offload FSDP parameters to CPU RAM - layer_type_dtypes: null # required for FSDP; use layer_autocast for mixed precision - torch_compile: inner # use inner or none for FSDP; use outer or none for DDP - world_size: 32 # The TOTAL number of GPUs across all nodes (e.g., 8 nodes * 4 GPUs = 32) - backend: nccl # 'nccl' is the standard and most efficient backend for NVIDIA GPUs - +run: + execution: + device: cuda + distribution: {strategy: fsdp, cpu_offload: false, world_size: 32, backend: nccl} + compilation: {torch_compile: inner} +dataset_training: + events: + parts: + train: {read_format: pt, metadata_config_path: metadata_configs/events.json} ``` When shards do not divide evenly across ranks, Sequifier automatically pads shorter ranks with repeated samples for step alignment. Those repeats are masked out of loss calculation, so each real sample contributes once. @@ -84,6 +83,6 @@ srun torchrun \ ### Important Considerations for Multi-Node -* **Batch Size:** The `batch_size` in your `train.yaml` is the **per-process** batch size. If `batch_size` is 100 and `world_size` is 32, each synchronized backward pass covers 3,200 samples. With gradient accumulation, the samples per optimizer update are `batch_size * world_size * accumulation_steps` (apart from a final partial accumulation window). +* **Batch Size:** The `batch_size` in your `train.yaml` is the **per-process** batch size. If `batch_size` is 100 and `world_size` is 32, each synchronized backward pass covers 3,200 samples. With gradient accumulation, the samples per optimizer update are `batch_size * world_size * accumulation_batches` (apart from a final partial accumulation window). * **Learning Rate:** You may need to scale your `learning_rate` up if you drastically increase your global batch size via distributed training. * **Data Access:** All nodes must have access to the same shared filesystem (e.g., NFS, GPFS) where the `project_root` and the sharded preprocessing output are stored. diff --git a/documentation/training/resource-controls.md b/documentation/training/resource-controls.md new file mode 100644 index 000000000..f9409c710 --- /dev/null +++ b/documentation/training/resource-controls.md @@ -0,0 +1,93 @@ +# Training resource controls + +The I12 source audit found these missing effects before this change: + +| Configuration | Missing behavior | +| --- | --- | +| `run.execution.data_loading.max_ram_gb` | No runtime consumer or data-loading budget. | +| `run.execution.precision.layer_autocast` | Ignored by local and DDP forward/validation; only FSDP applied mixed precision. | +| `run.execution.precision.float32_matmul_precision` | Set in the launcher, but lost in spawned training processes. | +| `run.reproducibility.enforce_determinism` | Requested deterministic algorithms in warning-only mode; false did not undo a previous true setting in the same process. | + +Device, distribution, FSDP offload, compilation, layer parameter dtypes, worker +counts, concatenation limits and episode-cache sizes already have consumers on +their applicable routes. Policy-containing plans reject unsupported resource +controls. `load_full_data_to_ram` selects eager/lazy **folder** loading; single +prepared files are eager. + +## Data-loading RAM + +`max_ram_gb` is a positive finite budget in decimal gigabytes for prepared-data +loading in each training rank, including its loader workers. It covers all cached +training/validation loaders together, decoded data, view/order indices, static +tables, decoded-frame caches, chunk/conversion working space and prefetched +batches. It is not an operating-system limit on total process RSS: model, +optimizer, activation, interpreter and third-party allocator memory require +additional headroom. Multiple ranks each have their own budget. + +Preflight uses decoded sizes rather than compressed file sizes and reserves +conversion/concatenation and worker/prefetch headroom. The largest applicable +phase batching/view requirements are considered before loading data. Actual +resident dataset storage is checked against the plan after construction, and +streamed shards are checked against their working allowance before gathering. An +over-budget plan raises `MemoryError` with required/available sizes and remedies; +it does not silently change an authored eager/lazy choice. Use lazy folders, +smaller shards, fewer workers, smaller batches or a larger budget as appropriate. + +Lazy folders gather at most a microbatch at a time, including when view expansion +produces many logical examples from one stored chunk. Chunk loading remains +bounded by the shard size, so an individual shard must fit its working allowance. + +## Policy run resource accounting + +Policy run reports include a `resource_accounting` section. Policy phase reports +carry the cumulative snapshot in `provenance.resource_accounting`. The snapshots +show current and peak estimated bytes by category and the peak of their combined +demand at a single point in execution. A resumed invocation starts a fresh +accounting session. + +Accounting covers prepared policy catalogs and partitions, decoded episode and +label caches, reconstructed scoring and update windows, rollout estimates and +archive working space, parent-side simulator request and response payloads, and +retained frozen-image features. Prepared partitions are shared through the last +phase that needs them. Each rollout is released after its update and any archive +or summary work, before evaluation begins. + +`episode_cache_size` still counts episodes, `max_buffer_bytes` still uses the +algorithm's conservative payload estimates, and `image_feature_cache_bytes` +still applies to each distinct cache. A cache shared by the actor and reference +contributes once. The combined estimate is observational. Model and optimizer +storage, activations, allocator overhead, and simulator process internals are +outside its scope. The supervised data-loading budget continues to apply to +supervised loaders. + +## Autocast and matrix precision + +`layer_autocast: true` applies Torch autocast to local/DDP training forward and +loss calculation, validation and warm-up. Backward runs outside autocast. +Parameter storage dtypes remain governed by `layer_type_dtypes`. A configured +linear FP16/BF16 dtype also selects that autocast dtype; otherwise the device's +Torch default is used (CPU BF16, CUDA FP16). MPS uses BF16 and requires macOS 14 +or newer; native MPS FP16 autocast is rejected because the native gradient-scaler +protocol supports CPU/CUDA. Unsupported device/dtype requests fail explicitly. +FSDP retains its BF16 mixed-precision policy by default. + +FP16 autocast with FP32 optimizer parameters enables gradient scaling. Explicit +FP16 parameters retain the existing unscaled update protocol. The scaler is +persisted/restored with native recovery. FSDP uses its sharded mixed-precision +policy rather than a second enclosing autocast context. + +`float32_matmul_precision` is applied inside each native training process before +model construction/compilation, including spawned ranks and direct builder use. +It is independent of autocast and parameter storage precision. + +## Determinism + +`enforce_determinism: true` enables deterministic algorithms with errors for +unsupported nondeterministic operations and configures CUDA workspace policy +before seeding. False disables Torch's deterministic-algorithm requirement, +including after an earlier deterministic run in the same process. Seeds and +intentional stochastic action selection remain independent controls. + +No tests have been written or run for this change, as requested. Runtime/device +and memory-estimation validation remains pending. diff --git a/documentation/training/runtime-architecture.md b/documentation/training/runtime-architecture.md index e36308f04..a2c7eb01e 100644 --- a/documentation/training/runtime-architecture.md +++ b/documentation/training/runtime-architecture.md @@ -58,11 +58,10 @@ precedes deep feature arithmetic in eager, compiled, and exported forwards. Unique layers in all temporal and depth stacks are compiled once with their shared aliases restored. Warm-up preserves the run RNG, including on failure. -New portable model artifacts use format version 2; version 1 flat models remain -readable. Tensor payload versions and stored-window versions remain separate. -Run checkpoints record initialization-policy and seed-derivation versions in -addition to resolved branch policies and the existing optimizer/scaler/RNG state. -Inactive legacy layout/payload fields are normalized in resume comparisons. +Sequifier validates portable model artifacts, tensor payloads, and stored-window +layouts independently. Run checkpoints retain the initialization contract, resolved +branch policies, and optimizer, scaler, and RNG state. +Resume comparisons use the current layout and payload fields. ## Export preflight and publication diff --git a/hooks/consolidate_docs.py b/hooks/consolidate_docs.py deleted file mode 100644 index abd9a8672..000000000 --- a/hooks/consolidate_docs.py +++ /dev/null @@ -1,44 +0,0 @@ -#!/usr/bin/env python3 -import sys -from pathlib import Path - -FILES_TO_READ = [ - "README.md", - "documentation/configs/preprocess.md", - "documentation/configs/train.md", - "documentation/configs/infer.md", - "documentation/commands/visualize-training.md", - "documentation/configs/hyperparameter-search.md", - "documentation/training/multi-gpu-training.md", -] - -OUTPUT_FILE = "src/sequifier/consolidated-docs.md" - - -def main(): - consolidated_content = [] - - for filepath in FILES_TO_READ: - path = Path(filepath) - if not path.is_file(): - print(f"Error: Required file not found - {filepath}") - sys.exit(1) - - consolidated_content.append(path.read_text(encoding="utf-8")) - - final_content = "\n\n".join(consolidated_content) - - output_path = Path(OUTPUT_FILE) - current_content = "" - if output_path.is_file(): - current_content = output_path.read_text(encoding="utf-8") - - if current_content != final_content: - output_path.parent.mkdir(parents=True, exist_ok=True) - output_path.write_text(final_content, encoding="utf-8") - print(f"Hook updated {OUTPUT_FILE}") - sys.exit(1) - - -if __name__ == "__main__": - main() diff --git a/pyproject.toml b/pyproject.toml index 311a61eba..86049ea87 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -45,7 +45,7 @@ Repository = "https://github.com/0xideas/sequifier" sequifier = "sequifier.sequifier:main" [tool.setuptools.package-data] -sequifier = ["consolidated-docs.md"] +sequifier = ["consolidated-docs.md", "templates/*.yaml"] [project.optional-dependencies] images = [ diff --git a/release-notes/unreleased-canonical-config-names.md b/release-notes/unreleased-canonical-config-names.md new file mode 100644 index 000000000..e117e4e8e --- /dev/null +++ b/release-notes/unreleased-canonical-config-names.md @@ -0,0 +1,24 @@ +# Canonical public configuration names + + +- Use named training interfaces, datasets, parts, bindings, and phases for every + cardinality. Remove singleton aliases and boolean evaluation shorthand. +- Standardize dimensions, counts, batch/optimizer/iteration clocks, selectors, + checkpoint intervals, paths, monitoring direction, and probability names. +- Require explicit inference `task` and list-valued `model_paths`. Named media + paths are mappings; preprocessing widths are always `window_lengths`. +- Nest registered component constructor settings under `arguments` and declare + initialization methods as mappings. Remove scalar, method, and dtype aliases. +- Give hyperparameter search explicit `$choices`, `$fixed`, `$range`, and + `$variants` operators with canonical training paths and independent sampler seed. +- Resolve run `seed: random` once. CLI flags mirror the canonical keys, with no + short aliases or abbreviations and explicit boolean override values. +- Update runtime consumers, fingerprints, scaffolding, templates, and guides. + +This is a breaking authoring change without an alias compatibility layer. See +[the complete naming reference](../documentation/configs/naming.md). Configurations +embedded in artifacts/checkpoints must also use the canonical reconstruction +schema; old embedded declarations are not silently translated. + +No tests were written or run for this change, as requested. Verification is limited +to static source syntax, reference searches, and diff inspection. diff --git a/release-notes/unreleased-configuration-ownership.md b/release-notes/unreleased-configuration-ownership.md new file mode 100644 index 000000000..94feff637 --- /dev/null +++ b/release-notes/unreleased-configuration-ownership.md @@ -0,0 +1,11 @@ +# Configuration ownership + +- Replace authored `global_training` and root resources with explicit `run`, durable `model.execution` and named interface input contracts. +- Keep dataset declarations focused on source identity and split policy. Move losses, metrics, curriculum and trainability to phases. +- Share typed catalog parameter selectors between optimizer routing and every algorithm's role trainability; supervised phases retain source-dependent masks. +- Put evaluation cadence, initial/snapshot triggers and batching on each phase. Keep retained snapshots phase-owned and rolling recovery run-owned. +- Add observation-based stopping with explicit `end_phase` or `end_plan` scope across native and mixed runners, with checkpointed patience and stop state. +- Update CLI mappings, search paths, preprocessing/inference resource ownership, scaffold templates, examples, notebooks and documentation. +- Bump native recovery formats because phase ownership and stopping state changed; portable execution artifacts remain usable for initialization/inference. + +No tests were written or run for this implementation, at the user's request. Validation is limited to static source inspection, syntax, lint and type analysis. diff --git a/release-notes/unreleased-contract-versions.md b/release-notes/unreleased-contract-versions.md new file mode 100644 index 000000000..d01f722b7 --- /dev/null +++ b/release-notes/unreleased-contract-versions.md @@ -0,0 +1,20 @@ +# Consolidated Sequifier contract versions + +Sequifier manages compatibility for generated artifacts, metadata, and adapter +protocols internally. Rollout configs select the adapter by factory. + +- Re-run preprocessing from raw data to rebuild dataset metadata, PT batches, + stored-window metadata, preprocessing manifests, depth ledgers, and video + frame indexes. +- Start a fresh training run. Pre-cutover checkpoints cannot be resumed. Use the + previous Sequifier release for any run or portable model you still need to + load from that release. Produce fresh portable models, backbone repository + revisions, reports, metrics, and ONNX exports for the current release. +- Re-export sequence data and regenerate trajectories and collection archives. + Authored preference-pair and trajectory-source manifests use a format tag. +- Custom adapters can inherit protocol metadata from + `ForecastRolloutAdapterBase` or `OnlineEnvironmentAdapterBase`. +- Initialization and policy collection use new seed namespaces. The same inputs + are deterministic within the new version and can produce different seeds from + earlier releases. Versioned identities change fingerprints and export + preflight cache keys. diff --git a/release-notes/unreleased-phase-ownership.md b/release-notes/unreleased-phase-ownership.md new file mode 100644 index 000000000..9d7939d62 --- /dev/null +++ b/release-notes/unreleased-phase-ownership.md @@ -0,0 +1,31 @@ +# Phase-owned training and explicit inference routes + +- Move batching and role-keyed optimization into each training phase. Supervised + uses `optimization.model`; policy roles remain algorithm-specific. Reject + legacy global optimization fields and remove the global reset switch. +- Add phase `optimizer_state: reset | continue`, with reset as the default and + validated compatible native continuation. Reject mixed-plan continuation and + policy OneCycle continuation. Preserve supervised epoch, update and metric + scheduling; resolve epoch OneCycle chain budgets and validate batch budgets, including interleaving and dataset-boundary flushes. +- Store resolved phase settings in checkpoint/config fingerprints and native + artifacts needed for recovery. +- Validate policy inference directly against its own schema and reject fields + belonging to another route. CLI overrides use the same validation. +- Rename inference execution determinism to `enforce_determinism` and policy + inference/evaluation selection to `action_selection`. Seeded sampling and + deterministic execution remain separate controls. +- Use `rollout.environment` and select rollout schemas from the artifact interface and `task`. + Forecast rollout and online policy environments use distinct adapter protocols. + Document forecast `seed_window_index` versus policy `seed_row_index` units. +- Update shipped examples, scaffolding, Python examples and consolidated docs. + CPU smoke coverage is required before release for native supervised and policy algorithms, mixed + handoff/recovery, inference, rollout and configuration rejection. No tests were written or run for the ownership implementation. + + +All algorithms own evaluation on their phase. Offline policy evaluation uses +`evaluation.sources` and `evaluation.monitor: {metric, mode}` without aliases. +Generative inference dispatches from the selected interface, with optional `task`; +`model_type` is `generative` or `embedding`. Rollout indices are `seed_window_index` +(forecast) and `seed_row_index` (policy), including trajectory metadata. +Scalar action mappings use `actions`/`previous_actions`; structured records use +`action_record`/`previous_action_record`. diff --git a/release-notes/unreleased-policy-training.md b/release-notes/unreleased-policy-training.md new file mode 100644 index 000000000..6b68237b7 --- /dev/null +++ b/release-notes/unreleased-policy-training.md @@ -0,0 +1,37 @@ +# Multi-phase categorical policy training (unreleased) + +- One `SequifierConfig` authoring schema with `training_plan.phases`, CLI dispatch and Python `run_training` entry point + support ordered imitation, IQL, PPO, DPO and GRPO phases with natural budgets. +- A frozen architecture/interface/codec/normalization/context/units contract spans + phases. Selected actors are copied at handoff; compatible native phases may explicitly + continue optimizer/critic state. Mixed plans reset optimization at handoffs. + Best/last selection is explicit, ties retain the earliest best, and final export + uses the final phase's selection. +- DPO adds versioned reward-free preference spans, connected-component-safe + partitions, branch-history validation, summed causal action likelihoods and + canonical sigmoid DPO. Chunked backward scoring bounds activation memory. +- GRPO adds explicit reproducible group-start capabilities, independent RNG + streams, bounded complete groups, population-normalized advantages, clipped + ratios and a separately named sampled frozen-reference penalty, without a critic. +- Versioned local online adapters serve PPO, GRPO and policy rollout. Explicit + seeded reset preserves supplied initial observations during inference. Exact continuation requires + validated snapshots or documented reproducible group starts; non-exact restarts + report a discontinuity. Failed/partial groups are never reused. +- One atomic policy recovery checkpoint owns phase progress, + role-keyed optimizer/network state, references, source identities and RNG. + Phase actors are durable exports referenced by checksum; reports are derived + and regenerated on resume. No phase manifests are needed. +- Shared phase sessions handle selection, checkpoints and optimization. Each + phase configures optimizer roles under `optimization`; gradient validation + and clipping precede all optimizer steps in a transaction. +- Actor-only portable artifacts remain usable for inference, rollout and cross-run + initialization. Default output is `models/.pt`; phase reports and + selected actors use indexed directories. +- Templates, examples, public exports and guides use the canonical schema. The + pre-existing supervised training runtime remains independent. + +Policy bindings support categorical and real action heads, including joint actions, +through the shared model execution schema. Policy algorithms currently execute +locally; distributed supervised execution remains available. See the current +[phase ownership notes](unreleased-phase-ownership.md) and +[policy guide](../documentation/policy-learning.md) for the supported surface. diff --git a/release-notes/unreleased-supervised-policy-unification.md b/release-notes/unreleased-supervised-policy-unification.md new file mode 100644 index 000000000..2a3f19eb4 --- /dev/null +++ b/release-notes/unreleased-supervised-policy-unification.md @@ -0,0 +1,23 @@ +# Supervised and policy model unification + +- Attach policy tasks directly to ordinary supervised model artifacts and + supported supervised run checkpoints, preserving prediction heads and inputs. +- Train named combinations of categorical and real action heads with imitation, + IQL, PPO, DPO, and GRPO. Retain autoregressive/composite decoders and hashed + categorical candidate semantics. +- Add bounded real distributions, optional point masses, conditional activity, + bounded global exploration scales, and transform-latent replay. +- Preserve full execution-schema payloads, including static, depth, image, and + mask inputs, throughout policy history and storage. +- Add mixed supervised/policy run plans with native phase recovery, atomic + handoffs, model identity checks, and source/environment preflight. +- Separate persisted forward attention from optimization objectives and allow + explicit prediction or policy inference views. +- Add shared real prediction distributions and supervised Gaussian NLL support. +- Write versioned model artifacts, typed policy trajectories, and collection payloads; + retain readers for earlier model artifacts and single-categorical interfaces. + +Focused CPU tests now cover phase-owned optimization, all policy algorithms, +checkpoint recovery, mixed handoffs and forecast/policy inference and rollout. +See [the usage guide](../documentation/supervised-policy-learning.md) and +[phase ownership notes](unreleased-phase-ownership.md) for the implemented contracts. diff --git a/release-notes/unreleased-training-artifact-contracts.md b/release-notes/unreleased-training-artifact-contracts.md new file mode 100644 index 000000000..ca59bb4ea --- /dev/null +++ b/release-notes/unreleased-training-artifact-contracts.md @@ -0,0 +1,20 @@ +# Unified training artifact contracts + +- Honour `run.outputs` for supervised-only, policy-only and mixed plans, including + a canonical final portable model and per-phase model/report records. +- Reject overlapping output files, reserved phase/log/archive trees, protected + inputs and generated destinations escaping through symlinks before publication. +- Store exact recovery behind a shared `sequifier_run_checkpoint` + envelope with an embedded executable model and runner-owned recovery payload. +- Publish versioned run manifests and phase reports, retaining + supervised manifest discovery beside the rolling checkpoint. +- Read policy units from the model interface across every export/loading route. +- Write policy observations through the supervised scalar metric tables with + explicit version, phase, algorithm, iteration and primary-metric fields. Add + `run.logging.log_interval_iterations` and support policy visualization. +- Align prediction identity columns, categorical probability files and + `output_path` across inference tasks. Wrap algorithm-specific training collection + archives in a shared, versioned envelope. + +This is a new major-version contract. Previous native checkpoint formats remain +readable; earlier metric schemas are not migrated. diff --git a/setup.py b/setup.py new file mode 100644 index 000000000..e000c575f --- /dev/null +++ b/setup.py @@ -0,0 +1,21 @@ +"""Build standalone package resources from their canonical sources.""" + +from pathlib import Path +from runpy import run_path + +from setuptools import setup +from setuptools.command.build_py import build_py + + +class BuildPy(build_py): + def run(self): + super().run() + root = Path(__file__).resolve().parent + documentation = run_path(str(root / "src/sequifier/docs.py")) + destination = Path(self.build_lib) / "sequifier/consolidated-docs.md" + destination.write_text( + documentation["source_documentation"](root), encoding="utf-8" + ) + + +setup(cmdclass={"build_py": BuildPy}) diff --git a/src/sequifier/api.py b/src/sequifier/api.py index b772d277f..cb156bb2e 100644 --- a/src/sequifier/api.py +++ b/src/sequifier/api.py @@ -4,6 +4,7 @@ ModelArtifact, ModelArtifactMetadata, ModelExecutionConfig, + load_execution_model, load_model_artifact, load_weights_from_run_checkpoint, ) @@ -11,6 +12,8 @@ canonical_parameter_name, canonicalize_state_dict, ) +from sequifier.model.execution import ModelExecution +from sequifier.model.execution_options import ModelExecutionOptions from sequifier.model.factory import build_transformer_network from sequifier.model.network import ( ComposableTransformerNetwork, @@ -39,7 +42,9 @@ "InterventionBinding", "ModelArtifact", "ModelArtifactMetadata", + "ModelExecution", "ModelExecutionConfig", + "ModelExecutionOptions", "ModelInterfaceModule", "ModelOutput", "ParameterCatalog", @@ -49,6 +54,7 @@ "build_transformer_network", "canonical_parameter_name", "canonicalize_state_dict", + "load_execution_model", "load_model_artifact", "load_weights_from_run_checkpoint", ] diff --git a/src/sequifier/artifacts/backbone_repository.py b/src/sequifier/artifacts/backbone_repository.py index 19b48c3cf..ae1ebbe54 100644 --- a/src/sequifier/artifacts/backbone_repository.py +++ b/src/sequifier/artifacts/backbone_repository.py @@ -1,7 +1,6 @@ import contextlib import hashlib import json -import os import uuid from collections.abc import Iterator from datetime import datetime, timezone @@ -9,12 +8,14 @@ from typing import Any import torch +from pydantic import BaseModel from torch import Tensor, nn +from sequifier.contract_versions import ContractVersions, require_contract_version +from sequifier.io.atomic import atomic_save, write_json_atomic from sequifier.typechecking import beartype ARTIFACT_TYPE = "sequifier_backbone" -FORMAT_VERSION = 1 SUPPORTED_DTYPES = { torch.bool, torch.int8, @@ -26,17 +27,14 @@ torch.bfloat16, torch.float32, torch.float64, + torch.float8_e4m3fn, + torch.float8_e5m2, } -SUPPORTED_DTYPES.update( - dtype - for name in ("float8_e4m3fn", "float8_e5m2") - if (dtype := getattr(torch, name, None)) is not None -) @beartype def canonical_architecture(architecture: Any) -> dict[str, Any]: - if hasattr(architecture, "model_dump"): + if isinstance(architecture, BaseModel): return architecture.model_dump(mode="json") if not isinstance(architecture, dict): raise TypeError("Backbone architecture must be a mapping or Pydantic model.") @@ -75,22 +73,6 @@ def _read_json(path: Path) -> dict[str, Any]: return value -@beartype -def _write_json_atomic(path: Path, value: dict[str, Any]) -> None: - path.parent.mkdir(parents=True, exist_ok=True) - temporary_path = path.with_name(f".{path.name}.{uuid.uuid4().hex}.tmp") - try: - with temporary_path.open("w") as file: - json.dump(value, file, sort_keys=True, indent=2) - file.write("\n") - file.flush() - os.fsync(file.fileno()) - os.replace(temporary_path, path) - finally: - with contextlib.suppress(OSError): - temporary_path.unlink() - - @beartype def latest_pointer(repository_path: Path) -> dict[str, Any] | None: path = repository_path / "latest.json" @@ -111,21 +93,26 @@ def latest_pointer(repository_path: Path) -> dict[str, Any] | None: @beartype -def select_revision(backbone_config: Any, project_root: str) -> dict[str, Any] | None: +def select_revision( + backbone_config: Any, project_root: str, *, revision_id: str | None = None +) -> dict[str, Any] | None: repository = backbone_config.repository if repository is None: return None repository_path = resolve_repository_path(repository.path, project_root) - pointer = latest_pointer(repository_path) - if pointer is None: + pointer = latest_pointer(repository_path) if revision_id is None else None + if revision_id is None and pointer is None: if repository.load_policy == "required": raise FileNotFoundError( f"Required backbone revision does not exist in {repository_path}." ) return None - revision_id = pointer["revision_id"] + if pointer is not None: + revision_id = pointer["revision_id"] + if not revision_id or Path(revision_id).name != revision_id: + raise ValueError("Invalid backbone revision identity") revision_path = repository_path / "revisions" / revision_id manifest_path = revision_path / "manifest.json" weights_path = revision_path / "weights.pt" @@ -134,6 +121,11 @@ def select_revision(backbone_config: Any, project_root: str) -> dict[str, Any] | f"Backbone revision {revision_id!r} is incomplete in {revision_path}." ) manifest = _read_json(manifest_path) + require_contract_version( + "backbone repository manifest", + manifest.get("format_version"), + ContractVersions.BACKBONE_MANIFEST, + ) expected_fingerprint = architecture_fingerprint(backbone_config.architecture) manifest_architecture = manifest.get("architecture") try: @@ -144,7 +136,6 @@ def select_revision(backbone_config: Any, project_root: str) -> dict[str, Any] | ) from exc checks = { "artifact_type": (manifest.get("artifact_type"), ARTIFACT_TYPE), - "format_version": (manifest.get("format_version"), FORMAT_VERSION), "backbone_id": (manifest.get("backbone_id"), repository.backbone_id), "revision_id": (manifest.get("revision_id"), revision_id), "architecture_fingerprint": ( @@ -155,11 +146,12 @@ def select_revision(backbone_config: Any, project_root: str) -> dict[str, Any] | manifest_fingerprint, expected_fingerprint, ), - "parent_revision_id": ( + } + if pointer is not None: + checks["parent_revision_id"] = ( manifest.get("parent_revision_id"), pointer.get("parent_revision_id"), - ), - } + ) mismatches = [ f"{name}: found {actual!r}, expected {expected!r}" for name, (actual, expected) in checks.items() @@ -278,23 +270,17 @@ def publish_revision( revision_path = revisions_path / revision_id revision_path.mkdir(exist_ok=False) weights_path = revision_path / "weights.pt" - temporary_weights_path = revision_path / f".weights.{uuid.uuid4().hex}.tmp" state_dict = { key: value.detach().cpu().clone() for key, value in backbone.state_dict().items() } _validate_state_dict(backbone, state_dict) - try: - torch.save(state_dict, temporary_weights_path) - os.replace(temporary_weights_path, weights_path) - finally: - with contextlib.suppress(OSError): - temporary_weights_path.unlink() + atomic_save(weights_path, state_dict) architecture = canonical_architecture(backbone_config.architecture) manifest = { "artifact_type": ARTIFACT_TYPE, - "format_version": FORMAT_VERSION, + "format_version": ContractVersions.BACKBONE_MANIFEST, "backbone_id": repository.backbone_id, "revision_id": revision_id, "parent_revision_id": parent_revision_id, @@ -303,8 +289,8 @@ def publish_revision( "source_run_id": source_run_id, "source_epoch": source_epoch, } - _write_json_atomic(revision_path / "manifest.json", manifest) - _write_json_atomic( + write_json_atomic(revision_path / "manifest.json", manifest) + write_json_atomic( repository_path / "latest.json", { "revision_id": revision_id, diff --git a/src/sequifier/artifacts/checkpoint_envelope.py b/src/sequifier/artifacts/checkpoint_envelope.py new file mode 100644 index 000000000..2cd37fe40 --- /dev/null +++ b/src/sequifier/artifacts/checkpoint_envelope.py @@ -0,0 +1,76 @@ +"""Shared execution envelope around runner-owned exact recovery state.""" + +from collections.abc import Mapping + +from sequifier.artifacts.model_artifact import ModelArtifact +from sequifier.contract_versions import ContractVersions, require_contract_version + + +def phase_boundary(run_id, *, selected=None, active=None): + """References made authoritative by the same write as the runner state.""" + return { + "format_version": ContractVersions.PHASE_BOUNDARY, + "run_id": run_id, + "selected": selected, + "active": active, + } + + +def selected_model_identity(completed): + if not completed: + return None + latest = completed[-1] + return {"path": latest["model_path"], "sha256": latest["model_sha256"]} + + +def checkpoint_envelope(recovery_type, recovery, model, *, boundary): + return { + "artifact_type": "sequifier_run_checkpoint", + "format_version": ContractVersions.RUN_CHECKPOINT, + "model": model.state_dict() if isinstance(model, ModelArtifact) else model, + "recovery_type": recovery_type, + "recovery": recovery, + "phase_boundary": boundary, + } + + +def validate_checkpoint_envelope(payload): + if not isinstance(payload, Mapping): + raise ValueError("Expected a Sequifier run checkpoint envelope mapping") + require_contract_version( + "run checkpoint envelope", + payload.get("format_version"), + ContractVersions.RUN_CHECKPOINT, + ) + if payload.get("artifact_type") != "sequifier_run_checkpoint": + raise ValueError("Expected a Sequifier run checkpoint envelope") + if payload.get("recovery_type") not in ("supervised", "policy", "learning"): + raise ValueError( + f"Unsupported checkpoint recovery kind {payload.get('recovery_type')!r}" + ) + boundary = payload.get("phase_boundary") + if boundary is not None: + if not isinstance(boundary, dict): + raise ValueError("Run checkpoint requires a phase boundary mapping") + require_contract_version( + "phase boundary", + boundary.get("format_version"), + ContractVersions.PHASE_BOUNDARY, + ) + if not isinstance(payload.get("recovery"), dict): + raise ValueError("Run checkpoint requires a recovery mapping") + + +def recovery_payload(payload, recovery_type): + """Validate the shared envelope before restoring runner-owned state.""" + validate_checkpoint_envelope(payload) + if payload["recovery_type"] != recovery_type: + raise ValueError( + f"Checkpoint recovery kind {payload['recovery_type']!r}; expected {recovery_type!r}" + ) + return payload["recovery"] + + +def require_phase_boundary(payload, expected): + if payload.get("phase_boundary") != expected: + raise ValueError("Phase boundary disagrees with checkpoint recovery") diff --git a/src/sequifier/artifacts/dependencies.py b/src/sequifier/artifacts/dependencies.py new file mode 100644 index 000000000..a70067528 --- /dev/null +++ b/src/sequifier/artifacts/dependencies.py @@ -0,0 +1,189 @@ +"""Content identities for inputs consumed by training.""" + +import json +from pathlib import Path + +from sequifier.trajectory.sources import file_checksum + + +class DependencyManifest: + def __init__(self): + self.identity: dict[str, str] = {} + self.roots: set[Path] = set() + self._media: dict[Path, tuple[str, ...]] = {} + self._paths: dict[Path, Path] = {} + self._embedded_references: set[str] = set() + + @property + def protected_paths(self): + return self.roots | {Path(path) for path in self.identity} + + def include(self, checksums): + for path, checksum in checksums.items(): + path = Path(path).absolute() + resolved = path.resolve() + self._paths[path] = resolved + previous = self.identity.setdefault(str(resolved), checksum) + if previous != checksum: + raise ValueError( + f"Training dependency changed during preparation: {path}" + ) + + def add(self, path): + resolved = Path(path).resolve() + if str(resolved) not in self.identity: + self.include({path: file_checksum(resolved)}) + else: + self._paths[Path(path).absolute()] = resolved + + def add_reference(self, path, checkpoint_checksum): + resolved = str(path.resolve()) + if checkpoint_checksum is None and resolved in self._embedded_references: + # A later phase can still need this same file to create its reference. + self.include({path: file_checksum(path)}) + self._embedded_references.remove(resolved) + return + try: + self.add(path) + except FileNotFoundError: + if checkpoint_checksum is None: + raise + self.include({path: checkpoint_checksum}) + self._embedded_references.add(resolved) + + def add_media_directory(self, directory, files): + self.roots.add(directory.resolve()) + self._media[directory] = tuple(file.name for file in files) + for path in files: + self.add(path) + + def merge(self, other): + self.include(other.identity) + self.roots.update(other.roots) + self._media.update(other._media) + self._paths.update(other._paths) + self._embedded_references.update(other._embedded_references) + + def validate(self, expected): + if self.identity != expected: + changed = sorted( + path + for path in self.identity.keys() | expected.keys() + if self.identity.get(path) != expected.get(path) + ) + raise ValueError( + f"Training dependencies differ from checkpoint: {changed[:10]}" + ) + + def revalidate(self): + from sequifier.io.images import media_files + + for directory, files in self._media.items(): + current = media_files(str(directory), str(directory.parent)) + if tuple(file.name for file in current) != files: + raise ValueError(f"Training media directory changed: {directory}") + for path, resolved in self._paths.items(): + if path.resolve() != resolved: + raise ValueError(f"Training dependency moved: {path}") + for path, checksum in self.identity.items(): + try: + current = file_checksum(path) + except FileNotFoundError: + if path in self._embedded_references: + continue + raise + if current != checksum: + raise ValueError(f"Training dependency changed: {path}") + + +def add_initialization(manifest, config, incoming=None): + if path := incoming or config.initialize_from: + manifest.add(Path(config.project_root) / path) + + +def _add_loader_media(manifest, view): + from sequifier.io.images import media_files + + root = Path(view.project_root) + if view.static_columns and view.static_features_path: + manifest.add(root / view.static_features_path) + media = [] + for source in view.image_sources.values(): + if source.path: + media.append(source.path) + if source.time_index_path: + manifest.add(root / source.time_index_path) + for path in source.video_index_paths.values(): + manifest.add(root / path) + if view.preconditioning_image: + media.extend(view.preconditioning_image.views.values()) + for path in media: + manifest.add_media_directory(root / path, media_files(path, view.project_root)) + + +def supervised_dependencies(config, incoming=None): + from sequifier.config.train_config import dataset_part_view + + manifest = DependencyManifest() + root = Path(config.project_root) + add_initialization(manifest, config, incoming) + for path in config.model_metadata_paths.values(): + manifest.add(path) + for name, dataset in config.dataset_training.items(): + for part_name, part in dataset.parts.items(): + if part.metadata_from_file: + manifest.add(root / part.metadata_config_path) + for split in (part.training_data_path, part.validation_data_path): + if split is None: + continue + path = Path(split) + if part.storage_form == "file": + manifest.add(path) + else: + manifest.roots.add(path.resolve()) + metadata = path / "metadata.json" + manifest.add(metadata) + for shard in json.loads(metadata.read_text())["batch_files"]: + manifest.add(path / shard["path"]) + _add_loader_media( + manifest, + dataset_part_view( + config, name, part_name, batch_size=1, window_stride=None + ), + ) + return manifest + + +def policy_dependencies( + config, prepared, inputs, *, saved=None, committed_reference_count=0 +): + manifest = DependencyManifest() + add_initialization(manifest, config) + for path in config.metadata_paths: + manifest.add(path) + for index, phase in enumerate(config.phases): + manifest.include(prepared.phase_checksums(config, phase, inputs)) + reference = phase.reference_config + if reference and reference.path: + path = Path(config.project_root) / reference.path + checksum = ( + saved.get(str(path.resolve())) + if saved is not None and index < committed_reference_count + else None + ) + manifest.add_reference(path, checksum) + return manifest + + +def add_backbone_revision(manifest, config, revision_id): + from sequifier.artifacts.backbone_repository import resolve_repository_path + + repository = config.model.backbone.repository + if repository is not None and revision_id is not None: + directory = ( + resolve_repository_path(repository.path, config.project_root) + / "revisions" + / revision_id + ) + manifest.add(directory / "manifest.json") + manifest.add(directory / "weights.pt") diff --git a/src/sequifier/artifacts/loading.py b/src/sequifier/artifacts/loading.py index 032c9d379..3ef7dcfc6 100644 --- a/src/sequifier/artifacts/loading.py +++ b/src/sequifier/artifacts/loading.py @@ -9,10 +9,8 @@ import torch from torch import nn -from sequifier.artifacts.model_artifact import ( - load_model_artifact, - load_weights_from_run_checkpoint, -) +from sequifier.artifacts.model_artifact import read_artifact, restore_model_artifact +from sequifier.model.execution import ModelExecution from sequifier.model.parameter_catalog import ParameterCatalog @@ -28,7 +26,7 @@ class ExecutionOptions: @dataclass class LoadedModel: network: nn.Module - config: Any + config: ModelExecution parameter_catalog: ParameterCatalog artifact_metadata: dict[str, Any] @@ -45,22 +43,12 @@ def load_model_for_analysis( if config is not None: raise ValueError("Current Sequifier artifacts embed their model configuration.") artifact_path = Path(path).expanduser().resolve() - payload = torch.load(artifact_path, map_location="cpu", weights_only=False) - artifact_type = payload.get("artifact_type") if isinstance(payload, dict) else None - if artifact_type == "sequifier_model": - network, resolved_config, artifact = load_model_artifact( - artifact_path, - device=options.device, - interface_name=interface_name, - ) - elif artifact_type == "sequifier_run_checkpoint": - network, resolved_config, artifact = load_weights_from_run_checkpoint( - artifact_path, - device=options.device, - interface_name=interface_name, - ) - else: - raise ValueError(f"Unsupported Sequifier artifact: {artifact_path}.") + artifact, artifact_type = read_artifact(artifact_path) + network, execution, artifact = restore_model_artifact( + artifact, + device=options.device, + interface_name=interface_name, + ) network.train(options.training_mode) from sequifier.model.layers import SelfAttention @@ -74,12 +62,11 @@ def load_model_for_analysis( network = torch.compile(network) return LoadedModel( network=network, - config=resolved_config, + config=execution, parameter_catalog=catalog, artifact_metadata={ "path": str(artifact_path), "artifact_type": artifact_type, - "format_version": artifact.format_version, "metadata": artifact.metadata, "selected_interface": interface_name, }, diff --git a/src/sequifier/artifacts/manifests.py b/src/sequifier/artifacts/manifests.py index e3786629c..890f67fb4 100644 --- a/src/sequifier/artifacts/manifests.py +++ b/src/sequifier/artifacts/manifests.py @@ -1,25 +1,10 @@ -import contextlib -import json -import os -import uuid from pathlib import Path from typing import Any +from sequifier.io.atomic import write_json_atomic from sequifier.typechecking import beartype @beartype def write_manifest(path: str | Path, manifest: dict[str, Any]) -> None: - path = Path(path) - path.parent.mkdir(parents=True, exist_ok=True) - temporary_path = path.with_name(f".{path.name}.{uuid.uuid4().hex}.tmp") - try: - with temporary_path.open("w") as file: - json.dump(manifest, file, indent=2, sort_keys=True, default=str) - file.write("\n") - file.flush() - os.fsync(file.fileno()) - os.replace(temporary_path, path) - finally: - with contextlib.suppress(OSError): - temporary_path.unlink() + write_json_atomic(path, manifest, sort_keys=True, default=str, allow_nan=True) diff --git a/src/sequifier/artifacts/model_artifact.py b/src/sequifier/artifacts/model_artifact.py index 8b4d111c7..2daea4cea 100644 --- a/src/sequifier/artifacts/model_artifact.py +++ b/src/sequifier/artifacts/model_artifact.py @@ -2,9 +2,6 @@ from __future__ import annotations -import contextlib -import os -import uuid from dataclasses import asdict, dataclass, field from pathlib import Path from typing import Any, Mapping @@ -12,16 +9,20 @@ import torch from torch import Tensor -from sequifier.artifacts.model_config import resolved_config_from_model_config +from sequifier.artifacts.model_config import ( + model_execution_from_model_config, + validate_interface_versions, +) from sequifier.artifacts.model_export import model_execution_config from sequifier.artifacts.state_dict import ( canonicalize_state_dict, validate_model_state_contract, ) +from sequifier.contract_versions import ContractVersions, require_contract_version +from sequifier.io.atomic import atomic_save +from sequifier.model.execution import ModelExecution from sequifier.model.factory import build_transformer_network -MODEL_ARTIFACT_FORMAT_VERSION = 2 - @dataclass(frozen=True) class ModelExecutionConfig: @@ -48,21 +49,13 @@ class ModelArtifact: metadata: ModelArtifactMetadata def validate(self) -> None: - if self.format_version not in (1, MODEL_ARTIFACT_FORMAT_VERSION): - raise ValueError( - f"Unsupported model artifact format {self.format_version}; " - f"expected {MODEL_ARTIFACT_FORMAT_VERSION}." - ) - if self.format_version == 1 and any( - i.get("depth_layouts") - for i in self.model_config.values.get("interfaces", {}).values() - ): - raise ValueError("Depth execution requires model artifact format version 2") - for name, interface in self.model_config.values.get("interfaces", {}).items(): - if interface.get("depth_layouts") and "execution_schema" not in interface: - raise ValueError( - f"Depth model interface {name!r} requires an execution schema" - ) + require_contract_version( + "portable model artifact", + self.format_version, + ContractVersions.MODEL_ARTIFACT, + ) + for interface in self.model_config.values.get("interfaces", {}).values(): + validate_interface_versions(interface) validate_model_state_contract(self.model_state_dict) def state_dict(self) -> dict[str, Any]: @@ -79,8 +72,13 @@ def state_dict(self) -> dict[str, Any]: def from_state_dict(cls, payload: Mapping[str, Any]) -> "ModelArtifact": if payload.get("artifact_type") != "sequifier_model": raise ValueError("Artifact is not a portable Sequifier model.") + require_contract_version( + "portable model artifact", + payload.get("format_version"), + ContractVersions.MODEL_ARTIFACT, + ) artifact = cls( - format_version=int(payload["format_version"]), + format_version=payload["format_version"], model_config=ModelExecutionConfig(dict(payload["model_config"])), model_state_dict=canonicalize_state_dict(payload["model_state_dict"]), metadata=ModelArtifactMetadata(**dict(payload.get("metadata", {}))), @@ -105,12 +103,19 @@ def build_model_artifact( f"{interface_name}:{site.name}" for site in network.trace_catalog_for(interface_name) ) + resolved_provenance = dict(provenance or {}) + from sequifier.config.train_config import ResolvedSequifierConfig + + if isinstance(config, ResolvedSequifierConfig) and config.training_plan: + resolved_provenance["training_plan"] = [ + phase.model_dump(mode="json") for phase in config.training_plan + ] artifact = ModelArtifact( - format_version=MODEL_ARTIFACT_FORMAT_VERSION, + format_version=ContractVersions.MODEL_ARTIFACT, model_config=ModelExecutionConfig.from_training_config(config), model_state_dict=state, metadata=ModelArtifactMetadata( - trace_sites=tuple(trace_sites), provenance=dict(provenance or {}) + trace_sites=tuple(trace_sites), provenance=resolved_provenance ), ) artifact.validate() @@ -119,14 +124,7 @@ def build_model_artifact( def save_model_artifact(artifact: ModelArtifact, path: str | Path) -> Path: destination = Path(path).expanduser().resolve() - destination.parent.mkdir(parents=True, exist_ok=True) - temporary = destination.with_name(f".{destination.name}.{uuid.uuid4().hex}.tmp") - try: - torch.save(artifact.state_dict(), temporary) - os.replace(temporary, destination) - finally: - with contextlib.suppress(OSError): - os.remove(temporary) + atomic_save(destination, artifact.state_dict()) return destination @@ -135,16 +133,23 @@ def load_model_artifact( *, device: str = "cpu", interface_name: str | None = None, -) -> tuple[Any, Any, ModelArtifact]: - """Load a portable artifact and return network, resolved config, and schema.""" +) -> tuple[Any, ModelExecution, ModelArtifact]: + """Load a portable artifact and return its network and execution contract.""" - payload = torch.load( - Path(path).expanduser().resolve(), - map_location=torch.device(device), - weights_only=False, + artifact, _ = read_artifact(path, expected_type="sequifier_model") + return restore_model_artifact( + artifact, device=device, interface_name=interface_name ) - artifact = ModelArtifact.from_state_dict(payload) - config, _ = resolved_config_from_model_config( + + +def restore_model_artifact( + artifact: ModelArtifact, + *, + device: str = "cpu", + interface_name: str | None = None, +) -> tuple[Any, ModelExecution, ModelArtifact]: + """Reconstruct execution identically for portable models and run checkpoints.""" + config, _ = model_execution_from_model_config( artifact.model_config.values, device=device, interface_name=interface_name, @@ -159,11 +164,55 @@ def load_model_artifact( return built.network, config, artifact +def read_artifact( + path: str | Path, *, expected_type: str | None = None +) -> tuple[ModelArtifact, str]: + """Read and validate an execution artifact, retaining its source kind.""" + payload = torch.load( + Path(path).expanduser().resolve(), map_location="cpu", weights_only=False + ) + artifact_type = payload.get("artifact_type") if isinstance(payload, dict) else None + if artifact_type in { + "sequifier_run_checkpoint", + "sequifier_policy_run", + "sequifier_learning_run", + }: + from sequifier.artifacts.checkpoint_envelope import validate_checkpoint_envelope + + validate_checkpoint_envelope(payload) + payload = payload.get("model") + if payload is None: + raise ValueError("Checkpoint precedes the first executable model") + if expected_type is not None and artifact_type != expected_type: + raise ValueError(f"Expected {expected_type} artifact, received {artifact_type}") + if not isinstance(payload, dict) or not isinstance(artifact_type, str): + raise ValueError( + "Sequifier model artifact must be a mapping with an artifact_type" + ) + return ModelArtifact.from_state_dict(payload), artifact_type + + +def read_execution_artifact(path: str | Path) -> ModelArtifact: + """Read the portable execution payload from either supported artifact kind.""" + return read_artifact(path)[0] + + +def load_execution_model( + path: str | Path | ModelArtifact, + *, + device: str = "cpu", + interface_name: str | None = None, +) -> tuple[Any, ModelExecution, ModelArtifact]: + """Load a network and its model-owned execution contract.""" + return restore_model_artifact( + path if isinstance(path, ModelArtifact) else read_execution_artifact(path), + device=device, + interface_name=interface_name, + ) + + def model_artifact_from_run_checkpoint(path: str | Path) -> ModelArtifact: - payload = torch.load(Path(path), map_location="cpu", weights_only=False) - if payload.get("artifact_type") != "sequifier_run_checkpoint": - raise ValueError("Artifact is not a Sequifier run checkpoint.") - return ModelArtifact.from_state_dict(payload["model"]) + return read_artifact(path, expected_type="sequifier_run_checkpoint")[0] def load_weights_from_run_checkpoint( @@ -171,18 +220,9 @@ def load_weights_from_run_checkpoint( *, device: str = "cpu", interface_name: str | None = None, -) -> tuple[Any, Any, ModelArtifact]: - artifact = model_artifact_from_run_checkpoint(path) - config, _ = resolved_config_from_model_config( - artifact.model_config.values, +) -> tuple[Any, ModelExecution, ModelArtifact]: + return restore_model_artifact( + model_artifact_from_run_checkpoint(path), device=device, interface_name=interface_name, ) - built = build_transformer_network( - config, device=torch.device(device), initialize=False - ) - built.network.load_state_dict(artifact.model_state_dict) - from sequifier.model.factory import validate_hash_contracts - - validate_hash_contracts(built.network, config) - return built.network, config, artifact diff --git a/src/sequifier/artifacts/model_config.py b/src/sequifier/artifacts/model_config.py index d729aee5c..4edb70d93 100644 --- a/src/sequifier/artifacts/model_config.py +++ b/src/sequifier/artifacts/model_config.py @@ -2,26 +2,97 @@ from __future__ import annotations -from pydantic import TypeAdapter +import json +from sequifier.config.metadata import validate_tensor_payload_contract +from sequifier.config.ownership import ExportSettings +from sequifier.config.policy_config import ResolvedPolicyBinding +from sequifier.config.run_capabilities import PublicationPlan from sequifier.config.train_config import ( BackboneComponentConfig, - DatasetFreezingSpecModel, - DecoderComponentConfig, - FeatureLayoutRegistryModel, - GlobalTrainingSpecModel, - IngestionComponentConfig, ModelInterfaceSpecModel, ModelSpecModel, - ResolvedDatasetTrainingSpec, + ResolvedInterfaceStorage, ResolvedModelInterface, ResolvedSequifierConfig, + TrainingRuntimeSettings, ) -from sequifier.helpers import ModelWindowView, StoredWindowLayout, resolve_window_view +from sequifier.contract_versions import ContractVersions, require_contract_version +from sequifier.helpers import ModelWindowView, resolve_window_view +from sequifier.model.execution import ModelExecution +from sequifier.model.execution_options import ( + DEFAULT_MAX_CONCAT_TENSORS, + ModelExecutionOptions, +) +from sequifier.model.execution_schema import ExecutionSchema +from sequifier.objectives import target_offset_for_objective from sequifier.special_tokens import SPECIAL_TOKEN_IDS from sequifier.typechecking import beartype +def validate_execution_attachment( + source: dict, current_execution: ModelExecution +) -> None: + """Require existing prediction semantics when reusing complete model weights.""" + from pydantic_core import to_jsonable_python + + from sequifier.artifacts.model_export import model_execution_config + + source_config, _ = model_execution_from_model_config(source, device="cpu") + source = model_execution_config(source_config) + current = model_execution_config(current_execution) + if source["backbone"]["architecture"] != current["backbone"]["architecture"]: + raise ValueError("model attachment cannot change backbone architecture") + if source.get("layer_type_dtypes") != current.get("layer_type_dtypes"): + raise ValueError("model attachment cannot silently change parameter precision") + if source["context_length"] != current["context_length"]: + raise ValueError("model attachment cannot change context length") + + if source["forward_attention"] != current["forward_attention"]: + raise ValueError("model attachment cannot change forward attention") + for name, interface in source["interfaces"].items(): + if name not in current["interfaces"]: + raise ValueError(f"initial model interface {name!r} was dropped") + for target, distribution in interface.get( + "prediction_distributions", {} + ).items(): + if to_jsonable_python(distribution) != to_jsonable_python( + current["interfaces"][name] + .get("prediction_distributions", {}) + .get(target) + ): + raise ValueError( + f"initial model prediction parameters differ at {name}.{target}" + ) + for key in ( + "input_columns", + "target_columns", + "ingestion", + "decoder", + "id_maps", + "target_decoder_ids", + "target_global_to_decoder", + "selected_columns_statistics", + "normalize_real_columns", + "depth_layouts", + "static_conditioning", + "feature_layout", + ): + default = ( + {} + if key == "depth_layouts" + else True + if key == "normalize_real_columns" + else None + ) + if to_jsonable_python(interface.get(key, default)) != to_jsonable_python( + current["interfaces"][name].get(key, default) + ): + raise ValueError( + f"initial model execution differs at interfaces.{name}.{key}" + ) + + @beartype def resolved_config_from_model_config( values: dict, @@ -29,7 +100,64 @@ def resolved_config_from_model_config( device: str, interface_name: str | None = None, ) -> tuple[ResolvedSequifierConfig, str]: - """Build an execution-only config and return its selected interface name.""" + """Public compatibility projection of a portable model's execution facts.""" + execution, selected = model_execution_from_model_config( + values, device=device, interface_name=interface_name, validate=False + ) + return resolved_training_config(execution), selected + + +def validate_interface_storage_versions(values: dict, depth_layouts=None) -> None: + storage = values.get("storage_layout") + require_contract_version( + "stored-window layout", + storage.get("version") if isinstance(storage, dict) else None, + ContractVersions.STORED_WINDOW_LAYOUT, + ) + validate_tensor_payload_contract( + values.get("tensor_payload_kind"), + values.get("tensor_payload_version"), + values.get("depth_layouts") if depth_layouts is None else depth_layouts, + projected=True, + ) + + +def split_interface_storage( + values: dict, +) -> tuple[dict, ResolvedInterfaceStorage | None]: + """Read the flat storage fields used by existing artifact and checkpoint formats.""" + if not ResolvedInterfaceStorage.model_fields.keys() & values.keys(): + return values, None + validate_interface_storage_versions(values) + model_values = dict(values) + storage = { + name: model_values.pop(name) + for name in ResolvedInterfaceStorage.model_fields + if name in model_values + } + return model_values, ResolvedInterfaceStorage.model_validate(storage) + + +def validate_interface_versions(interface: dict) -> None: + if ResolvedInterfaceStorage.model_fields.keys() & interface.keys(): + validate_interface_storage_versions(interface) + schema = interface.get("execution_schema") + require_contract_version( + "model execution schema", + schema.get("version") if isinstance(schema, dict) else None, + ContractVersions.MODEL_EXECUTION_SCHEMA, + ) + + +@beartype +def model_execution_from_model_config( + values: dict, + *, + device: str, + interface_name: str | None = None, + validate: bool = True, +) -> tuple[ModelExecution, str]: + """Resolve portable model execution without inventing datasets or losses.""" interface_values = values.get("interfaces") if not isinstance(interface_values, dict) or not interface_values: @@ -44,180 +172,197 @@ def resolved_config_from_model_config( context_length = int(values["context_length"]) target_offset = int(values.get("target_offset", 1)) objective = str(values["training_objective"]) - global_spec = GlobalTrainingSpecModel( - read_format=( - "pt" - if any(i.get("depth_layouts") for i in interface_values.values()) - else "parquet" - ), + options = ModelExecutionOptions( training_objective=objective, + forward_attention=values.get("forward_attention"), context_length=context_length, target_offset=target_offset, - inference_batch_size=1, - batch_size=1, - learning_rate=1e-3, layer_type_dtypes=values.get("layer_type_dtypes"), - next_occurrence_config=values.get("next_occurrence_config"), - torch_compile="none", + max_concat_tensors=values.get("max_concat_tensors", DEFAULT_MAX_CONCAT_TENSORS), ) authored_interfaces = {} - resolved_datasets = {} - fallback_storage_layout = StoredWindowLayout( - window_length=context_length + max(1, target_offset), - max_target_offset=max(1, target_offset), - version=2, - ) + resolved_interfaces = {} + interface_storage = {} window_view = ModelWindowView( context_length=context_length, objective=objective, - target_offset=(0 if objective == "bert" else target_offset), + target_offset=target_offset_for_objective(objective, target_offset), ) interface_order = [interface_name] + [ name for name in interface_values if name != interface_name ] for name in interface_order: interface = interface_values[name] - storage_layout_values = interface.get("storage_layout") - storage_layout = ( - StoredWindowLayout(**storage_layout_values) - if storage_layout_values is not None - else fallback_storage_layout - ) - resolve_window_view(storage_layout, window_view) - ingestion = TypeAdapter(IngestionComponentConfig).validate_python( - interface["ingestion"] - ) - decoder = TypeAdapter(DecoderComponentConfig).validate_python( - interface["decoder"] - ) - feature_layout = ( - FeatureLayoutRegistryModel.model_validate(interface["feature_layout"]) - if interface.get("feature_layout") is not None - else None - ) - authored_interfaces[name] = ModelInterfaceSpecModel( - input_columns=interface["input_columns"], - static_conditioning=interface.get("static_conditioning"), - preconditioning_image_encoder=interface.get( - "preconditioning_image_encoder" - ), - target_columns=interface["target_columns"], - categorical_hashing=interface.get("categorical_hashing", {}), - categorical_decoder_special_tokens=interface.get( - "categorical_decoder_special_tokens", {} - ), - feature_layout=feature_layout, - ingestion=ingestion, - decoder=decoder, - ) + validate_interface_versions(interface) + interface, storage = split_interface_storage(interface) + if storage is not None: + resolve_window_view(storage.storage_layout, window_view) + interface_storage[name] = storage + authored = { + key: value + for key, value in interface.items() + if key in ModelInterfaceSpecModel.model_fields + } + policy_spec = interface.get("policy") + authored_interfaces[name] = ModelInterfaceSpecModel.model_validate(authored) target_decoder_ids = interface.get("target_decoder_ids", {}) target_n_classes = interface.get( "target_n_classes", {column: len(ids) for column, ids in target_decoder_ids.items()}, ) - global_to_decoder = interface.get("target_global_to_decoder", {}) - resolved = ResolvedModelInterface( - depth_layouts=interface.get("depth_layouts", {}), - tensor_payload_version=interface.get("tensor_payload_version", 1), - split_paths=interface.get("split_paths", []), - prediction_aligned_splits=interface.get("prediction_aligned_splits", []), - prediction_alignment_steps=interface.get("prediction_alignment_steps", 1), - window_lengths=interface.get("window_lengths"), - prediction_length=interface.get("prediction_length"), - target_offset=interface.get("target_offset"), - name=name, - input_columns=interface["input_columns"], - static_conditioning=interface.get("static_conditioning"), - static_features_path=interface.get("static_features_path"), - image_sources=interface.get("image_sources", {}), - preconditioning_image=interface.get("preconditioning_image"), - preconditioning_image_encoder=interface.get( - "preconditioning_image_encoder" - ), - static_categorical_columns=interface.get("static_categorical_columns", []), - static_real_columns=interface.get("static_real_columns", []), - target_columns=interface["target_columns"], - categorical_hashing=interface.get("categorical_hashing", {}), - target_column_types=interface["target_column_types"], - column_data_types=interface["column_data_types"], - categorical_columns=interface["categorical_columns"], - real_columns=interface["real_columns"], - categorical_decoder_special_tokens=interface.get( - "categorical_decoder_special_tokens", {} - ), - feature_layout=feature_layout, - ingestion=ingestion, - decoder=decoder, - n_classes=interface.get("n_classes", target_n_classes), - id_maps=interface.get("id_maps", {}), - special_token_ids=interface.get( - "special_token_ids", SPECIAL_TOKEN_IDS.ids_by_label - ), - selected_columns_statistics=interface.get( - "selected_columns_statistics", {} - ), - normalize_real_columns=interface.get("normalize_real_columns", True), - target_decoder_ids=target_decoder_ids, - target_n_classes=target_n_classes, - target_global_to_decoder=global_to_decoder, - categorical_hash_contracts=interface.get("categorical_hash_contracts", {}), - storage_layout=storage_layout, - window_view=window_view, + resolved = ResolvedModelInterface.model_validate( + { + **{ + key: value + for key, value in interface.items() + if key in ResolvedModelInterface.model_fields + }, + "name": name, + "policy": ResolvedPolicyBinding.model_validate(policy_spec) + if policy_spec is not None + else None, + "n_classes": interface.get("n_classes", target_n_classes), + "target_decoder_ids": target_decoder_ids, + "target_n_classes": target_n_classes, + "target_global_to_decoder": interface.get( + "target_global_to_decoder", {} + ), + "id_maps": interface.get("id_maps", {}), + "special_token_ids": interface.get( + "special_token_ids", SPECIAL_TOKEN_IDS.ids_by_label + ), + "categorical_decoder_special_tokens": interface.get( + "categorical_decoder_special_tokens", {} + ), + "window_view": window_view, + } ) - if resolved.depth_layouts and resolved.tensor_payload_version not in {2, 3}: - raise ValueError( - "Depth model execution requires tensor_payload_version 2 or 3" - ) persisted_schema = interface.get("execution_schema") - if persisted_schema is not None: - from sequifier.model.execution_schema import ExecutionSchema - - actual_schema = ExecutionSchema.from_dict(persisted_schema) - expected_schema = ExecutionSchema.from_interface(resolved, context_length) - import json - - if json.dumps(actual_schema.to_dict(), sort_keys=True) != json.dumps( - expected_schema.to_dict(), sort_keys=True - ): - raise ValueError(f"Execution schema contradicts interface {name!r}") - criteria = { - target: ( - "CrossEntropyLoss" - if resolved.target_column_types[target] == "categorical" - else "MSELoss" - ) - for target in resolved.target_columns - } - resolved_datasets[name] = ResolvedDatasetTrainingSpec( - name=name, - model_interface=name, - interface=resolved, - parts={}, - criterion=criteria, - class_share_log_columns=[], - freeze=DatasetFreezingSpecModel(), - ) - model = ModelSpecModel( - backbone=backbone, - interfaces=authored_interfaces, - ) - config = ResolvedSequifierConfig( - project_root=".", - model_name="loaded-model", + actual_schema = ExecutionSchema.from_dict(persisted_schema) + expected_schema = ExecutionSchema.from_interface(resolved, context_length) + if json.dumps(actual_schema.to_dict(), sort_keys=True) != json.dumps( + expected_schema.to_dict(), sort_keys=True + ): + raise ValueError(f"Execution schema contradicts interface {name!r}") + resolved_interfaces[name] = resolved + config = ModelExecution( + ModelSpecModel(backbone=backbone, interfaces=authored_interfaces), + resolved_interfaces, + options, device=device, - seed=0, - global_training=global_spec, - model=model, - dataset_training=resolved_datasets, - training_plan=[], - evaluation_sources=[], - evaluation_monitor=None, - export_generative_model=True, - export_embedding_model=False, + interface_storage=interface_storage, embedding_layer_names=values.get( "embedding_layer_names", ["backbone.final_norm"] ), - export_onnx=False, - export_pt=False, ) - return config, interface_name + return config.validate() if validate else config, interface_name + + +def execution_config(model, interfaces, runtime_settings, *, device="cpu", **options): + """Adapt model execution to the existing public resolved training contract.""" + return resolved_training_config( + ModelExecution( + model, + interfaces, + ModelExecutionOptions.from_training(runtime_settings), + device=device, + ), + **options, + ) + + +def resolved_training_config( + execution: ModelExecution, **options +) -> ResolvedSequifierConfig: + """Project a model onto the resolved training schema for compatibility callers.""" + values = { + "project_root": execution.project_root, + "model_name": execution.model_name, + "seed": execution.seed, + "device": execution.device, + "runtime_settings": TrainingRuntimeSettings( + **execution.options.model_dump(), torch_compile="none" + ), + "model_options": execution.options, + "model": execution.model, + "interfaces": execution.interfaces, + "interface_storage": execution.interface_storage, + "dataset_training": {}, + "training_plan": [], + "publication": PublicationPlan( + exports=ExportSettings( + pt=False, embedding_layer_names=execution.embedding_layer_names + ) + ), + } + return ResolvedSequifierConfig(**{**values, **options}) + + +def extend_prediction_distributions(execution, declared_model): + """Allow explicit new scale parameters while preserving an initialized model.""" + exclude = {"interfaces": {"__all__": {"prediction_distributions"}}} + from sequifier.config.train_config import OwnedModelSpec + + if isinstance(declared_model, OwnedModelSpec): + declared_model = declared_model.architecture_model() + declared = declared_model.model_dump(exclude=exclude) + existing = execution.model.model_dump(exclude=exclude) + for name, interface in declared["interfaces"].items(): + if name in existing["interfaces"]: + tokens = interface["categorical_decoder_special_tokens"] + for target, value in existing["interfaces"][name][ + "categorical_decoder_special_tokens" + ].items(): + tokens.setdefault(target, value) + if declared != existing: + raise ValueError( + "declared model architecture differs from initialize_from model" + ) + for name, declared in declared_model.interfaces.items(): + interface = execution.interface(name) + distributions = dict(interface.prediction_distributions) + for target, field in declared.prediction_distributions.items(): + if target in distributions and distributions[target] != field: + raise ValueError( + f"{name}.{target}: declared prediction parameters differ from initialize_from" + ) + if ( + target not in interface.target_columns + or interface.target_column_types.get(target) != "real" + ): + raise ValueError( + f"{name}.{target}: prediction_distributions requires a real model target" + ) + distributions[target] = field + interface.prediction_distributions = distributions + execution.model.interfaces[name].prediction_distributions = dict(distributions) + return execution + + +def load_attached_model_weights(network, artifact): + """Restore source weights, initializing only newly declared model scale parameters.""" + from sequifier.model.prediction_distribution import PredictionScales + + expected = network.state_dict() + source = artifact.model_state_dict + missing = expected.keys() - source.keys() + allowed = set() + for name, route in network.interfaces.items(): + scales = route.prediction_scales + if not isinstance(scales, PredictionScales): + continue + original = artifact.model_config.values["interfaces"][name].get( + "prediction_distributions", {} + ) + allowed.update( + f"interfaces.{name}.prediction_scales.log_scales.{target}" + for target in scales.log_scales + if target not in original + ) + if missing - allowed: + raise ValueError( + f"initial model is missing weights: {sorted(missing - allowed)}" + ) + network.load_state_dict( + {**{key: expected[key] for key in missing}, **source}, strict=True + ) diff --git a/src/sequifier/artifacts/model_export.py b/src/sequifier/artifacts/model_export.py index f99a8e4fe..03dbb0174 100644 --- a/src/sequifier/artifacts/model_export.py +++ b/src/sequifier/artifacts/model_export.py @@ -1,111 +1,53 @@ from typing import Any +from sequifier.model.execution import ModelExecution +from sequifier.model.execution_options import DEFAULT_MAX_CONCAT_TENSORS +from sequifier.model.execution_schema import ExecutionSchema from sequifier.typechecking import beartype @beartype -def model_execution_config(training_config: Any) -> dict[str, Any]: - """Return only configuration required to reconstruct model execution.""" - - if not hasattr(training_config, "dataset_training"): - raise TypeError("Model export requires a canonical training config") - from sequifier.model.execution_schema import ExecutionSchema +def model_execution_config(config: Any) -> dict[str, Any]: + """Serialize the execution contract and its optional inference storage metadata.""" + execution = ModelExecution.from_config(config) + spec = execution.options interfaces = {} - for dataset in training_config.dataset_training.values(): - interface = dataset.interface - if interface.name in interfaces: - continue - interfaces[interface.name] = { - "execution_schema": ExecutionSchema.from_interface( - interface, training_config.global_training.context_length - ).to_dict(), - "depth_layouts": interface.depth_layouts.model_dump(mode="json"), - "tensor_payload_version": interface.tensor_payload_version, - "split_paths": interface.split_paths, - "prediction_aligned_splits": interface.prediction_aligned_splits, - "prediction_alignment_steps": interface.prediction_alignment_steps, - "window_lengths": interface.window_lengths, - "prediction_length": interface.prediction_length, - "target_offset": interface.target_offset, - "input_columns": interface.input_columns, - "static_conditioning": ( - interface.static_conditioning.model_dump(mode="python") - if interface.static_conditioning is not None - else None - ), - "static_features_path": interface.static_features_path, - "image_sources": { - name: source.model_dump(mode="python", exclude={"path"}) - for name, source in interface.image_sources.items() - }, - "preconditioning_image": ( - interface.preconditioning_image.model_dump( - mode="python", exclude={"path"} - ) - if interface.preconditioning_image is not None - else None - ), - "preconditioning_image_encoder": ( - interface.preconditioning_image_encoder.model_dump(mode="python") - if interface.preconditioning_image_encoder is not None - else None - ), - "static_categorical_columns": interface.static_categorical_columns, - "static_real_columns": interface.static_real_columns, - "target_columns": interface.target_columns, - "categorical_hashing": { - column: config.model_dump(mode="python") - for column, config in interface.categorical_hashing.items() - }, - "categorical_hash_contracts": interface.categorical_hash_contracts, - "target_column_types": interface.target_column_types, - "column_data_types": interface.column_data_types, - "categorical_columns": interface.categorical_columns, - "real_columns": interface.real_columns, - "categorical_decoder_special_tokens": ( - interface.categorical_decoder_special_tokens - ), - "feature_layout": ( - interface.feature_layout.model_dump(mode="python") - if interface.feature_layout is not None - else None - ), - "ingestion": interface.ingestion.model_dump( - mode="python", - ), - "decoder": interface.decoder.model_dump( - mode="python", - ), - "n_classes": interface.n_classes, - "id_maps": interface.id_maps, - "special_token_ids": interface.special_token_ids, - "selected_columns_statistics": (interface.selected_columns_statistics), - "normalize_real_columns": interface.normalize_real_columns, - "target_decoder_ids": interface.target_decoder_ids, - "target_n_classes": interface.target_n_classes, - "target_global_to_decoder": interface.target_global_to_decoder, - "storage_layout": { - "window_length": (interface.storage_layout.window_length), - "max_target_offset": interface.storage_layout.max_target_offset, - "version": interface.storage_layout.version, + for name, interface in execution.interfaces.items(): + values = interface.model_dump( + mode="python", + exclude={ + "name": True, + "window_view": True, + "policy": True, + "image_sources": {"__all__": {"path"}}, + "preconditioning_image": {"paths"}, }, - } - spec = training_config.global_training + ) + values["execution_schema"] = ExecutionSchema.from_interface( + interface, spec.context_length + ).to_dict() + values["depth_layouts"] = interface.depth_layouts.model_dump(mode="json") + if (storage := execution.interface_storage.get(name)) is not None: + values.update(storage.model_dump(mode="python")) + interfaces[name] = values + if interface.policy is not None: + values["policy"] = interface.policy.model_dump(mode="json") return { "training_objective": spec.training_objective, + "forward_attention": spec.attention, + **( + {"max_concat_tensors": spec.max_concat_tensors} + if spec.max_concat_tensors != DEFAULT_MAX_CONCAT_TENSORS + else {} + ), "context_length": spec.context_length, "target_offset": spec.target_offset, - "next_occurrence_config": ( - spec.next_occurrence_config.model_dump(mode="python") - if spec.next_occurrence_config is not None - else None - ), - "backbone": training_config.model.backbone.model_dump( + "backbone": execution.model.backbone.model_dump( mode="python", exclude={"repository"}, ), - "embedding_layer_names": list(training_config.embedding_layer_names), + "embedding_layer_names": list(execution.embedding_layer_names), "layer_type_dtypes": spec.layer_type_dtypes, "interfaces": interfaces, } diff --git a/src/sequifier/artifacts/phase.py b/src/sequifier/artifacts/phase.py new file mode 100644 index 000000000..6bd4853c5 --- /dev/null +++ b/src/sequifier/artifacts/phase.py @@ -0,0 +1,124 @@ +"""Selected models and sidecar identities at committed phase boundaries.""" + +from dataclasses import asdict, dataclass, replace +from pathlib import Path +from shutil import copyfile +from uuid import uuid4 + +from sequifier.contract_versions import ContractVersions, require_contract_version +from sequifier.fingerprints import fingerprint +from sequifier.io.atomic import atomic_path, write_json_atomic +from sequifier.trajectory.sources import file_checksum + + +@dataclass(frozen=True) +class PhaseArtifacts: + model: Path + + @classmethod + def for_phase(cls, directory, index, name, filename="model.pt"): + return cls(Path(directory) / f"{index:03d}-{name}" / filename) + + @property + def report(self): + return self.model.with_name("report.json") + + @property + def telemetry(self): + return self.model.parent / "telemetry" / "records.sqlite3" + + def describe(self): + return {"actor_path": str(self.model), "report_path": str(self.report)} + + +@dataclass(frozen=True) +class PhaseCommit: + """An immutable model and report payload published by a run checkpoint.""" + + model_path: str + model_sha256: str + result: dict + result_fingerprint: str + sidecars: dict + format_version: int = ContractVersions.PHASE_COMMIT + + @classmethod + def stage(cls, artifacts, result, model, *, telemetry=None): + from sequifier.artifacts.model_artifact import save_model_artifact + + path = artifacts.model.parent / "commits" / f"{uuid4().hex}.pt" + save_model_artifact(model, path) + sidecars = {"telemetry": telemetry} if telemetry is not None else {} + return cls( + str(path), + file_checksum(path), + result, + fingerprint({"result": result, "sidecars": sidecars}), + sidecars, + ) + + def with_result(self, result): + return replace( + self, + result=result, + result_fingerprint=fingerprint( + {"result": result, "sidecars": self.sidecars} + ), + ) + + def as_dict(self): + return asdict(self) + + @classmethod + def load(cls, values): + commit = cls(**values) + require_contract_version( + "phase commit", commit.format_version, ContractVersions.PHASE_COMMIT + ) + if fingerprint({"result": commit.result, "sidecars": commit.sidecars}) != ( + commit.result_fingerprint + ): + raise ValueError("Committed phase result changed") + verify_artifact( + commit.model_path, commit.model_sha256, "Committed phase model changed" + ) + return commit + + @property + def model_identity(self): + return {"path": self.model_path, "sha256": self.model_sha256} + + def report_result(self): + from sequifier.rl.telemetry import TelemetryJournal + + result = dict(self.result) + if result["algorithm"] == "supervised": + return result + sidecar = self.sidecars.get("telemetry") + history = ( + TelemetryJournal.read(Path(sidecar["path"]), sidecar["position"]) + if sidecar + else {} + ) + result["metrics"] = history.get("metrics", []) + if result["algorithm"] in {"ppo", "grpo"}: + result["provenance"] = { + **result["provenance"], + **{ + key: history.get(key, []) + for key in ("evaluations", "continuations") + }, + } + return result + + def publish(self, artifacts, report): + path = artifacts.model + if not path.is_file() or file_checksum(path) != self.model_sha256: + with atomic_path(path, durable=True) as temporary: + copyfile(self.model_path, temporary) + write_json_atomic(artifacts.report, report) + + +def verify_artifact(path, checksum, message): + if path and file_checksum(path) != checksum: + raise ValueError(message) diff --git a/src/sequifier/artifacts/reports.py b/src/sequifier/artifacts/reports.py new file mode 100644 index 000000000..26464789b --- /dev/null +++ b/src/sequifier/artifacts/reports.py @@ -0,0 +1,58 @@ +"""Common, versioned discovery records for every training runner.""" + +from pathlib import Path + +from sequifier.contract_versions import ContractVersions +from sequifier.io.atomic import write_json_atomic + + +def phase_report(*, index, name, algorithm, model_path, checkpoint_path, **details): + return { + "artifact_type": "sequifier_phase_report", + "index": index, + "name": name, + "algorithm": algorithm, + "model_path": str(model_path), + "checkpoint_path": str(checkpoint_path), + "monitor": None, + "direction": None, + "handoff": "last", + "selected_metric": None, + "selected_transaction": None, + "selected_weight_hash": None, + "clocks": {}, + "metrics": [], + "provenance": {}, + "stop_scope": None, + "completion_reason": "completed", + **details, + "format_version": ContractVersions.PHASE_REPORT, + } + + +def run_report(*, model_path, checkpoint_path, phases, **details): + return { + "artifact_type": "sequifier_run_manifest", + "model_path": str(model_path), + "checkpoint_path": str(checkpoint_path), + "phases": list(phases), + "run_id": None, + "session_id": None, + "status": "complete", + "completion_reason": "completed", + "exports": [], + "source_provenance": {}, + "initialization": {}, + "resume": {}, + **details, + "format_version": ContractVersions.RUN_MANIFEST, + } + + +def publish_run_report(report, path): + """Keep the supervised manifest discovery location on every route.""" + destination = Path(path).resolve() + manifest = Path(report["checkpoint_path"]).resolve().parent / "manifest.json" + write_json_atomic(destination, report) + if manifest != destination: + write_json_atomic(manifest, report) diff --git a/src/sequifier/artifacts/run_checkpoint.py b/src/sequifier/artifacts/run_checkpoint.py index 745daf666..7f960e4aa 100644 --- a/src/sequifier/artifacts/run_checkpoint.py +++ b/src/sequifier/artifacts/run_checkpoint.py @@ -2,10 +2,8 @@ from __future__ import annotations -import contextlib -import os -import uuid from collections.abc import Mapping +from copy import deepcopy from dataclasses import asdict, dataclass from pathlib import Path from typing import Any @@ -13,8 +11,121 @@ import torch from sequifier.artifacts.model_artifact import ModelArtifact +from sequifier.contract_versions import ContractVersions, require_contract_version +from sequifier.io.atomic import atomic_save -RUN_CHECKPOINT_FORMAT_VERSION = 2 + +def read_training_plan(saved: dict): + """Translate legacy embedded interface views at the recovery boundary.""" + from sequifier.artifacts.model_config import ( + split_interface_storage, + validate_interface_storage_versions, + ) + from sequifier.config.metadata import validate_persisted_metadata_versions + from sequifier.config.run_capabilities import PublicationPlan + from sequifier.config.train_config import ( + ResolvedSequifierConfig, + TrainingRuntimeSettings, + ) + from sequifier.model.execution_options import ModelExecutionOptions + + values = deepcopy(saved) + interfaces = values["interfaces"] + storage = values.setdefault("interface_storage", {}) + for name, interface in interfaces.items(): + model_interface, legacy_storage = split_interface_storage(interface) + interfaces[name] = model_interface + if legacy_storage is not None: + storage.setdefault(name, legacy_storage.model_dump()) + if name in storage: + validate_interface_storage_versions( + storage[name], model_interface.get("depth_layouts") or {} + ) + datasets = values["dataset_training"] + for dataset in datasets.values(): + if (legacy_view := dataset.pop("interface", None)) is not None: + _, legacy_storage = split_interface_storage(legacy_view) + if legacy_storage is not None: + dataset.setdefault("storage", legacy_storage.model_dump()) + if (dataset_storage := dataset.get("storage")) is not None: + interface = interfaces[dataset["model_interface"]] + validate_interface_storage_versions( + dataset_storage, interface.get("depth_layouts") or {} + ) + for part in dataset["parts"].values(): + validate_persisted_metadata_versions(part["metadata"]) + + if "model_metadata_paths" not in values: + authored_interfaces = (values.get("ownership", {}).get("model") or {}).get( + "interfaces", {} + ) + values["model_metadata_paths"] = { + name: str(Path(values["project_root"]) / path) + for name, spec in authored_interfaces.items() + if (path := spec.get("input_contract", {}).get("metadata_config_path")) + } + if "model_options" not in values: + values["model_options"] = ModelExecutionOptions.from_training( + TrainingRuntimeSettings.model_validate(values["runtime_settings"]) + ) + if "publication" not in values: + values["publication"] = PublicationPlan( + exports={ + "generative_model": values.pop("export_generative_model"), + "embedding_model": values.pop("export_embedding_model"), + "embedding_layer_names": values.pop("embedding_layer_names"), + "onnx": values.pop("export_onnx"), + "pt": values.pop("export_pt"), + "with_dropout": values.pop("export_with_dropout", False), + "batch_size": values["runtime_settings"].get("inference_batch_size", 1), + } + ) + for phase in values["training_plan"]: + if phase["algorithm"] == "supervised": + for source in phase["sources"] + phase.get("evaluation_sources", []): + if "parts" not in source: + source["parts"] = ( + [source["part"]] + if source.get("part") is not None + else list(datasets[source["dataset"]]["parts"]) + ) + return ResolvedSequifierConfig.model_validate(values) + + +def training_plan_identity(config): + """Retain the flat fingerprint contract used by existing coordinated runs.""" + from sequifier.fingerprints import fingerprint + + values = config.model_dump(mode="json") + values.pop("model_options") + values.pop("model_metadata_paths") + storage = values.pop("interface_storage") + for name, interface in values["interfaces"].items(): + interface.update(storage.get(name, {})) + for name, dataset in values["dataset_training"].items(): + dataset["interface"] = { + **config.dataset_interface(name).model_dump(mode="json"), + **config.dataset_storage(name).model_dump(mode="json"), + } + dataset.pop("storage") + exports = values.pop("publication")["exports"] + values.update( + export_generative_model=exports["generative_model"], + export_embedding_model=exports["embedding_model"], + embedding_layer_names=exports["embedding_layer_names"], + export_onnx=exports["onnx"], + export_pt=exports["pt"], + export_with_dropout=exports["with_dropout"], + ) + if values["actor_initialization"] == "attach": + values.pop("actor_initialization") + for phase in values["training_plan"]: + if phase["algorithm"] == "supervised": + for source in phase["sources"] + phase["evaluation_sources"]: + source.pop("parts") + values["runtime_settings"].pop("resume", None) + values.get("ownership", {}).get("run", {}).get("recovery", {}).pop("resume", None) + return fingerprint(values) @dataclass(frozen=True) @@ -51,19 +162,43 @@ class RunCheckpoint: loader_state: LoaderState integration_state: IntegrationState training_config: Any + dependencies: dict[str, str] def validate(self) -> None: - if self.format_version != RUN_CHECKPOINT_FORMAT_VERSION: - raise ValueError( - f"Unsupported run checkpoint format {self.format_version}; " - f"expected {RUN_CHECKPOINT_FORMAT_VERSION}." - ) + require_contract_version( + "supervised recovery", + self.format_version, + ContractVersions.SUPERVISED_RECOVERY, + ) self.model.validate() + def phase_boundary(self): + from sequifier.artifacts.checkpoint_envelope import ( + phase_boundary, + selected_model_identity, + ) + + state = self.run_state + completed = state["phase_commits"] + active = ( + { + "index": state.get("phase_offset", 0) + state["phase_index"], + "transaction": state["optimizer_step"], + } + if len(completed) <= state["phase_index"] + else None + ) + return phase_boundary( + state["run_id"], selected=selected_model_identity(completed), active=active + ) + def state_dict(self) -> dict[str, Any]: + from sequifier.artifacts.checkpoint_envelope import checkpoint_envelope + self.validate() - return { + recovery = { "artifact_type": "sequifier_run_checkpoint", + "recovery_type": "supervised", "format_version": self.format_version, "model": self.model.state_dict(), "optimization": asdict(self.optimization), @@ -71,39 +206,64 @@ def state_dict(self) -> dict[str, Any]: "random_state": {"states": self.random_state.states}, "loader_state": {"parts": self.loader_state.parts}, "integration_state": {"values": self.integration_state.values}, - "initialization_policy_version": 1, - "initialization_seed_derivation_version": 1, + "initialization_policy_version": ContractVersions.INITIALIZATION_POLICY, + "initialization_seed_derivation_version": ContractVersions.INITIALIZATION_SEED_DERIVATION, + "dependencies": self.dependencies, "training_config": self.training_config.model_dump(mode="python"), } + return checkpoint_envelope( + "supervised", recovery, self.model, boundary=self.phase_boundary() + ) @classmethod def from_state_dict(cls, payload: Mapping[str, Any]) -> "RunCheckpoint": + from sequifier.artifacts.checkpoint_envelope import ( + recovery_payload, + require_phase_boundary, + ) + + envelope = payload + payload = recovery_payload(envelope, "supervised") + require_contract_version( + "supervised recovery", + payload.get("format_version"), + ContractVersions.SUPERVISED_RECOVERY, + ) if payload.get("artifact_type") != "sequifier_run_checkpoint": - raise ValueError("Unsupported checkpoint: expected new Sequifier format.") - from sequifier.config.train_config import ResolvedSequifierConfig - - if ( - payload.get("initialization_policy_version") != 1 - or payload.get("initialization_seed_derivation_version") != 1 - ): - raise ValueError("Unsupported initialization provenance version") + raise ValueError("Expected a supervised checkpoint recovery payload") + if payload.get("recovery_type") != "supervised": + raise ValueError("Checkpoint belongs to another training runner") + + require_contract_version( + "initialization policy", + payload.get("initialization_policy_version"), + ContractVersions.INITIALIZATION_POLICY, + ) + require_contract_version( + "initialization seed derivation", + payload.get("initialization_seed_derivation_version"), + ContractVersions.INITIALIZATION_SEED_DERIVATION, + ) + model = ModelArtifact.from_state_dict(payload["model"]) + training_config = read_training_plan(payload["training_config"]) + loader_parts = payload["loader_state"]["parts"] checkpoint = cls( - format_version=int(payload["format_version"]), - model=ModelArtifact.from_state_dict(payload["model"]), + format_version=payload["format_version"], + model=model, optimization=OptimizationState(**dict(payload["optimization"])), run_state=dict(payload["run_state"]), random_state=DistributedRandomState( states=tuple(payload["random_state"]["states"]) ), - loader_state=LoaderState(parts=dict(payload["loader_state"]["parts"])), + loader_state=LoaderState(parts=dict(loader_parts)), integration_state=IntegrationState( values=dict(payload["integration_state"]["values"]) ), - training_config=ResolvedSequifierConfig.model_validate( - payload["training_config"] - ), + training_config=training_config, + dependencies=dict(payload["dependencies"]), ) checkpoint.validate() + require_phase_boundary(envelope, checkpoint.phase_boundary()) return checkpoint @@ -113,16 +273,32 @@ class RunCheckpointStore: def __init__(self, training_config: Any, model_name: str) -> None: self.latest_path = checkpoint_path(training_config) self.model_name = model_name - - def path_for(self, suffix: str | None) -> Path: - return ( - self.latest_path - if suffix in {None, "latest"} - else self.latest_path.with_name(f"{self.model_name}-{suffix}.pt") + self.phase_directory = training_config.phase_directory + self.phases = training_config.training_plan + + def path_for( + self, suffix: str | None, phase_index: int = 0, *, phase_offset: int = 0 + ) -> Path: + if suffix in {None, "latest"}: + return self.latest_path + from sequifier.artifacts.phase import PhaseArtifacts + + directory = ( + PhaseArtifacts.for_phase( + self.phase_directory, + phase_offset + phase_index, + self.phases[phase_index].name, + ).model.parent + / "checkpoints" ) + return directory / f"{self.model_name}-{suffix}.pt" def save(self, checkpoint: RunCheckpoint, suffix: str | None) -> Path: - output_path = self.path_for(suffix) + output_path = self.path_for( + suffix, + checkpoint.run_state["phase_index"], + phase_offset=checkpoint.run_state.get("phase_offset", 0), + ) output_path.parent.mkdir(parents=True, exist_ok=True) write_run_checkpoint(checkpoint, output_path, self.latest_path) return output_path @@ -142,29 +318,14 @@ def write_run_checkpoint( checkpoint: RunCheckpoint, output_path: Path, latest_path: Path ) -> None: payload = checkpoint.state_dict() - temporary = output_path.with_name(f".{output_path.name}.{uuid.uuid4().hex}.tmp") - try: - torch.save(payload, temporary) - os.replace(temporary, output_path) - finally: - with contextlib.suppress(OSError): - os.remove(temporary) - if output_path == latest_path: - return - latest_path.parent.mkdir(parents=True, exist_ok=True) - latest_temporary = latest_path.with_name( - f".{latest_path.name}.{uuid.uuid4().hex}.tmp" - ) - try: - torch.save(payload, latest_temporary) - os.replace(latest_temporary, latest_path) - finally: - with contextlib.suppress(OSError): - os.remove(latest_temporary) + atomic_save(output_path, payload) + if output_path != latest_path: + atomic_save(latest_path, payload) -def checkpoint_path(training_config: Any) -> Path: - resume = training_config.global_training.resume +def checkpoint_path(training_config: Any, *, resume=None) -> Path: + if resume is None: + resume = training_config.runtime_settings.resume configured_path = resume.checkpoint_path if resume is not None else None if configured_path: path = Path(configured_path) @@ -181,13 +342,37 @@ def checkpoint_path(training_config: Any) -> Path: return path.resolve() -def select_run_checkpoint(training_config: Any) -> Path | None: - resume = training_config.global_training.resume +def select_run_checkpoint(training_config: Any, *, resume=None) -> Path | None: + if resume is None: + resume = training_config.runtime_settings.resume if resume is None or resume.policy == "never": return None - path = checkpoint_path(training_config) + path = checkpoint_path(training_config, resume=resume) if path.is_file(): return path if resume.policy == "required": raise FileNotFoundError(f"Required run checkpoint does not exist: {path}") return None + + +def read_resume_seed(run) -> int | None: + """Read the saved run seed before resolving seed-dependent inputs.""" + from sequifier.artifacts.checkpoint_envelope import validate_checkpoint_envelope + + path = select_run_checkpoint(run, resume=run.recovery.resume) + if path is None: + return None + envelope = torch.load(path, map_location="cpu", weights_only=False) + validate_checkpoint_envelope(envelope) + recovery = envelope["recovery"] + if envelope["recovery_type"] == "supervised": + recovery = recovery["training_config"] + elif envelope["recovery_type"] == "policy": + recovery = recovery["plan"] + seed = recovery.get("seed") + if not isinstance(seed, int): + raise ValueError( + f"Checkpoint {path} has no valid run seed. " + "Set run.reproducibility.seed to the original integer." + ) + return seed diff --git a/src/sequifier/config/canonical_hyperparameter_search_config.py b/src/sequifier/config/canonical_hyperparameter_search_config.py index ebff5c2cc..037226155 100644 --- a/src/sequifier/config/canonical_hyperparameter_search_config.py +++ b/src/sequifier/config/canonical_hyperparameter_search_config.py @@ -17,33 +17,35 @@ from decimal import Decimal from typing import Any, Literal -from pydantic import ( - BaseModel, - ConfigDict, - Field, - PrivateAttr, - ValidationError, - model_validator, -) +from pydantic import ConfigDict, Field, PrivateAttr, ValidationError, model_validator from sequifier.config.composition import load_composed_yaml_config -from sequifier.config.train_config import ( - SequifierConfig, - load_train_config_with_source, - normalize_train_config_parameter_surface, -) +from sequifier.config.naming import ConfigModel, RunSeed +from sequifier.config.train_config import SequifierConfig, load_train_config_with_source from sequifier.typechecking import beartype + +def _validate_training_candidate(values: dict[str, Any]) -> SequifierConfig: + from sequifier.config.train_config import validate_training_contract + + candidate = SequifierConfig.model_validate(values) + validate_training_contract(candidate, candidate.training_plan.phases) + is_next = candidate.runtime_settings.training_objective == "next_occurrence" + for phase in candidate.training_plan.phases: + if phase.algorithm != "supervised": + continue + if (phase.learning.next_occurrence_config is not None) != is_next: + raise ValueError( + f"{phase.name}: learning.next_occurrence_config must be set " + "exactly for next_occurrence training" + ) + return candidate + + ConfigPath = tuple[str | int, ...] _MISSING = object() _DISTRIBUTION_KEYS = frozenset({"low", "high", "step", "log", "type"}) _PRIMITIVE_CATEGORICAL_TYPES = (str, int, float, bool, type(None)) -_NAME_DISCRIMINATOR_PATHS = frozenset( - { - ("global_training", "optimizer"), - ("global_training", "scheduler"), - } -) @beartype @@ -293,11 +295,15 @@ def spaces(self) -> list[_SearchSpace]: def _component_discriminator(path: ConfigPath) -> str | None: """Return the discriminator for a typed component mapping, if any.""" - if path in _NAME_DISCRIMINATOR_PATHS: + if ( + len(path) >= 3 + and path[-1] in {"optimizer", "scheduler"} + and "optimization" in path + ): return "name" if path and path[-1] in {"weight", "bias"} and "initialization" in path: return "method" - if path == ("global_training", "bert_spec", "span_masking"): + if path == ("runtime_settings", "bert_spec", "span_masking"): return "type" if len(path) < 4 or path[:2] != ("model", "interfaces"): return None @@ -553,26 +559,31 @@ def _list_index_mapping(expression: dict[Any, Any]) -> dict[int, Any] | None: @beartype def _compile_value(expression: Any, base: Any, path: ConfigPath) -> _CompiledValue: - if ( - isinstance(expression, dict) - and set(expression) in ({"choices"}, {"$choices"}) - and not (isinstance(base, dict) and next(iter(expression)) in base) - ): - key = "choices" if "choices" in expression else "$choices" - return _categorical_space(path, expression[key]) - if ( - isinstance(expression, dict) - and set(expression) in ({"fixed"}, {"$fixed"}) - and not (isinstance(base, dict) and next(iter(expression)) in base) - ): - key = "fixed" if "fixed" in expression else "$fixed" - return _LiteralValue(expression[key]) - if ( - isinstance(expression, dict) - and not isinstance(base, dict) - and _is_distribution(expression) - ): - return _distribution_space(path, expression, base) + if isinstance(expression, dict): + if set(expression) == {"$choices"}: + return _categorical_space(path, expression["$choices"]) + if set(expression) == {"$fixed"}: + if not isinstance(expression["$fixed"], (dict, list)): + raise ValueError(f"{_path_name(path)}: use primitive scalars directly") + return _LiteralValue(expression["$fixed"]) + if set(expression) == {"$range"}: + spec = expression["$range"] + if not isinstance(spec, dict) or not _is_distribution(spec): + raise ValueError( + f"{_path_name(path)}.$range requires low, high and optional step/log/type" + ) + return _distribution_space(path, spec, base) + if _is_distribution(expression) and not isinstance(base, dict): + raise ValueError( + f"{_path_name(path)}: wrap numeric distributions in $range" + ) + unsupported = { + key for key in expression if isinstance(key, str) and key.startswith("$") + } - {"$variants"} + if unsupported: + raise ValueError( + f"{_path_name(path)}: invalid search operator combination {sorted(unsupported)}" + ) if isinstance(base, list) and isinstance(expression, dict): indexed = _list_index_mapping(expression) @@ -594,12 +605,7 @@ def _compile_value(expression: Any, base: Any, path: ConfigPath) -> _CompiledVal base_mapping = copy.deepcopy(base) if isinstance(base, dict) else {} if isinstance(base, dict) and _changes_discriminator(base, expression, path): base_mapping = {} - variant_keys = {"variants", "$variants"} & set(expression) - if len(variant_keys) > 1: - raise ValueError( - f"{_path_name(path)} cannot configure both variants and $variants" - ) - variant_key = next(iter(variant_keys), None) + variant_key = "$variants" if "$variants" in expression else None child_expressions = { key: value for key, value in expression.items() if key != variant_key } @@ -633,22 +639,24 @@ def _compile_value(expression: Any, base: Any, path: ConfigPath) -> _CompiledVal return _MappingValue(path, base_mapping, mapping_children) if isinstance(expression, list): - if base is _MISSING: - return _LiteralValue(expression) - if isinstance(base, list): - # A list of lists is the established shorthand for selecting a - # complete list-valued field (for example input_columns). Other - # list-valued parameters are fixed replacements; ``choices`` is the - # unambiguous form for sampling arbitrary list values. - if expression and all(isinstance(value, list) for value in expression): - return _categorical_space(path, expression) - return _LiteralValue(expression) - return _categorical_space(path, expression) + raise ValueError( + f"{_path_name(path)}: use $choices to sample or $fixed to replace a list" + ) return _LiteralValue(expression) -class CanonicalHyperparameterSearchConfig(BaseModel): +class SearchOutputSettings(ConfigModel): + trial_config_directory: str = Field(min_length=1) + + +class SearchRunSettings(ConfigModel): + project_root: str = Field(min_length=1) + study_name: str = Field(min_length=1) + outputs: SearchOutputSettings + + +class CanonicalHyperparameterSearchConfig(ConfigModel): """Search controls and a recursive sampler over one canonical train config.""" model_config = ConfigDict( @@ -659,27 +667,26 @@ class CanonicalHyperparameterSearchConfig(BaseModel): base_config_path: str = Field(min_length=1) parameters: dict[str, Any] - project_root: str = Field(min_length=1) - - name: str = Field(min_length=1) - method: Literal["bayesian", "sample", "grid"] = "bayesian" - global_seed: int | None = None + run: SearchRunSettings + method: Literal["bayesian", "random", "grid"] = "bayesian" + sampler_seed: RunSeed | None = None trials: int | None = Field(default=None, gt=0) prune_trials: bool = True pruning_warmup_epochs: int | None = Field(default=None, ge=0) pruning_warmup_batches: int | None = Field(default=None, ge=0) - model_config_write_path: str = Field(min_length=1) - evaluation_inference_config: str | None = None - evaluation_script: str | None = None - evaluation_metric_directions: list[Literal["minimize", "maximize"]] | None = None + evaluation_inference_config_path: str | None = None + evaluation_script_path: str | None = None + evaluation_metric_directions: list[Literal["min", "max"]] | None = None evaluation_metrics: list[str] | None = None _compiled_config: _CompiledValue = PrivateAttr() @model_validator(mode="after") @beartype def validate_search_controls(self): - reserved_parameters = {"model_name", "project_root"} & set(self.parameters) + reserved_parameters = {"model_name", "project_root"} & set( + self.parameters.get("run", {}) + ) if reserved_parameters: raise ValueError( "Search parameters cannot set run-controlled " @@ -696,9 +703,9 @@ def validate_search_controls(self): if self.evaluation_metrics is not None: if not self.evaluation_metrics: raise ValueError("evaluation_metrics cannot be empty") - if self.evaluation_script is None: + if self.evaluation_script_path is None: raise ValueError( - "evaluation_script must be provided if evaluation_metrics " + "evaluation_script_path must be provided if evaluation_metrics " "is defined." ) if self.evaluation_metric_directions is None: @@ -711,26 +718,26 @@ def validate_search_controls(self): "evaluation_metrics and evaluation_metric_directions must have " "the same number of values" ) - if self.evaluation_inference_config is None: + if self.evaluation_inference_config_path is None: warnings.warn( - "Please provide evaluation_inference_config if your " - "evaluation_script requires inference outputs", + "Please provide evaluation_inference_config_path if your " + "evaluation_script_path requires inference outputs", stacklevel=2, ) - if self.evaluation_script is not None and not os.path.exists( - os.path.join(self.project_root, self.evaluation_script) + if self.evaluation_script_path is not None and not os.path.exists( + os.path.join(self.run.project_root, self.evaluation_script_path) ): raise ValueError( - f"evaluation_script {self.evaluation_script!r} does not exist " - f"under project_root {self.project_root!r}" + f"evaluation_script_path {self.evaluation_script_path!r} does not exist " + f"under run.project_root {self.run.project_root!r}" ) - if self.evaluation_inference_config is not None: - inference_path = self.evaluation_inference_config + if self.evaluation_inference_config_path is not None: + inference_path = self.evaluation_inference_config_path if not os.path.isabs(inference_path): - inference_path = os.path.join(self.project_root, inference_path) + inference_path = os.path.join(self.run.project_root, inference_path) if not os.path.exists(inference_path): raise ValueError( - f"evaluation_inference_config {self.evaluation_inference_config!r} " + f"evaluation_inference_config_path {self.evaluation_inference_config_path!r} " "does not exist" ) return self @@ -770,8 +777,8 @@ def validate_compiled_search(self) -> None: for description, trial in candidates: try: - SequifierConfig.model_validate(self._materialized_values(trial, 0)) - except ValidationError as error: + _validate_training_candidate(self._materialized_values(trial, 0)) + except (ValidationError, ValueError) as error: raise ValueError( "Search parameters produce an invalid " f"training config for {description}:\n{error}" @@ -802,17 +809,41 @@ def _materialized_values(self, trial: Any | None, run_index: int) -> dict[str, A if ( isinstance(architecture_override, dict) and "position_encoding" in architecture_override - and "positional_encoding_scope" not in architecture_override + and "position_encoding_scope" not in architecture_override ): architecture = values["model"]["backbone"]["architecture"] if architecture["position_encoding"]["type"] in { "range", "range_concat", }: - architecture["positional_encoding_scope"] = "global" - values["project_root"] = self.project_root + architecture["position_encoding_scope"] = "global" + values["run"]["project_root"] = self.run.project_root trial_count = self.trials if self.trials is not None else self.grid_size() - values["model_name"] = f"{self.name}-run-{run_index:0{len(str(trial_count))}d}" + values["run"]["model_name"] = ( + f"{self.run.study_name}-run-{run_index:0{len(str(trial_count))}d}" + ) + outputs = values["run"].get("outputs", {}) + run_name = values["run"]["model_name"] + for key in ("model_path", "report_path"): + if outputs.get(key): + from pathlib import Path + + path = Path(outputs[key]) + outputs[key] = str( + path.with_name(f"{path.stem}-{run_name}{path.suffix}") + ) + if outputs.get("phase_directory"): + outputs["phase_directory"] = os.path.join( + outputs["phase_directory"], run_name + ) + recovery = values["run"].get("recovery", {}).get("resume", {}) + if recovery.get("checkpoint_path"): + from pathlib import Path + + path = Path(recovery["checkpoint_path"]) + recovery["checkpoint_path"] = str( + path.with_name(f"{path.stem}-{run_name}{path.suffix}") + ) return values @beartype @@ -821,9 +852,12 @@ def sample_trial(self, trial: Any, run_index: int) -> SequifierConfig: values = self._materialized_values(trial, run_index) try: - return SequifierConfig.model_validate(values) - except ValidationError as error: - parameters = getattr(trial, "params", {}) + return _validate_training_candidate(values) + except (ValidationError, ValueError) as error: + try: + parameters = trial.params + except AttributeError: + parameters = {} raise ValueError( "Sampled hyperparameters produce an invalid canonical training " f"config for parameters {parameters!r}:\n{error}" @@ -872,19 +906,26 @@ def compile_canonical_hyperparameter_search_config( f"referenced by {config_path!r}: {error}" ) from error + if not isinstance(base_config, SequifierConfig) or any( + phase.algorithm != "supervised" for phase in base_config.training_plan.phases + ): + raise ValueError( + "hyperparameter search for trajectory training is not yet supported" + ) + search_values = copy.deepcopy(config_values) search_values["base_config_path"] = base_config_path - search_values.setdefault("project_root", base_config.project_root) + if isinstance(search_values.get("run"), dict): + search_values["run"].setdefault("project_root", base_config.run.project_root) try: base_values = base_config.model_dump(mode="python") - search_values["parameters"] = normalize_train_config_parameter_surface( - search_values.get("parameters"), - base_values, + base_values["run"] = base_config.run.model_dump( + mode="python", exclude_unset=True ) search_config = CanonicalHyperparameterSearchConfig.model_validate( search_values ) - base_values["project_root"] = search_config.project_root + base_values["run"]["project_root"] = search_config.run.project_root search_config._compiled_config = _compile_value( search_config.parameters, base_values, diff --git a/src/sequifier/config/command_ownership.py b/src/sequifier/config/command_ownership.py new file mode 100644 index 000000000..2df2032fc --- /dev/null +++ b/src/sequifier/config/command_ownership.py @@ -0,0 +1,71 @@ +"""Shared run settings and CLI overrides for command configurations.""" + +from copy import deepcopy + +from pydantic import Field + +from sequifier.config.ownership import ( + ExecutionSettings, + ReproducibilitySettings, + Settings, +) + + +class CommandExecutionSettings(ExecutionSettings): + batch_size: int = Field(default=1, gt=0) + + +class CommandRunSettings(Settings): + project_root: str = "." + execution: CommandExecutionSettings = Field( + default_factory=CommandExecutionSettings + ) + reproducibility: ReproducibilitySettings = Field( + default_factory=ReproducibilitySettings + ) + + def validate_for(self, command): + execution = self.execution + for field in ("distribution", "precision", "compilation", "max_concat_tensors"): + if field in execution.model_fields_set: + raise ValueError(f"{command} does not support run.execution.{field}") + loading = execution.data_loading + if command == "infer" and "data_loading" in execution.model_fields_set: + raise ValueError("inference does not support data_loading controls") + if command == "preprocess": + if ( + {"device", "batch_size"} & execution.model_fields_set + or "enforce_determinism" in self.reproducibility.model_fields_set + ): + raise ValueError( + "preprocessing supports run execution workers and a seed; device/batching/determinism controls are unsupported" + ) + if loading.model_fields_set - {"num_workers"}: + raise ValueError("preprocessing only supports data_loading.num_workers") + return self + + +def apply_run_overrides(values, overrides): + """CLI scalar conveniences write the same canonical paths as YAML.""" + values, overrides = deepcopy(values), dict(overrides) + run = values.setdefault("run", {}) + for key in ("project_root", "model_name"): + if key in overrides and overrides[key] is not None: + run[key] = overrides.pop(key) + if "seed" in overrides and overrides["seed"] is not None: + run.setdefault("reproducibility", {})["seed"] = overrides.pop("seed") + if "device" in overrides and overrides["device"] is not None: + run.setdefault("execution", {})["device"] = overrides.pop("device") + if ( + "enforce_determinism" in overrides + and overrides["enforce_determinism"] is not None + ): + run.setdefault("reproducibility", {})["enforce_determinism"] = overrides.pop( + "enforce_determinism" + ) + # The CLI uses an explicit None to clear either column selection. + return values, { + key: value + for key, value in overrides.items() + if value is not None or key in {"selected_columns", "input_columns"} + } diff --git a/src/sequifier/config/components.py b/src/sequifier/config/components.py index 0ed26d3b4..36caa981d 100644 --- a/src/sequifier/config/components.py +++ b/src/sequifier/config/components.py @@ -1,12 +1,10 @@ """Reusable configuration components shared by training and artifacts.""" import math -from collections.abc import Mapping from itertools import product from typing import Annotated, Any, Literal, Optional, TypeAlias, Union from pydantic import ( - BaseModel, ConfigDict, Field, RootModel, @@ -17,7 +15,9 @@ model_validator, ) +from sequifier.config.identifiers import validate_module_dict_key from sequifier.config.initialization_config import ModelInitializationConfig +from sequifier.config.naming import ConfigModel from sequifier.config.probabilities import ProbabilityDistribution from sequifier.typechecking import beartype @@ -25,7 +25,7 @@ NextOccurrenceTargetValue: TypeAlias = StrictInt | StrictStr -class CartesianLayoutModel(BaseModel): +class CartesianLayoutModel(ConfigModel): """Reusable coordinate annotation for flat feature columns.""" model_config = ConfigDict(extra="forbid") @@ -94,7 +94,7 @@ def __getitem__(self, key: str) -> CartesianLayoutModel: return self.root[key] -class IngestionComponentBase(BaseModel): +class IngestionComponentBase(ConfigModel): """Settings shared by every feature-ingestion component.""" model_config = ConfigDict(extra="forbid") @@ -125,7 +125,7 @@ def serialize_optional_seed(self, serializer): return values -class MultiHashEmbeddingConfig(BaseModel): +class MultiHashEmbeddingConfig(ConfigModel): """Embed one categorical variable through independent hash tables.""" model_config = ConfigDict(extra="forbid") @@ -136,7 +136,7 @@ class MultiHashEmbeddingConfig(BaseModel): seed: StrictInt = Field(default=0, ge=0, le=2**63 - 1) -class QRHashEmbeddingConfig(BaseModel): +class QRHashEmbeddingConfig(ConfigModel): """Embed one categorical variable with quotient and remainder tables.""" model_config = ConfigDict(extra="forbid") @@ -227,7 +227,7 @@ class GroupedIngestionConfig(IngestionComponentBase): def validate_groups(self): grouped_columns = [] for group_name, columns in self.groups.items(): - _validate_module_dict_key( + validate_module_dict_key( group_name, f"grouped ingestion group {group_name!r}" ) if not columns: @@ -276,7 +276,7 @@ class TemporalConvIngestionConfig(IngestionComponentBase): causal: bool = True activation: Literal["relu", "gelu", "silu"] = "gelu" dropout: float = Field(0.0, ge=0.0, lt=1.0) - post_conv_norm: Literal["layer_norm", "rmsnorm", "none"] = "layer_norm" + post_conv_normalization: Literal["layer_norm", "rmsnorm", "none"] = "layer_norm" orientation: Literal["within_column", "within_item_position"] = ( "within_item_position" ) @@ -333,7 +333,7 @@ def dilation_schedule(self) -> list[int]: return [self.dilation] * self.num_layers -class AxisProjectionBlockModel(BaseModel): +class AxisProjectionBlockModel(ConfigModel): """Flatten configured cartesian axes and project them with a linear layer.""" model_config = ConfigDict(extra="forbid") @@ -351,7 +351,7 @@ def validate_axes_unique(self): return self -class AxisConvBlockModel(BaseModel): +class AxisConvBlockModel(ConfigModel): """Sweep a native 1D/2D/3D convolution over configured cartesian axes.""" model_config = ConfigDict(extra="forbid") @@ -372,7 +372,7 @@ def validate_axes_unique(self): return self -class AxisAttentionBlockModel(BaseModel): +class AxisAttentionBlockModel(ConfigModel): """Apply self-attention over one or more configured cartesian axes.""" model_config = ConfigDict(extra="forbid") @@ -380,7 +380,7 @@ class AxisAttentionBlockModel(BaseModel): type: Literal["axis_attention"] axes: list[str] = Field(..., min_length=1) output_dim: int = Field(..., gt=0) - n_heads: int = Field(1, gt=0) + num_heads: int = Field(1, gt=0) dropout: float = Field(0.0, ge=0.0, lt=1.0) unshared_axes: list[str] = Field(default_factory=list) @@ -389,19 +389,19 @@ class AxisAttentionBlockModel(BaseModel): def validate_axes_unique(self): _validate_axis_list_unique(self.axes, "axes") _validate_axis_list_unique(self.unshared_axes, "unshared_axes") - if self.output_dim % self.n_heads != 0: - raise ValueError("axis_attention output_dim must be divisible by n_heads") + if self.output_dim % self.num_heads != 0: + raise ValueError("axis_attention output_dim must be divisible by num_heads") return self -class AxisPoolBlockModel(BaseModel): +class AxisPoolBlockModel(ConfigModel): """Reduce configured cartesian axes without changing the channel dimension.""" model_config = ConfigDict(extra="forbid") type: Literal["axis_pool"] axes: list[str] = Field(..., min_length=1) - mode: Literal["mean", "sum", "max"] = "mean" + pooling: Literal["mean", "sum", "max"] = "mean" @model_validator(mode="after") @beartype @@ -433,18 +433,6 @@ def _validate_column_list_unique(columns: list[str], field_name: str) -> None: raise ValueError(f"{field_name} cannot contain duplicate columns") -@beartype -def _validate_module_dict_key(key: str, usage: str) -> None: - if key == "": - raise ValueError(f"{usage} cannot be empty") - if "." in key: - raise ValueError(f"{usage} cannot contain '.'") - from torch import nn - - if hasattr(nn.ModuleDict(), key): - raise ValueError(f"{usage} collides with a ModuleDict attribute") - - @beartype def _validate_feature_embedding_dims( feature_embedding_dims: Optional[dict[str, int]], field_name: str @@ -458,7 +446,7 @@ def _validate_feature_embedding_dims( raise ValueError(f"{field_name} values must be positive: {invalid_dims}") -class AxisEmbeddingModel(BaseModel): +class AxisEmbeddingModel(ConfigModel): """Optional positional encoding for cartesian layout axes.""" model_config = ConfigDict(extra="forbid") @@ -503,7 +491,7 @@ class StructuredIngestionConfig(IngestionComponentBase): processing_blocks: list[StructuredProcessingBlock] = Field(default_factory=list) image_cells: dict[str, str] = Field(default_factory=dict) image_encoder: "ImageEncoderConfig | None" = None - image_frame_pool: Literal["mean", "attention"] = "mean" + image_frame_pooling: Literal["mean", "attention"] = "mean" @field_validator("axis_embeddings", mode="before") @classmethod @@ -524,7 +512,7 @@ def validate_image_cells(self): return self -class ImageEncoderConfig(BaseModel): +class ImageEncoderConfig(ConfigModel): model_config = ConfigDict(extra="forbid") type: Literal["resnet", "frozen"] = "resnet" @@ -536,6 +524,7 @@ class ImageEncoderConfig(BaseModel): groups: int = Field(default=1, gt=0) width_per_group: int = Field(default=64, gt=0) replace_stride_with_dilation: tuple[bool, bool, bool] = (False, False, False) + microbatch_size: int = Field(default=16, gt=0) @model_validator(mode="after") def validate_encoder(self): @@ -586,7 +575,7 @@ class ImageIngestionConfig(IngestionComponentBase): views: list[str] = Field(..., min_length=1) output_dim: int = Field(..., gt=0) encoder: ImageEncoderConfig = Field(default_factory=ImageEncoderConfig) - frame_pool: Literal["mean", "attention"] = "mean" + frame_pooling: Literal["mean", "attention"] = "mean" view_merge: Literal["mean", "concat"] = "concat" @field_validator("views") @@ -599,7 +588,7 @@ def validate_views(cls, views): StructuredIngestionConfig.model_rebuild() -class LinearDecodingConfig(BaseModel): +class LinearDecodingConfig(ConfigModel): """Project each support window directly to target outputs.""" model_config = ConfigDict(extra="forbid") @@ -616,7 +605,7 @@ def validate_target_columns(cls, v): return v -class MLPDecodingConfig(BaseModel): +class MLPDecodingConfig(ConfigModel): """Flatten support windows and decode targets with a shared MLP branch.""" model_config = ConfigDict(extra="forbid") @@ -652,7 +641,7 @@ def validate_hidden_dims(cls, v): return v -class AutoregressiveTransformerDecodingConfig(BaseModel): +class AutoregressiveTransformerDecodingConfig(ConfigModel): """Decode ordered targets autoregressively within each temporal position.""" model_config = ConfigDict(extra="forbid") @@ -710,13 +699,13 @@ def validate_decoder(self): DecodingSpecConfig = BranchDecodingConfig | dict[str, BranchDecodingConfig] -class DecoderComponentBase(BaseModel): +class DecoderComponentBase(ConfigModel): """Settings shared by every target-decoder component.""" model_config = ConfigDict(extra="forbid") prediction_length: int = Field(..., gt=0) - support: int = Field(1, gt=0) + support_length: int = Field(1, gt=0) initialization: ModelInitializationConfig = Field( default_factory=ModelInitializationConfig ) @@ -745,7 +734,7 @@ class CompositeDecoderComponentConfig(DecoderComponentBase): @beartype def validate_branch_names(cls, branches): for branch_name in branches: - _validate_module_dict_key( + validate_module_dict_key( branch_name, f"Target decoding branch {branch_name!r}" ) return branches @@ -762,60 +751,60 @@ def validate_branch_names(cls, branches): ] -class BackboneAttentionConfig(BaseModel): +class BackboneAttentionConfig(ConfigModel): model_config = ConfigDict(extra="forbid") type: Literal["mha", "mqa", "gqa"] = "mha" - n_heads: int = Field(..., gt=0) - n_kv_heads: Optional[int] = Field(default=None, gt=0) + num_heads: int = Field(..., gt=0) + num_kv_heads: Optional[int] = Field(default=None, gt=0) output_projection: bool = True @model_validator(mode="after") @beartype def validate_heads(self): - if self.n_kv_heads is None: + if self.num_kv_heads is None: if self.type == "gqa": - raise ValueError("gqa requires n_kv_heads") - self.n_kv_heads = 1 if self.type == "mqa" else self.n_heads - if self.n_heads % self.n_kv_heads != 0: + raise ValueError("gqa requires num_kv_heads") + self.num_kv_heads = 1 if self.type == "mqa" else self.num_heads + if self.num_heads % self.num_kv_heads != 0: raise ValueError( - f"n_heads {self.n_heads} must be divisible by n_kv_heads " - f"{self.n_kv_heads}" + f"num_heads {self.num_heads} must be divisible by num_kv_heads " + f"{self.num_kv_heads}" ) - if self.type == "mha" and self.n_kv_heads != self.n_heads: - raise ValueError("mha requires n_kv_heads to equal n_heads") - if self.type == "mqa" and self.n_kv_heads != 1: - raise ValueError("mqa requires n_kv_heads to equal 1") + if self.type == "mha" and self.num_kv_heads != self.num_heads: + raise ValueError("mha requires num_kv_heads to equal num_heads") + if self.type == "mqa" and self.num_kv_heads != 1: + raise ValueError("mqa requires num_kv_heads to equal 1") return self -class BackboneFeedForwardConfig(BaseModel): +class BackboneFeedForwardConfig(ConfigModel): model_config = ConfigDict(extra="forbid") - dim: int = Field(..., gt=0) - activation: Literal["relu", "gelu", "swiglu"] = "swiglu" + hidden_dim: int = Field(..., gt=0) + activation: Literal["relu", "gelu", "swiglu", "softplus"] = "swiglu" -class BackboneNormalizationConfig(BaseModel): +class BackboneNormalizationConfig(ConfigModel): model_config = ConfigDict(extra="forbid") type: Literal["layer_norm", "rmsnorm"] = "rmsnorm" norm_first: bool = True -class BackbonePositionEncodingConfig(BaseModel): +class BackbonePositionEncodingConfig(ConfigModel): model_config = ConfigDict(extra="forbid") type: Literal["learned", "rope", "range", "range_concat", "sinusoidal"] = "learned" theta: float = Field(10000.0, gt=0.0) -class TransformerEncoderArchitectureConfig(BaseModel): +class TransformerEncoderArchitectureConfig(ConfigModel): """Capacity-independent transformer architecture shared by both encoders.""" model_config = ConfigDict(extra="forbid") - dim_model: int = Field(..., gt=0) + model_dim: int = Field(..., gt=0) num_layers: int = Field(..., gt=0) attention: BackboneAttentionConfig feed_forward: BackboneFeedForwardConfig @@ -831,13 +820,13 @@ class TransformerEncoderArchitectureConfig(BaseModel): @model_validator(mode="after") @beartype def validate_architecture(self): - n_heads = self.attention.n_heads - if self.dim_model % n_heads != 0: + num_heads = self.attention.num_heads + if self.model_dim % num_heads != 0: raise ValueError( - f"dim_model {self.dim_model} must be divisible by n_heads {n_heads}" + f"model_dim {self.model_dim} must be divisible by num_heads {num_heads}" ) if self.position_encoding.type == "rope": - head_dim = self.dim_model // n_heads + head_dim = self.model_dim // num_heads if head_dim % 2 != 0: raise ValueError( f"RoPE requires an even head dimension, got {head_dim}" @@ -870,7 +859,7 @@ def validate_architecture(self): class BackboneArchitectureConfig(TransformerEncoderArchitectureConfig): max_context_length: int = Field(..., gt=0) - positional_encoding_scope: Literal["per_feature", "global"] = "per_feature" + position_encoding_scope: Literal["per_feature", "global"] = "per_feature" @model_validator(mode="before") @classmethod @@ -882,28 +871,30 @@ def default_global_position_scope(cls, values): position_type = ( position_encoding.get("type", "learned") if isinstance(position_encoding, dict) - else getattr(position_encoding, "type", "learned") + else position_encoding.type + if isinstance(position_encoding, BackbonePositionEncodingConfig) + else "learned" ) if ( position_type in {"range", "range_concat", "sinusoidal"} - and "positional_encoding_scope" not in values + and "position_encoding_scope" not in values ): values = dict(values) - values["positional_encoding_scope"] = "global" + values["position_encoding_scope"] = "global" return values @model_validator(mode="after") def validate_temporal_positions(self): if ( self.position_encoding.type in {"range", "range_concat", "sinusoidal"} - and self.positional_encoding_scope != "global" + and self.position_encoding_scope != "global" ): raise ValueError( f"position_encoding type {self.position_encoding.type!r} requires " - "positional_encoding_scope 'global'" + "position_encoding_scope 'global'" ) - if self.position_encoding.type == "range_concat" and self.dim_model < 2: - raise ValueError("range_concat requires dim_model to be at least 2") + if self.position_encoding.type == "range_concat" and self.model_dim < 2: + raise ValueError("range_concat requires model_dim to be at least 2") return self @@ -938,7 +929,7 @@ def validate_depth(self): return self -class IngestionMergeConfig(BaseModel): +class IngestionMergeConfig(ConfigModel): """How composite branch outputs are merged.""" model_config = ConfigDict(extra="forbid") @@ -969,7 +960,7 @@ class CompositeIngestionConfig(IngestionComponentBase): @beartype def validate_branch_names(cls, branches): for branch_name in branches: - _validate_module_dict_key( + validate_module_dict_key( branch_name, f"Composite ingestion branch {branch_name!r}" ) return branches @@ -997,7 +988,7 @@ def validate_branch_names(cls, branches): IngestionSpecConfig = IngestionComponentConfig | dict[str, IngestionComponentConfig] -class BackboneRepositoryConfig(BaseModel): +class BackboneRepositoryConfig(ConfigModel): model_config = ConfigDict(extra="forbid") backbone_id: StrictStr = Field(..., min_length=1) @@ -1007,7 +998,7 @@ class BackboneRepositoryConfig(BaseModel): conflict_policy: Literal["compare_and_swap"] = "compare_and_swap" -class BackboneComponentConfig(BaseModel): +class BackboneComponentConfig(ConfigModel): model_config = ConfigDict(extra="forbid") initialization_seed: Optional[StrictInt] = Field(default=None, ge=0, le=2**63 - 1) @@ -1027,7 +1018,7 @@ def serialize_optional_seed(self, serializer): return values -class ComponentSpec(BaseModel): +class ComponentSpec(ConfigModel): """A named component and its constructor arguments.""" model_config = ConfigDict(extra="forbid", frozen=True) @@ -1035,28 +1026,8 @@ class ComponentSpec(BaseModel): name: str = Field(min_length=1) arguments: dict[str, Any] = Field(default_factory=dict) - @model_validator(mode="before") - @classmethod - @beartype - def parse_flattened(cls, value): - if isinstance(value, cls): - return value - if not isinstance(value, Mapping): - raise TypeError("component configuration must be a mapping") - if "arguments" in value: - return value - return { - "name": value.get("name"), - "arguments": {key: item for key, item in value.items() if key != "name"}, - } - - @model_serializer - @beartype - def serialize_flattened(self) -> dict[str, Any]: - return {"name": self.name, **self.arguments} - -class ReplacementDistribution(BaseModel): +class ReplacementDistribution(ConfigModel): model_config = ConfigDict(extra="forbid") masked: float = Field(..., ge=0.0, le=1.0) @@ -1074,7 +1045,7 @@ def validate_sum(self): return self -class BERTSpecModel(BaseModel): +class BERTSpecModel(ConfigModel): model_config = ConfigDict(extra="forbid") masking_probability: float = Field(..., gt=0.0, le=1.0) @@ -1082,27 +1053,28 @@ class BERTSpecModel(BaseModel): span_masking: ProbabilityDistribution -class NextOccurrenceConfigModel(BaseModel): +class NextOccurrenceConfigModel(ConfigModel): model_config = ConfigDict(extra="forbid") column_name: str target_values: list[NextOccurrenceTargetValue] = Field(..., min_length=1) -class ResumeConfig(BaseModel): +class ResumeConfig(ConfigModel): """Policy for resuming the same run from its complete checkpoint.""" model_config = ConfigDict(extra="forbid") policy: Literal["never", "if_exists", "required"] = "never" checkpoint_path: Optional[str] = None + mode: Literal["exact", "reset_environments"] = "reset_environments" @model_validator(mode="after") @beartype def validate_checkpoint_path(self): if self.policy == "required" and not self.checkpoint_path: raise ValueError( - "global_training.resume.checkpoint_path is required when " + "run.recovery.resume.checkpoint_path is required when " "policy is 'required'" ) return self diff --git a/src/sequifier/config/composition.py b/src/sequifier/config/composition.py index 026b4c893..415216253 100644 --- a/src/sequifier/config/composition.py +++ b/src/sequifier/config/composition.py @@ -21,9 +21,7 @@ DEFAULT_ATOMIC_PATHS: frozenset[ConfigPath] = frozenset( { ("model", "backbone"), - ("global_training", "optimizer"), - ("global_training", "scheduler"), - ("global_training", "bert_spec", "span_masking"), + ("training_plan", "phases", "learning", "bert_spec", "span_masking"), } ) @@ -36,13 +34,7 @@ def _is_atomic(path: ConfigPath, atomic_paths: frozenset[ConfigPath]) -> bool: "model", "interfaces", ) - singleton_interface_component = len(path) == 3 and path[:2] == ( - "model", - "interface", - ) - return (named_interface_component or singleton_interface_component) and path[ - -1 - ] in {"ingestion", "decoder"} + return named_interface_component and path[-1] in {"ingestion", "decoder"} @beartype @@ -103,10 +95,10 @@ def load_composed_yaml_config( if not additional_paths: return entry_values - project_root = entry_values.get("project_root") + project_root = entry_values.get("run", {}).get("project_root", ".") if not isinstance(project_root, str) or not project_root.strip(): raise ValueError( - f"Config '{entry_path}' must define a non-empty string project_root " + f"Config '{entry_path}' must define a non-empty run.project_root " f"when {ADDITIONAL_CONFIG_PATHS_KEY} is configured." ) @@ -187,20 +179,18 @@ def _normalize_additional_config_paths( ) -> list[str]: if value is None: return [] - if isinstance(value, str): - paths = [value] - elif isinstance(value, list): + if isinstance(value, list): paths = value else: raise ValueError( f"Config '{config_path}' field '{ADDITIONAL_CONFIG_PATHS_KEY}' must " - "be a non-empty string, a list of non-empty strings, or null." + "be a list of non-empty strings, or null." ) if any(not isinstance(path, str) or not path.strip() for path in paths): raise ValueError( f"Config '{config_path}' field '{ADDITIONAL_CONFIG_PATHS_KEY}' must " - "be a non-empty string, a list of non-empty strings, or null." + "be a list of non-empty strings, or null." ) return paths @@ -327,7 +317,7 @@ def _deep_merge_dicts( def _changes_discriminator( base: Mapping[str, Any], override: Mapping[str, Any] ) -> bool: - for discriminator in ("type", "name"): + for discriminator in ("type", "name", "algorithm", "task"): if ( discriminator in base and discriminator in override diff --git a/src/sequifier/config/depth_layout.py b/src/sequifier/config/depth_layout.py index 470941575..f3bf9b71c 100644 --- a/src/sequifier/config/depth_layout.py +++ b/src/sequifier/config/depth_layout.py @@ -2,19 +2,21 @@ from typing import Literal -from pydantic import BaseModel, ConfigDict, Field, RootModel, model_validator -from torch import nn +from pydantic import ConfigDict, Field, RootModel, model_validator + +from sequifier.config.identifiers import validate_module_dict_key +from sequifier.config.naming import ConfigModel def depth_mask_metadata_key(layout: str) -> str: return f"depth_valid_mask:{layout}" -class DepthLayoutModel(BaseModel): +class DepthLayoutModel(ConfigModel): model_config = ConfigDict(extra="forbid") position_column: str columns: list[str] = Field(min_length=1) - context_length: int = Field(gt=0) + max_depth_length: int = Field(gt=0) position_base: Literal[0, 1] = 0 allow_gaps: bool = False @@ -41,10 +43,13 @@ class DepthLayoutRegistryModel(RootModel[dict[str, DepthLayoutModel]]): @model_validator(mode="after") def validate_registry(self): occupied = set() - storage = nn.ModuleDict() for name, layout in self.root.items(): - if not name or "." in name or hasattr(storage, name): - raise ValueError(f"Invalid depth layout module key: {name!r}") + try: + validate_module_dict_key(name, "Depth layout name") + except ValueError as error: + raise ValueError( + f"Invalid depth layout module key: {name!r}" + ) from error if occupied.intersection(layout.columns): raise ValueError("A feature cannot belong to multiple depth layouts") occupied.update(layout.columns) diff --git a/src/sequifier/config/freezing_config.py b/src/sequifier/config/freezing_config.py deleted file mode 100644 index 927fd8709..000000000 --- a/src/sequifier/config/freezing_config.py +++ /dev/null @@ -1,84 +0,0 @@ -"""Typed component fields for semantic layer freezing.""" - -from typing import Optional - -from pydantic import ( - BaseModel, - ConfigDict, - Field, - field_validator, - model_serializer, - model_validator, -) - -from sequifier.config.layer_groups import LayerGroup -from sequifier.typechecking import beartype - - -class LayerFreezingConfigFields(BaseModel): - """Mutually exclusive semantic layer-freezing selections.""" - - model_config = ConfigDict(extra="forbid") - - freeze: Optional[list[LayerGroup]] = None - freezing_except: Optional[list[LayerGroup]] = None - - @field_validator("freeze", "freezing_except") - @classmethod - @beartype - def validate_unique_groups( - cls, value: Optional[list[LayerGroup]] - ) -> Optional[list[LayerGroup]]: - if value is not None and len(value) != len(set(value)): - raise ValueError("layer freezing groups cannot contain duplicates") - return value - - @model_validator(mode="after") - @beartype - def validate_mutually_exclusive(self): - if self.freeze is not None and self.freezing_except is not None: - raise ValueError( - "freeze and freezing_except are mutually exclusive; " - "only one may be non-null" - ) - return self - - @model_serializer(mode="wrap") - @beartype - def serialize_freezing_fields(self, serializer): - """Omit inactive fields from canonical dataset freezing policies.""" - - values = serializer(self) - if self.freeze is None: - values.pop("freeze", None) - if self.freezing_except is None: - values.pop("freezing_except", None) - return values - - @property - @beartype - def has_freezing_policy(self) -> bool: - return self.freeze is not None or self.freezing_except is not None - - -class IngestionFreezingConfig(LayerFreezingConfigFields): - """Recursive policies following the declared composite branch tree.""" - - branches: dict[str, "IngestionFreezingConfig"] = Field(default_factory=dict) - - @property - def has_freezing_policy(self) -> bool: - return super().has_freezing_policy or any( - child.has_freezing_policy for child in self.branches.values() - ) - - @model_serializer(mode="wrap") - def serialize_ingestion_policy(self, serializer): - values = serializer(self) - if self.freeze is None: - values.pop("freeze", None) - if self.freezing_except is None: - values.pop("freezing_except", None) - if not self.branches: - values.pop("branches", None) - return values diff --git a/src/sequifier/config/identifiers.py b/src/sequifier/config/identifiers.py new file mode 100644 index 000000000..5050d9b46 --- /dev/null +++ b/src/sequifier/config/identifiers.py @@ -0,0 +1,27 @@ +import keyword + +from sequifier.typechecking import beartype + + +@beartype +def identifier(value: str, usage: str) -> str: + if "." in value or not value.isidentifier() or keyword.iskeyword(value): + raise ValueError( + f"{usage} {value!r} must be a valid identifier and cannot contain '.'." + ) + return value + + +@beartype +def validate_module_dict_key(key: str, usage: str) -> None: + if not key: + raise ValueError(f"{usage} cannot be empty") + if "." in key: + raise ValueError(f"{usage} cannot contain '.'") + + from torch import nn + + try: + nn.ModuleDict().add_module(key, None) + except KeyError as error: + raise ValueError(f"{usage} collides with a ModuleDict attribute") from error diff --git a/src/sequifier/config/image_config.py b/src/sequifier/config/image_config.py index 4149d8870..51ae56b8a 100644 --- a/src/sequifier/config/image_config.py +++ b/src/sequifier/config/image_config.py @@ -1,11 +1,13 @@ """Image source and frame-alignment configuration.""" -from typing import Any, Literal, Optional, Union +from typing import Any, Literal, Optional -from pydantic import BaseModel, ConfigDict, Field, model_validator +from pydantic import ConfigDict, Field, model_validator +from sequifier.config.naming import ConfigModel -class FrameAlignment(BaseModel): + +class FrameAlignment(ConfigModel): model_config = ConfigDict(extra="forbid") frames: int = Field(default=1, gt=0) @@ -46,7 +48,7 @@ def max_frames_per_item(self) -> int: return (self.frames + self.items - 1) // self.items -class ImageSourceConfig(BaseModel): +class ImageSourceConfig(ConfigModel): model_config = ConfigDict(extra="forbid") path: str | None = None @@ -79,10 +81,10 @@ def validate_size(self): return self -class PreconditioningImageConfig(BaseModel): +class PreconditioningImageConfig(ConfigModel): model_config = ConfigDict(extra="forbid") - path: str | dict[str, str] | None = None + paths: dict[str, str] | None = None view_names: list[str] = Field(default_factory=list) resize_size: int = Field(default=256, gt=0) crop_size: int = Field(default=224, gt=0) @@ -92,76 +94,61 @@ class PreconditioningImageConfig(BaseModel): def validate_size(self): if self.crop_size > self.resize_size: raise ValueError("crop_size must not exceed resize_size") - if isinstance(self.path, dict) and ( - not self.path + if self.paths is not None and ( + not self.paths or any( - not name.isidentifier() or not path for name, path in self.path.items() + not name.isidentifier() or not path for name, path in self.paths.items() ) ): raise ValueError("Preconditioning views require names and folder paths") - if isinstance(self.path, str) and not self.path: - raise ValueError("Preconditioning image path cannot be empty") - if self.path is None and not self.view_names: + if self.paths is None and not self.view_names: raise ValueError("Preconditioning image views are required") return self @property def views(self) -> dict[str, str]: - if isinstance(self.path, dict): - return self.path - if isinstance(self.path, str): - return {"default": self.path} - return {name: "" for name in self.view_names} + return ( + self.paths + if self.paths is not None + else {name: "" for name in self.view_names} + ) def compatibility_signature(self) -> dict: return { - **self.model_dump(exclude={"path", "signature", "view_names"}), + **self.model_dump(exclude={"paths", "signature", "view_names"}), "views": sorted(self.views), } -MediaPath = Optional[Union[str, dict[str, str]]] +MediaPath = Optional[dict[str, str]] def bind_image_paths( sources: dict[str, ImageSourceConfig], preconditioning: PreconditioningImageConfig | None, - image_path: MediaPath, - preconditioning_image_path: MediaPath, + image_paths: MediaPath, + preconditioning_image_paths: MediaPath, ) -> tuple[dict[str, ImageSourceConfig], PreconditioningImageConfig | None]: - """Attach explicit dataset media locations to path-free metadata.""" - if isinstance(image_path, str): - if len(sources) != 1: - raise ValueError( - "image_path must map view names when there are multiple views" - ) - paths = {next(iter(sources)): image_path} - else: - paths = image_path or {} + """Attach explicit named media locations to path-free metadata.""" + paths = image_paths or {} if set(paths) != set(sources) or any(not path for path in paths.values()): - raise ValueError("image_path must provide exactly the selected image views") + raise ValueError("image_paths must provide exactly the selected image views") bound = { name: source.model_copy(update={"path": paths[name]}) for name, source in sources.items() } if preconditioning is None: - if preconditioning_image_path is not None: - raise ValueError("preconditioning_image_path has no metadata source") + if preconditioning_image_paths is not None: + raise ValueError("preconditioning_image_paths has no metadata source") return bound, None - names = set(preconditioning.views) - if isinstance(preconditioning_image_path, str): - if names != {"default"}: - raise ValueError("preconditioning_image_path must map named views") - prefix_paths = preconditioning_image_path - else: - prefix_paths = preconditioning_image_path or {} - if set(prefix_paths) != names or any( - not path for path in prefix_paths.values() - ): - raise ValueError( - "preconditioning_image_path must provide exactly the selected views" - ) - return bound, preconditioning.model_copy(update={"path": prefix_paths}) + prefix_paths = preconditioning_image_paths or {} + if set(prefix_paths) != set(preconditioning.views) or any( + not path for path in prefix_paths.values() + ): + raise ValueError( + "preconditioning_image_paths must provide exactly the selected views" + ) + return bound, preconditioning.model_copy(update={"paths": prefix_paths}) def image_views(ingestion: Any) -> set[str]: diff --git a/src/sequifier/config/infer_config.py b/src/sequifier/config/infer_config.py index 5003fd6e6..f062ea327 100644 --- a/src/sequifier/config/infer_config.py +++ b/src/sequifier/config/infer_config.py @@ -1,34 +1,36 @@ import json import os import re -from typing import Any, Generic, Optional, TypeVar, Union +from typing import Any, Generic, Literal, Optional, TypeVar import numpy as np -from pydantic import ( - BaseModel, - ConfigDict, - Field, - PrivateAttr, - field_validator, - model_validator, -) +from pydantic import ConfigDict, Field, PrivateAttr, field_validator, model_validator from sequifier.config.composition import ( load_composed_yaml_config, merge_config_fragments, ) from sequifier.config.image_config import MediaPath, bind_image_paths, image_views +from sequifier.config.inference_common import ( + InferenceExecutionConfig, + inference_model_config, +) from sequifier.config.metadata import ( DatasetMetadata, extract_inline_metadata, load_dataset_metadata, + validate_persisted_metadata_versions, ) +from sequifier.config.policy_infer_config import ( + ResolvedPolicyInferenceConfig, + policy_inference_config, +) +from sequifier.config.rollout_config import RolloutConfig from sequifier.helpers import ( ModelWindowView, StoredWindowLayout, canonicalize_polars_dtype_name, derive_target_column_types, - metadata_config_path_from_preprocessing_data_path, normalize_path, resolve_window_view, try_catch_excess_keys, @@ -129,42 +131,43 @@ def _merge_loaded_execution_values( values[field] = loaded_value -@beartype -def _execution_source( - training: Any, dataset: Any -) -> tuple[dict[str, Any], DatasetMetadata]: - interface = dataset.interface - global_spec = training.global_training - layout = interface.storage_layout - return ( - { - "model_interface": dataset.model_interface, - "input_columns": list(interface.input_columns), - "static_columns": ( - list(interface.static_conditioning.columns) - if interface.static_conditioning - else [] - ), - "target_columns": list(interface.target_columns), - "column_data_types": dict(interface.column_data_types), - "target_column_types": dict(interface.target_column_types), - "training_objective": global_spec.training_objective, - "context_length": global_spec.context_length, - "target_offset": global_spec.target_offset, - "prediction_length": interface.decoder.prediction_length, - "decoder_support": interface.decoder.support, - }, - DatasetMetadata( +def _execution_values(settings: Any, interface: Any) -> dict[str, Any]: + return { + "model_interface": interface.name, + "input_columns": list(interface.input_columns), + "static_columns": ( + list(interface.static_conditioning.columns) + if interface.static_conditioning + else [] + ), + "target_columns": list(interface.target_columns), + "column_data_types": dict(interface.column_data_types), + "target_column_types": dict(interface.target_column_types), + "training_objective": settings.training_objective, + "context_length": settings.context_length, + "target_offset": settings.target_offset, + "prediction_length": interface.decoder.prediction_length, + "decoder_support_length": interface.decoder.support_length, + } + + +def _execution_metadata(interface: Any, storage: Any) -> DatasetMetadata: + if storage is None: + raise ValueError(f"{interface.name}: stored-window metadata is required") + layout = storage.storage_layout + return DatasetMetadata.model_validate( + dict( depth_layouts=interface.depth_layouts, - tensor_payload_version=interface.tensor_payload_version, - split_paths=interface.split_paths, + tensor_payload_kind=storage.tensor_payload_kind, + tensor_payload_version=storage.tensor_payload_version, + split_paths=storage.split_paths, column_data_types=dict(interface.column_data_types), static_columns=( list(interface.static_conditioning.columns) if interface.static_conditioning else [] ), - static_features_path=interface.static_features_path, + static_features_path=storage.static_features_path, image_sources=interface.image_sources, preconditioning_image=interface.preconditioning_image, n_classes=dict(interface.n_classes), @@ -172,19 +175,29 @@ def _execution_source( special_token_ids=dict(interface.special_token_ids), selected_columns_statistics=dict(interface.selected_columns_statistics), normalize_real_columns=interface.normalize_real_columns, - prediction_aligned_splits=interface.prediction_aligned_splits, - prediction_alignment_steps=interface.prediction_alignment_steps, - window_lengths=interface.window_lengths, - prediction_length=interface.prediction_length, - target_offset=interface.target_offset, + prediction_aligned_splits=storage.prediction_aligned_splits, + prediction_alignment_steps=storage.prediction_alignment_steps, + window_lengths=storage.window_lengths, + prediction_length=storage.prediction_length, + target_offset=storage.target_offset, window_length=( - interface.window_lengths[0] - if interface.window_lengths + storage.window_lengths[0] + if storage.window_lengths else layout.window_length ), max_target_offset=layout.max_target_offset, stored_window_layout_version=layout.version, ), + context={"projected_depth_layouts": True}, + ) + + +@beartype +def _execution_source( + settings: Any, interface: Any, storage: Any +) -> tuple[dict[str, Any], DatasetMetadata]: + return _execution_values(settings, interface), _execution_metadata( + interface, storage ) @@ -197,6 +210,8 @@ def _assert_metadata_matches_source( ) -> None: if _image_contract(loaded_metadata) != _image_contract(selected_metadata): raise ValueError(f"Inference image source contract does not match {source}") + loaded_fields = dict(loaded_metadata) + selected_fields = dict(selected_metadata) relevant = list(loaded_metadata.column_data_types) if loaded_metadata.depth_layouts.compatibility_signature( relevant @@ -208,16 +223,16 @@ def _assert_metadata_matches_source( "id_maps", "selected_columns_statistics", ): - expected = getattr(loaded_metadata, field) - found = getattr(selected_metadata, field) + expected = loaded_fields[field] + found = selected_fields[field] found = {key: found.get(key) for key in expected} if _comparable_value(field, found) != _comparable_value(field, expected): raise ValueError( f"Inference metadata field {field!r} does not match {source}." ) for field in ("special_token_ids", "normalize_real_columns"): - expected = getattr(loaded_metadata, field) - found = getattr(selected_metadata, field) + expected = loaded_fields[field] + found = selected_fields[field] if found != expected: raise ValueError( f"Inference metadata field {field!r} does not match {source}: " @@ -257,6 +272,7 @@ def _assert_folder_metadata_matches( ) with open(metadata_path) as metadata_file: folder_values = json.load(metadata_file) + validate_persisted_metadata_versions(folder_values) required_semantics = { "id_maps", "selected_columns_statistics", @@ -312,11 +328,8 @@ def _assert_folder_metadata_matches( ) ): mismatches.append("prediction_alignment") - if ( - folder_metadata.tensor_payload_version - != selected_metadata.tensor_payload_version - ): - mismatches.append("tensor_payload_version") + if folder_metadata.tensor_payload_kind != selected_metadata.tensor_payload_kind: + mismatches.append("tensor_payload_kind") if folder_metadata.depth_layouts.compatibility_signature( relevant_columns ) != selected_metadata.depth_layouts.compatibility_signature(relevant_columns): @@ -357,9 +370,14 @@ def _assert_folder_metadata_matches( if folder_classes != selected_classes: mismatches.append("n_classes") - for field in ("id_maps", "selected_columns_statistics"): - selected_values = getattr(selected_metadata, field) - folder_values = getattr(folder_metadata, field) + for field, selected_values, folder_values in ( + ("id_maps", selected_metadata.id_maps, folder_metadata.id_maps), + ( + "selected_columns_statistics", + selected_metadata.selected_columns_statistics, + folder_metadata.selected_columns_statistics, + ), + ): if { column: selected_values[column] for column in relevant_columns @@ -388,7 +406,7 @@ def _assert_folder_metadata_matches( @beartype def load_inferer_config( config_path: str, args_config: dict, skip_metadata: bool -) -> "ResolvedInferenceConfig": +) -> "ResolvedInferenceConfig | ResolvedPolicyInferenceConfig": """Compose inference YAML, load model defaults, and resolve metadata.""" config_values = load_composed_yaml_config(config_path) @@ -396,13 +414,26 @@ def load_inferer_config( cli_values = { key: value for key, value in args_config.items() if key != "skip_metadata" } - authored_values = merge_config_fragments((config_values, cli_values)) + from sequifier.config.command_ownership import apply_run_overrides + + config_values, cli_values = apply_run_overrides(config_values, cli_values) + canonical_values = merge_config_fragments((config_values, cli_values)) + authored_values = canonical_values + artifacts = {} + policy_config = policy_inference_config(authored_values, artifacts) + if policy_config is not None: + if skip_metadata: + raise ValueError("skip_metadata is not supported for policy inference") + return policy_config + forecast_values, _ = extract_inline_metadata(authored_values) + request = InferenceConfig.model_validate(forecast_values) + authored_values["run"] = request.run.model_dump(mode="python", exclude_unset=True) authored_values, extracted_metadata_values = extract_inline_metadata( authored_values ) inline_metadata_values = extracted_metadata_values if skip_metadata else None - project_root = authored_values["project_root"] + project_root = request.project_root selected_metadata = None selected_metadata_path = None artifact_metadata_source = None @@ -410,9 +441,7 @@ def load_inferer_config( image_selections: list[tuple[set[str], bool]] = [] dataset_name = authored_values.get("dataset") - model_paths = authored_values.get("model_path") - if isinstance(model_paths, str): - model_paths = [model_paths] + model_paths = authored_values.get("model_paths") training_config_path = authored_values.get("training_config_path") route_selected = ( dataset_name is not None or authored_values.get("model_interface") is not None @@ -421,7 +450,9 @@ def load_inferer_config( route_selected or ( isinstance(model_paths, list) - and ("model_path" not in cli_values or "training_config_path" in cli_values) + and ( + "model_paths" not in cli_values or "training_config_path" in cli_values + ) and any( isinstance(path, str) and path.lower().endswith(".pt") for path in model_paths @@ -435,38 +466,40 @@ def load_inferer_config( if dataset_name is not None: if dataset_name not in datasets: raise ValueError(f"Unknown inference dataset {dataset_name!r}") - dataset = datasets[dataset_name] + interface = training.dataset_interface(dataset_name) + storage = training.dataset_storage(dataset_name) + source = f"training config dataset {dataset_name!r}" elif authored_values.get("model_interface") is not None: interface_name = authored_values["model_interface"] - dataset = next( - ( - candidate - for candidate in datasets.values() - if candidate.model_interface == interface_name - ), - None, - ) - if dataset is None: + if interface_name not in training.interfaces: raise ValueError(f"Unknown model interface {interface_name!r}") + interface = training.interface(interface_name) + storage = training.interface_storage[interface_name] + source = f"training config interface {interface_name!r}" elif len(datasets) == 1: - dataset = next(iter(datasets.values())) + name = next(iter(datasets)) + interface = training.dataset_interface(name) + storage = training.dataset_storage(name) + source = f"training config dataset {name!r}" else: raise ValueError( "A dataset or model_interface selection is required for the " "training config" ) - loaded_values, training_metadata = _execution_source(training, dataset) + loaded_values, training_metadata = _execution_source( + training.model_options, interface, storage + ) image_selections.append( ( - image_views(dataset.interface.ingestion), - dataset.interface.preconditioning_image_encoder is not None, + image_views(interface.ingestion), + interface.preconditioning_image_encoder is not None, ) ) - source = f"training config dataset {dataset.name!r}" _merge_loaded_execution_values(authored_values, loaded_values, source) loaded_metadata_sources.append((training_metadata, source, True)) if dataset_name is not None: + dataset = datasets[dataset_name] part_name = authored_values.get("part") if part_name is None: if len(dataset.parts) != 1: @@ -481,67 +514,32 @@ def load_inferer_config( selected_metadata = part.metadata selected_metadata_path = part.metadata_config_path - import torch - - from sequifier.artifacts.model_config import resolved_config_from_model_config + from sequifier.config.inference_common import SelectedInferenceModel autoregressive_decoder_targets: set[str] = set() if isinstance(model_paths, list): for model_path in model_paths: if not isinstance(model_path, str): continue - if model_path.lower().endswith(".pt"): - payload = torch.load( - normalize_path(model_path, project_root), - map_location="cpu", - weights_only=False, - ) - model_config = payload.get("model_config") - elif model_path.lower().endswith(".onnx"): - import onnx - - from sequifier.model.execution_schema import ( - EXECUTION_SCHEMA_KEY, - MODEL_CONFIG_KEY, - ) - - graph = onnx.load( - normalize_path(model_path, project_root), - load_external_data=False, - ) - properties = {item.key: item.value for item in graph.metadata_props} - model_config = ( - json.loads(properties[MODEL_CONFIG_KEY]) - if MODEL_CONFIG_KEY in properties - else None - ) - if ( - any( - len(value.type.tensor_type.shape.dim) == 3 - for value in graph.graph.input - ) - and EXECUTION_SCHEMA_KEY not in properties - ): - raise ValueError( - "Depth ONNX graphs require execution-schema metadata" - ) - else: - continue + model_config = inference_model_config(model_path, project_root, artifacts) if model_config is None: continue if ( - "model_path" in cli_values + "model_paths" in cli_values and "training_config_path" not in cli_values and not route_selected ): authored_values["training_config_path"] = None - training, interface_name = resolved_config_from_model_config( + selection = SelectedInferenceModel.select( + model_path, model_config, - device=str(authored_values.get("device", "cpu")), interface_name=authored_values.get("model_interface"), + task=authored_values["task"], ) - dataset = training.dataset_training[interface_name] - decoder = dataset.interface.decoder + training = selection.resolve(str(authored_values.get("device", "cpu"))) + interface_name = selection.interface_name + interface = training.interface(interface_name) + decoder = interface.decoder branches = ( decoder.branches.values() if decoder.type == "composite" else (decoder,) ) @@ -551,11 +549,15 @@ def load_inferer_config( if branch.type == "autoregressive_transformer" for target in branch.target_columns ) - loaded_values, artifact_metadata = _execution_source(training, dataset) + loaded_values, artifact_metadata = _execution_source( + training.options, + interface, + training.interface_storage.get(interface_name), + ) image_selections.append( ( - image_views(dataset.interface.ingestion), - dataset.interface.preconditioning_image_encoder is not None, + image_views(interface.ingestion), + interface.preconditioning_image_encoder is not None, ) ) if artifact_metadata_source is None: @@ -602,7 +604,7 @@ def load_inferer_config( else: raise ValueError( f"Inference config '{config_path}' must define metadata_config_path " - "or preprocessing_data_path unless the PT artifact contains model " + "unless a selected model artifact contains model " "metadata." ) @@ -626,31 +628,32 @@ def load_inferer_config( layout_is_authoritative, ) - return resolve_inference_config( + resolved = resolve_inference_config( config, metadata, allow_wider_artifact_folder=artifact_only_metadata ) + resolved._artifacts = artifacts + return resolved @beartype def _effective_metadata_config_path(config: "InferenceConfig") -> Optional[str]: if config.metadata_config_path: return config.metadata_config_path - if config.preprocessing_data_path: - return metadata_config_path_from_preprocessing_data_path( - config.preprocessing_data_path - ) return None @beartype def resolve_inference_config( - config: "InferenceConfig", + config: Any, metadata: DatasetMetadata, *, allow_wider_artifact_folder: bool = False, ) -> "ResolvedInferenceConfig": """Return an inference config with all metadata-derived values populated.""" + if config.context_length is None or config.training_objective is None: + raise ValueError("Inference requires context_length and training_objective") + if config.prediction_alignment_steps != metadata.prediction_alignment_steps: raise ValueError( "Inference config prediction_alignment_steps does not match dataset " @@ -659,11 +662,6 @@ def resolve_inference_config( f"metadata {metadata.prediction_alignment_steps}." ) storage_layout = metadata.storage_layout - if storage_layout.version != 2: - raise ValueError( - "Inference requires metadata stored_window_layout_version=2, " - f"got {storage_layout.version}." - ) available_column_data_types = config.column_data_types or metadata.column_data_types if image_targets := set(config.target_columns) & set(metadata.image_sources): raise ValueError( @@ -692,7 +690,7 @@ def resolve_inference_config( ] if missing_columns: raise ValueError( - "Inference metadata is missing selected columns: " f"{missing_columns}" + f"Inference metadata is missing selected columns: {missing_columns}" ) temporal_columns = [ column @@ -731,7 +729,7 @@ def resolve_inference_config( if selected_layouts: if config.read_format != "pt": raise ValueError("Preprocessed depth inputs require read_format: pt") - if config.autoregressive: + if config.autoregressive_generation: raise ValueError("Depth inputs do not support autoregressive inference") target_column_types = config.target_column_types or derive_target_column_types( config.target_columns, column_data_types @@ -832,7 +830,7 @@ def resolve_inference_config( f"Aligned inference requires window width at least {required}" ) if aligned_data_path: - if config.autoregressive: + if config.autoregressive_generation: raise ValueError( "Prediction aligned inference does not support autoregressive output" ) @@ -853,6 +851,7 @@ def resolve_inference_config( "Prediction aligned inference requires window_stride: null" ) values = config.model_dump(mode="python") + values["run"] = config.run.model_dump(mode="python", exclude_unset=True) values.update( { "metadata_config_path": _effective_metadata_config_path(config), @@ -883,56 +882,58 @@ def resolve_inference_config( _DataPathT = TypeVar("_DataPathT") _InputColumnsT = TypeVar("_InputColumnsT") _ColumnTypesT = TypeVar("_ColumnTypesT") +_ObjectiveT = TypeVar("_ObjectiveT", bound=str | None) +_ContextLengthT = TypeVar("_ContextLengthT", bound=int | None) class _InferenceConfigBase( - BaseModel, Generic[_DataPathT, _InputColumnsT, _ColumnTypesT] + InferenceExecutionConfig, + Generic[_DataPathT, _InputColumnsT, _ColumnTypesT, _ObjectiveT, _ContextLengthT], ): """Shared fields and validation for authored and resolved inference config.""" model_config = ConfigDict(arbitrary_types_allowed=True, extra="forbid") - project_root: str - preprocessing_data_path: Optional[str] = None - metadata_config_path: Optional[str] = None - model_path: Union[str, list[str]] - model_type: str - training_objective: str + model_paths: list[str] = Field(min_length=1) + task: Literal["forecast", "embedding"] + training_objective: _ObjectiveT data_path: _DataPathT training_config_path: Optional[str] = None dataset: Optional[str] = None part: Optional[str] = None - model_interface: Optional[str] = None - read_format: str = Field(default="parquet") - write_format: str = Field(default="csv") input_columns: _InputColumnsT static_columns: list[str] = Field(default_factory=list) static_features_path: Optional[str] = None - image_path: MediaPath = None - preconditioning_image_path: MediaPath = None + image_paths: MediaPath = None + preconditioning_image_paths: MediaPath = None target_columns: list[str] column_data_types: _ColumnTypesT target_column_types: _ColumnTypesT - deterministic: bool = Field(default=False) output_probabilities: bool = Field(default=False) decode_categories: bool = Field(default=True) - seed: int = 1010 - device: str - context_length: int = Field(gt=0) + context_length: _ContextLengthT = Field(gt=0) target_offset: int = Field(default=1, ge=0) window_stride: Optional[int] = Field(default=None, gt=0) prediction_length: Optional[int] = None - decoder_support: int = Field(default=1, gt=0) + decoder_support_length: int = Field(default=1, gt=0) prediction_alignment_steps: int = Field(default=1, gt=0) - inference_batch_size: int = Field(default=1, gt=0) sample_from_distribution_columns: Optional[list[str]] = Field(default=None) infer_with_dropout: bool = Field(default=False) - autoregressive: bool = Field(default=False) + rollout: RolloutConfig | None = None + autoregressive_generation: bool = Field(default=False) generation_steps: Optional[int] = Field(default=None) + def input_paths(self) -> list[str]: + inputs = super().input_paths() + if self.static_features_path: + inputs.append(normalize_path(self.static_features_path, self.project_root)) + if self.training_config_path: + inputs.append(self.training_config_path) + return inputs + @model_validator(mode="after") @beartype def validate_route_selection(self): @@ -947,35 +948,21 @@ def validate_route_selection(self): raise ValueError("part selection requires dataset selection") return self - @field_validator("input_columns", mode="before") - @classmethod - @beartype - def normalize_single_input_column(cls, value): - if isinstance(value, str): - return [value] - return value - @field_validator("training_objective") @classmethod @beartype - def validate_authored_training_objective(cls, value: str) -> str: - if value not in ALLOWED_OBJECTIVE_NAMES: + def validate_authored_training_objective(cls, value: str | None) -> str | None: + if value is not None and value not in ALLOWED_OBJECTIVE_NAMES: raise ValueError( f"Only {OBJECTIVE_NAME_MESSAGE} are allowed, found {value}" ) return value - @field_validator("model_type") - @classmethod - @beartype - def validate_authored_model_type(cls, value: str) -> str: - if value not in {"embedding", "generative"}: - raise ValueError("model_type must be either embedding or generative") - return value - @model_validator(mode="after") @beartype def validate_authored_paths(self): + if self.context_length is None or self.training_objective is None: + return self ModelWindowView( context_length=self.context_length, objective=self.training_objective, @@ -987,16 +974,44 @@ def validate_authored_paths(self): class InferenceConfig( - _InferenceConfigBase[Optional[str], Optional[list[str]], Optional[dict[str, str]]] + _InferenceConfigBase[ + Optional[str], + Optional[list[str]], + Optional[dict[str, str]], + str | None, + int | None, + ] ): """User-authored configuration for one inference run.""" + @model_validator(mode="after") + def validate_embedding_options(self): + generative_fields = { + "sample_from_distribution_columns", + "decode_categories", + "output_probabilities", + "autoregressive_generation", + "generation_steps", + "rollout", + } + if self.task == "embedding" and ( + invalid := self.model_fields_set & generative_fields + ): + raise ValueError(f"embedding inference does not support {sorted(invalid)}") + return self + + training_objective: str | None = None + context_length: int | None = Field(default=None, gt=0) + input_columns: list[str] | None = None + target_columns: list[str] = Field(default_factory=list) data_path: Optional[str] = None column_data_types: Optional[dict[str, str]] = None target_column_types: Optional[dict[str, str]] = None -class ResolvedInferenceConfig(_InferenceConfigBase[str, list[str], dict[str, str]]): +class ResolvedInferenceConfig( + _InferenceConfigBase[str, list[str], dict[str, str], str, int] +): """Internal inference config after dataset metadata has been resolved.""" categorical_columns: list[str] @@ -1006,6 +1021,55 @@ class ResolvedInferenceConfig(_InferenceConfigBase[str, list[str], dict[str, str window_view: ModelWindowView dataset_metadata: Optional[DatasetMetadata] = Field(default=None, exclude=True) _image_store: Any = PrivateAttr(default=None) + _artifacts: dict[str, Any] = PrivateAttr(default_factory=dict) + + @model_validator(mode="after") + def validate_numeric_rollout(self): + if self.rollout is None: + return self + if self.autoregressive_generation or self.generation_steps is not None: + raise ValueError( + "rollout uses max_steps, not autoregressive/generation_steps" + ) + if self.device != "cpu" or self.task != "forecast": + raise ValueError("numeric rollout requires a CPU generative model") + if ( + self.training_objective not in {"causal", "final_value"} + or self.window_view.target_offset != 1 + or self.prediction_length not in (None, 1) + ): + raise ValueError( + "forecast rollout requires a one-step causal/final_value forecast with target_offset=1" + ) + if self.output_probabilities or self.sample_from_distribution_columns: + raise ValueError( + "numeric rollout does not support categorical probabilities or sampling" + ) + paths = self.model_paths + if len(paths) != 1: + raise ValueError("rollout requires exactly one entry in model_paths") + fields = self.input_columns + self.static_columns + self.target_columns + if not self.input_columns or any( + "float" not in self.column_data_types[column].lower() for column in fields + ): + raise ValueError( + "numeric rollout supports real scalar inputs, static inputs, and targets only" + ) + if set(self.input_columns) & set(self.static_columns): + raise ValueError("rollout static and changing inputs must be disjoint") + if any(kind != "real" for kind in self.target_column_types.values()): + raise ValueError("numeric rollout requires real forecast targets") + metadata = self.dataset_metadata + if metadata is None: + raise ValueError("rollout requires fixed training preprocessing metadata") + if ( + metadata.image_sources + or metadata.preconditioning_image is not None + or metadata.depth_layouts.relevant_layouts(fields) + ): + raise ValueError("numeric rollout does not support image or depth inputs") + self.rollout.validate_bindings(self.input_columns, self.target_columns) + return self def image_store(self): if self._image_store is None and self.dataset_metadata is not None: @@ -1014,8 +1078,8 @@ def image_store(self): sources, preconditioning = bind_image_paths( self.dataset_metadata.image_sources, self.dataset_metadata.preconditioning_image, - self.image_path, - self.preconditioning_image_path, + self.image_paths, + self.preconditioning_image_paths, ) self._image_store = ImageStore( sources, @@ -1042,13 +1106,6 @@ def derive_optional_config_values(cls, values): values["target_offset"] = window_view.get("target_offset", 1) elif "target_offset" not in values and isinstance(window_view, ModelWindowView): values["target_offset"] = window_view.target_offset - preprocessing_data_path = values.get("preprocessing_data_path") - if values.get("metadata_config_path") is None and preprocessing_data_path: - values["metadata_config_path"] = ( - metadata_config_path_from_preprocessing_data_path( - preprocessing_data_path - ) - ) if ( values.get("target_column_types") is None and values.get("column_data_types") is not None @@ -1082,11 +1139,13 @@ def normalize_prediction_length(self): self.window_view.context_length ) if objective_class.forward_looking: - decoded_length = self.window_view.context_length - self.decoder_support + 1 + decoded_length = ( + self.window_view.context_length - self.decoder_support_length + 1 + ) if not 1 <= self.prediction_length <= decoded_length: raise ValueError( "Forward inference prediction_length must be between 1 and " - f"context_length - decoder_support + 1 ({decoded_length})" + f"context_length - decoder_support_length + 1 ({decoded_length})" ) objective_class.validate_prediction_length( self.prediction_length, @@ -1105,24 +1164,11 @@ def validate_training_objective(cls, v): raise ValueError(f"Only {OBJECTIVE_NAME_MESSAGE} are allowed, found {v}") return v - @field_validator("model_type") - @classmethod - @beartype - def validate_model_type(cls, v: str) -> str: - if v not in [ - "embedding", - "generative", - ]: - raise ValueError( - f"model_type must be one of 'embedding' and 'generative, {v} isn't" - ) - return v - @field_validator("output_probabilities") @classmethod @beartype def validate_output_probabilities(cls, v: bool, info: Any) -> bool: - if v and info.data.get("model_type") == "embedding": + if v and info.data.get("task") == "embedding": raise ValueError( "For embedding models, 'output_probabilities' must be set to false" ) @@ -1131,9 +1177,7 @@ def validate_output_probabilities(cls, v: bool, info: Any) -> bool: @model_validator(mode="after") @beartype def validate_training_config_path(self): - model_paths = ( - self.model_path if isinstance(self.model_path, list) else [self.model_path] - ) + model_paths = self.model_paths if not any(path.lower().endswith(".pt") for path in model_paths): return self @@ -1147,19 +1191,19 @@ def validate_training_config_path(self): @classmethod @beartype def validate_generation_steps(cls, v: Optional[int], info: Any) -> Optional[int]: - if v is None and info.data.get("autoregressive") is True: + if v is None and info.data.get("autoregressive_generation") is True: raise ValueError( - "If autoregressive==True, 'generation_steps' needs to be set to an integer value." + "If autoregressive_generation==True, 'generation_steps' needs to be set to an integer value." ) if v is not None and v < 1: raise ValueError("generation_steps must by >= 1.") if v is not None and v > 1: - if not info.data.get("autoregressive"): + if not info.data.get("autoregressive_generation"): raise ValueError( - f"'generation_steps' can only be larger than 1 if 'autoregressive' is true: {info.data.get('autoregressive')}" + f"'generation_steps' can only be larger than 1 if 'autoregressive_generation' is true: {info.data.get('autoregressive_generation')}" ) - if not np.all( + if info.data.get("rollout") is None and not np.all( np.array(sorted(info.data.get("input_columns"))) == np.array(sorted(info.data.get("target_columns"))) ): @@ -1169,11 +1213,11 @@ def validate_generation_steps(cls, v: Optional[int], info: Any) -> Optional[int] return v - @field_validator("autoregressive") + @field_validator("autoregressive_generation") @classmethod @beartype def validate_autoregressive(cls, v: bool, info: Any): - if v and info.data.get("model_type") == "embedding": + if v and info.data.get("task") == "embedding": raise ValueError( "Autoregressive inference is not possible for embedding models" ) @@ -1185,9 +1229,13 @@ def validate_autoregressive(cls, v: bool, info: Any): raise ValueError( "Autoregressive inference is not possible for models with prediction_length > 1" ) - if v and not np.all( - np.array(sorted(info.data.get("input_columns"))) - == np.array(sorted(info.data.get("target_columns"))) + if ( + v + and info.data.get("rollout") is None + and not np.all( + np.array(sorted(info.data.get("input_columns"))) + == np.array(sorted(info.data.get("target_columns"))) + ) ): raise ValueError( "Autoregressive inference with non-identical 'input_columns' and 'target_columns' is possible but should not be performed" @@ -1212,7 +1260,7 @@ def validate_autoregressive(cls, v: bool, info: Any): def validate_data_path(cls, v: Optional[str], info: Any) -> Optional[str]: if v is None: return v - v2 = normalize_path(v, info.data.get("project_root")) + v2 = normalize_path(v, info.data["run"].project_root) if not os.path.exists(v2): raise ValueError(f"{v2} does not exist") return v diff --git a/src/sequifier/config/inference_common.py b/src/sequifier/config/inference_common.py new file mode 100644 index 000000000..baa82948d --- /dev/null +++ b/src/sequifier/config/inference_common.py @@ -0,0 +1,197 @@ +"""Execution options consumed by every inference route.""" + +import json +from dataclasses import dataclass +from pathlib import Path +from typing import TYPE_CHECKING, Any, Literal + +from pydantic import ConfigDict, Field, field_validator, model_validator + +from sequifier.config.command_ownership import CommandRunSettings +from sequifier.config.naming import ConfigModel +from sequifier.helpers import normalize_path +from sequifier.io.output_paths import validate_inference_destination + + +class InferenceExecutionConfig(ConfigModel): + model_config = ConfigDict(extra="forbid", arbitrary_types_allowed=True) + + run: CommandRunSettings = Field(default_factory=CommandRunSettings) + model_interface: str | None = None + read_format: str = "parquet" + write_format: str = "csv" + output_path: str | None = None + metadata_config_path: str | None = None + + if TYPE_CHECKING: + model_paths: list[str] + data_path: Any + task: Any + rollout: Any + + @property + def project_root(self): + return self.run.project_root + + @property + def device(self): + return self.run.execution.device + + @property + def seed(self): + return self.run.reproducibility.seed + + @property + def enforce_determinism(self): + return self.run.reproducibility.enforce_determinism + + @property + def inference_batch_size(self): + return self.run.execution.batch_size + + @model_validator(mode="after") + def validate_resources(self): + self.run.validate_for("infer") + if self.output_path is not None and len(self.model_paths) != 1: + raise ValueError("output_path requires exactly one model path") + rollout = self.rollout + if rollout is not None: + self.validate_output_destination( + normalize_path(rollout.trajectory_path, self.project_root), + label="trajectory_path", + ) + elif self.output_path is not None: + destination = ( + normalize_path(self.output_path, self.project_root) + if self.task == "policy" + else Path(self.project_root) / self.output_path + ) + self.validate_output_destination(destination, label="output_path") + return self + + def validate_output_destination( + self, destination: str | Path, *, label: str + ) -> None: + inputs = self.input_paths() + validate_inference_destination( + destination, + inputs, + label=label, + ) + + def input_paths(self) -> list[str]: + inputs = list(self.model_paths) + inputs.extend( + path for path in (self.data_path, self.metadata_config_path) if path + ) + return [normalize_path(path, self.project_root) for path in inputs] + + +class InferenceRouteConfig(ConfigModel): + """Validate artifact and interface selection before route-specific options.""" + + model_config = ConfigDict(extra="ignore") + run: CommandRunSettings = Field(default_factory=CommandRunSettings) + model_paths: list[str] = Field(min_length=1) + model_interface: str | None = None + task: Literal["forecast", "policy", "embedding"] + + @property + def project_root(self): + return self.run.project_root + + @field_validator("model_paths") + @classmethod + def validate_paths(cls, value): + paths = value + if not paths or any(not path.strip() for path in paths): + raise ValueError("model_paths requires at least one nonempty artifact path") + return value + + @field_validator("model_interface") + @classmethod + def validate_interface(cls, value): + if value is not None and not value.isidentifier(): + raise ValueError("model_interface must be a valid identifier without '.'") + return value + + +@dataclass(frozen=True) +class SelectedInferenceModel: + path: str + values: dict + interface_name: str + + @classmethod + def select(cls, path, values, *, interface_name=None, task=None): + interfaces = values.get("interfaces") + if not isinstance(interfaces, dict) or not interfaces: + raise ValueError("model_config.interfaces must be a non-empty mapping") + if interface_name is None: + policies = [ + name + for name, interface in interfaces.items() + if interface.get("policy") is not None + ] + if len(interfaces) > 1 and policies and task != "forecast": + raise ValueError( + "Artifact has multiple interfaces; select model_interface before choosing task" + ) + else: + interface_name = next(iter(interfaces)) + if interface_name not in interfaces: + raise ValueError(f"Unknown model_interface {interface_name!r}") + return cls(path, values, interface_name) + + @property + def policy_bound(self): + return self.values["interfaces"][self.interface_name].get("policy") is not None + + def resolve(self, device): + from sequifier.artifacts.model_config import model_execution_from_model_config + + return model_execution_from_model_config( + self.values, device=device, interface_name=self.interface_name + )[0] + + +def inference_model_config(path, project_root, artifacts): + """Read the interface contract once for PT artifacts, or from ONNX metadata.""" + from sequifier.artifacts.model_artifact import read_execution_artifact + + normalized = normalize_path(path, project_root) + if path.lower().endswith(".pt"): + if normalized not in artifacts: + artifacts[normalized] = artifacts.get(path) or read_execution_artifact( + normalized + ) + artifacts[path] = artifacts[normalized] + return artifacts[path].model_config.values + if not path.lower().endswith(".onnx"): + return None + import onnx + + from sequifier.model.execution_schema import ( + DROPOUT_MODE_KEY, + EXECUTION_SCHEMA_KEY, + MODEL_CONFIG_KEY, + ExecutionSchema, + ) + + graph = onnx.load(normalized, load_external_data=False) + properties = {item.key: item.value for item in graph.metadata_props} + schema = properties.get(EXECUTION_SCHEMA_KEY) + if ( + schema is not None + or MODEL_CONFIG_KEY in properties + or DROPOUT_MODE_KEY in properties + or any( + len(value.type.tensor_type.shape.dim) == 3 for value in graph.graph.input + ) + ): + ExecutionSchema.from_dict(json.loads(schema) if schema is not None else None) + return ( + json.loads(properties[MODEL_CONFIG_KEY]) + if MODEL_CONFIG_KEY in properties + else None + ) diff --git a/src/sequifier/config/initialization_config.py b/src/sequifier/config/initialization_config.py index a8e7778b6..7d428fd18 100644 --- a/src/sequifier/config/initialization_config.py +++ b/src/sequifier/config/initialization_config.py @@ -3,24 +3,17 @@ import math from typing import Annotated, Any, Literal, TypeAlias, Union -from pydantic import ( - BaseModel, - BeforeValidator, - ConfigDict, - Field, - RootModel, - field_serializer, - model_validator, -) +from pydantic import ConfigDict, Field, RootModel, field_serializer, model_validator from sequifier.config.layer_groups import LayerGroup +from sequifier.config.naming import ConfigModel from sequifier.typechecking import beartype ParameterKind: TypeAlias = Literal["weight", "bias"] InitializationTarget: TypeAlias = tuple[LayerGroup, ParameterKind] -class _InitializationMethod(BaseModel): +class _InitializationMethod(ConfigModel): """Base class for strictly validated initialization methods.""" model_config = ConfigDict(extra="forbid") @@ -52,12 +45,7 @@ def validate_bounds(self): class XavierUniformInitialization(_InitializationMethod): - method: Literal[ - "xavier", - "glorot", - "xavier_uniform", - "glorot_uniform", - ] + method: Literal["xavier_uniform"] gain: float = Field(1.0, ge=0.0) fan_mode: Literal["per_tensor", "joint"] = "per_tensor" @@ -68,7 +56,7 @@ def serialize_method(self, method): class XavierNormalInitialization(_InitializationMethod): - method: Literal["xavier_normal", "glorot_normal"] + method: Literal["xavier_normal"] gain: float = Field(1.0, ge=0.0) fan_mode: Literal["per_tensor", "joint"] = "per_tensor" @@ -113,11 +101,6 @@ class IdentityPlusNormalInitialization(_InitializationMethod): std: float = Field(0.02, ge=0.0) -@beartype -def _expand_method_shorthand(value: Any) -> Any: - return {"method": value} if isinstance(value, str) else value - - InitializationMethodConfig: TypeAlias = Annotated[ Union[ PreserveInitialization, @@ -133,11 +116,10 @@ def _expand_method_shorthand(value: Any) -> Any: IdentityPlusNormalInitialization, ], Field(discriminator="method"), - BeforeValidator(_expand_method_shorthand), ] -class LayerGroupInitialization(BaseModel): +class LayerGroupInitialization(ConfigModel): """Optional weight and bias overrides for one semantic layer group.""" model_config = ConfigDict(extra="forbid") @@ -145,6 +127,14 @@ class LayerGroupInitialization(BaseModel): weight: InitializationMethodConfig | None = None bias: InitializationMethodConfig | None = None + def parameter_kinds(self) -> tuple[ParameterKind, ...]: + kinds: list[ParameterKind] = [] + if self.weight is not None: + kinds.append("weight") + if self.bias is not None: + kinds.append("bias") + return tuple(kinds) + @model_validator(mode="after") @beartype def validate_not_empty(self): @@ -167,12 +157,11 @@ def configured_targets(self) -> set[InitializationTarget]: return { (group, parameter_kind) for group, initialization in self.root.items() - for parameter_kind in ("weight", "bias") - if getattr(initialization, parameter_kind) is not None + for parameter_kind in initialization.parameter_kinds() } -class InitializationMethodCandidates(BaseModel): +class InitializationMethodCandidates(ConfigModel): """Candidate initialization methods for one weight or bias group.""" model_config = ConfigDict(extra="forbid") @@ -186,7 +175,7 @@ class InitializationMethodCandidates(BaseModel): ] -class LayerGroupInitializationSampling(BaseModel): +class LayerGroupInitializationSampling(ConfigModel): """Fixed or sampled weight and bias initialization for one layer group.""" model_config = ConfigDict(extra="forbid") @@ -194,6 +183,14 @@ class LayerGroupInitializationSampling(BaseModel): weight: InitializationMethodSamplingConfig | None = None bias: InitializationMethodSamplingConfig | None = None + def parameter_kinds(self) -> tuple[ParameterKind, ...]: + kinds: list[ParameterKind] = [] + if self.weight is not None: + kinds.append("weight") + if self.bias is not None: + kinds.append("bias") + return tuple(kinds) + @model_validator(mode="after") @beartype def validate_not_empty(self): diff --git a/src/sequifier/config/input_contract_resolution.py b/src/sequifier/config/input_contract_resolution.py new file mode 100644 index 000000000..eeb1868fd --- /dev/null +++ b/src/sequifier/config/input_contract_resolution.py @@ -0,0 +1,45 @@ +"""Assert authored interface facts against frozen artifact/source metadata.""" + + +def validate_input_contracts(config, resolved, declarations): + if config.model is None: + return + from sequifier.config.train_config import _assert_compatible, _part_signature + + for name, spec in config.model.interfaces.items(): + declaration = spec.input_contract + datasets = [ + value + for value in resolved.dataset_training.values() + if value.model_interface == name + ] + if datasets and name in declarations: + expected = declarations[name] + for dataset in datasets: + for part in dataset.parts.values(): + _assert_compatible( + _part_signature(expected, config.model.interfaces[name]), + _part_signature(part.metadata, config.model.interfaces[name]), + f"model.interfaces.{name}.input_contract", + ) + normalization = declaration.normalization + if normalization is None: + continue + resolved.normalization_by_interface[name] = normalization + interface = resolved.interface(name) + expected = normalization.statistics + real_inputs = interface.real_columns + interface.static_real_columns + if normalization.source == "identity": + expected = {column: {"mean": 0.0, "std": 1.0} for column in real_inputs} + for column, values in expected.items(): + if column not in real_inputs: + raise ValueError( + f"{name}: normalization references unknown/non-real input {column!r}" + ) + actual = interface.selected_columns_statistics.get(column) + if not interface.normalize_real_columns: + actual = {"mean": 0.0, "std": 1.0} + if actual != values: + raise ValueError( + f"model.interfaces.{name}.input_contract.normalization.{column} differs from frozen metadata/artifact" + ) diff --git a/src/sequifier/config/learning_config.py b/src/sequifier/config/learning_config.py new file mode 100644 index 000000000..e5b6f0a6c --- /dev/null +++ b/src/sequifier/config/learning_config.py @@ -0,0 +1,57 @@ +"""Phase-owned supervised losses and sampling views.""" + +import math +from typing import Optional + +import torch +from pydantic import ConfigDict, Field, model_validator + +from sequifier.config.components import BERTSpecModel, NextOccurrenceConfigModel +from sequifier.config.naming import ConfigModel + + +class DatasetLearningConfig(ConfigModel): + model_config = ConfigDict(extra="forbid") + criterion: dict[str, str] = Field(default_factory=dict) + class_weights: Optional[dict[str, list[float]]] = None + loss_weights: Optional[dict[str, float]] = None + class_share_log_columns: list[str] = Field(default_factory=list) + + def validate_targets(self, targets, usage): + targets = set(targets) + if set(self.criterion) != targets: + raise ValueError(f"{usage}: criterion keys must equal interface targets") + for label, weights in ( + ("class_weights", self.class_weights), + ("loss_weights", self.loss_weights), + ): + if weights is not None and weights.keys() - targets: + raise ValueError(f"{usage}: {label} references unknown targets") + if self.loss_weights is not None and all( + self.loss_weights.get(target, 1.0) == 0 for target in targets + ): + raise ValueError( + f"{usage}: at least one target must have positive loss weight" + ) + + @model_validator(mode="after") + def losses(self): + for name in self.criterion.values(): + if name != "AccurateCrossEntropyLoss" and name not in vars(torch.nn): + raise ValueError(f"unknown criterion {name!r}") + for weights in (self.loss_weights, self.class_weights): + for value in (weights or {}).values(): + values = value if isinstance(value, list) else [value] + if any(not math.isfinite(x) or x < 0 for x in values): + raise ValueError( + "loss and class weights must be finite and nonnegative" + ) + return self + + +class SupervisedLearningConfig(ConfigModel): + model_config = ConfigDict(extra="forbid") + datasets: dict[str, DatasetLearningConfig] = Field(default_factory=dict) + bert_spec: BERTSpecModel | None = None + next_occurrence_config: NextOccurrenceConfigModel | None = None + window_stride: int | None = Field(default=None, gt=0) diff --git a/src/sequifier/config/metadata.py b/src/sequifier/config/metadata.py index 047938859..e81d7d87a 100644 --- a/src/sequifier/config/metadata.py +++ b/src/sequifier/config/metadata.py @@ -4,19 +4,14 @@ import copy import json -from typing import Any +from typing import Any, Literal -from pydantic import ( - AliasChoices, - BaseModel, - ConfigDict, - Field, - field_validator, - model_validator, -) +from pydantic import ConfigDict, Field, ValidationInfo, field_validator, model_validator from sequifier.config.depth_layout import DepthLayoutRegistryModel from sequifier.config.image_config import ImageSourceConfig, PreconditioningImageConfig +from sequifier.config.naming import ConfigModel +from sequifier.contract_versions import ContractVersions, require_contract_version from sequifier.helpers import ModelWindowView, StoredWindowLayout from sequifier.special_tokens import SPECIAL_TOKEN_IDS, validate_special_token_ids from sequifier.typechecking import beartype @@ -38,20 +33,67 @@ "image_sources", "time_alignment_column", "preconditioning_image", + "tensor_payload_kind", "tensor_payload_version", "prediction_aligned_splits", "split_index", } -class DatasetMetadata(BaseModel): +TensorPayloadKind = Literal["tabular", "depth"] + +_TENSOR_PAYLOAD_VERSIONS: dict[TensorPayloadKind, int] = { + "tabular": ContractVersions.DATASET_TENSOR_METADATA, + "depth": ContractVersions.DEPTH_TENSOR_METADATA, +} + + +def dataset_tensor_kind(depth_layouts) -> TensorPayloadKind: + return "depth" if depth_layouts else "tabular" + + +def dataset_tensor_version(kind: TensorPayloadKind) -> int: + return _TENSOR_PAYLOAD_VERSIONS[kind] + + +def validate_tensor_payload_contract(kind, version, depth_layouts, *, projected=False): + expected_kind = dataset_tensor_kind(depth_layouts) + # A shallow model interface may come from a dataset that has depth columns. + allowed = ( + ("tabular", "depth") if projected and not depth_layouts else (expected_kind,) + ) + if kind not in allowed: + raise ValueError( + f"tensor_payload_kind must be one of {allowed!r}, got {kind!r}" + ) + require_contract_version( + f"{kind} dataset tensor metadata", version, dataset_tensor_version(kind) + ) + + +def validate_persisted_metadata_versions(values: dict) -> None: + if not isinstance(values, dict): + raise ValueError("Dataset metadata must contain a mapping") + validate_tensor_payload_contract( + values.get("tensor_payload_kind"), + values.get("tensor_payload_version"), + values.get("depth_layouts"), + ) + require_contract_version( + "stored-window layout", + values.get("stored_window_layout_version"), + ContractVersions.STORED_WINDOW_LAYOUT, + ) + + +class DatasetMetadata(ConfigModel): """The stable subset of preprocessing metadata consumed by other commands.""" model_config = ConfigDict(extra="allow", populate_by_name=True) @model_validator(mode="before") @classmethod - def reject_legacy_placement(cls, values: Any) -> Any: + def normalize_inline_metadata(cls, values: Any) -> Any: if isinstance(values, dict) and { "window_strides", "window_placement", @@ -60,17 +102,24 @@ def reject_legacy_placement(cls, values: Any) -> Any: raise ValueError( "Preprocessing metadata uses obsolete placement settings; re-run preprocessing" ) + if isinstance(values, dict): + kind = dataset_tensor_kind(values.get("depth_layouts")) + values = { + "tensor_payload_kind": kind, + "tensor_payload_version": dataset_tensor_version(kind), + **values, + } return values depth_layouts: DepthLayoutRegistryModel = Field( default_factory=DepthLayoutRegistryModel ) - tensor_payload_version: int = Field(default=1, ge=1, le=3) + tensor_payload_kind: TensorPayloadKind + tensor_payload_version: int = Field(strict=True) split_paths: list[str] = Field(default_factory=list) split_index: int | None = Field(default=None, ge=0) column_data_types: dict[str, str] = Field( default_factory=dict, - validation_alias=AliasChoices("column_data_types", "column_types"), ) static_columns: list[str] = Field(default_factory=list) static_features_path: str | None = None @@ -95,12 +144,25 @@ def reject_legacy_placement(cls, values: Any) -> Any: window_length: int = Field(gt=0) window_lengths: list[int] | None = None max_target_offset: int = Field(default=1, ge=0) - stored_window_layout_version: int = Field(default=2, ge=2, le=2) + stored_window_layout_version: int = Field( + default=ContractVersions.STORED_WINDOW_LAYOUT, + strict=True, + json_schema_extra={"const": ContractVersions.STORED_WINDOW_LAYOUT}, + ) @model_validator(mode="after") - def validate_layout_metadata(self): - if self.depth_layouts and self.tensor_payload_version not in {2, 3}: - raise ValueError("Depth datasets require tensor_payload_version 2 or 3") + def validate_layout_metadata(self, info: ValidationInfo): + validate_tensor_payload_contract( + self.tensor_payload_kind, + self.tensor_payload_version, + self.depth_layouts, + projected=(info.context or {}).get("projected_depth_layouts", False), + ) + require_contract_version( + "stored-window layout", + self.stored_window_layout_version, + ContractVersions.STORED_WINDOW_LAYOUT, + ) if self.window_lengths is not None and ( (self.split_paths and len(self.window_lengths) != len(self.split_paths)) or ( @@ -191,15 +253,27 @@ def load_dataset_metadata(path: str) -> DatasetMetadata: values: Any = json.load(file) if not isinstance(values, dict): raise ValueError(f"Metadata config '{path}' must contain a JSON object.") + validate_persisted_metadata_versions(values) return DatasetMetadata.model_validate(values) +def validate_authored_metadata(values: dict) -> None: + layout = values.get("storage_layout") + if ( + "tensor_payload_version" in values + or "stored_window_layout_version" in values + or (isinstance(layout, dict) and "version" in layout) + ): + raise ValueError("Inline metadata cannot declare internal storage versions") + + @beartype def extract_inline_metadata( values: dict[str, Any], ) -> tuple[dict[str, Any], dict[str, Any] | None]: - """Split the historical ``skip_metadata`` representation into two mappings.""" + """Separate inline metadata facts from the authored request.""" + validate_authored_metadata(values) authored = copy.deepcopy(values) window_view = authored.get("window_view") if "context_length" not in authored and isinstance(window_view, dict): @@ -221,7 +295,11 @@ def extract_inline_metadata( if path is not None ], "depth_layouts": authored.get("depth_layouts", {}), - "tensor_payload_version": authored.get("tensor_payload_version", 1), + **( + {"tensor_payload_kind": authored["tensor_payload_kind"]} + if "tensor_payload_kind" in authored + else {} + ), "column_data_types": authored.get("column_data_types", {}), "static_columns": authored.get("static_columns", []), "static_features_path": authored.get("static_features_path"), @@ -251,7 +329,6 @@ def extract_inline_metadata( { "window_length": storage_layout.window_length, "max_target_offset": storage_layout.max_target_offset, - "stored_window_layout_version": storage_layout.version, } ) elif isinstance(storage_layout, dict): @@ -259,7 +336,6 @@ def extract_inline_metadata( { "window_length": storage_layout.get("window_length"), "max_target_offset": storage_layout.get("max_target_offset", 1), - "stored_window_layout_version": storage_layout.get("version", 2), } ) else: @@ -267,9 +343,6 @@ def extract_inline_metadata( { "window_length": authored.get("window_length"), "max_target_offset": authored.get("max_target_offset", 1), - "stored_window_layout_version": authored.get( - "stored_window_layout_version", 2 - ), } ) diff --git a/src/sequifier/config/naming.py b/src/sequifier/config/naming.py new file mode 100644 index 000000000..03d86befb --- /dev/null +++ b/src/sequifier/config/naming.py @@ -0,0 +1,21 @@ +"""Shared configuration defaults and run-seed resolution.""" + +import secrets +from typing import Annotated, Any + +from pydantic import BaseModel, BeforeValidator, ConfigDict + + +class ConfigModel(BaseModel): + model_config = ConfigDict(extra="forbid") + + +def resolve_run_seed(value: Any) -> int: + if value == "random": + return secrets.randbelow(1_000_000_000) + if isinstance(value, bool) or not isinstance(value, int): + raise ValueError("seed must be an integer or 'random'") + return value + + +RunSeed = Annotated[int, BeforeValidator(resolve_run_seed)] diff --git a/src/sequifier/config/normalization_config.py b/src/sequifier/config/normalization_config.py new file mode 100644 index 000000000..0479c04f5 --- /dev/null +++ b/src/sequifier/config/normalization_config.py @@ -0,0 +1,29 @@ +"""Frozen interface normalization facts, independent of learning algorithms.""" + +import math +from typing import Literal + +from pydantic import ConfigDict, Field, model_validator + +from sequifier.config.naming import ConfigModel + + +class FrozenNormalizationConfig(ConfigModel): + model_config = ConfigDict(extra="forbid") + source: Literal["identity", "statistics"] + statistics: dict[str, dict[str, float]] = Field(default_factory=dict) + + @model_validator(mode="after") + def validate_statistics(self): + if self.source == "identity" and self.statistics: + raise ValueError("identity normalization cannot include statistics") + for column, values in self.statistics.items(): + if ( + set(values) != {"mean", "std"} + or not all(math.isfinite(value) for value in values.values()) + or values["std"] < 0 + ): + raise ValueError( + f"{column}: statistics require finite mean and nonnegative std" + ) + return self diff --git a/src/sequifier/config/optimizer_config.py b/src/sequifier/config/optimizer_config.py index 92110343a..3c847c504 100644 --- a/src/sequifier/config/optimizer_config.py +++ b/src/sequifier/config/optimizer_config.py @@ -4,44 +4,16 @@ from typing import Any, Literal -from pydantic import BaseModel, ConfigDict, Field, model_validator +from pydantic import ConfigDict, Field, model_validator from sequifier.config.components import ComponentSpec +from sequifier.config.naming import ConfigModel +from sequifier.config.parameter_selection import ParameterSelector +OptimizerSelector = ParameterSelector -class OptimizerSelector(BaseModel): - """Match parameter catalog descriptors; supplied conditions are conjunctive.""" - model_config = ConfigDict(extra="forbid", frozen=True) - - semantic_groups: list[str] | None = None - component: Literal["ingestion", "backbone", "decoder"] | None = None - parameter_kind: Literal["weight", "bias", "other"] | None = None - ndim: int | None = Field(default=None, ge=0) - - @model_validator(mode="after") - def validate_selector(self) -> "OptimizerSelector": - if not any( - value is not None - for value in ( - self.semantic_groups, - self.component, - self.parameter_kind, - self.ndim, - ) - ): - raise ValueError( - "An optimizer selector must specify at least one condition." - ) - if self.semantic_groups is not None and ( - not self.semantic_groups - or any(not pattern for pattern in self.semantic_groups) - ): - raise ValueError("semantic_groups must contain nonempty patterns.") - return self - - -class OptimizerGroupSpec(BaseModel): +class OptimizerGroupSpec(ConfigModel): """One named route and the optimizer applied to its parameters.""" model_config = ConfigDict(extra="forbid", frozen=True) @@ -53,7 +25,7 @@ class OptimizerGroupSpec(BaseModel): base_learning_rate: float | None = Field(default=None, ge=0) -class OptimizerPlan(BaseModel): +class OptimizerPlan(ConfigModel): """Ordered, exhaustive optimizer routes with a final fallback.""" model_config = ConfigDict(extra="forbid", frozen=True) @@ -88,7 +60,7 @@ def resolve_plan_scheduler_arguments( if scheduler.name == "CyclicLR": if "base_lr" in arguments or "max_lr" in arguments: raise ValueError( - "global_training.scheduler.base_lr and max_lr must be omitted " + "optimization..scheduler.base_lr and max_lr must be omitted " "with an optimizer plan; set rates in each optimizer group." ) missing = [ @@ -116,7 +88,7 @@ def resolve_plan_scheduler_arguments( if scheduler.name == "OneCycleLR": if "max_lr" in arguments: raise ValueError( - "global_training.scheduler.max_lr must be omitted with an " + "optimization..scheduler.max_lr must be omitted with an " "optimizer plan; each group's learning_rate is its peak rate." ) arguments["max_lr"] = [group.learning_rate for group in plan.groups] diff --git a/src/sequifier/config/ownership.py b/src/sequifier/config/ownership.py new file mode 100644 index 000000000..e55503a6e --- /dev/null +++ b/src/sequifier/config/ownership.py @@ -0,0 +1,129 @@ +"""Run resources and durable model contracts, independent of learning algorithms.""" + +from typing import Literal + +from pydantic import ConfigDict, Field, StrictInt, field_validator, model_validator + +from sequifier.config.components import ResumeConfig +from sequifier.config.naming import ConfigModel, RunSeed +from sequifier.config.normalization_config import FrozenNormalizationConfig + + +class Settings(ConfigModel): + model_config = ConfigDict(extra="forbid", allow_inf_nan=False) + + +class DistributionSettings(Settings): + strategy: Literal["local", "ddp", "fsdp"] = "local" + world_size: int = Field(default=1, gt=0) + backend: str = "nccl" + cpu_offload: bool | None = None + + @model_validator(mode="after") + def coherent(self): + if self.strategy == "local" and self.world_size != 1: + raise ValueError("local distribution requires world_size: 1") + if self.strategy != "fsdp" and self.cpu_offload is not None: + raise ValueError("cpu_offload is only valid with fsdp") + if self.strategy == "fsdp" and self.cpu_offload is None: + raise ValueError("fsdp requires explicit cpu_offload") + return self + + +class PrecisionSettings(Settings): + layer_type_dtypes: dict[str, str] | None = None + layer_autocast: bool = False + float32_matmul_precision: Literal["highest", "high", "medium"] = "highest" + + +class CompilationSettings(Settings): + torch_compile: Literal["outer", "inner", "none"] = "outer" + + +class DataLoadingSettings(Settings): + num_workers: int = Field(default=0, ge=0) + load_full_data_to_ram: bool = True + max_ram_gb: float = Field(default=16, gt=0) + episode_cache_size: int = Field(default=8, gt=0) + + +class ExecutionSettings(Settings): + device: str = "cpu" + distribution: DistributionSettings = Field(default_factory=DistributionSettings) + precision: PrecisionSettings = Field(default_factory=PrecisionSettings) + compilation: CompilationSettings = Field(default_factory=CompilationSettings) + data_loading: DataLoadingSettings = Field(default_factory=DataLoadingSettings) + max_concat_tensors: int = Field(default=12, gt=0) + + +class TrainingReproducibilitySettings(Settings): + seed: StrictInt | Literal["random"] = 1010 + enforce_determinism: bool = False + + +class ReproducibilitySettings(TrainingReproducibilitySettings): + seed: RunSeed = 1010 + + +class RecoverySettings(Settings): + resume: ResumeConfig = Field(default_factory=ResumeConfig) + latest_checkpoint_interval_minutes: float | None = Field(default=None, gt=0) + + +class LoggingSettings(Settings): + log_interval_batches: int = Field(default=10, gt=0) + log_interval_iterations: int = Field(default=10, gt=0) + + +class OutputSettings(Settings): + model_path: str = "" + report_path: str = "" + phase_directory: str = "" + + +class ExportSettings(Settings): + generative_model: bool = True + embedding_model: bool = False + embedding_layer_names: list[str] = Field( + default_factory=lambda: ["backbone.final_norm"], min_length=1 + ) + onnx: bool = False + pt: bool = True + with_dropout: bool = False + batch_size: int = Field(default=1, gt=0) + + +class RunSettings(Settings): + project_root: str = "." + model_name: str + execution: ExecutionSettings = Field(default_factory=ExecutionSettings) + reproducibility: TrainingReproducibilitySettings = Field( + default_factory=TrainingReproducibilitySettings + ) + recovery: RecoverySettings = Field(default_factory=RecoverySettings) + logging: LoggingSettings = Field(default_factory=LoggingSettings) + outputs: OutputSettings = Field(default_factory=OutputSettings) + export: ExportSettings = Field(default_factory=ExportSettings) + + @field_validator("model_name") + @classmethod + def validate_model_name(cls, value): + if "embedding" in value: + raise ValueError("model_name cannot contain 'embedding'") + return value + + +class ModelExecutionSettings(Settings): + context_length: int | None = Field(default=None, gt=0) + target_offset: int = Field(default=1, ge=0) + training_objective: str | None = None + forward_attention: Literal["causal", "bidirectional"] | None = None + + +class InputContract(Settings): + metadata_config_path: str | None = None + normalization: FrozenNormalizationConfig | None = None + + +class SplitSettings(Settings): + validation_fraction: float = Field(default=0.2, gt=0, lt=1) diff --git a/src/sequifier/config/parameter_selection.py b/src/sequifier/config/parameter_selection.py new file mode 100644 index 000000000..8aa8b42e1 --- /dev/null +++ b/src/sequifier/config/parameter_selection.py @@ -0,0 +1,225 @@ +"""One catalog selector language for trainability and optimizer routing.""" + +from fnmatch import fnmatchcase +from typing import Literal + +from pydantic import ConfigDict, Field, model_validator + +from sequifier.config.naming import ConfigModel + + +class ParameterSelector(ConfigModel): + model_config = ConfigDict(extra="forbid", frozen=True) + semantic_groups: list[str] | None = None + exclude_modules: list[str] = Field(default_factory=list) + exclude_semantic_groups: list[str] = Field(default_factory=list) + component: Literal["ingestion", "backbone", "decoder"] | None = None + parameter_kind: Literal["weight", "bias", "other"] | None = None + ndim: int | None = Field(default=None, ge=0) + interface: str | None = None + branch_path: list[str] | None = None + module: str | None = None + + @model_validator(mode="after") + def conditions(self): + if all(value in (None, []) for _, value in self): + raise ValueError("a parameter selector requires at least one condition") + for key, value in ( + ("semantic_groups", self.semantic_groups), + ("branch_path", self.branch_path), + ("exclude_modules", self.exclude_modules), + ("exclude_semantic_groups", self.exclude_semantic_groups), + ): + if value is not None and ( + (not value and key in {"semantic_groups", "branch_path"}) + or any(not x for x in value) + ): + raise ValueError(f"{key} requires nonempty entries") + for key, value in (("module", self.module), ("interface", self.interface)): + if value is not None and ( + not value or any(not x for x in value.split(".")) + ): + raise ValueError(f"{key} must be nonempty") + return self + + def matches(self, descriptor): + names = list(descriptor.aliases) + for alias in descriptor.aliases: + parts = alias.split(".") + while len(parts) > 1 and (parts[0] == "network" or parts[0].isdigit()): + parts.pop(0) + normalized = ".".join(parts) + if normalized not in names: + names.append(normalized) + + def in_module(name, module): + return name == module or name.startswith(module + ".") + + if self.exclude_modules and descriptor.shared_parameter_id: + excluded = [ + any(in_module(name, path) for path in self.exclude_modules) + for name in descriptor.aliases + ] + if any(excluded) and not all(excluded): + raise ValueError( + f"conflicting selector exclusions for shared parameter {descriptor.parameter_id}" + ) + return ( + ( + self.semantic_groups is None + or any( + fnmatchcase(group, p) + for group in ( + descriptor.semantic_aliases or (descriptor.semantic_group,) + ) + for p in self.semantic_groups + ) + ) + and not any( + fnmatchcase(group, p) + for group in ( + descriptor.semantic_aliases or (descriptor.semantic_group,) + ) + for p in self.exclude_semantic_groups + ) + and not any(in_module(n, m) for n in names for m in self.exclude_modules) + and (self.component is None or descriptor.component == self.component) + and ( + self.parameter_kind is None + or descriptor.parameter_kind == self.parameter_kind + ) + and (self.ndim is None or len(descriptor.shape) == self.ndim) + and ( + self.branch_path is None + or tuple(self.branch_path) + == descriptor.branch_path[: len(self.branch_path)] + ) + and ( + self.interface is None + or any( + in_module(n, f"interfaces.{self.interface}") + or in_module(n, f"network.interfaces.{self.interface}") + for n in names + ) + ) + and (self.module is None or any(in_module(n, self.module) for n in names)) + ) + + def validate_semantic_patterns(self, descriptors): + """Require every literal/glob to match within this selector's scope. + + Validate inclusions before subtraction, so overlapping or repeated patterns + remain valid. Exclusions must match the positively selected scope. Never + normalize, autocorrect, or silently discard an unmatched pattern. + """ + structural = self.model_copy( + update={"semantic_groups": None, "exclude_semantic_groups": []} + ) + scoped = [ + descriptor for descriptor in descriptors if structural.matches(descriptor) + ] + positive = self.model_copy(update={"exclude_semantic_groups": []}) + included = [descriptor for descriptor in scoped if positive.matches(descriptor)] + for field, patterns, candidates in ( + ("semantic_groups", self.semantic_groups, scoped), + ("exclude_semantic_groups", self.exclude_semantic_groups, included), + ): + for pattern in patterns or []: + if not any( + fnmatchcase(group, pattern) + for descriptor in candidates + for group in ( + descriptor.semantic_aliases or (descriptor.semantic_group,) + ) + ): + raise ValueError( + f"{field} pattern {pattern!r} matched no parameters " + "in the selector's eligible scope" + ) + + +class RoleTrainability(ConfigModel): + model_config = ConfigDict(extra="forbid") + freeze: list[ParameterSelector] | None = None + freeze_except: list[ParameterSelector] | None = None + + @model_validator(mode="after") + def exclusive(self): + if self.freeze is not None and self.freeze_except is not None: + raise ValueError("freeze and freeze_except are mutually exclusive") + return self + + def resolve(self, module, *, eligible=None): + from sequifier.model.parameter_catalog import ParameterCatalog + + catalog = ParameterCatalog(module) + candidates = { + d.parameter_id + for d in catalog.descriptors() + if eligible is None or id(catalog.parameter(d.parameter_id)) in eligible + } + known_modules = set() + for name, _ in module.named_modules(remove_duplicate=False): + known_modules.add(name) + parts = name.split(".") + while len(parts) > 1 and (parts[0] == "network" or parts[0].isdigit()): + parts.pop(0) + known_modules.add(".".join(parts)) + selected = set() + for selector in ( + self.freeze if self.freeze is not None else self.freeze_except or [] + ): + paths = selector.exclude_modules + ( + [selector.module] if selector.module else [] + ) + if unknown := set(paths) - known_modules: + raise ValueError(f"unknown selector modules: {sorted(unknown)}") + eligible_descriptors = [ + descriptor + for descriptor in catalog.descriptors() + if descriptor.parameter_id in candidates + ] + selector.validate_semantic_patterns(eligible_descriptors) + matched = { + descriptor.parameter_id + for descriptor in eligible_descriptors + if selector.matches(descriptor) + } + if not matched: + raise ValueError( + f"trainability selector matched no parameters: {selector.model_dump(exclude_none=True)}" + ) + selected.update(matched) + if self.freeze_except is not None: + selected = candidates - selected + return frozenset(id(catalog.parameter(key)) for key in selected) + + +class RolePolicies(ConfigModel): + model_config = ConfigDict(extra="forbid") + roles: dict[str, RoleTrainability] = Field(default_factory=dict) + + +class TrainabilityConfig(RolePolicies): + by_dataset: dict[str, RolePolicies] = Field(default_factory=dict) + + def policy(self, role, dataset=None): + override = self.by_dataset.get(dataset) if dataset is not None else None + return ( + override.roles.get(role) + if override and role in override.roles + else self.roles.get(role) + ) or RoleTrainability() + + def validate_roles(self, roles, *, datasets=None): + for policy in [self, *self.by_dataset.values()]: + if unknown := policy.roles.keys() - set(roles): + raise ValueError(f"unsupported trainability roles: {sorted(unknown)}") + if self.by_dataset and datasets is None: + raise ValueError( + "source-dependent trainability requires supervised updates" + ) + if datasets is not None and (unknown := self.by_dataset.keys() - set(datasets)): + raise ValueError( + f"trainability overrides require training datasets: {sorted(unknown)}" + ) diff --git a/src/sequifier/config/phase_controls.py b/src/sequifier/config/phase_controls.py new file mode 100644 index 000000000..c249bb055 --- /dev/null +++ b/src/sequifier/config/phase_controls.py @@ -0,0 +1,139 @@ +"""Shared phase lifecycle declarations; algorithms only constrain capabilities.""" + +from typing import TYPE_CHECKING, Any, Literal + +from pydantic import ConfigDict, Field, model_validator + +from sequifier.config.naming import ConfigModel + +if TYPE_CHECKING: + from sequifier.config.policy_plan import OnlineEnvironmentConfig, ReferenceConfig + + +class Cadence(ConfigModel): + model_config = ConfigDict(extra="forbid") + epochs: int | None = Field(default=None, gt=0) + iterations: int | None = Field(default=None, gt=0) + + @model_validator(mode="after") + def one_clock(self): + if (self.epochs is None) == (self.iterations is None): + raise ValueError("cadence requires exactly one of epochs or iterations") + return self + + +class EvaluationTriggers(ConfigModel): + model_config = ConfigDict(extra="forbid") + cadence: Cadence | None = None + before_phase: bool = False + on_snapshot: bool = False + + +class PhaseCheckpoints(ConfigModel): + model_config = ConfigDict(extra="forbid", allow_inf_nan=False) + cadence: Cadence | None = None + snapshot_interval_minutes: float | None = Field(default=None, gt=0) + snapshot_interval_batches: int | None = Field(default=None, gt=0) + + +class ModelSelection(ConfigModel): + model_config = ConfigDict(extra="forbid") + handoff: Literal["best", "last"] = "last" + scope: Literal["compatible_phases", "phase"] = "compatible_phases" + + +class StoppingConfig(ConfigModel): + model_config = ConfigDict(extra="forbid", allow_inf_nan=False) + patience: int = Field(gt=0) + min_delta: float = Field(default=0.0, ge=0) + on_trigger: Literal["end_phase", "end_plan"] = "end_phase" + + +class CurriculumConfig(ConfigModel): + model_config = ConfigDict(extra="forbid") + column: str = Field(min_length=1) + + +class PhaseControls(ConfigModel): + if TYPE_CHECKING: + algorithm: Any + evaluation: Any + + model_config = ConfigDict(extra="forbid") + checkpoints: PhaseCheckpoints = Field(default_factory=PhaseCheckpoints) + model_selection: ModelSelection = Field(default_factory=ModelSelection) + stopping: StoppingConfig | None = None + + @property + def reference_config(self) -> "ReferenceConfig | None": + return None + + @property + def rollout_archive_directory(self) -> str | None: + return None + + @property + def training_environment(self) -> "OnlineEnvironmentConfig | None": + return None + + @property + def evaluation_environment(self) -> "OnlineEnvironmentConfig | None": + return None + + @property + def uses_actor_history(self) -> bool: + return False + + @property + def handoff(self): + return self.model_selection.handoff + + @property + def evaluation_interval_epochs(self): + cadence = self.evaluation.cadence if self.evaluation is not None else None + return cadence.epochs if cadence is not None else 1 + + @property + def checkpoint_interval_epochs(self): + return self.checkpoints.cadence.epochs if self.checkpoints.cadence else 1 + + @property + def evaluation_interval_iterations(self): + cadence = self.evaluation.cadence if self.evaluation is not None else None + return cadence.iterations if cadence is not None else 1 + + @property + def checkpoint_interval_iterations(self): + cadence = self.checkpoints.cadence + return cadence.iterations if cadence else 1 + + @model_validator(mode="after") + def validate_lifecycle(self): + if self.algorithm != "supervised" and self.model_selection.scope == "phase": + raise ValueError("model_selection.scope is a supervised phase setting") + clock = "epochs" if self.algorithm == "supervised" else "iterations" + for label, cadence in ( + ("checkpoints", self.checkpoints.cadence), + ("evaluation", self.evaluation.cadence if self.evaluation else None), + ): + if ( + cadence is not None + and (cadence.epochs if clock == "epochs" else cadence.iterations) + is None + ): + raise ValueError(f"{self.algorithm} {label}.cadence requires {clock}") + if self.evaluation is None and (self.stopping or self.handoff == "best"): + raise ValueError("stopping and best handoff require evaluation.monitor") + if ( + self.stopping + and self.evaluation is not None + and self.evaluation.monitor is None + and len(self.evaluation.sources or []) != 1 + ): + raise ValueError("stopping requires evaluation.monitor") + if self.algorithm != "supervised": + if self.checkpoints.snapshot_interval_batches is not None: + raise ValueError( + "policy snapshots use committed iterations, not batches" + ) + return self diff --git a/src/sequifier/config/policy_config.py b/src/sequifier/config/policy_config.py new file mode 100644 index 000000000..49b9d0ccd --- /dev/null +++ b/src/sequifier/config/policy_config.py @@ -0,0 +1,133 @@ +"""Policy alignment and standard model-component configuration.""" + +import math +from typing import Annotated, Any, Literal + +from pydantic import ConfigDict, Field, model_validator + +from sequifier.config.naming import ConfigModel +from sequifier.io.values import DomainValue + +NonemptyStr = Annotated[str, Field(min_length=1)] + + +class PolicyConfigBase(ConfigModel): + model_config = ConfigDict(extra="forbid", frozen=True, allow_inf_nan=False) + + +class ReturnDefinition(PolicyConfigBase): + gamma: float = Field(default=1.0, gt=0, le=1, allow_inf_nan=False) + scale: float = Field(default=1.0, gt=0, allow_inf_nan=False) + finite_horizon: int | None = Field(default=None, gt=0) + + +class FixedActionScale(PolicyConfigBase): + fixed: float = Field(gt=0) + + +class ActionFieldConfig(PolicyConfigBase): + """Task-specific action support and selection of model or fixed scale.""" + + type: Literal["categorical", "real"] + values: list[DomainValue] = Field(default_factory=list) + scale: Literal["model"] | FixedActionScale | None = None + lower: float | None = None + upper: float | None = None + active_when: dict[str, list[DomainValue]] = Field(default_factory=dict) + inactive_value: DomainValue = 0.0 + atoms: dict[float, float] = Field(default_factory=dict) + + @model_validator(mode="after") + def validate_distribution(self): + if self.type == "real": + if self.values: + raise ValueError("real actions cannot declare categorical values") + if self.scale is None: + raise ValueError( + "real actions require scale: model or scale: {fixed: ...}" + ) + elif ( + self.scale is not None + or self.lower is not None + or self.upper is not None + or self.atoms + ): + raise ValueError( + "categorical actions cannot configure real scale, bounds, or atoms" + ) + if len(set(self.values)) != len(self.values): + raise ValueError("action labels must be unique") + if ( + self.lower is not None + and self.upper is not None + and self.lower >= self.upper + ): + raise ValueError("action lower bound must be below upper bound") + if ( + any( + not math.isfinite(value) + or not math.isfinite(probability) + or probability <= 0 + for value, probability in self.atoms.items() + ) + or sum(self.atoms.values()) >= 1 + ): + raise ValueError( + "atom probabilities must be positive and sum to less than one" + ) + if any( + (self.lower is not None and value < self.lower) + or (self.upper is not None and value > self.upper) + for value in self.atoms + ): + raise ValueError("point masses must lie within the declared action domain") + return self + + +class PolicyBinding(PolicyConfigBase): + """A task attached to named targets of an ordinary model interface.""" + + actions: dict[str, ActionFieldConfig] = Field(min_length=1) + decision_position: int = Field(default=-1, le=-1) + previous_action_inputs: dict[str, str] = Field(default_factory=dict) + initial_inputs: dict[str, Any] = Field(default_factory=dict) + return_conditioned: bool = False + initial_desired_return: float | None = Field(default=None, allow_inf_nan=False) + returns: ReturnDefinition = Field(default_factory=ReturnDefinition) + units: dict[str, str] = Field(default_factory=dict) + + @model_validator(mode="after") + def validate_input_bindings(self): + if any(not name for name in self.actions): + raise ValueError("actions require nonempty target names") + if set(self.previous_action_inputs.values()) - set(self.actions): + raise ValueError("previous_action_inputs must reference declared actions") + if set(self.initial_inputs) - set(self.previous_action_inputs): + raise ValueError("initial_inputs must name previous_action_inputs") + return self + + +class ResolvedPolicyBinding(PolicyBinding): + """Fitted task binding persisted with a model's execution contract.""" + + context_length: int = Field(gt=0) + interface_name: NonemptyStr + observation_columns: list[str] = Field(default_factory=list) + static_columns: list[str] = Field(default_factory=list) + categorical_columns: list[str] = Field(default_factory=list) + observation_n_classes: dict[str, int] = Field(default_factory=dict) + static_n_classes: dict[str, int] = Field(default_factory=dict) + id_maps: dict[str, dict[str | int, int]] = Field(default_factory=dict) + categorical_encoding: Literal["global"] = "global" + action_order: list[str] = Field(default_factory=list) + input_schema: dict[str, Any] + input_statistics: dict[str, dict[str, float]] = Field(default_factory=dict) + input_encoding: Literal["raw", "model"] = "raw" + + @property + def action_columns(self): + return self.action_order or list(self.actions) + + @property + def categorical_n_classes(self): + return {**self.observation_n_classes, **self.static_n_classes} diff --git a/src/sequifier/config/policy_infer_config.py b/src/sequifier/config/policy_infer_config.py new file mode 100644 index 000000000..f8045f066 --- /dev/null +++ b/src/sequifier/config/policy_infer_config.py @@ -0,0 +1,172 @@ +"""Inference options for policy interfaces in ordinary portable model artifacts.""" + +from typing import Literal + +from pydantic import Field, PrivateAttr, model_validator + +from sequifier.artifacts.model_artifact import ModelArtifact +from sequifier.config.inference_common import ( + InferenceExecutionConfig, + InferenceRouteConfig, + SelectedInferenceModel, + inference_model_config, +) +from sequifier.config.policy_config import PolicyConfigBase +from sequifier.config.policy_plan import OnlineEnvironmentConfig +from sequifier.helpers import normalize_path + + +class PolicyRolloutConfig(PolicyConfigBase): + environment: OnlineEnvironmentConfig + max_steps: int = Field(gt=0) + trajectory_path: str = Field(min_length=1) + seed_row_index: int = Field(ge=0) + max_episode_steps: int | None = Field(default=None, gt=0) + + +class PolicyInferenceConfig(InferenceExecutionConfig): + model_paths: list[str] = Field(min_length=1, max_length=1) + task: Literal["policy"] + + data_path: str + output_probabilities: bool = False + action_selection: Literal["deterministic", "stochastic"] = "deterministic" + desired_return: float | None = Field(default=None, allow_inf_nan=False) + desired_return_column: str = "desired_return" + previous_action_columns: dict[str, str] = Field(default_factory=dict) + sequence_id_column: str = "sequenceId" + item_position_column: str = "itemPosition" + rollout: PolicyRolloutConfig | None = None + # Optional assertions against the portable artifact contract. + input_columns: list[str] | None = None + static_columns: list[str] | None = None + target_columns: list[str] | None = None + column_data_types: dict[str, str] | None = None + target_column_types: dict[str, str] | None = None + training_objective: str | None = None + context_length: int | None = Field(default=None, gt=0) + target_offset: int | None = Field(default=None, ge=0) + prediction_length: int | None = Field(default=None, gt=0) + decoder_support_length: int | None = Field(default=None, gt=0) + + @model_validator(mode="after") + def validate_execution(self): + if self.read_format not in {"csv", "parquet"} or self.write_format not in { + "csv", + "parquet", + }: + raise ValueError("policy inference formats must be csv or parquet") + if self.rollout is not None and self.output_probabilities: + raise ValueError( + "rollout writes a trajectory including action log probabilities" + ) + return self + + +class ResolvedPolicyInferenceConfig(PolicyInferenceConfig): + """Validated policy request with selected artifact/interface and assertions.""" + + _artifacts: dict[str, ModelArtifact] = PrivateAttr(default_factory=dict) + + +def _policy_inference_route( + route: InferenceRouteConfig, artifacts: dict[str, ModelArtifact] +) -> SelectedInferenceModel | None: + """Inspect artifact interfaces before applying either inference schema.""" + if route.task != "policy": + return None + paths = route.model_paths + bindings = [] + selection = None + for path in paths: + model_config = inference_model_config(path, route.project_root, artifacts) + if model_config is None: + bindings.append(False) + continue + selection = SelectedInferenceModel.select( + path, + model_config, + interface_name=route.model_interface, + task=route.task, + ) + bindings.append(selection.policy_bound) + if len(paths) != 1 or not all(bindings): + raise ValueError( + "task: policy requires one artifact with a selected policy-bound interface" + ) + if not paths[0].lower().endswith(".pt"): + raise ValueError( + "task: policy requires a portable .pt artifact; " + "use task: forecast for forecast inference" + ) + assert selection is not None + return selection + + +def policy_inference_config( + values: dict, artifacts: dict[str, ModelArtifact] | None = None +) -> ResolvedPolicyInferenceConfig | None: + """Dispatch from the persisted interface contract, without training YAML.""" + route = InferenceRouteConfig.model_validate(values) + from sequifier.config.infer_config import ( + _assert_metadata_matches_source, + _execution_metadata, + _execution_values, + ) + from sequifier.config.metadata import load_dataset_metadata + + artifacts = {} if artifacts is None else artifacts + selection = _policy_inference_route(route, artifacts) + if selection is None: + return None + model_path, selected = selection.path, selection.interface_name + request = PolicyInferenceConfig.model_validate(values) + execution = selection.resolve(request.device) + interface = execution.interface(selected) + policy = interface.policy + if policy is None: + raise ValueError("selected interface has no policy") + if ( + request.rollout is not None + and request.rollout.environment.units != policy.units + ): + raise ValueError("rollout environment units differ from policy artifact") + reserved = {"sequenceId", "subsequenceId", "windowStartOffset", "itemPosition"} + if reserved & set(policy.actions) or ( + request.output_probabilities and "action_log_probability" in policy.actions + ): + raise ValueError( + "policy action names overlap prediction identity/probability columns" + ) + if request.output_probabilities and any( + len({str(value) for value in field.values}) != len(field.values) + for field in policy.actions.values() + if field.type == "categorical" + ): + raise ValueError("action labels must have unique probability column names") + contract = _execution_values(execution.options, interface) + requested_fields = dict(request) + for key, actual in contract.items(): + if ( + key in request.model_fields_set + and requested_fields[key] is not None + and requested_fields[key] != actual + ): + raise ValueError(f"{key} differs from policy artifact") + if request.metadata_config_path: + explicit = load_dataset_metadata( + normalize_path(request.metadata_config_path, request.project_root) + ) + _assert_metadata_matches_source( + explicit, + _execution_metadata(interface, execution.interface_storage.get(selected)), + "policy metadata", + layout_is_authoritative=True, + ) + authored = request.model_dump() + authored["run"] = request.run.model_dump(mode="python", exclude_unset=True) + authored["model_paths"] = [model_path] + authored["model_interface"] = selected + config = ResolvedPolicyInferenceConfig.model_validate(authored) + config._artifacts = artifacts + return config diff --git a/src/sequifier/config/policy_inputs.py b/src/sequifier/config/policy_inputs.py new file mode 100644 index 000000000..38b8b2bb2 --- /dev/null +++ b/src/sequifier/config/policy_inputs.py @@ -0,0 +1,112 @@ +"""Internal policy feature layout shared by fitting and model attachment.""" + +from dataclasses import dataclass, replace + +from sequifier.config.policy_config import PolicyBinding, ResolvedPolicyBinding + + +@dataclass(frozen=True, kw_only=True) +class PolicyInputLayout: + observation_columns: list[str] + static_columns: list[str] + categorical_columns: list[str] + observation_n_classes: dict[str, int] + static_n_classes: dict[str, int] + id_maps: dict[str, dict[str | int, int]] + + @classmethod + def select( + cls, + temporal, + static, + binding, + categorical, + n_classes, + id_maps, + *, + features_only=False, + ): + observations = [ + name + for name in temporal + if name not in binding.previous_action_inputs + and (name != "__policy_return" or not binding.return_conditioned) + ] + vocabulary_columns = observations if features_only else temporal + columns = set(vocabulary_columns) | set(static) + return cls( + observation_columns=observations, + static_columns=list(static), + categorical_columns=list(categorical), + observation_n_classes={ + k: n for k, n in n_classes.items() if k in vocabulary_columns + }, + static_n_classes={k: n for k, n in n_classes.items() if k in static}, + id_maps={k: dict(v) for k, v in id_maps.items() if k in columns}, + ) + + @classmethod + def for_fitting(cls, temporal, static, binding, metadata): + layout = cls.select( + temporal, + static, + binding, + [], + metadata.n_classes, + metadata.id_maps, + features_only=True, + ) + return replace( + layout, + categorical_columns=[ + name + for name in layout.observation_columns + layout.static_columns + if "int" in metadata.column_data_types[name].lower() + ], + ) + + @property + def categorical_n_classes(self): + return {**self.observation_n_classes, **self.static_n_classes} + + def binding_values(self): + return { + "observation_columns": self.observation_columns, + "static_columns": self.static_columns, + "categorical_columns": self.categorical_columns, + "observation_n_classes": self.observation_n_classes, + "static_n_classes": self.static_n_classes, + "id_maps": self.id_maps, + } + + +@dataclass(frozen=True, kw_only=True) +class PolicyFittingSpec(PolicyInputLayout): + """Raw episode features before an executable model schema exists.""" + + binding: PolicyBinding + context_length: int + interface_name: str + + @property + def actions(self): + return self.binding.actions + + @property + def action_columns(self): + return list(self.actions) + + @property + def return_conditioned(self): + return self.binding.return_conditioned + + @property + def returns(self): + return self.binding.returns + + @property + def input_encoding(self): + return "raw" + + +PolicyEpisodeSpec = ResolvedPolicyBinding | PolicyFittingSpec diff --git a/src/sequifier/config/policy_optimization.py b/src/sequifier/config/policy_optimization.py new file mode 100644 index 000000000..6df2b84a4 --- /dev/null +++ b/src/sequifier/config/policy_optimization.py @@ -0,0 +1,182 @@ +"""Shared role optimization specs with explicit algorithm scheduling constraints.""" + +import inspect +from typing import Literal + +from pydantic import ConfigDict, Field, model_validator + +from sequifier.config.components import ComponentSpec +from sequifier.config.naming import ConfigModel +from sequifier.config.optimizer_config import ( + OptimizerPlan, + resolve_plan_scheduler_arguments, +) +from sequifier.optimizers.optimizers import get_optimizer_class, get_scheduler_class + + +def default_optimizer(): + return ComponentSpec(name="Adam") + + +def default_scheduler(): + return ComponentSpec(name="ConstantLR", arguments={"factor": 1.0, "total_iters": 1}) + + +class RoleOptimizationConfig(ConfigModel): + """One authored optimizer/scheduler configuration per trainable role.""" + + model_config = ConfigDict(extra="forbid", allow_inf_nan=False) + optimizer: OptimizerPlan | ComponentSpec = Field( + default_factory=default_optimizer, union_mode="left_to_right" + ) + scheduler: ComponentSpec = Field(default_factory=default_scheduler) + learning_rate: float | None = Field(default=None, gt=0) + gradient_clip_norm: float | None = Field(default=1.0, gt=0) + + @model_validator(mode="after") + def validate_components(self): + if isinstance(self.optimizer, OptimizerPlan): + if self.learning_rate is not None: + raise ValueError("optimizer plans set learning_rate in each group") + elif self.learning_rate is None: + self.learning_rate = 3e-4 + if self.scheduler.name == "OneCycleLR" and { + "epochs", + "steps_per_epoch", + }.intersection(self.scheduler.arguments): + raise ValueError( + "OneCycleLR uses the phase budget; configure total_steps instead of scheduler epochs/steps_per_epoch" + ) + specs = ( + [group.optimizer for group in self.optimizer.groups] + if isinstance(self.optimizer, OptimizerPlan) + else [self.optimizer] + ) + for spec in specs: + get_optimizer_class(spec.name) + if {"lr", "params"}.intersection(spec.arguments): + raise ValueError( + "optimizer ownership and learning rates must be configured explicitly" + ) + arguments = ( + resolve_plan_scheduler_arguments(self.optimizer, self.scheduler) + if isinstance(self.optimizer, OptimizerPlan) + else self.scheduler.arguments + ) + try: + inspect.signature(get_scheduler_class(self.scheduler.name)).bind( + object(), **arguments + ) + for spec in specs: + inspect.signature(get_optimizer_class(spec.name)).bind( + object(), lr=1.0, **spec.arguments + ) + except TypeError as error: + raise ValueError(str(error)) from error + return self + + def continuation_settings(self): + """Compare optimizer settings independently of optional budget assertions.""" + values = self.model_dump(mode="python") + if self.scheduler.name == "OneCycleLR": + values["scheduler"]["arguments"].pop("total_steps", None) + return values + + def validate_budget(self, step_budget): + if ( + step_budget is not None + and self.scheduler.name == "OneCycleLR" + and self.scheduler.arguments.get("total_steps") is not None + and self.scheduler.arguments["total_steps"] != step_budget + ): + raise ValueError( + f"OneCycleLR.total_steps must equal the maximum authored optimizer-step budget ({step_budget})" + ) + + def resolve_budget(self, step_budget): + if self.scheduler.name != "OneCycleLR": + return self + if step_budget is None or step_budget <= 0: + raise ValueError( + "OneCycleLR requires a positive resolved optimizer-step budget" + ) + self.validate_budget(step_budget) + return self.model_copy( + update={ + "scheduler": self.scheduler.model_copy( + update={ + "arguments": { + **self.scheduler.arguments, + "total_steps": step_budget, + } + } + ) + } + ) + + def build(self, module, *, step_budget=None): + from sequifier.model.parameter_catalog import ParameterCatalog + from sequifier.training.optimization import create_optimizer + + optimizer = create_optimizer( + self.optimizer, + [p for p in module.parameters() if p.requires_grad], + self.learning_rate, + parameter_catalog=ParameterCatalog(module) + if isinstance(self.optimizer, OptimizerPlan) + else None, + ) + scheduler = self.scheduler + if scheduler.name == "OneCycleLR": + scheduler = self.resolve_budget(step_budget).scheduler + arguments = ( + resolve_plan_scheduler_arguments(self.optimizer, scheduler) + if isinstance(self.optimizer, OptimizerPlan) + else scheduler.arguments + ) + return optimizer, get_scheduler_class(self.scheduler.name)( + optimizer, **arguments + ) + + +class OptimizationConfig(RoleOptimizationConfig): + """Policy roles advance their schedulers once per applied optimizer update.""" + + scheduler_step_on: Literal["optimizer_step"] = "optimizer_step" + + @model_validator(mode="after") + def validate_policy_scheduler(self): + if ( + self.scheduler.name + in {"ReduceLROnPlateau", "SequentialLR", "ChainedScheduler"} + or "optimizer" in self.scheduler.arguments + ): + raise ValueError( + "policy schedulers must advance independently on optimizer steps" + ) + return self + + +class SupervisedOptimizationConfig(RoleOptimizationConfig): + """Epoch scheduling includes metric-driven ReduceLROnPlateau.""" + + # Supervised training historically leaves gradients unclipped unless requested. + gradient_clip_norm: float | None = Field(default=None, gt=0) + + scheduler_step_on: Literal["epoch", "optimizer_step"] = "epoch" + scheduler: ComponentSpec = Field( + default_factory=lambda: ComponentSpec( + name="StepLR", arguments={"step_size": 1, "gamma": 0.99} + ) + ) + + @model_validator(mode="after") + def validate_supervised_scheduler(self): + if ( + self.scheduler.name == "ReduceLROnPlateau" + and self.scheduler_step_on != "epoch" + ): + raise ValueError("ReduceLROnPlateau requires scheduler_step_on: epoch") + if self.scheduler.name in {"SequentialLR", "ChainedScheduler"}: + raise ValueError("nested schedulers are not supported") + return self diff --git a/src/sequifier/config/policy_plan.py b/src/sequifier/config/policy_plan.py new file mode 100644 index 000000000..421c7f49d --- /dev/null +++ b/src/sequifier/config/policy_plan.py @@ -0,0 +1,388 @@ +"""Built-in policy phase schemas; budgets and objectives are algorithm-specific.""" + +from typing import Annotated, ClassVar, Literal + +from pydantic import ( + ConfigDict, + Field, + field_validator, + model_serializer, + model_validator, +) + +from sequifier.config.identifiers import identifier +from sequifier.config.naming import ConfigModel +from sequifier.config.normalization_config import ( # noqa: F401 + FrozenNormalizationConfig, +) +from sequifier.config.parameter_selection import TrainabilityConfig +from sequifier.config.phase_controls import ( + Cadence, + EvaluationTriggers, + PhaseCheckpoints, + PhaseControls, +) +from sequifier.config.policy_optimization import OptimizationConfig +from sequifier.config.sources import TrainingSourceSpecModel + + +class OfflineEvaluationMonitor(ConfigModel): + model_config = ConfigDict(extra="forbid") + metric: Literal["validation_action_nll"] = "validation_action_nll" + direction: Literal["min"] = "min" + + +class DPOEvaluationMonitor(ConfigModel): + model_config = ConfigDict(extra="forbid") + metric: Literal["validation_dpo_loss"] = "validation_dpo_loss" + direction: Literal["min"] = "min" + + +class PolicyEvaluationTriggers(EvaluationTriggers): + batch_size: int = Field(default=1, gt=0) + + +class OfflineEvaluationConfig(PolicyEvaluationTriggers): + model_config = ConfigDict(extra="forbid") + sources: list[TrainingSourceSpecModel] | None = Field(default=None, min_length=1) + monitor: OfflineEvaluationMonitor = Field(default_factory=OfflineEvaluationMonitor) + + +class DPOEvaluationConfig(PolicyEvaluationTriggers): + @model_validator(mode="after") + def serial_evaluation(self): + if self.batch_size != 1: + raise ValueError( + "DPO evaluation scores one preference pair at a time; evaluation.batch_size must be 1" + ) + return self + + model_config = ConfigDict(extra="forbid") + sources: list[TrainingSourceSpecModel] | None = Field(default=None, min_length=1) + monitor: DPOEvaluationMonitor = Field(default_factory=DPOEvaluationMonitor) + + +class PhaseConfig(PhaseControls): + model_config = ConfigDict(extra="forbid", allow_inf_nan=False) + name: str + optimizer_state: Literal["reset", "continue"] = "reset" + model_interface: str | None = None + trainability: TrainabilityConfig = Field(default_factory=TrainabilityConfig) + optimization_roles: ClassVar[tuple[str, ...]] = ("actor",) + optimization: dict[str, OptimizationConfig] = Field(default_factory=dict) + + def optimizer_step_budget(self, training_size=None): + raise NotImplementedError("built-in phase must declare its optimizer budget") + + def iteration_budget(self, optimizer_step_budget: int | None) -> int: + return optimizer_step_budget or 1 + + def optimization_configs(self): + unknown = self.optimization.keys() - set(self.optimization_roles) + if unknown: + raise ValueError(f"unsupported optimization roles: {sorted(unknown)}") + return { + role: self.optimization.get(role, OptimizationConfig()) + for role in self.optimization_roles + } + + @model_validator(mode="after") + def validate_optimization(self): + if self.uses_actor_history and self.optimizer_state == "continue": + raise ValueError("history_initialization requires optimizer_state: reset") + self.trainability.validate_roles(self.optimization_roles) + for config in self.optimization_configs().values(): + config.validate_budget(self.optimizer_step_budget()) + return self + + @model_serializer(mode="wrap") + def serialize_history_initialization(self, handler): + values = handler(self) + if values.get("history_initialization") == "random": + values.pop("history_initialization") + return values + + @field_validator("name") + @classmethod + def phase_name(cls, value): + return identifier(value, "Policy phase name") + + +class BatchedPhaseConfig(PhaseConfig): + batch_size: int = Field(default=32, gt=0) + accumulation_batches: int = Field(default=1, gt=0) + + +class OfflinePhaseConfig(PhaseConfig): + sources: list[TrainingSourceSpecModel] = Field(min_length=1) + evaluation: OfflineEvaluationConfig = Field(default_factory=OfflineEvaluationConfig) + + +class PolicyImitationPhaseConfig(BatchedPhaseConfig, OfflinePhaseConfig): + algorithm: Literal["policy_imitation"] + epochs: int = Field(gt=0) + + def optimizer_step_budget(self, training_size=None): + if training_size is None: + return None + size = self.batch_size * self.accumulation_batches + return self.epochs * ((training_size + size - 1) // size) + + +class IQLPhaseConfig(OfflinePhaseConfig): + optimization_roles: ClassVar[tuple[str, ...]] = ("actor", "value", "critic") + algorithm: Literal["iql"] + history_initialization: Literal["random", "phase_start_actor"] = "random" + evaluation: OfflineEvaluationConfig = Field( + default_factory=lambda: OfflineEvaluationConfig(cadence=Cadence(iterations=100)) + ) + checkpoints: PhaseCheckpoints = Field( + default_factory=lambda: PhaseCheckpoints(cadence=Cadence(iterations=100)) + ) + name: str = Field(default="offline_iql", min_length=1) + iterations: int = Field(gt=0) + batch_size: int = Field(default=64, gt=0) + gamma: float = Field(default=0.99, gt=0, le=1, allow_inf_nan=False) + expectile: float = Field(default=0.7, gt=0, lt=1, allow_inf_nan=False) + inverse_temperature: float = Field(default=3.0, gt=0, allow_inf_nan=False) + max_advantage_weight: float = Field(default=100.0, ge=1, allow_inf_nan=False) + target_update_rate: float = Field(default=0.005, gt=0, le=1, allow_inf_nan=False) + + @property + def uses_actor_history(self) -> bool: + return self.history_initialization == "phase_start_actor" + + def iteration_budget(self, optimizer_step_budget: int | None) -> int: + return self.iterations + + def optimizer_step_budget(self, training_size=None): + return self.iterations + + +class OnlineEnvironmentConfig(ConfigModel): + model_config = ConfigDict(extra="forbid") + factory: str + config: dict = Field(default_factory=dict) + + # Empty means unitless; adapters must report this exact declaration. + units: dict[str, str] = Field(default_factory=dict) + + @field_validator("factory") + @classmethod + def validate_factory(cls, value): + module, separator, attribute = value.partition(":") + if not separator or not module or not attribute.isidentifier(): + raise ValueError("environment.factory must use module:factory") + return value + + +class EvaluationEpisodesConfig(ConfigModel): + model_config = ConfigDict(extra="forbid") + environment: OnlineEnvironmentConfig + seeds: list[int] = Field(min_length=1) + max_episode_steps: int = Field(gt=0) + timeout_seconds: float = Field(gt=0, allow_inf_nan=False) + action_selection: Literal["deterministic", "stochastic"] = "deterministic" + + @field_validator("seeds") + @classmethod + def validate_seeds(cls, values): + if any(value < 0 or value >= 2**63 for value in values): + raise ValueError("evaluation seeds must be in [0, 2**63)") + return values + + +class OnlineEvaluationMonitor(ConfigModel): + model_config = ConfigDict(extra="forbid") + + metric: Literal[ + "mean_completed_return", "success_rate", "mean_completed_length", "score_mean" + ] = "mean_completed_return" + direction: Literal["min", "max"] = "max" + + +class OnlineEvaluationConfig(EvaluationEpisodesConfig, PolicyEvaluationTriggers): + @model_validator(mode="after") + def serial_evaluation(self): + if self.batch_size != 1: + raise ValueError( + "online evaluation runs serial episodes; evaluation.batch_size must be 1" + ) + return self + + monitor: OnlineEvaluationMonitor = Field(default_factory=OnlineEvaluationMonitor) + + +class OnlinePhaseConfig(PhaseConfig): + environment: OnlineEnvironmentConfig + archive_directory: str | None = None + + @property + def rollout_archive_directory(self) -> str | None: + return self.archive_directory + + @property + def training_environment(self) -> OnlineEnvironmentConfig: + return self.environment + + @property + def evaluation_environment(self) -> OnlineEnvironmentConfig: + return self.evaluation.environment + + +class PPOPhaseConfig(OnlinePhaseConfig): + optimization_roles: ClassVar[tuple[str, ...]] = ("actor", "value") + algorithm: Literal["ppo"] + history_initialization: Literal["random", "phase_start_actor"] = "random" + name: str = Field(default="online_ppo", min_length=1) + evaluation: OnlineEvaluationConfig + seed_policy: Literal["derived_per_reset"] = "derived_per_reset" + total_env_steps: int = Field(gt=0) + rollout_env_steps: int = Field(gt=0) + allow_partial_rollout: bool = True + optimization_epochs: int = Field(default=4, gt=0) + batch_size: int = Field(default=64, gt=0) + accumulation_batches: int = Field(default=1, gt=0) + gamma: float = Field(default=0.99, ge=0, le=1, allow_inf_nan=False) + gae_lambda: float = Field(default=0.95, ge=0, le=1, allow_inf_nan=False) + clip_epsilon: float = Field(default=0.2, gt=0, lt=1, allow_inf_nan=False) + value_clip_delta: float | None = Field(default=None, gt=0, allow_inf_nan=False) + value_coefficient: float = Field(default=0.5, gt=0, allow_inf_nan=False) + entropy_coefficient: float = Field(default=0.01, ge=0, allow_inf_nan=False) + old_policy_target_kl: float | None = Field(default=None, gt=0, allow_inf_nan=False) + normalize_advantages: bool = True + reward_scale: float = Field(default=1.0, gt=0, allow_inf_nan=False) + archive_directory: str | None = Field(default=None, min_length=1) + archive_scores: bool = False + max_buffer_bytes: int = Field(default=536870912, gt=0) + + @property + def uses_actor_history(self) -> bool: + return self.history_initialization == "phase_start_actor" + + def iteration_budget(self, optimizer_step_budget: int | None) -> int: + total, per_iteration = self.total_env_steps, self.rollout_env_steps + return (total + per_iteration - 1) // per_iteration + + def optimizer_step_budget(self, training_size=None): + full, remainder = divmod(self.total_env_steps, self.rollout_env_steps) + size = self.batch_size * self.accumulation_batches + return self.optimization_epochs * ( + full * ((self.rollout_env_steps + size - 1) // size) + + ((remainder + size - 1) // size if remainder else 0) + ) + + @model_validator(mode="after") + def validate_plans(self): + if ( + not self.allow_partial_rollout + and self.total_env_steps % self.rollout_env_steps + ): + raise ValueError( + "rollout_env_steps must divide total_env_steps unless allow_partial_rollout" + ) + return self + + +class ReferenceConfig(ConfigModel): + model_config = ConfigDict(extra="forbid") + source: Literal["phase_start", "artifact"] = "phase_start" + path: str | None = None + + @model_validator(mode="after") + def location(self): + if (self.source == "artifact") != (self.path is not None): + raise ValueError("reference.path is required only for source: artifact") + return self + + +class DPOPhaseConfig(BatchedPhaseConfig): + algorithm: Literal["dpo"] + preferences: TrainingSourceSpecModel + evaluation: DPOEvaluationConfig = Field(default_factory=DPOEvaluationConfig) + iterations: int = Field(gt=0) + beta: float = Field(default=0.1, gt=0) + scoring_chunk_size: int = Field(default=64, gt=0) + reference: ReferenceConfig = Field(default_factory=ReferenceConfig) + + @property + def reference_config(self) -> ReferenceConfig: + return self.reference + + def iteration_budget(self, optimizer_step_budget: int | None) -> int: + return self.iterations + + def optimizer_step_budget(self, training_size=None): + return self.iterations + + +class GRPOEvaluationConfig(OnlineEvaluationConfig): + monitor: OnlineEvaluationMonitor = Field( + default_factory=lambda: OnlineEvaluationMonitor(metric="score_mean") + ) + + +class GRPOPhaseConfig(BatchedPhaseConfig, OnlinePhaseConfig): + scoring_chunk_size: int = Field(default=64, gt=0) + # One preserves the original in-process collector, with no worker startup. + simulator_workers: int = Field(default=1, gt=0) + rollout_inference_batch_size: int = Field(default=32, gt=0) + worker_timeout_seconds: float = Field(default=120.0, gt=0, allow_inf_nan=False) + worker_threads: int = Field(default=1, gt=0) + image_feature_cache_bytes: int = Field(default=0, ge=0) + algorithm: Literal["grpo"] + evaluation: GRPOEvaluationConfig + reference: ReferenceConfig = Field(default_factory=ReferenceConfig) + total_groups: int = Field(gt=0) + groups_per_iteration: int = Field(default=1, gt=0) + group_size: int = Field(default=4, ge=2) + max_steps: int = Field(gt=0) + max_env_steps: int | None = Field(default=None, gt=0) + max_buffer_bytes: int = Field(default=536870912, gt=0) + optimization_epochs: int = Field(default=4, gt=0) + clip_epsilon: float = Field(default=0.2, gt=0, lt=1) + advantage_epsilon: float = Field(default=1e-6, gt=0) + reference_kl_coefficient: float = Field(default=0.04, ge=0) + old_policy_target_kl: float | None = Field(default=None, gt=0) + reward_scale: float = Field(default=1.0, gt=0) + + @model_serializer(mode="wrap") + def serialize_execution_options(self, handler): + values = handler(self) + # Omit default options from the serialized execution plan. + for name in ( + "simulator_workers", + "rollout_inference_batch_size", + "worker_timeout_seconds", + "worker_threads", + "image_feature_cache_bytes", + ): + if values.get(name) == type(self).model_fields[name].default: + values.pop(name, None) + return values + + @property + def reference_config(self) -> ReferenceConfig: + return self.reference + + def iteration_budget(self, optimizer_step_budget: int | None) -> int: + total, per_iteration = self.total_groups, self.groups_per_iteration + return (total + per_iteration - 1) // per_iteration + + def optimizer_step_budget(self, training_size=None): + full, remainder = divmod(self.total_groups, self.groups_per_iteration) + size = self.batch_size * self.accumulation_batches + return self.optimization_epochs * ( + full * ((self.groups_per_iteration * self.group_size + size - 1) // size) + + ((remainder * self.group_size + size - 1) // size if remainder else 0) + ) + + +PolicyPhase = Annotated[ + PolicyImitationPhaseConfig + | IQLPhaseConfig + | PPOPhaseConfig + | DPOPhaseConfig + | GRPOPhaseConfig, + Field(discriminator="algorithm"), +] diff --git a/src/sequifier/config/policy_train_config.py b/src/sequifier/config/policy_train_config.py new file mode 100644 index 000000000..2ab1759e3 --- /dev/null +++ b/src/sequifier/config/policy_train_config.py @@ -0,0 +1,129 @@ +"""Authored and resolved trajectory training contracts for the train command.""" + +import json +from pathlib import Path +from typing import Literal + +from pydantic import Field, PrivateAttr, model_validator + +from sequifier.config.policy_config import NonemptyStr, PolicyConfigBase + + +class PolicyTableColumns(PolicyConfigBase): + """Canonical feature names for tables without a model input schema.""" + + observations: list[NonemptyStr] = Field(default_factory=list) + static_inputs: list[NonemptyStr] = Field(default_factory=list) + + @model_validator(mode="after") + def validate_distinct_columns(self): + names = self.observations + self.static_inputs + if len(names) != len(set(names)): + raise ValueError("observation and static columns must be distinct") + return self + + +class SequenceSourceConfig(PolicyConfigBase): + _project_root: str = PrivateAttr(default=".") + _resolved_split_index: int | None = PrivateAttr(default=None) + units: dict[str, str] | None = None + paths: list[NonemptyStr] = Field(default_factory=list) + manifest_path: NonemptyStr | None = None + format: Literal["auto", "trajectory", "table", "prepared"] = "auto" + columns: PolicyTableColumns = Field(default_factory=PolicyTableColumns) + # A file boundary alone says nothing about task completion. + episode_end: Literal["unknown", "terminated", "truncated"] = "unknown" + episode_namespace: str = "" + metadata_config_path: str | None = None + encoding: Literal["raw", "model"] = "raw" + static_data_path: str | None = None + split_index: int | None = Field(default=None, ge=0) + action_semantics: Literal["policy", "executed_predictions"] = "policy" + + @property + def project_root(self) -> str: + """Run-owned media root, excluded from the authored and serialized schema.""" + return self._project_root + + @property + def selected_split_index(self) -> int | None: + """Keep the split selection available after expanding it into paths.""" + return ( + self.split_index + if self.split_index is not None + else self._resolved_split_index + ) + + @model_validator(mode="after") + def validate_location(self): + metadata_split = ( + self.metadata_config_path is not None and self.split_index is not None + ) + if sum((bool(self.paths), self.manifest_path is not None, metadata_split)) != 1: + raise ValueError("trajectory source requires either paths or one manifest") + if ( + self.split_index is not None + or self.encoding == "model" + or self.format == "prepared" + ) and self.metadata_config_path is None: + raise ValueError( + "prepared/model-encoded sources and split_index require metadata_config_path" + ) + return self + + +def resolve_sequence_source( + source: SequenceSourceConfig, root: Path, *, metadata=None +) -> SequenceSourceConfig: + values = source.model_dump() + values["paths"] = [str((root / path).resolve()) for path in source.paths] + for key in ("manifest_path", "metadata_config_path", "static_data_path"): + if values[key] is not None: + values[key] = str((root / values[key]).resolve()) + if source.metadata_config_path is not None and ( + source.static_data_path is None or source.split_index is not None + ): + from sequifier.config.metadata import ( + load_dataset_metadata, + validate_persisted_metadata_versions, + ) + + if metadata is None: + metadata = load_dataset_metadata(values["metadata_config_path"]) + prepared = ( + source.format == "prepared" + or source.split_index is not None + or any(Path(path).suffix.lower() == ".pt" for path in source.paths) + ) + if ( + prepared + and source.static_data_path is None + and metadata.static_features_path + ): + values["static_data_path"] = str( + (root / metadata.static_features_path).resolve() + ) + if source.split_index is not None: + if source.split_index >= len(metadata.split_paths): + raise ValueError( + "source split_index is outside the prepared metadata splits" + ) + path = (root / metadata.split_paths[source.split_index]).resolve() + paths = [str(path)] + if path.is_dir(): + manifest_path = path / "metadata.json" + folder_metadata = json.loads(manifest_path.read_text()) + validate_persisted_metadata_versions(folder_metadata) + paths = [ + str(path / shard["path"]) + for shard in folder_metadata["batch_files"] + ] + values.update( + paths=paths, + format="prepared", + split_index=None, + ) + resolved = SequenceSourceConfig.model_validate(values) + resolved._project_root = str(root) + resolved._resolved_split_index = source.selected_split_index + return resolved diff --git a/src/sequifier/config/prediction_config.py b/src/sequifier/config/prediction_config.py new file mode 100644 index 000000000..7bf8bc774 --- /dev/null +++ b/src/sequifier/config/prediction_config.py @@ -0,0 +1,33 @@ +"""Model-owned prediction parameters, independent of task bindings.""" + +from typing import Literal + +from pydantic import ConfigDict, Field, model_validator + +from sequifier.config.naming import ConfigModel + + +class PredictionConfigBase(ConfigModel): + model_config = ConfigDict(extra="forbid", frozen=True, allow_inf_nan=False) + + +class ModelScaleConfig(PredictionConfigBase): + """Model-owned Gaussian scale, including optional learned parameters.""" + + initial: float = Field(gt=0) + trainable: bool = False + min: float = Field(default=1e-4, gt=0) + max: float = Field(default=100.0, gt=0) + + @model_validator(mode="after") + def validate_range(self): + if not self.min <= self.initial <= self.max: + raise ValueError("scale.initial must lie between scale.min and scale.max") + return self + + +class PredictionDistributionConfig(PredictionConfigBase): + """A model output's Gaussian parameterization, independent of policy behavior.""" + + type: Literal["real"] = "real" + scale: ModelScaleConfig diff --git a/src/sequifier/config/preprocess_config.py b/src/sequifier/config/preprocess_config.py index f1ecc2478..397161d1f 100644 --- a/src/sequifier/config/preprocess_config.py +++ b/src/sequifier/config/preprocess_config.py @@ -4,21 +4,17 @@ from typing import Any, Optional, Union import numpy as np -from pydantic import ( - AliasChoices, - BaseModel, - ConfigDict, - Field, - field_validator, - model_validator, -) +from pydantic import ConfigDict, Field, field_validator, model_validator +from sequifier.config.command_ownership import CommandRunSettings from sequifier.config.composition import ( load_composed_yaml_config, merge_config_fragments, ) from sequifier.config.depth_layout import DepthLayoutRegistryModel from sequifier.config.image_config import ImageSourceConfig, PreconditioningImageConfig +from sequifier.config.naming import ConfigModel +from sequifier.config.sequence_import_config import SequenceImportConfig from sequifier.helpers import ( canonicalize_polars_dtype_name, is_float_dtype_name, @@ -31,58 +27,68 @@ @beartype def load_preprocessor_config( config_path: str, args_config: dict -) -> "PreprocessorModel": +) -> "PreprocessorModel | SequenceImportConfig": """Load preprocessing YAML plus CLI overrides.""" config_values = load_composed_yaml_config(config_path) + from sequifier.config.command_ownership import apply_run_overrides + + config_values, args_config = apply_run_overrides(config_values, args_config) config_values = merge_config_fragments((config_values, args_config)) - return try_catch_excess_keys(config_path, PreprocessorModel, config_values) + if config_values.get("operation") == "import_sequences": + return SequenceImportConfig.model_validate(config_values) + return try_catch_excess_keys( + config_path, + PreprocessorModel, + config_values, + ) -class CardinalityHashingModel(BaseModel): +class CardinalityHashingModel(ConfigModel): """Hash non-retained categorical values into a bounded set of buckets.""" model_config = ConfigDict(extra="forbid", populate_by_name=True) num_buckets: int = Field( ge=1, - validation_alias=AliasChoices("num_buckets", "buckets"), ) seed: int = 0 -class CardinalityLimitModel(BaseModel): +class CardinalityLimitModel(ConfigModel): """Bound the vocabulary produced for one categorical column.""" model_config = ConfigDict(extra="forbid", populate_by_name=True) - min_freq: Optional[int] = Field( + min_count: Optional[int] = Field( default=None, ge=1, - validation_alias=AliasChoices("min_freq", "min_count"), ) - top_k: Optional[int] = Field( + max_categories: Optional[int] = Field( default=None, ge=1, - validation_alias=AliasChoices("top_k", "k", "max_categories"), ) hashing: Optional[CardinalityHashingModel] = None @model_validator(mode="after") def validate_options(self) -> "CardinalityLimitModel": - if self.min_freq is not None and self.top_k is not None: + if self.min_count is not None and self.max_categories is not None: raise ValueError( - "top_k and min_freq are mutually exclusive cardinality limits" + "max_categories and min_count are mutually exclusive cardinality limits" ) - if self.min_freq is None and self.top_k is None and self.hashing is None: + if ( + self.min_count is None + and self.max_categories is None + and self.hashing is None + ): raise ValueError( - "cardinality configuration requires top_k, min_freq, or hashing" + "cardinality configuration requires max_categories, min_count, or hashing" ) return self -class PreprocessorModel(BaseModel): +class PreprocessorModel(ConfigModel): """Top-level preprocessing config.""" model_config = ConfigDict(arbitrary_types_allowed=True, extra="forbid") @@ -90,7 +96,7 @@ class PreprocessorModel(BaseModel): depth_layouts: DepthLayoutRegistryModel = Field( default_factory=DepthLayoutRegistryModel ) - project_root: str + run: CommandRunSettings = Field(default_factory=CommandRunSettings) preprocessing_data_path: str static_data_path: Optional[str] = None image_sources: dict[str, ImageSourceConfig] = Field(default_factory=dict) @@ -114,8 +120,7 @@ class PreprocessorModel(BaseModel): split_method: str = Field(default="within_sequence") split_column: Optional[str] = None split_values: Optional[list[Union[int, str, date, datetime]]] = None - window_length: Optional[int] = Field(default=None, gt=0) - window_lengths: Optional[list[int]] = None + window_lengths: list[int] = Field(min_length=1) max_target_offset: int = Field(default=1, ge=0) window_stride: Optional[int] = Field(default=None, gt=0) prediction_aligned_splits: list[int] = Field(default_factory=list) @@ -123,8 +128,6 @@ class PreprocessorModel(BaseModel): prediction_alignment_steps: int = Field(default=1, gt=0) target_offset: Optional[int] = Field(default=None, ge=0) max_rows: Optional[int] = None - seed: int = 1010 - n_cores: Optional[int] = None batches_per_file: int = 1024 process_by_file: bool = True continue_preprocessing: bool = False @@ -135,6 +138,26 @@ class PreprocessorModel(BaseModel): mask_column: Optional[str] = None curriculum_column: Optional[Union[str, list[str]]] = None + @property + def project_root(self): + return self.run.project_root + + @property + def seed(self): + return self.run.reproducibility.seed + + @property + def num_workers(self): + loading = self.run.execution.data_loading + return ( + loading.num_workers if "num_workers" in loading.model_fields_set else None + ) + + @model_validator(mode="after") + def validate_resources(self): + self.run.validate_for("preprocess") + return self + @field_validator("image_sources") @classmethod def validate_image_source_names(cls, sources): @@ -350,7 +373,7 @@ def validate_mask_column_requires_metadata(self) -> "PreprocessorModel": overlap = categorical & real if overlap: raise ValueError( - "Columns cannot be both categorical and real: " f"{sorted(overlap)}" + f"Columns cannot be both categorical and real: {sorted(overlap)}" ) declared = categorical | real @@ -434,7 +457,7 @@ def validate_mask_column_requires_metadata(self) -> "PreprocessorModel": raise ValueError("split_column cannot be sequenceId or itemPosition") if self.split_column == "__sequifier_split_value": raise ValueError( - "split_column cannot use reserved name " "'__sequifier_split_value'" + "split_column cannot use reserved name '__sequifier_split_value'" ) if self.split_column == self.mask_column: raise ValueError("split_column cannot also be mask_column") @@ -528,20 +551,7 @@ def validate_mask_column_requires_metadata(self) -> "PreprocessorModel": if self.split_values is not None else 0 ) - if (self.window_length is None) == (self.window_lengths is None): - raise ValueError("Specify exactly one of window_length or window_lengths") - if self.prediction_alignment_steps > 1 and self.window_lengths is None: - raise ValueError( - "window_lengths must be set when prediction_alignment_steps > 1" - ) - if self.window_lengths is not None: - widths = self.window_lengths - else: - if self.window_length is None: - raise ValueError( - "window_length is required when window_lengths is absent" - ) - widths = [self.window_length] * n_splits + widths = self.window_lengths if len(widths) != n_splits or any( isinstance(width, bool) or not isinstance(width, int) @@ -597,8 +607,5 @@ def validate_mask_column_requires_metadata(self) -> "PreprocessorModel": @beartype def __init__(self, **kwargs): if kwargs.get("window_stride") is None: - kwargs["window_stride"] = ( - kwargs.get("window_length") - or (kwargs.get("window_lengths") or [None])[0] - ) + kwargs["window_stride"] = (kwargs.get("window_lengths") or [None])[0] super().__init__(**kwargs) diff --git a/src/sequifier/config/probabilities.py b/src/sequifier/config/probabilities.py index 15d99e077..dafa37aa5 100644 --- a/src/sequifier/config/probabilities.py +++ b/src/sequifier/config/probabilities.py @@ -5,6 +5,7 @@ import torch from pydantic import BaseModel, ConfigDict, Field +from sequifier.config.naming import ConfigModel from sequifier.typechecking import beartype @@ -24,7 +25,7 @@ def sample( pass -class GeometricDistribution(BaseModel, ProbabilityDistributionBaseClass): +class GeometricDistribution(ConfigModel, ProbabilityDistributionBaseClass): model_config = ConfigDict(extra="forbid") type: Literal["geometric"] = "geometric" @@ -46,12 +47,12 @@ def sample( ) -class NormalDistributionDiscretizedFloor(BaseModel, ProbabilityDistributionBaseClass): +class NormalDistributionDiscretizedFloor(ConfigModel, ProbabilityDistributionBaseClass): model_config = ConfigDict(extra="forbid") - type: Literal["normal_discretized_floor"] = "normal_discretized_floor" + type: Literal["clipped_rounded_normal_plus_one"] = "clipped_rounded_normal_plus_one" mean: float - standard_deviation: float = Field(..., gt=0.0) + std: float = Field(..., gt=0.0) @beartype def sample( @@ -61,8 +62,7 @@ def sample( generator: Optional[torch.Generator] = None, ) -> torch.Tensor: val = ( - torch.randn(shape, device=device, generator=generator) - * self.standard_deviation + torch.randn(shape, device=device, generator=generator) * self.std + self.mean ) return torch.clamp(torch.round(val), min=0).long() + 1 @@ -73,9 +73,9 @@ class LogNormalDistributionDiscretizedFloor( ): model_config = ConfigDict(extra="forbid") - type: Literal["log_normal_discretized_floor",] = "log_normal_discretized_floor" + type: Literal["rounded_log_normal_plus_one",] = "rounded_log_normal_plus_one" mean: float - standard_deviation: float = Field(..., gt=0.0) + std: float = Field(..., gt=0.0) @beartype def sample( @@ -85,18 +85,17 @@ def sample( generator: Optional[torch.Generator] = None, ) -> torch.Tensor: normal = ( - torch.randn(shape, device=device, generator=generator) - * self.standard_deviation + torch.randn(shape, device=device, generator=generator) * self.std + self.mean ) val = torch.exp(normal) return torch.round(val).long() + 1 -class PoissonDistributionFloor(BaseModel, ProbabilityDistributionBaseClass): +class PoissonDistributionFloor(ConfigModel, ProbabilityDistributionBaseClass): model_config = ConfigDict(extra="forbid") - type: Literal["poisson_floor"] = "poisson_floor" + type: Literal["poisson_plus_one"] = "poisson_plus_one" rate: float = Field(..., gt=0.0) @beartype diff --git a/src/sequifier/config/rollout_config.py b/src/sequifier/config/rollout_config.py new file mode 100644 index 000000000..5f6580947 --- /dev/null +++ b/src/sequifier/config/rollout_config.py @@ -0,0 +1,71 @@ +"""Opt-in configuration for local, single-environment forecast rollouts.""" + +from typing import Any, Literal + +from pydantic import ConfigDict, Field, field_validator, model_validator + +from sequifier.config.naming import ConfigModel + + +class InputBinding(ConfigModel): + model_config = ConfigDict(extra="forbid", frozen=True) + + input: str = Field(min_length=1) + source: Literal["prediction", "environment"] + key: str = Field(min_length=1) + + +class AdapterConfig(ConfigModel): + model_config = ConfigDict(extra="forbid") + + factory: str + config: dict[str, Any] = Field(default_factory=dict) + + @field_validator("factory") + @classmethod + def validate_factory(cls, value: str) -> str: + module, separator, attribute = value.partition(":") + if not separator or not module or not attribute.isidentifier(): + raise ValueError( + "environment.factory must be a 'module:factory' import path" + ) + return value + + +class RolloutConfig(ConfigModel): + model_config = ConfigDict(extra="forbid") + + max_steps: int = Field(gt=0) + environment: AdapterConfig + bindings: list[InputBinding] + trajectory_path: str = Field(min_length=1) + # Index over prepared windows in dataset/chunk order, before seed filtering. + seed_window_index: int = Field(ge=0) + max_episode_steps: int | None = Field(default=None, gt=0) + + @model_validator(mode="after") + def reject_duplicate_producers(self): + names = [binding.input for binding in self.bindings] + if len(names) != len(set(names)): + raise ValueError("rollout bindings contain duplicate input producers") + return self + + def validate_bindings(self, inputs: list[str], targets: list[str]) -> None: + if len(inputs) != len(set(inputs)) or len(targets) != len(set(targets)): + raise ValueError("rollout input and target columns must be unique") + bound = {binding.input for binding in self.bindings} + if bound != set(inputs): + raise ValueError( + "rollout requires exactly one producer per changing input; " + f"missing={sorted(set(inputs) - bound)}, " + f"unknown/static={sorted(bound - set(inputs))}" + ) + unknown = { + binding.key + for binding in self.bindings + if binding.source == "prediction" and binding.key not in targets + } + if unknown: + raise ValueError( + f"rollout bindings reference unknown predictions: {sorted(unknown)}" + ) diff --git a/src/sequifier/config/run_capabilities.py b/src/sequifier/config/run_capabilities.py new file mode 100644 index 000000000..8e77952dc --- /dev/null +++ b/src/sequifier/config/run_capabilities.py @@ -0,0 +1,69 @@ +"""Runner and publication capabilities checked when compiling a training plan.""" + +import os +from typing import Literal + +from pydantic import Field + +from sequifier.config.naming import ConfigModel +from sequifier.config.ownership import ExportSettings, RunSettings + +RunMode = Literal["supervised", "policy", "coordinated"] + + +class PublicationPlan(ConfigModel): + mode: RunMode = "supervised" + exports: ExportSettings = Field(default_factory=ExportSettings) + + +def validate_runner_settings( + run: RunSettings, *, supervised: bool, policy: bool +) -> None: + execution = run.execution + if policy: + # These settings are consumed only by the supervised runner or finalizer. + supervised_only = ( + ( + "run.execution", + execution, + {"distribution", "compilation", "precision", "max_concat_tensors"}, + ), + ( + "run.execution.data_loading", + execution.data_loading, + {"num_workers", "load_full_data_to_ram", "max_ram_gb"}, + ), + ( + "run.export", + run.export, + { + "onnx", + "embedding_model", + "with_dropout", + "generative_model", + "pt", + "batch_size", + }, + ), + ) + for path, settings, fields in supervised_only: + if explicit := fields & settings.model_fields_set: + raise ValueError( + f"policy phases do not support {path} settings: {sorted(explicit)}" + ) + if not supervised and "log_interval_batches" in run.logging.model_fields_set: + raise ValueError("policy phases do not support batch logging intervals") + elif ( + not (run.export.generative_model or run.export.embedding_model) + and os.getenv("SEQUIFIER_PREVENT_EXPORT") is None + ): + raise ValueError("At least one model export must be enabled") + + +def publication_plan(run: RunSettings, phases) -> PublicationPlan: + policy = [phase for phase in phases if phase.algorithm != "supervised"] + if not policy: + return PublicationPlan(mode="supervised", exports=run.export) + if len(policy) == len(phases) and len({p.model_interface for p in policy}) == 1: + return PublicationPlan(mode="policy", exports=run.export) + return PublicationPlan(mode="coordinated", exports=run.export) diff --git a/src/sequifier/config/sequence_import_config.py b/src/sequifier/config/sequence_import_config.py new file mode 100644 index 000000000..ff5f07789 --- /dev/null +++ b/src/sequifier/config/sequence_import_config.py @@ -0,0 +1,78 @@ +"""Explicit sequence import with the same source and authored task contracts.""" + +from pathlib import Path +from typing import Literal + +from pydantic import Field, model_validator + +from sequifier.config.command_ownership import CommandRunSettings +from sequifier.config.policy_config import ( + PolicyBinding, + PolicyConfigBase, + ResolvedPolicyBinding, +) +from sequifier.config.policy_train_config import SequenceSourceConfig + + +class SequenceImportConfig(PolicyConfigBase): + operation: Literal["import_sequences"] + run: CommandRunSettings = Field(default_factory=CommandRunSettings) + source: SequenceSourceConfig + policy_binding: PolicyBinding + initialize_from: str | None = None + output_path: str + episodes_per_shard: int = Field(default=64, gt=0) + + @model_validator(mode="after") + def validate_resources(self): + unsupported = self.run.model_fields_set & {"execution", "reproducibility"} + if unsupported: + raise ValueError( + f"sequence import does not support run settings: {sorted(unsupported)}" + ) + return self + + @property + def project_root(self): + return self.run.project_root + + def resolved_source(self) -> SequenceSourceConfig: + from sequifier.config.policy_train_config import resolve_sequence_source + + return resolve_sequence_source(self.source, Path(self.project_root).resolve()) + + def resolved_policy(self) -> ResolvedPolicyBinding: + if self.initialize_from: + from sequifier.artifacts.model_artifact import read_execution_artifact + from sequifier.artifacts.model_config import ( + model_execution_from_model_config, + ) + from sequifier.rl.attachment import resolve_binding + + execution, name = model_execution_from_model_config( + read_execution_artifact( + Path(self.project_root) / self.initialize_from + ).model_config.values, + device="cpu", + ) + if len(execution.interface_names) != 1: + raise ValueError( + "sequence import initialize_from requires one model interface" + ) + return resolve_binding(execution, name, self.policy_binding) + observations = list(self.source.columns.observations) + static = list(self.source.columns.static_inputs) + if not observations: + raise ValueError( + "sequence import requires source.columns.observations or initialize_from" + ) + # Import validates and stores domain episodes, without constructing a model. + return ResolvedPolicyBinding( + **self.policy_binding.model_dump(), + context_length=1, + interface_name="default", + observation_columns=observations, + static_columns=static, + action_order=list(self.policy_binding.actions), + input_schema={}, + ) diff --git a/src/sequifier/config/sources.py b/src/sequifier/config/sources.py new file mode 100644 index 000000000..fffeae9a6 --- /dev/null +++ b/src/sequifier/config/sources.py @@ -0,0 +1,48 @@ +"""Named dataset and part references shared by training algorithms.""" + +from typing import Optional + +from pydantic import ConfigDict, Field, field_validator + +from sequifier.config.identifiers import identifier as _identifier +from sequifier.config.naming import ConfigModel +from sequifier.typechecking import beartype + + +def phase_sources(phase): + """All authored references, including held-out and preference sources.""" + if phase.algorithm in {"ppo", "grpo"}: + return [] + refs = [] if phase.algorithm == "dpo" else list(phase.sources) + if phase.evaluation is not None: + refs.extend(phase.evaluation.sources or []) + if phase.algorithm == "dpo": + refs.append(phase.preferences) + return refs + + +def validate_source(config, source, usage): + name, _, part = source.partition(".") + if name not in config.dataset_training or ( + part and part not in config.dataset_training[name].parts + ): + raise ValueError(f"unknown {usage} source {source!r}") + + +class TrainingSourceSpecModel(ConfigModel): + model_config = ConfigDict(extra="forbid") + + source: str + selection_weight: Optional[float] = Field(default=None, gt=0) + batches_per_selection: Optional[int] = Field(default=None, gt=0) + + @field_validator("source") + @classmethod + @beartype + def validate_source(cls, value): + parts = value.split(".") + if len(parts) > 2: + raise ValueError("Source refs use only dataset or dataset.part") + for part in parts: + _identifier(part, "Source reference component") + return value diff --git a/src/sequifier/config/train_config.py b/src/sequifier/config/train_config.py index 5a4809e56..4d8117f8b 100644 --- a/src/sequifier/config/train_config.py +++ b/src/sequifier/config/train_config.py @@ -1,40 +1,32 @@ -"""Canonical Sequifier training configuration and resolution. +"""Canonical named training configuration and resolution. -The concise singleton authoring surface is normalized to the canonical named -schema before validation. This module intentionally contains no migration -adapter for the historical flat single-dataset YAML schema. Runtime consumers -select a dataset, part, or model interface explicitly. +All cardinalities use named interfaces, datasets, parts and ordered phases. +Only the current ownership structure is accepted. """ from __future__ import annotations import copy import inspect -import keyword -import math import os -import warnings from dataclasses import dataclass from pathlib import Path from types import SimpleNamespace -from typing import Any, Literal, Optional +from typing import Annotated, Any, ClassVar, Literal, Optional -import torch from pydantic import ( - BaseModel, ConfigDict, Field, - StrictStr, + PrivateAttr, field_validator, + model_serializer, model_validator, ) -# Reusable schemas are independent of the canonical root configuration. from sequifier.config.components import ( BackboneComponentConfig, BERTSpecModel, CategoricalHashingConfig, - ComponentSpec, DecoderComponentConfig, FeatureLayoutRegistryModel, ImageEncoderConfig, @@ -43,10 +35,7 @@ ResumeConfig, ) from sequifier.config.depth_layout import DepthLayoutRegistryModel -from sequifier.config.freezing_config import ( - IngestionFreezingConfig, - LayerFreezingConfigFields, -) +from sequifier.config.identifiers import identifier as _identifier from sequifier.config.image_config import ( ImageSourceConfig, MediaPath, @@ -54,29 +43,53 @@ bind_image_paths, image_views, ) -from sequifier.config.metadata import DatasetMetadata, load_dataset_metadata -from sequifier.config.optimizer_config import ( - OptimizerPlan, - resolve_plan_scheduler_arguments, +from sequifier.config.learning_config import SupervisedLearningConfig +from sequifier.config.metadata import ( + DatasetMetadata, + TensorPayloadKind, + load_dataset_metadata, + validate_authored_metadata, +) +from sequifier.config.naming import ConfigModel +from sequifier.config.normalization_config import FrozenNormalizationConfig +from sequifier.config.parameter_selection import TrainabilityConfig +from sequifier.config.phase_controls import ( + CurriculumConfig, + EvaluationTriggers, + PhaseControls, ) +from sequifier.config.policy_config import PolicyBinding, ResolvedPolicyBinding +from sequifier.config.policy_optimization import SupervisedOptimizationConfig +from sequifier.config.policy_plan import ( + DPOPhaseConfig, + GRPOPhaseConfig, + IQLPhaseConfig, + PolicyImitationPhaseConfig, + PolicyPhase, + PPOPhaseConfig, +) +from sequifier.config.policy_train_config import SequenceSourceConfig +from sequifier.config.prediction_config import PredictionDistributionConfig +from sequifier.config.run_capabilities import PublicationPlan +from sequifier.config.sources import TrainingSourceSpecModel, phase_sources +from sequifier.config.sources import validate_source as _validate_source + +# Reusable schemas are independent of the canonical root configuration. from sequifier.helpers import ( ModelWindowView, StoredWindowLayout, derive_target_column_types, - metadata_config_path_from_preprocessing_data_path, normalize_path, resolve_window_view, ) from sequifier.model.embedding import validate_embedding_layer_names +from sequifier.model.execution_options import ExecutionOptions, ModelExecutionOptions from sequifier.objectives import ( - ALLOWED_OBJECTIVE_NAMES, - OBJECTIVE_NAME_MESSAGE, BERTObjective, NextOccurrenceObjective, get_objective_class, target_offset_for_objective, ) -from sequifier.optimizers.optimizers import get_optimizer_class, get_scheduler_class from sequifier.special_tokens import ( SPECIAL_TOKEN_IDS, SPECIAL_TOKEN_NAMES, @@ -105,15 +118,6 @@ def _validate_constructor_arguments( raise ValueError(f"Invalid {component} {name} arguments: {error}") from error -@beartype -def _identifier(value: str, usage: str) -> str: - if "." in value or not value.isidentifier() or keyword.iskeyword(value): - raise ValueError( - f"{usage} {value!r} must be a valid identifier and cannot contain '.'." - ) - return value - - @beartype def _unique_columns(value: list[str], usage: str) -> list[str]: if len(value) != len(set(value)): @@ -121,414 +125,37 @@ def _unique_columns(value: list[str], usage: str) -> list[str]: return value -_SINGLETON_CONFIG_NAME = "default" -_SINGLE_PHASE_NAME = "train" -_PHASE_FIELD_NAMES = frozenset({"name", "epochs", "mode", "selection", "sources"}) -_TRAINING_PLAN_POLICY_FIELDS = frozenset({"curriculum_training", "curriculum_column"}) - - -@beartype -def _normalize_dataset_part_surface(datasets: Any) -> None: - if not isinstance(datasets, dict): - return - for dataset_name, dataset in datasets.items(): - if not isinstance(dataset, dict): - continue - if "part" in dataset and "parts" in dataset: - raise ValueError( - f"Dataset {dataset_name!r} cannot define both 'part' and 'parts'." - ) - if "part" in dataset: - dataset["parts"] = {_SINGLETON_CONFIG_NAME: dataset.pop("part")} - - -@beartype -def _implicit_source(values: dict[str, Any]) -> str | None: - datasets = values.get("dataset_training") - if not isinstance(datasets, dict) or len(datasets) != 1: - return None - dataset_name, dataset = next(iter(datasets.items())) - if not isinstance(dataset, dict): - return None - parts = dataset.get("parts") - if not isinstance(parts, dict) or not parts: - return None - if len(parts) == 1: - return f"{dataset_name}.{next(iter(parts))}" - return dataset_name - - -@beartype -def normalize_train_config_surface(values: Any) -> Any: - """Expand the concise singleton authoring surface to canonical mappings.""" - - if not isinstance(values, dict): - return values - normalized = copy.deepcopy(values) - - model = normalized.get("model") - if isinstance(model, dict): - backbone = model.get("backbone") - global_training = normalized.get("global_training") - if isinstance(backbone, dict) and isinstance(global_training, dict): - architecture = backbone.get("architecture") - if isinstance(architecture, dict) and "context_length" in global_training: - architecture.setdefault( - "max_context_length", global_training["context_length"] - ) - if "interface" in model and "interfaces" in model: - raise ValueError("model cannot define both 'interface' and 'interfaces'.") - if "interface" in model: - model["interfaces"] = {_SINGLETON_CONFIG_NAME: model.pop("interface")} - - if "dataset" in normalized and "dataset_training" in normalized: - raise ValueError( - "Training config cannot define both 'dataset' and 'dataset_training'." - ) - if "dataset" in normalized: - normalized["dataset_training"] = { - _SINGLETON_CONFIG_NAME: normalized.pop("dataset") - } - - datasets = normalized.get("dataset_training") - _normalize_dataset_part_surface(datasets) - - interfaces = model.get("interfaces") if isinstance(model, dict) else None - if ( - isinstance(interfaces, dict) - and len(interfaces) == 1 - and isinstance(datasets, dict) - ): - interface_name = next(iter(interfaces)) - for dataset in datasets.values(): - if isinstance(dataset, dict): - dataset.setdefault("model_interface", interface_name) - - implicit_source = _implicit_source(normalized) - training_plan = normalized.get("training_plan") - if isinstance(training_plan, dict): - direct_fields = _PHASE_FIELD_NAMES & set(training_plan) - if "phases" in training_plan and direct_fields: - raise ValueError( - "training_plan cannot combine 'phases' with direct phase fields." - ) - if "phases" not in training_plan and direct_fields: - policy = { - key: training_plan[key] - for key in _TRAINING_PLAN_POLICY_FIELDS & set(training_plan) - } - phase = { - key: value - for key, value in training_plan.items() - if key not in _TRAINING_PLAN_POLICY_FIELDS - } - phase.setdefault("name", _SINGLE_PHASE_NAME) - phase.setdefault("mode", "sequential") - normalized["training_plan"] = {**policy, "phases": [phase]} - training_plan = normalized["training_plan"] - - phases = training_plan.get("phases") - if isinstance(phases, list) and implicit_source is not None: - for phase in phases: - if isinstance(phase, dict): - phase.setdefault("sources", [{"source": implicit_source}]) - - evaluation = normalized.get("evaluation") - if isinstance(evaluation, bool): - if not evaluation: - normalized["evaluation"] = None - elif implicit_source is None: - raise ValueError( - "evaluation=true requires exactly one configured dataset source." - ) - else: - normalized["evaluation"] = {"sources": [{"source": implicit_source}]} - - return normalized - - -@beartype -def _only_base_name(values: Any, usage: str) -> str: - if not isinstance(values, dict) or len(values) != 1: - raise ValueError( - f"The singleton {usage} override requires exactly one {usage} " - "in the base training config." - ) - return next(iter(values)) - - -@beartype -def normalize_train_config_parameter_surface( - parameters: Any, - base_values: dict[str, Any], -) -> Any: - """Translate singleton search parameters to canonical base paths.""" - - if not isinstance(parameters, dict): - return parameters - normalized = copy.deepcopy(parameters) - - model_override = normalized.get("model") - if isinstance(model_override, dict) and "interface" in model_override: - if "interfaces" in model_override: - raise ValueError( - "model parameters cannot define both 'interface' and 'interfaces'." - ) - interface_name = _only_base_name( - base_values.get("model", {}).get("interfaces"), - "model interface", - ) - model_override["interfaces"] = {interface_name: model_override.pop("interface")} - - if "dataset" in normalized: - if "dataset_training" in normalized: - raise ValueError( - "Training parameters cannot define both 'dataset' and " - "'dataset_training'." - ) - dataset_name = _only_base_name( - base_values.get("dataset_training"), - "dataset", - ) - normalized["dataset_training"] = {dataset_name: normalized.pop("dataset")} - - dataset_overrides = normalized.get("dataset_training") - base_datasets = base_values.get("dataset_training") - if isinstance(dataset_overrides, dict): - for dataset_name, dataset_override in dataset_overrides.items(): - if not isinstance(dataset_override, dict): - continue - if "part" in dataset_override and "parts" in dataset_override: - raise ValueError( - f"Dataset override {dataset_name!r} cannot define both " - "'part' and 'parts'." - ) - if "part" not in dataset_override: - continue - base_dataset = ( - base_datasets.get(dataset_name) - if isinstance(base_datasets, dict) - else None - ) - part_name = _only_base_name( - base_dataset.get("parts") if isinstance(base_dataset, dict) else None, - f"part for dataset {dataset_name!r}", - ) - dataset_override["parts"] = {part_name: dataset_override.pop("part")} - - plan_override = normalized.get("training_plan") - if isinstance(plan_override, dict): - direct_fields = _PHASE_FIELD_NAMES & set(plan_override) - if "phases" in plan_override and direct_fields: - raise ValueError( - "training_plan parameters cannot combine 'phases' with direct " - "phase fields." - ) - if "phases" not in plan_override and direct_fields: - base_phases = base_values.get("training_plan", {}).get("phases") - if not isinstance(base_phases, list) or len(base_phases) != 1: - raise ValueError( - "Direct training_plan parameters require exactly one phase " - "in the base training config." - ) - policy = { - key: plan_override[key] - for key in _TRAINING_PLAN_POLICY_FIELDS & set(plan_override) - } - phase_override = { - key: value - for key, value in plan_override.items() - if key not in _TRAINING_PLAN_POLICY_FIELDS - } - normalized["training_plan"] = { - **policy, - "phases": {0: phase_override}, - } - - return normalized - - -class GlobalTrainingSpecModel(BaseModel): - """Run-wide data, optimization, precision, and distribution settings.""" +class TrainingRuntimeSettings(ExecutionOptions): + """Internal native execution projection; never an authored ownership surface.""" model_config = ConfigDict(arbitrary_types_allowed=True, extra="forbid") - read_format: Literal["csv", "parquet", "pt"] = "parquet" - training_objective: str - context_length: int = Field(gt=0) - target_offset: int = Field(default=1, ge=0) + episode_cache_size: int = Field(default=8, gt=0) window_stride: Optional[int] = Field(default=None, gt=0) - inference_batch_size: int = Field(gt=0) - batch_size: int = Field(gt=0) - accumulation_steps: Optional[int] = Field(default=None, gt=0) - learning_rate: Optional[float] = Field(default=None, gt=0) - # Validate the plan branch first: ComponentSpec rejects non-mapping model - # instances with TypeError, which would otherwise abort union validation. - optimizer: OptimizerPlan | ComponentSpec = Field( - default_factory=lambda: ComponentSpec(name="Adam"), - union_mode="left_to_right", - ) - scheduler: ComponentSpec = Field( - default_factory=lambda: ComponentSpec( - name="StepLR", arguments={"step_size": 1, "gamma": 0.99} - ) - ) - scheduler_step_on: Literal["epoch", "batch"] = "epoch" - reset_optimization_on_phase: bool = True - gradient_clip: Optional[float] = Field(default=None, gt=0) + inference_batch_size: int = Field(default=1, gt=0) bert_spec: Optional[BERTSpecModel] = None next_occurrence_config: Optional[NextOccurrenceConfigModel] = None - device_max_concat_length: int = Field(default=12, gt=0) - log_interval: int = Field(default=10, gt=0) - early_stopping_epochs: Optional[int] = Field(default=None, gt=0) - save_interval_epochs: int = Field(default=1, gt=0) - save_latest_interval_minutes: Optional[float] = None - save_interval_minutes: Optional[float] = None - save_interval_batches: Optional[int] = None - save_interval_val_loss: bool = True - calculate_validation_loss_on_initialization: bool = True + log_interval_batches: int = Field(default=10, gt=0) + log_interval_iterations: int = Field(default=10, gt=0) + latest_checkpoint_interval_minutes: Optional[float] = None resume: Optional[ResumeConfig] = None enforce_determinism: bool = False distributed: bool = False load_full_data_to_ram: bool = True - max_ram_gb: int | float = 16 + max_ram_gb: float = Field(default=16, gt=0, allow_inf_nan=False) world_size: int = Field(default=1, gt=0) num_workers: int = Field(default=0, ge=0) backend: str = "nccl" - layer_type_dtypes: Optional[dict[str, str]] = None layer_autocast: bool = False data_parallelism: Optional[Literal["ddp", "fsdp"]] = None fsdp_cpu_offload: Optional[bool] = None torch_compile: Literal["outer", "inner", "none"] = "outer" float32_matmul_precision: Literal["highest", "high", "medium"] = "highest" - @field_validator("training_objective") - @classmethod - @beartype - def validate_objective(cls, value: str) -> str: - if value not in ALLOWED_OBJECTIVE_NAMES: - raise ValueError( - f"Only {OBJECTIVE_NAME_MESSAGE} are allowed, found {value}" - ) - return value - - @field_validator("optimizer", mode="before") - @classmethod - @beartype - def validate_optimizer(cls, value: Any) -> OptimizerPlan | ComponentSpec: - if isinstance(value, OptimizerPlan) or ( - isinstance(value, dict) and "groups" in value - ): - plan = OptimizerPlan.model_validate(value) - for group in plan.groups: - cls._validate_optimizer_spec( - group.optimizer, - group.learning_rate, - location=f"optimizer group {group.id!r}", - ) - return plan - spec = ComponentSpec.model_validate(value) - cls._validate_optimizer_spec(spec, 1.0, location="global_training") - return spec - - @classmethod - def _validate_optimizer_spec( - cls, spec: ComponentSpec, learning_rate: float, *, location: str - ) -> None: - name = spec.name - try: - optimizer_class = get_optimizer_class(name) - except ValueError: - raise ValueError(f"{name} not in the configured optimizer registries") - kwargs = dict(spec.arguments) - if "lr" in kwargs: - raise ValueError( - f"{location} must configure learning rate through its " - "learning_rate field, not optimizer.lr" - ) - kwargs["lr"] = learning_rate - _validate_constructor_arguments( - "optimizer", - name, - optimizer_class, - (object(),), - kwargs, - ) - - @model_validator(mode="after") - def validate_optimization_config(self) -> "GlobalTrainingSpecModel": - if isinstance(self.optimizer, OptimizerPlan): - if self.learning_rate is not None: - raise ValueError( - "global_training.learning_rate is only used with a single optimizer; " - "set learning_rate in each optimizer group instead." - ) - elif self.learning_rate is None: - raise ValueError( - "global_training.learning_rate is required with a single optimizer." - ) - - scheduler_class = get_scheduler_class(self.scheduler.name) - scheduler_arguments = dict(self.scheduler.arguments) - if isinstance(self.optimizer, OptimizerPlan): - scheduler_arguments = resolve_plan_scheduler_arguments( - self.optimizer, self.scheduler - ) - _validate_constructor_arguments( - "scheduler", - self.scheduler.name, - scheduler_class, - (object(),), - scheduler_arguments, - ) - return self - - @field_validator("scheduler", mode="before") - @classmethod - @beartype - def validate_scheduler(cls, value: Any) -> ComponentSpec: - return ComponentSpec.model_validate(value) - - @field_validator("layer_type_dtypes") - @classmethod - @beartype - def validate_layer_dtypes(cls, value: Optional[dict[str, str]]): - if value is None: - return value - allowed_keys = {"embedding", "linear", "conv", "norm", "decoder"} - allowed_types = { - "float32", - "float16", - "bfloat16", - "float64", - "float8_e4m3fn", - "float8_e5m2", - } - if invalid := set(value) - allowed_keys: - raise ValueError(f"Invalid layer_type_dtypes keys: {sorted(invalid)}") - if invalid := set(value.values()) - allowed_types: - raise ValueError(f"Invalid layer_type_dtypes values: {sorted(invalid)}") - return value - @model_validator(mode="after") @beartype def validate_distribution(self): - if ( - self.scheduler.name == "ReduceLROnPlateau" - and self.scheduler_step_on != "epoch" - ): - raise ValueError("ReduceLROnPlateau requires scheduler_step_on=epoch") - scheduler_total_steps = self.scheduler.arguments.get("total_steps") - if ( - scheduler_total_steps is not None - and self.reset_optimization_on_phase - and self.scheduler_step_on == "epoch" - ): - raise ValueError( - "global_training.scheduler.total_steps is managed per phase by " - "Sequifier for epoch-stepped schedulers and must not be provided." - ) if self.distributed and self.data_parallelism is None: raise ValueError("distributed=true requires data_parallelism") if self.data_parallelism != "fsdp" and self.fsdp_cpu_offload is not None: @@ -542,25 +169,22 @@ def validate_distribution(self): raise ValueError("FSDP requires torch_compile 'none' or 'inner'") if self.data_parallelism == "ddp" and self.torch_compile == "inner": raise ValueError("DDP requires torch_compile 'none' or 'outer'") - for name in ( - "save_latest_interval_minutes", - "save_interval_minutes", - "save_interval_batches", + interval = self.latest_checkpoint_interval_minutes + if ( + interval is not None + and interval <= 0 + and os.getenv("SEQUIFIER_TESTING") != "1" ): - value = getattr(self, name) - if ( - value is not None - and value <= 0 - and os.getenv("SEQUIFIER_TESTING") != "1" - ): - raise ValueError(f"{name} must be larger than zero") + raise ValueError( + "latest_checkpoint_interval_minutes must be larger than zero" + ) return self -class StaticConditioningConfig(BaseModel): +class StaticConditioningConfig(ConfigModel): model_config = ConfigDict(extra="forbid") columns: list[str] = Field(min_length=1) - embedding_size: int | None = Field(default=None, gt=0) + total_embedding_dim: int | None = Field(default=None, gt=0) feature_embedding_dims: dict[str, int] | None = None @field_validator("columns") @@ -581,21 +205,24 @@ def validate_embedding_dims(self): if any(dim <= 0 for dim in dims.values()): raise ValueError("Static feature embedding dimensions must be positive") if ( - self.embedding_size is not None - and sum(dims.values()) != self.embedding_size + self.total_embedding_dim is not None + and sum(dims.values()) != self.total_embedding_dim ): raise ValueError( - "Static feature embedding dimensions must sum to embedding_size" + "Static feature embedding dimensions must sum to total_embedding_dim" ) return self -class ModelInterfaceSpecModel(BaseModel): +class ModelInterfaceSpecModel(ConfigModel): """Architecture and selected-column contract for one named model route.""" model_config = ConfigDict(arbitrary_types_allowed=True, extra="forbid") input_columns: list[str] = Field(default_factory=list) + prediction_distributions: dict[str, PredictionDistributionConfig] = Field( + default_factory=dict + ) static_conditioning: StaticConditioningConfig | None = None preconditioning_image_encoder: ImageEncoderConfig | None = None target_columns: list[str] = Field(..., min_length=1) @@ -638,6 +265,8 @@ def validate_decoder_tokens(cls, value): @model_validator(mode="after") @beartype def validate_interface_contract(self): + if self.prediction_distributions.keys() - set(self.target_columns): + raise ValueError("prediction_distributions must name model targets") input_columns = set(self.input_columns) if image_targets := set(self.target_columns) & image_views(self.ingestion): raise ValueError( @@ -710,7 +339,7 @@ def image_cell_names(ingestion): return self -class ModelSpecModel(BaseModel): +class ModelSpecModel(ConfigModel): """One shared backbone and one or more named interfaces.""" model_config = ConfigDict(arbitrary_types_allowed=True, extra="forbid") @@ -727,110 +356,89 @@ def validate_interface_names(cls, value): return value -class DatasetPartSpecModel(BaseModel): +class DatasetPartSpecModel(ConfigModel): model_config = ConfigDict(extra="forbid") + type: Literal["prepared"] = "prepared" + read_format: Literal["csv", "parquet", "pt"] = "parquet" metadata_config_path: str - image_path: MediaPath = None - preconditioning_image_path: MediaPath = None + image_paths: MediaPath = None + preconditioning_image_paths: MediaPath = None file_order: Literal["shuffled", "name"] = "shuffled" -class DatasetFreezingSpecModel(BaseModel): +class SequencePartSpecModel(SequenceSourceConfig): + type: Literal["sequence"] + + +class PreferencePartSpecModel(ConfigModel): model_config = ConfigDict(extra="forbid") + type: Literal["preferences"] + manifest_path: str = Field(min_length=1) - backbone: LayerFreezingConfigFields = Field( - default_factory=LayerFreezingConfigFields - ) - ingestion: IngestionFreezingConfig = Field(default_factory=IngestionFreezingConfig) - ingestion_adapter: bool = False - decoder: LayerFreezingConfigFields = Field( - default_factory=LayerFreezingConfigFields - ) - @property - @beartype - def active(self) -> bool: - return self.ingestion_adapter or any( - value.has_freezing_policy - for value in (self.backbone, self.ingestion, self.decoder) - ) +DatasetPart = Annotated[ + DatasetPartSpecModel | SequencePartSpecModel | PreferencePartSpecModel, + Field(discriminator="type"), +] -class DatasetTrainingSpecModel(BaseModel): +class EvaluationMonitorSpecModel(ConfigModel): model_config = ConfigDict(extra="forbid") - model_interface: str - parts: dict[str, DatasetPartSpecModel] = Field(..., min_length=1) - criterion: dict[str, str] = Field(..., min_length=1) - class_weights: Optional[dict[str, list[float]]] = None - loss_weights: Optional[dict[str, float]] = None - class_share_log_columns: list[str] = Field(default_factory=list) - freeze: DatasetFreezingSpecModel = Field(default_factory=DatasetFreezingSpecModel) + source: str + metric: Literal["loss"] = "loss" + direction: Literal["min", "max"] = "min" - @field_validator("model_interface") - @classmethod - @beartype - def validate_interface_name(cls, value): - return _identifier(value, "Model interface reference") - @field_validator("parts") - @classmethod - @beartype - def validate_part_names(cls, value): - for name in value: - _identifier(name, "Dataset part name") - return value +class EvaluationSpecModel(EvaluationTriggers): + model_config = ConfigDict(extra="forbid") - @field_validator("criterion") - @classmethod - @beartype - def validate_criteria(cls, value): - for name in value.values(): - if name != "AccurateCrossEntropyLoss" and not hasattr(torch.nn, name): - raise ValueError(f"Criterion {name!r} not found in torch.nn") - return value + batch_size: int | None = Field(default=None, gt=0) + sources: list[TrainingSourceSpecModel] = Field(..., min_length=1) + monitor: Optional[EvaluationMonitorSpecModel] = None - @field_validator("loss_weights") - @classmethod + @model_validator(mode="after") @beartype - def validate_loss_weights(cls, value): - if value is None: - return value - invalid = { - target: weight - for target, weight in value.items() - if not math.isfinite(weight) or weight < 0 - } - if invalid: + def validate_monitor(self): + if any( + source.selection_weight is not None + or source.batches_per_selection is not None + for source in self.sources + ): raise ValueError( - "loss_weights must contain finite, non-negative values; " - f"found {invalid}." + "Evaluation sources cannot configure selection_weight or batches_per_selection" ) - return value + sources = [source.source for source in self.sources] + if len(sources) != len(set(sources)): + raise ValueError("Evaluation sources must be unique") + if self.monitor is not None and self.monitor.source not in sources: + raise ValueError("evaluation.monitor.source must be an evaluation source") + if len(sources) == 1 and self.monitor is None: + self.monitor = EvaluationMonitorSpecModel(source=sources[0]) + return self -class TrainingSourceSpecModel(BaseModel): +class TrainingPhaseSpecModel(PhaseControls): model_config = ConfigDict(extra="forbid") - source: str - weight: Optional[float] = Field(default=None, gt=0) - batches_per_selection: Optional[int] = Field(default=None, gt=0) + evaluation: EvaluationSpecModel | None = None + learning: SupervisedLearningConfig = Field(default_factory=SupervisedLearningConfig) + trainability: TrainabilityConfig = Field(default_factory=TrainabilityConfig) + curriculum: CurriculumConfig | None = None - @field_validator("source") - @classmethod - @beartype - def validate_source(cls, value): - parts = value.split(".") - if len(parts) > 2: - raise ValueError("Source refs use only dataset or dataset.part") - for part in parts: - _identifier(part, "Source reference component") - return value + algorithm: Literal["supervised"] = "supervised" + optimization_roles: ClassVar[tuple[str, ...]] = ("model",) + optimizer_state: Literal["reset", "continue"] = "reset" + batch_size: int = Field(default=32, gt=0) + accumulation_batches: int = Field(default=1, gt=0) + optimization: dict[Literal["model"], SupervisedOptimizationConfig] = Field( + default_factory=lambda: {"model": SupervisedOptimizationConfig()} + ) -class TrainingPhaseSpecModel(BaseModel): - model_config = ConfigDict(extra="forbid") + def optimization_configs(self): + return {"model": self.optimization.get("model", SupervisedOptimizationConfig())} name: str epochs: int = Field(gt=0) @@ -847,6 +455,16 @@ def validate_name(cls, value): @model_validator(mode="after") @beartype def validate_mode_fields(self): + datasets = {ref.source.partition(".")[0] for ref in self.sources} + self.trainability.validate_roles(self.optimization_roles, datasets=datasets) + learning_datasets = datasets | { + ref.source.partition(".")[0] + for ref in (self.evaluation.sources if self.evaluation else []) + } + if unknown := self.learning.datasets.keys() - learning_datasets: + raise ValueError( + f"learning declarations require training/evaluation datasets: {sorted(unknown)}" + ) if self.mode == "sequential": if self.selection is not None: raise ValueError("Sequential phases cannot configure selection") @@ -854,326 +472,232 @@ def validate_mode_fields(self): raise ValueError( "batches_per_selection is invalid in sequential phases" ) - if any(source.weight is not None for source in self.sources): - raise ValueError("weight is invalid in sequential phases") + if any(source.selection_weight is not None for source in self.sources): + raise ValueError("selection_weight is invalid in sequential phases") else: self.selection = self.selection or "round_robin" if self.selection == "round_robin" and any( - source.weight is not None for source in self.sources + source.selection_weight is not None for source in self.sources ): - raise ValueError("weight is only valid for weighted_random selection") + raise ValueError( + "selection_weight is only valid for weighted_random selection" + ) return self -class TrainingPlanModel(BaseModel): +TrainingPhase = Annotated[ + TrainingPhaseSpecModel + | PolicyImitationPhaseConfig + | IQLPhaseConfig + | PPOPhaseConfig + | DPOPhaseConfig + | GRPOPhaseConfig, + Field(discriminator="algorithm"), +] + + +class TrainingPlanModel(ConfigModel): model_config = ConfigDict(extra="forbid") - curriculum_training: bool = False - curriculum_column: Optional[str] = Field(default=None, min_length=1) - phases: list[TrainingPhaseSpecModel] = Field(..., min_length=1) + phases: list[TrainingPhase] = Field(..., min_length=1) @model_validator(mode="after") @beartype def validate_unique_names(self): - if self.curriculum_column is not None and not self.curriculum_training: - raise ValueError( - "curriculum_column may only be set when curriculum_training is true" - ) - if self.curriculum_training and self.curriculum_column is None: - raise ValueError( - "curriculum_column must be set when curriculum_training is true" - ) names = [phase.name for phase in self.phases] if len(names) != len(set(names)): raise ValueError("Training phase names must be unique") - return self - - -class EvaluationMonitorSpecModel(BaseModel): - model_config = ConfigDict(extra="forbid") - - source: str - metric: Literal["loss"] = "loss" - mode: Literal["min", "max"] = "min" - - -class EvaluationSpecModel(BaseModel): - model_config = ConfigDict(extra="forbid") - - sources: list[TrainingSourceSpecModel] = Field(..., min_length=1) - monitor: Optional[EvaluationMonitorSpecModel] = None - - @model_validator(mode="after") - @beartype - def validate_monitor(self): - if any( - source.weight is not None or source.batches_per_selection is not None - for source in self.sources - ): - raise ValueError( - "Evaluation sources cannot configure weight or batches_per_selection" - ) - sources = [source.source for source in self.sources] - if len(sources) != len(set(sources)): - raise ValueError("Evaluation sources must be unique") - if self.monitor is not None and self.monitor.source not in sources: - raise ValueError("evaluation.monitor.source must be an evaluation source") - if len(sources) == 1 and self.monitor is None: - self.monitor = EvaluationMonitorSpecModel(source=sources[0]) - return self - - -class SequifierConfig(BaseModel): - """Training configuration with singleton authoring normalization.""" - - model_config = ConfigDict(arbitrary_types_allowed=True, extra="forbid") - - project_root: str - model_name: str - device: str - seed: int = 1010 - global_training: GlobalTrainingSpecModel - model: ModelSpecModel - dataset_training: dict[str, DatasetTrainingSpecModel] = Field(..., min_length=1) - training_plan: TrainingPlanModel - evaluation: Optional[EvaluationSpecModel] = None - - export_generative_model: bool = True - export_embedding_model: bool = False - embedding_layer_names: list[StrictStr] = Field( - default_factory=lambda: ["backbone.final_norm"], min_length=1 - ) - export_onnx: bool = True - export_pt: bool = False - export_with_dropout: bool = False - - @model_validator(mode="before") - @classmethod - @beartype - def normalize_singleton_surface(cls, values): - return normalize_train_config_surface(values) - - @field_validator("dataset_training") - @classmethod - @beartype - def validate_dataset_names(cls, value): - for name in value: - _identifier(name, "Dataset name") - return value - - @field_validator("model_name") - @classmethod - @beartype - def validate_model_name(cls, value): - if "embedding" in value: - raise ValueError("model_name cannot contain 'embedding'") - return value - - @model_validator(mode="after") - @beartype - def validate_relationships(self): - if self.global_training.early_stopping_epochs is not None and ( - self.evaluation is None or self.evaluation.monitor is None - ): - raise ValueError("early stopping requires evaluation.monitor") + from sequifier.runtime.plan import validate_continuation - if self.global_training.scheduler.name == "ReduceLROnPlateau" and ( - self.evaluation is None or self.evaluation.monitor is None - ): - raise ValueError("ReduceLROnPlateau requires evaluation.monitor") - - scheduler_total_steps = self.global_training.scheduler.arguments.get( - "total_steps" - ) - if scheduler_total_steps is not None: - total_epochs = sum(phase.epochs for phase in self.training_plan.phases) - reset_on_phase = self.global_training.reset_optimization_on_phase - if reset_on_phase: - if self.global_training.scheduler_step_on == "batch": - warnings.warn( - "Batch-stepped scheduler " - f"total_steps={scheduler_total_steps} is applied " - "independently to every training phase.", - stacklevel=2, - ) - elif self.global_training.scheduler_step_on == "epoch": - if scheduler_total_steps != total_epochs: - raise ValueError( - "scheduler total steps: " - f"{scheduler_total_steps} != {total_epochs}: total epochs" - ) - else: - warnings.warn( - f"{scheduler_total_steps} scheduler steps at {total_epochs} " - "epochs implies " - f"{scheduler_total_steps / total_epochs:.2f} batches. " - "Does this seem correct?", - stacklevel=2, - ) - - for interface in self.model.interfaces.values(): - validate_embedding_layer_names( - self.embedding_layer_names, - SimpleNamespace( - backbone=self.model.backbone, - decoder=interface.decoder, - ), - ) - - referenced_interfaces = set() - for dataset_name, dataset in self.dataset_training.items(): - if dataset.model_interface not in self.model.interfaces: - raise ValueError( - f"Dataset {dataset_name!r} references unknown model interface " - f"{dataset.model_interface!r}." - ) - referenced_interfaces.add(dataset.model_interface) - interface = self.model.interfaces[dataset.model_interface] - if set(dataset.criterion) != set(interface.target_columns): - raise ValueError( - f"Dataset {dataset_name!r} criterion keys must equal interface " - f"target_columns." - ) - if dataset.loss_weights is not None and set(dataset.loss_weights) - set( - interface.target_columns + for index, phase in enumerate(self.phases): + if phase.optimizer_state != "continue": + continue + previous = self.phases[index - 1] if index else None + validate_continuation(previous, phase) + assert previous is not None + if phase.algorithm == "supervised" and ( + {ref.source.partition(".")[0] for ref in previous.sources} + != {ref.source.partition(".")[0] for ref in phase.sources} ): raise ValueError( - f"Dataset {dataset_name!r} loss_weights references unknown targets." + "supervised optimizer continuation requires identical dataset ownership" ) - if dataset.loss_weights is not None and all( - dataset.loss_weights.get(target, 1.0) == 0.0 - for target in interface.target_columns - ): + if { + role: value.continuation_settings() + for role, value in previous.optimization_configs().items() + } != { + role: value.continuation_settings() + for role, value in phase.optimization_configs().items() + }: raise ValueError( - f"Dataset {dataset_name!r} must have at least one target with " - "a positive loss weight." + "optimizer continuation requires identical role optimization settings" ) - if dataset.class_weights is not None and set(dataset.class_weights) - set( - interface.target_columns + if phase.algorithm != "supervised" and any( + item.scheduler.name == "OneCycleLR" + for item in phase.optimization_configs().values() ): raise ValueError( - f"Dataset {dataset_name!r} class_weights references unknown targets." + "policy OneCycleLR continuation is unsupported; use optimizer_state: reset" ) - unreferenced = set(self.model.interfaces) - referenced_interfaces - if unreferenced: - warnings.warn( - f"Unreferenced model interfaces: {sorted(unreferenced)}", - stacklevel=2, - ) + return self - for phase in self.training_plan.phases: - for source in phase.sources: - self._validate_source(source.source, f"training phase {phase.name!r}") - if self.evaluation is not None: - for source in self.evaluation.sources: - self._validate_source(source.source, "evaluation") - needs_monitor = len(self.evaluation.sources) > 1 and ( - self.global_training.save_interval_val_loss - or self.global_training.early_stopping_epochs is not None - ) - if needs_monitor and self.evaluation.monitor is None: + +def validate_training_contract(config, phases): + """Check model and phase requirements without reading training data.""" + for phase in phases: + evaluation = phase.evaluation + if phase.algorithm != "supervised": + if any( + ref.selection_weight is not None + or ref.batches_per_selection is not None + for ref in phase_sources(phase) + ): raise ValueError( - "Multiple evaluation sources require evaluation.monitor when " - "validation-based saving or early stopping is enabled." + "policy source references cannot configure supervised interleaving" ) - - objective = get_objective_class(self.global_training.training_objective) - is_bert = issubclass(objective, BERTObjective) - is_next = issubclass(objective, NextOccurrenceObjective) - if (self.global_training.bert_spec is not None) != is_bert: - raise ValueError("bert_spec must be configured exactly for BERT training") - if (self.global_training.next_occurrence_config is not None) != is_next: + continue + role = ( + phase.optimization_configs()["model"] + if isinstance(phase, TrainingPhaseSpecModel) + else phase.optimization["model"] + ) + needs_monitor = ( + phase.handoff == "best" + or role.scheduler.name == "ReduceLROnPlateau" + or phase.stopping is not None + or (evaluation is not None and evaluation.on_snapshot) + ) + if needs_monitor and (evaluation is None or evaluation.monitor is None): raise ValueError( - "next_occurrence_config must be configured exactly for " - "next_occurrence training" + f"{phase.name}: best handoff, plateau scheduling, validation saving, and early stopping require evaluation.monitor" ) if ( - len(self.dataset_training) > 1 - and self.global_training.data_parallelism == "fsdp" + role.scheduler.name == "ReduceLROnPlateau" + and phase.evaluation_interval_epochs != 1 ): - raise ValueError("Multi-dataset training does not support FSDP") - - context_length = self.global_training.context_length - prefix_positions = int( - any( - route.static_conditioning is not None - or route.preconditioning_image_encoder is not None - for route in self.model.interfaces.values() - ) + raise ValueError("ReduceLROnPlateau requires evaluation_interval_epochs: 1") + + supervised = [phase for phase in phases if phase.algorithm == "supervised"] + if not supervised: + return config + model = config.model + assert model is not None + if ( + config.runtime_settings.training_objective is None + or config.runtime_settings.context_length is None + ): + raise ValueError( + "supervised training requires training_objective and context_length" ) - if ( - context_length + prefix_positions - > self.model.backbone.architecture.max_context_length - ): + for interface in model.interfaces.values(): + validate_embedding_layer_names( + config.run.export.embedding_layer_names, + SimpleNamespace( + backbone=model.backbone, + decoder=interface.decoder, + ), + ) + + objective = get_objective_class(config.runtime_settings.training_objective) + is_bert = issubclass(objective, BERTObjective) + for phase in supervised: + if (phase.learning.bert_spec is not None) != is_bert: + raise ValueError("bert_spec must be configured exactly for BERT training") + + context_length = config.runtime_settings.context_length + prefix_positions = int( + any( + route.static_conditioning is not None + or route.preconditioning_image_encoder is not None + for route in model.interfaces.values() + ) + ) + if ( + context_length + prefix_positions + > model.backbone.architecture.max_context_length + ): + raise ValueError( + "context_length plus static prefix exceeds backbone max_context_length" + ) + for name, interface in model.interfaces.items(): + if interface.decoder.support_length > context_length: raise ValueError( - "context_length plus static prefix exceeds backbone max_context_length" + f"Interface {name!r} decoder support exceeds context_length" ) - for name, interface in self.model.interfaces.items(): - if interface.decoder.support > context_length: - raise ValueError( - f"Interface {name!r} decoder support exceeds context_length" - ) - decoded_length = context_length - interface.decoder.support + 1 - if interface.decoder.prediction_length > decoded_length: - raise ValueError( - f"Interface {name!r} prediction_length exceeds decoded length" - ) - objective.validate_prediction_length( - interface.decoder.prediction_length, - context_length, - usage="training", + decoded_length = context_length - interface.decoder.support_length + 1 + if interface.decoder.prediction_length > decoded_length: + raise ValueError( + f"Interface {name!r} prediction_length exceeds decoded length" ) - if ( - not self.export_generative_model - and not self.export_embedding_model - and os.getenv("SEQUIFIER_PREVENT_EXPORT") is None - ): - raise ValueError("At least one model export must be enabled") - return self - - @beartype - def _validate_source(self, source: str, usage: str) -> None: - dataset_name, _, part_name = source.partition(".") - dataset = self.dataset_training.get(dataset_name) - if dataset is None: - raise ValueError(f"Unknown {usage} dataset source {dataset_name!r}") - if part_name and part_name not in dataset.parts: - raise ValueError(f"Unknown {usage} part source {source!r}") + objective.validate_prediction_length( + interface.decoder.prediction_length, + context_length, + usage="training", + ) + return config -class ResolvedDatasetPart(BaseModel): +class ResolvedDatasetPart(ConfigModel): model_config = ConfigDict(arbitrary_types_allowed=True, extra="forbid") name: str metadata_config_path: str metadata: DatasetMetadata - image_path: MediaPath = None - preconditioning_image_path: MediaPath = None + metadata_from_file: bool + image_paths: MediaPath = None + preconditioning_image_paths: MediaPath = None training_data_path: str validation_data_path: Optional[str] = None + read_format: Literal["csv", "parquet", "pt"] = "parquet" storage_form: Literal["file", "folder"] file_order: Literal["shuffled", "name"] = "shuffled" -class ResolvedModelInterface(BaseModel): +class ResolvedInterfaceStorage(ConfigModel): model_config = ConfigDict(arbitrary_types_allowed=True, extra="forbid") - depth_layouts: DepthLayoutRegistryModel = Field( - default_factory=DepthLayoutRegistryModel - ) - tensor_payload_version: int = 1 + tensor_payload_kind: TensorPayloadKind + tensor_payload_version: int = Field(strict=True) split_paths: list[str] = Field(default_factory=list) prediction_aligned_splits: list[int] = Field(default_factory=list) prediction_alignment_steps: int = 1 window_lengths: Optional[list[int]] = None prediction_length: Optional[int] = None target_offset: Optional[int] = None - name: str - input_columns: list[str] - static_conditioning: StaticConditioningConfig | None = None static_features_path: str | None = None + storage_layout: StoredWindowLayout + + @classmethod + def from_metadata( + cls, metadata: DatasetMetadata, layout: StoredWindowLayout | None = None + ) -> ResolvedInterfaceStorage: + return cls( + tensor_payload_kind=metadata.tensor_payload_kind, + tensor_payload_version=metadata.tensor_payload_version, + split_paths=metadata.split_paths, + prediction_aligned_splits=metadata.prediction_aligned_splits, + prediction_alignment_steps=metadata.prediction_alignment_steps, + window_lengths=metadata.window_lengths, + prediction_length=metadata.prediction_length, + target_offset=metadata.target_offset, + static_features_path=metadata.static_features_path, + storage_layout=layout or metadata.storage_layout, + ) + + +class ResolvedModelInterface(ConfigModel): image_sources: dict[str, ImageSourceConfig] = Field(default_factory=dict) preconditioning_image: PreconditioningImageConfig | None = None + policy: ResolvedPolicyBinding | None = None + prediction_distributions: dict[str, PredictionDistributionConfig] = Field( + default_factory=dict + ) + depth_layouts: DepthLayoutRegistryModel = Field( + default_factory=DepthLayoutRegistryModel + ) + name: str + input_columns: list[str] + static_conditioning: StaticConditioningConfig | None = None preconditioning_image_encoder: ImageEncoderConfig | None = None static_categorical_columns: list[str] = Field(default_factory=list) static_real_columns: list[str] = Field(default_factory=list) @@ -1200,37 +724,65 @@ class ResolvedModelInterface(BaseModel): target_decoder_ids: dict[str, list[int]] target_n_classes: dict[str, int] target_global_to_decoder: dict[str, list[int]] - storage_layout: StoredWindowLayout window_view: ModelWindowView + @model_validator(mode="after") + def validate_prediction_targets(self): + for name in self.prediction_distributions: + if ( + name not in self.target_columns + or self.target_column_types.get(name) != "real" + ): + raise ValueError( + f"{name}: prediction_distributions requires a real model target" + ) + return self + -class ResolvedDatasetTrainingSpec(BaseModel): +class ResolvedDatasetTrainingSpec(ConfigModel): model_config = ConfigDict(arbitrary_types_allowed=True, extra="forbid") name: str model_interface: str - interface: ResolvedModelInterface + storage: ResolvedInterfaceStorage | None = None parts: dict[str, ResolvedDatasetPart] criterion: dict[str, str] class_weights: Optional[dict[str, list[float]]] = None loss_weights: Optional[dict[str, float]] = None class_share_log_columns: list[str] - freeze: DatasetFreezingSpecModel -class ResolvedTrainingSource(BaseModel): +class ResolvedTrainingSource(ConfigModel): model_config = ConfigDict(extra="forbid") source: str dataset: str part: Optional[str] = None - weight: float = 1.0 + parts: tuple[str, ...] + selection_weight: float = 1.0 batches_per_selection: int = 1 -class ResolvedTrainingPhase(BaseModel): +class ResolvedTrainingPhase(PhaseControls): + resolved_seed: int | None = None + trainability_summary: dict[str, list[str]] = Field(default_factory=dict) + model_config = ConfigDict(extra="forbid") + evaluation: EvaluationSpecModel | None = None + learning: SupervisedLearningConfig = Field(default_factory=SupervisedLearningConfig) + trainability: TrainabilityConfig = Field(default_factory=TrainabilityConfig) + curriculum: CurriculumConfig | None = None + evaluation_sources: list[ResolvedTrainingSource] = Field(default_factory=list) + evaluation_monitor: EvaluationMonitorSpecModel | None = None + + algorithm: Literal["supervised"] = "supervised" + + optimizer_state: Literal["reset", "continue"] = "reset" + batch_size: int = Field(default=32, gt=0) + accumulation_batches: int = Field(default=1, gt=0) + optimization: dict[str, SupervisedOptimizationConfig] + name: str epochs: int mode: Literal["sequential", "interleaved"] @@ -1238,8 +790,90 @@ class ResolvedTrainingPhase(BaseModel): sources: list[ResolvedTrainingSource] -class ResolvedSequifierConfig(BaseModel): - """Runtime configuration after all dataset parts and interfaces resolve.""" +def resolve_phases(authored_phases, datasets, *, seed: int): + """Resolve role defaults and epoch budgets without accessing datasets.""" + phases = [ + ResolvedTrainingPhase( + name=phase.name, + resolved_seed=seed + phase_index, + epochs=phase.epochs, + evaluation_sources=[ + _source(ref.source, datasets, 1) for ref in phase.evaluation.sources + ] + if phase.evaluation is not None + else [], + evaluation_monitor=phase.evaluation.monitor + if phase.evaluation is not None + else None, + evaluation=phase.evaluation, + learning=phase.learning, + trainability=phase.trainability, + curriculum=phase.curriculum, + checkpoints=phase.checkpoints, + model_selection=phase.model_selection, + stopping=phase.stopping, + optimizer_state=phase.optimizer_state, + batch_size=phase.batch_size, + accumulation_batches=phase.accumulation_batches, + optimization={ + role: value.model_copy(deep=True) + for role, value in phase.optimization_configs().items() + }, + mode=phase.mode, + selection=phase.selection or "round_robin", + sources=[ + _source( + source.source, + datasets, + phase.accumulation_batches, + selection_weight=source.selection_weight, + batches_per_selection=( + None + if phase.mode == "sequential" + else source.batches_per_selection + ), + ) + for source in phase.sources + ], + ) + if phase.algorithm == "supervised" + else phase + for phase_index, phase in enumerate(authored_phases) + ] + # A continued epoch scheduler spans the entire contiguous ownership chain. + for index, phase in enumerate(phases): + if phase.algorithm != "supervised": + continue + role = phase.optimization["model"] + if role.scheduler.name != "OneCycleLR" or role.scheduler_step_on != "epoch": + continue + start = index + while start > 0 and phases[start].optimizer_state == "continue": + start -= 1 + end = index + 1 + while end < len(phases) and phases[end].optimizer_state == "continue": + end += 1 + budget = sum(item.epochs for item in phases[start:end]) + explicit = role.scheduler.arguments.get("total_steps") + if explicit is not None and explicit != budget: + raise ValueError( + f"{phase.name}: OneCycleLR.total_steps must equal epoch budget {budget}" + ) + arguments = {**role.scheduler.arguments, "total_steps": budget} + role.scheduler = role.scheduler.model_copy(update={"arguments": arguments}) + return phases + + +class ResolvedSequifierConfig(ConfigModel): + """Compiled run, model, source, phase, and publication records.""" + + _prepared_inputs: Any = PrivateAttr(default=None) + + ownership: dict[str, Any] = Field(default_factory=dict) + source_validation_fractions: dict[str, float] = Field(default_factory=dict) + normalization_by_interface: dict[str, FrozenNormalizationConfig] = Field( + default_factory=dict + ) model_config = ConfigDict(arbitrary_types_allowed=True, extra="forbid") @@ -1247,84 +881,112 @@ class ResolvedSequifierConfig(BaseModel): model_name: str device: str seed: int - global_training: GlobalTrainingSpecModel + runtime_settings: TrainingRuntimeSettings + model_options: ModelExecutionOptions model: ModelSpecModel + interfaces: dict[str, ResolvedModelInterface] = Field(min_length=1) + interface_storage: dict[str, ResolvedInterfaceStorage] = Field(default_factory=dict) + model_metadata_paths: dict[str, str] = Field(default_factory=dict) dataset_training: dict[str, ResolvedDatasetTrainingSpec] - training_plan: list[ResolvedTrainingPhase] + authored: SequifierConfig | None = Field(default=None, exclude=True) + initialize_from: str | None = None + actor_initialization: Literal["attach", "history_only"] = "attach" + model_path: str = "" + report_path: str = "" + phase_directory: str = "" + sequence_sources: dict[str, SequenceSourceConfig] = Field(default_factory=dict) + preference_sources: dict[str, str] = Field(default_factory=dict) + policy_bindings: dict[str, ResolvedPolicyBinding] = Field(default_factory=dict) + training_plan: list[ResolvedTrainingPhase | PolicyPhase] curriculum_training: bool = False curriculum_column: Optional[str] = None - evaluation_sources: list[ResolvedTrainingSource] - evaluation_monitor: Optional[EvaluationMonitorSpecModel] - export_generative_model: bool - export_embedding_model: bool - embedding_layer_names: list[str] - export_onnx: bool - export_pt: bool - export_with_dropout: bool = False + publication: PublicationPlan - @model_validator(mode="after") - @beartype - def validate_next_occurrence_metadata(self): - objective = get_objective_class(self.global_training.training_objective) - if not issubclass(objective, NextOccurrenceObjective): - return self + @property + def evaluation_sources(self) -> list[ResolvedTrainingSource]: + """All supervised evaluation dependencies, for loader and metric setup.""" + return list( + { + source.source: source + for phase in self.training_plan + if phase.algorithm == "supervised" + for source in phase.evaluation_sources + }.values() + ) - next_config = self.global_training.next_occurrence_config - if next_config is None: - raise ValueError( - "next_occurrence_config must be set for next_occurrence training" - ) - column = next_config.column_name - for dataset_name, dataset in self.dataset_training.items(): - interface = dataset.interface - if column not in interface.target_columns: - raise ValueError( - "next_occurrence_config.column_name must be one of " - f"target_columns for dataset {dataset_name!r}, got {column!r}" - ) - if interface.target_column_types.get(column) != "categorical": - raise ValueError( - "next_occurrence_config.column_name must refer to a " - "categorical target column" - ) - if column not in interface.id_maps: - raise ValueError( - "next_occurrence_config.column_name must have a " - f"preprocessing id_map, got {column!r}" - ) - missing = [ - value - for value in next_config.target_values - if value not in interface.id_maps[column] - ] - if missing: + @property + def embedding_layer_names(self) -> list[str]: + return self.publication.exports.embedding_layer_names + + def validate_next_occurrence_phases(self): + objective = get_objective_class(self.runtime_settings.training_objective) + is_next = issubclass(objective, NextOccurrenceObjective) + for phase in self.training_plan: + if phase.algorithm != "supervised": + continue + next_config = phase.learning.next_occurrence_config + if (next_config is not None) != is_next: raise ValueError( - "next_occurrence_config.target_values must match keys in " - f"id_maps[{column!r}] exactly, missing {missing!r}" + f"{phase.name}: learning.next_occurrence_config must be set " + "exactly for next_occurrence training" ) - return self + if next_config is None: + continue + column = next_config.column_name + names = { + source.dataset for source in phase.sources + phase.evaluation_sources + } + for dataset_name in names: + dataset = self.dataset_training[dataset_name] + interface = self.interface(dataset.model_interface) + if column not in interface.target_columns: + raise ValueError( + f"{phase.name}: next_occurrence_config.column_name must be " + f"a target column for dataset {dataset_name!r}, got {column!r}" + ) + if interface.target_column_types.get(column) != "categorical": + raise ValueError( + f"{phase.name}: next_occurrence_config.column_name must refer " + "to a categorical target column" + ) + if column not in interface.id_maps: + raise ValueError( + f"{phase.name}: next_occurrence_config.column_name must have " + f"a preprocessing id_map, got {column!r}" + ) + missing = [ + value + for value in next_config.target_values + if value not in interface.id_maps[column] + ] + if missing: + raise ValueError( + f"{phase.name}: next_occurrence_config.target_values must " + f"match keys in id_maps[{column!r}], missing {missing!r}" + ) @model_validator(mode="after") @beartype def validate_model_execution_plans(self): """Compile each route while validation errors still retain Pydantic context.""" - from sequifier.model.decoders import resolve_decoding_plan - from sequifier.model.ingestion_compiler import resolve_ingestion_plan + from sequifier.model.execution import ModelExecution - validated_interfaces: set[str] = set() + for name, interface in self.interfaces.items(): + if name not in self.model.interfaces or name != interface.name: + raise ValueError(f"Invalid resolved model interface {name!r}") for dataset in self.dataset_training.values(): - if dataset.model_interface in validated_interfaces: - continue - if dataset.interface.depth_layouts: - if self.global_training.read_format != "pt": - raise ValueError( - "Preprocessed depth inputs require read_format: pt" - ) - view = interface_build_view(self, dataset.interface) - resolve_ingestion_plan(view) - resolve_decoding_plan(view) - validated_interfaces.add(dataset.model_interface) + if dataset.model_interface not in self.interfaces: + raise ValueError( + f"Dataset {dataset.name!r} references unknown interface " + f"{dataset.model_interface!r}" + ) + if self.interface(dataset.model_interface).depth_layouts and any( + part.read_format != "pt" for part in dataset.parts.values() + ): + raise ValueError("Preprocessed depth inputs require read_format: pt") + self.validate_next_occurrence_phases() + ModelExecution.from_config(self).validate() return self @property @@ -1335,7 +997,7 @@ def dataset_count(self) -> int: @property @beartype def interface_names(self) -> tuple[str, ...]: - return tuple(self.model.interfaces) + return tuple(self.interfaces) @beartype def dataset(self, name: Optional[str] = None) -> ResolvedDatasetTrainingSpec: @@ -1348,37 +1010,54 @@ def dataset(self, name: Optional[str] = None) -> ResolvedDatasetTrainingSpec: @beartype def interface(self, name: Optional[str] = None) -> ResolvedModelInterface: if name is None: - names = { - dataset.model_interface for dataset in self.dataset_training.values() - } - if len(names) != 1: + if len(self.interfaces) != 1: raise ValueError("A model interface selection is required") - name = next(iter(names)) - for dataset in self.dataset_training.values(): - if dataset.model_interface == name: - return dataset.interface - raise KeyError(name) + name = next(iter(self.interfaces)) + return self.interfaces[name] + + def dataset_interface(self, name: str) -> ResolvedModelInterface: + dataset = self.dataset_training[name] + interface = self.interface(dataset.model_interface) + if not dataset.parts: + return interface + metadata = next(iter(dataset.parts.values())).metadata + return interface.model_copy( + update={ + "image_sources": { + view: metadata.image_sources[view] + for view in interface.image_sources + }, + "preconditioning_image": metadata.preconditioning_image + if interface.preconditioning_image_encoder is not None + else None, + } + ) + + def dataset_storage(self, name: str) -> ResolvedInterfaceStorage: + dataset = self.dataset_training[name] + return dataset.storage or self.interface_storage[dataset.model_interface] @dataclass(frozen=True) class LoadedTrainConfig: - config: SequifierConfig + config: Any resolved: ResolvedSequifierConfig metadata: dict[str, DatasetMetadata] @beartype def _source( - source: str, accumulation_steps: Optional[int], **values: Any + source: str, datasets: dict, accumulation_batches: Optional[int], **values: Any ) -> ResolvedTrainingSource: dataset, _, part = source.partition(".") return ResolvedTrainingSource( source=source, dataset=dataset, part=part or None, - weight=values.get("weight") or 1.0, + parts=(part,) if part else tuple(datasets[dataset].parts), + selection_weight=values.get("selection_weight") or 1.0, batches_per_selection=( - values.get("batches_per_selection") or accumulation_steps or 1 + values.get("batches_per_selection") or accumulation_batches or 1 ), ) @@ -1408,23 +1087,6 @@ def _layout_for_part_path( return metadata.storage_layout -@beartype -def _evaluated_parts(config: SequifierConfig) -> set[str]: - selected: set[str] = set() - if config.evaluation is None: - return selected - for source in config.evaluation.sources: - dataset_name, _, part_name = source.source.partition(".") - if part_name: - selected.add(source.source) - else: - selected.update( - f"{dataset_name}.{name}" - for name in config.dataset_training[dataset_name].parts - ) - return selected - - @beartype def _part_signature( metadata: DatasetMetadata, interface: ModelInterfaceSpecModel @@ -1576,9 +1238,12 @@ def _resolve_interface( name: str, spec: ModelInterfaceSpecModel, metadata: DatasetMetadata, - global_spec: GlobalTrainingSpecModel, - storage_layout: StoredWindowLayout, + global_spec: TrainingRuntimeSettings, ) -> ResolvedModelInterface: + if global_spec.training_objective is None or global_spec.context_length is None: + raise ValueError( + "resolved training requires training_objective and context_length" + ) if any(metadata.depth_layouts.is_deep_column(c) for c in spec.target_columns): raise ValueError("Depth targets are not supported") signature = _part_signature(metadata, spec) @@ -1666,8 +1331,7 @@ def _resolve_interface( if global_id in collision_ids } raise ValueError( - f"categorical_hashing[{column!r}]: {error} " - f"(category labels: {labels})" + f"categorical_hashing[{column!r}]: {error} (category labels: {labels})" ) from error hash_contracts[column] = codec.contract() generated_metric_names = { @@ -1687,12 +1351,12 @@ def _resolve_interface( objective=global_spec.training_objective, target_offset=target_offset, ) - resolve_window_view(storage_layout, window_view) return ResolvedModelInterface( name=name, + policy=None, input_columns=spec.input_columns, + prediction_distributions=spec.prediction_distributions, static_conditioning=spec.static_conditioning, - static_features_path=metadata.static_features_path, image_sources={ name: metadata.image_sources[name] for name in sorted(image_views(spec.ingestion)) @@ -1717,13 +1381,6 @@ def _resolve_interface( for column, tokens in categorical_decoder_special_tokens.items() }, depth_layouts=metadata.depth_layouts.relevant_layouts(spec.input_columns), - tensor_payload_version=metadata.tensor_payload_version, - split_paths=metadata.split_paths, - prediction_aligned_splits=metadata.prediction_aligned_splits, - prediction_alignment_steps=metadata.prediction_alignment_steps, - window_lengths=metadata.window_lengths, - prediction_length=metadata.prediction_length, - target_offset=metadata.target_offset, feature_layout=spec.feature_layout, ingestion=spec.ingestion, decoder=spec.decoder, @@ -1745,12 +1402,12 @@ def _resolve_interface( if target_types[target] == "real" ] ) + if column in metadata.selected_columns_statistics }, normalize_real_columns=metadata.normalize_real_columns, target_decoder_ids=target_decoder_ids, target_n_classes=target_n_classes, target_global_to_decoder=target_global_to_decoder, - storage_layout=storage_layout, window_view=window_view, ) @@ -1764,14 +1421,6 @@ def _interface_semantics(interface: ResolvedModelInterface) -> dict[str, Any]: "ingestion", "decoder", "feature_layout", - "tensor_payload_version", - "split_paths", - "prediction_aligned_splits", - "prediction_alignment_steps", - "window_lengths", - "prediction_length", - "target_offset", - "static_features_path", "image_sources", "preconditioning_image", }, @@ -1800,44 +1449,41 @@ def _interface_semantics(interface: ResolvedModelInterface) -> dict[str, Any]: @beartype -def resolve_sequifier_config( - config: SequifierConfig, - metadata: DatasetMetadata | dict[str, DatasetMetadata], - *, - part_overrides: Optional[dict[str, dict[str, str]]] = None, -) -> ResolvedSequifierConfig: - """Resolve every dataset part, compatibility contract, and source.""" - - part_refs = [ - f"{dataset_name}.{part_name}" - for dataset_name, dataset in config.dataset_training.items() - for part_name in dataset.parts - ] - if isinstance(metadata, DatasetMetadata): - if len(part_refs) != 1: - raise ValueError( - "Metadata must be keyed by dataset.part for multiple parts" - ) - metadata_by_part = {part_refs[0]: metadata} - else: - metadata_by_part = metadata - missing_metadata = set(part_refs) - set(metadata_by_part) - if missing_metadata: - raise ValueError(f"Missing metadata for parts: {sorted(missing_metadata)}") - - evaluated = _evaluated_parts(config) - overrides = part_overrides or {} +def resolve_supervised_datasets( + config, metadata_by_part, phases, dataset_names, *, metadata_from_file: bool +): + """Resolve prepared dataset parts and their model compatibility contracts.""" + + evaluated = { + f"{source.dataset}.{part}" + for phase in phases + if phase.algorithm == "supervised" + for source in phase.evaluation_sources + for part in source.parts + } resolved_datasets: dict[str, ResolvedDatasetTrainingSpec] = {} - interface_semantics: dict[str, dict[str, Any]] = {} + interfaces = {} + interface_storage = {} + dataset_semantics: dict[str, dict[str, Any]] = {} for dataset_name, dataset_spec in config.dataset_training.items(): - interface_spec = config.model.interfaces[dataset_spec.model_interface] + if dataset_name not in dataset_names: + continue + learning = config.dataset_learning(dataset_name) + if dataset_spec.model_interface is None: + raise ValueError(f"Dataset {dataset_name!r} requires a model interface") + interface_name = dataset_spec.model_interface + interface_spec = config.model.interfaces[interface_name] resolved_parts = {} expected_signature = None expected_form = None first_metadata = None for part_name, part_spec in dataset_spec.parts.items(): ref = f"{dataset_name}.{part_name}" + if not isinstance(part_spec, DatasetPartSpecModel): + raise ValueError(f"{ref}: supervised sources require prepared parts") + if ref not in metadata_by_part: + raise ValueError(f"Missing metadata for part: {ref}") part_metadata = metadata_by_part[ref] signature = _part_signature(part_metadata, interface_spec) if expected_signature is None: @@ -1847,22 +1493,17 @@ def resolve_sequifier_config( _assert_compatible(expected_signature, signature, ref) split_paths = list(part_metadata.split_paths) - override = overrides.get(ref, {}) - training_path = override.get("data_path") or ( - split_paths[0] if split_paths else None - ) + training_path = split_paths[0] if split_paths else None if training_path is None: raise ValueError(f"Training part {ref!r} needs at least one split path") - validation_path = override.get("validation_data_path") or ( - split_paths[1] if len(split_paths) > 1 else None - ) + validation_path = split_paths[1] if len(split_paths) > 1 else None if ref in evaluated and validation_path is None: raise ValueError( f"Evaluation part {ref!r} requires a second split path" ) - training_path = normalize_path(training_path, config.project_root) + training_path = normalize_path(training_path, config.run.project_root) validation_path = ( - normalize_path(validation_path, config.project_root) + normalize_path(validation_path, config.run.project_root) if validation_path is not None else None ) @@ -1879,13 +1520,12 @@ def resolve_sequifier_config( ) resolved_parts[part_name] = ResolvedDatasetPart( name=part_name, - metadata_config_path=( - override.get("metadata_config_path") - or part_spec.metadata_config_path - ), + read_format=part_spec.read_format, + metadata_config_path=part_spec.metadata_config_path, metadata=part_metadata, - image_path=part_spec.image_path, - preconditioning_image_path=part_spec.preconditioning_image_path, + metadata_from_file=metadata_from_file, + image_paths=part_spec.image_paths, + preconditioning_image_paths=part_spec.preconditioning_image_paths, training_data_path=training_path, validation_data_path=validation_path, storage_form=form, @@ -1897,7 +1537,10 @@ def resolve_sequifier_config( interface_spec.target_columns, first_metadata.column_data_types, ) - for column in dataset_spec.class_share_log_columns: + learning.validate_targets( + interface_spec.target_columns, f"Dataset {dataset_name!r}" + ) + for column in learning.class_share_log_columns: if column not in interface_spec.target_columns: raise ValueError(f"Class-share column {column!r} must be a target") if target_types[column] != "categorical": @@ -1907,11 +1550,11 @@ def resolve_sequifier_config( if column not in first_metadata.id_maps: raise ValueError(f"Class-share column {column!r} needs an id_map") view = ModelWindowView( - context_length=config.global_training.context_length, - objective=config.global_training.training_objective, + context_length=config.runtime_settings.context_length, + objective=config.runtime_settings.training_objective, target_offset=target_offset_for_objective( - config.global_training.training_objective, - config.global_training.target_offset, + config.runtime_settings.training_objective, + config.runtime_settings.target_offset, ), ) first_training_layout: StoredWindowLayout | None = None @@ -1922,18 +1565,20 @@ def resolve_sequifier_config( selected_paths.append(resolved_part.validation_data_path) for path in selected_paths: layout = _layout_for_part_path( - resolved_part.metadata, path, config.project_root + resolved_part.metadata, path, config.run.project_root ) resolve_window_view(layout, view) if first_training_layout is None: first_training_layout = layout assert first_training_layout is not None interface = _resolve_interface( - dataset_spec.model_interface, + interface_name, interface_spec, first_metadata, - config.global_training, - first_training_layout, + config.runtime_settings, + ) + storage = ResolvedInterfaceStorage.from_metadata( + first_metadata, first_training_layout ) for part_name, resolved_part in resolved_parts.items(): ref = f"{dataset_name}.{part_name}" @@ -1954,7 +1599,7 @@ def resolve_sequifier_config( f"window width at least {required_aligned_width}." ) normalized_splits = [ - normalize_path(path, config.project_root) + normalize_path(path, config.run.project_root) for path in aligned_metadata.split_paths ] aligned_paths = { @@ -1981,18 +1626,23 @@ def resolve_sequifier_config( f"{aligned_metadata.prediction_length}, but the selected interface " f"uses {interface.decoder.prediction_length}." ) - semantic_contract = _interface_semantics(interface) - prior_contract = interface_semantics.get(dataset_spec.model_interface) + semantic_contract = { + **_interface_semantics(interface), + "storage_layout": first_training_layout, + } + prior_contract = dataset_semantics.get(interface_name) if prior_contract is not None: _assert_compatible( prior_contract, semantic_contract, - f"Datasets sharing interface {dataset_spec.model_interface!r}", + f"Datasets sharing interface {interface_name!r}", ) else: - interface_semantics[dataset_spec.model_interface] = semantic_contract + dataset_semantics[interface_name] = semantic_contract + interfaces.setdefault(interface_name, interface) + interface_storage.setdefault(interface_name, storage) - for target, criterion in dataset_spec.criterion.items(): + for target, criterion in learning.criterion.items(): if ( criterion == "AccurateCrossEntropyLoss" and interface.target_column_types[target] != "categorical" @@ -2001,8 +1651,8 @@ def resolve_sequifier_config( f"AccurateCrossEntropyLoss requires categorical target {target!r}." ) - if dataset_spec.class_weights is not None: - for column, weights in dataset_spec.class_weights.items(): + if learning.class_weights is not None: + for column, weights in learning.class_weights.items(): if column in interface.categorical_hashing: raise ValueError( f"class_weights[{column!r}] describes canonical classes, but hash heads " @@ -2023,7 +1673,7 @@ def resolve_sequifier_config( ) for column in interface.categorical_hashing: - if column in interface.target_columns and dataset_spec.criterion[ + if column in interface.target_columns and learning.criterion[ column ] not in {"CrossEntropyLoss", "AccurateCrossEntropyLoss"}: raise ValueError( @@ -2033,84 +1683,46 @@ def resolve_sequifier_config( resolved_datasets[dataset_name] = ResolvedDatasetTrainingSpec( name=dataset_name, - model_interface=dataset_spec.model_interface, - interface=interface, + model_interface=interface_name, + storage=storage, parts=resolved_parts, - criterion=dataset_spec.criterion, - class_weights=dataset_spec.class_weights, - loss_weights=dataset_spec.loss_weights, - class_share_log_columns=dataset_spec.class_share_log_columns, - freeze=dataset_spec.freeze, + criterion=learning.criterion, + class_weights=learning.class_weights, + loss_weights=learning.loss_weights, + class_share_log_columns=learning.class_share_log_columns, ) - accumulation = config.global_training.accumulation_steps - phases = [ - ResolvedTrainingPhase( - name=phase.name, - epochs=phase.epochs, - mode=phase.mode, - selection=phase.selection or "round_robin", - sources=[ - _source( - source.source, - accumulation, - weight=source.weight, - batches_per_selection=( - None - if phase.mode == "sequential" - else source.batches_per_selection - ), - ) - for source in phase.sources - ], - ) - for phase in config.training_plan.phases - ] - evaluation_sources = ( - [_source(source.source, accumulation) for source in config.evaluation.sources] - if config.evaluation is not None - else [] - ) - resolved = ResolvedSequifierConfig( - project_root=config.project_root, - model_name=config.model_name, - device=config.device, - seed=config.seed, - global_training=config.global_training, - model=config.model, - dataset_training=resolved_datasets, - training_plan=phases, - curriculum_training=config.training_plan.curriculum_training, - curriculum_column=config.training_plan.curriculum_column, - evaluation_sources=evaluation_sources, - evaluation_monitor=( - config.evaluation.monitor if config.evaluation is not None else None - ), - export_generative_model=config.export_generative_model, - export_embedding_model=config.export_embedding_model, - embedding_layer_names=config.embedding_layer_names, - export_onnx=config.export_onnx, - export_pt=config.export_pt, - export_with_dropout=config.export_with_dropout, - ) - return resolved + return resolved_datasets, interfaces, interface_storage + + +def resolve_sequifier_config( + config: SequifierConfig, + metadata: DatasetMetadata | dict[str, DatasetMetadata], +) -> ResolvedSequifierConfig: + """Compile a training plan with caller-provided dataset metadata.""" + from sequifier.config.training_resolution import resolve_training_run + + if isinstance(metadata, DatasetMetadata): + parts = [ + f"{name}.{part}" + for name, dataset in config.dataset_training.items() + for part in dataset.parts + ] + if len(parts) != 1: + raise ValueError( + "Metadata must be keyed by dataset.part for multiple parts" + ) + metadata = {parts[0]: metadata} + return resolve_training_run(config, metadata_by_part=metadata) -_SENSITIVE_OVERRIDES = { - "data_path", - "validation_data_path", - "metadata_config_path", - "preprocessing_data_path", - "input_columns", -} _INLINE_METADATA_KEYS = { "depth_layouts", "image_sources", "preconditioning_image", - "tensor_payload_version", + "tensor_payload_kind", "metadata_by_part", "column_data_types", - "column_types", "static_columns", "static_features_path", "n_classes", @@ -2121,7 +1733,6 @@ def resolve_sequifier_config( "window_length", "window_lengths", "max_target_offset", - "stored_window_layout_version", "storage_layout", "split_paths", "prediction_aligned_splits", @@ -2131,62 +1742,35 @@ def resolve_sequifier_config( } -@beartype -def _override_part(config: SequifierConfig) -> str: - if len(config.dataset_training) != 1: - raise ValueError( - "Dataset-sensitive legacy CLI overrides require exactly one dataset" - ) - dataset_name, dataset = next(iter(config.dataset_training.items())) - if len(dataset.parts) == 1: - return f"{dataset_name}.{next(iter(dataset.parts))}" - refs = { - source.source - for phase in config.training_plan.phases - for source in phase.sources - if source.source.startswith(f"{dataset_name}.") - } - if config.evaluation is not None: - refs.update( - source.source - for source in config.evaluation.sources - if source.source.startswith(f"{dataset_name}.") - ) - if len(refs) == 1: - return next(iter(refs)) - raise ValueError( - "Legacy metadata/data-path CLI override is ambiguous because multiple " - "dataset parts remain possible" - ) - - @beartype def _inline_metadata( - values: dict[str, Any], global_spec: GlobalTrainingSpecModel + values: dict[str, Any], global_spec: TrainingRuntimeSettings ) -> DatasetMetadata: + validate_authored_metadata(values) layout = values.get("storage_layout") if isinstance(layout, StoredWindowLayout): window_length = layout.window_length max_target_offset = layout.max_target_offset - layout_version = layout.version elif isinstance(layout, dict): window_length = layout.get("window_length") max_target_offset = layout.get("max_target_offset", 1) - layout_version = layout.get("version", 2) else: window_length = values.get("window_length") max_target_offset = values.get("max_target_offset", 1) - layout_version = values.get("stored_window_layout_version", 2) if window_length is None: + if global_spec.context_length is None: + raise ValueError("inline metadata requires context_length or window_length") window_length = global_spec.context_length + max(1, global_spec.target_offset) return DatasetMetadata.model_validate( { "depth_layouts": values.get("depth_layouts", {}), - "tensor_payload_version": values.get("tensor_payload_version", 1), - "split_paths": values.get("split_paths", []), - "column_data_types": values.get( - "column_data_types", values.get("column_types", {}) + **( + {"tensor_payload_kind": values["tensor_payload_kind"]} + if "tensor_payload_kind" in values + else {} ), + "split_paths": values.get("split_paths", []), + "column_data_types": values.get("column_data_types", {}), "static_columns": values.get("static_columns", []), "static_features_path": values.get("static_features_path"), "image_sources": values.get("image_sources", {}), @@ -2207,7 +1791,6 @@ def _inline_metadata( "target_offset": values.get("target_offset"), "window_length": window_length, "max_target_offset": max_target_offset, - "stored_window_layout_version": layout_version, } ) @@ -2228,71 +1811,49 @@ def load_train_config_with_source( metadata_inline_values = { key: args.pop(key) for key in list(args) if key in _INLINE_METADATA_KEYS } - sensitive = { - key: args.pop(key) for key in list(args) if key in _SENSITIVE_OVERRIDES + from sequifier.config.command_ownership import apply_run_overrides + + raw, args = apply_run_overrides(raw, args) + from sequifier.config.composition import merge_config_fragments + + overrides = { + key: args.pop(key) for key in ("run", "model", "training_plan") if key in args } - for key in ("model_name", "seed", "device"): - if key in args: - raw[key] = args.pop(key) + raw = merge_config_fragments((raw, overrides)) if args: raise ValueError(f"Unsupported training CLI overrides: {sorted(args)}") - config = try_catch_excess_keys(config_path, SequifierConfig, raw) - part_overrides: dict[str, dict[str, str]] = {} - if sensitive: - selected_part = _override_part(config) - dataset_name = selected_part.partition(".")[0] - if "input_columns" in sensitive: - interface_name = config.dataset_training[dataset_name].model_interface - config.model.interfaces[interface_name].input_columns = sensitive.pop( - "input_columns" - ) - override = {} - if "preprocessing_data_path" in sensitive: - override["metadata_config_path"] = ( - metadata_config_path_from_preprocessing_data_path( - sensitive.pop("preprocessing_data_path") - ) - ) - override.update(sensitive) - part_overrides[selected_part] = override + authored = try_catch_excess_keys(config_path, SequifierConfig, raw) + from sequifier.config.training_resolution import resolve_training_run metadata_by_part: dict[str, DatasetMetadata] = {} - all_part_refs = [ - (dataset_name, part_name, part) - for dataset_name, dataset in config.dataset_training.items() - for part_name, part in dataset.parts.items() - ] if skip_metadata: - if len(all_part_refs) != 1 and "metadata_by_part" not in metadata_inline_values: - raise ValueError( - "skip_metadata with multiple parts requires metadata_by_part" - ) + parts = [ + f"{name}.{part}" + for name, dataset in authored.dataset_training.items() + for part in dataset.parts + ] if "metadata_by_part" in metadata_inline_values: - for ref, values in metadata_inline_values["metadata_by_part"].items(): - metadata_by_part[ref] = _inline_metadata(values, config.global_training) - else: - dataset_name, part_name, _ = all_part_refs[0] - metadata_by_part[f"{dataset_name}.{part_name}"] = _inline_metadata( - metadata_inline_values, config.global_training - ) - else: - for dataset_name, part_name, part in all_part_refs: - ref = f"{dataset_name}.{part_name}" - configured_path = part_overrides.get(ref, {}).get( - "metadata_config_path", part.metadata_config_path + metadata_by_part = { + ref: _inline_metadata(values, authored.runtime_settings) + for ref, values in metadata_inline_values["metadata_by_part"].items() + } + elif len(parts) == 1: + metadata_by_part[parts[0]] = _inline_metadata( + metadata_inline_values, authored.runtime_settings ) - if configured_path is None: - raise ValueError(f"Part {ref!r} has no metadata_config_path") - metadata_by_part[ref] = load_dataset_metadata( - normalize_path(configured_path, config.project_root) + else: + raise ValueError( + "skip_metadata with multiple parts requires metadata_by_part" ) - resolved = resolve_sequifier_config( - config, metadata_by_part, part_overrides=part_overrides + resolved = resolve_training_run( + authored, + metadata_by_part=metadata_by_part if skip_metadata else None, + loaded_metadata=metadata_by_part, ) return LoadedTrainConfig( - config=config, resolved=resolved, metadata=metadata_by_part + config=authored, resolved=resolved, metadata=metadata_by_part ) @@ -2320,11 +1881,9 @@ class SelectedInterfaceConfig: model_name: str device: str seed: int - global_training: GlobalTrainingSpecModel training_objective: str input_columns: list[str] static_columns: list[str] - static_features_path: str | None image_sources: dict[str, ImageSourceConfig] preconditioning_image: PreconditioningImageConfig | None preconditioning_image_encoder: ImageEncoderConfig | None @@ -2338,22 +1897,29 @@ class SelectedInterfaceConfig: categorical_decoder_special_tokens: dict[str, list[str]] feature_layout: Optional[FeatureLayoutRegistryModel] depth_layouts: DepthLayoutRegistryModel - tensor_payload_version: int n_classes: dict[str, int] id_maps: dict[str, dict[str | int, int]] special_token_ids: dict[str, int] target_decoder_ids: dict[str, list[int]] target_n_classes: dict[str, int] target_global_to_decoder: dict[str, list[int]] - storage_layout: StoredWindowLayout window_view: ModelWindowView model: SelectedModelComponents @dataclass(frozen=True) -class SelectedDatasetPartConfig(SelectedInterfaceConfig): +class SelectedObjectiveConfig(SelectedInterfaceConfig): + runtime_settings: TrainingRuntimeSettings + + +@dataclass(frozen=True) +class SelectedDatasetPartConfig(SelectedObjectiveConfig): """Explicit dataset-part configuration used by data loaders.""" + static_features_path: str | None + tensor_payload_version: int + storage_layout: StoredWindowLayout + batch_size: int dataset_training: ResolvedDatasetTrainingSpec part: ResolvedDatasetPart curriculum_training: bool @@ -2368,7 +1934,8 @@ class SelectedDatasetPartConfig(SelectedInterfaceConfig): @beartype def interface_build_view( - config: ResolvedSequifierConfig, interface: ResolvedModelInterface + config: Any, + interface: ResolvedModelInterface, ) -> SelectedInterfaceConfig: """Select one interface for model and objective construction.""" @@ -2377,15 +1944,13 @@ def interface_build_view( model_name=config.model_name, device=config.device, seed=config.seed, - global_training=config.global_training, - training_objective=config.global_training.training_objective, - input_columns=interface.input_columns, + training_objective=interface.window_view.objective, static_columns=( interface.static_conditioning.columns if interface.static_conditioning else [] ), - static_features_path=interface.static_features_path, + input_columns=interface.input_columns, image_sources=interface.image_sources, preconditioning_image=interface.preconditioning_image, preconditioning_image_encoder=interface.preconditioning_image_encoder, @@ -2397,16 +1962,14 @@ def interface_build_view( categorical_columns=interface.categorical_columns, real_columns=interface.real_columns, categorical_decoder_special_tokens=interface.categorical_decoder_special_tokens, - depth_layouts=interface.depth_layouts, - tensor_payload_version=interface.tensor_payload_version, feature_layout=interface.feature_layout, + depth_layouts=interface.depth_layouts, n_classes=interface.n_classes, id_maps=interface.id_maps, special_token_ids=interface.special_token_ids, target_decoder_ids=interface.target_decoder_ids, target_n_classes=interface.target_n_classes, target_global_to_decoder=interface.target_global_to_decoder, - storage_layout=interface.storage_layout, window_view=interface.window_view, model=SelectedModelComponents( backbone=config.model.backbone, @@ -2421,14 +1984,19 @@ def dataset_part_view( config: ResolvedSequifierConfig, dataset_name: str, part_name: str, + *, + batch_size: int, + window_stride: int | None, + curriculum_column: str | None = None, ) -> SelectedDatasetPartConfig: """Select one dataset part for data loading.""" dataset = config.dataset_training[dataset_name] part = dataset.parts[part_name] - interface_config = interface_build_view(config, dataset.interface) + interface_config = interface_build_view( + config, config.interface(dataset.model_interface) + ) interface_values = vars(interface_config).copy() - interface_values["static_features_path"] = part.metadata.static_features_path selected_sources = { name: part.metadata.image_sources[name] for name in interface_config.image_sources @@ -2442,20 +2010,29 @@ def dataset_part_view( bind_image_paths( selected_sources, selected_prefix, - part.image_path, - part.preconditioning_image_path, + part.image_paths, + part.preconditioning_image_paths, ) ) return SelectedDatasetPartConfig( + runtime_settings=config.runtime_settings, + static_features_path=part.metadata.static_features_path, + tensor_payload_version=part.metadata.tensor_payload_version, + storage_layout=config.dataset_storage(dataset_name).storage_layout, + batch_size=batch_size, **interface_values, dataset_training=dataset, part=part, - curriculum_training=config.curriculum_training, - curriculum_column=config.curriculum_column, - epochs=sum(phase.epochs for phase in config.training_plan), - read_format=config.global_training.read_format, - window_stride=config.global_training.window_stride, + curriculum_training=curriculum_column is not None, + curriculum_column=curriculum_column, + epochs=sum( + phase.epochs + for phase in config.training_plan + if isinstance(phase, (ResolvedTrainingPhase, PolicyImitationPhaseConfig)) + ), + read_format=part.read_format, + window_stride=window_stride, data_path=part.training_data_path, validation_data_path=part.validation_data_path, metadata_config_path=part.metadata_config_path, @@ -2463,3 +2040,230 @@ def dataset_part_view( TrainModel = ResolvedSequifierConfig + + +# Public authored API; native schemas above are resolved execution projections. +from sequifier.config.ownership import ( # noqa: E402 + InputContract, + ModelExecutionSettings, + RunSettings, + SplitSettings, +) + + +class OwnedModelInterfaceSpec(ModelInterfaceSpecModel): + input_contract: InputContract = Field(default_factory=InputContract) + + +class OwnedModelSpec(ConfigModel): + model_config = ConfigDict(extra="forbid") + backbone: BackboneComponentConfig | None = None + execution: ModelExecutionSettings = Field(default_factory=ModelExecutionSettings) + interfaces: dict[str, OwnedModelInterfaceSpec] = Field(default_factory=dict) + + @classmethod + def from_execution(cls, execution, declared=None, *, settings=None): + """Project model facts without discarding authored input assertions.""" + values = execution.model.model_dump() + values["execution"] = settings or ( + declared.execution.model_dump(exclude_unset=True) if declared else {} + ) + if declared: + for name, interface in declared.interfaces.items(): + if name in values["interfaces"]: + values["interfaces"][name]["input_contract"] = ( + interface.input_contract + ) + return cls.model_validate(values) + + def architecture_model(self): + if self.backbone is None: + raise ValueError("model backbone has not been resolved") + return ModelSpecModel.model_validate( + self.model_dump( + exclude={ + "execution": True, + "interfaces": {"__all__": {"input_contract"}}, + } + ) + ) + + +class DatasetTrainingSpecModel(ConfigModel): + model_config = ConfigDict(extra="forbid") + model_interface: str | None = None + parts: dict[str, DatasetPart] = Field(min_length=1) + split: SplitSettings = Field(default_factory=SplitSettings) + + +class SequifierConfig(ConfigModel): + _runtime_settings: TrainingRuntimeSettings | None = PrivateAttr(default=None) + """The authored training configuration, with one owner for each setting.""" + + model_config = ConfigDict(extra="forbid") + run: RunSettings + model: OwnedModelSpec | None = None + initialize_from: str | None = None + actor_initialization: Literal["attach", "history_only"] = "attach" + dataset_training: dict[str, DatasetTrainingSpecModel] = Field(default_factory=dict) + policy_bindings: dict[str, PolicyBinding] = Field(default_factory=dict) + training_plan: TrainingPlanModel + + @model_serializer(mode="wrap") + def serialize_actor_initialization(self, handler): + values = handler(self) + if self.actor_initialization == "attach": + values.pop("actor_initialization", None) + return values + + @model_validator(mode="before") + @classmethod + def apply_defaults(cls, values): + if not isinstance(values, dict): + return values + values = copy.deepcopy(values) + model = values.get("model") + if isinstance(model, ConfigModel): + model = values["model"] = model.model_dump(exclude_unset=True) + model = model or {} + if not isinstance(model, dict): + return values + interfaces = model.get("interfaces", {}) + backbone = model.get("backbone") + context = model.get("execution", {}).get("context_length") + if isinstance(backbone, dict) and context is not None: + architecture = backbone.get("architecture") + if isinstance(architecture, dict): + architecture.setdefault("max_context_length", context) + datasets = values.get("dataset_training", {}) + if not isinstance(datasets, dict): + return values + for name, dataset in datasets.items(): + if isinstance(dataset, ConfigModel): + dataset = datasets[name] = dataset.model_dump(exclude_unset=True) + if not isinstance(dataset, dict): + continue + if isinstance(interfaces, dict) and len(interfaces) == 1: + dataset.setdefault("model_interface", next(iter(interfaces))) + for part in dataset.get("parts", {}).values(): + if isinstance(part, dict): + part.setdefault("type", "prepared") + plan = values.get("training_plan", {}) + if isinstance(plan, ConfigModel): + plan = values["training_plan"] = plan.model_dump(exclude_unset=True) + if not isinstance(plan, dict): + return values + implicit = None + if len(datasets) == 1: + name, dataset = next(iter(datasets.items())) + parts = dataset.get("parts", {}) + implicit = f"{name}.{next(iter(parts))}" if len(parts) == 1 else name + for phase in plan.get("phases", []): + phase.setdefault("algorithm", "supervised") + if implicit and phase["algorithm"] in { + "supervised", + "policy_imitation", + "iql", + }: + phase.setdefault("sources", [{"source": implicit}]) + return values + + @model_validator(mode="after") + def validate_contract(self): + if self.actor_initialization == "history_only" and ( + not self.initialize_from + or self.model is None + or self.model.backbone is None + or not self.model.interfaces + or not self.policy_bindings + or self.training_plan.phases[0].algorithm == "supervised" + ): + raise ValueError( + "actor_initialization: history_only requires initialize_from, " + "a complete downstream model, policy_bindings and a policy first phase" + ) + if ( + self.model is None or self.model.backbone is None + ) and self.initialize_from is None: + raise ValueError("a run requires a complete model or initialize_from") + for name, dataset in self.dataset_training.items(): + _identifier(name, "Dataset name") + for part in dataset.parts: + _identifier(part, "Part name") + for phase in self.training_plan.phases: + refs = phase_sources(phase) + for ref in refs: + _validate_source(self, ref.source, f"phase {phase.name!r}") + if phase.algorithm == "supervised": + for name in {ref.source.partition(".")[0] for ref in refs}: + if name not in phase.learning.datasets: + raise ValueError( + f"{phase.name}: declare learning.datasets.{name}" + ) + interface_name = self.dataset_training[name].model_interface + if self.model and interface_name in self.model.interfaces: + phase.learning.datasets[name].validate_targets( + self.model.interfaces[interface_name].target_columns, + f"{phase.name}: learning.datasets.{name}", + ) + return self + + def _build_runtime_settings(self): + """Resolve execution settings from their current owners.""" + run = self.run + execution = run.execution + contract = ( + self.model.execution.model_dump(exclude_unset=True) if self.model else {} + ) + training = { + **contract, + **execution.precision.model_dump(), + **execution.compilation.model_dump(), + **execution.data_loading.model_dump(), + **run.logging.model_dump(), + "max_concat_tensors": execution.max_concat_tensors, + "distributed": execution.distribution.strategy != "local", + "data_parallelism": None + if execution.distribution.strategy == "local" + else execution.distribution.strategy, + "world_size": execution.distribution.world_size, + "backend": execution.distribution.backend, + "fsdp_cpu_offload": execution.distribution.cpu_offload, + "resume": run.recovery.resume.model_dump(), + "latest_checkpoint_interval_minutes": run.recovery.latest_checkpoint_interval_minutes, + "enforce_determinism": run.reproducibility.enforce_determinism, + "inference_batch_size": run.export.batch_size, + } + supervised = [ + phase + for phase in self.training_plan.phases + if phase.algorithm == "supervised" + ] + if supervised: + training.update( + supervised[0].learning.model_dump( + exclude={"datasets"}, exclude_none=True + ) + ) + return TrainingRuntimeSettings.model_validate(training) + + @property + def runtime_settings(self): + if self._runtime_settings is None: + self._runtime_settings = self._build_runtime_settings() + return self._runtime_settings + + @runtime_settings.setter + def runtime_settings(self, value): + self._runtime_settings = value + + def dataset_learning(self, name): + for phase in self.training_plan.phases: + if phase.algorithm == "supervised" and name in phase.learning.datasets: + return phase.learning.datasets[name] + from sequifier.config.learning_config import DatasetLearningConfig + + return DatasetLearningConfig() + + +ResolvedSequifierConfig.model_rebuild() diff --git a/src/sequifier/config/training_resolution.py b/src/sequifier/config/training_resolution.py new file mode 100644 index 000000000..c4b53b4e3 --- /dev/null +++ b/src/sequifier/config/training_resolution.py @@ -0,0 +1,781 @@ +"""Resolve the common authored plan into model, dataset and task contracts.""" + +from pathlib import Path + +from sequifier.config.metadata import ( + DatasetMetadata, + dataset_tensor_kind, + dataset_tensor_version, + load_dataset_metadata, +) +from sequifier.config.naming import resolve_run_seed +from sequifier.config.policy_train_config import ( + SequenceSourceConfig, + resolve_sequence_source, +) +from sequifier.config.run_capabilities import publication_plan, validate_runner_settings +from sequifier.config.sources import phase_sources +from sequifier.config.train_config import ( + DatasetPartSpecModel, + OwnedModelSpec, + PreferencePartSpecModel, + ResolvedInterfaceStorage, + ResolvedSequifierConfig, + SequencePartSpecModel, + SequifierConfig, + TrainingRuntimeSettings, + _assert_compatible, + _interface_semantics, + _resolve_interface, + resolve_phases, + resolve_supervised_datasets, + validate_training_contract, +) +from sequifier.special_tokens import SPECIAL_TOKEN_IDS + + +def _normalization(config, name): + if config.model and name in config.model.interfaces: + return config.model.interfaces[name].input_contract.normalization + return None + + +def _source_fractions(config): + return { + f"{name}.{part}": spec.split.validation_fraction + for name, spec in config.dataset_training.items() + for part in spec.parts + } + + +def source_parts(config, reference): + """Expand a reference without silently filtering out incompatible parts.""" + dataset, _, part = reference.partition(".") + spec = config.dataset_training[dataset] + return [ + (f"{dataset}.{name}", value) + for name, value in spec.parts.items() + if not part or name == part + ] + + +class TrainingSources: + """Named authored parts shared by metadata fitting and run resolution.""" + + def __init__(self, config, root): + self.root = root + self.parts = { + f"{dataset_name}.{part_name}": (dataset.model_interface, part) + for dataset_name, dataset in config.dataset_training.items() + for part_name, part in dataset.parts.items() + } + + def resolve(self, metadata_by_part): + sequences, preferences = {}, {} + for ref, (_, part) in self.parts.items(): + if isinstance(part, PreferencePartSpecModel): + path = self.root / part.manifest_path + preferences[ref] = str(path.resolve()) + continue + if isinstance(part, SequencePartSpecModel): + source = SequenceSourceConfig.model_validate( + part.model_dump(exclude={"type"}) + ) + else: + source = SequenceSourceConfig( + metadata_config_path=part.metadata_config_path, + split_index=0, + format="prepared", + encoding="model", + ) + sequences[ref] = resolve_sequence_source( + source, self.root, metadata=metadata_by_part.get(ref) + ) + return sequences, preferences + + +def _resolve_phase_sources(config, phase, available): + """Expand named sources for both metadata fitting and final run resolution.""" + preference = phase.algorithm == "dpo" + if phase.algorithm in {"ppo", "grpo"}: + return + + def expand(refs): + expanded = [ + ref.model_copy(update={"source": full}) + for ref in refs + for full, _ in source_parts(config, ref.source) + ] + if not expanded or any(ref.source not in available for ref in expanded): + raise ValueError( + f"{phase.name}: select {'preference' if preference else 'sequence'} parts" + ) + return expanded + + if preference: + expanded = expand([phase.preferences]) + if len(expanded) != 1: + raise ValueError(f"{phase.name}: select one training preference part") + phase.preferences = expanded[0] + else: + phase.sources = expand(phase.sources) + if phase.evaluation.sources is not None: + phase.evaluation.sources = expand(phase.evaluation.sources) + + +def phase_interface(config, phase): + names = { + config.dataset_training[ref.source.partition(".")[0]].model_interface + for ref in phase_sources(phase) + } + explicit = phase.model_interface if phase.algorithm != "supervised" else None + if explicit is not None: + names.add(explicit) + if not names: + names = set(config.policy_bindings) or set(config.model.interfaces) + if len(names) != 1 or None in names: + raise ValueError( + f"training_plan.phases.{phase.name}: select one model_interface" + ) + name = next(iter(names)) + if name not in config.model.interfaces: + raise ValueError(f"unknown model interface {name!r}") + return name + + +def _inferred_metadata(config, name, binding): + """Identity numeric inputs and explicitly declared categorical action codecs.""" + interface = config.model.interfaces[name] + columns = set(interface.input_columns + interface.target_columns) + if interface.static_conditioning is not None: + columns.update(interface.static_conditioning.columns) + types = dict.fromkeys(columns, "Float32") + counts, maps = {}, {} + for column, action in binding.actions.items(): + if action.type == "categorical": + if not action.values: + raise ValueError( + f"{column}: fresh categorical targets need values or metadata" + ) + types[column] = "Int64" + counts[column] = SPECIAL_TOKEN_IDS.user_start + len(action.values) + maps[column] = { + str(label): SPECIAL_TOKEN_IDS.user_start + index + for index, label in enumerate(action.values) + } + for column, action in binding.previous_action_inputs.items(): + if action not in binding.actions: + raise ValueError( + f"{column}: previous_action_inputs references an unknown action" + ) + if action in maps: + labels = list(binding.actions[action].values) + initial = binding.initial_inputs.get(column) + if initial is not None and initial not in labels: + labels.append(initial) + maps[column] = { + str(label): SPECIAL_TOKEN_IDS.user_start + index + for index, label in enumerate(labels) + } + counts[column] = SPECIAL_TOKEN_IDS.user_start + len(labels) + types[column] = "Int64" + normalization = _normalization(config, name) + statistics = ( + normalization.statistics + if normalization and normalization.source == "statistics" + else {} + ) + return DatasetMetadata( + window_length=config.runtime_settings.context_length + + max(1, config.runtime_settings.target_offset), + column_data_types=types, + n_classes=counts, + id_maps=maps, + selected_columns_statistics={ + column: statistics.get(column, {"mean": 0.0, "std": 1.0}) + for column in columns + if column not in counts + and (normalization is not None or column not in interface.input_columns) + }, + ) + + +def _history_initialized_metadata(config, name, binding, source, storage): + """Resolve new targets while sourcing observation preprocessing from the artifact.""" + metadata = _inferred_metadata(config, name, binding) + static = source.static_conditioning + inputs = set(source.input_columns + (static.columns if static else [])) + targets = set(config.model.interfaces[name].target_columns) + for key, values, original in ( + ("column_data_types", metadata.column_data_types, source.column_data_types), + ("n_classes", metadata.n_classes, source.n_classes), + ("id_maps", metadata.id_maps, source.id_maps), + ( + "selected_columns_statistics", + metadata.selected_columns_statistics, + source.selected_columns_statistics, + ), + ): + for column in inputs: + if column in targets and column in values and column in original: + if ( + key != "selected_columns_statistics" + and values[column] != original[column] + ): + raise ValueError( + f"history transfer differs at {name}.{key}.{column}" + ) + values.pop(column, None) + if column in original: + values[column] = original[column] + tensor_kind = ( + storage.tensor_payload_kind + if storage + else dataset_tensor_kind(source.depth_layouts) + ) + return metadata.model_copy( + update={ + "static_columns": static.columns if static else [], + "static_features_path": storage.static_features_path if storage else None, + "image_sources": source.image_sources, + "preconditioning_image": source.preconditioning_image, + "special_token_ids": source.special_token_ids, + "depth_layouts": source.depth_layouts, + "tensor_payload_kind": tensor_kind, + "tensor_payload_version": ( + storage.tensor_payload_version + if storage + else dataset_tensor_version(tensor_kind) + ), + "normalize_real_columns": source.normalize_real_columns, + } + ) + + +def _fit_input_metadata( + config, + name, + binding, + metadata, + sources, + preferences, + source_fractions, + prepared_inputs, +): + """Fit missing raw codecs/moments on the same episode-safe training split.""" + from types import SimpleNamespace + + from sequifier.config.policy_inputs import PolicyFittingSpec, PolicyInputLayout + + phase = next( + ( + phase + for phase in config.training_plan.phases + if phase.algorithm != "supervised" and phase.model_interface == name + ), + None, + ) + if phase is None or phase.algorithm not in {"policy_imitation", "iql", "dpo"}: + return metadata + interface = config.model.interfaces[name] + normalization = _normalization(config, name) + static = ( + interface.static_conditioning.columns if interface.static_conditioning else [] + ) + layout = PolicyInputLayout.for_fitting( + interface.input_columns, static, binding, metadata + ) + observations, categorical = layout.observation_columns, layout.categorical_columns + missing_codecs = set(categorical) - metadata.n_classes.keys() + missing_statistics = ( + set(observations + static) + - set(categorical) + - metadata.selected_columns_statistics.keys() + ) + if not missing_codecs and (not missing_statistics or normalization is not None): + return metadata + provisional = PolicyFittingSpec( + **layout.binding_values(), + binding=binding, + context_length=config.runtime_settings.context_length, + interface_name=name, + ) + available = preferences if phase.algorithm == "dpo" else sources + if any(ref.source not in available for ref in phase_sources(phase)): + return metadata # Prepared dataset parts already own frozen preprocessing. + run = SimpleNamespace( + validation_fraction=0.2, + source_validation_fractions=source_fractions, + seed=config.run.reproducibility.seed, + preferences=preferences, + sources=sources, + episode_cache_size=config.runtime_settings.episode_cache_size, + ) + inputs = prepared_inputs.resolve(run) + inputs.revalidate(inputs.identity, f"interfaces.{name}.fitting_sources") + from sequifier.io.episode_features import EpisodeFeatures + + with prepared_inputs.fitting_phase(run, phase, provisional, inputs) as data: + fitting = data.fitting_dataset() + fitted = fitting.fit_categorical_config() + values = metadata.model_dump() + values["n_classes"].update(fitted.categorical_n_classes) + values["id_maps"].update(fitted.id_maps) + if ( + normalization is None + and not metadata.depth_layouts + and not metadata.image_sources + ): + features = EpisodeFeatures(fitting.episodes, fitted, fitting.lengths) + features.validate_features() + for column, statistics in features.normalization_statistics().items(): + values["selected_columns_statistics"].setdefault(column, statistics) + return DatasetMetadata.model_validate(values) + + +def resolve_training_run( + authored: SequifierConfig, + *, + metadata_by_part: dict[str, DatasetMetadata] | None = None, + loaded_metadata: dict[str, DatasetMetadata] | None = None, +) -> ResolvedSequifierConfig: + """Compile the authored run into the records consumed by both runners.""" + from sequifier.artifacts.model_artifact import read_execution_artifact + from sequifier.artifacts.model_config import model_execution_from_model_config + from sequifier.model.execution import ModelExecution + from sequifier.model.execution_options import ModelExecutionOptions + from sequifier.model.history_transfer import validate_history_transfer + from sequifier.rl.attachment import apply_binding, resolve_binding + from sequifier.rl.prepared_inputs import PreparedTrainingInputs + + prepared_inputs = PreparedTrainingInputs() + config = authored.model_copy(deep=True) + phases = config.training_plan.phases + supervised = [phase for phase in phases if phase.algorithm == "supervised"] + has_policy = len(supervised) != len(phases) + validate_runner_settings(config.run, supervised=bool(supervised), policy=has_policy) + seed = config.run.reproducibility.seed + if seed == "random": + from sequifier.artifacts.run_checkpoint import read_resume_seed + + saved_seed = read_resume_seed(config.run) + seed = resolve_run_seed(seed) if saved_seed is None else saved_seed + config.run.reproducibility.seed = seed + needs_policy_contracts = has_policy or bool( + config.policy_bindings or config.initialize_from + ) + supervised_names = { + ref.source.partition(".")[0] + for phase in supervised + for ref in phase_sources(phase) + } + if not needs_policy_contracts: + supervised_names.update(config.dataset_training) + if ( + len(supervised_names) > 1 + and config.run.execution.distribution.strategy == "fsdp" + ): + raise ValueError("Multi-dataset training does not support FSDP") + root = Path(config.run.project_root).resolve() + if needs_policy_contracts: + config.run.project_root = str(root) + training = config.runtime_settings + source = None + history_only = config.actor_initialization == "history_only" + if config.initialize_from: + config.initialize_from = str((root / config.initialize_from).resolve()) + source, _ = model_execution_from_model_config( + read_execution_artifact(config.initialize_from).model_config.values, + device=config.run.execution.device, + ) + if config.model is not None and not history_only: + from sequifier.artifacts.model_config import extend_prediction_distributions + + extend_prediction_distributions(source, config.model) + if not history_only: + config.model = OwnedModelSpec.from_execution(source, config.model) + incoming = source.options + resolved_fields = dict(training) + inherited = {} + for field, actual in ( + ("context_length", incoming.context_length), + ("training_objective", incoming.training_objective), + ("target_offset", incoming.target_offset), + ("forward_attention", incoming.forward_attention), + ): + if ( + history_only + and field in {"training_objective", "target_offset"} + and field in training.model_fields_set + ): + continue + if field in training.model_fields_set and resolved_fields[field] != actual: + raise ValueError( + f"model.execution.{field} differs from initialize_from" + ) + inherited[field] = actual + training = training.model_copy(update=inherited) + source_dtypes = source.options.layer_type_dtypes + if ( + "layer_type_dtypes" in config.run.execution.precision.model_fields_set + and training.layer_type_dtypes != source_dtypes + ): + raise ValueError( + "run.execution.precision.layer_type_dtypes differs from initialize_from" + ) + training.layer_type_dtypes = source_dtypes + if "max_concat_tensors" not in config.run.execution.model_fields_set: + training.max_concat_tensors = source.options.max_concat_tensors + assert config.model is not None + assert config.model.backbone is not None + for dataset_name, dataset in config.dataset_training.items(): + if dataset.model_interface is None: + if len(config.model.interfaces) != 1: + raise ValueError("dataset model_interface is required when ambiguous") + dataset.model_interface = next(iter(config.model.interfaces)) + if dataset.model_interface not in config.model.interfaces: + raise ValueError( + f"Dataset {dataset_name!r} references unknown model interface " + f"{dataset.model_interface!r}." + ) + if training.context_length is None: + training.context_length = config.model.backbone.architecture.max_context_length + if training.training_objective is None: + if supervised: + raise ValueError( + "supervised phases require model.execution.training_objective" + ) + training.training_objective = "policy" + training.target_offset = 0 + training = TrainingRuntimeSettings.model_validate(training.model_dump()) + config.runtime_settings = training + if needs_policy_contracts: + config.model.execution.context_length = training.context_length + config.model.execution.training_objective = training.training_objective + config.model.execution.target_offset = training.target_offset + config.model.execution.forward_attention = training.forward_attention + + # Keep each physical metadata read shared by model assertions and source resolution. + metadata_cache: dict[Path, DatasetMetadata] = {} + + def read_metadata(path): + resolved = (root / path).resolve() + if resolved not in metadata_cache: + metadata_cache[resolved] = load_dataset_metadata(str(resolved)) + return metadata_cache[resolved] + + metadata_paths = { + name: str(root / path) + for name, spec in config.model.interfaces.items() + if (path := spec.input_contract.metadata_config_path) + } + declarations = {name: read_metadata(path) for name, path in metadata_paths.items()} + part_metadata = {} + for dataset_name, dataset in config.dataset_training.items(): + for part_name, part in dataset.parts.items(): + ref = f"{dataset_name}.{part_name}" + path = ( + part.metadata_config_path + if not isinstance(part, PreferencePartSpecModel) + else None + ) + if metadata_by_part is not None and isinstance(part, DatasetPartSpecModel): + if ref in metadata_by_part: + part_metadata[ref] = metadata_by_part[ref] + elif path: + part_metadata[ref] = read_metadata(path) + if loaded_metadata is not None: + loaded_metadata.update(part_metadata) + + source_fractions = _source_fractions(config) + sources = TrainingSources(config, root) + sequences, preferences = ( + sources.resolve(metadata_by_part=part_metadata) + if needs_policy_contracts + else ({}, {}) + ) + fitting_sequences = { + ref: sequence + for ref, sequence in sequences.items() + if isinstance(sources.parts[ref][1], SequencePartSpecModel) + } + + for phase in phases: + if phase.algorithm == "supervised": + continue + phase.optimization = phase.optimization_configs() + phase.model_interface = phase_interface(config, phase) + _resolve_phase_sources( + config, phase, preferences if phase.algorithm == "dpo" else sequences + ) + reference = phase.reference_config + if reference and reference.path: + reference.path = str((root / reference.path).resolve()) + archive = phase.rollout_archive_directory + if archive: + phase.archive_directory = str((root / archive).resolve()) + + from sequifier.runtime.plan import validate_continuation + + for index, phase in enumerate(phases): + validate_continuation( + phases[index - 1] if index else None, phase, resolved=True + ) + + resolved_phases = resolve_phases(phases, config.dataset_training, seed=seed) + validate_training_contract(config, resolved_phases) + + datasets, dataset_interfaces, dataset_storage = resolve_supervised_datasets( + config, + part_metadata, + resolved_phases, + supervised_names, + metadata_from_file=metadata_by_part is None, + ) + metadata = dict(declarations) + if needs_policy_contracts: + for ref, value in part_metadata.items(): + name = config.dataset_training[ref.partition(".")[0]].model_interface + if name in metadata: + expected_fields, actual_fields = dict(metadata[name]), dict(value) + for key in ( + "column_data_types", + "n_classes", + "id_maps", + "selected_columns_statistics", + "normalize_real_columns", + ): + if expected_fields[key] != actual_fields[key]: + raise ValueError(f"{ref}: interface metadata differs at {key}") + metadata[name] = value + + interfaces = {} + storage = {} + history_interface = phases[0].model_interface if history_only else None + if history_only: + assert source is not None + if history_interface not in source.interfaces: + raise ValueError( + f"history transfer requires interface {history_interface!r}" + ) + interface_order = dict.fromkeys( + config.model.interfaces + if needs_policy_contracts + else [*dataset_interfaces, *config.model.interfaces] + ) + for name in interface_order: + spec = config.model.interfaces[name] + value = metadata.get(name) + binding = config.policy_bindings.get(name) + if source is not None and not history_only: + interface = source.interface(name) + if value is not None: + candidate = _resolve_interface(name, spec, value, training) + expected_fields, actual_fields = dict(interface), dict(candidate) + for key in ( + "column_data_types", + "id_maps", + "selected_columns_statistics", + "target_decoder_ids", + ): + if expected_fields[key] != actual_fields[key]: + raise ValueError( + f"{name}: metadata differs from initialization at {key}" + ) + selected_storage = source.interface_storage.get(name) + elif name in dataset_interfaces and not needs_policy_contracts: + interface = dataset_interfaces[name] + selected_storage = dataset_storage[name] + else: + if name == history_interface and value is None: + assert source is not None + if binding is None: + raise ValueError(f"{name}: history_only requires a policy binding") + value = _history_initialized_metadata( + config, + name, + binding, + source.interface(name), + source.interface_storage.get(name), + ) + if value is None: + if binding is None: + raise ValueError(f"{name}: model interface requires metadata") + value = _inferred_metadata(config, name, binding) + if binding is not None and name != history_interface: + value = _fit_input_metadata( + config, + name, + binding, + value, + fitting_sequences, + preferences, + source_fractions, + prepared_inputs, + ) + interface = _resolve_interface(name, spec, value, training) + selected_storage = ResolvedInterfaceStorage.from_metadata(value) + from sequifier.helpers import resolve_window_view + + resolve_window_view(selected_storage.storage_layout, interface.window_view) + if name in dataset_interfaces and interface is not dataset_interfaces[name]: + canonical = _interface_semantics(interface) + for key in ("window_view", "policy", "prediction_distributions"): + canonical.pop(key) + _assert_compatible( + canonical, _interface_semantics(dataset_interfaces[name]), name + ) + if selected_storage is not None: + storage[name] = selected_storage + interfaces[name] = interface + + execution = ModelExecution( + config.model.architecture_model(), + interfaces, + ModelExecutionOptions.from_training(training), + interface_storage=storage, + ) + if history_only: + validate_history_transfer(source, execution, history_interface) + for name, binding in config.policy_bindings.items(): + if name not in interfaces: + raise ValueError(f"policy_bindings: unknown interface {name!r}") + apply_binding(execution, name, resolve_binding(execution, name, binding)) + + for phase in phases: + if phase.algorithm == "supervised": + continue + name = phase.model_interface + policy = interfaces[name].policy + if policy is None: + raise ValueError( + f"{phase.name}: interface {name!r} requires a policy binding" + ) + for environment in ( + phase.training_environment, + phase.evaluation_environment, + ): + if environment and environment.units != policy.units: + raise ValueError( + f"{phase.name}: environment units differ from policy units" + ) + + curriculum = supervised[0].curriculum if supervised else None + resolved = ResolvedSequifierConfig( + ownership=config.model_dump(mode="json"), + authored=config, + project_root=config.run.project_root, + model_name=config.run.model_name, + device=config.run.execution.device, + seed=seed, + runtime_settings=training, + model_options=execution.options, + model=execution.model, + interfaces=interfaces, + interface_storage=storage, + model_metadata_paths=metadata_paths, + dataset_training=datasets, + training_plan=resolved_phases, + initialize_from=config.initialize_from, + actor_initialization=config.actor_initialization, + model_path=config.run.outputs.model_path, + report_path=config.run.outputs.report_path, + phase_directory=config.run.outputs.phase_directory, + sequence_sources=sequences, + preference_sources=preferences, + source_validation_fractions=source_fractions, + policy_bindings={ + name: interface.policy + for name, interface in interfaces.items() + if interface.policy + }, + curriculum_training=curriculum is not None, + curriculum_column=curriculum.column if curriculum else None, + publication=publication_plan(config.run, resolved_phases), + ) + if has_policy: + resolved._prepared_inputs = prepared_inputs + from sequifier.config.input_contract_resolution import validate_input_contracts + from sequifier.io.output_paths import resolve_training_output_paths + + validate_input_contracts(config, resolved, declarations) + resolve_training_output_paths(resolved) + return resolved + + +def policy_run_config( + config: ResolvedSequifierConfig, + phases, + *, + incoming=None, + index=0, + destination=None, + resuming=None, +): + """Select resolved inputs for one native policy lifecycle.""" + from sequifier.artifacts.run_checkpoint import checkpoint_path + from sequifier.config.components import ResumeConfig + from sequifier.model.execution import ModelExecution + from sequifier.rl.run_config import ResolvedPolicyRun + + names = {phase.model_interface for phase in phases} + if len(names) != 1: + raise ValueError("a native policy lifecycle requires one interface") + name = next(iter(names)) + execution = ModelExecution.from_config(config) + initialization = ( + incoming if index or incoming is not None else config.initialize_from + ) + actor_initialization = config.actor_initialization if index == 0 else "attach" + binding = execution.interface(name).policy + if binding is None: + raise ValueError("selected interface has no policy binding") + training = config.runtime_settings + resume = training.resume + checkpoint = checkpoint_path(config) + if resuming is not None: + checkpoint = ( + Path(config.phase_directory) / f"{index:03d}-{phases[0].name}" / "run.pt" + ) + return ResolvedPolicyRun( + project_root=config.project_root, + model_name=config.model_name, + device=config.device, + seed=config.seed, + phase_offset=index, + enforce_determinism=training.enforce_determinism, + policy=binding, + execution=execution, + policy_interface=name, + initialize_from=initialization, + metadata_paths=[config.model_metadata_paths[name]] + if name in config.model_metadata_paths + else [], + actor_initialization=actor_initialization, + normalization=config.normalization_by_interface.get(name), + units=binding.units, + sources=config.sequence_sources, + preferences=config.preference_sources, + phases=phases, + validation_fraction=0.2, + source_validation_fractions=config.source_validation_fractions, + latest_checkpoint_interval_minutes=training.latest_checkpoint_interval_minutes, + episode_cache_size=training.episode_cache_size, + log_interval_iterations=training.log_interval_iterations, + resume=ResumeConfig( + checkpoint_path=str(checkpoint), + policy=("required" if resuming else "never") + if resuming is not None + else (resume.policy if resume else "never"), + mode=resume.mode if resume else "reset_environments", + ), + embedding_layer_names=config.embedding_layer_names, + model_path=str(destination or config.model_path), + report_path=config.report_path + if destination is None + else str(Path(destination).with_name("report.json")), + checkpoint_path=str(checkpoint), + phase_directory=config.phase_directory + if destination is None + else str(Path(destination).parent / "native"), + ) diff --git a/src/sequifier/consolidated-docs.md b/src/sequifier/consolidated-docs.md deleted file mode 100644 index 19c32a374..000000000 --- a/src/sequifier/consolidated-docs.md +++ /dev/null @@ -1,2281 +0,0 @@ -

- Sequifier -

- -# What is sequifier? - -Sequifier is the short, efficient, scalable path from tabular sequences to your own transformer model. - -Its three core commands, `preprocess`, `train`, and `infer`, are configurable. -Preprocessing can use multiple CPU workers for eligible workloads, and training -supports multiple GPUs. - -They enable you to go from multivariate sequence data to a model that ingests such data, and emits either (a) this data, (b) a subset, (c) other variables or (d) embeddings. - -If input and target variables are the same, it supports full autoregressive inference. - -Through configuration, most modern architectural variants such as RoPE, GQA, SwiGLU, and RMSNorm are supported. The ambition is to follow the frontier when industrial-grade implementations become available. - -The process looks like this: - -

- Sequifier preprocessing, training, and inference workflow -

- -## Examples from the literature - -Multivariate tabular transformers have *many* applications. - -These independent references illustrate some of them; inclusion does not imply that the cited work used or is affiliated with sequifier. - -**Finance** - -- Transaction Foundation Models ([Overview](https://sistemalabs.com/blog/shaping-the-problem), [EWE-1](https://sistemalabs.com/blog/introducing-ewe-1)) -- Order Book Transformers ([TransLOB](https://arxiv.org/abs/2003.00130)) -- Volatility Forecasting ([paper](https://www.sciencedirect.com/science/article/abs/pii/S0952197624003816)) -- Macroeconomic Models ([BISTRO](https://www.bis.org/publications/bistro-general-purpose-oracle-macroeconomic-time-series)) - -**Health/Bio** - -- Health Records ([BEHRT](https://www.nature.com/articles/s41598-020-62922-y), [Med-BERT](https://www.nature.com/articles/s41746-021-00455-y)) -- Treatment Outcome Prediction ([G-Transformer](https://arxiv.org/abs/2406.05504)) -- Health Trajectory Prediction ([ETHOS](https://www.nature.com/articles/s41746-024-01235-0)) -- Glucose Forecasting ([GluForecast](https://arxiv.org/html/2606.18640v1)) -- ECG rhythm/arrhythmia monitoring ([Heart Language Model](https://www.nature.com/articles/s41598-024-84270-x)) - -**Cybersecurity** - -- Network Intrusion Detection ([paper](https://arxiv.org/pdf/2309.01070)) -- Encrypted Traffic Analysis ([Criss-Cross Traffic Transformer](https://ieeexplore.ieee.org/document/11396346)) -- Insider Threat Detection ([paper](https://www.nature.com/articles/s41598-025-12063-x)) -- In-Vehicle Intrusion Detection ([paper](https://www.sciencedirect.com/science/article/abs/pii/S0950705124007251)) -- IoT Anomaly Detection ([paper](https://www.sciopen.com/article/10.32604/cmc.2024.053765)) -- System Logs Anomaly Detection ([DeepEAD](https://ieeexplore.ieee.org/document/10279563)) - -**Industrial & IoT** - -- Remaining Useful Life Estimation ([paper](https://www.sciencedirect.com/science/article/abs/pii/S095219762400633X)) -- Fault Diagnosis ([paper](https://www.sciencedirect.com/science/article/abs/pii/S0957582026001485)) -- Anomaly Detection for Industrial Control Systems ([paper](https://onlinelibrary.wiley.com/doi/10.1155/2024/5459452)) -- Soft Sensing ([Debutanizer](https://www.sciencedirect.com/science/article/abs/pii/S1876107024003249)) -- Battery State-of-Health Estimation ([DS-transformer](https://www.nature.com/articles/s41598-026-52202-6)) -- Production Line Modelling ([paper](https://www.sciencedirect.com/science/article/pii/S2212827126007791?)) - -**Agriculture & Environment** - -- Soil Moisture Forecasting ([paper](https://link.springer.com/chapter/10.1007/978-3-032-19763-4_18)) -- Crop Water Demand ([paper](https://www.mdpi.com/2073-4395/16/16/1600)) -- Rainfall–runoff modelling ([paper](https://www.sciencedirect.com/science/article/abs/pii/S0022169422003560)) -- Drought Forecasting ([paper](https://www.sciencedirect.com/science/article/abs/pii/S1364815225000787)) -- Land Surface Dynamics ([paper](https://www.sciencedirect.com/science/article/abs/pii/S0022169425002446)) - -**Neuroscience** - -- Neural Population Dynamics ([Neural Data Transformer](https://pmc.ncbi.nlm.nih.gov/articles/PMC10541112/)) -- Neural + Motor Modelling ([Intracortical Motor Decoder](https://www.biorxiv.org/content/10.1101/2025.02.02.634313v1)) -- fMRI State Prediction ([paper](https://arxiv.org/abs/2412.19814)) -- Seizure Detection ([BIOT](https://arxiv.org/abs/2305.10351)) -- Magnetoencephalography Data ([MEG-GPT](https://arxiv.org/abs/2510.18080)) - -**Animal Communication** - -- Sperm Whales ([whale-gpt](https://github.com/0xideas/whale-gpt), [WhAM](https://github.com/Project-CETI/wham/)) -- Bengalese Finches ([paper](https://pmc.ncbi.nlm.nih.gov/articles/PMC8746767/)) -- Zebra Finches ([ZF-Aim](https://www.biorxiv.org/content/10.64898/2026.02.12.705387v1)) - -Sequifier aims to standardise model implementation across these fields, to make them comparable, transfer learnings between domains, and converge on optimal solutions faster. - -## Value Proposition - -For the individual researcher, sequifier cuts the development time of a model significantly: typically, some preprocessing and the subsequent model evaluation are specific to the modelling problem, but all the steps in between are taken care of. - -This enables: - -- rapid prototyping on a configurable architecture -- trusted implementation (you can't create bugs inadvertently) -- scaling preprocessing across cores and training across GPUs and nodes -- hyperparameter optimization using Optuna (Bayesian, Random, or Grid search) - -## The Seven Commands - -There are seven standalone commands within sequifier: `make`, `preprocess`, `train`, `infer`, `hyperparameter-search`, `visualize-training`, and `docs`. - -| Command | Purpose | -| --- | --- | -| `make` | Create a new sequifier project with config templates. | -| `preprocess` | Convert input data into fixed-length subsequences. | -| `train` | Train a model on preprocessed data. | -| `infer` | Generate predictions, probabilities, or embeddings. | -| `hyperparameter-search` | Use Optuna to find optimal configurations across multiple training runs. | -| `visualize-training` | Generate interactive HTML plots from structured training metrics. | -| `docs` | Print the consolidated documentation, or the guide for `train`, `preprocess`, or `infer`. | - -Run `sequifier docs` for all documentation, or `sequifier docs train`, `sequifier docs preprocess`, or `sequifier docs infer` for one command guide. - -The processing and visualization commands have dedicated documentation pages: - -- [preprocess documentation](./documentation/configs/preprocess.md) -- [train documentation](./documentation/configs/train.md) -- [infer documentation](./documentation/configs/infer.md) -- [hyperparameter-search documentation](./documentation/configs/hyperparameter-search.md) -- [visualize-training documentation](./documentation/commands/visualize-training.md) - -## Other Materials - -To get the full documentation, visit [sequifier.com](https://sequifier.com) - -For changes in each published version, see the [release notes](https://github.com/0xideas/sequifier/releases). - -## Structure of a Sequifier Project - -Sequifier is designed with a specific folder structure in mind: - -```text -YOUR_PROJECT_NAME/ -├── configs/ -│ ├── preprocess.yaml -│ ├── train.yaml -│ └── infer.yaml -├── data/ -│ └── (Place your CSV/Parquet files here) -├── models/ -├── checkpoints/ -├── outputs/ -│ ├── embeddings(?) -│ ├── predictions(?) -│ ├── probabilities(?) -│ └── visualization/ -├── logs/ -├── state/ (created when hyperparameter search runs) -└── scripts/ -``` - -The `sequifier` commands should typically be run in the project root. - -Within YOUR_PROJECT_NAME, you can also add other folders for additional steps, such as `notebooks` or `scripts` for pre- or postprocessing, and `analysis`, `visualizations` or `evals` for files you generate in other, manual steps. - -### Data Transformations in Sequifier - -The basic input data format is this: - -| sequenceId | itemPosition | column1 | column2 | ... | -| --- | --- | --- | --- | --- | -| 0 | 0 | "high" | 12.3 | ... | -| 0 | 1 | "high" | 10.2 | ... | -| ... | ... | ... | ... | ... | -| 1 | 0 | "medium" | 20.6 | ... | -| ... | ... | ... | ... | ... | - -The two columns "sequenceId" and "itemPosition" have to be present, and there must be one or more feature columns. - -`sequifier preprocess` splits sequences into subsequences, normalises real variables and maps categorical variables to integers/tokens. Each stored subsequence contains `window_length` positions; `max_target_offset` reserves positions within that window for future targets. - -| sequenceId | subsequenceId | startItemPosition | leftPadLength | inputCol | [window_length - 1] | [window_length - 2] | ... | 0 | -| --- | --- | --- | --- | --- | --- | --- | --- | --- | -| 0 | 0 | 0 | 0 | column1 | "high" | "high" | ... | "low" | -| 0 | 0 | 0 | 0 | column2 | 12.3 | 10.2 | ... | 14.9 | -| ... | ... | ... | ... | ... | ... | ... | ... | ... | -| 1 | 0 | 15 | 0 | column1 | "medium" | "high" | ... | "medium" | -| 1 | 0 | 15 | 0 | column2 | 20.6 | 18.5 | ... | 21.6 | -| ... | ... | ... | ... | ... | ... | ... | ... | ... | - -Generative inference returns a row-oriented table with the predicted target -columns plus identifiers for the source sequence and model window: - -| sequenceId | subsequenceId | windowStartOffset | itemPosition | column1 | column2 | ... | -| --- | --- | --- | --- | --- | --- | --- | -| 0 | 0 | 0 | 963 | "medium" | 8.9 | ... | -| 0 | 0 | 0 | 964 | "low" | 6.3 | ... | -| ... | ... | ... | ... | ... | ... | ... | -| 1 | 4 | 0 | 732 | "medium" | 14.4 | ... | -| ... | ... | ... | ... | ... | ... | ... | - -### Complete Example of Training and Inferring a Transformer Model - -Once you have your data in the input format described above, you can train a transformer model in a couple of steps: - -1. **Install sequifier.** Create and activate an environment with Python >=3.10, then run: - - ```console - pip install sequifier - ``` - -2. **Create a project.** Generate a project folder with config templates in its `configs` subfolder: - - ```console - sequifier make YOUR_PROJECT_NAME - ``` - -3. **Add your data.** Change into the `YOUR_PROJECT_NAME` folder, create a `data` folder, add your data, and adapt `preprocessing_data_path` in `preprocess.yaml` to point to it. - -4. **Preprocess the data.** Run: - - ```console - sequifier preprocess - ``` - -5. **Reference the generated metadata.** Preprocessing outputs metadata at `configs/metadata_configs/[INPUT BASENAME].json`. For a single dataset and part, reference that file from `dataset.part.metadata_config_path` in `train.yaml`; named configurations use `dataset_training..parts..metadata_config_path`. Inference may still use `preprocessing_data_path` or `metadata_config_path`. - -6. **Configure and train the model.** Adapt `train.yaml` with your transformer hyperparameters, then run: - - ```console - sequifier train - ``` - -7. **Configure inference.** Point `model_path` in `infer.yaml` at the default ONNX export. Keep the scaffold's explicit contract, or replace it with `training_config_path` and `dataset`; see the [ONNX/PT trade-offs](./documentation/configs/infer.md#onnx-or-pt). - - **GPU ONNX inference:** `device: cuda` requires `onnxruntime-gpu` built for - your CUDA and cuDNN versions. The CPU-only `onnxruntime` package does not - provide `CUDAExecutionProvider`. - -8. **Run inference.** Run: - - ```console - sequifier infer - ``` - -9. **Find your predictions.** They are written to `[PROJECT ROOT]/outputs/predictions/[EXPORTED_MODEL_BASENAME]/part-000.[FORMAT]`, for example `outputs/predictions/your-model-best-3/part-000.csv`. - -## Key Features - -### Causal Modelling - -Causal transformers are the core architecture in sequifier. Their training objective is to 'predict the next token', or, in the multivariate case, the next value for each target variable. - -To train a causal transformer, set `training_objective: causal`. If the target value of interest is further in the future, set `target_offset` to a value higher than 1. - -`prediction_length` controls how many consecutive positions at the end of a -window are returned during inference. For example, with `target_offset: 1` and -`prediction_length: 3`, the three outputs predict the next token after each of -the last three input positions. This is not LLM-style multi-token prediction, -where one input position predicts several future tokens. During training, the -current loss calculation uses all valid decoded positions, regardless of -`prediction_length`. - -### Autoregressive Inference - -Autoregressive inference is allowed when the model is causal, all input variables are target variables, `target_offset` equals 1 and `prediction_length` equals 1. It is enabled by setting `autoregressive: true`, and `generation_steps` to the desired integer value. The number of generation_steps will be generated starting from the *first* complete subsequence in a sequence. - -It iteratively predicts future values, by returning predictions at step t-1 as input for generating a prediction at t. Predictions for categorical target variables can be made using argmax or sampling. - -For dependencies between multiple outputs at the same time step, the -`autoregressive_transformer` decoder models targets in explicit -`target_columns` order. It supports categorical and real targets, parallel -teacher forcing during training and validation, greedy generation during -inference, composite decoder branches, shared categorical tables, and optional -input/output weight tying. Probabilities from this decoder are conditioned on -the preceding greedy same-step predictions, so external per-column sampling is -rejected for its targets. - -### Causal Modelling Variants - -#### Final-value Causal Modelling - -In final-value causal models, the final value of each target variable within the subsequence is projected back in time as target. The idea is that the sequence of events leading up to the final value is a continuous accrual of evidence for an outcome, with the final value being the resolution. For example, the sequence of clicks through an online shop are in search of a product, and the product that is actually purchased at the end is the resolution. - -#### Next-occurrence Causal Modelling - -Next-occurrence causal modelling is a generalisation of final-value causal modelling: instead of taking the last value of each target variable as target, it takes the next value at a position where another categorical variable matches a criterion value as target. To illustrate this, final-value causal modelling is equivalent to next-occurrence causal modelling where the criterion variable is 'is_last', which is '0' up to the last position, where it is '1', and the criterion value is '1'. The values at the last position are projected 'back' across the subsequence, only now, we also have the option to use a different criterion variable, set it to '1' at multiple locations, and train the model to predict 'next relevant event', rather than just 'last event'. - -### Causal Embedding Model - -sequifier also supports the export of causal embeddings, instead of predictions. It requires the following settings: `export_embedding_model: true` in the training config and `model_type: embedding` in the inference config. Selected activations are restricted to the configured final `prediction_length` positions and concatenated in configuration order along the feature dimension. - -If you are interested in activations *other* than the last backbone layer, you can configure the exact layers you want to contribute to the export using `embedding_layer_names`. You can pass an ordered list, such as `[backbone.layers.1, decoder.branches.default.hidden_blocks.0]`, and the activations of these layers will be concatenated and output. - -Layer names follow the network hierarchy using zero-based indices: `backbone.layers.` selects a transformer block output, `backbone.final_norm` the normalized backbone output, and `decoder.branches..hidden_blocks.` an MLP decoder hidden-block output; the same scheme applies to BERT embedding models. - -Backbone selectors contribute `dim_model` activations. Decoder MLP hidden-block selectors contribute their configured hidden width and receive the same flattened `decoding_support * dim_model` windows used during training. The default, `embedding_layer_names: [backbone.final_norm]`, preserves the final normalised backbone representation. - -### BERT Model - -Sequifier also supports training and inference of BERT-style masked reconstruction models. - -Configuration: - -- Preprocessing: Set `max_target_offset: 0` for equal-width input and target windows. -- Training: Set `training_objective: bert`, configure `bert_spec`, and set decoder `prediction_length` equal to `context_length`. Enable generative and/or embedding export according to the desired inference. -- Inference: Set `model_type: generative` to reconstruct explicitly masked input, or `model_type: embedding` to output contextual representations. - -Technical Details: BERT-style models use bidirectional attention and learn by reconstructing positions sampled according to `bert_spec`. Inference does not apply random masking; inputs that should be reconstructed must be masked explicitly, for example using `mask_column` during preprocessing. Embedding inference returns one contextual representation for every valid position in the input window. - -### Structured Ingestion - -Structured ingestion allows the model to learn relationships for configured sets of input variables *before* they are passed to the transformer backbone. - -This enables more constrained representation learning within these subspaces, and provides a structural inductive bias and may reduce parameters, depending on configuration. It can be helpful to think of them as smaller submodules that learn local structure before passing the extracted information to the transformer for longer-range dependencies. - -The key modalities are self-attention, pooling, 1D, 2D and 3D convolutions, and adding learned or rotary axis embeddings. - -Image ingestion reads named image or video views from their original folders. -Each view uses a frame rate relative to `itemPosition` or timestamps aligned -through a configured time column; a trainable -ResNet or frozen pretrained backbone embeds each image. Views can enter a -composite branch or occupy cells in a structured layout. A separate single -image can condition the whole sequence through one prefix embedding. See the -[image source configuration](documentation/configs/preprocess.md#image-and-video-sources) -and [image ingestion configuration](documentation/configs/train.md#image-ingestion). - -#### Repeated child collections - -Named depth layouts represent a fixed-capacity collection of child rows at each -outer sequence position. A `depth_transformer` ingestion branch encodes the -valid children, pools them through a CLS token, and emits one vector per outer -position for the temporal transformer backbone. It can be combined with shallow -features in nested composite ingestion branches. - -The preprocessing metadata and PT payload carry each layout's capacity and -validity mask. Portable PT and ONNX models retain that execution contract. See -the [preprocessing guide](documentation/configs/preprocess.md#named-depth-layouts), -[training guide](documentation/configs/train.md#depth-encoders-and-nested-composites), -and [inference guide](documentation/configs/infer.md#portable-depth-models-and-dropout-modes). - -#### Temporal Convolution - -Separately, `temporal_conv` enables temporal convolutions on pass-through or embedded real or categorical variables. - -`temporal_conv` uses causal, left-padded convolutions by default (`causal: true`). -For causal models, keep the default `orientation: within_item_position` when -using `post_conv_norm: layer_norm` or `rmsnorm`. With -`orientation: within_column`, these norms operate across the full time axis, so -earlier positions can depend on later ones despite the causal convolution and -the transformer's attention mask. If `within_column` is needed in a causal -model, set `post_conv_norm: none`. - -#### Categorical multi-hash embeddings and targets - -An interface can hash canonical categorical IDs for input embeddings and target -supervision. Configure `categorical_hashing` on the model interface, keyed by -logical column name: - -```yaml -model: - interfaces: - default: - input_columns: [product_id] - target_columns: [product_id] - categorical_hashing: - product_id: - type: multi_hash - num_buckets: 4096 - num_hashes: 3 - seed: 42 - ingestion: - type: embedding - output_dim: 128 -``` - -The `categorical_hashing` field belongs to the interface. Its settings drive -both the input embedding and target heads for `product_id`; do not put a -`hashing` field under `ingestion`. See the -[training guide](documentation/configs/train.md#categorical-multi-hash-embeddings-and-targets) -for a two-column example and the required target loss configuration. - -`type: qr` uses quotient and remainder heads; `num_hashes` is always 2. Each -hash component gets an equal share of the logical target's `loss_weights` -entry. Class weights are unavailable for hash targets. The model resolves -component predictions against the allowed canonical IDs and reports the -logical column as usual. The resolver contributes no training loss. No hash -columns are needed in stored data. Preprocessing cardinality hashing remains -independent: a category collapsed there still decodes as `[hash_bucket:k]`. -Configure model hashing through the interface-level `categorical_hashing` field. - -In an `autoregressive_transformer` decoder branch, a hashed target must be the -last target in that branch's `target_columns` order. - -### Multi-Part Datasets - -Multi-part datasets let a model train on separately preprocessed data with -distinct metadata configs. Parts must share the schema, categorical mappings, -normalization policy, `max_target_offset`, per-split window widths, alignment -settings, dtypes, and file/folder storage form. Set `metadata_config_path` to -the first part's metadata when preprocessing later parts to reuse its fitted -category and normalization meanings. - -### Composable Configs - -Between different training runs and hyperparameter searches, a lot of configuration can get duplicated, and it becomes hard to follow what differentiates them and where they overlap. One approach to address this is to create different config fragments, and compose them into full training and hyperparameter tuning configurations using separate 'top-level' configs, and assemble the fragments by listing them as value in `additional_config_paths`. Fragments can contribute disjoint nested fields, while duplicate fields, recursive fragment inclusion, and repeated files are rejected; command-line overrides are applied after composition. - -### Distributed Training - -Sequifier supports distributed training using torch `DistributedDataParallel` and `FullyShardedDataParallel`. To make use of multi gpu support, the preprocessing step must write sharded output with `merge_output: false`. `write_format: pt` is the recommended file format; sharded `parquet` is also supported but currently considered beta for distributed training. - -For the full guide on how to configure a distributed run, check the [multi-GPU training guide](./documentation/training/multi-gpu-training.md). - -### System Requirements - -Sequifier currently runs on MacOS and Ubuntu. - -## Citation - -Please cite with: - -```bibtex -@software{sequifier_2025, - author = {Luithlen, Leon}, - title = {sequifier - transformers for multivariate sequence generation and representation learning}, - year = {2025}, - publisher = {GitHub}, - version = {v3.1.2.0}, - url = {https://github.com/0xideas/sequifier} -} - -``` - - -# Preprocess Command Guide - -The `sequifier preprocess` command transforms raw tabular data (CSV or Parquet) into the specific sequence format required for training transformer sequence models. It handles windowing, data splitting (train/validation/test), categorical encoding, and optional numerical standardization. - -## Usage - -```console -sequifier preprocess --config-path configs/preprocess.yaml -``` - -## CLI Overrides - -Values passed on the command line override the YAML before validation. - -| Flag | Overrides / Action | -| :--- | :--- | -| `-r`, `--randomize` | Generates a random `seed`. The seed affects `between_sequence` split assignment. | -| `-dp`, `--data-path` | Overrides `preprocessing_data_path`. | -| `-sc`, `--selected-columns` | Overrides `selected_columns` with a space-separated list. Use `None` to process all columns. | - -## Composable Configuration Files - -A preprocessing entry config may set `additional_config_paths` to one -non-empty string, a list of non-empty strings, or `null`. Relative paths -resolve against the entry config's `project_root`; absolute paths are used -directly. Fragments are direct only and cannot include further fragments. They -may share nested containers when their child fields are disjoint, but duplicate -fields are errors. CLI values override the completed file composition. - -## Configuration Fields - -The configuration is defined in a YAML file (e.g., `preprocess.yaml`). Below are the available fields, their requirements, and their functions. - -### 1\. File System & Input/Output - -| Field | Type | Mandatory | Default | Description | -| :--- | :--- | :--- | :--- | :--- | -| `project_root` | `str` | **Yes** | - | The root directory of your Sequifier project. Usually `.` | -| `additional_config_paths` | `str`, `list[str]`, or `null` | No | `null` | Direct complementary YAML fragments. Relative paths resolve against `project_root`; recursive composition and duplicate fields are rejected. | -| `preprocessing_data_path` | `str` | **Yes** | - | Path to the raw input file or folder. | -| `static_data_path` | `str` | No | `null` | Separate CSV or Parquet table with exactly one row per `sequenceId`. Preprocessing requires its ID set to match the sequence data in both directions and writes a processed static feature table. | -| `static_categorical_columns` | `list[str]` | No | `[]` | Static columns to treat as categorical when their source dtype would not make that intent clear. | -| `static_real_columns` | `list[str]` | No | `[]` | Static numeric columns to treat as real valued, including integer measurements. | -| `read_format` | `str` | No | `csv` | Format of input data (`csv`, `parquet`). | -| `write_format` | `str` | No | `parquet` | Format of output data (`csv`, `parquet`, `pt`). | -| `merge_output` | `bool` | No | `true` | Whether to merge split files into single files or keep them sharded. | -| `continue_preprocessing`| `bool` | No | `false` | If `true`, resumes from an existing preprocessing temp folder created by an interrupted run. | - - -> **Important Constraint on `write_format`:** -> -> * If `write_format` is **`pt`** (PyTorch tensors), `merge_output` must be **`false`**. -> * If `write_format` is **`parquet`**, `merge_output` can be **`false`** or **`true`**. -> * If `write_format` is **`csv`**, `merge_output` must be **`true`**. -> For distributed training, `merge_output` must be set to **`false`**. - -Static columns are read from `static_data_path`, not `selected_columns` or the -sequence rows. The static table may reside in the sequence input folder; it is -excluded from sequence-file discovery. Its `sequenceId` column must be an -integer without nulls or duplicates. Static feature columns must have finite, -non-null values and must not share names with temporal columns. Categorical -maps and real normalization are fitted using sequences represented in training -split 0, unless `normalize_on_all_data` is enabled. The processed feature -table has one row per sequence and is referenced by the generated metadata. - -### Image and video sources - -Install `sequifier[images]` to use image sources. Temporal images live in one -flat folder per named view. Untimed images are named -`{sequence_id}-{frame_index}` plus an image extension; an untimed video view has -one `{sequence_id}` video per sequence. Decoded video frames are numbered from -zero in decode order. A separate `preconditioning_image` -source contains one `{sequence_id}` image per sequence in each folder. Its -`path` can be one folder or a map of named view folders. - -```yaml -image_sources: - front: - path: data/front_images - type: images - alignment: {frames: 3, items: 2} - side: - path: data/side_videos - type: videos - alignment: {frames: 1, items: 2} -preconditioning_image: - path: - front: data/sequence_front_images - side: data/sequence_side_images -``` - -`alignment` defaults to `{frames: 1, items: 1, phase: 0}`. `frames` and -`items` are positive integers; `phase` is an integer from zero through -`items - 1`. Every sequence starts at `itemPosition: 0`, and image frame indices -start at zero. With `p = frames`, `q = items`, and `h = phase`: - -- For `p >= q`, item `i` receives frames from boundary `b(i)` through - `b(i+1) - 1`, where `b(i) = ceil((i*p+h)/q) - ceil(h/q)`. -- For `p < q`, item `i` uses frame `floor((i*p+h)/q)`, repeating the latest - image until the next frame becomes available. - -For irregular item times, set `time_alignment_column` at the top level of the -preprocessing config. Timed media filenames use one of these patterns for every -sequence ID in a view: - -- Videos: `{sequence_id}-{time}` (one video per sequence). The filename gives - the first frame's time; subsequent frame times come from video presentation - timestamps. -- Images with a known interval: frame zero is - `{sequence_id}-0-{time}-{frame_interval_ms}`; later frames are - `{sequence_id}-{frame_index}`. The positive interval is in milliseconds. -- Images with every frame timestamped: - `{sequence_id}-{frame_index}-{time}` for all frames. - -`time` is either `HHmmssSSS` (relative time) or `yyyyMMddHHmmssSSS` -(absolute time). The time column must use the same kind of value for all rows: -relative `HHmmssSSS` or absolute timestamps with millisecond precision, including -the 17-digit filename format. A `Date` value represents midnight on that date; -date-only ISO strings use the same convention. Relative media and item times start at -`000000000` for every sequence. Absolute values are compared as written; users -must keep their inputs consistent. Sequifier performs no timezone conversion. -Filename modes and time representations must be uniform across sequence IDs in -each view. Timed views require a positive per-view `max_frames_per_item` and do -not use `alignment` to determine frame counts. -If `max_frames_per_item` is set, it must be at least -`ceil(alignment.frames / alignment.items)`; untimed views otherwise lose frames. -Leave `alignment` at its default for timed views. - -At each item, images since the previous item time (including images exactly at -the current item time) are pooled. If more than `max_frames_per_item` arrived, -the most recent images are kept. If none arrived, the latest prior image is -repeated. The first media frame must be available at or before the first item. -Preprocessing writes an item-time index beside the processed data; this file -must remain available for training and inference. - -```yaml -time_alignment_column: captureTime -image_sources: - front: - path: data/front_images - type: images - max_frames_per_item: 4 -``` - -`time_alignment_column` is used for alignment and is excluded from model -features. It may be omitted from `selected_columns`. - -Preprocessing checks all sequence IDs, contiguous image indices, exact frame -counts for untimed views, and decodability against the raw item grid. Untimed -views may have different frame counts when their configured rates cover the -same grid. Metadata records view names, alignment, transforms, and a file-list -signature, but omits the original folder paths. Supply those paths again when -training or inferring; original media is neither modified nor copied. Loading -fails if the source file list, sizes, or modification times change after -preprocessing. `resize_size` (default 256) and -`crop_size` (default 224) can be set per source; images are resized by their -shorter edge and then center cropped during loading. -Preprocessing writes a versioned video index for each source video beside the -processed data and records its path in metadata. Each index contains frame -timestamps, keyframe seek positions, and frame times for timed alignment. -Workers load an index when they first need its video and reject it if the media -signature or stream time base changed. Videos without reliable seeking use a -temporary indexed cache of transformed frames. Older metadata without video -indexes remains readable through a worker-local scan. - -### 2\. Column Selection & Filtering - -| Field | Type | Mandatory | Default | Description | -| :--- | :--- | :--- | :--- | :--- | -| `time_alignment_column` | `str` | For timed media | `null` | Raw item-time column used to align timed images and video. Stored in a separate item-time index and excluded from model features. | -| `selected_columns` | `list[str]` | No | `null` | A specific list of columns to process. If `null`, all columns (except metadata) are processed. | -| `categorical_columns` | `list[str]` | No | `null` | Explicitly classify supported discrete processed columns as categorical instead of relying on dtype inference. Undeclared columns retain dtype-based inference. Columns cannot also appear in `real_columns`. When `column_data_types` is set, these columns must use an integer output dtype. | -| `real_columns` | `list[str]` | No | `null` | Explicitly classify processed numeric columns as real-valued. This is useful for integer-valued amounts, counts, and epoch timestamps that must retain ordinal meaning. Undeclared columns retain dtype-based inference. Columns cannot also appear in `categorical_columns`. When `column_data_types` is set, these columns must use a floating-point output dtype. | -| `cardinality_config` | `dict[str, object]` | No | `{}` | Per-categorical-column rules for retaining frequent values and optionally hashing the rest into buckets. See [Categorical cardinality and hashing](#4-categorical-cardinality-and-hashing). | -| `column_data_types` | `dict[str, str]` | No | `null` | Optional output dtype map for processed columns, such as `Float32`, `Float64`, `Int32`, or `Int64`. A `Date` or `Datetime...` dtype is accepted for the `value_cutoff` `split_column` or `time_alignment_column`. If set, every processed column must be included. Parquet uses one unified sequence dtype; `pt` writes each variable to its configured tensor dtype. | -| `normalize_real_columns` | `bool` | No | `true` | If `true`, Z-score normalizes real-valued columns. Set to `false` to preserve their original values. Statistics are still recorded in metadata. | -| `normalize_on_all_data` | `bool` | No | `false` | If `false`, numeric statistics and dynamic categorical vocabularies are fitted only on split 0; values seen only in later splits map to `[other]` or a configured hash bucket. Set to `true` to fit on all splits. | -| `max_rows` | `int` | No | `null` | Limits processing to the first N rows. Useful for rapid debugging. | -| `metadata_config_path` | `Optional[str]` | No | `null` | Use a preexisting metadata config for tokenizing discrete columns and, when enabled, standardizing real-valued columns. | -| `mask_column` | `Optional[str]` | No | `null` | Optional input column used as a row-level mask. If set, `metadata_config_path` must also be set, and it cannot also be `split_column`. | -| `curriculum_column` | `Optional[str \| list[str]]` | No | `null` | One or more optional integer input columns to preserve as per-subsequence curriculum metadata. | -| `use_precomputed_maps`| `list[str]` | No | `null` | If not `null`, enforces the use of precomputed maps for the variables in the list. | - -When `curriculum_column` is set, each named column must be integer-valued and -constant within every generated subsequence. A list preserves several columns -so that a later training run can select any one of them. The values are stored -as metadata, not as model features. Different subsequences of one `sequenceId` -may use different values. For depth input, the values must also agree across -all repeated child rows for an outer item; depth PT output carries the same -per-window metadata. Names must be unique and non-empty, must exist in every -input file, and cannot be `sequenceId`, `itemPosition`, the mask column, or a -depth feature/position column. Names beginning with -`__sequifier_curriculum_value_` are reserved. - -### 3\. Sequence Logic & Splitting - -| Field | Type | Mandatory | Default | Description | -| :--- | :--- | :--- | :--- | :--- | -| `window_length` | `int` | Conditional | - | One physical width for every split. Specify exactly one of `window_length` or `window_lengths`. | -| `window_lengths` | `list[int]` | Conditional | - | Physical widths in split order, with one entry per split. Required when `prediction_alignment_steps > 1`; widths from split 2 onward must equal the first width plus `prediction_alignment_steps - 1`. | -| `max_target_offset` | `int` | No | `1` | Number of future items retained after the model input window. Use `0` for BERT-style same-width inputs and targets; use `1` for causal next-item training. | -| `split_ratios` | `list[float]`| Conditional | `null` | Ordered split proportions for `within_sequence` and `between_sequence`. Must sum to 1.0 and must be omitted for `value_cutoff`. | -| `split_method` | `str` | No | `within_sequence` | How rows are assigned to splits: `within_sequence`, `between_sequence`, or `value_cutoff`. | -| `split_column` | `str` | Conditional | `null` | Required for `value_cutoff`. Names the integer, Date, or Datetime column compared with `split_values`; it cannot be a mask, curriculum, sequence ID, or item-position column. | -| `split_values` | `list[int \| timestamp]` | Conditional | `null` | Required for `value_cutoff`. Strictly increasing boundaries that create `len(split_values) + 1` splits. Values must all be integers or all be ISO timestamp values. | -| `window_stride` | `int` | No | First split width | Stored-window stride for distributed splits. | -| `prediction_aligned_splits` | `list[int]` | No | `[]` | Zero-based split indices whose prediction groups are anchored to each split end. All other splits use distributed placement. | -| `prediction_alignment_steps` | `int` | No | `1` | Number of consecutive prediction groups represented by each aligned stored window. | -| `prediction_length` | `int` | Conditional | `null` | Required for prediction-aligned splits; number of output positions per window. | -| `target_offset` | `int` | Conditional | `null` | Required for prediction-aligned splits; must equal `max_target_offset`. | -| `allow_sequence_splitting` | `bool` | No | `false` | If `false`, a single sequence is kept within one preprocessing batch. | - -All newly preprocessed windows store their absolute start position and split -target bounds, including distributed windows and depth-layout PT windows. Dataset -loaders require these fields and reject outputs created with an older payload -schema; re-run preprocessing to migrate such data. - -`value_cutoff` applies the same boundaries to every sequence. Values equal to a -boundary belong to the later split, and the split column must be non-decreasing -within each sequence. Integer columns require integer boundaries. Timestamp -columns support Polars `Date` and `Datetime` values, or ISO timestamp strings -that can be parsed as UTC; `Time` and `Duration` are not supported. A timestamp -split column may be retained in preprocessing output by including it in -`selected_columns` (or by leaving `selected_columns: null`), but timestamp -columns cannot be model inputs or targets in training or inference. Any other -typed temporal column is rejected. The name `__sequifier_split_value` is -reserved for preprocessing internals. - -To align validation or test predictions exactly to their split positions, configure -those split indices and the model's prediction view: - -```yaml -window_length: 129 -max_target_offset: 1 -window_stride: 128 -prediction_aligned_splits: [1, 2] -prediction_length: 2 -target_offset: 1 -``` - -Aligned splits place the last prediction group at the split end and step backward -by `prediction_length`. The first group may begin before the split; its earlier -predictions are masked from loss, metrics, and inference output. Inputs may use -preceding rows from the same sequence. Missing history at the sequence start is -left-padded. The stored windows never read beyond the split end. An aligned -split requires `target_offset == max_target_offset` and -`allow_sequence_splitting: false`. Training and inference must use matching -`target_offset` and `prediction_length`. Training can sample additional model -views with `global_training.window_stride`; aligned inference requires -`window_stride: null`. Empty splits produce no windows. - -Prediction-aligned inference supports generative and embedding output without -autoregressive generation. Aligned embeddings are labeled with the target row -predicted from each activation and use the target validity mask. For a causal -model with `target_offset: 1`, the embedding labeled as row N represents the -context before row N; it does not include row N's value. Embeddings from other -splits retain their input-position labels. - -Distributed splits retain isolated, evenly spread windows and use the scalar -preprocessing `window_stride`. The same placement rules apply to depth-layout -PT output. - -### 4\. Performance & System - -| Field | Type | Mandatory | Default | Description | -| :--- | :--- | :--- | :--- | :--- | -| `seed` | `int` | No | `1010` | Random seed for reproducibility. | -| `n_cores` | `int` | No | Max Cores | Maximum worker processes for eligible preprocessing stages. Depth preprocessing parallelizes a single Parquet file only when it has multiple row groups and `max_rows` is unset. | -| `batches_per_file` | `int` | No | `1024` | Buffer flush threshold per split. For flat input, counts sequence-window groups in PT and Parquet output (including merged output); a byte limit can flush earlier. For depth input, counts windows in PT output. | -| `process_by_file` | `bool` | No | `true` | Memory optimization. If `true`, processes one input file at a time. | - ------ - -## Key Trade-offs and Decisions - -### 1\. `write_format`: `parquet` vs. `pt` - - * **Choose `parquet` (default):** Unless you have a specific reason, use `parquet`. *Note: If you are doing distributed training, Parquet support is currently in **Beta**.* - * **Choose `pt`:** Use `pt` data loading if speed and CPU overhead are your primary bottlenecks, **or if you are running multi-GPU distributed training.** This format is the most stable choice for high-throughput scaling. - -### 2\. Stored windows and model windows - -Preprocessing stores windows of the configured width for each split. Distributed -splits use `window_stride`; aligned stored windows step by -`prediction_alignment_steps × prediction_length`. Training's `context_length` -is the model input width. Each selected split width must be at least -`context_length + max_target_offset`. Training's -`window_stride` is separate: `null` uses one right-aligned model view per stored -window, while a positive integer samples additional views *within* a longer -stored window. If the two widths are equal, `window_stride` adds no views. - -Aligned inference reads `prediction_alignment_steps` model views from each -stored window, moving right by `prediction_length` positions for each view. -Preprocessing requires room for all prediction groups after `max_target_offset`; -training and inference also check the width against the model's `context_length`. -To use the entire window, choose its width as -`context_length + target_offset + (prediction_alignment_steps - 1) × prediction_length`. -Earlier positions outside the split are masked from prediction output. -Set `window_lengths` when `prediction_alignment_steps > 1`. The first two -split widths can differ; every later split width must equal the first width -plus `prediction_alignment_steps - 1`, even when that split uses distributed -placement. - -For example, with `context_length: 128`, `target_offset: 1`, and three splits, -the following gives aligned splits room for three two-position prediction -groups: - -```yaml -window_lengths: [131, 133, 133] -max_target_offset: 1 -prediction_aligned_splits: [1, 2] -prediction_length: 2 -prediction_alignment_steps: 3 -target_offset: 1 -``` - -| Scenario | Suggested settings | Trade-off | -| --- | --- | --- | -| Many short or varied-length sequences | Store the minimum width (for example, `window_length: 129` for `context_length: 128`, `max_target_offset: 1`), use a preprocessing `window_stride` near 128, and set training `window_stride: null`. | Limits padding and stores roughly one copy of long sequences, but targets near window boundaries have less preceding context. | -| More overlap during training | Keep that width; reduce the preprocessing `window_stride` to a fraction of the training `context_length`. | Roughly 2× or 4× as many stored events for long sequences. | -| Long sequences, several model views per stored window | Use a longer stored width (for example, `window_length: 513`, `window_stride: 384`, `context_length: 128`, training config `window_stride: 128`). | About 1.3× stored events on long sequences; short sequences pad to 513, and more model views cost more compute. | -| Dense evaluation with nearly full preceding context at each window's right edge | Use `prediction_aligned_splits` for exact split coverage, or a small preprocessing `window_stride` for distributed evaluation. | More stored windows and more evaluation compute. | - -Self-attention lets the model use relationships between positions within a -window, so heavily overlapping stored windows are often unnecessary. Small -preprocessing strides increase storage and compute and may put very similar -windows in a training batch. Sequifier currently computes training loss over -all valid decoded positions of each model view, regardless of -`prediction_length`. With causal next-event targets, each decoded position -predicts its own next event. A smaller stride gives more targets a view with -substantial preceding context, even though adjacent inputs overlap. -`prediction_length` selects the final consecutive output positions during -inference; it does not make one position predict several future events. Short -sequences are left-padded to their split's stored width regardless of stride; -inspect the sequence-length distribution before choosing a long stored width. - -### 3\. Distributed and prediction-aligned placement - -Distributed placement adjusts starts to cover each split evenly and includes -the final available window. Prediction-aligned placement anchors stored windows -to the split end, then steps backward by -`prediction_alignment_steps × prediction_length`. Each stored window contains -`prediction_alignment_steps` consecutive prediction groups. When the split -length is not divisible by this spacing, the first window includes positions -before the split; those predictions are masked. - -### 4. Categorical cardinality and hashing - -Use `cardinality_config` to limit the number of distinct IDs created for a -categorical column. Each key is a column name. For example, keep the 10,000 most -frequent product IDs and hash every other product ID into 4,096 buckets: - -```yaml -categorical_columns: [product_id] -cardinality_config: - product_id: - top_k: 10000 - hashing: - num_buckets: 4096 - seed: 42 -``` - -`top_k` retains at most that many values, ranked by frequency. Alternatively, -`min_freq` retains every value occurring at least that many times. Both must be -positive integers and cannot be used together. Retained values keep individual -IDs; all remaining non-null values, including values first seen in later -splits, map to a hash bucket when `hashing` is configured. Without `hashing`, -non-retained values map to `[other]` instead. To hash all ordinary categorical -values, omit both retention settings: - -```yaml -cardinality_config: - product_id: - hashing: - num_buckets: 4096 -``` - -`num_buckets` is a positive integer. The optional `hashing.seed` defaults to -`0` and is separate from the top-level preprocessing `seed`. Hashing is stable -for strings, booleans, and integers. Multiple values can land in the same -bucket, so their original identities cannot be recovered from preprocessed -data; decoded bucket values appear as `[hash_bucket:k]`. Null values retain the -reserved `[unknown]` ID. - -By default, frequencies and retained values are fitted on split 0. Set -`normalize_on_all_data: true` to fit them on all data. A configured column must -be categorical, must appear in `selected_columns` when that list is set, and -cannot also appear in `use_precomputed_maps`. If `categorical_columns` is set, -include the configured column there. Preprocessing hashing is independent of -the model interface's `categorical_hashing` setting described in the -[training guide](train.md#categorical-multi-hash-embeddings-and-targets). - -### 5. Advanced: Static Vocabularies (Custom ID Maps) - -By default, Sequifier dynamically builds ID maps from the data found in the input file. However, in production systems, you often need a **fixed vocabulary** to ensure that ID "105" always maps to "Item_X", regardless of the daily training batch. - -To use a static vocabulary: -1. Create a folder `configs/id_maps/` in your project root. -2. Add JSON files named `{COLUMN_NAME}.json`. -3. The format must be a dictionary mapping ordinary data values to integers **starting at 3**. Reserved labels may be included only with their fixed IDs. - -> **Reserved Indices:** -> * **0**: Reserved for `[unknown]` (padding/missing). -> * **1**: Reserved for `[other]` (unseen values not in your map). -> * **2**: Reserved for `[mask]`. -> * **3+**: Your data. - -**Example `configs/id_maps/itemId.json`:** -```json -{ - "apple": 3, - "banana": 4, - "cherry": 5 -} -``` ------ - -## Outputs - -After running `preprocess`, the following are generated: - -1. **Data Files:** Located in `data/`. Depending on your configuration, these will be merged files such as `[NAME]-split0.parquet` (Training), `[NAME]-split1.parquet` (Validation), etc., or split folders such as `[NAME]-split0/` containing `.pt` or `.parquet` shards. -2. **Metadata Config:** Located in `configs/metadata_configs/[NAME].json`. - * **Crucial:** This file contains the integer mappings for categorical variables (`id_maps`), statistics for real variables (`selected_columns_statistics`), and whether those variables were normalized (`normalize_real_columns`). - * **Next Step:** Reference this file from `dataset.part.metadata_config_path` in a singleton training config, or from `dataset_training..parts..metadata_config_path` in a named training config. In inference, either `preprocessing_data_path` or `metadata_config_path` can locate the metadata and its split paths. - -## Named depth layouts - -Repeated rows can describe one outer item with a fixed-capacity child collection. -Depth behavior is explicit; a column named `subItemPosition` alone remains an -ordinary flat feature. - -```yaml -project_root: . -preprocessing_data_path: data/raw-items -read_format: parquet -write_format: pt -merge_output: false -selected_columns: [accountType, subitemType, subitemAmount, nextAction] -depth_layouts: - subitems: - position_column: subItemPosition - columns: [subitemType, subitemAmount] - context_length: 16 - position_base: 0 - allow_gaps: false -window_length: 129 -max_target_offset: 1 -split_ratios: [0.8, 0.1, 0.1] -window_stride: 128 -``` - -Every file must contain the depth position column, which is read automatically -and excluded from feature statistics and output types. Configured -`curriculum_column` values are also read automatically and cannot be used as -the depth position or a depth feature. Configure at most one raw layout, use PT output -without merging, and omit `mask_column`. Reused metadata must have the same -complete layout definition, output types, and normalization policy. Sequence -identifiers and item positions must have integer source types and fit signed -Int64; curriculum and depth positions must have integer source types. - -Within each source file, rows for one `(sequenceId, itemPosition)` must be -adjacent, and distinct item coordinates must increase in that order. An item -may occur in only one file. Files may cover interleaved coordinate ranges; -the adapter merges their ordered item streams without a temporary database. -For large file counts, the merge uses bounded fan-in and temporary sorted runs. -`max_rows` counts complete outer items in global coordinate order. Ordering -and cross-file uniqueness are checked across the full input, including items -beyond `max_rows`. Shallow features must agree across every child row before -casting or mapping. Shallow statistics count each item once; deep -statistics count occupied child slots. Both populations are selected before -split extraction. Materialization encodes each item once and uses bounded -windows and output batches; `batches_per_file` bounds the number of windows -accumulated per split. For a single Parquet file with multiple row groups and -no `max_rows`, this adapter uses `n_cores` to parallelize auditing, metadata -fitting, and window materialization. Other depth inputs use the sequential -merge path. - -Child positions map to physical slots by subtracting `position_base`. Without -`allow_gaps`, occupied slots must be a prefix starting at zero. Tail padding is -always allowed. With gaps enabled, physical slots remain unchanged. Outer item -positions must be continuous within each selected sequence. An item in this raw -format must have at least one child; null child rows do not encode emptiness. - -All PT files use version 5 of the `sequifier_tensor_batch` envelope. The payload -stores shallow tensors as `[N,W]`, deep tensors as `[N,W,D]`, boolean masks under -`metadata.depth_valid_masks.`, and the absolute window and split-boundary -positions needed to enforce split ownership. Optional signed Int64 curriculum -values are stored under `metadata.sample_positions`; their source names are -stored under `metadata.curriculum_columns`, and multiple preserved columns use -shape `[N,C]`. Public metadata records the model-facing -`tensor_payload_version` separately from this internal storage envelope. Readers -reject legacy tuple payloads and envelope versions 2 through 4; re-run -preprocessing to migrate them. -Categorical padding is the existing unknown-token ID (zero); real padding is -finite zero after normalization. Temporal padding has false depth masks. - -External tensor payloads may contain several layouts with different capacities -and independently empty collections. Use `StoredTensorBatch`, `save_pt_payload`, -and `load_pt_payload` in `sequifier.io.pt_payload`, supplying the complete layout -registry and categorical vocabulary sizes. All stored feature values, including -masked slots, must have legal categorical indices and finite real values. Empty -collections use an all-false mask. Missing masks and forbidden internal gaps are -errors. Selected interfaces compare only relevant layout feature membership and -layout properties, so unused stored layouts/features can be added independently. - -To consume a named layout during training, configure a `depth_transformer` -ingestion branch whose `layout` names this entry and whose `columns` are selected -features from it. See [the training guide](train.md#depth-encoders-and-nested-composites). - - -# Train Command Guide - -`sequifier train` trains one shared transformer backbone through one or more -named model interfaces. An interface is an ingestion module, its generated -adapter, and a decoder. Datasets own data and training policy; `model` -owns architecture. - -```console -sequifier train --config-path configs/train.yaml -``` - -## Singleton configuration - -When a run has one model interface, one dataset, one part, and one phase, those -values can be authored directly without routing names or references: - -```yaml -project_root: . -model_name: event-model -device: cuda -seed: 1010 - -global_training: - read_format: parquet - training_objective: causal - context_length: 128 - inference_batch_size: 256 - batch_size: 64 - learning_rate: 0.0001 - -model: - backbone: - architecture: - dim_model: 128 - max_context_length: 512 - num_layers: 6 - attention: {type: mha, n_heads: 8, n_kv_heads: 8, output_projection: true} - feed_forward: {dim: 512, activation: swiglu} - normalization: {type: rmsnorm, norm_first: true} - position_encoding: {type: rope, theta: 10000} - dropout: 0.1 - shared_layer_groups: [] - interface: - input_columns: [event] - target_columns: [event] - ingestion: {type: embedding, output_dim: 128} - decoder: {type: linear, prediction_length: 1, support: 1} - -dataset: - part: {metadata_config_path: configs/metadata_configs/events.json} - criterion: {event: CrossEntropyLoss} - -training_plan: - epochs: 5 - -evaluation: true -``` - -The concise form expands before validation: - -| Authored field | Canonical form | -| --- | --- | -| `model.interface` | `model.interfaces.default` | -| `dataset.part` | `dataset_training.default.parts.default` | -| `training_plan.epochs` | One sequential phase named `train` | -| `evaluation: true` | Evaluate the inferred single source | - -These shortcuts can be used independently. A unique interface and training -source are inferred; multiple interfaces or datasets require explicit names and -references. Singular and plural spellings cannot be combined. Because errors -are reported after expansion, they may use the canonical paths above. - -To condition every model window on features from a separate per-sequence -table, preprocess with `static_data_path` and configure the interface: - -```yaml -model: - interface: - input_columns: [event] - static_conditioning: - columns: [account_type, account_age] - feature_embedding_dims: {account_type: 16, account_age: 8} - target_columns: [event] - ingestion: {type: embedding, output_dim: 128} - decoder: {type: linear, prediction_length: 1, support: 1} -``` - -The static columns are separate from temporal `input_columns` and targets. -They are looked up by `sequenceId` for every sampled window, including windows -from the middle of a sequence. -`feature_embedding_dims` assigns a width to each static feature; real features -with width greater than one use a learned projection. Alternatively, -`embedding_size` distributes a total width automatically when all static -features are categorical or all are real. For mixed types, specify -`feature_embedding_dims`. If neither option is set, categorical widths use -the existing cardinality-based default and real features use one channel. -The concatenated features are projected to the backbone `dim_model`. -The model creates one prefix token at runtime -and removes its output before decoding. Set backbone `max_context_length` to -at least `global_training.context_length + 1`. - -## Canonical configuration - -```yaml -project_root: . -model_name: event-model -device: cuda -seed: 1010 - -global_training: - read_format: parquet - training_objective: causal - context_length: 128 - target_offset: 1 - window_stride: 1 - inference_batch_size: 256 - batch_size: 64 - accumulation_steps: 4 - learning_rate: 0.0001 - optimizer: {name: AdamW, weight_decay: 0.01} - scheduler: {name: StepLR, step_size: 1, gamma: 0.99} - scheduler_step_on: epoch - reset_optimization_on_phase: true - gradient_clip: 1.0 - save_interval_epochs: 1 - -model: - backbone: - architecture: - dim_model: 128 - max_context_length: 512 - num_layers: 6 - attention: {type: mha, n_heads: 8, n_kv_heads: 8, output_projection: true} - feed_forward: {dim: 512, activation: swiglu} - normalization: {type: rmsnorm, norm_first: true} - position_encoding: {type: rope, theta: 10000} - dropout: 0.1 - shared_layer_groups: [] - interfaces: - event_prediction: - input_columns: [event] - target_columns: [event] - categorical_decoder_special_tokens: {event: [other]} - ingestion: {type: embedding, output_dim: 128} - decoder: {type: linear, prediction_length: 1, support: 1} - -dataset_training: - events: - model_interface: event_prediction - parts: - original: {metadata_config_path: configs/metadata_configs/events.json} - increment: {metadata_config_path: configs/metadata_configs/events-increment.json} - criterion: {event: CrossEntropyLoss} - loss_weights: {event: 1.0} - freeze: - backbone: {freeze: [attention.qkv]} - -training_plan: - phases: - - name: incremental_finetuning - epochs: 2 - mode: sequential - sources: [{source: events.increment}] - - name: complete_retraining - epochs: 5 - mode: interleaved - selection: round_robin - sources: [{source: events, batches_per_selection: 4}] - -evaluation: - sources: [{source: events}] - -export_generative_model: true -export_embedding_model: false -export_onnx: true -export_pt: false -export_with_dropout: false -``` - -The historical flat training schema is not accepted. In particular, -`training_spec`, top-level dataset paths/columns, `model.ingestion`, -`model.decoder`, and architecture-owned freezing are not canonical fields. - -For a causal objective, `target_offset` sets the target position relative to -each input position. With `target_offset: 1`, every decoded position predicts -its next token. A decoder `prediction_length` greater than 1 selects several -consecutive positions for inference; it does not make one position predict -several future tokens as in LLM multi-token prediction. Training currently -computes loss over all valid decoded positions, regardless of -`prediction_length`. - -## Ownership and resolution - -- `global_training` owns objective, window, optimizer, precision, - distribution, compilation, checkpoint, and data-loader behavior. Phase - entries own `epochs`. -- `model` contains exactly one backbone and one or more named interfaces. - Different interface names create distinct ingestion and decoder weights; - repeated references to one name share those weights. -- `dataset_training` owns parts, criterion/weights, class-share logging, - freezing, and the interface reference. -- Preprocessing metadata owns split paths, data types, class counts, ID maps, - special-token IDs, normalization facts, and stored-window layout. - -For categorical hard-label targets, set a target's criterion to -`AccurateCrossEntropyLoss` to preserve small loss values and target-logit -gradients at high confidence. `CrossEntropyLoss` retains the existing PyTorch -behavior. For example: - -```yaml -dataset_training: - events: - criterion: {event: AccurateCrossEntropyLoss} -``` - -The choice is per target and is checked when resuming a run. Class -weights and `loss_weights` retain their usual meaning: the target class weight -multiplies each example's loss, then the target loss weight scales the masked -sum divided by the number of valid positions. The criterion also works with -hashed target heads, which divide the target loss weight among their heads; -class weights remain unsupported for hashed targets. - -Every part of a dataset must resolve to the same schema, categorical semantics, -normalization contract, per-split window widths, alignment settings, and -file/folder storage form. Training validates the model view against each -selected training or evaluation split width. A source -named `events` iterates all parts in declaration order; `events.increment` -iterates only that part. Only parts selected by `evaluation.sources` require a -validation split. - -### Categorical multi-hash embeddings and targets - -Add `categorical_hashing` to a model interface, keyed by categorical column -name. The setting applies to an input column's embedding and, if the column is -also a target, to its prediction heads. For example, add these fields to a -training config with a backbone and dataset part: - -```yaml -model: - interfaces: - default: - input_columns: [accountId, merchantId] - target_columns: [accountId] - categorical_hashing: - accountId: - type: multi_hash - num_buckets: 50000 - num_hashes: 4 - seed: 1010 - merchantId: - type: qr - num_buckets: 1000 - ingestion: - type: embedding - output_dim: 128 - decoder: - type: linear - prediction_length: 1 - support: 1 -dataset_training: - default: - model_interface: default - criterion: - accountId: CrossEntropyLoss - loss_weights: - accountId: 1.0 -``` - -Here `accountId` uses both hashed input embeddings and hashed target heads; -`merchantId` uses hashed input embeddings only. `multi_hash` creates -`num_hashes` independently seeded tables of `num_buckets` rows and adds their -outputs. `num_hashes` must be positive; `seed` defaults to 0. `qr` uses the -quotient and remainder of the category ID as two table indices and multiplies -their embeddings. Its hash count is always two. Do not add `hashing` under -`ingestion`: the embedding ingestion reads the interface setting, including -when used by `temporal_conv` with `base_ingestion: embedding`. - -When `feature_embedding_dims` is omitted for a categorical-only embedding -ingestion, each column receives at least two dimensions. The remaining -`output_dim` positions are split in proportion to the logarithm of each -column's cardinality, rounding down first and assigning leftover positions -by largest fractional remainder (ties follow column order). For -`multi_hash`, the effective cardinality is the smaller of the original -cardinality and `num_buckets` per table. Other categorical columns use their -original cardinality. The output width must be at least twice the number of -categorical columns. Explicit `feature_embedding_dims` still controls each -column's width directly. - -A hashed target requires `CrossEntropyLoss` or `AccurateCrossEntropyLoss`. Its -loss weight is divided equally across the hash heads; `class_weights` cannot be -used for that target. The -stored data still contains the original categorical IDs, without extra hash -columns. Hash codes for classes eligible for prediction must be unique; config -resolution reports a collision if the chosen bucket count and hash count do -not distinguish them. - -In an `autoregressive_transformer` decoder branch, a hashed target must be the -last target in that branch's `target_columns` order. Earlier targets use a -full-size feedback embedding inside the decoder. - -Folder dataset parts accept `file_order: shuffled` (the default) or -`file_order: name`. For curriculum training, these respectively reshuffle file -blocks each epoch or keep them in lexicographic path order; curriculum order is -local to each file. Without curriculum training, eager `shuffled` loading keeps -the ordinary global sample shuffle. The setting has no effect on single-file -parts. -`training_plan` also accepts `curriculum_training: false` (the default). When -set to `true`, `curriculum_column` must name one column available in the -preprocessed data. Training visits lower values first inside each physical file -and reshuffles equal-valued samples each epoch. Curriculum ordering applies only -to training loaders for parts selected by the training plan; validation keeps -its ordinary unshuffled order. Ordering is intentionally not synchronized -across files, loader workers, or distributed ranks. -`curriculum_column` defaults to `null` and cannot be set when curriculum -training is disabled. When disabled, curriculum values are ignored and the -ordinary sample shuffle is preserved. - -```yaml -dataset: - part: - metadata_config_path: configs/metadata_configs/events.json - file_order: name -training_plan: - curriculum_training: true - curriculum_column: difficulty - epochs: 5 -``` - -`loss_weights` scales each target's contribution to the training and reported -aggregate loss. A weight of `0.0` disables that target's backward-loss -component while retaining its output and per-target accounting. At least one -target in each dataset must have a positive weight. - -## Optimizer plans - -The single-optimizer form above continues to apply one optimizer to all -parameters and uses `global_training.learning_rate`. To select parameters or -use multiple optimizers, replace it with an ordered plan. Each group has its -own learning rate; omit the run-wide `learning_rate` in this form: - -```yaml -global_training: - optimizer: - groups: - - id: hidden_matrices - select: - semantic_groups: ["attention.*", "feed_forward.*"] - component: backbone - parameter_kind: weight - ndim: 2 - optimizer: {name: Muon, momentum: 0.95} - learning_rate: 0.02 - - id: remainder - select: otherwise - optimizer: {name: AdamW, weight_decay: 0.01} - learning_rate: 0.0003 -``` - -Each supplied selector condition must match. `semantic_groups` matches the -parameter catalog's semantic names using shell-style `*` patterns; a parameter -matches the list if it matches any pattern. Other available conditions are -`component` (`ingestion`, `backbone`, or `decoder`), `parameter_kind` (`weight`, -`bias`, or `other`), and `ndim`. At least one condition is required. The final -group must be `otherwise`, which receives every parameter not selected above. -Routes must be disjoint, all groups must receive parameters, and each trainable -parameter is assigned once. Group IDs are available to integration directives. - -The plan also supports different settings for one optimizer, for example two -AdamW groups with different weight decay. The named optimizer must be available -in Sequifier's optimizer registry. Muon availability depends on the installed -PyTorch version. The configured scheduler applies to every group; training -metrics report the first group's learning rate. With `OneCycleLR`, each group's -`learning_rate` is its peak rate. Omit `scheduler.max_lr` in plan mode; -Sequifier supplies the group rates in plan order. `OneCycleLR` derives the -starting rates from those peaks and `div_factor`. - -With `CyclicLR`, each group's `learning_rate` is also its peak rate. Add a -`base_learning_rate` below that peak to every optimizer group, and omit -`scheduler.base_lr` and `scheduler.max_lr`. For the plan above, the two base -rates could be `0.002` and `0.00003`, respectively. Configure the scheduler -with `name: CyclicLR`, `step_size_up: 2000`, and `scheduler_step_on: batch`. -Momentum cycling defaults to off for both cycle schedulers in plan mode; -explicit `cycle_momentum: true` is unsupported. `base_learning_rate` is only -valid with `CyclicLR`. - -## Optimization across phases - -By default, every training phase starts with a new optimizer, scheduler, and -gradient scaler. Model weights and global progress counters continue across -phase boundaries, while optimizer moments and scheduler position do not. Set -`global_training.reset_optimization_on_phase: false` to keep the optimizer, -scheduler, and gradient scaler operating continuously across all phases. A -checkpoint restores the active optimization lifecycle exactly in either mode. - -For an epoch-stepped scheduler in reset mode, do not provide -`global_training.scheduler.total_steps` because Sequifier owns each phase's -scheduler duration. When a scheduler such as `OneCycleLR` accepts `total_steps`, -Sequifier sets it to the active phase's `epochs` in reset mode, or the sum of all -phase epochs in continuous mode. A continuous epoch-stepped scheduler may still -provide `total_steps` for compatibility, but it must equal the sum of all phase -epochs. Other scheduler arguments remain shared across phases. This example uses -a single optimizer: - -```yaml -global_training: - scheduler: - name: OneCycleLR - max_lr: 0.003 - pct_start: 0.2 - cycle_momentum: false - scheduler_step_on: epoch - reset_optimization_on_phase: false -``` - -Batch-stepped schedulers may configure `total_steps` explicitly because phase -update counts can depend on source interleaving, dataset transitions, and -gradient accumulation. In reset mode, that value applies independently to each -phase; configuration emits a warning to make that scope explicit. In continuous -mode, it applies once across the full run. - -## Training plans - -A sequential phase exhausts each source in listed order. An interleaved phase -uses `round_robin` or `weighted_random`. Each selection consumes at most -`batches_per_selection`; every source is still exhausted once per phase epoch. -Weights affect order and burst frequency, not the amount of data consumed. - -Gradient accumulation may cross part or source boundaries only while the -dataset stays the same. A dataset transition flushes a partial window using its -actual microbatch count. Dataset-specific frozen gradients are removed before -the optimizer step, preventing momentum and weight decay from changing frozen -parameters. - -With several evaluation sources, configure an explicit monitor when -validation-based saving or early stopping is enabled: - -```yaml -evaluation: - sources: [{source: events}, {source: telemetry.main}] - monitor: {source: events, metric: loss, mode: min} -``` - -Checkpoint interval fields retain distinct behavior. `save_interval_epochs`, -`save_interval_batches`, and `save_interval_minutes` create persistent -epoch/batch snapshots and refresh the rolling `latest` checkpoint; -`save_latest_interval_minutes` refreshes only `latest`. -`save_interval_val_loss` controls whether timed or batch snapshots also run -validation and record its monitored loss. Every checkpoint is written after -flushing any partial gradient-accumulation window. - -## Composable YAML files - -An entry file may declare complementary fragments with -`additional_config_paths`. Relative paths resolve against the entry file's -`project_root`. Fragments can contribute disjoint children under containers -such as `global_training`, `model.interfaces`, and -`dataset_training`; duplicate fields are rejected. CLI overrides are -applied after composition and before metadata resolution. - -Singleton fragments may likewise contribute disjoint fields under -`model.interface` or `dataset`. Normalization occurs after fragment -composition, so all fragments in one training config must consistently use the -singular or named spelling at each level. - -The training command accepts `--model-name`, `--seed`, and `--skip-metadata` as -configuration overrides. Dataset paths, columns, metadata paths, and device -selection must use their canonical YAML locations. - -## Artifacts - -ONNX is exported by default; PT inference bundles are opt-in. ONNX favors a -portable deployment runtime, while PT embeds its execution contract and retains -PyTorch behavior. See the "ONNX or PT?" section of the inference guide for the -trade-offs. - -Single-dataset filenames use ``, while multi-dataset logs, metrics, and -ONNX files use `-`. Part names are metric-row fields, not -filename components. PT inference bundles and exact-resume checkpoints remain -run-wide. Generated filenames do not use a `sequifier-` prefix. - -The PT inference bundle contains `artifact_type`, `format_version`, -`model_state_dict`, an execution-only `model_config`, and export metadata -(trace-site names and provenance). Optimizers, paths, parts, training plans, -evaluation policy, and dataset bindings remain outside that bundle. -`export_with_dropout` affects ONNX export only: enabling it exports -the ONNX graph in training mode and disables constant folding so dropout remains -active. - -## Image ingestion - -Image views are declared during preprocessing and selected by name in an -ingestion branch. They can be the only temporal inputs (`input_columns: []`) or -can be merged with tabular branches. Each configured view is required for every -sequence. An image encoder returns one embedding per source image; its frame -pool reduces the images assigned to one `itemPosition` to one embedding. - -```yaml -model: - interfaces: - prediction: - input_columns: [temperature] - target_columns: [nextAction] - preconditioning_image_encoder: - type: frozen - model: resnet18 - weights: IMAGENET1K_V1 - ingestion: - type: composite - branches: - measurements: - type: embedding - columns: [temperature] - output_dim: 32 - cameras: - type: images - views: [front, side] - output_dim: 128 - encoder: {type: resnet, depth: 18} - frame_pool: attention - view_merge: concat - merge: {type: concat} - decoder: {type: linear, prediction_length: 1} -``` - -Bind the selected views to their original folders in each dataset part. The -preprocessing metadata deliberately omits folder paths. A single selected view -may use a string `image_path`; multiple views require a name-to-path map. -Preconditioning images use `preconditioning_image_path` in the same way: - -```yaml -dataset_training: - camera_data: - model_interface: prediction - parts: - main: - metadata_config_path: configs/metadata_configs/camera_data.json - image_path: {front: data/front_images, side: data/side_videos} - preconditioning_image_path: - front: data/sequence_front_images - side: data/sequence_side_images - criterion: {nextAction: CrossEntropyLoss} -``` - -Image view names and structured image cell names cannot be target columns; -the available decoder heads produce tabular targets. For BERT training, temporal -image frames at selected positions receive the configured masked, random, or -unchanged replacement. Masked frames become black and random frames receive -random pixel values; frame validity stays the same. Sequence preconditioning -images remain available as static context. - -The trainable `resnet` supports depths 18, 34, 50, 101, and 152, or a custom -`block` (`basic` or `bottleneck`) with four `layers` depths. Bottleneck models -also support `groups`, `width_per_group`, and `replace_stride_with_dilation`. -Trainable ResNets use per-image group normalization so later frames cannot -affect earlier image embeddings during causal training. -The `frozen` encoder supports TorchVision `resnet18` and `resnet50` with -an explicit weight version. Its backbone stays frozen and in evaluation mode; -its output projection trains. The default image size and normalization match -the V1 ImageNet weight transforms. ResNet-50 V2 requires `resize_size: 232` -on its source; model resolution checks this. `frame_pool` is `mean` (default) or -`attention`; `view_merge` is `concat` (default) or `mean`. - -Image embeddings may also occupy cells in an existing structured layout. The -cell names are part of `feature_layout`; `image_cells` maps those names to image -view names. The usual structured processing blocks then combine image and -tabular cells: - -```yaml -feature_layout: - sensors: - axes: {source: [camera, measurement]} - columns: - camera_cell: {source: camera} - temperature: {source: measurement} -ingestion: - type: structured - layout: sensors - output_dim: 128 - image_cells: {camera_cell: front} - image_encoder: {type: frozen, model: resnet50, weights: IMAGENET1K_V1} - processing_blocks: - - {type: axis_pool, axes: [source], mode: mean} -``` - -The preconditioning source is independent of temporal views. One or more -preconditioning views are encoded and merged into a single sequence prefix -embedding. When scalar static conditioning is also set, both are fused into -that one prefix position. - -## Depth encoders and nested composites - -Use `global_training.read_format: pt` for preprocessed depth inputs. Layout -capacities come from dataset metadata. The following singleton model fragment -combines one shallow branch with a depth encoder; its metadata comes from the -preprocessing example in [preprocess.md](preprocess.md#named-depth-layouts). - -At every outer time step, a `depth_transformer` branch embeds the valid child -slots from its named layout, applies a masked transformer across the child -dimension, and uses the CLS output as that collection's single temporal token. -The shared backbone then models relationships across outer time steps. - -```yaml -model: - backbone: - initialization_seed: 20260910 - architecture: - dim_model: 128 - max_context_length: 128 - num_layers: 4 - attention: {type: mha, n_heads: 8} - feed_forward: {dim: 512, activation: swiglu} - position_encoding: {type: rope} - interface: - input_columns: [accountType, subitemType, subitemAmount] - target_columns: [nextAction] - ingestion: - type: composite - branches: - item: - type: embedding - columns: [accountType] - output_dim: 32 - subitems: - type: depth_transformer - layout: subitems - columns: [subitemType, subitemAmount] - feature_embedding_dims: {subitemType: 48, subitemAmount: 16} - output_dim: 96 - dropout: 0.1 - initialization_seed: 12345 - initialization: - attention.qkv: - weight: {method: xavier_uniform, gain: 1.0} - free_parameter: - weight: {method: normal, mean: 0.0, std: 0.02} - architecture: - dim_model: 64 - num_layers: 2 - attention: {type: gqa, n_heads: 4, n_kv_heads: 2} - feed_forward: {dim: 256, activation: swiglu} - normalization: {type: rmsnorm, norm_first: true} - position_encoding: {type: learned} - dropout: 0.1 - merge: {type: concat} - decoder: {type: linear, prediction_length: 1} -``` - -For a `gated` or `attention` merge, `merge.softmax` accepts `standard` -(the default) or `max_excluding`. The latter uses the [max-excluding softmax](https://arxiv.org/abs/2610.31642) -to retain small gradients when one branch score dominates, at additional -compute cost. Set it independently for each composite, including nested ones. -`concat` and `sum` merges require `standard`. - -Depth encoders support learned, sinusoidal, or rotary positions; MHA, MQA, or -GQA; LayerNorm or RMSNorm; the shared feed-forward activations and layer-sharing -configuration. `architecture.dropout` controls depth position and transformer -sites. The ingestion-level `dropout` controls the pooled output. Mixed -categorical/real features require explicit feature widths; homogeneous features -can divide `architecture.dim_model` using the ordinary ingestion width rules. - -For categorical-only depth features, automatic allocation gives each column at -least two dimensions, so `architecture.dim_model` must be at least twice the -number of columns. To use smaller shares, set positive widths for every column -in `feature_embedding_dims`. For example, with two categorical columns, -`feature_embedding_dims: {first: 1, second: 1}` permits one dimension each. -Explicit widths may sum to a value different from `architecture.dim_model`; -the depth input projection handles the difference. The pooled projection handles -differences between `architecture.dim_model` and `output_dim`. - -CLS occupies position zero, and physical slot `s` occupies position `s+1`. -Empty collections have a learnable CLS-only representation. Depth columns -cannot be targets because there is no depth decoder head. BERT training masks -valid child slots at selected outer time steps, using the configured masked, -random, or unchanged replacement; padded child slots stay untouched. -Autoregressive depth inference is excluded. - -A composite branch may itself be a composite. Every nested composite requires -`output_dim`; an omitted root composite width retains the existing backbone -input width. Child widths resolve recursively before merge projections are -built. `allow_shared_columns` applies to the immediate children of the node -where it appears, including overlaps between their descendant features. -`allow_unused_input_columns` and `auxiliary_input_columns` belong at the root; -non-default child policies are rejected. Temporal per-feature positions occur -at leaf outputs only, and global temporal positions occur in the backbone. -Composite merges do not add another temporal position stage. - -## Ordered autoregressive target decoding - -Use `type: autoregressive_transformer` when outputs at the same temporal -position depend on one another. `target_columns` is an ordered contract: each -target is conditioned on the backbone state and all targets before it. Training -and validation use parallel teacher forcing; inference generates that prefix -greedily inside the decoder. Categorical and real targets can be mixed. - -```yaml -decoder: - type: autoregressive_transformer - prediction_length: 1 - support: 1 - target_columns: [patch_length, byte_0, byte_1, magnitude] - architecture: - dim_model: 256 - num_layers: 2 - attention: {type: mha, n_heads: 4} - feed_forward: {dim: 768, activation: swiglu} - normalization: {type: rmsnorm, norm_first: true} - position_encoding: {type: rope, theta: 10000} - dropout: 0.0 - shared_categorical_target_groups: [[byte_0, byte_1]] - tie_input_output_embeddings: true -``` - -Shared groups must contain categorical targets with identical decoder-ID -mappings. Weight tying aliases each categorical input table with its output -projection. A composite decoder may mix this branch with ordinary linear or -MLP branches. For variable-length byte patches, predict length first and encode -unused slots with an explicit padding category. - -For BERT objectives, autoregressive transformer targets must not include `mask` -in `categorical_decoder_special_tokens`. Inference excludes mask predictions, -which would invalidate the prefix used to generate later targets. The default -decoder vocabulary includes `other` and excludes `mask`. Configure an explicit -empty token list for a categorical target to exclude `other` as well. - -Initialization overrides inherit per semantic group and per weight/bias target. -A child overrides only the targets it specifies; `preserve` keeps the constructed -value. Parameters are initialized once per identity. The ingestion adapter -inherits the root ingestion policy. Branch overrides that older versions ignored -now take effect; exact resume of those older runs is rejected. - -`initialization_seed` accepts integers from zero through `2**63-1`, excluding -booleans. An explicit seed isolates constructor and custom-initializer randomness -from the run stream. Descendants derive seeds using SHA-256 over version 1, -namespace seed, relative branch path, and phase; explicit child seeds start a new -namespace. Omitted seeds preserve the existing RNG stream and initialization -traversal. Initialization seeds do not control runtime dropout. Dropout continues -using the run RNG and checkpointed per-rank state. - -Dataset freezing can follow the same tree: - -```yaml -dataset: - part: {metadata_config_path: configs/metadata_configs/raw-items.json} - criterion: {nextAction: CrossEntropyLoss} - freeze: - backbone: - freeze: [attention.qkv] - ingestion: - branches: - subitems: - freezing_except: [free_parameter, ingestion.output_projection] - item: - freeze: [] -``` - -A local child selector replaces the inherited decision, including selective -unfreezing. `freeze: []` unfreezes the scope; `freezing_except: []` freezes it. -A node containing only `branches` delegates without changing inherited decisions. -Merge parameters retain their parent's policy; the adapter has its own dataset -selector. Unknown branches, policies below leaves, and contradictory alias -selections fail. Freezing leaves dropout active and permits gradients through -frozen modules. At optimizer boundaries, frozen gradients are removed **before** -AMP unscaling/overflow detection, so a frozen-only overflow cannot suppress a -healthy active update. Momentum and weight decay cannot update parameters whose -gradients are absent. - - -# Infer Command Guide - -Models with `static_conditioning` read the processed static feature table -named in the preprocessing metadata. Each inference window receives the -configured static values for its `sequenceId`. -Autoregressive generation retains these values while temporal inputs slide -forward. - -Models with image ingestion need `image_path` for the original image or video -folders selected by the model. Use a string for one temporal view or a map from -view name to folder for multiple views. When the model has a preconditioning -image encoder, also set `preconditioning_image_path` to a string for the -`default` view or a map for named views. Preprocessing metadata records view -contracts and signatures, but omits folder paths. The folders must remain -available and unchanged after preprocessing. The inference loader applies the -same frame-to-`itemPosition` or time-column alignment and image transform used -during training. Timed sources also require the item-time index written by preprocessing. -PT and ONNX model execution accept image tensors and frame-valid masks; media -decoding stays outside the exported model. - -```yaml -image_path: {front: data/front_images, side: data/side_videos} -preconditioning_image_path: - front: data/sequence_front_images - side: data/sequence_side_images -``` - -`sequifier infer` produces predictions, probabilities, or embeddings from a -PyTorch (`.pt`) or ONNX (`.onnx`) model. - -```console -sequifier infer --config-path configs/infer.yaml -``` - -## Start here: ONNX - -> **GPU ONNX inference requires `onnxruntime-gpu`.** If you set `device: cuda`, -> install a build compatible with your CUDA and cuDNN versions. The CPU-only -> `onnxruntime` package does not provide `CUDAExecutionProvider`; setting -> `device: cuda` alone does not enable GPU inference. See the -> [ONNX Runtime CUDA compatibility guide](https://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html). - -ONNX is the default training export and the deployment-oriented inference path. -Exports embed their execution contract. Select a training route when providing -model configuration through a training config: - -```yaml -project_root: . -model_path: models/event-model-best-5.onnx -training_config_path: configs/train.yaml -dataset: events -data_path: data/events-test.parquet -model_type: generative -device: cuda -``` - -The route supplies columns, types, objective, window sizes, and preprocessing -metadata. Add `part` when the dataset has several parts. Current ONNX exports -also embed this contract and metadata, so the route is optional. Older exports -need a training route or the full contract and metadata supplied explicitly. - -## ONNX or PT? - -| | ONNX (default) | PT | -| --- | --- | --- | -| Best fit | Portable, deployment-oriented inference. | Python/PyTorch workflows and easier configuration. | -| Runtime | ONNX Runtime on CPU or CUDA (with a CUDA-enabled ONNX Runtime installation). | PyTorch on CPU, CUDA, or MPS. | -| Configuration | Current exports embed their contract and metadata; older exports need a training route or explicit values. | Embeds its contract and metadata. | -| Behavior | Runs the exported graph; dropout requires a dropout-preserving export. | Retains PyTorch behavior and supports self-describing, multi-interface bundles. | - -Benchmark the target workload rather than assuming either runtime is faster. -Choose PT when portability matters less than a compact, self-contained config: - -```yaml -project_root: . -model_path: models/event-model-best-5.pt -data_path: data/events-test.parquet -model_type: generative -device: cuda -``` - -For multi-interface PT, add `model_interface`. `model_type` stays explicit for -both formats because inference may generate outputs or extract embeddings. -`project_root` is required (normally `.`), and `inference_batch_size` defaults -to `1`. - -## Effective configuration and validation - -Sequifier resolves inference configuration in this order: - -1. Compose `additional_config_paths`, then apply CLI overrides. -2. Fill omitted model-contract fields from a selected training route and/or a - self-describing PT artifact. -3. Load metadata from an explicit metadata path, selected dataset part, or PT - artifact, in that order. -4. Apply defaults and validate the complete configuration. - -Explicit values are assertions, not silent overrides. If an authored column, -type, objective, context, prediction length, interface, or metadata value -disagrees with its training config or portable artifact, inference stops and names the -conflicting field and source. Multiple model paths must share one contract. - -Relative model, data, and metadata paths resolve under `project_root`. -`additional_config_paths` also resolve there; fragments cannot include further -fragments or define the same field twice. - -## Fields - -### Model, data, and routing - -| Field | Default | Purpose | -| --- | --- | --- | -| `model_path` | Required | Model path, or a list of compatible model paths. | -| `model_type` | Required | `generative` or `embedding`. | -| `device` | Required | ONNX: `cpu` or `cuda`; PT also supports `mps`. ONNX CUDA requires an installed runtime exposing `CUDAExecutionProvider`. | -| `project_root` | Required | Base for project paths; normally `.`. | -| `data_path` | Metadata test/last split | Input file or folder. Usually required with artifact-only inference because artifacts do not store split paths. | -| `preprocessing_data_path` | `null` | Derives the generated metadata path. | -| `metadata_config_path` | `null` | Explicit preprocessing metadata. | -| `image_path` | `null` | Original folder for one temporal view, or a view-to-folder map for every selected temporal view. Required when the model consumes temporal images. | -| `preconditioning_image_path` | `null` | Original folder for the `default` preconditioning view, or a view-to-folder map for named views. Required when the model uses preconditioning images. | -| `training_config_path` | `null` | Training config used to resolve a route. | -| `dataset` / `part` | `null` | Select a dataset and optional part from the training config. | -| `model_interface` | Implicit when unique | Select a route from a training config or portable artifact. | -| `read_format` | `parquet` | `csv`, `parquet`, or folder-based `pt`. | -| `write_format` | `csv` | `csv` or `parquet`. | -| `inference_batch_size` | `1` | Sequences processed per batch. | - -### Model contract - -The following fields are optional when supplied by a PT artifact or training -route, and otherwise required as applicable: `input_columns`, `target_columns`, -`column_data_types`, `target_column_types`, `training_objective`, -`context_length`, `target_offset`, and `prediction_length`. - -Preprocessing metadata may contain a Date/Datetime split column retained for -traceability. Inference ignores temporal metadata columns when `input_columns` -is omitted and rejects them if they are explicitly selected as model inputs or -targets. Other numeric columns from the same dataset remain usable normally. - -`window_stride` optionally evaluates several model windows inside each -stored preprocessing row. `null` uses the legacy right-aligned view. -`prediction_alignment_steps` defaults to `1` and must match the value in -dataset metadata. For aligned splits, it sets the number of consecutive -prediction groups evaluated per stored window. Aligned inference requires -`window_stride: null` and a stored width of at least `context_length + -target_offset + (prediction_alignment_steps - 1) × prediction_length`. -With `model_type: embedding`, aligned splits label each embedding by its target -`itemPosition` and omit positions outside the split. For causal models, the -embedding represents the context used to predict that row. Other splits label -embeddings by the input activation position. - -Folder inputs must carry `metadata.json` with matching storage and semantic -metadata, including category maps and normalization statistics. When using -metadata embedded in a portable artifact alone, inference can accept a wider -stored folder window if the remaining layout and semantics match. - -### Output and runtime options - -| Field | Default | Purpose | -| --- | --- | --- | -| `output_probabilities` | `false` | Write full distributions for categorical targets. Invalid for embeddings. | -| `decode_categories` | `true` | Decode categorical predictions to their original values. | -| `sample_from_distribution_columns` | `null` | Sample these categorical targets instead of using argmax. | -| `infer_with_dropout` | `false` | Enable dropout at inference; ONNX also requires dropout-preserving export. | -| `deterministic` | `false` | Request deterministic PyTorch algorithms. | -| `seed` | `1010` | Random seed. | -| `autoregressive` | `false` | Feed predictions back for multi-step generation. | -| `generation_steps` | `null` | Required positive step count when autoregressive is enabled. | - -For an `autoregressive_transformer` decoder branch, same-position targets are -generated greedily in configured `target_columns` order. Returned categorical -probabilities are conditional on the preceding greedy predictions; they are not -independent marginals that can be resampled into a coherent joint result. -Consequently, `sample_from_distribution_columns` must not include a target from -an autoregressive transformer branch. External sampling remains available for -ordinary decoder branches. - -Autoregressive inference requires a forward-looking generative model, prediction length -`1`, and identical input and target columns. For tabular CSV or Parquet input, -it begins at the first input window for each sequence and generates the same -number of steps for every sequence. Folder-based PT input is processed using -its stored windows rather than this first-window reduction. - -## CLI overrides - -`--data-path`, `--input-columns`, `--metadata-config-path`, `--model-path`, -`--seed`, `--dataset`, `--part`, and `--model-interface` override YAML values. -`--randomize` takes precedence over `--seed`. `--skip-metadata` skips external -metadata loading; inline metadata, a selected training route, or a self-describing -PT artifact must then provide the required metadata. - -## Outputs - -- Generative predictions: `outputs/predictions//part-NNN.`. -- Categorical probabilities, when enabled: - `outputs/probabilities///part-NNN.`. -- Embeddings: `outputs/embeddings//part-NNN.`. - -`` is the model artifact filename without its extension. Prediction and -embedding outputs contain sequence and window identifiers. Probability files -contain only the class-probability columns for their target. Categorical -predictions are decoded when `decode_categories` is enabled, and normalized -real predictions are restored to their original scale. Every input writes one -or more numbered parts. - -## Portable depth models and dropout modes - -New ONNX artifacts embed the selected interface, vocabulary/normalization -metadata, layouts, input name mapping, output descriptors, and fixed capacities. -They can resolve inference without the training YAML. Older flat ONNX files keep -the existing training-config/explicit-metadata fallback. - -The `depth_transformer` ingestion branch is part of the saved model contract; -inference does not configure it again. Inference supplies the matching named -layout tensors and masks through the preprocessed PT input. - -```yaml -project_root: . -model_path: models/item-model-best-5.onnx -model_type: generative -read_format: pt -data_path: data/raw-items-split2 -device: cpu -infer_with_dropout: false -seed: 1010 -``` - -Use the corresponding embedding artifact with `model_type: embedding`, or change -`model_path` to a portable PT model. `read_format: pt` describes preprocessed -input; it is independent of the model artifact extension. Deep inputs retain -`[B,T,D]` and explicit boolean depth masks through batching. New graphs accept -symbolic batch size with fixed temporal/depth capacities, including a partial -last batch. Legacy static graphs repeat features and masks together and trim -synthetic outputs. Empty input chunks do not invoke ONNX Runtime. - -New ONNX graphs have a fixed dropout mode: - -| `export_with_dropout` | `infer_with_dropout` | Behavior | -| --- | --- | --- | -| false | false | Evaluation graph; deterministic dropout behavior. | -| false | true | Error: export a stochastic graph. | -| true | true | Stochastic graph; runtime dropout remains enabled. | -| true | false | Error: export an evaluation graph. | - -PT artifacts retain executable modules and can switch either way at inference. -Stochastic ONNX CLI inference seeds ORT once before session creation, then lets -successive calls advance the session RNG. Separate processes with the same seed, -provider, toolchain, graph, and call/batch ordering define the intended -repeatability boundary. Cross-provider identity and PT/ORT matching random draws -are not promised. The direct `Inferer` API never reseeds the process-global ORT -RNG. Legacy ONNX graphs lacking mode metadata cannot make this guarantee and -emit a warning when stochastic inference is requested. - -Selected metadata must match layout names, selected deep membership, capacity, -position column/base, and gap policy. Input validation rejects missing masks, -incorrect ranks/capacities, forbidden gaps, illegal categories, and non-finite -values. Legal masked values are sanitized before narrowing and inside the graph. -The declared provider must be available; CUDA inference requires a CUDA-enabled -ORT installation. Outputs stay indexed by outer item coordinates. - - -# Visualize Training Command Guide - -The `sequifier visualize-training` command reads the structured metric files generated during training and hyperparameter search to create interactive Plotly HTML visualizations of the training and validation losses. It supports viewing a single model's progress or comparing multiple models side-by-side. - -## Usage - -```console -# Visualize a single model -sequifier visualize-training my-model-name - -# Visualize multiple models side-by-side -sequifier visualize-training model-A,model-B,model-C - -# Visualize every run from a hyperparameter search -sequifier visualize-training my-hyperparameter-search - -# Visualize models listed in a text file -sequifier visualize-training path/to/models.txt --log-scale - -``` - -## Arguments - -Unlike other commands that rely on a YAML config, `visualize-training` is configured directly via command-line arguments. - -| Argument | Type | Default | Description | -| --- | --- | --- | --- | -| `models` | `str` | **Required** | A model name, hyperparameter-search name, comma-separated list of model names, or path to a `.txt` file containing model names (one per line). A search name includes all models named `[SEARCH]-run-[NUMBER]`. | -| `--log-scale` | `flag` | `False` | Use a logarithmic scale on the y-axis for the loss curves. | -| `--bucket-training-batches` | `int` | `null` | Smooths the training loss curve by averaging the loss over a specified number of batches. **Must be a multiple of the logged batch interval** used during training. | -| `--project-root` | `str` | `.` | The root directory of your Sequifier project. | - -For a single-dataset model, the command reads `logs/[MODEL_NAME]/[MODEL_NAME]-training-full.csv` and `logs/[MODEL_NAME]/[MODEL_NAME]-validation-full.csv`. The corresponding files without `-full` contain only condensed global-loss records. - -## Outputs - -The interactive HTML reports are saved in the `outputs/visualization/` directory. - -* **Single Model:** `outputs/visualization/[MODEL_NAME]-training-visualization.html` (Includes global losses and normalized variable validation losses if applicable). -* **Multiple Models:** `outputs/visualization/multi-model-training-visualization.html` (Side-by-side comparison of validation and training losses across all specified models). -* **Hyperparameter Search:** `outputs/visualization/[SEARCH_NAME].html` (Includes all valid runs and lists skipped invalid runs and their reasons). - -If every run in a hyperparameter search is invalid, Sequifier still creates the report with an empty plot and the invalid-run list. - -When comparing multiple models, their initial baseline validation loss must match unless `SKIP_BASELINE_CHECK` or `SEQUIFIER_SKIP_BASELINE_CHECK` is set. - - -# Hyperparameter Search Command Guide - -`sequifier hyperparameter-search` searches over complete canonical training -configurations with Optuna. It supports Bayesian optimization, random sampling, -finite grid search, cooperative pruning, and custom single- or multi-objective -evaluation. - -```console -sequifier hyperparameter-search --config-path configs/hyperparameter-search.yaml -``` - -## Minimal search configuration - -Every search starts from a canonical training config named by -`base_config_path`. The `parameters` tree describes fixed replacements and -search spaces using the same paths as the training schema. - -```yaml -base_config_path: train.yaml -name: transformer-width-search -model_config_write_path: configs/hp-search -method: bayesian -trials: 40 - -parameters: - global_training: - context_length: [64, 128] - batch_size: [16, 32] - learning_rate: - low: 0.0001 - high: 0.001 - log: true - model: - backbone: - architecture: - num_layers: {low: 4, high: 8, step: 2} - training_plan: - epochs: [2, 4] -``` - -`base_config_path` resolves relative to the hyperparameter-search entry file. -The base may itself be a composed training config. If the search config supplies -`project_root`, that value is used in every generated trial; otherwise the base -training config's root is inherited. - -For a singleton base, use `model.interface`, `dataset.part`, and -`training_plan.epochs`. Named paths remain available for multi-value bases. - -The historical self-contained search schema and historical flat training base -configs are not accepted. Every generated trial is validated as an authored -canonical `SequifierConfig` before training begins, so unknown paths, invalid -references, incompatible component types, and cross-field violations fail with -their canonical validation paths. - -`model_name` and `project_root` cannot appear in `parameters`. Generated model -names use `[name]-run-[index]`, and `project_root` is controlled by the -top-level search field. - -## Parameter expressions - -Fields omitted from `parameters` retain their base values. Parameter expressions -have these forms: - -| Form | Meaning | -| --- | --- | -| Scalar | Fixed replacement. | -| List on a scalar field | Categorical choices. | -| `{low, high, step?, log?, type?}` | Integer or float distribution. `type` may be `int` or `float`; otherwise the base value and bounds determine it. | -| `{choices: [...]}` or `{$choices: [...]}` | Categorical choices for an entire value, including mappings and lists. | -| `{fixed: value}` or `{$fixed: value}` | Unambiguous fixed replacement for a mapping or list. | -| Numeric keys under a base list | Recursive replacement of zero-based list entries. | -| `{variants: [...]}` or `{$variants: [...]}` | Paired partial mapping variants, optionally followed by independently sampled sibling fields. | - -Integer ranges default to `step: 1`. Grid search requires a `step` for float -ranges because an unstepped float interval is infinite. Logarithmic integer -ranges require `step: 1`; logarithmic float ranges cannot use `step`. - -A direct list of lists samples a complete list-valued field. Prefer the explicit -`choices` wrapper for mapping or list candidates: - -```yaml -parameters: - model: - backbone: - architecture: - choices: - - dim_model: 128 - max_context_length: 512 - num_layers: 4 - attention: {type: mha, n_heads: 8} - feed_forward: {dim: 512, activation: swiglu} - - dim_model: 256 - max_context_length: 512 - num_layers: 6 - attention: {type: gqa, n_heads: 16, n_kv_heads: 4} - feed_forward: {dim: 1024, activation: swiglu} -``` - -Each choice replaces the complete overridden subtree, keeping coupled values -valid. The same mechanism can select complete ingestion or decoder components, -training phases, or other canonical subtrees. - -Use `variants` when candidates should patch the base mapping and other sibling -fields should remain independent. A variant that changes a component's `type` -starts from that new component shape rather than retaining fields belonging to -the old type. - -```yaml -parameters: - model: - interfaces: - event_prediction: - ingestion: - variants: - - {type: embedding, output_dim: 128} - - {type: passthrough, output_dim: 128} - dropout: [0.0, 0.1] -``` - -## Search controls - -| Field | Type | Required | Default | Description | -| --- | --- | --- | --- | --- | -| `base_config_path` | `str` | Yes | - | Canonical training config used as the base. | -| `parameters` | `mapping` | Yes | - | Recursive fixed values and search spaces. May be empty. | -| `project_root` | `str` | No | Base value | Root written to every generated training config. | -| `additional_config_paths` | `str`, `list[str]`, or `null` | No | `null` | Complementary direct fragments of the search config. Relative paths resolve against the entry config's `project_root`; recursive composition and duplicate fields are rejected. | -| `name` | `str` | Yes | - | Study name and generated-run prefix. | -| `model_config_write_path` | `str` | Yes | - | Directory, under `project_root`, for generated trial configs. | -| `method` | `bayesian`, `sample`, or `grid` | No | `bayesian` | Optuna TPE, random, or exhaustive finite-grid sampling. | -| `trials` | `int` | Except grid | - | Target total number of completed or pruned runs in the persisted study. | -| `global_seed` | `int` or `null` | No | `null` | Sampler seed. Training seeds belong in canonical `parameters`. | -| `prune_trials` | `bool` | No | `true` | Enables cooperative pruning. Distributed pruning remains experimental. | -| `pruning_warmup_epochs` | `int` or `null` | No | `null` | Complete epochs before pruning may begin. Mutually exclusive with batch warmup. | -| `pruning_warmup_batches` | `int` or `null` | No | `null` | Training batches before pruning may begin. Mutually exclusive with epoch warmup. | -| `evaluation_metrics` | `list[str]` or `null` | No | `null` | Metric names expected from a custom evaluation script. | -| `evaluation_metric_directions` | `list[minimize\|maximize]` or `null` | With metrics | `null` | One optimization direction per metric. | -| `evaluation_script` | `str` or `null` | With metrics | `null` | Script invoked with the best exported model's evaluation ID. | -| `evaluation_inference_config` | `str` or `null` | No | `null` | Inference config run before the custom evaluation script. | - -For grid search, omitting `trials` runs the complete finite grid. If it is -provided, it must exactly equal the grid size. For Bayesian and random search, -`trials` is a target total across invocations of the persisted study. If an -identical completed or pruned parameter set is proposed again, it is recorded -as a failed duplicate and does not consume a generated run number or count -toward the target. - -## Custom evaluation - -Without custom metrics, Sequifier minimizes the best validation loss. With one -metric it performs single-objective optimization; with several it records the -Pareto front. - -```yaml -evaluation_metrics: [accuracy, latency_ms] -evaluation_metric_directions: [maximize, minimize] -evaluation_inference_config: configs/infer-validation.yaml -evaluation_script: scripts/evaluate.py -``` - -The evaluation script receives the exported model's evaluation ID, formatted -as `-best-`, as its only -argument. It must write -`outputs/evaluations/[evaluation-id].json` under `project_root`, containing all -configured metric names. Extra metrics are allowed but produce a warning and -are ignored by the optimization. - -## CLI and outputs - -The search definition comes from YAML. `--skip-metadata` is the only -configuration-related command flag: it validates the canonical base without -loading metadata-derived values, so all required authored fields must already -be present. - -Generated canonical training configs are written below -`model_config_write_path`. The Optuna SQLite study is persisted at -`state/optuna/[name].db` under `project_root`, allowing later -invocations to continue toward the configured total. - - -# Distributed and Multi-Node Training in Sequifier - -Sequifier natively supports multi-GPU and multi-node training using PyTorch's `DistributedDataParallel` (DDP) and `FullyShardedDataParallel` (FSDP). - -## 1. Prerequisites: Preprocessing for Distributed Training - -To use distributed training, your data must be sharded into multiple files so that different GPUs can read different chunks simultaneously without memory bottlenecks. - -In your `preprocess.yaml`, you **must** write sharded output: - -```yaml -merge_output: false -``` - -For production multi-GPU training, use PyTorch tensor shards: - -```yaml -write_format: pt -``` - -*Note: Distributed training is not supported if your data is kept as a single `csv` or `parquet` file. You must use `merge_output: false` to generate a folder of sharded files.* - -> **Beta Notice for Parquet in Distributed Training:** -> While `write_format: parquet` is supported for distributed training, it is currently considered **Beta**. Because Parquet chunk reading relies on Polars' multi-threading, using it alongside PyTorch's multiprocess `DataLoader` in heavy multi-GPU environments can lead to CPU thread contention, high RAM usage, or NCCL timeouts. -> **Recommendation:** For production multi-GPU runs, use `write_format: pt`. It relies on native PyTorch serialization and is significantly more stable under heavy hardware loads. - - -## 2. Configuration: `train.yaml` - -Once your data is preprocessed into `.pt` shards, or beta `.parquet` shards, you need to tell the Sequifier training engine to expect a distributed environment. - -In your `train.yaml`, configure the canonical `global_training` block: - -```yaml -global_training: - read_format: pt # or parquet for beta sharded Parquet loading - distributed: true - data_parallelism: fsdp # or ddp - fsdp_cpu_offload: false # omit if using ddp; set true to offload FSDP parameters to CPU RAM - layer_type_dtypes: null # required for FSDP; use layer_autocast for mixed precision - torch_compile: inner # use inner or none for FSDP; use outer or none for DDP - world_size: 32 # The TOTAL number of GPUs across all nodes (e.g., 8 nodes * 4 GPUs = 32) - backend: nccl # 'nccl' is the standard and most efficient backend for NVIDIA GPUs - -``` - -When shards do not divide evenly across ranks, Sequifier automatically pads shorter ranks with repeated samples for step alignment. Those repeats are masked out of loss calculation, so each real sample contributes once. - -## 3. Launching the Training Job - -How you launch the training depends on whether you are using a single machine with multiple GPUs, or multiple machines (nodes) connected over a network. - -### Scenario A: Single-Node, Multi-GPU - -If you are running on a single machine that has multiple GPUs (e.g., an AWS EC2 instance with 4x A100s), Sequifier can handle process generation internally using `torch.multiprocessing.spawn`. - -You simply run the standard command: - -```bash -sequifier train --config-path configs/train.yaml - -``` - -Sequifier will read the `world_size` config parameter and automatically spawn that exact number of worker processes. - -### Scenario B: Multi-Node, Multi-GPU (HPC / Slurm) - -Sequifier cannot automatically spawn Python processes across physical network boundaries. For multi-node training, you must use an external cluster manager (like Slurm) combined with PyTorch's `torchrun` utility. - -When `sequifier` detects `torchrun` environment variables (like `RANK` and `WORLD_SIZE`), it bypasses its internal spawner and attaches to the distributed network established by the cluster. In that mode, the environment `WORLD_SIZE` is used. - -Here is a standard `sbatch` script template for launching Sequifier across multiple nodes: - -```bash -#!/bin/bash -#[SBATCH COMMANDS] - -MASTER_NODE=$(scontrol show hostnames $SLURM_JOB_NODELIST | head -n 1) - -srun torchrun \ - [-- torchrun args]... - $(which sequifier) train --config-path=configs/train.yaml -``` - -### Important Considerations for Multi-Node - -* **Batch Size:** The `batch_size` in your `train.yaml` is the **per-process** batch size. If `batch_size` is 100 and `world_size` is 32, each synchronized backward pass covers 3,200 samples. With gradient accumulation, the samples per optimizer update are `batch_size * world_size * accumulation_steps` (apart from a final partial accumulation window). -* **Learning Rate:** You may need to scale your `learning_rate` up if you drastically increase your global batch size via distributed training. -* **Data Access:** All nodes must have access to the same shared filesystem (e.g., NFS, GPFS) where the `project_root` and the sharded preprocessing output are stored. diff --git a/src/sequifier/contract_versions.py b/src/sequifier/contract_versions.py new file mode 100644 index 000000000..c377d51ae --- /dev/null +++ b/src/sequifier/contract_versions.py @@ -0,0 +1,67 @@ +"""Current versions scoped to individual Sequifier contracts.""" + + +class ContractVersions: + # Checkpoint envelope and runner-owned payloads. + RUN_CHECKPOINT = 100 + SUPERVISED_RECOVERY = 100 + POLICY_RECOVERY = 100 + POLICY_PHASE_STATE = 100 + LEARNING_RUN_STATE = 100 + + # Portable artifacts and training telemetry. + MODEL_ARTIFACT = 100 + BACKBONE_MANIFEST = 100 + PHASE_REPORT = 100 + RUN_MANIFEST = 100 + METRICS_SCHEMA = 100 + + INITIALIZATION_POLICY = 100 + INITIALIZATION_SEED_DERIVATION = 100 + + # Dataset storage and preprocessing resume records. + PT_TENSOR_BATCH = 100 + DATASET_TENSOR_METADATA = 100 + DEPTH_TENSOR_METADATA = 100 + STORED_WINDOW_LAYOUT = 100 + PREPROCESSING_MANIFEST = 100 + VIDEO_FRAME_INDEX = 100 + SERIAL_DEPTH_LEDGER = 100 + PARALLEL_DEPTH_LEDGER = 100 + + # Rollout records, algorithm archives, and offline source manifests. + FORECAST_TRAJECTORY = 100 + JOINT_POLICY_TRAJECTORY = 100 + COLLECTION_ARCHIVE = 100 + PPO_COLLECTION = 100 + GRPO_ARCHIVE = 100 + SEQUENCE_DATA_MANIFEST = 100 + SCALAR_POLICY_TRAJECTORY = 100 + + FORECAST_ROLLOUT_ADAPTER = 100 + ONLINE_POLICY_ADAPTER = 100 + MODEL_EXECUTION_SCHEMA = 100 + EXPORT_PREFLIGHT = 100 + + # Algorithm and implementation identities used in fingerprints and seeds. + HASH_CODEC = 100 + RESERVED_CODE_SPECIAL_TOKENS = 100 + INJECTIVE_SPECIAL_TOKENS = 100 + POLICY_COLLECTION_SEED = 100 + EXPORT_ATTENTION_LOWERING = 100 + CATEGORICAL_CONTROL = 100 + + PHASE_COMMIT = 100 + PHASE_BOUNDARY = 100 + + # External ONNX operator set. + ONNX_OPSET = 18 + + +FORECAST_ROLLOUT_PROTOCOL = "sequifier.forecast-rollout-adapter" +ONLINE_POLICY_PROTOCOL = "sequifier.online-policy-adapter" + + +def require_contract_version(contract: str, actual, expected: int) -> None: + if not isinstance(actual, int) or isinstance(actual, bool) or actual != expected: + raise ValueError(f"Unsupported {contract} version") diff --git a/src/sequifier/docs.py b/src/sequifier/docs.py index 0f16853ec..ac6536330 100644 --- a/src/sequifier/docs.py +++ b/src/sequifier/docs.py @@ -1,18 +1,44 @@ """Read the documentation bundled with the installed sequifier package.""" from importlib.resources import files +from pathlib import Path + +DOCUMENTATION_SOURCES = ( + "README.md", + "documentation/configs/preprocess.md", + "documentation/configs/train.md", + "documentation/configs/infer.md", + "documentation/commands/visualize-training.md", + "documentation/configs/hyperparameter-search.md", + "documentation/configs/naming.md", + "documentation/configs/ownership.md", + "documentation/training/multi-gpu-training.md", + "documentation/policy-learning.md", + "documentation/supervised-policy-learning.md", + "documentation/rollout.md", +) GUIDE_HEADINGS = { "preprocess": "# Preprocess Command Guide", "train": "# Train Command Guide", "infer": "# Infer Command Guide", + "policy": "# Policy learning", } +def source_documentation(root: Path) -> str: + return "\n\n".join( + (root / name).read_text(encoding="utf-8") for name in DOCUMENTATION_SOURCES + ) + + def docs(topic: str | None = None) -> None: """Print all documentation or the guide for a single command.""" + resource = files("sequifier").joinpath("consolidated-docs.md") content = ( - files("sequifier").joinpath("consolidated-docs.md").read_text(encoding="utf-8") + resource.read_text(encoding="utf-8") + if resource.is_file() + else source_documentation(Path(__file__).resolve().parents[2]) ) if topic is not None: diff --git a/src/sequifier/evaluation/compatibility.py b/src/sequifier/evaluation/compatibility.py new file mode 100644 index 000000000..828c7c0d1 --- /dev/null +++ b/src/sequifier/evaluation/compatibility.py @@ -0,0 +1,111 @@ +"""Resolved scoring contracts for supervised best-model selection.""" + +from pathlib import Path + +from sequifier.fingerprints import fingerprint + + +def evaluation_batch_size(phase): + evaluation = phase.evaluation + if evaluation is not None and evaluation.batch_size is not None: + return evaluation.batch_size + return phase.batch_size + + +def evaluation_fingerprint(config, phase_index): + """Exclude learning mechanics while retaining everything used for scoring. + + Contracts are compared within one run, whose model architecture and execution + precision are fixed. Dataset interfaces and validation bindings are included + explicitly so different routes or held-out sources cannot share a history. + """ + phase = config.training_plan[phase_index] + if phase.algorithm != "supervised" or phase.evaluation_monitor is None: + return None + sources = [] + for source in phase.evaluation_sources: + dataset = config.dataset_training[source.dataset] + interface = config.dataset_interface(source.dataset) + learning = phase.learning.datasets.get(source.dataset) + if learning is None: + return None + sources.append( + { + "source": source.source, + "interface": dataset.model_interface, + "interface_contract": { + **interface.model_dump(mode="json"), + **config.dataset_storage(source.dataset).model_dump(mode="json"), + }, + "parts": [ + { + "name": name, + "validation_path": str( + Path(dataset.parts[name].validation_data_path).resolve() + ) + if dataset.parts[name].validation_data_path is not None + else None, + "read_format": dataset.parts[name].read_format, + "storage_form": dataset.parts[name].storage_form, + "file_order": dataset.parts[name].file_order, + "metadata": dataset.parts[name].metadata.model_dump( + mode="json" + ), + "image_paths": dataset.parts[name].image_paths, + "preconditioning_image_paths": dataset.parts[ + name + ].preconditioning_image_paths, + } + for name in source.parts + ], + "criterion": learning.criterion, + "class_weights": learning.class_weights, + "loss_weights": { + target: (learning.loss_weights or {}).get(target, 1.0) + for target in interface.target_columns + }, + } + ) + runtime = config.runtime_settings + return fingerprint( + { + "model": config.model.model_dump(mode="json"), + "monitor": phase.evaluation_monitor.model_dump(mode="json"), + "sources": sources, + "objective": runtime.training_objective, + "context_length": runtime.context_length, + "target_offset": runtime.target_offset, + "window_stride": phase.learning.window_stride, + "bert_spec": phase.learning.bert_spec.model_dump(mode="json") + if phase.learning.bert_spec is not None + else None, + # EvaluationService seeds corruption by phase, epoch and batch index. + "mask_seed_phase": phase_index + if runtime.training_objective == "bert" + else None, + "next_occurrence_config": phase.learning.next_occurrence_config.model_dump( + mode="json" + ) + if phase.learning.next_occurrence_config is not None + else None, + "batch_size": evaluation_batch_size(phase), + "device": config.device, + "layer_type_dtypes": runtime.layer_type_dtypes, + "layer_autocast": runtime.layer_autocast, + "float32_matmul_precision": runtime.float32_matmul_precision, + "distributed": runtime.distributed, + "data_parallelism": runtime.data_parallelism, + "world_size": runtime.world_size, + "num_workers": runtime.num_workers, + } + ) + + +def compatible_supervised_phases(config, previous_index, current_index): + current = config.training_plan[current_index] + if current.model_selection.scope == "phase": + return False + previous = evaluation_fingerprint(config, previous_index) + return previous is not None and previous == evaluation_fingerprint( + config, current_index + ) diff --git a/src/sequifier/evaluation/service.py b/src/sequifier/evaluation/service.py index 9c15b8ea0..9c9783dfa 100644 --- a/src/sequifier/evaluation/service.py +++ b/src/sequifier/evaluation/service.py @@ -4,15 +4,19 @@ import json import time +from contextlib import nullcontext from dataclasses import asdict, dataclass, field, is_dataclass -from typing import Any, Sequence +from typing import Any import torch +from pydantic import BaseModel from torch.nn.functional import one_hot from sequifier.helpers import construct_index_maps from sequifier.model.network import ModelOutput +from sequifier.runtime.precision import PrecisionPolicy from sequifier.training.loss import LossService, PreparedBatch +from sequifier.training.plan import SupervisedPhaseView from sequifier.training.runtime import DatasetRuntimeRegistry, build_source_runtime @@ -66,20 +70,22 @@ class BaselineAccounting: class EvaluationService: - def __init__(self, loss_service: LossService) -> None: + def __init__( + self, loss_service: LossService, *, precision: PrecisionPolicy | None = None + ) -> None: self.loss_service = loss_service + self.precision = precision self._baselines: dict[tuple[str, str], BaselineAccounting] = {} @staticmethod def _baseline_definition(dataset: Any) -> str: """Identify inputs that affect raw baseline loss statistics.""" - interface = dataset.config.interface - training = getattr(dataset.objective.config, "global_training", None) + interface = dataset.interface + training = dataset.objective.config.runtime_settings def model_value(value: Any) -> Any: - dump = getattr(value, "model_dump", None) - if callable(dump): - return dump(mode="json") + if isinstance(value, BaseModel): + return value.model_dump(mode="json") if is_dataclass(value) and not isinstance(value, type): return asdict(value) return value @@ -87,13 +93,11 @@ def model_value(value: Any) -> Any: definition = { "objective": dataset.objective.name, "objective_options": { - "bert_spec": model_value(getattr(training, "bert_spec", None)), - "next_occurrence_config": model_value( - getattr(training, "next_occurrence_config", None) - ), + "bert_spec": model_value(training.bert_spec), + "next_occurrence_config": model_value(training.next_occurrence_config), }, "window_view": model_value(interface.window_view), - "prediction_length": getattr(interface.decoder, "prediction_length", None), + "prediction_length": interface.decoder.prediction_length, "target_columns": interface.target_columns, "target_column_types": interface.target_column_types, "target_decoder_ids": interface.target_decoder_ids, @@ -106,7 +110,7 @@ def model_value(value: Any) -> Any: def _baseline_output( self, prepared: PreparedBatch, dataset: Any, decoded_length: int, network: Any ) -> ModelOutput: - interface = dataset.config.interface + interface = dataset.interface logits = {} auxiliary_logits = {} for target in interface.target_columns: @@ -118,10 +122,12 @@ def _baseline_output( ).transpose(0, 1)[:, -decoded_length:] if interface.target_column_types[target] == "categorical": global_ids = values.to(torch.int64) - if target in getattr(interface, "categorical_hashing", {}): - codec = network.resolve_interface( - dataset.interface_name - ).decoder.hash_codecs[target] + if target in interface.categorical_hashing: + codec = ( + network.resolve_interface(dataset.interface_name) + .require_decoder() + .hash_codec(target) + ) codes = codec.encode(global_ids) auxiliary_logits[target] = tuple( one_hot(codes[..., index], width).to(torch.float32) @@ -146,7 +152,7 @@ def _baseline_output( def evaluate( self, network: Any, - sources: Sequence[Any], + phase: SupervisedPhaseView, datasets: DatasetRuntimeRegistry, context: EvaluationContext, ) -> EvaluationResult: @@ -154,13 +160,14 @@ def evaluate( was_training = network.training network.eval() try: - with torch.no_grad(): - for source_config in sources: + autocast = self.precision.autocast() if self.precision else nullcontext() + with torch.no_grad(), autocast: + for source_config in phase.spec().evaluation_sources: started = time.perf_counter() - source = build_source_runtime(source_config, datasets) + source = build_source_runtime(source_config, datasets, phase) source.set_epoch(context.phase_epoch, "validation") dataset = source.dataset - targets = list(dataset.config.interface.target_columns) + targets = list(dataset.interface.target_columns) dtype = ( torch.float32 if context.device.type == "mps" else torch.float64 ) @@ -186,10 +193,8 @@ def evaluate( target: torch.zeros( (), device=context.device, dtype=torch.int64 ) - for target in getattr( - dataset.config.interface, "categorical_hashing", {} - ) - if target in dataset.config.interface.target_decoder_ids + for target in dataset.interface.categorical_hashing + if target in dataset.interface.target_decoder_ids } accuracy_count = torch.zeros( (), device=context.device, dtype=torch.int64 @@ -365,7 +370,7 @@ def evaluate( } distributions: dict[str, list[dict[str, Any]]] = {} index_maps = construct_index_maps( - dataset.config.interface.id_maps, + dataset.interface.id_maps, list(class_counts), True, ) diff --git a/src/sequifier/export/onnx.py b/src/sequifier/export/onnx.py index 212e4b97d..3045d3e2f 100644 --- a/src/sequifier/export/onnx.py +++ b/src/sequifier/export/onnx.py @@ -17,6 +17,7 @@ import torch from torch import Tensor, nn +from sequifier.contract_versions import ContractVersions from sequifier.model.execution_schema import ( DROPOUT_MODE_KEY, EXECUTION_SCHEMA_KEY, @@ -308,10 +309,10 @@ def export( state = manager.capture_local() try: # All graph lowering and dtype conversion happen on a disposable copy. - model = copy.deepcopy(model).cpu().float() + model = copy.deepcopy(model).cpu() for module in model.modules(): if isinstance(module, SelfAttention): - module._sequifier_export_attention = True + module.export_attention = True schema = ExecutionSchema.from_interface( interface, context_length, embedding=interface_name is None ) @@ -356,7 +357,7 @@ def export( inputs, staged, export_params=True, - opset_version=18, + opset_version=ContractVersions.ONNX_OPSET, dynamo=True, external_data=False, optimize=False, @@ -449,7 +450,7 @@ def export( expected.detach().numpy(), actual, rtol=2e-4, atol=2e-5 ): raise ValueError( - "ONNX artifact differs from its FP32 export reference" + "ONNX artifact differs from its export reference" ) published = [] try: diff --git a/src/sequifier/export/preflight.py b/src/sequifier/export/preflight.py index a5c6cbef7..0dca9a99f 100644 --- a/src/sequifier/export/preflight.py +++ b/src/sequifier/export/preflight.py @@ -9,23 +9,24 @@ import subprocess import sys import tempfile +from dataclasses import replace from importlib.metadata import version from pathlib import Path from sequifier.artifacts.manifests import write_manifest from sequifier.artifacts.model_export import model_execution_config +from sequifier.contract_versions import ContractVersions +from sequifier.model.execution import ModelExecution -PROTOCOL_VERSION = 1 PREFLIGHT_SEED = 20260910 -def preflight_description(config): +def preflight_description(config, execution): import torch versions = { name: version(name) for name in ("torch", "onnx", "onnxscript", "onnxruntime") } - execution = model_execution_config(config) # Initial weights and their provenance do not affect graph capability. def graph_only(value): @@ -41,6 +42,7 @@ def graph_only(value): "selected_columns_statistics", "split_paths", "storage_layout", + "tensor_payload_kind", "tensor_payload_version", } } @@ -48,26 +50,27 @@ def graph_only(value): return [graph_only(item) for item in value] return value + export = config.publication.exports return { - "protocol_version": PROTOCOL_VERSION, + "protocol_version": ContractVersions.EXPORT_PREFLIGHT, "representative_batch_policy": "publication_clamped_2_to_4_preflight_2", "execution": graph_only(execution), - "generative": config.export_generative_model, - "embedding": config.export_embedding_model, + "generative": export.generative_model, + "embedding": export.embedding_model, "embedding_sites": config.embedding_layer_names, - "dropout_mode": "stochastic" if config.export_with_dropout else "evaluation", + "dropout_mode": "stochastic" if export.with_dropout else "evaluation", "export_policy": { "dynamo": True, - "opset": 18, + "opset": ContractVersions.ONNX_OPSET, "dtype": "float32", "external_data": "unique_external_files_graph_committed_last", "optimize": False, - "attention_lowering": "explicit_matmul_softmax_dropout_v1", + "attention_lowering": f"explicit_matmul_softmax_dropout_v{ContractVersions.EXPORT_ATTENTION_LOWERING}", "dynamic_shapes": "shared_batch_fixed_capacities", }, "versions": versions, "provider": "CPUExecutionProvider", - "session_optimization": "disabled" if config.export_with_dropout else "all", + "session_optimization": "disabled" if export.with_dropout else "all", "device": { "machine": platform.machine(), "system": platform.platform(), @@ -91,17 +94,26 @@ def graph_only(value): def ensure_export_preflight(config, distributed): - if not config.export_onnx or not any( - dataset.interface.depth_layouts - or dataset.interface.image_sources - or dataset.interface.preconditioning_image is not None + export = config.publication.exports + if config.publication.mode != "supervised" or not export.onnx: + return + execution = ModelExecution.from_config(config) + interfaces = { + dataset.model_interface: execution.interface(dataset.model_interface) for dataset in config.dataset_training.values() + } + if not any( + interface.depth_layouts + or interface.image_sources + or interface.preconditioning_image is not None + for interface in interfaces.values() ): return failure = None if distributed.rank == 0: try: - description = preflight_description(config) + contract = model_execution_config(replace(execution, interfaces=interfaces)) + description = preflight_description(config, contract) encoded = json.dumps( description, sort_keys=True, separators=(",", ":"), allow_nan=False ).encode() @@ -111,9 +123,17 @@ def ensure_export_preflight(config, distributed): record_path = cache / f"{fingerprint}.json" if record_path.exists(): record = json.loads(record_path.read_text()) + cached_description = record.get("description") + cached_protocol = ( + cached_description.get("protocol_version") + if isinstance(cached_description, dict) + else None + ) if ( record.get("success") is True - and record.get("description") == description + and isinstance(cached_protocol, int) + and not isinstance(cached_protocol, bool) + and cached_description == description ): return_record = True else: @@ -128,10 +148,10 @@ def ensure_export_preflight(config, distributed): request.write_text( json.dumps( { - "execution": model_execution_config(config), - "generative": config.export_generative_model, - "embedding": config.export_embedding_model, - "dropout": config.export_with_dropout, + "execution": contract, + "generative": export.generative_model, + "embedding": export.embedding_model, + "dropout": export.with_dropout, "seed": PREFLIGHT_SEED, } ) @@ -173,7 +193,7 @@ def _worker(request_path): import onnxruntime import torch - from sequifier.artifacts.model_config import resolved_config_from_model_config + from sequifier.artifacts.model_config import model_execution_from_model_config from sequifier.export.embedding import EmbeddingModelExporter from sequifier.export.onnx import OnnxModelExporter from sequifier.helpers import configure_determinism @@ -183,33 +203,31 @@ def _worker(request_path): request = json.loads(request_path.read_text()) configure_determinism(request["seed"]) onnxruntime.set_seed(request["seed"]) - config, _ = resolved_config_from_model_config(request["execution"], device="cpu") + config, _ = model_execution_from_model_config(request["execution"], device="cpu") network = build_transformer_network( config, device=torch.device("cpu") ).network.float() exporter = OnnxModelExporter() - for index, dataset in enumerate(config.dataset_training.values()): + for index, (name, interface) in enumerate(config.interfaces.items()): if request["generative"]: exporter.export( network, - dataset.interface, + interface, request_path.parent / f"{index}-generative.onnx", - interface_name=dataset.model_interface, + interface_name=name, batch_size=2, - context_length=config.global_training.context_length, + context_length=config.options.context_length, training=request["dropout"], ) if request["embedding"]: - embedding = EmbeddingModelExporter().build( - network, dataset.model_interface, config - ) + embedding = EmbeddingModelExporter().build(network, name, config) exporter.export( embedding, - dataset.interface, + interface, request_path.parent / f"{index}-embedding.onnx", interface_name=None, batch_size=2, - context_length=config.global_training.context_length, + context_length=config.options.context_length, training=request["dropout"], ) print( diff --git a/src/sequifier/export/service.py b/src/sequifier/export/service.py index b7c2bc430..0f8d3cb21 100644 --- a/src/sequifier/export/service.py +++ b/src/sequifier/export/service.py @@ -2,9 +2,8 @@ from __future__ import annotations -import copy import json -from dataclasses import dataclass, field +from dataclasses import dataclass, field, replace from pathlib import Path from typing import Any, Protocol @@ -16,6 +15,7 @@ from sequifier.export.pytorch import PyTorchModelExporter from sequifier.logging_paths import model_artifact_path from sequifier.model.embedding import ONNX_EMBEDDING_LAYER_NAMES_KEY +from sequifier.model.execution import ModelExecution from sequifier.model.execution_schema import MODEL_CONFIG_KEY from sequifier.model.factory import build_transformer_network from sequifier.runtime.random_state import RandomStateManager @@ -40,13 +40,6 @@ def export( ) -> Path: ... -def _onnx_execution_config(config, interface_name): - execution = model_execution_config(config) - execution["interfaces"] = {interface_name: execution["interfaces"][interface_name]} - execution["layer_type_dtypes"] = None - return execution - - class ExportService: def __init__(self, config: Any, rank: int) -> None: self.config = config @@ -72,11 +65,8 @@ def _export( if self.rank != 0: return ExportResult() del network - export_config = self.config.model_copy(deep=True) - export_config.device = "cpu" - export_config.global_training.distributed = False - export_config.global_training.data_parallelism = None - export_config.global_training.torch_compile = "none" + execution = ModelExecution.from_config(self.config) + export_config = replace(execution, device="cpu") export_network = build_transformer_network( export_config, device=torch.device("cpu"), initialize=False ).network @@ -86,7 +76,8 @@ def _export( validate_hash_contracts(export_network, export_config) export_network.eval() paths: list[Path] = [] - if self.config.export_pt and self.config.export_generative_model: + export = self.config.publication.exports + if export.pt and export.generative_model: path = model_artifact_path( self.config.project_root, self.config.model_name, @@ -97,7 +88,7 @@ def _export( export_network, self.config, path, options ) paths.append(exported_path) - if self.config.export_pt and self.config.export_embedding_model: + if export.pt and export.embedding_model: embedding_options = ExportOptions( f"{options.suffix}-embedding", options.epoch ) @@ -112,77 +103,66 @@ def _export( export_network, self.config, path, embedding_options ) ) - if self.config.export_onnx: - onnx_network = export_network - if any( - parameter.dtype in (torch.float16, torch.bfloat16, torch.float64) - for parameter in export_network.parameters() - ): - onnx_network = copy.deepcopy(export_network).float() - count = len(self.config.dataset_training) - for dataset_name, dataset in self.config.dataset_training.items(): - if self.config.export_generative_model: + if export.onnx: + contract = model_execution_config(execution) + settings = execution.options + datasets = self.config.dataset_training + dataset_count = len(datasets) + for dataset_name, dataset in datasets.items(): + name = dataset.model_interface + interface = execution.interface(name) + metadata = { + MODEL_CONFIG_KEY: json.dumps( + {**contract, "interfaces": {name: contract["interfaces"][name]}} + ), + ONNX_CATEGORICAL_TARGET_CODECS_KEY: json.dumps( + interface.target_decoder_ids + ), + } + if export.generative_model: path = model_artifact_path( self.config.project_root, self.config.model_name, f"{options.suffix}-{options.epoch}", "onnx", dataset_name=dataset_name, - dataset_count=count, + dataset_count=dataset_count, ) paths.append( self.onnx.export( - onnx_network, - dataset.interface, + export_network, + interface, path, - interface_name=dataset.model_interface, - batch_size=self.config.global_training.inference_batch_size, - context_length=self.config.global_training.context_length, - training=self.config.export_with_dropout, - metadata={ - MODEL_CONFIG_KEY: json.dumps( - _onnx_execution_config( - self.config, dataset.model_interface - ) - ), - ONNX_CATEGORICAL_TARGET_CODECS_KEY: json.dumps( - dataset.interface.target_decoder_ids - ), - }, + interface_name=name, + batch_size=export.batch_size, + context_length=settings.context_length, + training=export.with_dropout, + metadata=metadata, ) ) - if self.config.export_embedding_model: - embedding = self.embedding.build( - onnx_network, dataset.model_interface, self.config - ) + if export.embedding_model: + embedding = self.embedding.build(export_network, name, execution) path = model_artifact_path( self.config.project_root, self.config.model_name, f"{options.suffix}-embedding-{options.epoch}", "onnx", dataset_name=dataset_name, - dataset_count=count, + dataset_count=dataset_count, ) paths.append( self.onnx.export( embedding, - dataset.interface, + interface, path, interface_name=None, - batch_size=self.config.global_training.inference_batch_size, - context_length=self.config.global_training.context_length, - training=self.config.export_with_dropout, + batch_size=export.batch_size, + context_length=settings.context_length, + training=export.with_dropout, metadata={ - MODEL_CONFIG_KEY: json.dumps( - _onnx_execution_config( - self.config, dataset.model_interface - ) - ), - ONNX_CATEGORICAL_TARGET_CODECS_KEY: json.dumps( - dataset.interface.target_decoder_ids - ), + **metadata, ONNX_EMBEDDING_LAYER_NAMES_KEY: json.dumps( - self.config.embedding_layer_names + execution.embedding_layer_names ), }, ) diff --git a/src/sequifier/fingerprints.py b/src/sequifier/fingerprints.py new file mode 100644 index 000000000..2cec39eac --- /dev/null +++ b/src/sequifier/fingerprints.py @@ -0,0 +1,10 @@ +"""Deterministic identities for JSON-compatible content.""" + +import hashlib +import json + + +def fingerprint(value): + return hashlib.sha256( + json.dumps(value, sort_keys=True, allow_nan=False).encode() + ).hexdigest() diff --git a/src/sequifier/helpers.py b/src/sequifier/helpers.py index 2dd5ca642..af35ad7d0 100644 --- a/src/sequifier/helpers.py +++ b/src/sequifier/helpers.py @@ -16,6 +16,7 @@ from pydantic import ValidationError from torch import Tensor +from sequifier.contract_versions import ContractVersions, require_contract_version from sequifier.logging_paths import ( dataset_artifact_prefix, model_artifact_path, @@ -70,11 +71,7 @@ "UInt8": pl.UInt8, } -POLARS_NUMERIC_DTYPE_ALIASES = { - alias: canonical - for canonical in POLARS_NUMERIC_DTYPES - for alias in (canonical, canonical.lower()) -} +POLARS_NUMERIC_DTYPE_ALIASES = {name: name for name in POLARS_NUMERIC_DTYPES} FLOAT_TYPE_ORDER = ("Float16", "Float32", "Float64") INTEGER_TYPE_ORDER = ( @@ -231,10 +228,9 @@ class StoredWindowLayout: @beartype def __post_init__(self) -> None: - if self.version != 2: - raise ValueError( - f"stored_window_layout_version must be 2, got {self.version}" - ) + require_contract_version( + "stored-window layout", self.version, ContractVersions.STORED_WINDOW_LAYOUT + ) if self.window_length < 1: raise ValueError("window_length must be a positive integer") if self.max_target_offset < 0: @@ -596,8 +592,11 @@ def resolve_window_sampling_plan( @beartype def configured_window_stride(config: Any) -> Optional[int]: - """Read the optional stride from validated configs or legacy test doubles.""" - value = getattr(config, "window_stride", None) + """Read the optional stride from dataset or inference settings.""" + try: + value = config.window_stride + except AttributeError: + return None return value if isinstance(value, int) else None @@ -612,10 +611,15 @@ def validate_stored_window_width(tensor: Tensor, window_length: int) -> None: @beartype def stored_window_layout_from_metadata(metadata: dict) -> StoredWindowLayout: + require_contract_version( + "stored-window layout", + metadata.get("stored_window_layout_version"), + ContractVersions.STORED_WINDOW_LAYOUT, + ) return StoredWindowLayout( window_length=int(metadata["window_length"]), max_target_offset=int(metadata["max_target_offset"]), - version=int(metadata["stored_window_layout_version"]), + version=metadata["stored_window_layout_version"], ) @@ -945,6 +949,9 @@ def configure_logger( @beartype def configure_determinism(seed: int, strict: bool = False) -> None: """Enforces deterministic execution for reproducibility.""" + if strict: + # Set before CUDA RNG initialization can create a cuBLAS context. + os.environ["CUBLAS_WORKSPACE_CONFIG"] = ":4096:8" random.seed(seed) np.random.seed(seed) torch.manual_seed(seed) @@ -952,10 +959,7 @@ def configure_determinism(seed: int, strict: bool = False) -> None: torch.backends.cudnn.benchmark = False torch.backends.cudnn.deterministic = True - if strict: - torch.use_deterministic_algorithms(True, warn_only=True) - - os.environ["CUBLAS_WORKSPACE_CONFIG"] = ":4096:8" + torch.use_deterministic_algorithms(strict, warn_only=False) @beartype @@ -966,14 +970,10 @@ def get_torch_dtype(dtype_str: str) -> torch.dtype: "float16": torch.float16, "bfloat16": torch.bfloat16, "float64": torch.float64, + "float8_e4m3fn": torch.float8_e4m3fn, + "float8_e5m2": torch.float8_e5m2, } - # Add float8 support if available in this PyTorch version - if hasattr(torch, "float8_e4m3fn"): - dtype_map["float8_e4m3fn"] = torch.float8_e4m3fn - if hasattr(torch, "float8_e5m2"): - dtype_map["float8_e5m2"] = torch.float8_e5m2 - if dtype_str not in dtype_map: raise ValueError( f"dtype '{dtype_str}' not supported or available. Options: {list(dtype_map.keys())}" @@ -1031,9 +1031,17 @@ def get_last_training_batch_timedelta( ) if os.path.exists(metrics_path): + from sequifier.training.metrics import ( + validate_metrics_csv_header, + validate_metrics_csv_row, + ) + observations = [] with open(metrics_path, "r", encoding="utf-8", newline="") as file: - for row in csv.DictReader(file): + reader = csv.DictReader(file) + validate_metrics_csv_header(reader.fieldnames, metrics_path) + for row in reader: + validate_metrics_csv_row(row, metrics_path) if row.get("metric") != "loss" or row.get("target") != "__total__": continue timestamp = row.get("timestamp_utc") diff --git a/src/sequifier/hyperparameter_search.py b/src/sequifier/hyperparameter_search.py index affd91443..3faaada83 100644 --- a/src/sequifier/hyperparameter_search.py +++ b/src/sequifier/hyperparameter_search.py @@ -1,12 +1,14 @@ import csv import ctypes import json +import math import os import signal import subprocess import sys import time import warnings +from pathlib import Path from typing import Any, Union import optuna @@ -21,16 +23,17 @@ from sequifier.config.hyperparameter_search_config import ( # noqa: E402 load_hyperparameter_search_config, ) -from sequifier.helpers import ( # noqa: E402 - get_best_model_path, - get_last_training_batch_timedelta, -) +from sequifier.helpers import get_last_training_batch_timedelta # noqa: E402 from sequifier.io.yaml import TrainModelDumper # noqa: E402 from sequifier.logging_paths import ( # noqa: E402 dataset_artifact_prefix, model_log_directory, ) -from sequifier.training.metrics import VALIDATION_FIELDS # noqa: E402 +from sequifier.training.metrics import ( # noqa: E402 + VALIDATION_FIELDS, + validate_metrics_csv_header, + validate_metrics_csv_row, +) _DUPLICATE_OF_USER_ATTR = "sequifier_duplicate_of" _MAX_CONSECUTIVE_DUPLICATE_PROPOSALS = 1000 @@ -38,12 +41,18 @@ @beartype def _monitored_dataset(run_config: Any) -> tuple[str | None, int]: + if any( + phase.algorithm != "supervised" for phase in run_config.training_plan.phases + ): + raise ValueError( + "hyperparameter search for trajectory training is not yet supported" + ) dataset_names = tuple(run_config.dataset_training) dataset_count = len(dataset_names) if dataset_count == 1: return None, dataset_count - evaluation = getattr(run_config, "evaluation", None) - monitor = getattr(evaluation, "monitor", None) + evaluation = run_config.training_plan.phases[-1].evaluation + monitor = evaluation.monitor if evaluation is not None else None if monitor is None: raise ValueError( "Multi-dataset hyperparameter search requires evaluation.monitor" @@ -58,16 +67,20 @@ def _monitored_dataset(run_config: Any) -> tuple[str | None, int]: @beartype def create_sampler(config: Any) -> optuna.samplers.BaseSampler: - strategy = getattr(config, "method", "bayesian") - global_seed = getattr(config, "global_seed", None) - if strategy in ["sample"]: - return optuna.samplers.RandomSampler(seed=global_seed) + strategy = config.method + sampler_seed = config.sampler_seed + if strategy == "random": + return optuna.samplers.RandomSampler(seed=sampler_seed) if strategy == "grid": - if hasattr(optuna.samplers, "BruteForceSampler"): - return optuna.samplers.BruteForceSampler(seed=global_seed) - raise RuntimeError("Grid search requires Optuna >= 3.1 for BruteForceSampler.") + try: + from optuna.samplers import BruteForceSampler + except ImportError as error: + raise RuntimeError( + "Grid search requires Optuna >= 3.1 for BruteForceSampler." + ) from error + return BruteForceSampler(seed=sampler_seed) return optuna.samplers.TPESampler( - seed=global_seed, + seed=sampler_seed, multivariate=True, ) @@ -87,48 +100,65 @@ def objective( """Run one Optuna trial through the CLI trainer and validation metrics.""" if run_config is None: run_config = config.sample_trial(trial, accepted_trials) - run_name = run_config.model_name + run_name = run_config.run.model_name monitored_dataset, dataset_count = _monitored_dataset(run_config) config_path = os.path.join( - config.project_root, config.model_config_write_path, f"{run_name}.yaml" + config.run.project_root, + config.run.outputs.trial_config_directory, + f"{run_name}.yaml", ) os.makedirs(os.path.dirname(config_path), exist_ok=True) + final_phase = run_config.training_plan.phases[-1] + monitor = final_phase.evaluation.monitor if final_phase.evaluation else None + monitor_metric = monitor.metric if monitor is not None else "loss" + monitor_direction = monitor.direction if monitor is not None else "min" + monitor_part = monitor.source.partition(".")[2] if monitor is not None else "" + expected_direction = (config.evaluation_metric_directions or ["min"])[0] + if not config.evaluation_script_path and expected_direction != monitor_direction: + raise ValueError( + "search direction must match the final phase evaluation monitor" + ) + serialized = run_config.model_dump(mode="python") + serialized["run"] = run_config.run.model_dump(mode="python", exclude_unset=True) with open(config_path, "w") as f: yaml.dump( - run_config.model_dump(mode="python"), + serialized, f, Dumper=TrainModelDumper, sort_keys=False, ) - log_dir = model_log_directory(config.project_root, run_name) + metrics_name = run_name + log_dir = model_log_directory(config.run.project_root, metrics_name) log_dir.mkdir(parents=True, exist_ok=True) validation_path = ( str( dataset_artifact_prefix( - config.project_root, - run_name, + config.run.project_root, + metrics_name, dataset_name=monitored_dataset, dataset_count=dataset_count, ) ) + "-validation-full.csv" ) - prune_path = str(log_dir / f"{run_name}.prune") + prune_path = str(log_dir / f"{metrics_name}.prune") consumed_evaluation_ids: set[str] = set() if os.path.exists(validation_path): with open(validation_path, "r", encoding="utf-8", newline="") as file: reader = csv.DictReader(file) + validate_metrics_csv_header(reader.fieldnames, validation_path) if reader.fieldnames != VALIDATION_FIELDS: raise ValueError( f"Unexpected validation metrics schema in {validation_path}: " f"{reader.fieldnames!r}" ) - consumed_evaluation_ids.update( - row["evaluation_id"] for row in reader if row.get("evaluation_id") - ) + for row in reader: + validate_metrics_csv_row(row, validation_path) + if row.get("evaluation_id"): + consumed_evaluation_ids.add(row["evaluation_id"]) env = os.environ.copy() env["SEQUIFIER_HYPERPARAMETER_SEARCH_RUN"] = "1" @@ -139,7 +169,7 @@ def objective( preexec_fn=set_pdeathsig if sys.platform.startswith("linux") else None, ) - best_val_loss = float("inf") + best_val_loss = float("inf") if monitor_direction == "min" else float("-inf") completed_epochs = 0 @beartype @@ -150,17 +180,26 @@ def consume_metrics( if os.path.exists(validation_path): with open(validation_path, "r", encoding="utf-8", newline="") as file: reader = csv.DictReader(file) + validate_metrics_csv_header(reader.fieldnames, validation_path) if reader.fieldnames != VALIDATION_FIELDS: raise ValueError( f"Unexpected validation metrics schema in " f"{validation_path}: {reader.fieldnames!r}" ) for data in reader: + if None in data.values(): + continue # The training process may still be writing this row. + validate_metrics_csv_row(data, validation_path) evaluation_id = data.get("evaluation_id") if ( not evaluation_id or evaluation_id in consumed_evaluation_ids - or data.get("metric") != "loss" + or data.get("metric") != monitor_metric + or (data.get("phase") and data["phase"] != final_phase.name) + or ( + monitor is not None + and (data.get("part") or "") != monitor_part + ) or data.get("target") != "__total__" or not data.get("value") or not data.get("global_step") @@ -171,7 +210,7 @@ def consume_metrics( val_loss = float(data["value"]) global_step = int(data["global_step"]) metric_epoch = int(data["epoch"]) - if data.get("evaluation_kind") == "epoch_end": + if data.get("evaluation_kind") in {"epoch_end", "iteration_end"}: completed_epochs = max(completed_epochs, metric_epoch) is_multi_objective = ( @@ -180,7 +219,9 @@ def consume_metrics( ) if not is_multi_objective: trial.report(val_loss, global_step) - best_val_loss = min(best_val_loss, val_loss) + best_val_loss = (min if monitor_direction == "min" else max)( + best_val_loss, val_loss + ) if config.pruning_warmup_batches is not None: warmup_complete = ( @@ -198,7 +239,7 @@ def consume_metrics( try: try: timedelta = get_last_training_batch_timedelta( - run_name, 0, config.project_root + metrics_name, 0, config.run.project_root ) timeout_val = (timedelta * 2) + 30 except (ValueError, FileNotFoundError): @@ -230,41 +271,49 @@ def consume_metrics( elif exit_code != 0: raise RuntimeError(f"Training failed with exit code {exit_code}") - model_type = "onnx" if run_config.export_onnx else "pt" - model_path, _last_epoch = get_best_model_path( - config.project_root, - run_name, - model_type, - dataset_name=(monitored_dataset if model_type == "onnx" else None), - dataset_count=(dataset_count if model_type == "onnx" else 1), + report_path = Path(config.run.project_root) / ( + run_config.run.outputs.report_path or f"outputs/training/{run_name}/report.json" ) + report = json.loads(report_path.read_text()) + phases = report["phases"] + if not phases: + raise ValueError("Training completed without a selected phase") + final_report = phases[-1] + best_val_loss = final_report.get("selected_metric") + if not config.evaluation_script_path: + if final_report["name"] != final_phase.name: + raise ValueError("Search plan stopped before its objective phase") + if best_val_loss is None or not math.isfinite(best_val_loss): + raise ValueError("Search requires a finite final phase selection metric") + model_path = report["model_path"] evaluation_id = os.path.splitext(os.path.basename(model_path))[0] - if config.evaluation_inference_config: - evaluation_inference_config = config.evaluation_inference_config - if not os.path.isabs(evaluation_inference_config) and not os.path.exists( - evaluation_inference_config + if config.evaluation_inference_config_path: + inference_model_path = os.path.relpath(model_path, config.run.project_root) + evaluation_inference_config_path = config.evaluation_inference_config_path + if not os.path.isabs(evaluation_inference_config_path) and not os.path.exists( + evaluation_inference_config_path ): - evaluation_inference_config = os.path.join( - config.project_root, - evaluation_inference_config, + evaluation_inference_config_path = os.path.join( + config.run.project_root, + evaluation_inference_config_path, ) subprocess.run( [ "sequifier", "infer", - f"--config-path={evaluation_inference_config}", - f"--model-path={model_path}", + f"--config-path={evaluation_inference_config_path}", + f"--model-paths={inference_model_path}", ], check=True, ) - if config.evaluation_script and config.evaluation_metrics: - eval_script_path = config.evaluation_script + if config.evaluation_script_path and config.evaluation_metrics: + eval_script_path = config.evaluation_script_path cmd = [sys.executable, eval_script_path, evaluation_id] eval_process = subprocess.run( - cmd, capture_output=True, text=True, cwd=config.project_root + cmd, capture_output=True, text=True, cwd=config.run.project_root ) if eval_process.returncode != 0: @@ -273,7 +322,7 @@ def consume_metrics( ) eval_json_path = os.path.join( - config.project_root, + config.run.project_root, "outputs", "evaluations", f"{evaluation_id}.json", @@ -411,11 +460,11 @@ def hyperparameter_search(config_path: str, skip_metadata: bool) -> None: """Load config, create Optuna study, and optimize trials.""" config = load_hyperparameter_search_config(config_path, skip_metadata) - os.makedirs(os.path.join(config.project_root, "state", "optuna"), exist_ok=True) + os.makedirs(os.path.join(config.run.project_root, "state", "optuna"), exist_ok=True) sampler = create_sampler(config) storage_path = os.path.join( - config.project_root, "state", "optuna", f"{config.name}.db" + config.run.project_root, "state", "optuna", f"{config.run.study_name}.db" ) is_multivariate = ( @@ -423,9 +472,13 @@ def hyperparameter_search(config_path: str, skip_metadata: bool) -> None: ) if is_multivariate: + assert config.evaluation_metric_directions is not None study = optuna.create_study( - study_name=config.name, - directions=config.evaluation_metric_directions, + study_name=config.run.study_name, + directions=[ + {"min": "minimize", "max": "maximize"}[value] + for value in config.evaluation_metric_directions + ], sampler=sampler, storage=f"sqlite:///{storage_path}", load_if_exists=True, @@ -437,11 +490,11 @@ def hyperparameter_search(config_path: str, skip_metadata: bool) -> None: config.evaluation_metric_directions and len(config.evaluation_metric_directions) == 1 ) - else "minimize" + else "min" ) study = optuna.create_study( - study_name=config.name, - direction=direction, + study_name=config.run.study_name, + direction={"min": "minimize", "max": "maximize"}[direction], sampler=sampler, storage=f"sqlite:///{storage_path}", load_if_exists=True, diff --git a/src/sequifier/infer.py b/src/sequifier/infer.py index c23dc58a4..9b33463db 100644 --- a/src/sequifier/infer.py +++ b/src/sequifier/infer.py @@ -12,8 +12,9 @@ from beartype.typing import Iterator from loguru import logger +from sequifier.artifacts.model_artifact import read_execution_artifact from sequifier.config.infer_config import ResolvedInferenceConfig, load_inferer_config -from sequifier.config.train_config import ResolvedSequifierConfig as TrainModel +from sequifier.config.inference_common import InferenceExecutionConfig from sequifier.helpers import ( PANDAS_TO_TORCH_TYPES, configure_determinism, @@ -27,6 +28,9 @@ validate_stored_window_width, write_data, ) +from sequifier.io.output_paths import require_file_destinations, unique_paths +from sequifier.io.preprocessing import denormalize_values +from sequifier.model.execution_schema import ExecutionSchema from sequifier.objectives import get_objective_class from sequifier.special_tokens import ( ONNX_CATEGORICAL_TARGET_CODECS_KEY, @@ -167,6 +171,14 @@ def infer(args: Any, args_config: dict[str, Any]) -> None: skip_metadata = args_config.get("skip_metadata", False) config = load_inferer_config(config_path, args_config, skip_metadata) + from sequifier.config.policy_infer_config import ResolvedPolicyInferenceConfig + + if isinstance(config, ResolvedPolicyInferenceConfig): + from sequifier.rl.inference import infer_policy + + infer_policy(config) + return + if config.decode_categories or "real" in config.target_column_types.values(): metadata = config.dataset_metadata if metadata is None: @@ -182,7 +194,7 @@ def infer(args: Any, args_config: dict[str, Any]) -> None: selected_columns_statistics = {} normalize_real_columns = True - configure_determinism(config.seed, config.deterministic) + configure_determinism(config.seed, config.enforce_determinism) # Seed once for this CLI run, before creating any ORT sessions. onnxruntime.set_seed(config.seed) @@ -207,12 +219,15 @@ def load_pt_dataset(data_path: str, start_pct: float, end_pct: float) -> Iterato for pt_file in pt_files[start_idx:end_idx]: from sequifier.config.depth_layout import DepthLayoutRegistryModel + from sequifier.config.metadata import validate_persisted_metadata_versions from sequifier.io.pt_payload import load_pt_payload metadata_path = Path(data_path) / "metadata.json" folder_metadata = ( json.loads(metadata_path.read_text()) if metadata_path.exists() else {} ) + if metadata_path.exists(): + validate_persisted_metadata_versions(folder_metadata) yield load_pt_payload( pt_file, layouts=DepthLayoutRegistryModel.model_validate( @@ -357,8 +372,8 @@ def _windowed_inference_batch_from_storage( config.window_view, configured_window_stride(config), ) - if getattr(config, "aligned_data_path", False): - steps = getattr(config, "prediction_alignment_steps", 1) + if config.aligned_data_path: + steps = config.prediction_alignment_steps spacing = config.prediction_length if spacing is None: raise ValueError("Aligned inference requires prediction_length") @@ -605,6 +620,7 @@ def infer_worker( is_folder_input = os.path.isdir( normalize_path(config.data_path, config.project_root) ) + start_pct, end_pct = percentage_limits or (0.0, 100.0) dataset = None if not is_folder_input: # Standalone Single-File Path Execution @@ -628,87 +644,32 @@ def infer_worker( ) ] - model_paths = ( - config.model_path - if isinstance(config.model_path, list) - else [config.model_path] - ) + model_paths = config.model_paths for model_path in model_paths: target_decoder_ids = None route_args = dict(args_config) - for key in ("dataset", "part", "model_interface"): - value = getattr(config, key, None) + for key, value in ( + ("dataset", config.dataset), + ("part", config.part), + ("model_interface", config.model_interface), + ): if value is not None: route_args[key] = value + artifact = config._artifacts.get(model_path) if model_path.lower().endswith(".pt"): - model_state = torch.load( - normalize_path(model_path, config.project_root), - map_location="cpu", - weights_only=False, - ) - embedded_model_config = model_state.get("model_config") - if embedded_model_config is not None: - interface_name = route_args.get("model_interface") - if ( - interface_name is None - and len(embedded_model_config.get("interfaces", {})) == 1 - ): - interface_name = next(iter(embedded_model_config["interfaces"])) - if interface_name is not None: - interface = embedded_model_config["interfaces"].get(interface_name) - if interface is None: - raise ValueError( - f"Unknown PT model interface {interface_name!r}" - ) - target_decoder_ids = interface.get("target_decoder_ids", {}) - elif model_state.get("training_config") is not None: - training_config = TrainModel.model_validate( - model_state["training_config"] + if artifact is None: + artifact = read_execution_artifact( + normalize_path(model_path, config.project_root) ) - selected_dataset = route_args.get("dataset") - selected_interface = route_args.get("model_interface") - datasets = training_config.dataset_training - if selected_dataset is not None: - if selected_dataset not in datasets: - raise ValueError( - f"Unknown inference dataset {selected_dataset!r}" - ) - dataset_config = datasets[selected_dataset] - if ( - selected_interface is not None - and dataset_config.model_interface != selected_interface - ): - raise ValueError( - f"Dataset {selected_dataset!r} maps to interface " - f"{dataset_config.model_interface!r}, not " - f"{selected_interface!r}" - ) - elif selected_interface is not None: - dataset_config = next( - ( - dataset - for dataset in datasets.values() - if dataset.model_interface == selected_interface - ), - None, - ) - if dataset_config is None: - raise ValueError( - "No execution route for model interface " - f"{selected_interface!r}" - ) - elif len(datasets) == 1: - dataset_config = next(iter(datasets.values())) - else: - raise ValueError( - "A dataset or model_interface selection is required for " - "multi-dataset checkpoint inference" - ) - target_decoder_ids = dict(dataset_config.interface.target_decoder_ids) - else: - raise ValueError( - "PyTorch artifact has neither model_config nor a resolved " - "training_config." + interfaces = artifact.model_config.values["interfaces"] + interface_name = route_args.get("model_interface") + if interface_name is None and len(interfaces) == 1: + interface_name = next(iter(interfaces)) + if interface_name is not None: + if interface_name not in interfaces: + raise ValueError(f"Unknown PT model interface {interface_name!r}") + target_decoder_ids = interfaces[interface_name].get( + "target_decoder_ids", {} ) if is_folder_input: @@ -716,8 +677,6 @@ def infer_worker( raise ValueError( "percentage_limits must be provided for folder-based read formats" ) - start_pct, end_pct = percentage_limits - # Direct folders to their respective lazy loaders based on file format if config.read_format == "pt": dataset = load_pt_dataset(config.data_path, start_pct, end_pct) @@ -741,7 +700,7 @@ def infer_worker( ) inferer = Inferer( - config.model_type, + ("embedding" if config.task == "embedding" else "generative"), model_path, config.project_root, id_maps, @@ -762,6 +721,7 @@ def infer_worker( training_objective=config.training_objective, normalize_real_columns=normalize_real_columns, target_decoder_ids=target_decoder_ids, + artifact=artifact, ) column_data_types = _torch_column_types(config) @@ -769,9 +729,23 @@ def infer_worker( model_id = Path(model_path).stem logger.info(f"Inferring for {model_id}") - if config.model_type == "generative": + if config.task == "forecast": + if config.output_path is not None and config.output_probabilities: + part_count = 1 + if is_folder_input: + total = len( + list(Path(config.data_path).glob(f"*.{config.read_format}")) + ) + part_count = len( + range(total)[ + int(total * start_pct / 100) : int(total * end_pct / 100) + ] + ) + validate_probability_destinations( + config, model_id, inferer.target_column_types, part_count + ) infer_generative(config, inferer, model_id, dataset, column_data_types) - if config.model_type == "embedding": + if config.task == "embedding": infer_embedding(config, inferer, model_id, dataset, column_data_types) logger.info("--- Inference Complete ---") @@ -887,20 +861,101 @@ def _autoregressive_seed_dataframe( @beartype +def _inference_destination( + project_root: str, + write_format: str, + artifact_type: str, + model_id: str, + data_id: int, + target_column: str | None = None, + output_path: str | None = None, +) -> tuple[Path, str]: + if output_path is not None and artifact_type in {"predictions", "embeddings"}: + destination = Path(project_root) / output_path + if data_id: + destination = destination.with_name( + f"{destination.stem}-part-{data_id:03d}{destination.suffix}" + ) + label = "output_path" + else: + output_dir = Path(project_root) / "outputs" / artifact_type / model_id + if target_column is not None: + output_dir /= target_column + destination = output_dir / f"part-{data_id:03d}.{write_format}" + label = f"{artifact_type} output" + return destination, label + + def inference_output_path( project_root: str, write_format: str, artifact_type: str, model_id: str, data_id: int, - target_column: Optional[str] = None, + target_column: str | None = None, + output_path: str | None = None, + *, + config: InferenceExecutionConfig | None = None, ) -> str: - """Return a canonical inference output path and create its directory.""" - output_dir = Path(project_root) / "outputs" / artifact_type / model_id - if target_column is not None: - output_dir /= target_column - output_dir.mkdir(parents=True, exist_ok=True) - return str(output_dir / f"part-{data_id:03d}.{write_format}") + """Prepare an inference output path, checking inputs before creating directories.""" + destination, label = _inference_destination( + project_root, + write_format, + artifact_type, + model_id, + data_id, + target_column, + output_path, + ) + if config is not None: + config.validate_output_destination(destination, label=label) + destination.parent.mkdir(parents=True, exist_ok=True) + return str(destination) + + +def validate_probability_destinations(config, model_id, target_types, part_count): + """Reserve all prediction and probability files before writing either.""" + output_path = config.output_path + if config.task == "policy" and output_path is not None: + output_path = str( + Path(normalize_path(output_path, config.project_root)).absolute() + ) + paths = [] + for index in range(part_count): + paths.append( + _inference_destination( + config.project_root, + config.write_format, + "predictions", + model_id, + index, + output_path=output_path, + )[0] + ) + paths.extend( + _inference_destination( + config.project_root, + config.write_format, + "probabilities", + model_id, + index, + target, + )[0] + for target, kind in target_types.items() + if kind == "categorical" + ) + paths = unique_paths(paths, "prediction and probability outputs must be distinct") + require_file_destinations(paths) + existing_files = set() + for path in paths: + try: + stat = path.stat() + except FileNotFoundError: + continue + identity = (stat.st_dev, stat.st_ino) + if identity in existing_files: + raise ValueError("prediction and probability outputs must be distinct") + existing_files.add(identity) @beartype @@ -1023,6 +1078,8 @@ def infer_embedding( "embeddings", model_id, data_id, + output_path=config.output_path, + config=config, ) logger.info(f"Writing embeddings to '{embeddings_path}'") write_data( @@ -1041,16 +1098,19 @@ def infer_generative( column_data_types: dict[str, torch.dtype], ): """Write generative predictions/probabilities for each dataset chunk.""" + if config.rollout is not None: + infer_numeric_rollout(config, inferer, model_id, dataset, column_data_types) + return data_path = config.data_path if data_path is None: raise ValueError("data_path must be provided or resolved from metadata") for data_id, data in enumerate(dataset): - if config.autoregressive and isinstance(data, pl.DataFrame): + if config.autoregressive_generation and isinstance(data, pl.DataFrame): data = _autoregressive_seed_dataframe(config, data) windowed = _windowed_inference_batch(config, data, column_data_types) if windowed.sequence_ids.numel() == 0: continue - if config.autoregressive and inferer.prediction_length != 1: + if config.autoregressive_generation and inferer.prediction_length != 1: raise ValueError( "prediction_length must be 1 for autoregressive inference, " f"got {inferer.prediction_length}" @@ -1058,7 +1118,7 @@ def infer_generative( total_steps = ( config.generation_steps - if config.autoregressive and config.generation_steps is not None + if config.autoregressive_generation and config.generation_steps is not None else 1 ) if ( @@ -1072,10 +1132,7 @@ def infer_generative( ) ) and configured_window_stride(config) is None - and ( - not getattr(config, "aligned_data_path", False) - or getattr(config, "prediction_alignment_steps", 1) == 1 - ) + and (not config.aligned_data_path or config.prediction_alignment_steps == 1) ): probs, preds = get_probs_preds_from_df( config, @@ -1186,6 +1243,7 @@ def infer_generative( model_id, data_id, target_column, + config=config, ) logger.info(f"Writing probabilities to '{probabilities_path}'") # Step 5: Finalize Output and I/O (write_data now handles Polars DF) @@ -1224,6 +1282,8 @@ def infer_generative( "predictions", model_id, data_id, + output_path=config.output_path, + config=config, ) logger.info(f"Writing predictions to '{predictions_path}'") write_data( @@ -1233,6 +1293,126 @@ def infer_generative( ) +def infer_numeric_rollout(config, inferer, model_id, dataset, column_data_types): + """Run the opt-in single-seed path through the shared forecast coordinator.""" + from sequifier.model.execution_schema import ExecutionSchema + from sequifier.rollout.coordinator import collect_rollout + from sequifier.rollout.factories import load_adapter + from sequifier.rollout.preprocessing import FixedNumericPreprocessing + from sequifier.trajectory.store import save_trajectory + + rollout = config.rollout + remaining = rollout.seed_window_index + selected = None + for data_id, data in enumerate(dataset): + windowed = _windowed_inference_batch(config, data, column_data_types) + count = windowed.sequence_ids.numel() + if remaining >= count: + remaining -= count + continue + selected = (data_id, windowed, remaining) + break + if selected is None: + raise ValueError( + f"rollout seed_window_index {rollout.seed_window_index} is outside the prepared dataset windows" + ) + data_id, windowed, row = selected + metadata = config.dataset_metadata + columns = list( + dict.fromkeys( + config.input_columns + config.static_columns + config.target_columns + ) + ) + codec = FixedNumericPreprocessing( + columns, + metadata.selected_columns_statistics, + metadata.normalize_real_columns, + { + column: torch.empty((), dtype=column_data_types[column]).numpy().dtype + for column in columns + }, + ) + seed_inputs = { + key: value[row : row + 1].numpy() for key, value in windowed.inputs.items() + } + seed_mask = windowed.metadata["attention_valid_mask"][row : row + 1].numpy() + seed = codec.seed_from_model( + seed_inputs, seed_mask, config.input_columns, config.static_columns + ) + schema = ( + ExecutionSchema.from_interface( + inferer.inference_model.interface, config.window_view.context_length + ) + if inferer.inference_model_type == "pt" + else inferer.execution_schema + ) + if schema is None: + raise ValueError("numeric rollout requires a model execution schema") + + def validate_history(history): + features, masks = codec.model_inputs(history) + schema.validate( + {key: torch.from_numpy(value) for key, value in features.items()}, + {key: torch.from_numpy(value) for key, value in masks.items()}, + ) + + def predict(history): + inputs, masks = codec.model_inputs(history) + predictions = inferer.infer_generative( + inputs, masks, column_data_types=column_data_types, return_probs=False + ) + decoded = {} + for column in config.target_columns: + value = np.asarray(predictions[column]) + if value.size != 1: + raise ValueError( + f"rollout predictor must produce one scalar for {column!r}" + ) + decoded[column] = float(codec.decode(column, value).reshape(-1)[0]) + return decoded + + # Instantiate only after validating the seed; collect_rollout owns close(), + # including reset, model, adapter, successor validation, and collection failures. + adapter = load_adapter(rollout.environment) + trajectory = collect_rollout( + predict, + adapter, + seed, + rollout, + config.target_columns, + rollout.max_steps, + random_seed=config.seed, + history_validator=validate_history, + metadata={ + "model_id": model_id, + "model_paths": config.model_paths, + "model_interface": config.model_interface, + "training_objective": config.training_objective, + "target_offset": config.window_view.target_offset, + "context_length": config.window_view.context_length, + "preprocessing": codec.to_dict(), + "rollout": rollout.model_dump(mode="json"), + "collection_budget": rollout.max_steps, + "seed": config.seed, + "enforce_determinism": config.enforce_determinism, + "seed_origin": { + "data_id": data_id, + "seed_window_index": rollout.seed_window_index, + "sequence_id": int(windowed.sequence_ids[row]), + "subsequence_id": int(windowed.subsequence_ids[row]), + "model_start_position": int(windowed.model_start_positions[row]), + "window_start_offset": int(windowed.window_start_offsets[row]), + }, + }, + ) + path = normalize_path(rollout.trajectory_path, config.project_root) + config.validate_output_destination(path, label="trajectory_path") + save_trajectory(trajectory, path) + logger.info( + f"Wrote {len(trajectory.transitions)} rollout transitions to '{path}'; return={trajectory.total_return}" + ) + + @beartype def get_probs_preds_from_dict( config: Any, @@ -1264,6 +1444,9 @@ def get_probs_preds_from_dict( all_preds_list = {col: [] for col in target_cols} metadata_for_step = metadata_np + + if not isinstance(total_steps, int) or total_steps < 1: + raise ValueError("generation budget must be a positive integer") for i in range(total_steps): if config.output_probabilities: probs_for_step = inferer.infer_generative( @@ -1284,7 +1467,6 @@ def get_probs_preds_from_dict( column_data_types=column_data_types, return_probs=False, ) - for col in target_cols: all_preds_list[col].append(preds_for_step[col]) @@ -1416,6 +1598,8 @@ def verify_variable_order(data: pl.DataFrame) -> None: class Inferer: """Inference runtime for PT/ONNX sequifier models.""" + execution_schema: "ExecutionSchema | None" = None + @beartype def __init__( self, @@ -1440,6 +1624,7 @@ def __init__( training_objective: Optional[str] = None, normalize_real_columns: bool = True, target_decoder_ids: Optional[dict[str, list[int]]] = None, + artifact: Any = None, ): """Load a PT or ONNX backend and postprocessing state.""" self.model_type = model_type @@ -1512,23 +1697,19 @@ def __init__( raise ValueError( f"ONNX graph provides {mode} dropout mode, but inference requests {requested}; export a graph for the requested mode" ) - self.execution_schema = ( - ExecutionSchema.from_dict(json.loads(properties[EXECUTION_SCHEMA_KEY])) - if EXECUTION_SCHEMA_KEY in properties - else None - ) - if self.execution_schema is not None: - import onnx + import onnx - from sequifier.export.onnx import validate_graph_contract + from sequifier.export.onnx import validate_graph_contract - graph = onnx.load( - normalize_path(model_path, project_root), - load_external_data=False, - ) - validate_graph_contract(graph, self.execution_schema) - elif any(len(value.shape) == 3 for value in self.ort_session.get_inputs()): - raise ValueError("Depth ONNX inputs require execution schema metadata") + schema = properties.get(EXECUTION_SCHEMA_KEY) + self.execution_schema = ExecutionSchema.from_dict( + json.loads(schema) if schema is not None else None + ) + graph = onnx.load( + normalize_path(model_path, project_root), + load_external_data=False, + ) + validate_graph_contract(graph, self.execution_schema) self.symbolic_batch = all( not isinstance(value.shape[0], int) for value in self.ort_session.get_inputs() @@ -1556,28 +1737,11 @@ def __init__( self.args_config, self.device, self.infer_with_dropout, + artifact=artifact, ) - route_model = getattr( - self.inference_model, - "transformer_model", - self.inference_model, - ) - self.target_decoder_ids = dict(route_model.target_decoder_ids) + self.target_decoder_ids = self.inference_model.target_decoder_ids if self.sample_from_distribution_columns: - from sequifier.model.decoders import ( - AutoregressiveTransformerDecoderBranch, - ) - - network = self.inference_model.network - network = getattr(network, "_orig_mod", network) - route = network.resolve_interface(self.inference_model.interface_name) - autoregressive_targets = { - target - for branch in route.decoder.branches.values() - if isinstance(branch, AutoregressiveTransformerDecoderBranch) - for target in branch.target_columns - } - sampled = autoregressive_targets.intersection( + sampled = self.inference_model.autoregressive_targets.intersection( self.sample_from_distribution_columns ) if sampled: @@ -1595,9 +1759,9 @@ def invert_normalization( """Invert target-column Z-score normalization.""" if not self.normalize_real_columns: return values - std = self.selected_columns_statistics[target_column]["std"] - mean = self.selected_columns_statistics[target_column]["mean"] - return (values * (std + 1e-9)) + mean + return denormalize_values( + values, self.selected_columns_statistics[target_column] + ) @beartype def _exclude_mask_token( @@ -1865,7 +2029,7 @@ def infer_pure( ) -> list[np.ndarray]: """Run one ONNX batch and flatten sequence-major outputs.""" metadata = metadata or {} - schema = getattr(self, "execution_schema", None) + schema = self.execution_schema if schema is not None: schema.validate( {key: torch.from_numpy(value) for key, value in x.items()}, diff --git a/src/sequifier/integration/callbacks.py b/src/sequifier/integration/callbacks.py index 3d85daa5d..71c2e954b 100644 --- a/src/sequifier/integration/callbacks.py +++ b/src/sequifier/integration/callbacks.py @@ -1,6 +1,5 @@ from __future__ import annotations -import importlib from dataclasses import dataclass, replace from typing import Any, Protocol, cast, runtime_checkable @@ -13,6 +12,7 @@ TrainingEvent, ) from sequifier.integration.controls import TrainingDirective +from sequifier.integration.factories import load_factory from sequifier.integration.specifications import ( ExecutionRequirements, IntegrationSpec, @@ -64,13 +64,42 @@ class _RegisteredIntegration: state_policy: StatePolicy rank_policy: str + def __post_init__(self): + instance = self.instance + try: + self.handle = instance.handle + except AttributeError: + self.handle = None + try: + self.on_gradients_unscaled = instance.on_gradients_unscaled + except AttributeError: + self.on_gradients_unscaled = None + try: + self.capture_request = instance.capture_request + except AttributeError: + self.capture_request = None + try: + self.interventions = instance.interventions + except AttributeError: + self.interventions = None + try: + self.local_gradient_summary = instance.local_gradient_summary + except AttributeError: + self.local_gradient_summary = None + try: + self.state_dict = instance.state_dict + except AttributeError: + self.state_dict = None + try: + self.load_state_dict = instance.load_state_dict + except AttributeError: + self.load_state_dict = None + @beartype def _load_factory(path: str) -> Any: - module_name, attribute_name = path.split(":", 1) - module = importlib.import_module(module_name) try: - return getattr(module, attribute_name) + return load_factory(path) except AttributeError as error: raise ImportError(f"Integration factory {path!r} does not exist.") from error @@ -101,7 +130,10 @@ def __init__( continue factory = _load_factory(spec.factory) instance = factory(dict(spec.config)) - actual_id = getattr(instance, "integration_id", None) + try: + actual_id = instance.integration_id + except AttributeError: + actual_id = None if actual_id != spec.integration_id: raise ValueError( f"Factory {spec.factory!r} returned integration_id " @@ -121,7 +153,10 @@ def __init__( self._observers: list[_RegisteredIntegration] = [] self._controller: _RegisteredIntegration | None = None for item in registered: - integration_id = getattr(item.instance, "integration_id", None) + try: + integration_id = item.instance.integration_id + except AttributeError: + integration_id = None if not isinstance(integration_id, str) or not integration_id: raise ValueError( "Every integration must define a non-empty integration_id." @@ -129,9 +164,9 @@ def __init__( if integration_id in ids: raise ValueError(f"Duplicate integration_id: {integration_id!r}.") ids.add(integration_id) - if callable(getattr(item.instance, "handle", None)): + if callable(item.handle): self._observers.append(item) - if callable(getattr(item.instance, "on_gradients_unscaled", None)): + if callable(item.on_gradients_unscaled): if self._controller is not None: raise ValueError( "At most one TrainingController may be configured." @@ -168,7 +203,9 @@ def control_enabled(self) -> bool: @beartype def emit(self, event: TrainingEvent) -> None: for item in self._observers: - item.instance.handle(event) + handle = item.handle + assert callable(handle) + handle(event) @beartype def forward_trace(self, event: BatchPrepared) -> TraceContext | None: @@ -178,7 +215,7 @@ def forward_trace(self, event: BatchPrepared) -> TraceContext | None: if self._controller is not None and self._controller not in items: items.append(self._controller) for item in items: - request_provider = getattr(item.instance, "capture_request", None) + request_provider = item.capture_request if callable(request_provider): request = request_provider(event) if request is not None and not isinstance(request, CaptureRequest): @@ -187,7 +224,7 @@ def forward_trace(self, event: BatchPrepared) -> TraceContext | None: ) if request is not None: requests.append(request) - intervention_provider = getattr(item.instance, "interventions", None) + intervention_provider = item.interventions if callable(intervention_provider): bindings = intervention_provider(event) if not isinstance(bindings, tuple) or not all( @@ -226,7 +263,11 @@ def positions_equal(left: Any, right: Any) -> bool: clone=any(item.clone for item in requests), positions=positions, ) - trace_catalog = getattr(event.access.model, "trace_catalog", ()) + trace_catalog: Any + try: + trace_catalog = event.access.model.trace_catalog + except AttributeError: + trace_catalog = () available = {site.name for site in trace_catalog} requested = set(request.sites).union(binding.site for binding in interventions) unknown = requested.difference(available) @@ -246,9 +287,10 @@ def validate_execution( if self._controller is not None and self._controller not in items: items.append(self._controller) for item in items: - requirements = getattr( - item.instance, "execution_requirements", ExecutionRequirements() - ) + try: + requirements = item.instance.execution_requirements + except AttributeError: + requirements = ExecutionRequirements() if not isinstance(requirements, ExecutionRequirements): raise TypeError( f"Integration {item.instance.integration_id!r} has invalid " @@ -258,13 +300,13 @@ def validate_execution( requirements.activation_tracing or requirements.interventions or requirements.higher_order_gradients - or callable(getattr(item.instance, "capture_request", None)) - or callable(getattr(item.instance, "interventions", None)) + or callable(item.capture_request) + or callable(item.interventions) ) if needs_eager and torch_compile != "none": raise ValueError( f"Integration {item.instance.integration_id!r} requires eager " - "execution; set global_training.torch_compile to 'none'." + "execution; set runtime_settings.torch_compile to 'none'." ) if self.distributed and needs_eager and item.rank_policy == "rank_zero": raise ValueError( @@ -289,9 +331,7 @@ def directive(self, event: GradientsUnscaled) -> TrainingDirective | None: directive: TrainingDirective | None = None controller_event = event if self._controller is not None: - summary_provider = getattr( - self._controller.instance, "local_gradient_summary", None - ) + summary_provider = self._controller.local_gradient_summary if callable(summary_provider): local_summary = summary_provider(event) if not isinstance(local_summary, dict) or not all( @@ -324,9 +364,10 @@ def directive(self, event: GradientsUnscaled) -> TrainingDirective | None: device=parameter.device, ) dist.all_reduce(values, op=dist.ReduceOp.SUM) - reduction = getattr( - self._controller.instance, "gradient_reduction", "mean" - ) + try: + reduction = self._controller.instance.gradient_reduction + except AttributeError: + reduction = "mean" if reduction == "mean": values /= self.world_size elif reduction != "sum": @@ -339,13 +380,14 @@ def directive(self, event: GradientsUnscaled) -> TrainingDirective | None: } controller_event = replace(event, reduced_summary=reduced_summary) if self._controller is not None and (not self.distributed or self.rank == 0): - directive = self._controller.instance.on_gradients_unscaled( - controller_event - ) - if directive is not None and not isinstance(directive, TrainingDirective): + decide = self._controller.on_gradients_unscaled + assert callable(decide) + decision = decide(controller_event) + if decision is not None and not isinstance(decision, TrainingDirective): raise TypeError( "TrainingController must return TrainingDirective or None." ) + directive = decision if self.distributed: if not dist.is_available() or not dist.is_initialized(): @@ -366,7 +408,7 @@ def state_dict(self) -> dict[str, dict[str, Any]]: items.append(self._controller) for item in items: integration_id = item.instance.integration_id - state_dict = getattr(item.instance, "state_dict", None) + state_dict = item.state_dict result[integration_id] = { "factory": item.factory, "state": state_dict() if callable(state_dict) else {}, @@ -442,7 +484,7 @@ def load_state_dict(self, saved: dict[str, Any] | None) -> None: raise ValueError( f"Checkpoint integration state for {integration_id!r} is invalid." ) - load_state_dict = getattr(item.instance, "load_state_dict", None) + load_state_dict = item.load_state_dict if callable(load_state_dict): load_state_dict(state) unknown = set(saved).difference(configured_ids) diff --git a/src/sequifier/integration/contexts.py b/src/sequifier/integration/contexts.py index b93646f1c..8475614a7 100644 --- a/src/sequifier/integration/contexts.py +++ b/src/sequifier/integration/contexts.py @@ -18,7 +18,7 @@ class StepIdentity: global_batch_step: int optimizer_step: int accumulation_index: int - accumulation_steps: int + accumulation_batches: int rank: int world_size: int diff --git a/src/sequifier/integration/factories.py b/src/sequifier/integration/factories.py new file mode 100644 index 000000000..097572e31 --- /dev/null +++ b/src/sequifier/integration/factories.py @@ -0,0 +1,14 @@ +"""Resolve configured module-level factories at the plugin boundary.""" + +from importlib import import_module + + +def load_factory(path: str): + module_name, symbol = path.split(":", 1) + module = import_module(module_name) + try: + return vars(module)[symbol] + except KeyError as error: + raise AttributeError( + f"module {module_name!r} has no attribute {symbol!r}" + ) from error diff --git a/src/sequifier/io/actions.py b/src/sequifier/io/actions.py new file mode 100644 index 000000000..9ef5f95d6 --- /dev/null +++ b/src/sequifier/io/actions.py @@ -0,0 +1,149 @@ +"""Typed action conversion shared by datasets, collection, and algorithms.""" + +from collections.abc import Mapping + +import torch + +from sequifier.io.values import domain_scalar, numeric_scalar + + +def _metadata_key(name, config, prefix): + key = prefix + name + if key not in config.actions: + return key + + # Give longer action names first claim on their keys so a shorter name + # cannot take a key reached by adding the same prefix to a longer name. + used = set(config.actions) + longer_real_names = ( + other + for other, field in config.actions.items() + if field.type == "real" and len(other) > len(name) + ) + for other in sorted(longer_real_names, key=len, reverse=True): + other_key = prefix + other + while other_key in used: + other_key = prefix + other_key + used.add(other_key) + + while key in used: + key = prefix + key + return key + + +def action_latent_key(name, config): + return _metadata_key(name, config, "__policy_latent:") + + +def action_component_key(name, config): + return _metadata_key(name, config, "__policy_component:") + + +def action_record_fields(config): + return set(config.actions) | { + key(name, config) + for name, field in config.actions.items() + if field.type == "real" + for key in (action_latent_key, action_component_key) + } + + +def inactive_action(field): + if field.type == "categorical": + return field.values.index(field.inactive_value) if field.active_when else 0 + return field.inactive_value + + +def encode_action(value, config): + if not isinstance(value, Mapping) or { + key for key in value if key in config.actions + } != set(config.actions): + raise ValueError("actions must contain exactly the declared named fields") + result = {} + for name, field in config.actions.items(): + item = domain_scalar(value[name], name) + if field.type == "categorical": + if item not in field.values: + raise ValueError(f"{name}: action is outside the executable vocabulary") + item = field.values.index(item) + else: + item = numeric_scalar(item, name) + result[name] = item + allowed = set(config.actions) + for name, field in config.actions.items(): + if field.type != "real": + continue + key = action_latent_key(name, config) + allowed.add(key) + component_key = action_component_key(name, config) + allowed.add(component_key) + if component_key in value and value[component_key] is not None: + component = numeric_scalar(value[component_key], component_key) + if ( + component != int(component) + or component < -2 + or component >= len(field.atoms) + ): + raise ValueError("invalid real mixture component") + result[component_key] = int(component) + if key in value and value[key] is not None: + result[key] = numeric_scalar(value[key], key) + if set(value) - allowed: + raise ValueError("action record contains unknown fields") + return result + + +def action_batch(values, config, *, device=None, encoded=True): + values = list(values) + if not encoded: + values = [encode_action(value, config) for value in values] + real_dtype = ( + torch.float32 + if device is not None and torch.device(device).type == "mps" + else torch.float64 + ) + + def tensor(key, dtype, *, default=None, required=False): + return torch.tensor( + [value[key] if required else value.get(key, default) for value in values], + dtype=dtype, + device=device, + ) + + result = { + name: tensor( + name, + torch.long if field.type == "categorical" else real_dtype, + required=True, + ) + for name, field in config.actions.items() + } + latent_keys = { + action_latent_key(name, config) + for name, field in config.actions.items() + if field.type == "real" + } + for key in latent_keys: + result[key] = tensor(key, real_dtype, default=float("nan")) + for name, field in config.actions.items(): + if field.type == "real": + key = action_component_key(name, config) + result[key] = tensor(key, torch.long, default=-2) + return result + + +def action_record(actions, index=0): + if isinstance(actions, dict): + return {name: value[index].item() for name, value in actions.items()} + return int(actions[index]) + + +def decode_action(action, config): + return { + name: decode_action_value(action[name], field) + for name, field in config.actions.items() + } + + +def decode_action_value(value, field): + return field.values[int(value)] if field.type == "categorical" else float(value) diff --git a/src/sequifier/io/atomic.py b/src/sequifier/io/atomic.py new file mode 100644 index 000000000..db03e9e24 --- /dev/null +++ b/src/sequifier/io/atomic.py @@ -0,0 +1,41 @@ +"""Atomic publication shared by model, checkpoint and trajectory writers.""" + +import json +import os +import uuid +from contextlib import contextmanager +from pathlib import Path + + +@contextmanager +def atomic_path(path, *, durable=False): + path = Path(path) + path.parent.mkdir(parents=True, exist_ok=True) + temporary = path.with_name(f".{path.name}.{uuid.uuid4().hex}.tmp") + try: + yield temporary + if durable: + with temporary.open("rb") as stream: + os.fsync(stream.fileno()) + temporary.replace(path) + if durable and os.name != "nt": + descriptor = os.open(path.parent, os.O_RDONLY) + try: + os.fsync(descriptor) + finally: + os.close(descriptor) + finally: + temporary.unlink(missing_ok=True) + + +def atomic_save(path, payload, *, durable=True): + import torch + + with atomic_path(path, durable=durable) as temporary: + torch.save(payload, temporary) + + +def write_json_atomic(path, payload, **options): + options = {"indent": 2, "allow_nan": False, **options} + with atomic_path(path, durable=True) as temporary: + temporary.write_text(json.dumps(payload, **options) + "\n", encoding="utf-8") diff --git a/src/sequifier/io/batch.py b/src/sequifier/io/batch.py index 76a2845a1..b65bd1795 100644 --- a/src/sequifier/io/batch.py +++ b/src/sequifier/io/batch.py @@ -3,19 +3,39 @@ import torch +from sequifier.io.tensors import TensorRecord, collate_tensor_records from sequifier.typechecking import beartype REQUIRED_METADATA_KEYS = frozenset({"attention_valid_mask", "target_valid_mask"}) @dataclass(frozen=True) -class SequifierBatch: +class SequifierInputs(TensorRecord): + """Named model tensors, independent of labels or learning algorithm.""" + + inputs: dict[str, torch.Tensor] + metadata: dict[str, torch.Tensor] + + @property + def attention_valid_mask(self) -> torch.Tensor: + return self.metadata["attention_valid_mask"] + + +collate_model_inputs = collate_tensor_records + + +@dataclass(frozen=True) +class SequifierBatch(TensorRecord): inputs: dict[str, torch.Tensor] targets: dict[str, torch.Tensor] metadata: dict[str, torch.Tensor] sequence_ids: Optional[torch.Tensor] = None subsequence_ids: Optional[torch.Tensor] = None + @property + def model_inputs(self) -> SequifierInputs: + return SequifierInputs(self.inputs, self.metadata) + @beartype def __post_init__(self) -> None: metadata_keys = self.metadata.keys() if self.metadata is not None else set() @@ -26,20 +46,5 @@ def __post_init__(self) -> None: f"SequifierBatch metadata is missing required keys: {missing}" ) - @beartype - def pin_memory(self) -> "SequifierBatch": - return SequifierBatch( - inputs={k: v.pin_memory() for k, v in self.inputs.items()}, - targets={k: v.pin_memory() for k, v in self.targets.items()}, - metadata={k: v.pin_memory() for k, v in self.metadata.items()}, - sequence_ids=( - self.sequence_ids.pin_memory() - if self.sequence_ids is not None - else None - ), - subsequence_ids=( - self.subsequence_ids.pin_memory() - if self.subsequence_ids is not None - else None - ), - ) + +collate_sequence_batches = collate_tensor_records diff --git a/src/sequifier/io/config.py b/src/sequifier/io/config.py deleted file mode 100644 index ac7aa894d..000000000 --- a/src/sequifier/io/config.py +++ /dev/null @@ -1,13 +0,0 @@ -"""Small compatibility boundary for standalone data-loader tests and callers.""" - -from typing import Any - -from sequifier.typechecking import beartype - - -@beartype -def global_training(config: Any) -> Any: - training = vars(config).get("global_training") - if training is not None: - return training - return config.training_spec diff --git a/src/sequifier/io/depth_preprocess.py b/src/sequifier/io/depth_preprocess.py index 022b9e2b0..6cfa722e7 100644 --- a/src/sequifier/io/depth_preprocess.py +++ b/src/sequifier/io/depth_preprocess.py @@ -21,6 +21,8 @@ from loguru import logger from sequifier.config.depth_layout import DepthLayoutRegistryModel +from sequifier.config.metadata import validate_persisted_metadata_versions +from sequifier.contract_versions import ContractVersions, require_contract_version from sequifier.helpers import PANDAS_TO_TORCH_TYPES from sequifier.io.pt_payload import ( StoredTensorBatch, @@ -68,19 +70,17 @@ def _file_items( def source_rows(): if read_format == "csv": - if hasattr(pl.LazyFrame, "collect_batches"): - batches = ( - pl.scan_csv(filename) - .select(columns) - .collect_batches(chunk_size=1024) - ) - for batch in batches: - yield from batch.iter_rows(named=True) - else: + frame = pl.scan_csv(filename).select(columns) + try: + collect_batches = frame.collect_batches + except AttributeError: reader = pl.read_csv_batched(filename, columns=columns, batch_size=1024) while batches := reader.next_batches(1): for batch in batches: yield from batch.iter_rows(named=True) + else: + for batch in collect_batches(chunk_size=1024): + yield from batch.iter_rows(named=True) else: parquet = _parquet_file(filename) for batch in parquet.iter_batches( @@ -312,7 +312,7 @@ def _audit_parquet_row_group(job): ) -def _audit_parquet_groups(path, capacity, n_cores): +def _audit_parquet_groups(path, capacity, num_workers): """Count complete items in parallel, joining row groups at item boundaries.""" from sequifier.preprocess import _create_preprocess_pool @@ -320,10 +320,10 @@ def _audit_parquet_groups(path, capacity, n_cores): logger.info( "Depth audit: {} Parquet row groups across {} workers", group_count, - min(n_cores, group_count), + min(num_workers, group_count), ) jobs = ((path, index, capacity) for index in range(group_count)) - with _create_preprocess_pool(min(n_cores, group_count)) as pool: + with _create_preprocess_pool(min(num_workers, group_count)) as pool: audits = list(pool.imap(_audit_parquet_row_group, jobs)) counts = Counter() first_positions = {} @@ -383,7 +383,9 @@ def _validated_item_rows(item, columns, shallow, layout, curriculum_columns): layout.position_column, ) if not ( - layout.position_base <= child < layout.position_base + layout.context_length + layout.position_base + <= child + < layout.position_base + layout.max_depth_length ): raise ValueError(f"Depth position {child} is outside layout capacity") slot = child - layout.position_base @@ -457,7 +459,7 @@ def _fit_parquet_group(job): path, "parquet", required, - layout.context_length, + layout.max_depth_length, row_groups=row_groups, ): sid, pos = item.sequence_id, item.position @@ -574,7 +576,7 @@ def flush(split, index): path, "parquet", job["required"], - layout.context_length, + layout.max_depth_length, row_groups=job["row_groups"], sequence_id=sid, ) @@ -670,14 +672,16 @@ def advance_before(position): encode_until(absolute_start + width - 1) tensors = { c: torch.zeros( - (1, width, layout.context_length) + (1, width, layout.max_depth_length) if c in layout.columns else (1, width), dtype=PANDAS_TO_TORCH_TYPES[col_types[c]], ) for c in columns } - mask = torch.zeros((1, width, layout.context_length), dtype=torch.bool) + mask = torch.zeros( + (1, width, layout.max_depth_length), dtype=torch.bool + ) sample_position = None for time in range(window_pad, width): position = absolute_start + time @@ -770,9 +774,7 @@ def preprocess_depth(owner, selected_columns): if Path(owner.preprocessing_data_path).is_dir() else [owner.preprocessing_data_path] ) - configured_curriculum_columns = curriculum_columns( - getattr(owner, "curriculum_column", None) - ) + configured_curriculum_columns = curriculum_columns(owner.curriculum_column) reserved_columns = { "sequenceId", "itemPosition", @@ -839,10 +841,9 @@ def preprocess_depth(owner, selected_columns): raise ValueError(f"{filename}: missing columns {sorted(missing)}") for coordinate in ("sequenceId", "itemPosition"): dtype = schema[coordinate] - assert dtype.is_integer(), ( - f"{coordinate} must have an integer dtype in {filename}; " - f"found {dtype}." - ) + assert ( + dtype.is_integer() + ), f"{coordinate} must have an integer dtype in {filename}; found {dtype}." types = {c: schema[c] for c in columns} if schema_types is not None and types != schema_types: raise ValueError( @@ -889,7 +890,7 @@ def check_sources(): owner.read_format == "parquet" and len(files) == 1 and owner.max_rows is None - and owner.n_cores > 1 + and owner.num_workers > 1 and pq.ParquetFile(files[0]).num_row_groups > 1 ): ( @@ -897,7 +898,7 @@ def check_sources(): first_positions, parallel_groups, parallel_audits, - ) = _audit_parquet_groups(files[0], layout.context_length, owner.n_cores) + ) = _audit_parquet_groups(files[0], layout.max_depth_length, owner.num_workers) selected_count = sum(sequence_counts.values()) previous = parallel_audits[-1].last else: @@ -905,7 +906,7 @@ def check_sources(): files, owner.read_format, ["sequenceId", "itemPosition"], - layout.context_length, + layout.max_depth_length, ): if owner.max_rows is not None and selected_count >= owner.max_rows: continue @@ -926,7 +927,7 @@ def check_sources(): def selected_items(): check_sources() stream = _merged_items( - files, owner.read_format, required, layout.context_length, selected_stop + files, owner.read_format, required, layout.max_depth_length, selected_stop ) try: for _ in range(selected_count): @@ -985,6 +986,7 @@ def item_rows(item): existing = json.loads( (Path(owner.project_root) / owner.metadata_config_path).read_text() ) + validate_persisted_metadata_versions(existing) owner.metadata_fitted_on_all_data = existing.get("normalize_on_all_data", True) if ( DepthLayoutRegistryModel.model_validate(existing.get("depth_layouts", {})) @@ -1072,9 +1074,11 @@ def fit_jobs(): logger.info( "Depth metadata fit: {} source partitions across {} workers", len(parallel_groups), - min(owner.n_cores, len(parallel_groups)), + min(owner.num_workers, len(parallel_groups)), ) - with _create_preprocess_pool(min(owner.n_cores, len(parallel_groups))) as pool: + with _create_preprocess_pool( + min(owner.num_workers, len(parallel_groups)) + ) as pool: for local_stats, local_sets, local_counts in pool.imap( _fit_parquet_group, fit_jobs() ): @@ -1205,34 +1209,44 @@ def fit_jobs(): owner._write_or_validate_resume_manifest( selected_columns, "pt", columns, id_maps, n_classes, col_types, stats ) - owner._export_metadata(id_maps, n_classes, col_types, stats) # A shard is reusable only after both its payload and its ledger entry have # been committed. The ledger also ties completed shards to this source set. ledger_path = scratch_root / "depth-shards.json" source_identity = [list(entry) for entry in source_snapshot] parallel_render = parallel_groups is not None + ledger_versions = { + "serial": ContractVersions.SERIAL_DEPTH_LEDGER, + "parallel": ContractVersions.PARALLEL_DEPTH_LEDGER, + } if owner.continue_preprocessing and ledger_path.exists(): ledger = json.loads(ledger_path.read_text()) - if ( - ledger.get("version") not in (1, 2) - or ledger.get("sources") != source_identity - ): + mode = ledger.get("mode") if isinstance(ledger, dict) else None + if not isinstance(mode, str) or mode not in ledger_versions: + raise ValueError("Depth resume ledger requires serial or parallel mode") + require_contract_version( + "depth ledger", ledger.get("version"), ledger_versions[mode] + ) + if ledger.get("sources") != source_identity: raise ValueError( "Depth resume source files differ from the completed shards" ) - parallel_render = parallel_render and ledger["version"] == 2 - if ledger["version"] == 2 and not parallel_render: + if mode == "parallel" and not parallel_render: raise ValueError( "Depth parallel resume requires the original Parquet row groups " - "and n_cores > 1" + "and num_workers > 1" ) + parallel_render = mode == "parallel" else: + mode = "parallel" if parallel_render else "serial" ledger = { - "version": 2 if parallel_render else 1, + "mode": mode, + "version": ledger_versions[mode], "sources": source_identity, "shards": {}, } + owner._export_metadata(id_maps, n_classes, col_types, stats) + def shard_path(split, index): return Path(owner.split_paths[split]).stem + f"-0-{index:08d}.pt" @@ -1337,9 +1351,11 @@ def render_jobs(): logger.info( "Depth windows: {} sequences across {} workers", len(sequence_counts), - min(owner.n_cores, len(sequence_counts)), + min(owner.num_workers, len(sequence_counts)), ) - with _create_preprocess_pool(min(owner.n_cores, len(sequence_counts))) as pool: + with _create_preprocess_pool( + min(owner.num_workers, len(sequence_counts)) + ) as pool: for names, records in pool.imap_unordered( _render_parquet_sequence, render_jobs() ): @@ -1494,7 +1510,7 @@ def advance_before(sid, position): encode_until(sid, absolute_start + width - 1) tensors = { c: torch.zeros( - (1, width, layout.context_length) + (1, width, layout.max_depth_length) if c in layout.columns else (1, width), dtype=PANDAS_TO_TORCH_TYPES[col_types[c]], @@ -1502,7 +1518,7 @@ def advance_before(sid, position): for c in columns } mask = torch.zeros( - (1, width, layout.context_length), dtype=torch.bool + (1, width, layout.max_depth_length), dtype=torch.bool ) sample_position = None for time in range(window_pad, width): diff --git a/src/sequifier/io/episode_cache.py b/src/sequifier/io/episode_cache.py new file mode 100644 index 000000000..ade26bef5 --- /dev/null +++ b/src/sequifier/io/episode_cache.py @@ -0,0 +1,35 @@ +"""Count-bounded LRU storage for decoded episodes and decision labels.""" + +from collections import OrderedDict + +from sequifier.io.memory_budget import resident_bytes + + +class EpisodeCache: + def __init__(self, capacity, storage=None): + self.capacity = capacity + self.storage = storage + self.entries = OrderedDict() + self.bytes = 0 + + def get(self, key, load): + if key not in self.entries: + value = load(key) + size = resident_bytes(value) if self.storage is not None else 0 + self.entries[key] = value, size + self.bytes += size + if self.storage is not None: + self.storage.resize(self.bytes) + if len(self.entries) > self.capacity: + _, (_, evicted) = self.entries.popitem(last=False) + self.bytes -= evicted + if self.storage is not None: + self.storage.resize(self.bytes) + self.entries.move_to_end(key) + return self.entries[key][0] + + def clear(self): + self.entries.clear() + self.bytes = 0 + if self.storage is not None: + self.storage.resize(0) diff --git a/src/sequifier/io/episode_corpus.py b/src/sequifier/io/episode_corpus.py new file mode 100644 index 000000000..1f2465009 --- /dev/null +++ b/src/sequifier/io/episode_corpus.py @@ -0,0 +1,159 @@ +"""Indexed episode storage and immutable metadata shared by learning views.""" + +import hashlib +import json +from dataclasses import dataclass +from decimal import Decimal + +from sequifier.config.policy_train_config import SequenceSourceConfig +from sequifier.fingerprints import fingerprint +from sequifier.io.episode_cache import EpisodeCache +from sequifier.io.policy_window import action_index +from sequifier.io.sequence import SequenceEpisode +from sequifier.trajectory.importers import source_episodes +from sequifier.trajectory.sources import ResolvedSequenceSource + + +def _load(source) -> SequenceEpisode: + if isinstance(source, SequenceEpisode): + return SequenceEpisode.model_validate(source.model_dump()) + return source.load() + + +@dataclass(frozen=True) +class EpisodeMetadata: + episode_id: str + group_id: str | None + length: int + content_key: str + fingerprint: str + + +@dataclass(frozen=True) +class EpisodeRecords: + """Indexed source readers and metadata for one episode interpretation.""" + + sources: tuple + metadata: tuple[EpisodeMetadata, ...] + + +class EpisodeCorpus: + """Own one bounded decoded cache over immutable source records.""" + + def __init__( + self, sources, config, cache_size=8, *, cache_owner=None, records=None + ): + if cache_size < 1: + raise ValueError("episode_cache_size must be positive") + self.cache = ( + cache_owner.episode_cache(cache_size) + if cache_owner is not None + else EpisodeCache(cache_size) + ) + self.records = records if records is not None else self._index(sources, config) + + def _index(self, sources, config): + sources = tuple( + episode + for source in sources + for episode in ( + source_episodes(source, config) + if isinstance(source, (SequenceSourceConfig, ResolvedSequenceSource)) + else [source] + ) + ) + metadata = [] + seen = set() + for index in range(len(sources)): + episode = self.cache.get(index, lambda i: _load(sources[i])) + if episode.episode_id in seen: + raise ValueError(f"duplicate episode ID {episode.episode_id!r}") + seen.add(episode.episode_id) + if set(episode.observations) != set(config.observation_columns) or set( + episode.static_inputs + ) != set(config.static_columns): + raise ValueError( + "sequence observations/static inputs do not match the policy schema" + ) + for value in episode.actions: + action_index(value, config) + if episode.previous_action is not None: + action_index(episode.previous_action, config) + metadata.append( + EpisodeMetadata( + episode.episode_id, + episode.group_id, + len(episode.actions), + episode_content_key(episode), + fingerprint(episode.model_dump(mode="json")), + ) + ) + return EpisodeRecords(sources, tuple(metadata)) + + def __len__(self): + return len(self.records.sources) + + def __getitem__(self, index): + return self.cache.get(index, self._load_episode) + + def _load_episode(self, index): + return _load(self.records.sources[index]) + + def view(self): + return EpisodeView([(self, i) for i in range(len(self))]) + + +class EpisodeView: + """A lightweight ordered partition; storage and metadata remain corpus-owned.""" + + def __init__(self, entries): + self.entries = tuple(entries) + + def __len__(self): + return len(self.entries) + + def __iter__(self): + return (self[i] for i in range(len(self))) + + def __getitem__(self, index): + corpus, position = self.entries[index] + return corpus[position] + + def metadata(self, index): + corpus, position = self.entries[index] + return corpus.records.metadata[position] + + def subset(self, indices): + return EpisodeView([self.entries[i] for i in indices]) + + @classmethod + def combine(cls, views): + return cls([entry for view in views for entry in view.entries]) + + +def _canonical(value): + if isinstance(value, dict): + return {key: _canonical(item) for key, item in value.items()} + if isinstance(value, (tuple, list)): + return [_canonical(item) for item in value] + if isinstance(value, (int, float)) and not isinstance(value, bool): + return ["number", str(Decimal(str(value)).normalize())] + return value + + +def episode_content_key(episode, *, length=None, returns=None) -> str: + """Hash learning content, independently of log format, IDs and provenance.""" + length = len(episode.actions) if length is None else length + values = { + "observations": { + column: entries[:length] for column, entries in episode.observations.items() + }, + "actions": episode.actions[:length], + "static_inputs": episode.static_inputs, + "previous_action": episode.previous_action, + "starts_episode": episode.starts_episode, + "returns": returns, + } + return hashlib.sha256( + json.dumps(_canonical(values), sort_keys=True, allow_nan=False).encode() + ).hexdigest() diff --git a/src/sequifier/io/episode_features.py b/src/sequifier/io/episode_features.py new file mode 100644 index 000000000..5240fb1b9 --- /dev/null +++ b/src/sequifier/io/episode_features.py @@ -0,0 +1,141 @@ +"""Fit and validate frozen policy features over an episode view.""" + +from copy import deepcopy +from dataclasses import replace + +import polars as pl + +from sequifier.config.policy_config import ResolvedPolicyBinding +from sequifier.config.policy_inputs import PolicyFittingSpec +from sequifier.io.policy_window import encode_feature +from sequifier.io.preprocessing import ( + _finite_mean_and_std, + create_id_map, + get_combined_statistics, +) +from sequifier.trajectory.returns import return_to_go + + +def return_labels(episode, length, gamma): + if episode.transitions is None or episode.transitions.rewards is None: + raise ValueError("return-conditioned examples require recorded rewards") + return return_to_go(episode.transitions.rewards[:length], gamma) + + +class EpisodeFeatures: + """Selected episode lengths define fitting coverage independently of batching.""" + + def __init__(self, episodes, config, lengths=None): + self.episodes, self.config = episodes, config + self.lengths = ( + lengths + if lengths is not None + else [episodes.metadata(i).length for i in range(len(episodes))] + ) + self._features_validated = False + self._validated_statistics = None + ids = [episodes.metadata(i).episode_id for i in range(len(episodes))] + if len(ids) != len(set(ids)): + raise ValueError("duplicate episode IDs in fitting view") + + def values(self, episode, length, column): + return ( + episode.observations[column][:length] + if column in episode.observations + else [episode.static_inputs[column]] + ) + + def fit_categorical_config(self) -> ResolvedPolicyBinding | PolicyFittingSpec: + """Fit only declared raw categorical features, using the training episodes.""" + values = deepcopy( + { + "id_maps": self.config.id_maps, + "observation_n_classes": self.config.observation_n_classes, + "static_n_classes": self.config.static_n_classes, + } + ) + for column in self.config.categorical_columns: + if ( + column in self.config.id_maps + or column in self.config.categorical_n_classes + ): + continue + vocabulary = set() + for episode, length in zip(self.episodes, self.lengths): + vocabulary.update(self.values(episode, length, column)) + if not vocabulary: + raise ValueError(f"cannot fit empty categorical column {column!r}") + if len({isinstance(value, str) for value in vocabulary}) > 1: + raise ValueError( + f"{column}: mixed string/numeric category labels are ambiguous" + ) + if any( + not isinstance(value, str) and value != int(value) + for value in vocabulary + ): + raise ValueError( + f"{column}: categorical numeric labels must be integers" + ) + mapping = create_id_map(pl.DataFrame({column: sorted(vocabulary)}), column) + values["id_maps"][column] = mapping + key = ( + "observation_n_classes" + if column in self.config.observation_columns + else "static_n_classes" + ) + values[key][column] = max(mapping.values()) + 1 + if isinstance(self.config, PolicyFittingSpec): + return replace(self.config, **values) + return ResolvedPolicyBinding.model_validate( + {**self.config.model_dump(), **values} + ) + + def validate_features(self) -> None: + if self._features_validated: + return + config = self.config + # Validate held-out values against the fixed training codec before updates. + for episode, length in zip(self.episodes, self.lengths): + for column, values in episode.observations.items(): + for value in values[:length]: + encode_feature(value, column, config) + for column, value in episode.static_inputs.items(): + encode_feature(value, column, config) + + self._features_validated = True + + def validate_normalization(self, statistics): + """Check FP32 transforms for decisions in the selected episode prefixes.""" + if self._validated_statistics == statistics: + return + from sequifier.io.policy_window import PolicyWindowView + + view = PolicyWindowView(self.config) + for episode, length in zip(self.episodes, self.lengths): + for step in range(length): + view.episode_inputs(episode, step, None) + self._validated_statistics = {k: dict(v) for k, v in statistics.items()} + return + + def normalization_statistics(self): + """Use the same stable sample moments as supervised preprocessing.""" + moments = {} + columns = self.config.observation_columns + self.config.static_columns + for episode, length in zip(self.episodes, self.lengths): + for column in columns: + if column in self.config.categorical_n_classes: + continue + values = self.values(episode, length, column) + mean, std = _finite_mean_and_std(pl.Series(column, values), column) + size = len(values) + if column in moments: + count, old_mean, old_std = moments[column] + mean, std = get_combined_statistics( + size, mean, std, count, old_mean, old_std + ) + size += count + moments[column] = size, mean, std + return { + column: {"mean": mean, "std": std} + for column, (_, mean, std) in moments.items() + } diff --git a/src/sequifier/io/folder_dataset.py b/src/sequifier/io/folder_dataset.py index c37f917d4..c381ef0fc 100644 --- a/src/sequifier/io/folder_dataset.py +++ b/src/sequifier/io/folder_dataset.py @@ -12,6 +12,7 @@ from torch import Tensor from torch.utils.data import IterableDataset, get_worker_info +from sequifier.config.metadata import validate_persisted_metadata_versions from sequifier.helpers import ( WindowSampleIndex, configured_window_stride, @@ -20,7 +21,6 @@ stored_window_layout_from_metadata, ) from sequifier.io.batch import SequifierBatch -from sequifier.io.config import global_training from sequifier.io.images import ImageStore from sequifier.io.iteration_state import ( read_shared_int, @@ -45,6 +45,9 @@ class FolderDataset(IterableDataset): """Initialize the metadata and state shared by every folder loader.""" + total_batches: int + target_samples: int + def __init__(self, data_path: str, config: Any, shuffle: bool, format_hint: str): super().__init__() # DataLoader spawn workers do not inherit the process group. @@ -60,7 +63,7 @@ def __init__(self, data_path: str, config: Any, shuffle: bool, format_hint: str) if config.image_sources or config.preconditioning_image is not None else None ) - self.batch_size = global_training(config).batch_size + self.batch_size = config.batch_size self.shuffle = shuffle self._epoch_state = shared_int(0) self._start_batch_state = shared_int(0) @@ -74,6 +77,7 @@ def __init__(self, data_path: str, config: Any, shuffle: bool, format_hint: str) with open(metadata_path, "r") as f: metadata = json.load(f) + validate_persisted_metadata_versions(metadata) validate_folder_curriculum(config, metadata, self.data_dir) self.metadata = metadata @@ -89,7 +93,7 @@ def _initialize_window_layout(self) -> None: @beartype def _calculate_total_batches(self, target_samples: int) -> int: - num_workers = global_training(self.config).num_workers + num_workers = self.config.runtime_settings.num_workers num_workers_to_use = num_workers if num_workers > 0 else 1 total_batches = 0 @@ -118,6 +122,9 @@ def __len__(self) -> int: class EagerFolderDataset(FolderDataset): """Shared sample sharding for fully materialized folders.""" + n_samples: int + file_sample_orders: list[tuple[int, SampleOrderPlan]] + @beartype def _get_target_samples(self) -> int: world_size = self.world_size @@ -193,6 +200,10 @@ class LazyFolderChunk: class LazyFolderDataset(FolderDataset): """Schedule, order, and batch streamed folder chunks.""" + batch_files_info: list[dict[str, Any]] + chunk_format: str + ram_chunk_limit: int | None = None + @beartype def _get_target_samples(self) -> int: world_size = self.world_size @@ -291,6 +302,17 @@ def _iter_batches(self, worker_info: Any) -> Iterator[SequifierBatch]: file_path = os.path.join(self.data_dir, self.batch_files_info[f_id]["path"]) chunk = self._read_chunk(file_path) + if self.ram_chunk_limit is not None: + from sequifier.io.memory_budget import resident_bytes + + actual = resident_bytes(chunk) + if actual > self.ram_chunk_limit: + raise MemoryError( + f"Decoded shard {file_path!r} uses {actual / 1e9:.3f} GB, " + f"exceeding its planned {self.ram_chunk_limit / 1e9:.3f} GB " + "working allowance under max_ram_gb. Use smaller shards " + "or increase the data-loading RAM budget." + ) if len(chunk.sample_index) != file_samples: raise RuntimeError( f"Expanded sample count mismatch for {file_path}: " @@ -307,69 +329,67 @@ def _iter_batches(self, worker_info: Any) -> Iterator[SequifierBatch]: worker_file_start_idx = max(0, worker_start_sample - file_start) worker_file_end_idx = min(file_samples, worker_end_sample - file_start) worker_indices = indices[worker_file_start_idx:worker_file_end_idx] - logical_positions = torch.arange( - file_start + worker_file_start_idx, - file_start + worker_file_end_idx, - dtype=torch.int64, - ) - # Repeated file cycles are padding even when their source rows are real. - sample_is_real = logical_positions < rank_real_samples if len(worker_indices) == 0: + del chunk continue - - new_batch = build_window_batch( - chunk.sequences, - self.config.input_columns, - self.config.target_columns, - chunk.sample_index, - worker_indices, - sample_is_real, - depth_valid_masks=chunk.depth_valid_masks, - start_item_positions=chunk.start_item_positions, - split_start_item_positions=chunk.split_start_item_positions, - split_end_item_positions=chunk.split_end_item_positions, - sequence_ids=chunk.sequence_ids, - static_table=self.static_table, - image_store=self.image_store, - ) - new_seq = new_batch.inputs - new_tgt = new_batch.targets - new_meta = new_batch.metadata - if buffer_len == 0: - seq_buffer = new_seq - tgt_buffer = new_tgt - meta_buffer = new_meta - else: - seq_buffer = { - k: torch.cat([seq_buffer[k], new_seq[k]], dim=0) for k in seq_buffer - } - tgt_buffer = { - k: torch.cat([tgt_buffer[k], new_tgt[k]], dim=0) for k in tgt_buffer - } - if set(meta_buffer) != set(new_meta): - raise RuntimeError( - "Inconsistent leftPadLength metadata across " - f"{self.chunk_format} chunks." - ) - meta_buffer = { - k: torch.cat([meta_buffer[k], new_meta[k]], dim=0) - for k in meta_buffer - } - buffer_len += len(worker_indices) - - while buffer_len >= self.batch_size: + offset = 0 + while offset < len(worker_indices): if yielded_samples >= worker_target_samples: break - yield SequifierBatch( - inputs={k: v[: self.batch_size] for k, v in seq_buffer.items()}, - targets={k: v[: self.batch_size] for k, v in tgt_buffer.items()}, - metadata={k: v[: self.batch_size] for k, v in meta_buffer.items()}, + # Expanding an entire shard here can dwarf the stored payload, + # particularly with images, depth or multiple views per row. + count = min(self.batch_size - buffer_len, len(worker_indices) - offset) + positions = torch.arange( + file_start + worker_file_start_idx + offset, + file_start + worker_file_start_idx + offset + count, + dtype=torch.int64, + ) + new_batch = build_window_batch( + chunk.sequences, + self.config.input_columns, + self.config.target_columns, + chunk.sample_index, + worker_indices[offset : offset + count], + positions < rank_real_samples, + depth_valid_masks=chunk.depth_valid_masks, + start_item_positions=chunk.start_item_positions, + split_start_item_positions=chunk.split_start_item_positions, + split_end_item_positions=chunk.split_end_item_positions, + sequence_ids=chunk.sequence_ids, + static_table=self.static_table, + image_store=self.image_store, ) - yielded_samples += self.batch_size - seq_buffer = {k: v[self.batch_size :] for k, v in seq_buffer.items()} - tgt_buffer = {k: v[self.batch_size :] for k, v in tgt_buffer.items()} - meta_buffer = {k: v[self.batch_size :] for k, v in meta_buffer.items()} - buffer_len -= self.batch_size + if buffer_len == 0: + seq_buffer = new_batch.inputs + tgt_buffer = new_batch.targets + meta_buffer = new_batch.metadata + else: + if set(meta_buffer) != set(new_batch.metadata): + raise RuntimeError( + f"Inconsistent metadata across {self.chunk_format} chunks." + ) + seq_buffer = { + k: torch.cat([seq_buffer[k], new_batch.inputs[k]], dim=0) + for k in seq_buffer + } + tgt_buffer = { + k: torch.cat([tgt_buffer[k], new_batch.targets[k]], dim=0) + for k in tgt_buffer + } + meta_buffer = { + k: torch.cat([meta_buffer[k], new_batch.metadata[k]], dim=0) + for k in meta_buffer + } + offset += count + buffer_len += count + if buffer_len == self.batch_size: + yield SequifierBatch(seq_buffer, tgt_buffer, meta_buffer) + yielded_samples += buffer_len + seq_buffer, tgt_buffer, meta_buffer = {}, {}, {} + buffer_len = 0 + del new_batch + # Do not retain the previous chunk while decoding its successor. + del chunk if buffer_len > 0 and yielded_samples < worker_target_samples: final_yield_size = min(buffer_len, worker_target_samples - yielded_samples) diff --git a/src/sequifier/io/image_feature_cache.py b/src/sequifier/io/image_feature_cache.py new file mode 100644 index 000000000..5cecc6d91 --- /dev/null +++ b/src/sequifier/io/image_feature_cache.py @@ -0,0 +1,28 @@ +"""Scoped, transient identities for owned policy frames; never checkpoint state.""" + +from contextlib import contextmanager +from contextvars import ContextVar +from itertools import count + +IMAGE_CACHE_PREFIX = "__image_cache_id:" +_enabled = ContextVar("sequifier_image_feature_cache", default=False) +_identities = count(1) + + +def image_feature_cache_enabled(): + return _enabled.get() + + +def new_image_identity(): + # Frames own their image data. A monotonic process-local identity avoids + # hashing pixels or conflating visually identical but independently owned rows. + return next(_identities) + + +@contextmanager +def image_identity_scope(enabled): + token = _enabled.set(enabled) + try: + yield + finally: + _enabled.reset(token) diff --git a/src/sequifier/io/images.py b/src/sequifier/io/images.py index 5efe73735..004799a36 100644 --- a/src/sequifier/io/images.py +++ b/src/sequifier/io/images.py @@ -16,6 +16,7 @@ import torch from sequifier.config.image_config import ImageSourceConfig, PreconditioningImageConfig +from sequifier.contract_versions import ContractVersions, require_contract_version from sequifier.helpers import normalize_path IMAGE_EXTENSIONS = {".jpg", ".jpeg", ".png", ".webp", ".bmp", ".tif", ".tiff"} @@ -32,9 +33,6 @@ class _VideoIndex: by_pts: dict[int, int] -_VIDEO_INDEX_VERSION = 1 - - def _media_signature(path: Path) -> dict[str, int]: stat = path.stat() return {"size": stat.st_size, "mtime_ns": stat.st_mtime_ns} @@ -93,7 +91,7 @@ def _time_ms(value: Any, *, absolute: bool) -> int: return ((hours * 60 + minutes) * 60 + seconds) * 1000 + millis -def _files(path: str, project_root: str) -> list[Path]: +def media_files(path: str, project_root: str) -> list[Path]: directory = Path(normalize_path(path, project_root)) if not directory.is_dir(): raise ValueError(f"Image source is not a directory: {directory}") @@ -106,7 +104,7 @@ def _files(path: str, project_root: str) -> list[Path]: def _signature(path: str, project_root: str) -> str: digest = hashlib.sha256() - for file in _files(path, project_root): + for file in media_files(path, project_root): stat = file.stat() digest.update(f"{file.name}\0{stat.st_size}\0{stat.st_mtime_ns}\n".encode()) return digest.hexdigest() @@ -141,7 +139,7 @@ def _index( extensions = IMAGE_EXTENSIONS if kind == "images" else VIDEO_EXTENSIONS result: dict[int, dict[int, Path]] = {} timing: dict[int, dict[int, tuple[str | None, int | None]]] = {} - for file in _files(path, project_root): + for file in media_files(path, project_root): if file.suffix.lower() not in extensions: raise ValueError(f"Unsupported file in image source: {file}") if preconditioning: @@ -247,7 +245,7 @@ def _scan_video(path: Path) -> dict[str, Any]: else None ) return { - "version": _VIDEO_INDEX_VERSION, + "version": ContractVersions.VIDEO_FRAME_INDEX, "media": _media_signature(path), "time_base": [base.numerator, base.denominator] if base else None, "frame_count": len(stamps), @@ -424,7 +422,7 @@ def validate_sources( "video_index_paths": video_index_paths.get(name, {}), } if preconditioning is not None: - if preconditioning.path is None: + if preconditioning.paths is None: raise ValueError("Preconditioning images need a source path") for name, path in preconditioning.views.items(): index, _, _ = _index( @@ -443,7 +441,7 @@ def validate_sources( f"Cannot decode preconditioning image {frames[0]}: {error}" ) from error manifest["preconditioning_image"] = { - **preconditioning.model_dump(mode="json", exclude={"path"}), + **preconditioning.model_dump(mode="json", exclude={"paths"}), "view_names": sorted(preconditioning.views), "signature": _preconditioning_signature(preconditioning, project_root), } @@ -463,14 +461,14 @@ def __init__( self.preconditioning = preconditioning for name, source in sources.items(): if source.path is None: - raise ValueError(f"Image view {name!r} needs image_path") + raise ValueError(f"Image view {name!r} needs image_paths") if ( source.signature is not None and _signature(source.path, project_root) != source.signature ): raise ValueError(f"Image view {name!r} changed after preprocessing") - if preconditioning is not None and preconditioning.path is None: - raise ValueError("Preconditioning images need preconditioning_image_path") + if preconditioning is not None and preconditioning.paths is None: + raise ValueError("Preconditioning images need preconditioning_image_paths") if ( preconditioning is not None and preconditioning.signature is not None @@ -486,7 +484,7 @@ def __init__( self.item_times: dict[int, list[int]] | None = None for name, source in sources.items(): if source.path is None: - raise ValueError(f"Image view {name!r} needs image_path") + raise ValueError(f"Image view {name!r} needs image_paths") index, timing, mode = _index(source.path, project_root, kind=source.type) self.indices[name] = index if source.type == "videos": @@ -629,6 +627,11 @@ def _video_record(self, path: Path) -> dict[str, Any]: try: with index_path.open() as file: record = json.load(file) + require_contract_version( + "video frame index", + record.get("version") if isinstance(record, dict) else None, + ContractVersions.VIDEO_FRAME_INDEX, + ) import av with av.open(str(path)) as container: @@ -636,7 +639,6 @@ def _video_record(self, path: Path) -> dict[str, Any]: time_base = [base.numerator, base.denominator] if base else None if ( not isinstance(record, dict) - or record.get("version") != _VIDEO_INDEX_VERSION or record.get("media") != _media_signature(path) or record.get("time_base") != time_base or not isinstance(record.get("pts"), list) @@ -681,7 +683,7 @@ def _video_record(self, path: Path) -> dict[str, Any]: ) ) ): - raise ValueError("invalid version, signature, or frame data") + raise ValueError("invalid signature or frame data") except (OSError, ValueError, TypeError, KeyError) as error: raise ValueError( f"Video index for {path} is missing, stale, or invalid: {error}" diff --git a/src/sequifier/io/iql_dataset.py b/src/sequifier/io/iql_dataset.py new file mode 100644 index 000000000..632af227b --- /dev/null +++ b/src/sequifier/io/iql_dataset.py @@ -0,0 +1,118 @@ +"""Fixed offline transitions with causal current and successor contexts.""" + +from dataclasses import dataclass + +import torch + +from sequifier.io.actions import action_batch +from sequifier.io.batch import SequifierInputs +from sequifier.io.policy_window import PolicyWindowView +from sequifier.io.tensors import TensorRecord +from sequifier.io.values import numeric_scalar + + +@dataclass(frozen=True) +class IQLBatch(TensorRecord): + current: SequifierInputs + successor: SequifierInputs + action: torch.Tensor | dict[str, torch.Tensor] + reward: torch.Tensor + bootstrap: torch.Tensor + + +class IQLTransitionDataset: + """Index eligible decisions while retaining the source's bounded episode cache.""" + + def __init__(self, decisions): + self.decisions = decisions + self.view = PolicyWindowView(decisions.config) + self.cumulative_lengths = decisions.cumulative_lengths + self.episode_ids = decisions.episode_ids + for episode_index, episode in enumerate(decisions.episodes): + source = episode_index + transitions = episode.transitions + if transitions is None or any( + values is None + for values in ( + transitions.rewards, + transitions.terminated, + transitions.truncated, + transitions.bootstrap_allowed, + ) + ): + raise ValueError( + f"IQL source {source!r} episode {episode.episode_id!r} " + "needs complete transition columns" + ) + for step in range(len(episode.actions)): + location = ( + f"IQL source {source!r} episode {episode.episode_id!r} " + f"step {episode.item_positions[step]}" + ) + reward = numeric_scalar(transitions.rewards[step], f"{location} reward") + if not torch.isfinite(torch.tensor(reward, dtype=torch.float32)): + raise ValueError(f"{location}: reward exceeds finite float32 range") + terminated = transitions.terminated[step] + bootstrap = transitions.bootstrap_allowed[step] + successor = self._successor(episode, step) + if not terminated and not bootstrap: + raise ValueError( + f"{location}: nonterminal transition must bootstrap" + ) + if bootstrap and successor is None: + raise ValueError(f"{location}: bootstrap needs the true successor") + + def validate_normalization(self, actor): + # Interior successors are ordinary observations, already checked by the + # policy dataset. Only the final successor can contain additional values. + for episode in self.decisions.episodes: + step = len(episode.actions) - 1 + successor = self._successor(episode, step) + if successor is None: + continue + inputs = self.view.successor_inputs(episode, step, successor) + features, _ = actor.model_inputs(inputs.to(actor.device)) + if any( + t.is_floating_point() and not torch.isfinite(t).all() + for t in features.values() + ): + raise ValueError( + f"episode {episode.episode_id}: non-finite normalized IQL successor" + ) + + @staticmethod + def _successor(episode, step): + transitions = episode.transitions + if transitions is not None and transitions.successors is not None: + supplied = transitions.successors[step] + if supplied is not None: + return supplied + if step + 1 < len(episode.actions): + return episode.observation(step + 1) + return None + + def __len__(self): + return len(self.decisions) + + def __getitem__(self, index): + episode_index, step = self.decisions.locate(index) + episode = self.decisions.episodes[episode_index] + transitions = episode.transitions + successor = self._successor(episode, step) + current = self.view.episode_inputs(episode, step, None) + next_inputs = self.view.successor_inputs( + episode, + step, + successor if successor is not None else episode.observation(step), + ) + return IQLBatch( + current=current, + successor=next_inputs, + action=action_batch( + [episode.actions[step]], self.decisions.config, encoded=False + ), + reward=torch.tensor([transitions.rewards[step]], dtype=torch.float32), + bootstrap=torch.tensor( + [transitions.bootstrap_allowed[step]], dtype=torch.bool + ), + ) diff --git a/src/sequifier/io/memory_budget.py b/src/sequifier/io/memory_budget.py new file mode 100644 index 000000000..68c2aae6b --- /dev/null +++ b/src/sequifier/io/memory_budget.py @@ -0,0 +1,316 @@ +"""Preflight and resident-storage accounting for prepared-data loading. + +The budget belongs to one training rank and its loader workers. It does not cap +model/optimizer memory or the Python process's address space. Estimates reserve +decode/conversion and prefetch headroom before any dataset is materialized. +""" + +from __future__ import annotations + +import csv +import json +import math +import sys +import zipfile +from dataclasses import dataclass, replace +from decimal import Decimal +from pathlib import Path +from typing import Any + +import torch +from loguru import logger + +from sequifier.config.metadata import validate_persisted_metadata_versions +from sequifier.helpers import PANDAS_TO_TORCH_TYPES, normalize_path + + +def resident_bytes(value: Any, seen: set[Any] | None = None) -> int: + """Count CPU backing stores once, including aliased views and Python indices.""" + if seen is None: + seen = set() + if id(value) in seen: + return 0 + seen.add(id(value)) + if isinstance(value, torch.Tensor): + if value.device.type != "cpu": + return 0 + storage = value.untyped_storage() + identity = ("storage", storage._cdata) + if identity in seen: + return 0 + seen.add(identity) + return sys.getsizeof(value) + storage.nbytes() + size = sys.getsizeof(value) + if isinstance(value, dict): + return size + sum( + resident_bytes(key, seen) + resident_bytes(item, seen) + for key, item in value.items() + if key != "config" + ) + if isinstance(value, (tuple, list, set, frozenset)): + return size + sum(resident_bytes(item, seen) for item in value) + if isinstance(value, type): + return size + try: + fields = vars(value) + except TypeError: + return size + return size + resident_bytes(fields, seen) + + +def _parquet_size(path: Path, string_width: int) -> tuple[int, int]: + import pyarrow as pa + import pyarrow.parquet as pq + + metadata = pq.ParquetFile(path).metadata + schema = pq.read_schema(path) + row_width = 0 + for field in schema: + dtype = field.type + if pa.types.is_dictionary(dtype): + dtype = dtype.value_type + if pa.types.is_boolean(dtype): + row_width += 1 + elif pa.types.is_integer(dtype) or pa.types.is_floating(dtype): + row_width += dtype.bit_width // 8 + else: + # Prepared tables contain numeric values plus feature-name strings. + row_width += string_width + 8 + row_width += 1 # Validity and alignment headroom. + decoded = metadata.num_rows * row_width + pages = sum( + metadata.row_group(i).total_byte_size for i in range(metadata.num_row_groups) + ) + return max(decoded, pages), metadata.num_rows + + +def _source_size( + path: Path, read_format: str, string_width: int +) -> tuple[int, int | None]: + if read_format == "parquet": + return _parquet_size(path, string_width) + if read_format == "pt": + if not zipfile.is_zipfile(path): + raise ValueError( + f"RAM-budget preflight requires a current ZIP-based PT payload: {path}" + ) + with zipfile.ZipFile(path) as archive: + return sum(item.file_size for item in archive.infolist()), None + if read_format == "csv": + # Streaming inspection counts quoted records correctly without loading + # feature values into a DataFrame just to estimate its memory demand. + rows, decoded = 0, 0 + with path.open(newline="", encoding="utf-8-sig") as stream: + reader = csv.reader(stream) + next(reader, None) + for row in reader: + rows += 1 + decoded += sum(max(8, len(item.encode("utf-8")) + 8) for item in row) + return decoded, rows + raise ValueError(f"Cannot plan RAM for prepared format {read_format!r}") + + +def _column_bytes(config: Any, column: str) -> int: + dtype = PANDAS_TO_TORCH_TYPES[config.column_data_types[column]] + width = torch.empty((), dtype=dtype).element_size() + layout = config.depth_layouts.layout_for_column(column) + return width * (layout.max_depth_length if layout is not None else 1) + + +def _batch_bytes(config: Any) -> int: + context = config.window_view.context_length + values = context * sum( + _column_bytes(config, column) + for column in config.input_columns + config.target_columns + ) + values += context * ( + 32 + sum(layout.max_depth_length for _, layout in config.depth_layouts.items()) + ) + values += sum(_column_bytes(config, column) for column in config.static_columns) + for source in config.image_sources.values(): + values += context * source.frame_capacity * (3 * source.crop_size**2 + 1) + prefix = config.preconditioning_image + if prefix is not None: + values += len(prefix.views) * 3 * prefix.crop_size**2 + return config.batch_size * (values + 64) + + +@dataclass(frozen=True) +class LoaderMemoryEstimate: + resident: int + working: int + chunk_limit: int + + @property + def total(self) -> int: + return self.resident + self.working + + +def _estimate_loader(path: str, config: Any, eager: bool) -> LoaderMemoryEstimate: + source = Path(normalize_path(path, config.project_root)) + width = config.storage_layout.window_length + infos = None + metadata_bytes = 0 + if source.is_dir(): + with (source / "metadata.json").open() as stream: + metadata = json.load(stream) + validate_persisted_metadata_versions(metadata) + metadata_bytes = resident_bytes(metadata) + width = metadata["window_length"] + infos = metadata["batch_files"] + files = ( + [(source / info["path"], int(info["samples"])) for info in infos] + if infos is not None + else [(source, None)] + ) + columns = set(config.input_columns + config.target_columns) + stored_row = width * sum(_column_bytes(config, column) for column in columns) + 48 + stored_row += width * sum( + layout.max_depth_length for _, layout in config.depth_layouts.items() + ) + stride = config.window_stride + views = ( + 1 + if stride is None + else max( + 1, + ( + width + - config.storage_layout.max_target_offset + - config.window_view.context_length + ) + // stride + + 1, + ) + ) + string_width = ( + max((len(name.encode("utf-8")) for name in config.column_data_types), default=0) + + 64 + ) + residents, decode_peak = [], 0 + for file, samples in files: + decoded, rows = _source_size(file, config.read_format, string_width) + if samples is None: + samples = ( + math.ceil(rows / max(1, len(columns))) + if rows is not None + else math.ceil(decoded / max(1, stored_row)) + ) + # Window indices, curriculum order/tie groups and epoch shuffles include + # Python lists as well as tensor indices. Reserve their expanded size. + indices = samples * (64 + 160 * views) + resident = samples * stored_row + indices + residents.append(resident) + decode_peak = max(decode_peak, 3 * decoded + 2 * resident) + static = 0 + if config.static_features_path is not None and config.static_columns: + static_path = Path( + normalize_path(config.static_features_path, config.project_root) + ) + static = 3 * _parquet_size(static_path, string_width)[0] + image_sources = list(config.image_sources.values()) + if config.preconditioning_image is not None: + image_sources.append(config.preconditioning_image) + cache = 128 * max((3 * item.crop_size**2 for item in image_sources), default=0) + workers = config.runtime_settings.num_workers + copies = max(1, workers) + batch = _batch_bytes(config) + # PyTorch's worker queue has four prefetched batches per worker. Pinning + # can overlap the queued copy; reserve both it and the consumer's batch. + prefetch = 2 * (4 * workers + 1) * batch + base = metadata_bytes + static + if eager: + resident = (sum(residents) + base) * (workers + 1) + working = decode_peak + copies * cache + prefetch + else: + resident = base * (workers + 1) + working = copies * (decode_peak + cache + 2 * batch) + prefetch + return LoaderMemoryEstimate(resident, working, decode_peak) + + +@dataclass +class DataLoadingBudget: + limit: int + estimates: dict[str, LoaderMemoryEstimate] + + def check(self) -> None: + required = sum(estimate.total for estimate in self.estimates.values()) + if required > self.limit: + raise MemoryError( + "Prepared-data loading exceeds run.execution.data_loading.max_ram_gb: " + f"planned {required / 1e9:.3f} GB, budget {self.limit / 1e9:.3f} GB " + "per training rank including loader workers. Use lazy folders " + "(load_full_data_to_ram: false), smaller shards/batches, fewer " + "workers, or increase max_ram_gb. Model/optimizer memory is additional." + ) + + def validate_dataset(self, key: str, dataset: Any, workers: int) -> None: + estimate = self.estimates[key] + measured = resident_bytes(dataset) * (workers + 1) + self.estimates[key] = replace( + estimate, resident=max(estimate.resident, measured) + ) + self.check() + # Plain integers survive DataLoader spawn without serializing the + # run-wide budget or retaining references to other loaders. + dataset.ram_chunk_limit = estimate.chunk_limit + + +def plan_data_loading(config: Any) -> DataLoadingBudget: + """Budget every cached source and the largest phase working requirements.""" + from sequifier.config.train_config import dataset_part_view + from sequifier.evaluation.compatibility import evaluation_batch_size + + settings = config.runtime_settings + if not math.isfinite(settings.max_ram_gb) or settings.max_ram_gb <= 0: + raise ValueError("max_ram_gb must be finite and positive") + budget = DataLoadingBudget(int(Decimal(str(settings.max_ram_gb)) * 10**9), {}) + phases = config.training_plan + batch = max( + [phase.batch_size for phase in phases] + + [evaluation_batch_size(phase) for phase in phases], + default=1, + ) + strides = [ + phase.learning.window_stride + for phase in phases + if phase.learning.window_stride is not None + ] + stride = min(strides) if strides else None + for name, dataset in config.dataset_training.items(): + for part_name, part in dataset.parts.items(): + selected = dataset_part_view( + config, + name, + part_name, + batch_size=batch, + window_stride=stride, + ) + for split, path in ( + ("training", part.training_data_path), + ("validation", part.validation_data_path), + ): + if path is None: + continue + normalized = normalize_path(path, config.project_root) + splits = [ + normalize_path(item, config.project_root) + for item in part.metadata.split_paths + ] + layout = ( + part.metadata.layout_for_split(splits.index(normalized)) + if normalized in splits + else selected.storage_layout + ) + view = replace(selected, storage_layout=layout) + eager = part.storage_form == "file" or settings.load_full_data_to_ram + budget.estimates[f"{name}.{part_name}.{split}"] = _estimate_loader( + path, view, eager + ) + budget.check() + logger.info( + "Prepared-data RAM plan: {:.3f} GB / {:.3f} GB per rank (including workers)", + sum(item.total for item in budget.estimates.values()) / 1e9, + budget.limit / 1e9, + ) + return budget diff --git a/src/sequifier/io/output_paths.py b/src/sequifier/io/output_paths.py new file mode 100644 index 000000000..7e8ed7539 --- /dev/null +++ b/src/sequifier/io/output_paths.py @@ -0,0 +1,187 @@ +"""Shared path geometry and destination checks.""" + +import re +from pathlib import Path + + +def unique_paths(paths, message): + paths = [Path(path).resolve() for path in paths] + if len(set(paths)) != len(paths): + raise ValueError(message) + return paths + + +def contains_path(directory, path): + return directory == path or directory in path.parents + + +def paths_overlap(left, right): + return contains_path(left, right) or contains_path(right, left) + + +def validate_path_conflicts(paths, protected, *, label, check_each_other=True): + protected = {Path(path).resolve() for path in protected} + paths = [Path(path).resolve() for path in paths] + for path in paths: + others = protected | (set(paths) - {path} if check_each_other else set()) + for other in others: + if paths_overlap(path, other): + raise ValueError(f"{label} {path} conflicts with input/output {other}") + + +def validate_inference_destination(destination, inputs, *, label): + entry = Path(destination) + # Atomic replacement acts on the directory entry even if it is a symlink. + entry = entry.parent.resolve() / entry.name + resolved = entry.resolve() + for input_path in inputs: + source = Path(input_path).resolve() + conflict = paths_overlap(resolved, source) or paths_overlap(entry, source) + if not conflict: + try: + conflict = resolved.samefile(source) + except (FileNotFoundError, NotADirectoryError): + pass + if conflict: + raise ValueError( + f"{label} {destination!s} conflicts with input {input_path}" + ) + + +def require_file_destinations(paths): + for path in paths: + if path.is_dir() or any(p.exists() and not p.is_dir() for p in path.parents): + raise ValueError(f"output requires a file destination: {path}") + + +def require_directory_destinations(paths): + for path in paths: + if path.exists() and not path.is_dir(): + raise ValueError(f"archive requires a directory: {path}") + + +def validate_training_destinations(config, checkpoint, protected, *, phase_offset=0): + """Reserve generated trees as well as explicit files before any publication.""" + checkpoint = Path(checkpoint).resolve() + files = unique_paths( + [config.model_path, config.report_path, checkpoint], + "training model, report and checkpoint paths must be distinct", + ) + manifest = checkpoint.parent / "manifest.json" + if manifest != Path(config.report_path).resolve(): + files = unique_paths([*files, manifest], "manifest overlaps a training output") + directories = [Path(config.phase_directory).resolve()] + from sequifier.logging_paths import model_log_directory + + directories.append( + model_log_directory(config.project_root, config.model_name).resolve() + ) + from sequifier.rl.run_config import ResolvedPolicyRun + + if isinstance(config, ResolvedPolicyRun): + phases, resume = config.phases, config.resume + dataset_names = () + else: + phases, resume = config.training_plan, config.runtime_settings.resume + dataset_names = tuple(config.dataset_training) + from sequifier.artifacts.phase import PhaseArtifacts + + generated = [] + for index, phase in enumerate(phases, phase_offset): + artifacts = PhaseArtifacts.for_phase(config.phase_directory, index, phase.name) + for path in ( + artifacts.model, + artifacts.report, + artifacts.model.parent / "commits", + artifacts.model.with_name("run.pt"), + artifacts.model.parent / "native", + artifacts.model.parent / "checkpoints", + artifacts.telemetry.parent, + artifacts.telemetry, + ): + resolved = path.resolve() + if not contains_path(directories[0], resolved): + raise ValueError(f"phase destination escapes phase_directory: {path}") + generated.append(resolved) + validate_path_conflicts( + generated, protected, label="phase output", check_each_other=False + ) + directories.extend( + Path(archive).resolve() + for phase in phases + if (archive := phase.rollout_archive_directory) + ) + directories = unique_paths( + directories, "training output directories must be distinct" + ) + require_file_destinations(files) + require_directory_destinations(directories) + # A generated snapshot may be selected as the rolling checkpoint on resume. + snapshot_phases = { + ( + PhaseArtifacts.for_phase( + config.phase_directory, index, phase.name + ).model.parent + / "checkpoints" + ).resolve(): phase + for index, phase in enumerate(phases, phase_offset) + } + snapshot_phase = snapshot_phases.get(checkpoint.parent) + snapshot_name = snapshot_phase is not None and ( + checkpoint.name.startswith(f"{config.model_name}-") + if snapshot_phase.algorithm == "supervised" + else re.fullmatch(r"transaction-\d{8,}\.pt", checkpoint.name) is not None + ) + snapshot_resume = ( + resume is not None + and resume.policy != "never" + and checkpoint.is_file() + and snapshot_name + and checkpoint.suffix == ".pt" + and manifest != Path(config.report_path).resolve() + ) + if snapshot_resume: + external_files = [path for path in files if path not in {checkpoint, manifest}] + validate_path_conflicts( + [*external_files, *directories], protected, label="training output" + ) + validate_path_conflicts( + [checkpoint, manifest], + protected | set(external_files) | set(directories[1:]), + label="training output", + ) + else: + validate_path_conflicts( + [*files, *directories], protected, label="training output" + ) + if any(phase.algorithm == "supervised" for phase in phases): + export_directory = (Path(config.project_root) / "models").resolve() + names = [config.model_name] + [ + f"{config.model_name}-{name}" for name in dataset_names + ] + pattern = re.compile( + "(?:" + + "|".join(re.escape(name) for name in names) + + r")-(?:best|last)(?:-embedding)?-\d+\.(?:pt|onnx)" + ) + for path in [*files, *directories, *(Path(p).resolve() for p in protected)]: + if contains_path(path, export_directory) or ( + path.parent == export_directory and pattern.fullmatch(path.name) + ): + raise ValueError( + f"training destination/input overlaps generated exports: {path}" + ) + + +def resolve_training_output_paths(config): + root = Path(config.project_root) + output_dir = Path("outputs/training") / config.model_name + config.model_path = str( + (root / (config.model_path or f"models/{config.model_name}.pt")).resolve() + ) + config.report_path = str( + (root / (config.report_path or output_dir / "report.json")).resolve() + ) + config.phase_directory = str( + (root / (config.phase_directory or output_dir / "phases")).resolve() + ) diff --git a/src/sequifier/io/partitioning.py b/src/sequifier/io/partitioning.py new file mode 100644 index 000000000..395bd97a6 --- /dev/null +++ b/src/sequifier/io/partitioning.py @@ -0,0 +1,35 @@ +"""Leakage-safe connected components and deterministic held-out partitions.""" + +import random + + +def split_components(records, leakage_keys, validation_fraction, seed): + if not 0 < validation_fraction < 1: + raise ValueError("validation_fraction must be in (0,1)") + records = list(records) + parents = list(range(len(records))) + + def root(index): + while parents[index] != index: + parents[index] = parents[parents[index]] + index = parents[index] + return index + + seen = {} + for index, record in enumerate(records): + for key in leakage_keys(record): + if key in seen: + parents[root(index)] = root(seen[key]) + seen[key] = index + groups = {} + for index, record in enumerate(records): + groups.setdefault(root(index), []).append(record) + components = list(groups.values()) + if len(components) < 2: + raise ValueError("splitting requires at least two independent components") + random.Random(seed).shuffle(components) + count = min(len(components) - 1, max(1, int(len(components) * validation_fraction))) + return tuple( + [record for group in groups for record in group] + for groups in (components[count:], components[:count]) + ) diff --git a/src/sequifier/io/policy_dataset.py b/src/sequifier/io/policy_dataset.py new file mode 100644 index 000000000..a4f250a00 --- /dev/null +++ b/src/sequifier/io/policy_dataset.py @@ -0,0 +1,260 @@ +"""Episode-safe policy views over ordinary sequences and imported rollout logs.""" + +from bisect import bisect_right +from collections.abc import Sequence + +import torch +from torch.utils.data import Dataset, Sampler + +from sequifier.config.policy_inputs import PolicyEpisodeSpec +from sequifier.io.batch import SequifierBatch +from sequifier.io.episode_cache import EpisodeCache +from sequifier.io.episode_corpus import EpisodeCorpus, EpisodeView, episode_content_key +from sequifier.io.episode_features import EpisodeFeatures, return_labels +from sequifier.io.partitioning import split_components +from sequifier.io.policy_window import PolicyWindowView, action_index + + +class EpisodeDecisions(EpisodeFeatures): + """Episode-safe decision indexing shared by imitation and transition views.""" + + def __init__( + self, + sources: Sequence | EpisodeView, + config: PolicyEpisodeSpec, + *, + episode_cache_size: int = 8, + cache_owner=None, + ): + if episode_cache_size < 1: + raise ValueError("episode_cache_size must be positive") + self.config = config + self.episode_cache_size = episode_cache_size + self.exclusions = {} + self.all_episode_ids = [] + self.episode_ids = [] + self.demonstration_keys = [] + self.group_ids = [] + self.lengths = [] + self.cumulative_lengths = [] + self.cache_owner = cache_owner + self._labels = ( + cache_owner.episode_cache(episode_cache_size) + if cache_owner is not None + else EpisodeCache(episode_cache_size) + ) + view = ( + sources + if isinstance(sources, EpisodeView) + else EpisodeCorpus( + sources, config, episode_cache_size, cache_owner=cache_owner + ).view() + ) + eligible = [] + seen = set() + for index in range(len(view)): + record = view.metadata(index) + episode_id = record.episode_id + if episode_id in seen: + raise ValueError(f"duplicate episode ID {episode_id!r}") + seen.add(episode_id) + self.all_episode_ids.append(episode_id) + length = record.length + demonstration = record.content_key + if config.return_conditioned: + episode = view[index] + length = episode.return_length(config.returns.finite_horizon) + if length is None: + self.exclusions[episode_id] = ( + "missing_rewards" + if episode.transitions is None + or episode.transitions.rewards is None + else "unknown_return_tail" + ) + continue + demonstration = episode_content_key(episode, length=length) + eligible.append(index) + self.episode_ids.append(episode_id) + self.demonstration_keys.append(demonstration) + self.group_ids.append(record.group_id) + self.lengths.append(length) + self.cumulative_lengths.append( + length + (self.cumulative_lengths[-1] if self.cumulative_lengths else 0) + ) + self.episodes = view.subset(eligible) + self.episode_fingerprints = [ + self.episodes.metadata(i).fingerprint for i in range(len(self.episodes)) + ] + super().__init__(self.episodes, config, self.lengths) + + def __len__(self): + return self.cumulative_lengths[-1] if self.cumulative_lengths else 0 + + def sampler(self, generator): + return EpisodeWindowSampler(self, generator) + + def subset(self, indices): + return type(self)( + self.episodes.subset(indices), + self.config, + episode_cache_size=self.episode_cache_size, + cache_owner=self.cache_owner, + ) + + def _episode_labels(self, index: int): + return self._labels.get(index, self._load_labels) + + def _load_labels(self, index): + episode = self.episodes[index] + length = self.lengths[index] + actions = [ + action_index(value, self.config) for value in episode.actions[:length] + ] + returns = ( + return_labels(episode, length, self.config.returns.gamma) + if self.config.return_conditioned + else None + ) + return actions, returns + + def locate(self, index): + if index < 0: + index += len(self) + if not 0 <= index < len(self): + raise IndexError(index) + episode_index = bisect_right(self.cumulative_lengths, index) + step = index - ( + self.cumulative_lengths[episode_index - 1] if episode_index else 0 + ) + return episode_index, step + + +class EpisodeWindowSampler(Sampler): + """Shuffle episodes and their decisions while keeping disk access local.""" + + def __init__(self, dataset, generator): + self.dataset, self.generator = dataset, generator + + def __len__(self): + return len(self.dataset) + + def __iter__(self): + for episode in torch.randperm( + len(self.dataset.lengths), generator=self.generator + ).tolist(): + offset = self.dataset.cumulative_lengths[episode - 1] if episode else 0 + for step in torch.randperm( + self.dataset.lengths[episode], generator=self.generator + ).tolist(): + yield offset + step + + +def episode_partition_indices( + dataset, validation_fraction=0.2, seed=1010, *, group_identical=False +): + def keys(index): + result = ( + [] + if dataset.group_ids[index] is None + else [("group", dataset.group_ids[index])] + ) + if group_identical: + result.append(("content", dataset.demonstration_keys[index])) + return result + + return tuple( + tuple(indices) + for indices in split_components( + range(len(dataset.episodes)), keys, validation_fraction, seed + ) + ) + + +def split_by_episode( + dataset, validation_fraction=0.2, seed=1010, *, group_identical=False +): + return tuple( + dataset.subset(indices) + for indices in episode_partition_indices( + dataset, validation_fraction, seed, group_identical=group_identical + ) + ) + + +class PolicyDataCatalog: + """Run-owned corpora and reusable decision views for named input sources.""" + + def __init__( + self, + sources, + config, + cache_size, + *, + preference_manifests=None, + episode_records=None, + cache_owner=None, + ): + self.sources, self.config, self.cache_size = sources, config, cache_size + self.cache_owner = cache_owner + self.corpora = {} + self.episode_records = episode_records if episode_records is not None else {} + self.datasets = {} + self.preference_manifests = ( + preference_manifests if preference_manifests is not None else {} + ) + + def preference_pairs(self, path): + if path not in self.preference_manifests: + from sequifier.trajectory.preferences import load_preferences + + self.preference_manifests[path] = load_preferences(path).pairs + return self.preference_manifests[path] + + def dataset(self, names): + key = tuple(names) + if key not in self.datasets: + for name in key: + if name not in self.corpora: + corpus = EpisodeCorpus( + [self.sources[name]], + self.config, + self.cache_size, + cache_owner=self.cache_owner, + records=self.episode_records.get(name), + ) + self.corpora[name] = corpus + self.episode_records[name] = corpus.records + self.datasets[key] = EpisodeDecisions( + EpisodeView.combine([self.corpora[name].view() for name in key]), + self.config, + episode_cache_size=self.cache_size, + cache_owner=self.cache_owner, + ) + return self.datasets[key] + + +class PolicySequenceDataset(EpisodeDecisions, Dataset): + """A supervised batch view over the common episode decision index.""" + + def __getitem__(self, index): + return imitation_example(self, index) + + +def imitation_example(dataset, index) -> SequifierBatch: + episode_index, step = dataset.locate(index) + episode = dataset.episodes[episode_index] + action_ids, returns = dataset._episode_labels(episode_index) + inputs = PolicyWindowView(dataset.config).episode_inputs( + episode, step, returns, action_ids + ) + from sequifier.io.actions import action_batch + + actions = action_batch([action_ids[step]], dataset.config) + metadata = dict(inputs.metadata) + metadata["target_valid_mask"] = torch.zeros_like(inputs.attention_valid_mask) + metadata["target_valid_mask"][0, -1] = True + return SequifierBatch( + inputs.inputs, + {name: value[:, None] for name, value in actions.items()}, + metadata, + ) diff --git a/src/sequifier/io/policy_materialization.py b/src/sequifier/io/policy_materialization.py new file mode 100644 index 000000000..0601c8e6e --- /dev/null +++ b/src/sequifier/io/policy_materialization.py @@ -0,0 +1,105 @@ +"""Materialize policy histories using the persisted model execution schema.""" + +import torch + +from sequifier.io.actions import decode_action +from sequifier.io.batch import SequifierInputs +from sequifier.model.execution_schema import ExecutionSchema + +_HISTORY_DTYPES = { + "bool": torch.bool, + "float32": torch.float32, + "int64": torch.int64, + "uint8": torch.uint8, +} + + +def encode_value(value, column, config, *, force_raw=False): + # Numeric/image payloads have no categorical codec. Avoid recursively + # walking every pixel in every frame of each reconstructed history. + if not config.id_maps.get(column): + return value + if isinstance(value, (list, tuple)): + return [ + encode_value(item, column, config, force_raw=force_raw) for item in value + ] + mapping = config.id_maps.get(column) + if mapping and (force_raw or config.input_encoding == "raw"): + if value in mapping: + return mapping[value] + if str(value) in mapping: + return mapping[str(value)] + from sequifier.special_tokens import SPECIAL_TOKEN_IDS + + special = { + label: number for number, label in SPECIAL_TOKEN_IDS.labels_by_id.items() + } + if value in special and special[value] < config.categorical_n_classes[column]: + return special[value] + unknown = SPECIAL_TOKEN_IDS.unknown + if unknown in mapping.values() or unknown < config.categorical_n_classes.get( + column, 0 + ): + return unknown + raise ValueError(f"{column}: value is outside the persisted categorical codec") + return value + + +def materialize_history(config, observations, previous_actions, static): + schema = ExecutionSchema.from_dict(config.input_schema) + rows = observations[-config.context_length :] + previous = previous_actions[-config.context_length :] + padding = config.context_length - len(rows) + features, metadata = {}, {} + for descriptor in schema.inputs: + key = descriptor.key + dtype = _HISTORY_DTYPES[descriptor.dtype] + if key == "attention_valid_mask": + metadata[key] = torch.tensor([[False] * padding + [True] * len(rows)]) + continue + if key in config.static_columns: + if key not in static: + raise ValueError(f"missing static model input {key!r}") + value = torch.as_tensor( + encode_value(static[key], key, config), dtype=dtype + ).unsqueeze(0) + else: + shape = (1, config.context_length, *descriptor.shape[2:]) + value = torch.zeros(shape, dtype=dtype) + for index, row in enumerate(rows): + if key in config.previous_action_inputs: + action = previous[index] + if action is None: + if key not in config.initial_inputs: + raise ValueError( + f"{key}: lagged action requires initial_inputs at episode start" + ) + raw = config.initial_inputs[key] + else: + raw = decode_action(action, config)[ + config.previous_action_inputs[key] + ] + else: + if key not in row: + raise ValueError(f"missing temporal model input {key!r}") + raw = row[key] + feedback = key in config.previous_action_inputs + encoded = torch.as_tensor( + encode_value(raw, key, config, force_raw=feedback), dtype=dtype + ) + if ( + feedback + and config.input_encoding == "model" + and key in config.input_statistics + ): + from sequifier.io.preprocessing import normalize_values + + encoded = normalize_values(encoded, config.input_statistics[key]) + if tuple(encoded.shape) != tuple(shape[2:]): + raise ValueError( + f"{key}: payload shape differs from the execution schema" + ) + value[0, padding + index] = encoded + (metadata if descriptor.role == "metadata" else features)[key] = value + schema.validate(features, metadata) + return SequifierInputs(features, metadata) diff --git a/src/sequifier/io/policy_window.py b/src/sequifier/io/policy_window.py new file mode 100644 index 000000000..6417affdf --- /dev/null +++ b/src/sequifier/io/policy_window.py @@ -0,0 +1,152 @@ +"""Policy alignment over the shared named-tensor and model-window contracts.""" + +from collections.abc import Mapping, Sequence + +from sequifier.config.policy_config import ResolvedPolicyBinding +from sequifier.config.policy_inputs import PolicyEpisodeSpec +from sequifier.contract_versions import ContractVersions +from sequifier.helpers import ModelWindowView, StoredWindowLayout, resolve_window_view +from sequifier.io.batch import SequifierInputs +from sequifier.io.preprocessing import normalize_values + +PREVIOUS_ACTION = "__policy_previous_action" +DESIRED_RETURN = "__policy_return" + + +def normalize_policy_features(features, metadata, interface): + """Apply persisted policy transforms identically in training and prediction.""" + features = dict(features) + mask = metadata["attention_valid_mask"] + if interface.policy.return_conditioned and interface.policy.input_encoding == "raw": + features[DESIRED_RETURN] = ( + features[DESIRED_RETURN] / interface.policy.returns.scale + ) + for column, statistics in interface.selected_columns_statistics.items(): + if column not in features or (not interface.normalize_real_columns): + continue + if interface.policy.input_encoding == "model": + continue + value = normalize_values(features[column], statistics) + if value.ndim >= 2 and value.shape[:2] == mask.shape: + valid = mask.reshape(*mask.shape, *([1] * (value.ndim - 2))) + value = value.masked_fill(~valid, 0.0) + features[column] = value + return features + + +def action_index(value, config: PolicyEpisodeSpec) -> int | dict[str, int | float]: + from sequifier.io.actions import encode_action + + return encode_action(value, config) + + +def encode_feature(value, column: str, config: PolicyEpisodeSpec): + from sequifier.io.policy_materialization import encode_value + + return encode_value(value, column, config) + + +class PolicyWindowView: + """At decision t, expose o_t and a_(t-1), and supervise a_t separately. + + Previous actions are decoder-local IDs at this boundary. They are converted + to the persisted input vocabulary here; the start token is input-only. + """ + + def __init__(self, config: ResolvedPolicyBinding): + self.config = config + self.view = ModelWindowView(config.context_length, "policy", 0) + self.resolved = resolve_window_view( + StoredWindowLayout( + config.context_length, 0, ContractVersions.STORED_WINDOW_LAYOUT + ), + self.view, + ) + + def episode_inputs( + self, episode, step: int, returns, action_ids=None + ) -> SequifierInputs: + """Expose one causal decision from a validated episode in either lifecycle.""" + if self.config.return_conditioned and returns is None: + from sequifier.io.episode_features import return_labels + + length = episode.return_length(self.config.returns.finite_horizon) + if length is None: + raise ValueError("conditioned likelihood requires a known reward tail") + returns = return_labels(episode, length, self.config.returns.gamma) + start = max(0, step + 1 - self.config.context_length) + initial = ( + (None) + if episode.starts_episode + else action_index(episode.previous_action, self.config) + ) + previous = [initial] if start == 0 else [] + if action_ids is None: + previous += [ + action_index(value, self.config) + for value in episode.actions[max(0, start - 1) : step] + ] + else: + previous += action_ids[max(0, start - 1) : step] + return self.inputs( + [episode.observation(position) for position in range(start, step + 1)], + previous, + returns[start : step + 1] if returns is not None else None, + episode.static_inputs, + ) + + def successor_inputs(self, episode, step: int, successor) -> SequifierInputs: + """Build h_(t+1) from the true successor and action a_t.""" + start = max(0, step + 2 - self.config.context_length) + observations = [ + episode.observation(position) for position in range(start, step + 1) + ] + [successor] + previous = [ + ( + (None) + if episode.starts_episode + else action_index(episode.previous_action, self.config) + ) + if position == 0 + else action_index(episode.actions[position - 1], self.config) + for position in range(start, step + 2) + ] + budgets = None + if self.config.return_conditioned: + from sequifier.io.episode_features import return_labels + from sequifier.trajectory.returns import remaining_return + + length = episode.return_length(self.config.returns.finite_horizon) + if length is None: + raise ValueError("conditioned successors require a known reward tail") + returns = return_labels(episode, length, self.config.returns.gamma) + following = remaining_return( + returns[step], + episode.transitions.rewards[step], + self.config.returns.gamma, + ) + budgets = returns[start : step + 1] + [following] + return self.inputs(observations, previous, budgets, episode.static_inputs) + + def inputs( + self, + observations: Sequence[Mapping], + previous_actions: Sequence[int | dict[str, int | float] | None], + desired_returns: Sequence[float] | None, + static_inputs: Mapping, + ) -> SequifierInputs: + config = self.config + from sequifier.io.policy_materialization import materialize_history + + if config.return_conditioned: + if desired_returns is None or len(desired_returns) != len(observations): + raise ValueError( + "return-conditioned histories require one return per observation" + ) + observations = [ + {**row, DESIRED_RETURN: value} + for row, value in zip(observations, desired_returns) + ] + return materialize_history( + config, observations, previous_actions, static_inputs + ) diff --git a/src/sequifier/io/preference_dataset.py b/src/sequifier/io/preference_dataset.py new file mode 100644 index 000000000..a13460575 --- /dev/null +++ b/src/sequifier/io/preference_dataset.py @@ -0,0 +1,240 @@ +"""Lazy preference storage and leakage-safe partition views.""" + +import hashlib +from dataclasses import dataclass + +import torch + +from sequifier.fingerprints import fingerprint +from sequifier.io.episode_corpus import EpisodeView, episode_content_key +from sequifier.io.episode_features import EpisodeFeatures +from sequifier.io.partitioning import split_components +from sequifier.io.policy_window import PolicyWindowView, action_index + + +@dataclass(frozen=True) +class PreferenceData: + training: "PreferenceDataset" + validation: "PreferenceDataset" + + @property + def contract(self): + return [self.training.contract, self.validation.contract] + + def fitting_dataset(self): + return self.training.fitting_dataset() + + def validate(self, actor): + for partition in (self.training, self.validation): + partition.validate_features() + fitting = partition.fitting_dataset() + fitting.validate_features() + fitting.validate_normalization(actor.interface.selected_columns_statistics) + + +class PreferenceDataset: + def __init__(self, pairs, corpora, config): + self.pairs, self.corpora, self.config = tuple(pairs), corpora, config + self.lookup = { + name: {episode: i for i, episode in enumerate(data.episode_ids)} + for name, data in corpora.items() + } + self._fitting = None + self._features_validated = False + self.contract = [] + for pair in self.pairs: + entries = [] + for span in (pair.chosen, pair.rejected): + episode = self.episode(span) + if span.end > len(episode.actions): + raise ValueError( + f"pair {pair.pair_id}: span exceeds episode {span.episode_id}" + ) + entries.append( + {"span": span.model_dump(), "content": episode_content_key(episode)} + ) + if self.alternative_key(pair.chosen) == self.alternative_key(pair.rejected): + raise ValueError( + f"pair {pair.pair_id}: chosen/rejected alternatives have identical decision content" + ) + if ( + pair.chosen.source == pair.rejected.source + and pair.chosen.episode_id == pair.rejected.episode_id + and max(pair.chosen.start, pair.rejected.start) + < min(pair.chosen.end, pair.rejected.end) + ): + raise ValueError(f"pair {pair.pair_id}: chosen/rejected spans overlap") + self.contract.append({"pair": pair.model_dump(), "alternatives": entries}) + + def __len__(self): + return len(self.pairs) + + def episode(self, span): + try: + return self.corpora[span.source].episodes[ + self.lookup[span.source][span.episode_id] + ] + except KeyError as exc: + raise ValueError( + f"preference references unknown source/episode {span.source}:{span.episode_id}" + ) from exc + + def alternative_key(self, span): + episode = self.episode(span) + start = max(0, span.start + 1 - self.config.context_length) + return fingerprint( + { + "observations": { + k: v[start : span.end] for k, v in episode.observations.items() + }, + "actions": episode.actions[max(0, start - 1) : span.end], + "static": episode.static_inputs, + "starts_episode": episode.starts_episode, + "previous_action": episode.previous_action, + } + ) + + def components(self, pair): + keys = [("context", pair.context_id)] + if pair.group_id is not None: + keys.append(("group", pair.group_id)) + for span in (pair.chosen, pair.rejected): + episode = self.episode(span) + keys += [ + ("episode", span.episode_id), + ("content", episode_content_key(episode)), + ("alternative", self.alternative_key(span)), + ] + if episode.group_id is not None: + keys.append(("group", episode.group_id)) + return keys + + def fitting_dataset(self): + if self._fitting is None: + entries = {} + for pair in self.pairs: + for span in (pair.chosen, pair.rejected): + index = self.lookup[span.source][span.episode_id] + entries[(span.source, span.episode_id)] = self.corpora[ + span.source + ].episodes.entries[index] + self._fitting = EpisodeFeatures( + EpisodeView(entries.values()), + self.config, + ) + return self._fitting + + def validate_features(self): + if self._features_validated: + return + config = self.config + view = PolicyWindowView(config) + contexts = {} + for pair in self.pairs: + initial = [] + alternatives = [] + for span in (pair.chosen, pair.rejected): + episode = self.episode(span) + digest = hashlib.sha256() + # Validate every selected decision and all its causal inputs before training. + for step in range(span.start, span.end): + inputs = view.episode_inputs(episode, step, None) + action = action_index(episode.actions[step], config) + digest.update( + fingerprint( + { + "features": { + k: v.tolist() for k, v in inputs.inputs.items() + }, + "masks": { + k: v.tolist() for k, v in inputs.metadata.items() + }, + "action": action, + } + ).encode() + ) + if step == span.start: + initial.append(inputs) + alternatives.append(digest.hexdigest()) + if alternatives[0] == alternatives[1]: + raise ValueError( + f"pair {pair.pair_id}: alternatives have identical canonical executable decisions" + ) + context_key = fingerprint( + { + "features": {k: v.tolist() for k, v in initial[0].inputs.items()}, + "masks": {k: v.tolist() for k, v in initial[0].metadata.items()}, + } + ) + if pair.context_id in contexts and contexts[pair.context_id] != context_key: + raise ValueError( + f"context {pair.context_id}: ID refers to different canonical initial histories" + ) + contexts[pair.context_id] = context_key + left, right = initial + for mapping, lmap, rmap in ( + ("inputs", left.inputs, right.inputs), + ("metadata", left.metadata, right.metadata), + ): + if lmap.keys() != rmap.keys() or any( + not torch.equal(lmap[k], rmap[k]) for k in lmap + ): + raise ValueError( + f"pair {pair.pair_id}: initial {mapping} differ under frozen codec" + ) + + self._features_validated = True + + +def prepare_preferences(run, phase, catalog, cache, membership): + from sequifier.rl.offline_data import phase_validation_fraction + + key = ( + "preferences", + phase.preferences.source, + tuple(ref.source for ref in (phase.evaluation.sources or [])), + ) + if key in cache: + return cache[key] + config = catalog.config + pairs = catalog.preference_pairs(run.preferences[phase.preferences.source]) + validation = ( + [ + pair + for ref in phase.evaluation.sources + for pair in catalog.preference_pairs(run.preferences[ref.source]) + ] + if phase.evaluation.sources + else None + ) + all_pairs = pairs + (validation or []) + ids = [pair.pair_id for pair in all_pairs] + if len(ids) != len(set(ids)): + raise ValueError("preference pair IDs overlap across manifests") + names = {span.source for pair in all_pairs for span in (pair.chosen, pair.rejected)} + if names - run.sources.keys(): + raise ValueError( + f"preferences reference unknown sources: {sorted(names - run.sources.keys())}" + ) + corpora = {name: catalog.dataset([name]) for name in sorted(names)} + if key not in membership: + data = PreferenceDataset(all_pairs, corpora, config) + if validation is not None: + left = {key for pair in pairs for key in data.components(pair)} + right = {key for pair in validation for key in data.components(pair)} + if left & right: + raise ValueError( + f"preference train/validation components overlap: {next(iter(left & right))}" + ) + else: + pairs, validation = split_components( + pairs, data.components, phase_validation_fraction(run, phase), run.seed + ) + membership[key] = tuple(pairs), tuple(validation) + pairs, validation = membership[key] + result = PreferenceData( + PreferenceDataset(pairs, corpora, config), + PreferenceDataset(validation, corpora, config), + ) + cache[key] = result + return result diff --git a/src/sequifier/io/prepared_policy_data.py b/src/sequifier/io/prepared_policy_data.py new file mode 100644 index 000000000..c7609bce5 --- /dev/null +++ b/src/sequifier/io/prepared_policy_data.py @@ -0,0 +1,21 @@ +"""The small data contract shared by policy phase preparation.""" + +from collections.abc import Sized +from typing import TYPE_CHECKING, Protocol + +from sequifier.io.episode_features import EpisodeFeatures + +if TYPE_CHECKING: + from sequifier.rl.policy import Policy + + +class PreparedPolicyData(Protocol): + @property + def training(self) -> Sized: ... + + @property + def contract(self) -> list: ... + + def fitting_dataset(self) -> EpisodeFeatures: ... + + def validate(self, actor: "Policy") -> None: ... diff --git a/src/sequifier/io/preprocessing.py b/src/sequifier/io/preprocessing.py new file mode 100644 index 000000000..7fa5c0c8b --- /dev/null +++ b/src/sequifier/io/preprocessing.py @@ -0,0 +1,155 @@ +"""Shared categorical vocabulary and stable sample-statistics fitting.""" + +import math +import warnings +from collections.abc import Mapping +from typing import Union + +import numpy as np +import polars as pl + +from sequifier.special_tokens import SPECIAL_TOKEN_IDS +from sequifier.typechecking import beartype + + +@beartype +def _finite_mean_and_std(column: pl.Series, column_name: str) -> tuple[float, float]: + """Compute sample statistics without overflowing intermediate squares.""" + values = column.to_numpy().astype(np.float64, copy=False) + if values.size == 0: + raise ValueError(f"Cannot compute statistics for empty column {column_name!r}.") + if not np.isfinite(values).all(): + raise ValueError(f"Column {column_name!r} contains non-finite values.") + + scale = float(np.max(np.abs(values))) + if scale == 0.0: + return 0.0, 0.0 + + scaled_values = values / scale + mean = float(np.mean(scaled_values) * scale) + std = 0.0 if values.size <= 1 else float(np.std(scaled_values, ddof=1) * scale) + if not math.isfinite(mean) or not math.isfinite(std): + raise ValueError( + f"Column {column_name!r} has a numeric range too large for finite " + "normalization statistics." + ) + return mean, std + + +@beartype +def get_combined_statistics( + n1: int, mean1: float, std1: float, n2: int, mean2: float, std2: float +) -> tuple[float, float]: + """Combine two mean/std summaries.""" + if n1 == 0: + return mean2, std2 + if n2 == 0: + return mean1, std1 + + scale = max(abs(mean1), abs(mean2), abs(std1), abs(std2)) + if not math.isfinite(scale): + raise ValueError("Cannot combine non-finite normalization statistics.") + if scale == 0.0: + return 0.0, 0.0 + + scaled_mean1 = mean1 / scale + scaled_mean2 = mean2 / scale + scaled_std1 = std1 / scale + scaled_std2 = std2 / scale + combined_mean_scaled = (n1 * scaled_mean1 + n2 * scaled_mean2) / (n1 + n2) + combined_mean = combined_mean_scaled * scale + + if n1 + n2 <= 1: + return combined_mean, 0.0 + + sum_of_squares1 = (n1 - 1) * scaled_std1**2 + n1 * ( + scaled_mean1 - combined_mean_scaled + ) ** 2 + sum_of_squares2 = (n2 - 1) * scaled_std2**2 + n2 * ( + scaled_mean2 - combined_mean_scaled + ) ** 2 + + combined_std = ( + math.sqrt((sum_of_squares1 + sum_of_squares2) / (n1 + n2 - 1)) * scale + ) + if not math.isfinite(combined_mean) or not math.isfinite(combined_std): + raise ValueError( + "Combined numeric range is too large for finite normalization statistics." + ) + + return combined_mean, combined_std + + +@beartype +def create_id_map(data: pl.DataFrame, column: str) -> dict[Union[str, int], int]: + """Map sorted user values to IDs after reserved tokens.""" + ids = sorted( + [int(x) if not isinstance(x, str) else x for x in np.unique(data[column])] + ) # type: ignore + + if isinstance(ids[0], str): + if SPECIAL_TOKEN_IDS.labels_by_id[SPECIAL_TOKEN_IDS.mask] in ids: + raise ValueError( + f"Found value '{SPECIAL_TOKEN_IDS.labels_by_id[SPECIAL_TOKEN_IDS.mask]}' in {column}, this is invalid" + ) + + for special_val in [ + SPECIAL_TOKEN_IDS.labels_by_id[SPECIAL_TOKEN_IDS.unknown], + SPECIAL_TOKEN_IDS.labels_by_id[SPECIAL_TOKEN_IDS.other], + ]: + if special_val in ids: + warnings.warn( + f"Found special value {special_val} in {column}, these will be combined with the sequifier-internal special value {special_val}" + ) + ids = [ + id_ + for id_ in ids + if id_ + not in [ + SPECIAL_TOKEN_IDS.labels_by_id[SPECIAL_TOKEN_IDS.unknown], + SPECIAL_TOKEN_IDS.labels_by_id[SPECIAL_TOKEN_IDS.other], + ] + ] + id_map = {id_: i + SPECIAL_TOKEN_IDS.user_start for i, id_ in enumerate(ids)} + id_map[SPECIAL_TOKEN_IDS.labels_by_id[SPECIAL_TOKEN_IDS.unknown]] = ( + SPECIAL_TOKEN_IDS.unknown + ) + id_map[SPECIAL_TOKEN_IDS.labels_by_id[SPECIAL_TOKEN_IDS.other]] = ( + SPECIAL_TOKEN_IDS.other + ) + else: + id_map = {id_: i + SPECIAL_TOKEN_IDS.user_start for i, id_ in enumerate(ids)} + return dict(id_map) + + +NORMALIZATION_EPSILON = 1e-9 + + +def normalization_statistics(statistics: Mapping) -> dict[str, dict[str, float]]: + """Validate persisted statistics once, independently of the array backend.""" + result = {} + for column, values in statistics.items(): + mean, std = values.get("mean"), values.get("std") + if ( + isinstance(mean, bool) + or isinstance(std, bool) + or not isinstance(mean, (int, float)) + or not isinstance(std, (int, float)) + or not math.isfinite(mean) + or not math.isfinite(std) + or std < 0 + ): + raise ValueError( + f"{column}: normalization requires finite means and nonnegative scales" + ) + result[column] = {"mean": float(mean), "std": float(std)} + return result + + +def normalize_values(values, statistics): + """Standardize scalar, NumPy, Torch, or Polars values with fixed statistics.""" + return (values - statistics["mean"]) / (statistics["std"] + NORMALIZATION_EPSILON) + + +def denormalize_values(values, statistics): + return values * (statistics["std"] + NORMALIZATION_EPSILON) + statistics["mean"] diff --git a/src/sequifier/io/pt_payload.py b/src/sequifier/io/pt_payload.py index ae35139f3..9aa1299da 100644 --- a/src/sequifier/io/pt_payload.py +++ b/src/sequifier/io/pt_payload.py @@ -6,6 +6,7 @@ from torch import Tensor from sequifier.config.depth_layout import DepthLayoutRegistryModel +from sequifier.contract_versions import ContractVersions, require_contract_version def validate_tensor_inputs( @@ -23,7 +24,7 @@ def validate_tensor_inputs( raise ValueError("Features require batch and temporal axes") for column, values in sequences.items(): layout = layouts.layout_for_column(column) - expected = (*shape, layout.context_length) if layout else shape + expected = (*shape, layout.max_depth_length) if layout else shape if tuple(values.shape) != tuple(expected): raise ValueError( f"{column}: expected {tuple(expected)}, got {tuple(values.shape)}" @@ -56,10 +57,10 @@ def validate_tensor_inputs( mask = masks[name] if mask.dtype != torch.bool or tuple(mask.shape) != ( *shape, - layout.context_length, + layout.max_depth_length, ): raise ValueError( - f"{name}: depth mask must be bool [batch, time, {layout.context_length}]" + f"{name}: depth mask must be bool [batch, time, {layout.max_depth_length}]" ) if not layout.allow_gaps and (mask[..., 1:] & ~mask[..., :-1]).any().item(): raise ValueError( @@ -87,6 +88,31 @@ class StoredTensorBatch: sample_positions: Tensor | None = None curriculum_columns: tuple[str, ...] = () + def slice_rows(self, start: int, stop: int) -> "StoredTensorBatch": + # Own just the selected storage so saving a shard cannot serialize its parent. + rows = slice(start, stop) + return StoredTensorBatch( + sequences={ + name: value[rows].clone() for name, value in self.sequences.items() + }, + depth_valid_masks={ + name: value[rows].clone() + for name, value in self.depth_valid_masks.items() + }, + sample_positions=( + self.sample_positions[rows].clone() + if self.sample_positions is not None + else None + ), + curriculum_columns=self.curriculum_columns, + sequence_ids=self.sequence_ids[rows].clone(), + subsequence_ids=self.subsequence_ids[rows].clone(), + start_item_positions=self.start_item_positions[rows].clone(), + left_pad_lengths=self.left_pad_lengths[rows].clone(), + split_start_item_positions=self.split_start_item_positions[rows].clone(), + split_end_item_positions=self.split_end_item_positions[rows].clone(), + ) + def validate(self, layouts=None, *, n_classes=None): layouts = DepthLayoutRegistryModel.model_validate(layouts or {}) missing = set(layouts.deep_columns) - set(self.sequences) @@ -100,21 +126,14 @@ def validate(self, layouts=None, *, n_classes=None): if first.ndim < 2: raise ValueError("Stored tensors require batch and temporal axes") n, width = first.shape[:2] - for key in ( - "sequence_ids", - "subsequence_ids", - "start_item_positions", - "left_pad_lengths", + for key, value in ( + ("sequence_ids", self.sequence_ids), + ("subsequence_ids", self.subsequence_ids), + ("start_item_positions", self.start_item_positions), + ("left_pad_lengths", self.left_pad_lengths), + ("split_start_item_positions", self.split_start_item_positions), + ("split_end_item_positions", self.split_end_item_positions), ): - value = getattr(self, key) - if ( - not isinstance(value, Tensor) - or value.dtype != torch.int64 - or tuple(value.shape) != (n,) - ): - raise ValueError(f"{key} must be int64 [{n}]") - for key in ("split_start_item_positions", "split_end_item_positions"): - value = getattr(self, key) if ( not isinstance(value, Tensor) or value.dtype != torch.int64 @@ -170,38 +189,68 @@ def validate(self, layouts=None, *, n_classes=None): return self +def tabular_tensor_batch( + data, column_data_types, columns, window_length +) -> StoredTensorBatch: + """Read the legacy tabular representation into the standard stored batch.""" + import polars as pl + + from sequifier.helpers import numpy_storage_to_pytorch + + bounds = ("startItemPosition", "splitStartItemPosition", "splitEndItemPosition") + if not set(bounds) <= set(data.columns): + raise ValueError( + "prepared windows require explicit start positions and split bounds" + ) + sequences, pads = numpy_storage_to_pytorch( + data, column_data_types, columns, window_length + ) + positions = ( + data.group_by(["sequenceId", "subsequenceId"]) + .agg(*(pl.col(name).first() for name in bounds)) + .sort(["sequenceId", "subsequenceId"]) + ) + ids, subsequences, starts, lows, highs = ( + torch.tensor(positions[name].to_numpy(), dtype=torch.int64) + for name in ("sequenceId", "subsequenceId", *bounds) + ) + return StoredTensorBatch(sequences, ids, subsequences, starts, pads, lows, highs) + + def load_pt_payload(path, *, layouts=None, n_classes=None) -> StoredTensorBatch: value = torch.load(path, map_location="cpu", weights_only=True) - if isinstance(value, dict): - version = value.get("version") - if value.get("format") != "sequifier_tensor_batch" or version != 5: - raise ValueError(f"Unsupported PT tensor payload format/version in {path}") - expected = { - "format", - "version", - "sequences", - "metadata", - "sequence_ids", - "subsequence_ids", - "start_item_positions", - "left_pad_lengths", - } - expected.update({"split_start_item_positions", "split_end_item_positions"}) - expected_metadata = { - "depth_valid_masks", - "sample_positions", - "curriculum_columns", - } - if set(value) != expected or set(value["metadata"]) != expected_metadata: - raise ValueError(f"Inconsistent version {version} tensor payload schema") - batch = StoredTensorBatch( - **{key: value[key] for key in expected - {"format", "version", "metadata"}}, - depth_valid_masks=value["metadata"]["depth_valid_masks"], - sample_positions=value["metadata"].get("sample_positions"), - curriculum_columns=tuple(value["metadata"].get("curriculum_columns", ())), - ) - else: - raise ValueError(f"Unsupported PT tensor payload in {path}") + version = value.get("version") if isinstance(value, dict) else None + require_contract_version( + "PT tensor-batch envelope", version, ContractVersions.PT_TENSOR_BATCH + ) + if value.get("format") != "sequifier_tensor_batch": + raise ValueError(f"Unsupported PT tensor payload format in {path}") + expected = { + "format", + "version", + "sequences", + "metadata", + "sequence_ids", + "subsequence_ids", + "start_item_positions", + "left_pad_lengths", + "split_start_item_positions", + "split_end_item_positions", + } + expected_metadata = {"depth_valid_masks", "sample_positions", "curriculum_columns"} + metadata = value.get("metadata") + if ( + set(value) != expected + or not isinstance(metadata, dict) + or set(metadata) != expected_metadata + ): + raise ValueError("Inconsistent tensor payload schema") + batch = StoredTensorBatch( + **{key: value[key] for key in expected - {"format", "version", "metadata"}}, + depth_valid_masks=metadata["depth_valid_masks"], + sample_positions=metadata["sample_positions"], + curriculum_columns=tuple(metadata["curriculum_columns"]), + ) return batch.validate(layouts, n_classes=n_classes) @@ -210,24 +259,19 @@ def save_pt_payload(batch: StoredTensorBatch, path, *, layouts=None, n_classes=N torch.save( { "format": "sequifier_tensor_batch", - "version": 5, + "version": ContractVersions.PT_TENSOR_BATCH, "sequences": batch.sequences, "metadata": { "depth_valid_masks": batch.depth_valid_masks, "sample_positions": batch.sample_positions, "curriculum_columns": list(batch.curriculum_columns), }, - **{ - key: getattr(batch, key) - for key in ( - "sequence_ids", - "subsequence_ids", - "start_item_positions", - "left_pad_lengths", - "split_start_item_positions", - "split_end_item_positions", - ) - }, + "sequence_ids": batch.sequence_ids, + "subsequence_ids": batch.subsequence_ids, + "start_item_positions": batch.start_item_positions, + "left_pad_lengths": batch.left_pad_lengths, + "split_start_item_positions": batch.split_start_item_positions, + "split_end_item_positions": batch.split_end_item_positions, }, path, ) @@ -267,13 +311,10 @@ def concatenate_pt_batches(batches: list[StoredTensorBatch]) -> StoredTensorBatc split_end_item_positions=torch.cat( [b.split_end_item_positions for b in batches] ), - **{ - key: torch.cat([getattr(b, key) for b in batches]) - for key in ( - "sequence_ids", - "subsequence_ids", - "start_item_positions", - "left_pad_lengths", - ) - }, + sequence_ids=torch.cat([batch.sequence_ids for batch in batches]), + subsequence_ids=torch.cat([batch.subsequence_ids for batch in batches]), + start_item_positions=torch.cat( + [batch.start_item_positions for batch in batches] + ), + left_pad_lengths=torch.cat([batch.left_pad_lengths for batch in batches]), ) diff --git a/src/sequifier/io/sample_order.py b/src/sequifier/io/sample_order.py index 169b6f85d..e21b1d848 100644 --- a/src/sequifier/io/sample_order.py +++ b/src/sequifier/io/sample_order.py @@ -1,6 +1,7 @@ """Deterministic sample ordering for ordinary and curriculum datasets.""" from dataclasses import dataclass +from typing import Any import polars as pl import torch @@ -127,26 +128,35 @@ def indices_for_epoch(self, *, seed: int, epoch: int, shuffle: bool) -> Tensor: return indices -def configured_file_order(config: object) -> str: - """Read the validated per-part file-order policy, tolerating test doubles.""" - value = getattr(getattr(config, "part", None), "file_order", "shuffled") +def configured_file_order(config: Any) -> str: + """Read the optional per-part file-order policy.""" + try: + value = config.part.file_order + except AttributeError: + return "shuffled" return value if value in {"shuffled", "name"} else "shuffled" -def curriculum_training_enabled(config: object) -> bool: +def curriculum_training_enabled(config: Any) -> bool: """Return the validated training-plan curriculum switch.""" - value = getattr(config, "curriculum_training", False) + try: + value = config.curriculum_training + except AttributeError: + return False return value if isinstance(value, bool) else False -def configured_curriculum_column(config: object) -> str | None: +def configured_curriculum_column(config: Any) -> str | None: """Read the validated training-plan curriculum column.""" - value = getattr(config, "curriculum_column", None) + try: + value = config.curriculum_column + except AttributeError: + return None return value if isinstance(value, str) else None def curriculum_sample_positions( - config: object, + config: Any, sample_positions: Tensor | None, source: str, stored_columns: tuple[str, ...] = (), @@ -172,7 +182,7 @@ def curriculum_sample_positions( def curriculum_positions_from_parquet( - config: object, data: pl.DataFrame, source: str + config: Any, data: pl.DataFrame, source: str ) -> Tensor | None: """Read curriculum values only when curriculum training is enabled.""" if not curriculum_training_enabled(config): @@ -183,7 +193,7 @@ def curriculum_positions_from_parquet( ) -def validate_folder_curriculum(config: object, metadata: dict, source: str) -> None: +def validate_folder_curriculum(config: Any, metadata: dict, source: str) -> None: """Fail before iteration when a folder lacks curriculum-capable metadata.""" if not curriculum_training_enabled(config): return diff --git a/src/sequifier/io/sequence.py b/src/sequifier/io/sequence.py new file mode 100644 index 000000000..84f0dbdfb --- /dev/null +++ b/src/sequifier/io/sequence.py @@ -0,0 +1,141 @@ +"""Columnar demonstrations with optional transition capabilities. + +These records describe data, independently of the simulator, collection context +length, prediction bindings, preprocessing or the algorithm that will consume it. +""" + +from pydantic import ( + BaseModel, + ConfigDict, + Field, + JsonValue, + StrictBool, + StrictInt, + model_validator, +) + +from sequifier.io.values import DomainValue + + +def ordered_sequence_rows(frame, position_column="itemPosition"): + """Share explicit temporal ordering and gap validation across table consumers.""" + if position_column not in frame.columns: + return frame + positions = frame[position_column] + if not positions.dtype.is_integer() or positions.null_count(): + raise ValueError(f"{position_column} must contain non-null integers") + frame = frame.sort(position_column) + values = frame[position_column].to_list() + if values and ( + values[0] < 0 + or any(right != left + 1 for left, right in zip(values, values[1:])) + ): + raise ValueError( + f"{position_column} must be nonnegative, unique and contiguous within a sequence" + ) + return frame + + +class TransitionColumns(BaseModel): + model_config = ConfigDict(extra="forbid", frozen=True, allow_inf_nan=False) + + rewards: tuple[float, ...] | None = None + terminated: tuple[StrictBool, ...] | None = None + truncated: tuple[StrictBool, ...] | None = None + # None at a position means the true successor was not observed. Never a reset. + successors: tuple[dict[str, JsonValue] | None, ...] | None = None + bootstrap_allowed: tuple[StrictBool, ...] | None = None + + def at(self, index: int) -> dict: + """Read a transition without inventing absent capabilities.""" + return { + name: values[index] if values is not None else None for name, values in self + } + + +class SequenceEpisode(BaseModel): + model_config = ConfigDict(extra="forbid", frozen=True, allow_inf_nan=False) + + episode_id: str = Field(min_length=1) + item_positions: tuple[StrictInt, ...] = Field(min_length=1) + observations: dict[str, tuple[JsonValue, ...]] = Field(min_length=1) + actions: tuple[DomainValue | dict[str, DomainValue], ...] + static_inputs: dict[str, JsonValue] = Field(default_factory=dict) + starts_episode: StrictBool = True + previous_action: DomainValue | dict[str, DomainValue] | None = None + group_id: str | None = None + transitions: TransitionColumns | None = None + + @model_validator(mode="after") + def validate_sequence(self): + length = len(self.item_positions) + if self.item_positions[0] < 0 or any( + right != left + 1 + for left, right in zip(self.item_positions, self.item_positions[1:]) + ): + raise ValueError( + "itemPosition must be nonnegative, unique and contiguous within an episode" + ) + if len(self.actions) != length or any( + len(values) != length for values in self.observations.values() + ): + raise ValueError( + "sequence feature/action lengths must match item positions" + ) + if set(self.observations) & set(self.static_inputs): + raise ValueError("static and temporal features must be disjoint") + if self.starts_episode: + if self.item_positions[0] != 0 or self.previous_action is not None: + raise ValueError( + "episode starts require position zero and no preceding action" + ) + elif self.previous_action is None: + raise ValueError("fragments require an explicit preceding action") + transitions = self.transitions + if transitions is None: + return self + for field, values in transitions: + if values is not None and len(values) != length: + raise ValueError(f"transition {field} must match sequence length") + for index in range(length): + step = transitions.at(index) + terminated, truncated = step["terminated"], step["truncated"] + if (terminated or truncated) and index != length - 1: + raise ValueError( + "a sequence cannot contain a decision after an episode boundary" + ) + successor = step["successors"] + if successor is not None: + if successor.keys() != self.observations.keys(): + raise ValueError( + "successor observation fields must match observations" + ) + if index + 1 < length and successor != self.observation(index + 1): + raise ValueError("explicit successor breaks sequence continuity") + bootstrap = step["bootstrap_allowed"] + if terminated and bootstrap: + raise ValueError("task termination must disable bootstrap") + if bootstrap and index + 1 == length and successor is None: + raise ValueError( + "bootstrapped final decisions require an observed successor" + ) + return self + + def observation(self, index: int) -> dict: + return {column: values[index] for column, values in self.observations.items()} + + def return_length(self, finite_horizon: int | None) -> int | None: + if self.transitions is None or self.transitions.rewards is None: + return None + length = len(self.actions) + if finite_horizon is not None and ( + self.starts_episode or self.item_positions[0] > 0 + ): + remaining = finite_horizon - self.item_positions[0] + if remaining <= 0: + return None + if length >= remaining: + return remaining + if self.transitions.terminated is not None and self.transitions.terminated[-1]: + return length + return None diff --git a/src/sequifier/io/sequifier_dataset_from_file.py b/src/sequifier/io/sequifier_dataset_from_file.py index 9aba96fa9..6c7e85751 100644 --- a/src/sequifier/io/sequifier_dataset_from_file.py +++ b/src/sequifier/io/sequifier_dataset_from_file.py @@ -2,7 +2,6 @@ from collections.abc import Iterator from typing import Any -import polars as pl import torch import torch.distributed as dist from loguru import logger @@ -11,12 +10,10 @@ from sequifier.helpers import ( PANDAS_TO_TORCH_TYPES, configured_window_stride, - numpy_storage_to_pytorch, read_data, resolve_window_sampling_plan, ) from sequifier.io.batch import SequifierBatch -from sequifier.io.config import global_training from sequifier.io.images import ImageStore from sequifier.io.iteration_state import ( read_shared_int, @@ -47,7 +44,7 @@ class SequifierDatasetFromFile(IterableDataset): def __init__(self, data_path: str, config: Any, shuffle: bool = True): super().__init__() self.config = config - self.batch_size = global_training(config).batch_size + self.batch_size = config.batch_size self.shuffle = shuffle self._epoch_state = shared_int(0) self._start_batch_state = shared_int(0) @@ -92,7 +89,10 @@ def __init__(self, data_path: str, config: Any, shuffle: bool = True): if config.read_format == "pt": from sequifier.io.pt_payload import load_pt_payload - part = getattr(config, "part", None) + try: + part = config.part + except AttributeError: + part = None metadata = part.metadata if part is not None else None payload = load_pt_payload( data_path, @@ -117,38 +117,19 @@ def __init__(self, data_path: str, config: Any, shuffle: bool = True): self.split_end_item_positions = payload.split_end_item_positions else: assert data_df is not None - all_tensors, left_pad_lengths = numpy_storage_to_pytorch( - data=data_df, - column_data_types=column_data_types, - all_columns=all_columns, - window_length=config.storage_layout.window_length, + from sequifier.io.pt_payload import tabular_tensor_batch + + payload = tabular_tensor_batch( + data_df, + column_data_types, + all_columns, + config.storage_layout.window_length, ) - if { - "startItemPosition", - "splitStartItemPosition", - "splitEndItemPosition", - } <= set(data_df.columns): - positions = ( - data_df.group_by(["sequenceId", "subsequenceId"]) - .agg( - pl.col("startItemPosition").first(), - pl.col("splitStartItemPosition").first(), - pl.col("splitEndItemPosition").first(), - ) - .sort(["sequenceId", "subsequenceId"]) - ) - self.start_item_positions = torch.tensor( - positions["startItemPosition"].to_numpy(), dtype=torch.int64 - ) - self.sequence_ids = torch.tensor( - positions["sequenceId"].to_numpy(), dtype=torch.int64 - ) - self.split_start_item_positions = torch.tensor( - positions["splitStartItemPosition"].to_numpy(), dtype=torch.int64 - ) - self.split_end_item_positions = torch.tensor( - positions["splitEndItemPosition"].to_numpy(), dtype=torch.int64 - ) + all_tensors, left_pad_lengths = payload.sequences, payload.left_pad_lengths + self.start_item_positions = payload.start_item_positions + self.sequence_ids = payload.sequence_ids + self.split_start_item_positions = payload.split_start_item_positions + self.split_end_item_positions = payload.split_end_item_positions validate_split_bounds_available( self.start_item_positions, self.split_start_item_positions, @@ -205,7 +186,7 @@ def set_start_batch(self, start_batch: int): @beartype def __len__(self) -> int: - num_workers = max(1, global_training(self.config).num_workers) + num_workers = max(1, self.config.runtime_settings.num_workers) total_batches = 0 for worker_id in range(num_workers): worker_samples = self.n_samples // num_workers + ( diff --git a/src/sequifier/io/sequifier_dataset_from_folder_pt.py b/src/sequifier/io/sequifier_dataset_from_folder_pt.py index 9d9f717df..c051c5978 100644 --- a/src/sequifier/io/sequifier_dataset_from_folder_pt.py +++ b/src/sequifier/io/sequifier_dataset_from_folder_pt.py @@ -35,9 +35,7 @@ def __init__(self, data_path: str, config: Any, shuffle: bool = True): self.depth_layouts = DepthLayoutRegistryModel.model_validate( metadata.get("depth_layouts", {}) ) - selected_layouts: DepthLayoutRegistryModel | None = getattr( - config, "depth_layouts", None - ) + selected_layouts: DepthLayoutRegistryModel | None = config.depth_layouts if selected_layouts is None: selected_layouts = DepthLayoutRegistryModel() if self.depth_layouts.compatibility_signature( diff --git a/src/sequifier/io/sequifier_dataset_from_folder_pt_lazy.py b/src/sequifier/io/sequifier_dataset_from_folder_pt_lazy.py index eb5fa7282..cadb557bf 100644 --- a/src/sequifier/io/sequifier_dataset_from_folder_pt_lazy.py +++ b/src/sequifier/io/sequifier_dataset_from_folder_pt_lazy.py @@ -33,9 +33,7 @@ def __init__(self, data_path: str, config: Any, shuffle: bool = True): self.depth_layouts = DepthLayoutRegistryModel.model_validate( metadata.get("depth_layouts", {}) ) - selected_layouts: DepthLayoutRegistryModel | None = getattr( - config, "depth_layouts", None - ) + selected_layouts: DepthLayoutRegistryModel | None = config.depth_layouts if selected_layouts is None: selected_layouts = DepthLayoutRegistryModel() if self.depth_layouts.compatibility_signature( diff --git a/src/sequifier/io/static_features.py b/src/sequifier/io/static_features.py index 191655b65..3c0940009 100644 --- a/src/sequifier/io/static_features.py +++ b/src/sequifier/io/static_features.py @@ -54,10 +54,16 @@ def _load( def table_for_config(config: Any) -> StaticFeatureTable | None: - columns: tuple[str, ...] = tuple(getattr(config, "static_columns", ())) + try: + columns = tuple(config.static_columns) + except AttributeError: + return None if not columns: return None - path = getattr(config, "static_features_path", None) + try: + path = config.static_features_path + except AttributeError: + path = None if path is None: raise ValueError("Static conditioning requires static_features_path") dtypes = tuple(config.column_data_types[column] for column in columns) diff --git a/src/sequifier/io/tensors.py b/src/sequifier/io/tensors.py new file mode 100644 index 000000000..42832c126 --- /dev/null +++ b/src/sequifier/io/tensors.py @@ -0,0 +1,75 @@ +"""Device transfer and collation for records of named or optional tensors.""" + +from dataclasses import dataclass, fields +from typing import TypeVar, cast + +import torch + +Record = TypeVar("Record", bound="TensorRecord") + + +def _map_tensors(value, operation): + if value is None: + return None + if isinstance(value, dict): + return {key: _map_tensors(item, operation) for key, item in value.items()} + if isinstance(value, TensorRecord): + return type(value)( + **{ + field.name: _map_tensors(vars(value)[field.name], operation) + for field in fields(value) + } + ) + return operation(value) + + +@dataclass(frozen=True) +class TensorRecord: + """Mixin for dataclasses containing tensors, dictionaries and optional fields.""" + + def to(self: Record, device: str | torch.device) -> Record: + target = torch.device(device) + return cast( + Record, + _map_tensors( + self, + lambda value: value.to( + target, + dtype=torch.float32 + if target.type == "mps" and value.dtype == torch.float64 + else value.dtype, + non_blocking=True, + ), + ), + ) + + def pin_memory(self: Record) -> Record: + return cast(Record, _map_tensors(self, lambda value: value.pin_memory())) + + +def collate_tensor_records(values): + """Concatenate corresponding tensors, rejecting inconsistent record schemas.""" + if not values: + raise ValueError("cannot collate empty tensor records") + first = values[0] + if any(type(value) is not type(first) for value in values): + raise ValueError("tensor records must share types and optional fields") + if first is None: + return None + if isinstance(first, dict): + if any(value.keys() != first.keys() for value in values): + raise ValueError("tensor records must share feature and metadata schemas") + return { + key: collate_tensor_records([value[key] for value in values]) + for key in first + } + if isinstance(first, TensorRecord): + return type(first)( + **{ + field.name: collate_tensor_records( + [vars(value)[field.name] for value in values] + ) + for field in fields(first) + } + ) + return torch.cat(values) diff --git a/src/sequifier/io/values.py b/src/sequifier/io/values.py new file mode 100644 index 000000000..07550621a --- /dev/null +++ b/src/sequifier/io/values.py @@ -0,0 +1,72 @@ +"""Domain scalars shared by table import, preprocessing and environment adapters.""" + +import math +from collections.abc import Mapping +from numbers import Integral, Real +from typing import Annotated, Any, TypeAlias + +from pydantic import BeforeValidator, FiniteFloat, StrictInt, StrictStr + +DomainValue: TypeAlias = Annotated[ + StrictInt | FiniteFloat | StrictStr, + BeforeValidator(lambda value: domain_scalar(value, "domain value")), +] + + +def numeric_scalar(value, label: str) -> float: + if isinstance(value, bool) or not isinstance(value, Real): + raise ValueError(f"{label} must be a finite numeric scalar, got {value!r}") + try: + number = float(value) + except (ValueError, OverflowError) as exc: + raise ValueError(f"{label} is outside the supported numeric range") from exc + if not math.isfinite(number): + raise ValueError(f"{label} must be finite, got {value!r}") + return number + + +def portable_value(value): + """Own JSON-compatible copies of array, tensor, and nested input payloads.""" + try: + tolist = value.tolist + except AttributeError: + pass + else: + value = tolist() + if isinstance(value, dict): + return {key: portable_value(item) for key, item in value.items()} + if isinstance(value, (tuple, list)): + return [portable_value(item) for item in value] + if isinstance(value, float) and not math.isfinite(value): + raise ValueError("model payloads must contain finite values") + if value is None or isinstance(value, (str, bool, int, float)): + return value + raise ValueError(f"unsupported model payload type: {type(value).__name__}") + + +def domain_scalar(value, label: str): + if isinstance(value, str): + return value + number = numeric_scalar(value, label) + return int(value) if isinstance(value, Integral) else number + + +def domain_fields(values, expected: set[str], label: str) -> dict: + if not isinstance(values, Mapping) or set(values) != expected: + raise ValueError( + f"{label} must supply exactly these fields: {sorted(expected)}" + ) + return { + key: domain_scalar(values[key], f"{label}.{key}") for key in sorted(expected) + } + + +def numeric_fields(values, expected: set[str], label: str) -> dict[str, float]: + return { + key: numeric_scalar(value, f"{label}.{key}") + for key, value in domain_fields(values, expected, label).items() + } + + +def portable_fields(values: Mapping[str, Any]) -> dict[str, Any]: + return {name: portable_value(value) for name, value in values.items()} diff --git a/src/sequifier/io/yaml.py b/src/sequifier/io/yaml.py index 7f6ba3887..bc7f3b528 100644 --- a/src/sequifier/io/yaml.py +++ b/src/sequifier/io/yaml.py @@ -10,14 +10,6 @@ def represent_sequifier_object(dumper, data): """Represent sequifier config objects as plain YAML mappings.""" values = data.model_dump() if isinstance(data, BaseModel) else vars(data).copy() - for field_name in ("freeze", "freezing_except"): - if values.get(field_name) is None: - values.pop(field_name, None) - for component_name in ("ingestion", "backbone", "decoder"): - for suffix in ("freeze", "freezing_except"): - field_name = f"{component_name}_{suffix}" - if values.get(field_name) is None: - values.pop(field_name, None) return dumper.represent_dict(values) diff --git a/src/sequifier/make.py b/src/sequifier/make.py index 7339d6ba9..9a01ea7b0 100644 --- a/src/sequifier/make.py +++ b/src/sequifier/make.py @@ -1,94 +1,153 @@ import os +from importlib.resources import files +from pathlib import Path from sequifier.typechecking import beartype -preprocess_config_string = """project_root: . -preprocessing_data_path: PLEASE FILL +preprocess_config_string = """preprocessing_data_path: PLEASE FILL read_format: csv write_format: parquet -selected_columns: [EXAMPLE_INPUT_COLUMN_NAME] # should include all target column, can include additional columns -column_data_types: null # optional map of selected columns to output dtypes, e.g. {EXAMPLE_INPUT_COLUMN_NAME: Float32} +selected_columns: +- EXAMPLE_INPUT_COLUMN_NAME +column_data_types: null mask_column: null - split_ratios: - 0.8 - 0.1 - 0.1 -split_method: within_sequence # one of within_sequence, between_sequence, value_cutoff -# For value_cutoff, set split_ratios: null and configure both fields below. -# split_column: eventTimestamp -# split_values: [2024-01-01T00:00:00Z, 2024-07-01T00:00:00Z] # creates 3 splits -# Timestamp split columns support Date/Datetime (or ISO strings), not Time/Duration, -# and cannot be used as training or inference inputs/targets. -window_length: 49 +split_method: within_sequence max_target_offset: 1 window_stride: 49 -# prediction_aligned_splits: [1, 2] -# prediction_length: 1 -# target_offset: 1 max_rows: null +window_lengths: +- 49 +- 49 +- 49 +run: + project_root: . """ -train_config_string = """project_root: . -model_name: PLEASE FILL -device: cuda - -global_training: - training_objective: causal - context_length: 48 - inference_batch_size: 10 - batch_size: 10 - learning_rate: 0.0001 - optimizer: {name: AdamW} - scheduler: {name: StepLR, step_size: 1, gamma: 0.99} - scheduler_step_on: epoch - reset_optimization_on_phase: true - -model: +train_config_string = """model: backbone: architecture: - dim_model: 128 + model_dim: 128 num_layers: 3 - attention: {n_heads: 16} - feed_forward: {dim: 128} + attention: + num_heads: 16 + feed_forward: + hidden_dim: 128 dropout: 0.2 - interface: - input_columns: [EXAMPLE_INPUT_COLUMN_NAME] - target_columns: [EXAMPLE_TARGET_COLUMN_NAME] - ingestion: {type: embedding, output_dim: 128} - decoder: {type: linear, prediction_length: 1} - -dataset: - part: {metadata_config_path: PLEASE FILL} - criterion: {EXAMPLE_TARGET_COLUMN_NAME: MSELoss} - -training_plan: {epochs: 10} - -evaluation: true - -export_generative_model: PLEASE FILL # true or false -export_embedding_model: PLEASE FILL # true or false + interfaces: + default: + input_columns: + - EXAMPLE_INPUT_COLUMN_NAME + target_columns: + - EXAMPLE_TARGET_COLUMN_NAME + ingestion: + type: embedding + output_dim: 128 + decoder: + type: linear + prediction_length: 1 + execution: + context_length: 48 + training_objective: causal +training_plan: + phases: + - name: supervised + algorithm: supervised + epochs: 10 + mode: sequential + sources: + - source: default.default + batch_size: 10 + accumulation_batches: 1 + optimization: + model: + learning_rate: 0.0001 + optimizer: + name: AdamW + arguments: {} + scheduler: + name: StepLR + arguments: + step_size: 1 + gamma: 0.99 + scheduler_step_on: epoch + evaluation: + sources: + - source: default.default + cadence: + epochs: 1 + batch_size: 10 + before_phase: true + on_snapshot: true + model_selection: + handoff: last + checkpoints: + cadence: + epochs: 1 + learning: + datasets: + default: + criterion: + EXAMPLE_TARGET_COLUMN_NAME: MSELoss +dataset_training: + default: + parts: + default: + metadata_config_path: PLEASE FILL + read_format: parquet + split: + validation_fraction: 0.2 +run: + project_root: . + model_name: PLEASE FILL + reproducibility: + seed: 1010 + enforce_determinism: false + execution: + device: cuda + distribution: + strategy: local + world_size: 1 + backend: nccl + precision: {} + data_loading: {} + compilation: + torch_compile: outer + recovery: {} + logging: {} + outputs: {} + export: + generative_model: true + embedding_model: false + onnx: true + pt: false + batch_size: 10 """ -infer_config_string = """project_root: . -preprocessing_data_path: PLEASE FILL -model_type: PLEASE FILL # generative or embedding -model_path: PLEASE FILL - -input_columns: [EXAMPLE_INPUT_COLUMN_NAME] # should include all target column, can include additional columns -target_columns: [EXAMPLE_TARGET_COLUMN_NAME] +infer_config_string = """input_columns: +- EXAMPLE_INPUT_COLUMN_NAME +target_columns: +- EXAMPLE_TARGET_COLUMN_NAME target_column_types: EXAMPLE_TARGET_COLUMN_NAME: real - training_objective: causal output_probabilities: false decode_categories: true -device: cpu context_length: 48 -inference_batch_size: 10 - -autoregressive: true generation_steps: 5 +task: forecast +model_paths: +- PLEASE FILL +metadata_config_path: configs/metadata_configs/PLEASE_FILL.json +autoregressive_generation: true +run: + project_root: . + execution: + device: cpu + batch_size: 10 """ gitignore_string = """models/ @@ -122,3 +181,16 @@ def make(args): with open(f"{project_name}/configs/infer.yaml", "w") as f: f.write(infer_config_string) + + for name in ( + "train-policy", + "infer-policy", + "train-iql", + "train-ppo", + "train-dpo", + "train-grpo", + ): + template = files("sequifier").joinpath("templates", f"{name}.yaml") + Path(project_name, "configs", f"{name}.yaml").write_text( + template.read_text(encoding="utf-8"), encoding="utf-8" + ) diff --git a/src/sequifier/model/backbone.py b/src/sequifier/model/backbone.py index 37db03ca0..0ac3aa8f9 100644 --- a/src/sequifier/model/backbone.py +++ b/src/sequifier/model/backbone.py @@ -14,7 +14,7 @@ class TransformerBackbone(TransformerEncoderStack): """Dataset-independent temporal transformer. The input and output contract is batch-first - ``[batch, time, input_dim]``. ``input_dim`` normally equals ``dim_model``; + ``[batch, time, input_dim]``. ``input_dim`` normally equals ``model_dim``; ``range_concat`` reserves the final model channel for its coordinate. Attention policy and padding are supplied by the caller as one broadcastable attention mask. @@ -26,11 +26,11 @@ def __init__(self, architecture: Any): architecture, architecture.max_context_length, defer_layers=True ) self.architecture = architecture - self.dim_model = architecture.dim_model + self.model_dim = architecture.model_dim self.max_context_length = architecture.max_context_length self.position_encoding_type = architecture.position_encoding.type - self.positional_encoding_scope = architecture.positional_encoding_scope - self.input_dim = self.dim_model - int( + self.position_encoding_scope = architecture.position_encoding_scope + self.input_dim = self.model_dim - int( self.position_encoding_type == "range_concat" ) self.position_dropout = nn.Dropout(architecture.dropout) @@ -39,17 +39,17 @@ def __init__(self, architecture: Any): self.range_projection: nn.Module | None = None if ( self.position_encoding_type == "learned" - and self.positional_encoding_scope == "global" + and self.position_encoding_scope == "global" ): self.position_embedding = nn.Embedding( - self.max_context_length, self.dim_model + self.max_context_length, self.model_dim ) elif self.position_encoding_type == "range": - self.range_projection = nn.Linear(self.dim_model + 1, self.dim_model) + self.range_projection = nn.Linear(self.model_dim + 1, self.model_dim) sinusoidal = self._sinusoidal_positions( self.max_context_length, - self.dim_model, + self.model_dim, architecture.position_encoding.theta, ) self.register_buffer("sinusoidal_positions", sinusoidal, persistent=False) @@ -85,7 +85,7 @@ def _add_temporal_position(self, x: Tensor) -> Tensor: if ( self.position_encoding_type == "learned" - and self.positional_encoding_scope == "global" + and self.position_encoding_scope == "global" ): positions = torch.arange(sequence_length, device=x.device) if self.position_embedding is None: @@ -139,7 +139,7 @@ def _forward_with_activations( ) -> tuple[Tensor, dict[int | str, Tensor]]: if x.ndim != 3: raise ValueError( - "TransformerBackbone expects [batch, time, dim_model], got " + "TransformerBackbone expects [batch, time, model_dim], got " f"shape {tuple(x.shape)}." ) if not torch.onnx.is_in_onnx_export() and x.shape[-1] != self.input_dim: @@ -152,9 +152,9 @@ def _forward_with_activations( if not torch.onnx.is_in_onnx_export() and prefix.shape != ( x.shape[0], 1, - self.dim_model, + self.model_dim, ): - raise ValueError("Prefix must have shape [batch, 1, dim_model]") + raise ValueError("Prefix must have shape [batch, 1, model_dim]") if ( not torch.onnx.is_in_onnx_export() and x.shape[1] + 1 > self.max_context_length @@ -166,7 +166,7 @@ def _forward_with_activations( "backbone.positioned", x, axes=("batch", "time", "channel"), - width=self.dim_model, + width=self.model_dim, ) return self._run_layers( x, attention_mask, layer_indices, capture_final_norm, trace diff --git a/src/sequifier/model/decoders.py b/src/sequifier/model/decoders.py index dded50d25..1aeb00259 100644 --- a/src/sequifier/model/decoders.py +++ b/src/sequifier/model/decoders.py @@ -30,8 +30,6 @@ def __init__(self, input_dim: int, codec: CategoricalHashCodec): self.layers = nn.ModuleList( nn.Linear(input_dim, width) for width in codec.widths ) - for layer in self.layers: - layer._sequifier_decoder_output = True # type: ignore[attr-defined] def forward(self, hidden: Tensor) -> Tensor: return torch.cat( @@ -99,8 +97,6 @@ def __init__( if target_column in self.hash_codecs else nn.Linear(layer_input_dim, output_dim) ) - if isinstance(output_layer, nn.Linear): - output_layer._sequifier_decoder_output = True # type: ignore[attr-defined] self.output_layers[target_column] = output_layer @staticmethod @@ -151,7 +147,13 @@ def project_hidden_with_activations( @conditional_beartype def decode(self, target_column: str, x: Tensor) -> Tensor: + return self._project_target(target_column, self._project_hidden(x)) + + def decode_targets(self, x: Tensor, targets: tuple[str, ...]) -> dict[str, Tensor]: hidden = self._project_hidden(x) + return {target: self._project_target(target, hidden) for target in targets} + + def _project_target(self, target_column: str, hidden: Tensor) -> Tensor: output_layer = self.output_layers[target_column] if isinstance(output_layer, HashTargetHead): return output_layer(hidden) @@ -198,14 +200,7 @@ def forward( ) outputs = {} for target_column in self.target_columns: - output_layer = self.output_layers[target_column] - output = ( - output_layer(hidden) - if isinstance(output_layer, HashTargetHead) - else output_layer( - cast_floating_to_module_dtype(hidden, output_layer) - ).float() - ) + output = self._project_target(target_column, hidden) if trace is not None: output = trace.emit( f"decoder.branch.{branch_name}.logits.{target_column}", @@ -241,7 +236,7 @@ def __init__( self.input_dim = input_dim self.hidden_dims: list[int] = [] self.hidden_weight_l2 = 0.0 - self.width = architecture.dim_model + self.width = architecture.model_dim sequence_length = len(target_columns) + 1 self.context_projection = nn.Linear(input_dim, self.width) @@ -343,8 +338,6 @@ def __init__( output = nn.Linear(self.width, 1) else: raise ValueError(f"Unknown target column type {kind!r}.") - if isinstance(output, nn.Linear): - output._sequifier_decoder_output = True # type: ignore[attr-defined] if embedding is not None: self.value_embeddings[target] = embedding self.output_layers[target] = output @@ -375,7 +368,7 @@ def _embed_value( layer = cast(nn.Linear, module) values = values.to(dtype=layer.weight.dtype).unsqueeze(-1) return layer(values) - lookup = getattr(self, self._lookup_buffer_names[target]) + lookup = self.get_buffer(self._lookup_buffer_names[target]) global_ids = values.to(torch.long) if valid_mask is None: effective_valid_mask = torch.ones_like(global_ids, dtype=torch.bool) @@ -409,6 +402,45 @@ def _project_target(self, target: str, hidden: Tensor) -> Tensor: return layer(hidden) return layer(cast_floating_to_module_dtype(hidden, layer)).float() + def _initial_slots(self, context: Tensor) -> tuple[Tensor, Tensor]: + projected = self.context_projection( + cast_floating_to_module_dtype(context, self.context_projection) + ).reshape(-1, 1, self.width) + return projected, self.bos_embedding.expand(projected.shape[0], 1, self.width) + + def start_prefix( + self, context: Tensor, teachers: dict[str, Tensor] | None = None + ) -> Tensor: + """Build a decision prefix from context, BOS, and any recorded values.""" + slots = list(self._initial_slots(context)) + if teachers is not None: + slots.extend( + self._prefix_embedding(target, values) + for target, values in teachers.items() + ) + return torch.cat(slots, dim=1) + + def _prefix_embedding(self, target: str, values: Tensor) -> Tensor: + if self.target_column_types[target] == "categorical": + return self.value_embeddings[target](values).unsqueeze(1) + return self._embed_value(target, values).unsqueeze(1) + + def append_prefix(self, prefix: Tensor, target: str, values: Tensor) -> Tensor: + """Append real teacher values or already resolved categorical decoder IDs.""" + return torch.cat([prefix, self._prefix_embedding(target, values)], dim=1) + + def decode_prefix( + self, prefix: Tensor, targets: tuple[str, ...] + ) -> dict[str, Tensor]: + """Decode the requested trailing slots of a causal decision prefix.""" + hidden = self._run_transformer( + prefix, batch=prefix.shape[0], time=1, trace=None, branch_name="" + )[:, -len(targets) :] + return { + target: self._project_target(target, hidden[:, index]) + for index, target in enumerate(targets) + } + def trace_sites(self, branch_name: str) -> tuple[Any, ...]: from sequifier.model.tracing import TraceSite @@ -497,10 +529,7 @@ def _teacher_forced( raise ValueError( f"Missing teacher targets for decoder branch: {sorted(missing)!r}." ) - projected = self.context_projection( - cast_floating_to_module_dtype(context, self.context_projection) - ).reshape(batch * time, 1, self.width) - shifted = [self.bos_embedding.expand(batch * time, 1, self.width)] + shifted = list(self._initial_slots(context)) valid_mask = ( None if teacher_valid_mask is None else teacher_valid_mask[:, -time:] ) @@ -520,7 +549,7 @@ def _teacher_forced( batch * time, 1, self.width ) shifted.append(embedded) - sequence = torch.cat([projected, *shifted], dim=1) + sequence = torch.cat(shifted, dim=1) if trace is not None: shaped = trace.emit( f"decoder.branch.{branch_name}.slot_input", @@ -559,12 +588,7 @@ def _greedy( branch_name: str, ) -> dict[str, Tensor]: batch, time, _ = context.shape - projected = self.context_projection( - cast_floating_to_module_dtype(context, self.context_projection) - ).reshape(batch * time, 1, self.width) - sequence = torch.cat( - [projected, self.bos_embedding.expand(batch * time, 1, self.width)], dim=1 - ) + sequence = torch.cat(self._initial_slots(context), dim=1) outputs: dict[str, Tensor] = {} for index, target in enumerate(self.target_columns): transformer_input = sequence @@ -609,13 +633,7 @@ def _greedy( if self.target_column_types[target] == "categorical" else logits.squeeze(-1) ) - if self.target_column_types[target] == "categorical": - embedded = cast(nn.Embedding, self.value_embeddings[target])( - generated.to(torch.long) - ) - else: - embedded = self._embed_value(target, generated) - sequence = torch.cat([sequence, embedded.unsqueeze(1)], dim=1) + sequence = self.append_prefix(sequence, target, generated) return outputs def forward( @@ -668,7 +686,12 @@ def decode(self, target_column: str, x: Tensor) -> Tensor: TargetDecoderBranch, self.branches[self.target_to_branch[target_column]], ) - raw = branch.decode(target_column, x) + return self.resolve_target(target_column, branch.decode(target_column, x)) + + def hash_codec(self, target_column: str) -> CategoricalHashCodec: + return cast(CategoricalHashCodec, self.hash_codecs[target_column]) + + def resolve_target(self, target_column: str, raw: Tensor) -> Tensor: if target_column in self.hash_codecs: codec = self.hash_codecs[target_column] return codec.resolve(torch.split(raw, codec.widths, dim=-1)) @@ -820,7 +843,7 @@ def resolve_decoding_plan(hparams: Any) -> DecodingPlan: f"target_columns: {sorted(missing_columns)}" ) if branch_config.type == "autoregressive_transformer": - if getattr(hparams, "training_objective", None) == "bert": + if hparams.training_objective == "bert": mask_targets = [ target for target in target_columns @@ -888,12 +911,12 @@ def build_target_decoding( decoder_n_classes = ( hparams.n_classes if target_n_classes is None else target_n_classes ) - input_dim = model.backbone.architecture.dim_model * model.decoder.support + input_dim = model.backbone.architecture.model_dim * model.decoder.support_length hash_codecs = { target: CategoricalHashCodec( config, hparams.n_classes[target], hparams.target_decoder_ids[target] ) - for target, config in getattr(hparams, "categorical_hashing", {}).items() + for target, config in hparams.categorical_hashing.items() if target in hparams.target_decoder_ids } @@ -930,9 +953,13 @@ def build_target_decoding( n_classes=decoder_n_classes, input_dim=input_dim, hidden_dims=DECODER_HIDDEN_DIMS[branch_config.type](branch_config), - activation=getattr(branch_config, "activation", "relu"), - dropout=getattr(branch_config, "dropout", 0.0), - hidden_weight_l2=getattr(branch_config, "hidden_weight_l2", 0.0), + activation=branch_config.activation + if branch_config.type == "mlp" + else "relu", + dropout=branch_config.dropout if branch_config.type == "mlp" else 0.0, + hidden_weight_l2=branch_config.hidden_weight_l2 + if branch_config.type == "mlp" + else 0.0, hash_codecs={ target: hash_codecs[target] for target in branch.target_columns diff --git a/src/sequifier/model/depth_ingestion.py b/src/sequifier/model/depth_ingestion.py index e9719c9d6..1b064b0f4 100644 --- a/src/sequifier/model/depth_ingestion.py +++ b/src/sequifier/model/depth_ingestion.py @@ -35,7 +35,7 @@ def __init__( self.categorical_columns = frozenset(categorical_columns) self.mask_key = depth_mask_metadata_key(config.layout) architecture = config.architecture - width = architecture.dim_model + width = architecture.model_dim dims = config.feature_embedding_dims or get_feature_embedding_dims( width, categorical_columns, real_columns, n_classes ) @@ -55,21 +55,21 @@ def __init__( ) self.cls = nn.Parameter(torch.zeros(1, 1, width)) self.position_embedding = ( - nn.Embedding(layout.context_length + 1, width) + nn.Embedding(layout.max_depth_length + 1, width) if architecture.position_encoding.type == "learned" else None ) self.register_buffer( "sinusoidal_positions", TransformerBackbone._sinusoidal_positions( - layout.context_length + 1, + layout.max_depth_length + 1, width, architecture.position_encoding.theta, ), persistent=False, ) self.position_dropout = nn.Dropout(architecture.dropout) - self.stack = TransformerEncoderStack(architecture, layout.context_length + 1) + self.stack = TransformerEncoderStack(architecture, layout.max_depth_length + 1) self.output_projection_layer = ( nn.Linear(width, config.output_dim) if width != config.output_dim @@ -81,9 +81,6 @@ def __init__( else None ) self.drop = nn.Dropout(config.dropout) - # Ownership affects new optimizer IDs without changing legacy attention ingestions. - for parameter in self.parameters(): - parameter._sequifier_depth_parameter = True def forward(self, src, metadata): valid = metadata[self.mask_key] diff --git a/src/sequifier/model/encoder_stack.py b/src/sequifier/model/encoder_stack.py index ae444a16d..f89f1b7dd 100644 --- a/src/sequifier/model/encoder_stack.py +++ b/src/sequifier/model/encoder_stack.py @@ -19,7 +19,7 @@ def __init__( ): super().__init__() self.architecture = architecture - self.dim_model = architecture.dim_model + self.model_dim = architecture.model_dim self.max_context_length = max_context_length if not defer_layers: self._construct_layers() @@ -40,7 +40,7 @@ def _construct_layers(self): normalization_type = architecture.normalization.type norm_class = RMSNorm if normalization_type == "rmsnorm" else nn.LayerNorm norm_eps = 1e-6 if normalization_type == "rmsnorm" else 1e-3 - self.final_norm = norm_class(self.dim_model, eps=norm_eps) + self.final_norm = norm_class(self.model_dim, eps=norm_eps) else: self.final_norm = nn.Identity() @@ -62,7 +62,7 @@ def _run_layers( f"{site_prefix}.layer.{index}.input", x, axes=axes, - width=self.dim_model, + width=self.model_dim, ) x = layer( x, @@ -75,7 +75,7 @@ def _run_layers( f"{site_prefix}.layer.{index}.output", x, axes=axes, - width=self.dim_model, + width=self.model_dim, ) if index in selected_indices: activations[index] = x @@ -85,7 +85,7 @@ def _run_layers( f"{site_prefix}.final_norm", x, axes=axes, - width=self.dim_model, + width=self.model_dim, ) if capture_final_norm: activations["final_norm"] = x diff --git a/src/sequifier/model/execution.py b/src/sequifier/model/execution.py new file mode 100644 index 000000000..21c0828eb --- /dev/null +++ b/src/sequifier/model/execution.py @@ -0,0 +1,69 @@ +"""Model execution facts, independent of dataset ownership and learning phases.""" + +from __future__ import annotations + +from dataclasses import dataclass, field + +from sequifier.config.train_config import ( + ModelSpecModel, + ResolvedInterfaceStorage, + ResolvedModelInterface, + ResolvedSequifierConfig, +) +from sequifier.model.execution_options import ModelExecutionOptions + + +@dataclass +class ModelExecution: + model: ModelSpecModel + interfaces: dict[str, ResolvedModelInterface] + options: ModelExecutionOptions + device: str = "cpu" + project_root: str = "." + model_name: str = "loaded-model" + seed: int = 0 + embedding_layer_names: list[str] = field( + default_factory=lambda: ["backbone.final_norm"] + ) + + interface_storage: dict[str, ResolvedInterfaceStorage] = field(default_factory=dict) + + @classmethod + def from_config( + cls, config: ResolvedSequifierConfig | ModelExecution + ) -> ModelExecution: + if isinstance(config, ModelExecution): + return config + return cls( + model=config.model, + interfaces=config.interfaces, + options=config.model_options, + interface_storage=config.interface_storage, + device=config.device, + project_root=config.project_root, + model_name=config.model_name, + seed=config.seed, + embedding_layer_names=config.embedding_layer_names, + ) + + @property + def interface_names(self) -> tuple[str, ...]: + return tuple(self.interfaces) + + def interface(self, name: str | None = None) -> ResolvedModelInterface: + if name is None: + if len(self.interfaces) != 1: + raise ValueError("A model interface selection is required") + name = next(iter(self.interfaces)) + return self.interfaces[name] + + def validate(self) -> ModelExecution: + from sequifier.config.train_config import interface_build_view + from sequifier.model.decoders import resolve_decoding_plan + from sequifier.model.ingestion_compiler import resolve_ingestion_plan + + for interface in self.interfaces.values(): + view = interface_build_view(self, interface) + resolve_ingestion_plan(view) + resolve_decoding_plan(view) + return self diff --git a/src/sequifier/model/execution_options.py b/src/sequifier/model/execution_options.py new file mode 100644 index 000000000..f6f4df633 --- /dev/null +++ b/src/sequifier/model/execution_options.py @@ -0,0 +1,80 @@ +"""Model-owned options shared by training models and portable artifacts.""" + +from __future__ import annotations + +from typing import Literal + +from pydantic import Field, field_validator + +from sequifier.config.naming import ConfigModel + +DEFAULT_MAX_CONCAT_TENSORS = 12 + + +class _SharedExecutionOptions(ConfigModel): + target_offset: int = Field(default=1, ge=0) + forward_attention: Literal["causal", "bidirectional"] | None = None + layer_type_dtypes: dict[str, str] | None = None + max_concat_tensors: int = Field(default=DEFAULT_MAX_CONCAT_TENSORS, gt=0) + + @field_validator("training_objective", check_fields=False) + @classmethod + def validate_objective(cls, value: str | None) -> str | None: + if value is not None: + from sequifier.objectives import get_objective_class + + get_objective_class(value) + return value + + @field_validator("layer_type_dtypes") + @classmethod + def validate_layer_dtypes(cls, value: dict[str, str] | None): + if value is None: + return value + allowed_keys = {"embedding", "linear", "conv", "norm", "decoder"} + allowed_types = { + "float32", + "float16", + "bfloat16", + "float64", + "float8_e4m3fn", + "float8_e5m2", + } + if invalid := set(value) - allowed_keys: + raise ValueError(f"Invalid layer_type_dtypes keys: {sorted(invalid)}") + if invalid := set(value.values()) - allowed_types: + raise ValueError(f"Invalid layer_type_dtypes values: {sorted(invalid)}") + return value + + +class ExecutionOptions(_SharedExecutionOptions): + context_length: int | None = Field(default=None, gt=0) + training_objective: str | None = None + + +class ModelExecutionOptions(_SharedExecutionOptions): + context_length: int = Field(gt=0) + training_objective: str + + @property + def attention(self) -> Literal["causal", "bidirectional"]: + return self.forward_attention or ( + "bidirectional" if self.training_objective == "bert" else "causal" + ) + + @classmethod + def from_training(cls, settings: ExecutionOptions) -> ModelExecutionOptions: + context_length = settings.context_length + training_objective = settings.training_objective + if context_length is None or training_objective is None: + raise ValueError( + "Model execution requires context_length and training_objective" + ) + return cls( + context_length=context_length, + training_objective=training_objective, + target_offset=settings.target_offset, + forward_attention=settings.forward_attention, + layer_type_dtypes=settings.layer_type_dtypes, + max_concat_tensors=settings.max_concat_tensors, + ) diff --git a/src/sequifier/model/execution_schema.py b/src/sequifier/model/execution_schema.py index c7ede7fc9..037c43c6a 100644 --- a/src/sequifier/model/execution_schema.py +++ b/src/sequifier/model/execution_schema.py @@ -9,6 +9,7 @@ depth_mask_metadata_key, ) from sequifier.config.image_config import image_views +from sequifier.contract_versions import ContractVersions, require_contract_version EXECUTION_SCHEMA_KEY = "sequifier.execution_schema" MODEL_CONFIG_KEY = "sequifier.model_config" @@ -30,23 +31,28 @@ class ExecutionSchema: outputs: tuple[dict, ...] depth_layouts: dict n_classes: dict - version: int = 1 + version: int = ContractVersions.MODEL_EXECUTION_SCHEMA batch_policy: str = "symbolic" + def __post_init__(self): + require_contract_version( + "model execution schema", + self.version, + ContractVersions.MODEL_EXECUTION_SCHEMA, + ) + @classmethod def from_interface( cls, interface, context_length, *, embedding=False, embedding_width=None ): - layouts: DepthLayoutRegistryModel | None = getattr( - interface, "depth_layouts", None - ) + layouts: DepthLayoutRegistryModel | None = interface.depth_layouts if layouts is None: layouts = DepthLayoutRegistryModel() inputs = [] for index, column in enumerate(interface.input_columns): layout = layouts.layout_for_column(column) shape = ( - ("batch", context_length, layout.context_length) + ("batch", context_length, layout.max_depth_length) if layout else ("batch", context_length) ) @@ -61,7 +67,7 @@ def from_interface( ) static_columns = ( interface.static_conditioning.columns - if getattr(interface, "static_conditioning", None) is not None + if interface.static_conditioning is not None else [] ) for index, column in enumerate(static_columns): @@ -107,7 +113,7 @@ def from_interface( ("batch", context_length, capacity), ) ) - if getattr(interface, "preconditioning_image_encoder", None) is not None: + if interface.preconditioning_image_encoder is not None: source = interface.preconditioning_image inputs.append( InputDescriptor( @@ -134,7 +140,7 @@ def from_interface( depth_mask_metadata_key(name), "metadata", "bool", - ("batch", context_length, layout.context_length), + ("batch", context_length, layout.max_depth_length), ) ) outputs = ( @@ -179,8 +185,13 @@ def to_dict(self): @classmethod def from_dict(cls, values): - if values.get("version") != 1 or values.get("batch_policy") != "symbolic": - raise ValueError("Unsupported execution schema version or batch policy") + require_contract_version( + "model execution schema", + values.get("version") if isinstance(values, dict) else None, + ContractVersions.MODEL_EXECUTION_SCHEMA, + ) + if values.get("batch_policy") != "symbolic": + raise ValueError("Unsupported execution schema batch policy") descriptors = tuple( InputDescriptor(**{**value, "shape": tuple(value["shape"])}) for value in values["inputs"] diff --git a/src/sequifier/model/factory.py b/src/sequifier/model/factory.py index a09473742..a0b5e0e45 100644 --- a/src/sequifier/model/factory.py +++ b/src/sequifier/model/factory.py @@ -2,24 +2,27 @@ import logging from dataclasses import dataclass -from typing import Any +from typing import Any, cast import torch from torch import nn +from sequifier.config.initialization_config import ModelInitializationConfig from sequifier.helpers import get_torch_dtype from sequifier.model.backbone import TransformerBackbone from sequifier.model.decoders import build_target_decoding +from sequifier.model.execution import ModelExecution from sequifier.model.ingestion_compiler import compile_feature_ingestion +from sequifier.model.ingestions import BaseFeatureIngestion, IngestionOutputProjection from sequifier.model.initialization import ( initialize_model_weights, initialize_owned_ingestion, ) from sequifier.model.initialization_seed import derived_seed, isolated_initialization -from sequifier.model.layers import RMSNorm +from sequifier.model.layers import RMSNorm, SequifierEncoderLayer from sequifier.model.network import ComposableTransformerNetwork, ModelInterfaceModule from sequifier.model.static_conditioning import StaticConditionEncoder -from sequifier.objectives import CausalObjective, create_objective +from sequifier.objectives import BERTObjective, CausalObjective from sequifier.special_tokens import resolve_categorical_decoder_ids from sequifier.typechecking import beartype @@ -61,7 +64,6 @@ def target_global_to_decoder(self) -> dict[str, list[int]]: @dataclass class BuiltModel: network: ComposableTransformerNetwork - objectives: dict[str, Any] runtime_metadata: ModelRuntimeMetadata @@ -69,11 +71,7 @@ def validate_hash_contracts(network: ComposableTransformerNetwork, config: Any) """Check persisted codec buffers after weights are restored.""" from sequifier.model.ingestions import EmbeddingFeatureIngestion, MultiHashEmbedding - interfaces = { - dataset.model_interface: dataset.interface - for dataset in config.dataset_training.values() - } - for name, interface in interfaces.items(): + for name, interface in ModelExecution.from_config(config).interfaces.items(): route = network.resolve_interface(name) for column, expected in interface.categorical_hash_contracts.items(): from sequifier.model.hash_codec import CategoricalHashCodec @@ -88,7 +86,7 @@ def validate_hash_contracts(network: ComposableTransformerNetwork, config: Any) f"Resolved hash contract for {column!r} differs from its configuration" ) if column in interface.target_decoder_ids: - codec = route.decoder.hash_codecs[column] + codec = route.require_decoder().hash_codec(column) if codec.contract() != expected: raise ValueError( f"Persisted hash codec for {column!r} differs from its resolved contract" @@ -125,7 +123,7 @@ def compile_unique_layers(layers: nn.ModuleList) -> None: @beartype def _apply_layer_dtypes(network: nn.Module, config: Any) -> None: - layer_config = config.global_training.layer_type_dtypes + layer_config = config.options.layer_type_dtypes if not layer_config: return for name, module in network.named_modules(): @@ -164,29 +162,35 @@ def _apply_layer_dtypes(network: nn.Module, config: Any) -> None: module.to(dtype=get_torch_dtype(layer_config["norm"])) +@dataclass(frozen=True) +class _ComponentInitialization: + module: nn.Module + policy: ModelInitializationConfig + seed: int | None = None + + @beartype def _initialize_components( - components: dict[str, tuple[nn.Module, Any]], logger: Any + components: dict[str, _ComponentInitialization], logger: Any ) -> None: configured_targets: set[tuple[str, str]] = set() matched_targets: set[tuple[str, str]] = set() - for module, initialization in components.values(): + for component in components.values(): + module, initialization = component.module, component.policy configured_targets.update(initialization.configured_targets()) - initializer = ( - initialize_owned_ingestion - if hasattr(module, "_sequifier_initialization_config") - else initialize_model_weights - ) - with isolated_initialization( - getattr(module, "_sequifier_initialization_seed", None) - ): - matched_targets.update( - initializer( - module, - initialization, - warn_unmatched=False, + with isolated_initialization(component.seed): + if ( + isinstance(module, BaseFeatureIngestion) + and module.initialization_config is not None + ): + matched = initialize_owned_ingestion( + module, initialization, warn_unmatched=False ) - ) + else: + matched = initialize_model_weights( + module, initialization, warn_unmatched=False + ) + matched_targets.update(matched) unmatched = configured_targets.difference(matched_targets) if unmatched: targets = ", ".join(f"{group}.{kind}" for group, kind in sorted(unmatched)) @@ -200,7 +204,7 @@ def _decoder_metadata(view: Any) -> InterfaceRuntimeMetadata: for column in view.target_columns if view.target_column_types[column] == "categorical" } - persisted_decoder_ids = getattr(view, "target_decoder_ids", None) + persisted_decoder_ids = view.target_decoder_ids if persisted_decoder_ids: if set(persisted_decoder_ids) != categorical_targets: raise ValueError( @@ -255,49 +259,45 @@ def _decoder_metadata(view: Any) -> InterfaceRuntimeMetadata: @beartype def _build_composable_network( - config: Any, + config: ModelExecution, *, device: torch.device, initialize: bool, + restored_history_interface: str | None, logger: Any, + encoder_interface: str | None = None, ) -> BuiltModel: from sequifier.config.train_config import interface_build_view - resolved_interfaces = {} - for dataset in config.dataset_training.values(): - resolved_interfaces.setdefault(dataset.model_interface, dataset.interface) - objectives = {} - for name, interface in resolved_interfaces.items(): - objective_view = interface_build_view(config, interface) - # A lean inference bundle may omit next-occurrence loss metadata. Its - # forward attention policy remains the ordinary causal policy. - if ( - config.global_training.training_objective == "next_occurrence" - and config.global_training.next_occurrence_config is None - ): - objectives[name] = CausalObjective(objective_view) - else: - objectives[name] = create_objective(objective_view) - objective = next(iter(objectives.values())) + interfaces = ( + config.interfaces + if encoder_interface is None + else {encoder_interface: config.interface(encoder_interface)} + ) backbone_seed = config.model.backbone.initialization_seed with isolated_initialization(derived_seed(backbone_seed, (), "constructor")): backbone = TransformerBackbone(config.model.backbone.architecture) - backbone._sequifier_initialization_seed = derived_seed( - backbone_seed, (), "initializer" + backbone_initialization = _ComponentInitialization( + backbone, + config.model.backbone.initialization, + derived_seed(backbone_seed, (), "initializer"), ) routes = {} runtime_metadata = {} - route_initialization_components: dict[str, tuple[nn.Module, Any]] = {} - for name, interface in resolved_interfaces.items(): + route_initialization_components: dict[str, _ComponentInitialization] = {} + for name, interface in interfaces.items(): view = interface_build_view(config, interface) + load_pretrained = ( + initialize and device.type != "meta" and name != restored_history_interface + ) built_ingestion = compile_feature_ingestion( hparams=view, - direct_real_dtype_provider=lambda: backbone.layers[ - 0 - ].ff.get_first_layer_dtype(), - device_max_concat_length=config.global_training.device_max_concat_length, - load_pretrained=initialize, + direct_real_dtype_provider=lambda: cast( + SequifierEncoderLayer, backbone.layers[0] + ).ff.get_first_layer_dtype(), + max_concat_tensors=config.options.max_concat_tensors, + load_pretrained=load_pretrained, ) from sequifier.model.ingestions import EmbeddingFeatureIngestion @@ -323,29 +323,30 @@ def _build_composable_network( adapter: nn.Module = ( nn.Identity() if built_ingestion.width == backbone.input_dim - else nn.Linear(built_ingestion.width, backbone.input_dim) + else IngestionOutputProjection( + built_ingestion.width, backbone.input_dim + ) ) - adapter._sequifier_initialization_seed = derived_seed( - interface.ingestion.initialization_seed, ("adapter",), "initializer" - ) - adapter._sequifier_layer_group = "ingestion.output_projection" # type: ignore[attr-defined] - metadata = _decoder_metadata(view) - decoder = build_target_decoding( - view, - target_n_classes=metadata.target_n_classes, - target_global_to_decoder=metadata.target_global_to_decoder, - ) - for column, expected in interface.categorical_hash_contracts.items(): - actual = ( - decoder.hash_codecs[column].contract() - if column in decoder.hash_codecs - else None + decoder = None + if encoder_interface is None: + metadata = _decoder_metadata(view) + runtime_metadata[name] = metadata + decoder = build_target_decoding( + view, + target_n_classes=metadata.target_n_classes, + target_global_to_decoder=metadata.target_global_to_decoder, ) - if column in interface.target_decoder_ids and actual != expected: - raise ValueError( - f"Persisted categorical hash contract for {column!r} differs " - "from the resolved decoder codec." + for column, expected in interface.categorical_hash_contracts.items(): + actual = ( + decoder.hash_codecs[column].contract() + if column in decoder.hash_codecs + else None ) + if column in interface.target_decoder_ids and actual != expected: + raise ValueError( + f"Persisted categorical hash contract for {column!r} differs " + "from the resolved decoder codec." + ) static_encoder = None if interface.static_conditioning is not None: static_seed = interface.ingestion.initialization_seed @@ -357,25 +358,27 @@ def _build_composable_network( interface.static_categorical_columns, interface.static_real_columns, interface.n_classes, - backbone.dim_model, - interface.static_conditioning.embedding_size, + backbone.model_dim, + interface.static_conditioning.total_embedding_dim, interface.static_conditioning.feature_embedding_dims, ) - static_encoder._sequifier_initialization_seed = derived_seed( - static_seed, ("static_conditioning",), "initializer" - ) image_prefix_encoder = None if interface.preconditioning_image_encoder is not None: from sequifier.model.images import ImagePrefixEncoder + image_source = interface.preconditioning_image + if image_source is None: + raise ValueError( + "Metadata lacks the configured preconditioning image source" + ) image_prefix_encoder = ImagePrefixEncoder( interface.preconditioning_image_encoder, - backbone.dim_model, - len(interface.preconditioning_image.views), - initialize, + backbone.model_dim, + len(image_source.views), + load_pretrained, ) prefix_projection = ( - nn.Linear(2 * backbone.dim_model, backbone.dim_model) + nn.Linear(2 * backbone.model_dim, backbone.model_dim) if static_encoder is not None and image_prefix_encoder is not None else None ) @@ -383,65 +386,90 @@ def _build_composable_network( ingestion=built_ingestion.module, ingestion_adapter=adapter, decoder=decoder, - decoder_input_width=interface.decoder.support * backbone.dim_model, - decoding_support=interface.decoder.support, + decoder_input_width=interface.decoder.support_length * backbone.model_dim, + decoding_support=interface.decoder.support_length, prediction_length=interface.decoder.prediction_length, target_columns=tuple(interface.target_columns), target_column_types=dict(interface.target_column_types), static_condition_encoder=static_encoder, preconditioning_image_encoder=image_prefix_encoder, prefix_projection=prefix_projection, + depth_layouts=interface.depth_layouts, + input_columns=tuple(interface.input_columns), + input_n_classes=interface.n_classes, ) - routes[name].depth_layouts = interface.depth_layouts - routes[name].input_columns = tuple(interface.input_columns) - routes[name].input_n_classes = interface.n_classes - runtime_metadata[name] = metadata + from sequifier.model.prediction_distribution import PredictionScales + + distributions = interface.prediction_distributions + if encoder_interface is None and distributions: + routes[name].prediction_scales = PredictionScales(distributions).to(device) route_initialization_components.update( { - f"interfaces.{name}.ingestion": ( + f"interfaces.{name}.ingestion": _ComponentInitialization( built_ingestion.module, interface.ingestion.initialization, + built_ingestion.module.initialization_seed, ), - f"interfaces.{name}.ingestion_adapter": ( + f"interfaces.{name}.ingestion_adapter": _ComponentInitialization( adapter, interface.ingestion.initialization, - ), - f"interfaces.{name}.decoder": ( - decoder, - interface.decoder.initialization, + derived_seed( + interface.ingestion.initialization_seed, + ("adapter",), + "initializer", + ), ), } ) - if routes[name].static_condition_encoder is not None: + if decoder is not None: + route_initialization_components[f"interfaces.{name}.decoder"] = ( + _ComponentInitialization( + decoder, + interface.decoder.initialization, + ) + ) + if static_encoder is not None: route_initialization_components[ f"interfaces.{name}.static_condition_encoder" - ] = ( - routes[name].static_condition_encoder, + ] = _ComponentInitialization( + static_encoder, interface.ingestion.initialization, + derived_seed( + interface.ingestion.initialization_seed, + ("static_conditioning",), + "initializer", + ), ) if image_prefix_encoder is not None: route_initialization_components[ f"interfaces.{name}.preconditioning_image_encoder" - ] = (image_prefix_encoder, interface.ingestion.initialization) + ] = _ComponentInitialization( + image_prefix_encoder, interface.ingestion.initialization + ) if prefix_projection is not None: route_initialization_components[f"interfaces.{name}.prefix_projection"] = ( - prefix_projection, - interface.ingestion.initialization, + _ComponentInitialization( + prefix_projection, + interface.ingestion.initialization, + ) ) + attention_objective = ( + CausalObjective if config.options.attention == "causal" else BERTObjective + ) network = ComposableTransformerNetwork( backbone=backbone, interfaces=routes, - attention_mask_policy=objective.build_attention_mask_policy( - config.global_training.context_length + attention_mask_policy=attention_objective.build_attention_mask_policy( + config.options.context_length ), - attention_mask_policy_with_prefix=objective.build_attention_mask_policy( - config.global_training.context_length + 1 + attention_mask_policy_with_prefix=attention_objective.build_attention_mask_policy( + config.options.context_length + 1 ), - context_length=config.global_training.context_length, + context_length=config.options.context_length, ) if initialize: - if len(routes) == 1: + if len(config.interfaces) == 1: # Preserve the legacy single-interface seeded initialization order. # The order is observable because default initializers consume RNG. name = next(iter(routes)) @@ -454,18 +482,14 @@ def _build_composable_network( f"interfaces.{name}.ingestion_adapter" ] ), - "backbone": ( - backbone, - config.model.backbone.initialization, - ), - f"interfaces.{name}.decoder": route_initialization_components[ - f"interfaces.{name}.decoder" - ], + "backbone": backbone_initialization, } - if routes[name].static_condition_encoder is not None: - key = f"interfaces.{name}.static_condition_encoder" - initialization_components[key] = route_initialization_components[key] - for component in ("preconditioning_image_encoder", "prefix_projection"): + for component in ( + "decoder", + "static_condition_encoder", + "preconditioning_image_encoder", + "prefix_projection", + ): key = f"interfaces.{name}.{component}" if key in route_initialization_components: initialization_components[key] = route_initialization_components[ @@ -473,10 +497,7 @@ def _build_composable_network( ] else: initialization_components = { - "backbone": ( - backbone, - config.model.backbone.initialization, - ), + "backbone": backbone_initialization, **route_initialization_components, } _initialize_components(initialization_components, logger) @@ -484,7 +505,6 @@ def _build_composable_network( network.to(device) return BuiltModel( network=network, - objectives=objectives, runtime_metadata=ModelRuntimeMetadata( interfaces=runtime_metadata, device=device, @@ -498,14 +518,34 @@ def build_transformer_network( *, device: torch.device, initialize: bool = True, + restored_history_interface: str | None = None, logger: Any | None = None, ) -> BuiltModel: """Build one shared backbone and every distinct named model interface.""" resolved_logger: Any = logger if logger is not None else logging.getLogger(__name__) return _build_composable_network( - config, + ModelExecution.from_config(config), device=device, initialize=initialize, + restored_history_interface=restored_history_interface, logger=resolved_logger, ) + + +def build_history_encoder( + config: Any, + interface_name: str, + *, + device: torch.device, + restored_history: bool, +) -> ComposableTransformerNetwork: + """Build the selected ingestion path and backbone, using the actor's namespaces.""" + return _build_composable_network( + ModelExecution.from_config(config), + device=device, + initialize=True, + restored_history_interface=interface_name if restored_history else None, + logger=logging.getLogger(__name__), + encoder_interface=interface_name, + ).network diff --git a/src/sequifier/model/frozen_image_cache.py b/src/sequifier/model/frozen_image_cache.py new file mode 100644 index 000000000..60595a7b0 --- /dev/null +++ b/src/sequifier/model/frozen_image_cache.py @@ -0,0 +1,212 @@ +"""Bounded, phase-scoped frozen-backbone features, outside model state dicts.""" + +from collections import OrderedDict +from contextlib import contextmanager +from dataclasses import dataclass, field + +import torch + +from sequifier.io.image_feature_cache import ( + image_feature_cache_enabled, + image_identity_scope, +) +from sequifier.runtime.resource_accounting import Allocation, allocation + + +@dataclass +class FrozenImageCache: + budget: int + entries: OrderedDict = field(default_factory=OrderedDict) + retained_bytes: int = 0 + hits: int = 0 + misses: int = 0 + storage: Allocation | None = field(default=None, repr=False, compare=False) + owners: int = field(default=0, init=False, repr=False, compare=False) + + @classmethod + def scoped(cls, budget): + return cls(budget, storage=allocation("frozen_image_features")) + + def acquire(self): + self.owners += 1 + return self + + def release(self): + if self.owners: + self.owners -= 1 + if not self.owners: + self.clear() + if self.storage is not None: + self.storage.close() + + def replacement(self): + if self.storage is None: + return type(self)(self.budget) + self.release() + return type(self).scoped(self.budget).acquire() + + def __deepcopy__(self, memo): + return type(self)(self.budget) + + def clear(self): + self.entries.clear() + self._resize(0) + + def _resize(self, size): + self.retained_bytes = size + if self.storage is not None: + self.storage.resize(size) + + def get(self, key): + feature = self.entries.get(key) + if feature is not None: + self.entries.move_to_end(key) + self.hits += 1 + else: + self.misses += 1 + return feature + + def put(self, key, feature): + size = feature.numel() * feature.element_size() + if size > self.budget: + return + previous = self.entries.pop(key, None) + if previous is not None: + self._resize( + self.retained_bytes - previous.numel() * previous.element_size() + ) + while self.retained_bytes + size > self.budget: + _, evicted = self.entries.popitem(last=False) + self._resize(self.retained_bytes - evicted.numel() * evicted.element_size()) + # Own a row, not a view retaining an entire microbatch's GPU storage. + with torch.inference_mode(False), torch.no_grad(): + self.entries[key] = feature.detach().clone() + self._resize(self.retained_bytes + size) + + +def encoder_signature(encoder): + """Track ordinary in-place updates, weight loading and device/dtype moves. + + Inference tensors lack version counters, so caching is bypassed for them. + Updating frozen weights via `.data` requires explicit clear_feature_cache(). + """ + if any(module.training for module in encoder.backbone.modules()) or any( + p.requires_grad for p in encoder.backbone.parameters() + ): + return None + tensors = [ + *encoder.backbone.parameters(), + *encoder.backbone.buffers(), + encoder.mean, + encoder.std, + ] + try: + tensor_state = tuple( + (id(value), value.data_ptr(), value._version, value.device, value.dtype) + for value in tensors + ) + device_type = encoder.mean.device.type + autocast = torch.is_autocast_enabled(device_type) + return ( + tensor_state, + autocast, + torch.get_autocast_dtype(device_type) if autocast else None, + torch.get_float32_matmul_precision(), + torch.backends.cudnn.allow_tf32, + ) + except RuntimeError: + return None + + +def _same_backbone(left, right): + a = {**left.backbone.state_dict(), "__mean": left.mean, "__std": left.std} + b = {**right.backbone.state_dict(), "__mean": right.mean, "__std": right.std} + return a.keys() == b.keys() and all( + a[key].shape == b[key].shape + and a[key].dtype == b[key].dtype + and a[key].device == b[key].device + and torch.equal(a[key], b[key]) + for key in a + ) + + +def _active_caches(models): + if not image_feature_cache_enabled(): + return {} + from sequifier.model.images import ImageEncoder + + return { + id(encoder._feature_cache): encoder._feature_cache + for model in models + for encoder in model.modules() + if isinstance(encoder, ImageEncoder) and encoder._feature_cache is not None + } + + +def image_cache_statistics(*models): + caches = _active_caches(models) + if not caches: + return {} + return { + "image_cache_hits": sum(cache.hits for cache in caches.values()), + "image_cache_misses": sum(cache.misses for cache in caches.values()), + "image_cache_retained_bytes": sum( + cache.retained_bytes for cache in caches.values() + ), + } + + +def release_image_cache_entries(*models): + # GRPO never reuses trajectories after their update. Do not retain features + # from previous iterations merely because the phase's LRU budget has space. + for cache in _active_caches(models).values(): + cache.clear() + + +@contextmanager +def frozen_image_cache_scope(actor, reference, budget): + """Install optional GRPO caches, releasing all entries on success or failure. + + Sharing requires matching module names and exact frozen backbone state. + Each owner checks its own tensor versions before reuse and detaches from a + shared cache if its weights, device, dtype or normalization have changed. + """ + from sequifier.model.images import ImageEncoder + + if not budget: + with image_identity_scope(False): + yield + return + saved, active = [], {} + try: + for model in (actor, reference): + for name, encoder in model.named_modules(): + if not isinstance(encoder, ImageEncoder): + continue + if not encoder.frozen: + continue + signature = encoder_signature(encoder) + if signature is None: + continue + other = active.get(name) + shared = ( + other._feature_cache + if other is not None and _same_backbone(encoder, other) + else None + ) + saved.append( + (encoder, encoder._feature_cache, encoder._cache_signature) + ) + encoder._feature_cache = ( + shared if shared is not None else FrozenImageCache.scoped(budget) + ).acquire() + encoder._cache_signature = signature + active[name] = encoder + with image_identity_scope(bool(saved)): + yield + finally: + for encoder, cache, signature in reversed(saved): + if encoder._feature_cache is not None: + encoder._feature_cache.release() + encoder._feature_cache = cache + encoder._cache_signature = signature diff --git a/src/sequifier/model/hash_codec.py b/src/sequifier/model/hash_codec.py index 0036d1f32..6f9ce4180 100644 --- a/src/sequifier/model/hash_codec.py +++ b/src/sequifier/model/hash_codec.py @@ -5,7 +5,8 @@ import torch from torch import Tensor, nn -HASH_CODEC_VERSION = 1 +from sequifier.contract_versions import ContractVersions + HASH_PRIME = 2_147_483_647 @@ -77,8 +78,8 @@ def __init__(self, config, n_classes: int, decoder_ids: list[int] | None = None) self.kind = config.type self.num_buckets = config.num_buckets self.num_hashes = config.num_hashes - self.seed = getattr(config, "seed", 0) - self.version = HASH_CODEC_VERSION + self.seed = config.seed if config.type == "multi_hash" else 0 + self.version = ContractVersions.HASH_CODEC self.widths = ( [config.num_buckets] * config.num_hashes if config.type == "multi_hash" @@ -182,7 +183,7 @@ def contract(self) -> dict: "offsets": self.offsets.tolist(), "codebook": self.codebook.tolist(), "decoder_ids": self.decoder_ids.tolist(), - "special_token_policy": "reserve_complete_codes_0_1_2_v1" + "special_token_policy": f"reserve_complete_codes_0_1_2_v{ContractVersions.RESERVED_CODE_SPECIAL_TOKENS}" if self.kind == "multi_hash" - else "qr_injective_v1", + else f"qr_injective_v{ContractVersions.INJECTIVE_SPECIAL_TOKENS}", } diff --git a/src/sequifier/model/history_transfer.py b/src/sequifier/model/history_transfer.py new file mode 100644 index 000000000..3ab47a9ea --- /dev/null +++ b/src/sequifier/model/history_transfer.py @@ -0,0 +1,140 @@ +"""Compatibility and weight copying for the model's decision-history path.""" + +from sequifier.model.execution import ModelExecution +from sequifier.model.execution_schema import ExecutionSchema + +_HISTORY_COMPONENTS = ( + "ingestion", + "ingestion_adapter", + "static_condition_encoder", + "preconditioning_image_encoder", + "prefix_projection", +) + + +def _ingestion_contract(ingestion): + values = ingestion.model_dump(exclude={"initialization", "initialization_seed"}) + if ingestion.type == "composite": + values["branches"] = [ + (name, _ingestion_contract(branch)) + for name, branch in ingestion.branches.items() + ] + return values + + +def _history_contract(config, interface_name): + execution = ModelExecution.from_config(config) + if interface_name not in execution.interfaces: + raise ValueError(f"history transfer requires interface {interface_name!r}") + interface = execution.interface(interface_name) + settings = execution.options + static = interface.static_conditioning + columns = interface.input_columns + (static.columns if static else []) + media_fields = { + "path", + "signature", + "time_index_path", + "time_index_signature", + "video_index_paths", + } + return { + "backbone.architecture": execution.model.backbone.architecture, + "layer_type_dtypes": settings.layer_type_dtypes, + "context_length": settings.context_length, + "forward_attention": settings.attention, + "inputs": ExecutionSchema.from_interface( + interface, settings.context_length + ).inputs, + "input_columns": interface.input_columns, + "ingestion": _ingestion_contract(interface.ingestion), + "static_conditioning": static, + "feature_layout": interface.feature_layout, + "depth_layouts": interface.depth_layouts.compatibility_signature(columns), + "image_sources": { + name: item.model_dump(exclude=media_fields) + for name, item in interface.image_sources.items() + }, + "preconditioning_image": interface.preconditioning_image.compatibility_signature() + if interface.preconditioning_image is not None + else None, + "preconditioning_image_encoder": interface.preconditioning_image_encoder, + "normalize_real_columns": interface.normalize_real_columns, + "special_token_ids": interface.special_token_ids, + "input_metadata": { + column: { + "column_data_types": interface.column_data_types.get(column), + "n_classes": interface.n_classes.get(column), + "id_maps": interface.id_maps.get(column), + "categorical_hashing": interface.categorical_hashing.get(column), + "selected_columns_statistics": interface.selected_columns_statistics.get( + column + ), + } + for column in columns + }, + } + + +def validate_history_transfer(source, destination, interface_name): + expected = _history_contract(source, interface_name) + actual = _history_contract(destination, interface_name) + for field, value in expected.items(): + if value != actual[field]: + if field == "input_metadata": + field = next( + f"{field}.{column}.{key}" + for column, metadata in value.items() + for key, entry in metadata.items() + if entry != actual[field][column][key] + ) + raise ValueError(f"history transfer differs at {interface_name}.{field}") + + +def history_components(network, interface_name): + route = network.resolve_interface(interface_name) + return { + "backbone": network.backbone, + **{ + f"interfaces.{interface_name}.{name}": component + for name, component in ( + ("ingestion", route.ingestion), + ("ingestion_adapter", route.ingestion_adapter), + ("static_condition_encoder", route.static_condition_encoder), + ("preconditioning_image_encoder", route.preconditioning_image_encoder), + ("prefix_projection", route.prefix_projection), + ) + if component is not None + }, + } + + +def history_parameter_ids(network, interface_name): + return { + id(parameter) + for module in history_components(network, interface_name).values() + for parameter in module.parameters() + } + + +def copy_history_weights(source, destination, interface_name): + """Copy parameters and persistent buffers without adopting source storage or flags.""" + source_modules = history_components(source, interface_name) + destination_modules = history_components(destination, interface_name) + if source_modules.keys() != destination_modules.keys(): + raise ValueError(f"history transfer components differ at {interface_name}") + weights = {} + for path, module in destination_modules.items(): + incoming = source_modules[path].state_dict() + expected = module.state_dict() + if incoming.keys() != expected.keys(): + raise ValueError(f"history transfer tensor keys differ at {path}") + for name, tensor in expected.items(): + if ( + tensor.shape != incoming[name].shape + or tensor.dtype != incoming[name].dtype + ): + raise ValueError(f"history transfer tensor differs at {path}.{name}") + weights[path] = incoming + for path, module in destination_modules.items(): + module.load_state_dict(weights[path], strict=True) + return list(weights) diff --git a/src/sequifier/model/images.py b/src/sequifier/model/images.py index 8f90f2fb0..cbe25b9d8 100644 --- a/src/sequifier/model/images.py +++ b/src/sequifier/model/images.py @@ -6,7 +6,9 @@ import torch from torch import Tensor, nn +from sequifier.io.image_feature_cache import IMAGE_CACHE_PREFIX from sequifier.model.dtypes import cast_floating_to_module_dtype +from sequifier.model.frozen_image_cache import FrozenImageCache, encoder_signature from sequifier.model.ingestions import BaseFeatureIngestion, _AxisPoolBlock @@ -27,6 +29,9 @@ def __init__(self, config: Any, output_dim: int, load_pretrained: bool = True): ) from error self.frozen = config.type == "frozen" + self.microbatch_size = config.microbatch_size + self._feature_cache: FrozenImageCache | None = None + self._cache_signature = None if self.frozen: weights_type = models.get_model_weights(config.model) try: @@ -82,7 +87,13 @@ def train(self, mode: bool = True): self.backbone.eval() return self - def _encode(self, images: Tensor) -> Tensor: + def clear_feature_cache(self): + """Invalidate transient features after an external frozen-weight change.""" + if self._feature_cache is not None: + self._feature_cache = self._feature_cache.replacement() + self._cache_signature = None + + def _backbone_microbatch(self, images: Tensor) -> Tensor: images = images.float() / 255.0 normalized = (images - self.mean) / self.std normalized = normalized.to(dtype=next(self.backbone.parameters()).dtype) @@ -91,23 +102,99 @@ def _encode(self, images: Tensor) -> Tensor: encoded = self.backbone(normalized) else: encoded = self.backbone(normalized) - return self.projection(encoded.to(dtype=self.projection.weight.dtype)) + return encoded + + def _encode_microbatch(self, images: Tensor) -> Tensor: + return self.projection( + self._backbone_microbatch(images).to(dtype=self.projection.weight.dtype) + ) + + def _encode(self, images: Tensor) -> Tensor: + if torch.onnx.is_in_onnx_export(): + return self._encode_microbatch(images) + if images.shape[0] == 0: + return self.projection.weight.new_zeros(0, self.output_dim) + # Bound backbone call sizes independently of policy scoring batches. + return torch.cat( + [ + self._encode_microbatch(chunk) + for chunk in images.split(self.microbatch_size) + ] + ) @torch.compiler.disable - def _encode_valid(self, flat: Tensor, valid: Tensor) -> Tensor: + def _encode_valid(self, flat: Tensor, valid: Tensor, cache_ids=None) -> Tensor: """Skip padded frames while preserving their positions in the output.""" indices = valid.reshape(-1).nonzero().flatten() if indices.numel() == 0: return self.projection.weight.new_zeros(flat.shape[0], self.output_dim) - selected = self._encode(flat.index_select(0, indices)) + if self._feature_cache is not None and cache_ids is not None: + signature = encoder_signature(self) + if signature is not None: + if signature != self._cache_signature: + self.clear_feature_cache() + self._cache_signature = signature + return self._encode_cached(flat, indices, cache_ids) + self.clear_feature_cache() + # Gather one microbatch at a time, avoiding a second full image tensor. + selected = torch.cat( + [ + self._encode_microbatch(flat.index_select(0, chunk)) + for chunk in indices.split(self.microbatch_size) + ] + ) output = selected.new_zeros(flat.shape[0], self.output_dim) return output.index_copy(0, indices, selected) - def forward(self, images: Tensor, valid: Tensor | None = None) -> Tensor: + def _encode_cached(self, flat, indices, cache_ids): + cache = self._feature_cache + assert cache is not None + if cache_ids.numel() != flat.shape[0]: + raise ValueError("image cache identities must match the frame axes") + identities = cache_ids.reshape(-1).index_select(0, indices).cpu().tolist() + positions = indices.cpu().tolist() + if any(identity <= 0 for identity in identities): + raise ValueError("valid images require positive transient identities") + shape = tuple(flat.shape[1:]) + keys = [(shape, identity) for identity in identities] + features, missing = {}, {} + for position, key in zip(positions, keys): + if key in features or key in missing: + continue + feature = cache.get(key) + if feature is None: + missing[key] = position + else: + features[key] = feature + items = list(missing.items()) + for start in range(0, len(items), self.microbatch_size): + chunk = items[start : start + self.microbatch_size] + rows = torch.tensor([position for _, position in chunk], device=flat.device) + with torch.inference_mode(False), torch.no_grad(): + encoded = self._backbone_microbatch(flat.index_select(0, rows)) + for (key, _), feature in zip(chunk, encoded): + features[key] = feature + cache.put(key, feature) + # Only the frozen backbone is cached. This projection remains in autograd. + selected = self.projection( + torch.stack([features[key] for key in keys]).to( + dtype=self.projection.weight.dtype + ) + ) + output = selected.new_zeros(flat.shape[0], self.output_dim) + return output.index_copy(0, indices, selected) + + def forward( + self, images: Tensor, valid: Tensor | None = None, cache_ids=None + ) -> Tensor: shape = images.shape[:-3] flat = images.reshape(-1, *images.shape[-3:]) - if self.training and valid is not None and not torch.onnx.is_in_onnx_export(): - return self._encode_valid(flat, valid).reshape(*shape, -1) + if ( + (self.training or self.frozen) + and valid is not None + and not torch.onnx.is_in_onnx_export() + ): + return self._encode_valid(flat, valid, cache_ids).reshape(*shape, -1) return self._encode(flat).reshape(*shape, -1) @@ -117,7 +204,7 @@ def __init__( views: list[str], encoder_config: Any, output_dim: int, - frame_pool: str, + frame_pooling: str, load_pretrained: bool = True, ): super().__init__() @@ -125,7 +212,7 @@ def __init__( self.encoder = ImageEncoder(encoder_config, output_dim, load_pretrained) self.view_embedding = nn.Parameter(torch.zeros(len(views), output_dim)) self.frame_score = ( - nn.Linear(output_dim, 1) if frame_pool == "attention" else None + nn.Linear(output_dim, 1) if frame_pooling == "attention" else None ) self.frame_sum = _AxisPoolBlock( axes=["frame"], mode="sum", active_axes=["frame"] @@ -138,7 +225,9 @@ def forward( for index, view in enumerate(self.views): images = inputs[f"__image:{view}"] valid = metadata[f"__image_mask:{view}"].bool() - encoded = self.encoder(images, valid) + encoded = self.encoder( + images, valid, metadata.get(IMAGE_CACHE_PREFIX + view) + ) frame_score = self.frame_score if frame_score is None: weights = valid.to(encoded.dtype) @@ -189,7 +278,7 @@ def __init__( config.views, config.encoder, config.output_dim, - config.frame_pool, + config.frame_pooling, load_pretrained, ) self.merge = ( diff --git a/src/sequifier/model/ingestion_compiler.py b/src/sequifier/model/ingestion_compiler.py index f1f5780ba..54acd0634 100644 --- a/src/sequifier/model/ingestion_compiler.py +++ b/src/sequifier/model/ingestion_compiler.py @@ -63,7 +63,7 @@ class BuiltIngestion: class IngestionBuildContext: hparams: Any direct_real_dtype_provider: Callable[[], torch.dtype] - device_max_concat_length: int + max_concat_tensors: int add_ingestion_position: bool load_pretrained: bool @@ -301,18 +301,18 @@ def resolve_ingestion_plan(hparams: Any) -> IngestionPlan: model = hparams.model ingestion_spec = model.ingestion architecture = model.backbone.architecture - objective_name = getattr(hparams, "training_objective", None) + objective_name = hparams.training_objective uses_causal_attention = ( get_objective_class(objective_name).uses_causal_attention if objective_name is not None else False ) - transformer_input_width = architecture.dim_model - int( + transformer_input_width = architecture.model_dim - int( architecture.position_encoding.type == "range_concat" ) from sequifier.config.depth_layout import DepthLayoutRegistryModel - layouts: DepthLayoutRegistryModel | None = getattr(hparams, "depth_layouts", None) + layouts: DepthLayoutRegistryModel | None = hparams.depth_layouts if layouts is None: layouts = DepthLayoutRegistryModel() @@ -443,7 +443,9 @@ def _build_embedding_handler( real_columns=list(branch.real_columns), n_classes=context.hparams.n_classes, context_length=context.hparams.window_view.context_length, - embedding_size=None if feature_embedding_dims is not None else branch.width, + total_embedding_dim=None + if feature_embedding_dims is not None + else branch.width, feature_embedding_dims=feature_embedding_dims, add_ingestion_position=context.add_ingestion_position, dropout=branch.config.dropout, @@ -453,7 +455,7 @@ def _build_embedding_handler( if column in context.hparams.categorical_hashing }, embedding_dim=branch.width, - device_max_concat_length=context.device_max_concat_length, + max_concat_tensors=context.max_concat_tensors, ) @@ -471,7 +473,7 @@ def _build_passthrough_module( dropout=branch.config.dropout, projection_dim=projection_dim, direct_real_dtype_provider=context.direct_real_dtype_provider, - device_max_concat_length=context.device_max_concat_length, + max_concat_tensors=context.max_concat_tensors, ) @@ -503,7 +505,7 @@ def _build_temporal_conv_handler( causal=branch.config.causal, activation=branch.config.activation, dropout=branch.config.dropout, - post_conv_norm=branch.config.post_conv_norm, + post_conv_normalization=branch.config.post_conv_normalization, orientation=branch.config.orientation, context_length=context.hparams.window_view.context_length, ) @@ -554,7 +556,7 @@ def _build_structured_handler( processing_blocks=branch.config.processing_blocks, image_cells=branch.config.image_cells, image_encoder_config=branch.config.image_encoder, - image_frame_pool=branch.config.image_frame_pool, + image_frame_pooling=branch.config.image_frame_pooling, load_pretrained=context.load_pretrained, **_common_branch_kwargs(branch, context), ) @@ -581,7 +583,7 @@ def _build_image_handler(branch, context): def _depth_columns(hparams, config, allow_default_columns): from sequifier.config.depth_layout import DepthLayoutRegistryModel - layouts: DepthLayoutRegistryModel | None = getattr(hparams, "depth_layouts", None) + layouts: DepthLayoutRegistryModel | None = hparams.depth_layouts if layouts is None: layouts = DepthLayoutRegistryModel() if config.layout not in layouts: @@ -600,7 +602,7 @@ def _validate_depth(hparams, usage, columns, config, layout): ) if config.feature_embedding_dims is None: get_feature_embedding_dims( - config.architecture.dim_model, + config.architecture.model_dim, categorical, real, hparams.n_classes, @@ -669,17 +671,17 @@ def compile_feature_ingestion( *, hparams: Any, direct_real_dtype_provider: Callable[[], torch.dtype], - device_max_concat_length: int, + max_concat_tensors: int, load_pretrained: bool = True, ) -> BuiltIngestion: plan = resolve_ingestion_plan(hparams) context = IngestionBuildContext( hparams=hparams, direct_real_dtype_provider=direct_real_dtype_provider, - device_max_concat_length=device_max_concat_length, + max_concat_tensors=max_concat_tensors, add_ingestion_position=( hparams.model.backbone.architecture.position_encoding.type == "learned" - and hparams.model.backbone.architecture.positional_encoding_scope + and hparams.model.backbone.architecture.position_encoding_scope == "per_feature" ), load_pretrained=load_pretrained, @@ -710,10 +712,8 @@ def build(branch, inherited_seed=None, path=()): ) else: module = INGESTION_HANDLERS[branch.config.type].build(branch, context) - module._sequifier_initialization_config = branch.config - module._sequifier_initialization_seed = derived_seed( - inherited_seed, path, "initializer" - ) + module.initialization_config = branch.config + module.initialization_seed = derived_seed(inherited_seed, path, "initializer") return module built_branches = { @@ -737,8 +737,8 @@ def build(branch, inherited_seed=None, path=()): merge_dim=plan.transformer_input_width, merge_softmax=cast(str, plan.merge_softmax), ) - module._sequifier_initialization_config = root_config - module._sequifier_initialization_seed = derived_seed(root_seed, (), "initializer") + module.initialization_config = root_config + module.initialization_seed = derived_seed(root_seed, (), "initializer") return BuiltIngestion(module, plan.transformer_input_width) @@ -747,11 +747,11 @@ def build_feature_ingestion( *, hparams: Any, direct_real_dtype_provider: Callable[[], torch.dtype], - device_max_concat_length: int, + max_concat_tensors: int, ) -> BaseFeatureIngestion: """Compatibility wrapper returning only the compiled runtime module.""" return compile_feature_ingestion( hparams=hparams, direct_real_dtype_provider=direct_real_dtype_provider, - device_max_concat_length=device_max_concat_length, + max_concat_tensors=max_concat_tensors, ).module diff --git a/src/sequifier/model/ingestions.py b/src/sequifier/model/ingestions.py index f6a194a84..4dd61d868 100644 --- a/src/sequifier/model/ingestions.py +++ b/src/sequifier/model/ingestions.py @@ -1,5 +1,5 @@ import math -from collections.abc import Callable +from collections.abc import Callable, Iterable from dataclasses import dataclass from itertools import product from typing import Any, Optional, cast @@ -10,6 +10,8 @@ from torch.nn import ModuleDict from sequifier.accurate_softmax import accurate_softmax +from sequifier.config.components import IngestionComponentBase +from sequifier.config.identifiers import validate_module_dict_key from sequifier.model.dtypes import ( cast_floating_to_dtype, cast_floating_to_module_dtype, @@ -48,21 +50,9 @@ def embedding_safe_indices(indices: Tensor) -> Tensor: return indices.to(dtype=target_dtype) -@beartype -def _validate_module_dict_key(key: str, usage: str) -> None: - if key == "": - raise ValueError(f"{usage} cannot be empty") - if "." in key: - raise ValueError(f"{usage} cannot contain '.'") - from torch import nn - - if hasattr(nn.ModuleDict(), key): - raise ValueError(f"{usage} collides with a ModuleDict attribute") - - @beartype def get_feature_embedding_dims( - embedding_size: int, + total_embedding_dim: int, categorical_columns: list[str], real_columns: list[str], n_classes: Optional[dict[str, int]] = None, @@ -72,9 +62,9 @@ def get_feature_embedding_dims( raise ValueError("No columns found") if len(categorical_columns) == 0 and len(real_columns) > 0: - if embedding_size < len(real_columns): + if total_embedding_dim < len(real_columns): raise ValueError( - f"embedding_size ({embedding_size}) is smaller than the " + f"total_embedding_dim ({total_embedding_dim}) is smaller than the " f"number of real input columns ({len(real_columns)}). " "Cannot allocate at least 1 dimension per column." ) @@ -82,22 +72,22 @@ def get_feature_embedding_dims( feature_embedding_dims = {col: 1 for col in real_columns} column_index = dict(enumerate(real_columns)) - remaining_dims = embedding_size - len(real_columns) + remaining_dims = total_embedding_dim - len(real_columns) for i in range(remaining_dims): j = i % len(real_columns) feature_embedding_dims[column_index[j]] += 1 - if sum(feature_embedding_dims.values()) != embedding_size: + if sum(feature_embedding_dims.values()) != total_embedding_dim: raise ValueError( "Auto-calculated embedding dimensions " f"({sum(feature_embedding_dims.values())}) do not sum to " - f"embedding_size ({embedding_size})." + f"total_embedding_dim ({total_embedding_dim})." ) elif len(real_columns) == 0 and len(categorical_columns) > 0: minimum_size = 2 * len(categorical_columns) - if embedding_size < minimum_size: + if total_embedding_dim < minimum_size: raise ValueError( - f"embedding_size ({embedding_size}) is smaller than the " + f"total_embedding_dim ({total_embedding_dim}) is smaller than the " f"minimum of 2 dimensions per categorical column ({minimum_size})." ) if n_classes is None: @@ -116,7 +106,7 @@ def get_feature_embedding_dims( weights = [math.log(cardinalities[col]) for col in categorical_columns] if not any(weights): weights = [1.0] * len(categorical_columns) - remaining = embedding_size - minimum_size + remaining = total_embedding_dim - minimum_size weight_total = sum(weights) quotas = [remaining * weight / weight_total for weight in weights] extra_dims = [math.floor(quota) for quota in quotas] @@ -141,6 +131,12 @@ def get_feature_embedding_dims( class BaseFeatureIngestion(nn.Module): + initialization_config: IngestionComponentBase | None = None + initialization_seed: int | None = None + + def initialization_children(self) -> Iterable["BaseFeatureIngestion"]: + return () + @conditional_beartype def forward(self, src: dict[str, Tensor], metadata: dict[str, Tensor]) -> Tensor: raise NotImplementedError @@ -151,6 +147,10 @@ def _scale_embedding(x: Tensor, embedding_dim: int) -> Tensor: return x * math.sqrt(embedding_dim) +class IngestionOutputProjection(nn.Linear): + """Adapt an ingestion representation to its consumer's width.""" + + class RealFeatureProjection(nn.Linear): """Project one standardized real-valued feature into an embedding space.""" @@ -184,13 +184,23 @@ def __init__( self.register_buffer( "multipliers", torch.randint( - 1, HASH_PRIME, (num_hashes,), generator=generator, dtype=torch.int64 + 1, + HASH_PRIME, + (num_hashes,), + generator=generator, + dtype=torch.int64, + device="cpu", ), ) self.register_buffer( "offsets", torch.randint( - 0, HASH_PRIME, (num_hashes,), generator=generator, dtype=torch.int64 + 0, + HASH_PRIME, + (num_hashes,), + generator=generator, + dtype=torch.int64, + device="cpu", ), ) @@ -239,13 +249,13 @@ def __init__( real_columns: list[str], n_classes: dict[str, int], context_length: int, - embedding_size: Optional[int], + total_embedding_dim: Optional[int], feature_embedding_dims: Optional[dict[str, int]], add_ingestion_position: bool, dropout: float, hashing: Optional[dict[str, Any]] = None, embedding_dim: Optional[int] = None, - device_max_concat_length: int = 12, + max_concat_tensors: int = 12, ): super().__init__() self.categorical_columns = categorical_columns @@ -255,18 +265,18 @@ def __init__( self.add_ingestion_position = add_ingestion_position self.drop = nn.Dropout(dropout) self.hashing = hashing or {} - self.device_max_concat_length = device_max_concat_length + self.max_concat_tensors = max_concat_tensors if feature_embedding_dims is not None: self.feature_embedding_dims = feature_embedding_dims else: - if embedding_size is None: + if total_embedding_dim is None: raise ValueError( "embedding ingestion requires an embedding dimension when " "feature_embedding_dims is not configured" ) self.feature_embedding_dims = get_feature_embedding_dims( - embedding_size, + total_embedding_dim, categorical_columns, real_columns, n_classes, @@ -274,7 +284,7 @@ def __init__( ) self.input_dim = sum(self.feature_embedding_dims.values()) - self.embedding_size = self.input_dim + self.total_embedding_dim = self.input_dim self.embedding_dim = embedding_dim or self.input_dim self.encoder = ModuleDict() @@ -324,14 +334,12 @@ def __init__( @conditional_beartype def _recursive_concat(self, srcs: list[Tensor]) -> Tensor: - if len(srcs) <= self.device_max_concat_length: + if len(srcs) <= self.max_concat_tensors: return torch.cat(srcs, 2) srcs_inner = [] - for start in range(0, len(srcs), self.device_max_concat_length): - src = self._recursive_concat( - srcs[start : start + self.device_max_concat_length] - ) + for start in range(0, len(srcs), self.max_concat_tensors): + src = self._recursive_concat(srcs[start : start + self.max_concat_tensors]) srcs_inner.append(src) return self._recursive_concat(srcs_inner) @@ -386,7 +394,7 @@ def __init__( dropout: float, projection_dim: Optional[int] = None, direct_real_dtype_provider: Optional[Callable[[], torch.dtype]] = None, - device_max_concat_length: int = 12, + max_concat_tensors: int = 12, ): super().__init__() if not real_columns: @@ -398,10 +406,10 @@ def __init__( self.add_ingestion_position = add_ingestion_position self.drop = nn.Dropout(dropout) self.input_dim = len(real_columns) - self.embedding_size = self.input_dim + self.total_embedding_dim = self.input_dim self.projection_dim = projection_dim or self.input_dim self.direct_real_dtype_provider = direct_real_dtype_provider - self.device_max_concat_length = device_max_concat_length + self.max_concat_tensors = max_concat_tensors if self.add_ingestion_position: self.pos_encoder = ModuleDict() @@ -419,14 +427,12 @@ def __init__( @conditional_beartype def _recursive_concat(self, srcs: list[Tensor]) -> Tensor: - if len(srcs) <= self.device_max_concat_length: + if len(srcs) <= self.max_concat_tensors: return torch.cat(srcs, 2) srcs_inner = [] - for start in range(0, len(srcs), self.device_max_concat_length): - src = self._recursive_concat( - srcs[start : start + self.device_max_concat_length] - ) + for start in range(0, len(srcs), self.max_concat_tensors): + src = self._recursive_concat(srcs[start : start + self.max_concat_tensors]) srcs_inner.append(src) return self._recursive_concat(srcs_inner) @@ -482,7 +488,7 @@ def __init__( causal: bool, activation: str, dropout: float, - post_conv_norm: str, + post_conv_normalization: str, orientation: str, context_length: int, ): @@ -515,7 +521,7 @@ def __init__( norm_dim = ( context_length if self.orientation == "within_column" else self.channels ) - self.post_conv_norm = self._norm(post_conv_norm, norm_dim) + self.post_conv_normalization = self._norm(post_conv_normalization, norm_dim) @staticmethod @conditional_beartype @@ -537,19 +543,19 @@ def _norm(name: str, output_dim: int) -> nn.Module: return RMSNorm(output_dim) if name == "none": return nn.Identity() - raise ValueError(f"Unknown temporal_conv post_conv_norm: {name}") + raise ValueError(f"Unknown temporal_conv post_conv_normalization: {name}") @conditional_beartype def _apply_post_conv_norm(self, output: Tensor) -> Tensor: if self.orientation == "within_column": output = output.transpose(1, 2) - output = self.post_conv_norm( - cast_floating_to_module_dtype(output, self.post_conv_norm) + output = self.post_conv_normalization( + cast_floating_to_module_dtype(output, self.post_conv_normalization) ) return output.transpose(1, 2) if self.orientation == "within_item_position": - return self.post_conv_norm( - cast_floating_to_module_dtype(output, self.post_conv_norm) + return self.post_conv_normalization( + cast_floating_to_module_dtype(output, self.post_conv_normalization) ) raise ValueError( f"Unknown temporal_conv normalization orientation: {self.orientation}" @@ -675,7 +681,7 @@ def __init__( categorical_set = set(categorical_columns) real_set = set(real_columns) for group_name, group_columns in self.groups.items(): - _validate_module_dict_key( + validate_module_dict_key( group_name, f"grouped ingestion group {group_name!r}" ) self.group_ingestions[group_name] = FeaturePoolFeatureIngestion( @@ -1034,7 +1040,7 @@ def __init__( *, input_dim: int, output_dim: int, - n_heads: int, + num_heads: int, dropout: float, ): super().__init__() @@ -1045,7 +1051,7 @@ def __init__( ) self.attention = nn.MultiheadAttention( embed_dim=output_dim, - num_heads=n_heads, + num_heads=num_heads, dropout=dropout, batch_first=True, ) @@ -1069,7 +1075,7 @@ def __init__( axes: list[str], unshared_axes: list[str], output_dim: int, - n_heads: int, + num_heads: int, dropout: float, active_axes: list[str], axis_sizes: dict[str, int], @@ -1079,7 +1085,7 @@ def __init__( self.axes = axes self.unshared_axes = unshared_axes self.output_dim = output_dim - self.n_heads = n_heads + self.num_heads = num_heads self.active_axes = active_axes self.output_axes = list(active_axes) self.axis_sizes = axis_sizes @@ -1093,7 +1099,7 @@ def __init__( _module_key(indices): _AxisAttentionLayer( input_dim=self.input_dim, output_dim=self.output_dim, - n_heads=self.n_heads, + num_heads=self.num_heads, dropout=dropout, ) for indices in self.unshared_indices @@ -1301,7 +1307,7 @@ def _build_axis_attention( axes=block.axes, unshared_axes=block.unshared_axes, output_dim=block.output_dim, - n_heads=block.n_heads, + num_heads=block.num_heads, dropout=block.dropout, active_axes=list(shape.active_axes), axis_sizes=axis_sizes, @@ -1315,7 +1321,7 @@ def _build_axis_pool( ) -> nn.Module: return _AxisPoolBlock( axes=block.axes, - mode=block.mode, + mode=block.pooling, active_axes=list(shape.active_axes), ) @@ -1352,7 +1358,7 @@ def __init__( processing_blocks: Optional[list[Any]] = None, image_cells: Optional[dict[str, str]] = None, image_encoder_config: Optional[Any] = None, - image_frame_pool: str = "mean", + image_frame_pooling: str = "mean", load_pretrained: bool = True, ): self.layout = layout @@ -1399,7 +1405,7 @@ def __init__( list(dict.fromkeys(self.image_cells.values())), image_encoder_config, self.cell_dim, - image_frame_pool, + image_frame_pooling, load_pretrained, ) if self.add_ingestion_position: @@ -1641,7 +1647,7 @@ class CompositeFeatureIngestion(BaseFeatureIngestion): def __init__( self, *, - branches: dict[str, nn.Module], + branches: dict[str, BaseFeatureIngestion], branch_widths: dict[str, int], merge_type: str, merge_dim: int, @@ -1649,7 +1655,7 @@ def __init__( ): super().__init__() for branch_name in branches: - _validate_module_dict_key( + validate_module_dict_key( branch_name, f"Composite ingestion branch {branch_name!r}" ) self.branches = ModuleDict(branches) @@ -1660,6 +1666,9 @@ def __init__( merge_softmax, ) + def initialization_children(self) -> Iterable[BaseFeatureIngestion]: + return cast(Iterable[BaseFeatureIngestion], self.branches.values()) + @conditional_beartype def forward(self, src: dict[str, Tensor], metadata: dict[str, Tensor]) -> Tensor: branch_outputs = { @@ -1709,7 +1718,7 @@ def compile_feature_ingestion( *, hparams: Any, direct_real_dtype_provider: Callable[[], torch.dtype], - device_max_concat_length: int, + max_concat_tensors: int, load_pretrained: bool = True, ): from sequifier.model.ingestion_compiler import compile_feature_ingestion as compile @@ -1717,7 +1726,7 @@ def compile_feature_ingestion( return compile( hparams=hparams, direct_real_dtype_provider=direct_real_dtype_provider, - device_max_concat_length=device_max_concat_length, + max_concat_tensors=max_concat_tensors, load_pretrained=load_pretrained, ) @@ -1727,11 +1736,11 @@ def build_feature_ingestion( *, hparams: Any, direct_real_dtype_provider: Callable[[], torch.dtype], - device_max_concat_length: int, + max_concat_tensors: int, ) -> BaseFeatureIngestion: """Compatibility wrapper returning only the compiled runtime module.""" return compile_feature_ingestion( hparams=hparams, direct_real_dtype_provider=direct_real_dtype_provider, - device_max_concat_length=device_max_concat_length, + max_concat_tensors=max_concat_tensors, ).module diff --git a/src/sequifier/model/initialization.py b/src/sequifier/model/initialization.py index 955791b68..f971498a3 100644 --- a/src/sequifier/model/initialization.py +++ b/src/sequifier/model/initialization.py @@ -15,12 +15,19 @@ ModelInitializationConfig, ParameterKind, ) -from sequifier.model.decoders import TargetDecoderBranch, TargetDecoding from sequifier.model.ingestions import ( + BaseFeatureIngestion, RealFeatureProjection, TemporalConvFeatureIngestion, ) from sequifier.model.layers import FeedForward, RMSNorm, SelfAttention +from sequifier.model.module_roles import ( + decoder_module, + decoder_output_ids, + ingestion_projection_ids, + position_embedding_ids, + range_position_projection, +) from sequifier.typechecking import beartype EMBEDDING_INIT_STD = 0.02 @@ -59,16 +66,9 @@ def __init__( self.initialization = initialization or ModelInitializationConfig() self.initialized_parameter_ids: set[int] = set() self.matched_override_targets: set[InitializationTarget] = set() - self.position_embedding_module_ids = self._position_embedding_module_ids() - self.ingestion_projection_module_ids = self._ingestion_projection_module_ids() - self.decoder_output_module_ids = self._decoder_output_module_ids() - - @beartype - def _decoder_module(self) -> Optional[nn.Module]: - if isinstance(self.model, (TargetDecoding, TargetDecoderBranch)): - return self.model - decoder = getattr(self.model, "decoder", None) - return decoder if isinstance(decoder, nn.Module) else None + self.position_embedding_module_ids = position_embedding_ids(model) + self.ingestion_projection_module_ids = ingestion_projection_ids(model) + self.decoder_output_module_ids = decoder_output_ids(decoder_module(model)) @beartype def initialize(self) -> None: @@ -82,79 +82,6 @@ def initialize(self) -> None: self._initialize_remaining_modules() self._initialize_free_parameters() - @beartype - def _position_embedding_module_ids(self) -> set[int]: - position_embeddings: set[int] = set() - for module in self.model.modules(): - for attribute in ( - "pos_encoder", - "position_embedding", - "global_position_encoder", - "axis_embedding_layers", - ): - value = getattr(module, attribute, None) - if not isinstance(value, nn.Module): - continue - value_module = cast(nn.Module, value) - position_embeddings.update( - id(child) - for child in value_module.modules() - if isinstance(child, nn.Embedding) - ) - return position_embeddings - - @beartype - def _ingestion_projection_module_ids(self) -> set[int]: - projections: set[int] = set() - for module in self.model.modules(): - if ( - isinstance(module, nn.Linear) - and getattr(module, "_sequifier_layer_group", None) - == "ingestion.output_projection" - ): - projections.add(id(module)) - for attribute in ( - "output_projection_layer", - "input_projection", - "concat_projection", - "branch_projections", - ): - value = getattr(module, attribute, None) - if not isinstance(value, nn.Module): - continue - value_module = cast(nn.Module, value) - projections.update( - id(child) - for child in value_module.modules() - if isinstance(child, nn.Linear) - ) - return projections - - @beartype - def _decoder_output_module_ids(self) -> set[int]: - decoder = self._decoder_module() - if decoder is None: - return set() - marked = { - id(module) - for module in decoder.modules() - if isinstance(module, nn.Linear) - and getattr(module, "_sequifier_decoder_output", False) - } - output_module_ids = { - id(output_layer) - for module in decoder.modules() - if isinstance(module, TargetDecoderBranch) - for output_layer in module.output_layers.values() - if isinstance(output_layer, nn.Linear) - } - if output_module_ids or marked: - return output_module_ids | marked - # Compatibility with the original per-target ModuleDict decoder. - return { - id(module) for module in decoder.modules() if isinstance(module, nn.Linear) - } - @beartype def _is_initialized(self, parameter: Optional[Tensor]) -> bool: return parameter is None or id(parameter) in self.initialized_parameter_ids @@ -230,18 +157,13 @@ def _apply_method( elif method == "uniform": for parameter in tensors: nn.init.uniform_(parameter, a=config.low, b=config.high) - elif method in { - "xavier", - "glorot", - "xavier_uniform", - "glorot_uniform", - }: + elif method == "xavier_uniform": if config.fan_mode == "joint": self._joint_xavier_uniform_raw(tensors, gain=config.gain) else: for parameter in tensors: nn.init.xavier_uniform_(parameter, gain=config.gain) - elif method in {"xavier_normal", "glorot_normal"}: + elif method == "xavier_normal": if config.fan_mode == "joint": self._joint_xavier_normal_raw(tensors, gain=config.gain) else: @@ -445,7 +367,7 @@ def _initialize_feed_forward(self, feed_forward: FeedForward) -> None: @beartype def _initialize_decoder(self) -> None: - decoder = self._decoder_module() + decoder = decoder_module(self.model) if decoder is None: return @@ -461,32 +383,31 @@ def _initialize_decoder(self) -> None: @beartype def _initialize_range_position_projection(self) -> None: - projection = getattr(self.model, "range_position_projection", None) - if projection is None: - projection = getattr(self.model, "range_projection", None) + projection = range_position_projection(self.model) if projection is None: return if not isinstance(projection, nn.Linear): raise TypeError("range_position_projection must be an nn.Linear") - if projection.out_features + 1 != projection.in_features: + linear_projection = cast(nn.Linear, projection) + if linear_projection.out_features + 1 != linear_projection.in_features: raise ValueError( - "range_position_projection must map dim_model + 1 to dim_model" + "range_position_projection must map model_dim + 1 to model_dim" ) if not self._apply_override( - "position.range_projection", "weight", [projection.weight] + "position.range_projection", "weight", [linear_projection.weight] ): - if not self._is_initialized(projection.weight): + if not self._is_initialized(linear_projection.weight): self._identity_plus_normal_raw( - projection.weight, + linear_projection.weight, mean=0.0, std=EMBEDDING_INIT_STD, ) - self._mark_initialized(projection.weight) + self._mark_initialized(linear_projection.weight) if not self._apply_override( - "position.range_projection", "bias", [projection.bias] + "position.range_projection", "bias", [linear_projection.bias] ): - self._zero_(projection.bias) + self._zero_(linear_projection.bias) @beartype def _initialize_real_feature_projections(self) -> None: @@ -569,12 +490,12 @@ def _initialize_remaining_modules(self) -> None: nn.BatchNorm3d, ), ): - weight = getattr(module, "weight", None) + weight = module.weight if not self._apply_override("normalization", "weight", [weight]): if isinstance(weight, Tensor) and not self._is_initialized(weight): nn.init.ones_(weight) self._mark_initialized(weight) - bias = getattr(module, "bias", None) + bias = module.bias if not self._apply_override("normalization", "bias", [bias]): self._zero_(bias) elif isinstance(module, nn.Linear): @@ -627,16 +548,16 @@ def initialize_model_weights( return initializer.matched_override_targets -def initialize_owned_ingestion(model, initialization, *, warn_unmatched=True): - from sequifier.config.initialization_config import ModelInitializationConfig +def initialize_owned_ingestion( + model: BaseFeatureIngestion, initialization, *, warn_unmatched=True +): from sequifier.model.initialization_seed import isolated_initialization def active(node): - config = getattr(node, "_sequifier_initialization_config", None) - if getattr(node, "_sequifier_initialization_seed", None) is not None: + if node.initialization_seed is not None: return True - for child in getattr(node, "branches", {}).values(): - child_config = getattr(child, "_sequifier_initialization_config", config) + for child in node.initialization_children(): + child_config = child.initialization_config if ( child_config is not None and child_config.initialization.configured_targets() @@ -653,7 +574,7 @@ def active(node): requests = {} def visit(node, inherited): - config = getattr(node, "_sequifier_initialization_config", None) + config = node.initialization_config values = inherited.model_dump(mode="python", exclude_none=True) if config is not None: for group, targets in config.initialization.model_dump( @@ -661,12 +582,12 @@ def visit(node, inherited): ).items(): values[group] = {**values.get(group, {}), **targets} effective = ModelInitializationConfig.model_validate(values) - children = getattr(node, "branches", {}) - excluded = {id(p) for child in children.values() for p in child.parameters()} + branches = node.initialization_children() + excluded = {id(p) for child in branches for p in child.parameters()} owned = {id(p) for p in node.parameters()} - excluded request = ( effective.model_dump(mode="python"), - getattr(node, "_sequifier_initialization_seed", None), + node.initialization_seed, ) for identity in owned: if identity in requests and requests[identity] != request: @@ -679,9 +600,12 @@ def visit(node, inherited): groups = semantic_parameter_groups(node) biases = { - id(module.bias) + id(parameter) for module in node.modules() - if isinstance(getattr(module, "bias", None), Tensor) + for name, parameter in module.named_parameters( + recurse=False, remove_duplicate=False + ) + if name == "bias" } unmatched = { (group, kind) @@ -705,13 +629,11 @@ def visit(node, inherited): if not parameter.requires_grad } ) - with isolated_initialization( - getattr(node, "_sequifier_initialization_seed", None) - ): + with isolated_initialization(node.initialization_seed): initializer.initialize() visited.update(id(p) for p in node.parameters() if id(p) not in excluded) all_matched.update(initializer.matched_override_targets) - for child in children.values(): + for child in branches: visit(child, effective) visit(model, initialization) diff --git a/src/sequifier/model/initialization_seed.py b/src/sequifier/model/initialization_seed.py index 8a2bff91e..bbe2f2f52 100644 --- a/src/sequifier/model/initialization_seed.py +++ b/src/sequifier/model/initialization_seed.py @@ -1,4 +1,4 @@ -"""Version 1 stable initialization namespaces, independent of runtime dropout.""" +"""Stable initialization namespaces, independent of runtime dropout.""" import hashlib import json @@ -8,14 +8,15 @@ import numpy as np import torch -SEED_DERIVATION_VERSION = 1 +from sequifier.contract_versions import ContractVersions def derived_seed(seed: int | None, path: tuple[str, ...], phase: str) -> int | None: if seed is None: return None value = json.dumps( - [SEED_DERIVATION_VERSION, seed, path, phase], separators=(",", ":") + [ContractVersions.INITIALIZATION_SEED_DERIVATION, seed, path, phase], + separators=(",", ":"), ) return int.from_bytes(hashlib.sha256(value.encode()).digest()[:8], "big") % (2**63) diff --git a/src/sequifier/model/layers.py b/src/sequifier/model/layers.py index 74b8e583a..4c82d7352 100644 --- a/src/sequifier/model/layers.py +++ b/src/sequifier/model/layers.py @@ -15,6 +15,7 @@ def __init__(self, dim: int, eps: float = 1e-6): super().__init__() self.eps = eps self.weight = nn.Parameter(torch.ones(dim)) + self.bias = None @conditional_beartype def forward(self, x): @@ -74,7 +75,7 @@ def apply_rotary_pos_emb(q, k, cos, sin): class FeedForward(nn.Module): @beartype - def __init__(self, dim_model, dim_feedforward, activation, dropout): + def __init__(self, model_dim, dim_feedforward, activation, dropout): super().__init__() self.activation = activation @@ -82,13 +83,15 @@ def __init__(self, dim_model, dim_feedforward, activation, dropout): # SwiGLU requires 2 gates, so we often adjust dim_feedforward or keep it # but implement the GLU split. Commonly SwiGLU hidden dim is 2/3 of standard. # Here we strictly follow config dim_feedforward. - self.w1 = nn.Linear(dim_model, dim_feedforward) - self.w2 = nn.Linear(dim_model, dim_feedforward) # Gate - self.w3 = nn.Linear(dim_feedforward, dim_model) # Output + self.w1 = nn.Linear(model_dim, dim_feedforward) + self.w2 = nn.Linear(model_dim, dim_feedforward) # Gate + self.w3 = nn.Linear(dim_feedforward, model_dim) # Output else: - self.linear1 = nn.Linear(dim_model, dim_feedforward) - self.linear2 = nn.Linear(dim_feedforward, dim_model) - self.act = nn.GELU() if activation == "gelu" else nn.ReLU() + self.linear1 = nn.Linear(model_dim, dim_feedforward) + self.linear2 = nn.Linear(dim_feedforward, model_dim) + self.act = {"gelu": nn.GELU, "relu": nn.ReLU, "softplus": nn.Softplus}[ + activation + ]() self.dropout = nn.Dropout(dropout) @@ -146,9 +149,9 @@ class SelfAttention(nn.Module): @beartype def __init__( self, - dim_model, - n_heads, - n_kv_heads, + model_dim, + num_heads, + num_kv_heads, attention_type, dropout, context_length, @@ -157,18 +160,19 @@ def __init__( output_projection=True, ): super().__init__() - self.n_heads = n_heads - self.n_kv_heads = n_kv_heads if n_kv_heads is not None else n_heads - self.dim_model = dim_model - self.head_dim = dim_model // n_heads + self.export_attention = False + self.num_heads = num_heads + self.num_kv_heads = num_kv_heads if num_kv_heads is not None else num_heads + self.model_dim = model_dim + self.head_dim = model_dim // num_heads self.attention_type = attention_type self.use_rope = use_rope - self.wq = nn.Linear(dim_model, n_heads * self.head_dim, bias=False) - self.wk = nn.Linear(dim_model, self.n_kv_heads * self.head_dim, bias=False) - self.wv = nn.Linear(dim_model, self.n_kv_heads * self.head_dim, bias=False) + self.wq = nn.Linear(model_dim, num_heads * self.head_dim, bias=False) + self.wk = nn.Linear(model_dim, self.num_kv_heads * self.head_dim, bias=False) + self.wv = nn.Linear(model_dim, self.num_kv_heads * self.head_dim, bias=False) self.wo = ( - nn.Linear(n_heads * self.head_dim, dim_model, bias=False) + nn.Linear(num_heads * self.head_dim, model_dim, bias=False) if output_projection else nn.Identity() ) @@ -200,17 +204,17 @@ def forward( xq = ( self.wq(xq_input) - .view(batch_size, seq_len, self.n_heads, self.head_dim) + .view(batch_size, seq_len, self.num_heads, self.head_dim) .transpose(1, 2) ) xk = ( self.wk(xk_input) - .view(batch_size, seq_len, self.n_kv_heads, self.head_dim) + .view(batch_size, seq_len, self.num_kv_heads, self.head_dim) .transpose(1, 2) ) xv = ( self.wv(xv_input) - .view(batch_size, seq_len, self.n_kv_heads, self.head_dim) + .view(batch_size, seq_len, self.num_kv_heads, self.head_dim) .transpose(1, 2) ) @@ -219,8 +223,8 @@ def forward( xq, xk = apply_rotary_pos_emb(xq, xk, cos, sin) # Handle GQA/MQA by repeating keys/values - if self.n_kv_heads != self.n_heads: - n_rep = self.n_heads // self.n_kv_heads + if self.num_kv_heads != self.num_heads: + n_rep = self.num_heads // self.num_kv_heads xk = xk.repeat_interleave(n_rep, dim=1) xv = xv.repeat_interleave(n_rep, dim=1) @@ -238,7 +242,7 @@ def forward( analysis_requested = trace is not None and any( trace.requires(f"{site_prefix}.{suffix}") for suffix in analysis_sites ) - if getattr(self, "_sequifier_export_attention", False): + if self.export_attention: # Explicit export lowering retains attention-weight dropout independently # of residual and feed-forward dropout. Eager SDPA is unchanged. scores = torch.matmul(xq, xk.transpose(-2, -1)) / math.sqrt(self.head_dim) @@ -291,7 +295,7 @@ def forward( f"{site_prefix}.update", output, axes=("batch", "time", "channel"), - width=self.dim_model, + width=self.model_dim, ) return output @@ -300,22 +304,22 @@ class SequifierEncoderLayer(nn.Module): @beartype def __init__(self, architecture, max_context_length=None): super().__init__() - dim_model = architecture.dim_model - self.dim_model = dim_model + model_dim = architecture.model_dim + self.model_dim = model_dim self.norm_first = architecture.normalization.norm_first # Normalization normalization_type = architecture.normalization.type NormClass = RMSNorm if normalization_type == "rmsnorm" else nn.LayerNorm norm_eps = 1e-6 if normalization_type == "rmsnorm" else 1e-3 - self.norm1 = NormClass(dim_model, eps=norm_eps) - self.norm2 = NormClass(dim_model, eps=norm_eps) + self.norm1 = NormClass(model_dim, eps=norm_eps) + self.norm2 = NormClass(model_dim, eps=norm_eps) # Attention self.attn = SelfAttention( - dim_model=dim_model, - n_heads=architecture.attention.n_heads, - n_kv_heads=architecture.attention.n_kv_heads, + model_dim=model_dim, + num_heads=architecture.attention.num_heads, + num_kv_heads=architecture.attention.num_kv_heads, attention_type=architecture.attention.type, dropout=architecture.dropout, context_length=( @@ -330,8 +334,8 @@ def __init__(self, architecture, max_context_length=None): # Feed Forward self.ff = FeedForward( - dim_model, - architecture.feed_forward.dim, + model_dim, + architecture.feed_forward.hidden_dim, architecture.feed_forward.activation, architecture.dropout, ) @@ -365,7 +369,7 @@ def forward( f"{site_prefix}.attention.norm_input", normed_src, axes=("batch", "time", "channel"), - width=self.dim_model, + width=self.model_dim, ) attention_update = self.dropout( self.attn( @@ -384,7 +388,7 @@ def forward( f"{site_prefix}.attention.output", x, axes=("batch", "time", "channel"), - width=self.dim_model, + width=self.model_dim, ) normed_x = self.norm2(cast_floating_to_module_dtype(x, self.norm2)) if trace is not None: @@ -392,7 +396,7 @@ def forward( f"{site_prefix}.mlp.norm_input", normed_x, axes=("batch", "time", "channel"), - width=self.dim_model, + width=self.model_dim, ) mlp_update = self.dropout( self.ff(normed_x, trace=trace, site_prefix=f"{site_prefix}.mlp") @@ -402,7 +406,7 @@ def forward( f"{site_prefix}.mlp.update", mlp_update, axes=("batch", "time", "channel"), - width=self.dim_model, + width=self.model_dim, ) x = self._residual_add(x, mlp_update) else: @@ -412,7 +416,7 @@ def forward( f"{site_prefix}.attention.norm_input", attention_input, axes=("batch", "time", "channel"), - width=self.dim_model, + width=self.model_dim, ) x = self._residual_add( src, @@ -431,13 +435,13 @@ def forward( f"{site_prefix}.attention.output", x, axes=("batch", "time", "channel"), - width=self.dim_model, + width=self.model_dim, ) x = trace.emit( f"{site_prefix}.mlp.norm_input", x, axes=("batch", "time", "channel"), - width=self.dim_model, + width=self.model_dim, ) mlp_update = self.dropout( self.ff(x, trace=trace, site_prefix=f"{site_prefix}.mlp") @@ -447,7 +451,7 @@ def forward( f"{site_prefix}.mlp.update", mlp_update, axes=("batch", "time", "channel"), - width=self.dim_model, + width=self.model_dim, ) x = self._residual_add(x, mlp_update) x = self.norm2(cast_floating_to_module_dtype(x, self.norm2)) diff --git a/src/sequifier/model/module_roles.py b/src/sequifier/model/module_roles.py new file mode 100644 index 000000000..d1a630ad1 --- /dev/null +++ b/src/sequifier/model/module_roles.py @@ -0,0 +1,92 @@ +"""Structural module roles shared by initialization and parameter selection.""" + +from torch import nn + +from sequifier.model.decoders import ( + AutoregressiveTransformerDecoderBranch, + HashTargetHead, + TargetDecoderBranch, + TargetDecoding, +) +from sequifier.model.ingestions import IngestionOutputProjection + +_POSITION_MODULES = { + "pos_encoder", + "position_embedding", + "global_position_encoder", + "axis_embedding_layers", +} +_INGESTION_PROJECTION_MODULES = { + "output_projection_layer", + "input_projection", + "concat_projection", + "branch_projections", +} + + +def decoder_module(model: nn.Module) -> nn.Module | None: + if isinstance(model, (TargetDecoding, TargetDecoderBranch)): + return model + try: + return model.get_submodule("decoder") + except AttributeError: + return None + + +def position_embedding_ids(model: nn.Module) -> set[int]: + return { + id(module) + for name, module in model.named_modules(remove_duplicate=False) + if isinstance(module, nn.Embedding) + and _POSITION_MODULES.intersection(name.split(".")) + } + + +def ingestion_projection_ids(model: nn.Module) -> set[int]: + return { + id(module) + for name, module in model.named_modules(remove_duplicate=False) + if isinstance(module, nn.Linear) + and ( + isinstance(module, IngestionOutputProjection) + or _INGESTION_PROJECTION_MODULES.intersection(name.split(".")) + ) + } + + +def decoder_output_ids(decoder: nn.Module | None) -> set[int]: + if decoder is None: + return set() + ids = { + id(output) + for branch in decoder.modules() + if isinstance( + branch, (TargetDecoderBranch, AutoregressiveTransformerDecoderBranch) + ) + for output in branch.output_layers.values() + if isinstance(output, nn.Linear) + } + ids.update( + id(layer) + for head in decoder.modules() + if isinstance(head, HashTargetHead) + for layer in head.layers + ) + # Legacy decoders were per-target ModuleDicts. + return ids or { + id(module) for module in decoder.modules() if isinstance(module, nn.Linear) + } + + +def range_position_projection(model: nn.Module) -> object: + # Accept both historical names for standalone backbones. + try: + projection = model.range_position_projection + except AttributeError: + projection = None + if projection is not None: + return projection + try: + return model.range_projection + except AttributeError: + return None diff --git a/src/sequifier/model/network.py b/src/sequifier/model/network.py index fb92203b0..5f26eb2e8 100644 --- a/src/sequifier/model/network.py +++ b/src/sequifier/model/network.py @@ -6,7 +6,16 @@ import torch from torch import Tensor, nn +from sequifier.config.depth_layout import DepthLayoutRegistryModel +from sequifier.model.backbone import TransformerBackbone +from sequifier.model.decoders import ( + AutoregressiveTransformerDecoderBranch, + TargetDecoderBranch, + TargetDecoding, +) from sequifier.model.dtypes import cast_floating_to_module_dtype +from sequifier.model.layers import SequifierEncoderLayer +from sequifier.model.prediction_distribution import PredictionScales from sequifier.model.tracing import ( CaptureRequest, ForwardContext, @@ -51,7 +60,7 @@ class TracedModelOutput: class ModelInterfaceModule(nn.Module): - """One named ingestion/adapter/decoder route around the shared backbone.""" + """One named ingestion route, optionally with a prediction decoder.""" @beartype def __init__( @@ -59,7 +68,7 @@ def __init__( *, ingestion: nn.Module, ingestion_adapter: nn.Module, - decoder: nn.Module, + decoder: TargetDecoding | None, decoder_input_width: int, decoding_support: int, prediction_length: int, @@ -68,6 +77,9 @@ def __init__( static_condition_encoder: nn.Module | None = None, preconditioning_image_encoder: nn.Module | None = None, prefix_projection: nn.Module | None = None, + depth_layouts: DepthLayoutRegistryModel | None = None, + input_columns: tuple[str, ...] = (), + input_n_classes: dict[str, int] | None = None, ) -> None: super().__init__() if decoding_support <= 0: @@ -75,6 +87,10 @@ def __init__( self.ingestion = ingestion self.ingestion_adapter = ingestion_adapter self.decoder = decoder + self.prediction_scales: PredictionScales | None = None + self.depth_layouts = depth_layouts + self.input_columns = input_columns + self.input_n_classes = input_n_classes self.decoder_input_width = decoder_input_width self.decoding_support = decoding_support self.prediction_length = prediction_length @@ -84,6 +100,18 @@ def __init__( self.preconditioning_image_encoder = preconditioning_image_encoder self.prefix_projection = prefix_projection + def require_decoder(self) -> TargetDecoding: + if self.decoder is None: + raise ValueError("This model interface has no prediction decoder.") + return self.decoder + + def prediction_scale(self, target: str): + if self.prediction_scales is None: + raise ValueError( + f"{target}: scale: model requires a prediction_distributions entry" + ) + return self.prediction_scales.scale(target) + def prefix(self, features: dict[str, Tensor]) -> Tensor | None: parts = [] if self.static_condition_encoder is not None: @@ -105,7 +133,7 @@ def prefix(self, features: dict[str, Tensor]) -> Tensor | None: def ingest( self, features: dict[str, Tensor], metadata: dict[str, Tensor] ) -> Tensor: - layouts = getattr(self, "depth_layouts", None) + layouts = self.depth_layouts if ( layouts and not torch.compiler.is_compiling() @@ -206,7 +234,7 @@ def decode_with_auxiliary( axes=("batch", "time", "channel"), width=self.decoder_input_width, ) - decoded = self.decoder( + decoded = self.require_decoder()( decoder_input, teacher_targets=teacher_targets, teacher_valid_mask=teacher_valid_mask, @@ -240,7 +268,7 @@ class ComposableTransformerNetwork(nn.Module): def __init__( self, *, - backbone: nn.Module, + backbone: TransformerBackbone, interfaces: dict[str, ModelInterfaceModule], attention_mask_policy: Tensor, attention_mask_policy_with_prefix: Tensor, @@ -263,8 +291,8 @@ def __init__( @property @conditional_beartype - def dim_model(self) -> int: - return int(self.backbone.dim_model) + def model_dim(self) -> int: + return int(self.backbone.model_dim) @property @conditional_beartype @@ -288,19 +316,18 @@ def regularization_loss(self, interface_name: str | None = None) -> Tensor: """Return decoder regularization through an explicit model contract.""" route = self.resolve_interface(interface_name) - regularization = getattr(route.decoder, "regularization_loss", None) - if not callable(regularization): - parameter = next(route.decoder.parameters(), None) - device = parameter.device if parameter is not None else torch.device("cpu") - return torch.zeros((), device=device) - return regularization() + if route.decoder is not None: + return route.decoder.regularization_loss() + parameter = next(route.parameters(), None) + device = parameter.device if parameter is not None else torch.device("cpu") + return torch.zeros((), device=device) def structural_metadata(self) -> dict[str, object]: return { "state_dict_prefixes": ("backbone.", "interfaces."), "interfaces": tuple(self.interfaces), "context_length": self.context_length, - "dim_model": self.dim_model, + "model_dim": self.model_dim, } @conditional_beartype @@ -308,25 +335,29 @@ def trace_catalog_for( self, interface_name: str | None = None ) -> tuple[TraceSite, ...]: route = self.resolve_interface(interface_name) + decoder = route.require_decoder() branch_counts = { - name: tuple(branch.hidden_dims) - for name, branch in route.decoder.branches.items() + name: tuple( + cast( + TargetDecoderBranch | AutoregressiveTransformerDecoderBranch, branch + ).hidden_dims + ) + for name, branch in decoder.branches.items() } - first_attention = self.backbone.layers[0].attn + first_attention = cast(SequifierEncoderLayer, self.backbone.layers[0]).attn sites = list( trace_sites( num_layers=len(self.backbone.layers), - model_width=self.dim_model, + model_width=self.model_dim, attention_width=int(first_attention.head_dim), decoder_input_width=route.decoder_input_width, decoder_branches=branch_counts, - target_branches=dict(route.decoder.target_to_branch), + target_branches=dict(decoder.target_to_branch), ) ) - for branch_name, branch in route.decoder.branches.items(): - describe = getattr(branch, "trace_sites", None) - if callable(describe): - sites.extend(cast(tuple[TraceSite, ...], describe(branch_name))) + for branch_name, branch in decoder.branches.items(): + if isinstance(branch, AutoregressiveTransformerDecoderBranch): + sites.extend(branch.trace_sites(branch_name)) return tuple(sites) @property @@ -446,10 +477,10 @@ def decode_representation( route = self.resolve_interface(interface_name) if ( not torch.onnx.is_in_onnx_export() - and representation.shape[-1] != self.dim_model + and representation.shape[-1] != self.model_dim ): raise ValueError( - f"Representation width must be {self.dim_model}, got " + f"Representation width must be {self.model_dim}, got " f"{representation.shape[-1]}." ) if request.apply_final_norm: diff --git a/src/sequifier/model/parameter_catalog.py b/src/sequifier/model/parameter_catalog.py index 43e308d24..9134a0d34 100644 --- a/src/sequifier/model/parameter_catalog.py +++ b/src/sequifier/model/parameter_catalog.py @@ -29,6 +29,7 @@ class ParameterDescriptor: shared_parameter_id: str | None depth_parameter: bool = False branch_path: tuple[str, ...] = () + semantic_aliases: tuple[str, ...] = () class ParameterCatalog: @@ -42,6 +43,47 @@ def __init__(self, model: nn.Module): for parameter in parameters } + semantic_aliases = { + identity: {group} for identity, group in semantic_by_identity.items() + } + # Component-local discovery preserves decoder and range-position semantics + # when the catalog is built for a complete multi-interface network. + from sequifier.model.decoders import TargetDecoderBranch, TargetDecoding + from sequifier.model.depth_ingestion import DepthTransformerIngestion + from sequifier.model.module_roles import range_position_projection + + depth_parameters = { + id(parameter) + for module in model.modules() + if isinstance(module, DepthTransformerIngestion) + for parameter in module.parameters() + } + for name, module in model.named_modules(): + component = name.rsplit(".", 1)[-1] + if ( + component not in {"backbone", "ingestion", "decoder"} + and not isinstance(module, (TargetDecoding, TargetDecoderBranch)) + and range_position_projection(module) is None + ): + continue + for group, component_parameters in semantic_parameter_groups( + module + ).items(): + for parameter in component_parameters: + identity = id(parameter) + previous = semantic_by_identity.get(identity) + group_aliases = semantic_aliases.setdefault(identity, set()) + if ( + previous + in {"fallback.linear", "fallback.convolution", "free_parameter"} + and previous != group + ): + group_aliases.discard(previous) + semantic_by_identity[identity] = group + elif previous is None: + semantic_by_identity[identity] = group + group_aliases.add(group) + aliases_by_identity: dict[int, list[str]] = {} parameter_by_identity: dict[int, nn.Parameter] = {} for name, parameter in model.named_parameters(remove_duplicate=False): @@ -67,13 +109,12 @@ def __init__(self, model: nn.Module): semantic_group=str( semantic_by_identity.get(identity, "free_parameter") ), + semantic_aliases=tuple(sorted(semantic_aliases.get(identity, ()))), parameter_kind=self._kind(canonical_name), shape=tuple(parameter.shape), dtype=parameter.dtype, shared_parameter_id=parameter_id if len(names) > 1 else None, - depth_parameter=bool( - getattr(parameter, "_sequifier_depth_parameter", False) - ), + depth_parameter=bool(identity in depth_parameters), branch_path=tuple( part for i, part in enumerate(canonical_name.split(".")) @@ -92,6 +133,8 @@ def __init__(self, model: nn.Module): @beartype def _component(name: str) -> ParameterComponent: parts = name.split(".") + while len(parts) > 1 and (parts[0] == "network" or parts[0].isdigit()): + parts = parts[1:] root = parts[0] route_component = parts[2] if root == "interfaces" and len(parts) > 2 else root if route_component in { @@ -102,7 +145,7 @@ def _component(name: str) -> ParameterComponent: "prefix_projection", }: return "ingestion" - if route_component == "decoder": + if route_component in {"decoder", "prediction_scales"}: return "decoder" return "backbone" @@ -141,7 +184,11 @@ def select( descriptor for descriptor in self._descriptors if (component is None or descriptor.component == component) - and (semantic_group is None or descriptor.semantic_group == semantic_group) + and ( + semantic_group is None + or semantic_group + in (descriptor.semantic_aliases or (descriptor.semantic_group,)) + ) ) @beartype diff --git a/src/sequifier/model/parameter_groups.py b/src/sequifier/model/parameter_groups.py index 270d0c3cf..34a3324c7 100644 --- a/src/sequifier/model/parameter_groups.py +++ b/src/sequifier/model/parameter_groups.py @@ -2,17 +2,23 @@ from collections import defaultdict from collections.abc import Iterable -from typing import Optional, cast +from typing import Optional from torch import Tensor, nn from sequifier.config.layer_groups import LayerGroup -from sequifier.model.decoders import TargetDecoderBranch, TargetDecoding from sequifier.model.ingestions import ( RealFeatureProjection, TemporalConvFeatureIngestion, ) from sequifier.model.layers import FeedForward, RMSNorm, SelfAttention +from sequifier.model.module_roles import ( + decoder_module, + decoder_output_ids, + ingestion_projection_ids, + position_embedding_ids, + range_position_projection, +) from sequifier.typechecking import beartype CONVOLUTION_TYPES = (nn.Conv1d, nn.Conv2d, nn.Conv3d) @@ -26,9 +32,9 @@ def __init__(self, model: nn.Module): self.model = model self._claimed_parameter_ids: set[int] = set() self._groups: dict[LayerGroup, list[nn.Parameter]] = defaultdict(list) - self._position_embedding_module_ids = self._position_embedding_ids() - self._ingestion_projection_module_ids = self._ingestion_projection_ids() - self._decoder_output_module_ids = self._decoder_output_ids() + self._position_embedding_module_ids = position_embedding_ids(model) + self._ingestion_projection_module_ids = ingestion_projection_ids(model) + self._decoder_output_module_ids = decoder_output_ids(decoder_module(model)) @beartype def collect(self) -> dict[LayerGroup, tuple[nn.Parameter, ...]]: @@ -59,92 +65,17 @@ def _claim( self._claimed_parameter_ids.add(parameter_id) self._groups[group].append(parameter) - @beartype - def _decoder_module(self) -> Optional[nn.Module]: - if isinstance(self.model, (TargetDecoding, TargetDecoderBranch)): - return self.model - decoder = getattr(self.model, "decoder", None) - return decoder if isinstance(decoder, nn.Module) else None - - @beartype - def _position_embedding_ids(self) -> set[int]: - position_embeddings: set[int] = set() - for module in self.model.modules(): - for attribute in ( - "pos_encoder", - "position_embedding", - "global_position_encoder", - "axis_embedding_layers", - ): - value = getattr(module, attribute, None) - if not isinstance(value, nn.Module): - continue - position_embeddings.update( - id(child) - for child in cast(nn.Module, value).modules() - if isinstance(child, nn.Embedding) - ) - return position_embeddings - - @beartype - def _ingestion_projection_ids(self) -> set[int]: - projections: set[int] = set() - for module in self.model.modules(): - if ( - isinstance(module, nn.Linear) - and getattr(module, "_sequifier_layer_group", None) - == "ingestion.output_projection" - ): - projections.add(id(module)) - for attribute in ( - "output_projection_layer", - "input_projection", - "concat_projection", - "branch_projections", - ): - value = getattr(module, attribute, None) - if not isinstance(value, nn.Module): - continue - projections.update( - id(child) - for child in cast(nn.Module, value).modules() - if isinstance(child, nn.Linear) - ) - return projections - - @beartype - def _decoder_output_ids(self) -> set[int]: - marked = { - id(module) - for module in self.model.modules() - if isinstance(module, nn.Linear) - and getattr(module, "_sequifier_decoder_output", False) - } - decoder = self._decoder_module() - if decoder is None: - return marked - output_module_ids = { - id(output_layer) - for module in decoder.modules() - if isinstance(module, TargetDecoderBranch) - for output_layer in module.output_layers.values() - if isinstance(output_layer, nn.Linear) - } - if output_module_ids or marked: - return output_module_ids | marked - return { - id(module) for module in decoder.modules() if isinstance(module, nn.Linear) - } - @beartype def _claim_linear(self, layer: nn.Linear, group: LayerGroup) -> None: self._claim(group, (layer.weight, layer.bias)) @beartype - def _claim_convolution(self, layer: nn.Module, group: LayerGroup) -> None: + def _claim_convolution( + self, layer: nn.Conv1d | nn.Conv2d | nn.Conv3d, group: LayerGroup + ) -> None: self._claim( group, - (getattr(layer, "weight", None), getattr(layer, "bias", None)), + (layer.weight, layer.bias), ) @beartype @@ -175,7 +106,7 @@ def _collect_transformer_layers(self) -> None: @beartype def _collect_decoder(self) -> None: - decoder = self._decoder_module() + decoder = decoder_module(self.model) if decoder is None: return for module in decoder.modules(): @@ -190,9 +121,7 @@ def _collect_decoder(self) -> None: @beartype def _collect_range_position_projection(self) -> None: - projection = getattr(self.model, "range_position_projection", None) - if projection is None: - projection = getattr(self.model, "range_projection", None) + projection = range_position_projection(self.model) if isinstance(projection, nn.Linear): self._claim_linear(projection, "position.range_projection") @@ -253,7 +182,7 @@ def _collect_remaining_modules(self) -> None: ): self._claim( "normalization", - (getattr(module, "weight", None), getattr(module, "bias", None)), + (module.weight, module.bias), ) elif isinstance(module, nn.Linear): group = ( diff --git a/src/sequifier/model/prediction_distribution.py b/src/sequifier/model/prediction_distribution.py new file mode 100644 index 000000000..6e19092a7 --- /dev/null +++ b/src/sequifier/model/prediction_distribution.py @@ -0,0 +1,219 @@ +"""Scalar prediction distributions, independent of the learning objective.""" + +import math +from types import SimpleNamespace + +import torch +from torch import nn +from torch.nn import functional as F + + +class PredictionScales(nn.Module): + """Gaussian scales owned and persisted by the model interface.""" + + def __init__(self, fields): + super().__init__() + self.fields = dict(fields) + self.log_scales = nn.ParameterDict( + { + name: nn.Parameter(torch.tensor(math.log(field.scale.initial))) + for name, field in sorted(fields.items()) + if field.scale.trainable + } + ) + + def ensure_fields(self, fields): + for name, field in sorted(fields.items()): + if field.scale.trainable and name not in self.log_scales: + self.log_scales[name] = nn.Parameter( + torch.tensor(math.log(field.scale.initial)) + ) + self.fields.update(fields) + + def scale(self, name): + if name not in self.fields: + raise ValueError( + f"{name}: scale: model requires a prediction_distributions entry" + ) + spec = self.fields[name].scale + if not spec.trainable: + return spec.initial + return self.log_scales[name].clamp(math.log(spec.min), math.log(spec.max)).exp() + + +class RealPredictionDistribution: + """A Gaussian, optionally mapped bijectively onto an open real domain. + + Values and densities use physical coordinates. Explicit point masses can + represent endpoints; continuous samples retain transform latents for replay. + """ + + def __init__(self, location, scale, field=None): + self.dtype = torch.float32 if location.device.type == "mps" else torch.float64 + self.location = location.to(self.dtype) + self.scale = torch.as_tensor(scale, device=location.device).to(self.dtype) + self.field = ( + field + if field is not None + else SimpleNamespace(lower=None, upper=None, atoms={}) + ) + if ( + not torch.isfinite(self.location).all() + or not torch.isfinite(self.scale).all() + or (self.scale <= 0).any() + ): + raise ValueError( + "real prediction locations and positive scales must be finite" + ) + self.normal = torch.distributions.Normal(self.location, self.scale) + + def _transform(self, latent): + lower, upper = self.field.lower, self.field.upper + if lower is not None and upper is not None: + return lower + (upper - lower) * torch.sigmoid(latent) + if lower is not None: + return lower + F.softplus(latent) + if upper is not None: + return upper - F.softplus(latent) + return latent + + def mode(self): + # Deterministic location; a transformed location need not be its mode. + return self._transform(self.location) + + @torch.no_grad() + def sample(self, generator=None): + """Sample a tensor, optionally with an independent generator per batch row. + + Only random draws are split by environment. Transforms and mixture + selection remain batched, and the scalar-generator path is unchanged. + """ + if isinstance(generator, (list, tuple)): + if self.location.ndim == 0 or len(generator) != self.location.shape[0]: + raise ValueError("one action RNG is required per leading batch row") + if any(not isinstance(rng, torch.Generator) for rng in generator): + raise TypeError("batched action RNGs must be torch.Generator instances") + + def draw(random, **options): + if isinstance(generator, (list, tuple)): + if not generator: + return torch.empty( + self.location.shape, device=self.location.device, **options + ) + return torch.cat( + [ + random( + self.location[index : index + 1].shape, + device=rng.device, + generator=rng, + **options, + ).to(self.location.device) + for index, rng in enumerate(generator) + ] + ) + device = generator.device if generator is not None else self.location.device + return random( + self.location.shape, device=device, generator=generator, **options + ).to(self.location.device) + + noise = draw(torch.randn, dtype=self.dtype) + self.sampled_latent = self.location + self.scale * noise + value = self._transform(self.sampled_latent) + self.sampled_component = torch.full_like(value, -1, dtype=torch.long) + if self.field.atoms: + choice = draw(torch.rand) + cumulative = 0.0 + for index, (atom, probability) in enumerate( + sorted(self.field.atoms.items()) + ): + selected = (choice >= cumulative) & (choice < cumulative + probability) + value = torch.where(selected, value.new_tensor(atom), value) + self.sampled_component = torch.where( + selected, + self.sampled_component.new_tensor(index), + self.sampled_component, + ) + cumulative += probability + return value + + def log_prob(self, value, latent=None, component=None): + value = value.to(self.dtype) + if not torch.isfinite(value).all(): + raise ValueError("real actions must be finite") + atoms = torch.zeros_like(value, dtype=torch.bool) + atom_score = torch.zeros_like(value) + if component is not None and ( + component.shape != value.shape + or (component < -2).any() + or (component >= len(self.field.atoms)).any() + ): + raise ValueError("invalid real mixture component") + for index, (atom, probability) in enumerate(sorted(self.field.atoms.items())): + selected = value == atom + if component is not None: + if ((component == index) & ~selected).any(): + raise ValueError( + "recorded mixture component differs from its action" + ) + selected &= (component == -2) | (component == index) + atoms |= selected + atom_score = torch.where( + selected, value.new_tensor(math.log(probability)), atom_score + ) + supplied = torch.zeros_like(atoms) if latent is None else ~torch.isnan(latent) + if latent is not None: + latent = latent.to(self.dtype) + if ( + latent.shape != value.shape + or not torch.isfinite(latent[supplied]).all() + ): + raise ValueError( + "real transform latents must be finite and match actions" + ) + comparable = supplied & ~atoms + if not torch.equal(self._transform(latent[comparable]), value[comparable]): + raise ValueError( + "recorded real latent does not reconstruct its executed action" + ) + lower, upper = self.field.lower, self.field.upper + invert = ~supplied & ~atoms + if lower is not None and (value[invert] <= lower).any(): + raise ValueError( + "real endpoints require a declared atom or a recorded transform latent" + ) + if upper is not None and (value[invert] >= upper).any(): + raise ValueError( + "real endpoints require a declared atom or a recorded transform latent" + ) + safe = torch.where(invert, value, self._transform(torch.zeros_like(value))) + if lower is not None and upper is not None: + fraction = (safe - lower) / (upper - lower) + inverse = fraction.log() - torch.log1p(-fraction) + elif lower is not None or upper is not None: + distance = safe - lower if lower is not None else upper - safe + inverse = distance + torch.log(-torch.expm1(-distance)) + else: + inverse = safe + z = torch.where(supplied, latent, inverse) if latent is not None else inverse + z = torch.where(atoms, torch.zeros_like(z), z) + if lower is not None and upper is not None: + jacobian = math.log(upper - lower) + F.logsigmoid(z) + F.logsigmoid(-z) + elif lower is not None or upper is not None: + jacobian = F.logsigmoid(z) + else: + jacobian = 0.0 + score = ( + self.normal.log_prob(z) + - jacobian + + math.log1p(-sum(self.field.atoms.values())) + ) + return torch.where(atoms, atom_score, score) + + def entropy(self): + if ( + self.field.lower is not None + or self.field.upper is not None + or self.field.atoms + ): + raise ValueError("transformed entropy requires a sampling estimator") + return self.normal.entropy() diff --git a/src/sequifier/model/static_conditioning.py b/src/sequifier/model/static_conditioning.py index f9d49b326..47d78c4d9 100644 --- a/src/sequifier/model/static_conditioning.py +++ b/src/sequifier/model/static_conditioning.py @@ -6,7 +6,10 @@ from torch import Tensor, nn from sequifier.model.dtypes import cast_floating_to_module_dtype -from sequifier.model.ingestions import get_feature_embedding_dims +from sequifier.model.ingestions import ( + IngestionOutputProjection, + get_feature_embedding_dims, +) class StaticConditionEncoder(nn.Module): @@ -16,8 +19,8 @@ def __init__( categorical_columns: list[str], real_columns: list[str], n_classes: dict[str, int], - dim_model: int, - embedding_size: int | None = None, + model_dim: int, + total_embedding_dim: int | None = None, feature_embedding_dims: dict[str, int] | None = None, ) -> None: super().__init__() @@ -26,9 +29,9 @@ def __init__( self.real_columns = tuple(real_columns) if feature_embedding_dims is not None: widths = feature_embedding_dims - elif embedding_size is not None: + elif total_embedding_dim is not None: widths = get_feature_embedding_dims( - embedding_size, categorical_columns, real_columns, n_classes + total_embedding_dim, categorical_columns, real_columns, n_classes ) else: widths = { @@ -53,8 +56,7 @@ def __init__( raise ValueError( "Every static conditioning column must be categorical or real" ) - self.projection = nn.Linear(input_width, dim_model) - self.projection._sequifier_layer_group = "ingestion.output_projection" + self.projection = IngestionOutputProjection(input_width, model_dim) def forward(self, features: dict[str, Tensor]) -> Tensor: parts = [ diff --git a/src/sequifier/model/tracing.py b/src/sequifier/model/tracing.py index d682b377f..aef797b72 100644 --- a/src/sequifier/model/tracing.py +++ b/src/sequifier/model/tracing.py @@ -7,6 +7,7 @@ import torch from torch import Tensor +from torch._dynamo import OptimizedModule from sequifier.typechecking import beartype @@ -178,9 +179,13 @@ def analysis_execution( trace: CaptureRequest | None = None, interventions: tuple[InterventionBinding, ...] = (), ): - if hasattr(model, "_orig_mod"): + if isinstance(model, OptimizedModule): raise ValueError("analysis_execution requires an eager, uncompiled model.") - available = {site.name for site in getattr(model, "trace_catalog", ())} + try: + trace_catalog = model.trace_catalog + except AttributeError: + trace_catalog = () + available = {site.name for site in trace_catalog} requested = set(trace.sites if trace is not None else ()).union( binding.site for binding in interventions ) @@ -216,7 +221,7 @@ def functional_forward( *args: Any, **kwargs: Any, ) -> Any: - if hasattr(model, "_orig_mod"): + if isinstance(model, OptimizedModule): raise ValueError("functional_forward requires an eager, uncompiled model.") return torch.func.functional_call(model, (parameters, buffers), args, kwargs) diff --git a/src/sequifier/objectives.py b/src/sequifier/objectives.py index 07263d3a9..ebdf2bd5b 100644 --- a/src/sequifier/objectives.py +++ b/src/sequifier/objectives.py @@ -148,6 +148,34 @@ class CausalObjective(Objective): name = "causal" +class PolicyObjective(Objective): + """Causal attention with same-position action labels, never future labels.""" + + name = "policy" + forward_looking = False + + @classmethod + @beartype + def default_target_offset(cls) -> int: + return 0 + + @classmethod + @beartype + def validate_window_view(cls, context_length: int, target_offset: int) -> None: + if target_offset != 0: + raise ValueError("Policy views require target_offset=0") + + @classmethod + @beartype + def item_positions( + cls, start_positions: np.ndarray, context_length: int, prediction_length: int + ) -> np.ndarray: + return ( + start_positions[:, None] + + np.arange(context_length - prediction_length, context_length)[None, :] + ).reshape(-1) + + class FinalValueObjective(CausalObjective): name = "final_value" @@ -165,9 +193,7 @@ class NextOccurrenceObjective(CausalObjective): @beartype def __init__(self, config: Any) -> None: super().__init__(config) - training = getattr(config, "global_training", None) - if training is None: - training = config.training_spec + training = config.runtime_settings next_occurrence_config = training.next_occurrence_config self.next_occurrence_column = next_occurrence_config.column_name @@ -326,11 +352,14 @@ def item_positions( OBJECTIVE_REGISTRY: dict[str, type[Objective]] = { CausalObjective.name: CausalObjective, + PolicyObjective.name: PolicyObjective, BERTObjective.name: BERTObjective, FinalValueObjective.name: FinalValueObjective, NextOccurrenceObjective.name: NextOccurrenceObjective, } -OBJECTIVE_NAME_MESSAGE = "'causal', 'bert', 'final_value', and 'next_occurrence'" +OBJECTIVE_NAME_MESSAGE = ( + "'causal', 'bert', 'policy', 'final_value', and 'next_occurrence'" +) ALLOWED_OBJECTIVE_NAMES = frozenset(OBJECTIVE_REGISTRY) @@ -492,9 +521,7 @@ def apply_bert_masking( seeded_generator.manual_seed(eval_seed) generator = seeded_generator - training = getattr(config, "global_training", None) - if training is None: - training = config.training_spec + training = config.runtime_settings bert_spec = training.bert_spec if bert_spec is None: raise ValueError("bert_spec must be configured for BERT training") @@ -524,7 +551,7 @@ def apply_bert_masking( ) masked_data = dict(data_batch) - depth_layouts = getattr(config, "depth_layouts", None) + depth_layouts = config.depth_layouts depth_column_layout = depth_layouts.column_to_layout if depth_layouts else {} for col, tensor in data_batch.items(): diff --git a/src/sequifier/optimizers/plan.py b/src/sequifier/optimizers/plan.py index c9c4d4a65..98a8aa8e0 100644 --- a/src/sequifier/optimizers/plan.py +++ b/src/sequifier/optimizers/plan.py @@ -3,7 +3,6 @@ from __future__ import annotations from dataclasses import dataclass -from fnmatch import fnmatchcase from typing import Any, Iterable import torch @@ -24,21 +23,7 @@ class RoutedOptimizerGroup: def _matches(descriptor: ParameterDescriptor, selector: OptimizerSelector) -> bool: - return ( - ( - selector.semantic_groups is None - or any( - fnmatchcase(descriptor.semantic_group, pattern) - for pattern in selector.semantic_groups - ) - ) - and (selector.component is None or descriptor.component == selector.component) - and ( - selector.parameter_kind is None - or descriptor.parameter_kind == selector.parameter_kind - ) - and (selector.ndim is None or len(descriptor.shape) == selector.ndim) - ) + return selector.matches(descriptor) def route_optimizer_parameters( @@ -55,8 +40,16 @@ def route_optimizer_parameters( seen: set[int] = set() explicit = plan.groups[:-1] fallback = plan.groups[-1] - - for descriptor in catalog.descriptors(): + eligible_descriptors = [ + descriptor + for descriptor in catalog.descriptors() + if id(catalog.parameter(descriptor.parameter_id)) in supplied + ] + for group in explicit: + if isinstance(group.select, OptimizerSelector): + group.select.validate_semantic_patterns(eligible_descriptors) + + for descriptor in eligible_descriptors: parameter = catalog.parameter(descriptor.parameter_id) identity = id(parameter) if identity not in supplied or identity in seen: diff --git a/src/sequifier/preprocess.py b/src/sequifier/preprocess.py index 8bfd013bb..f80b77b83 100644 --- a/src/sequifier/preprocess.py +++ b/src/sequifier/preprocess.py @@ -22,21 +22,42 @@ from sequifier.config.depth_layout import DepthLayoutRegistryModel from sequifier.config.image_config import ImageSourceConfig, PreconditioningImageConfig +from sequifier.config.metadata import ( + dataset_tensor_kind, + dataset_tensor_version, + validate_persisted_metadata_versions, +) from sequifier.config.preprocess_config import ( CardinalityLimitModel, load_preprocessor_config, ) +from sequifier.contract_versions import ContractVersions, require_contract_version from sequifier.helpers import ( + FLOAT_TYPE_ORDER, PANDAS_TO_TORCH_TYPES, StoredWindowLayout, +) +from sequifier.helpers import _highest_ranked_type as _highest_ranked_type +from sequifier.helpers import _resolve_integer_sequence_type +from sequifier.helpers import ( + _smallest_float_covering_integer_range as _smallest_float_covering_integer_range, +) +from sequifier.helpers import ( assign_sequence_to_split, canonicalize_polars_dtype_name, is_float_dtype_name, is_integer_dtype_name, polars_dtype_from_name, read_data, + resolve_unified_polars_numeric_dtype, ) from sequifier.io.images import _time_ms, validate_sources +from sequifier.io.preprocessing import ( + _finite_mean_and_std, + create_id_map, + get_combined_statistics, + normalize_values, +) from sequifier.io.pt_payload import StoredTensorBatch, load_pt_payload, save_pt_payload from sequifier.io.sample_order import ( CURRICULUM_COLUMN_PREFIX, @@ -56,14 +77,12 @@ INPUT_METADATA_COLUMNS = ("sequenceId", "itemPosition") SPLIT_VALUE_COLUMN = "__sequifier_split_value" REAL_MASK_VALUE = 0.0 -CURRENT_STORED_WINDOW_LAYOUT_VERSION = 2 MAX_WINDOW_BUFFER_BYTES = 256 * 1024 * 1024 HASH_BUCKET_LABEL_PREFIX = "[hash_bucket:" STABLE_HASH_OFFSET_BASIS = 14695981039346656037 STABLE_HASH_PRIME = 1099511628211 UINT64_MASK = (1 << 64) - 1 -FLOAT_TYPE_ORDER = ("Float16", "Float32", "Float64") INTEGER_TYPE_ORDER = ( "Int8", "UInt8", @@ -74,26 +93,7 @@ "Int64", "UInt64", ) -INTEGER_TYPE_INFO = { - "Int8": np.iinfo(np.int8), - "Int16": np.iinfo(np.int16), - "Int32": np.iinfo(np.int32), - "Int64": np.iinfo(np.int64), - "UInt8": np.iinfo(np.uint8), - "UInt16": np.iinfo(np.uint16), - "UInt32": np.iinfo(np.uint32), - "UInt64": np.iinfo(np.uint64), -} -FLOAT_TYPE_INFO = { - "Float16": np.finfo(np.float16), - "Float32": np.finfo(np.float32), - "Float64": np.finfo(np.float64), -} -FLOAT_EXACT_INTEGER_LIMITS = { - "Float16": 2**11, - "Float32": 2**24, - "Float64": 2**53, -} + INT64_INFO = np.iinfo(np.int64) @@ -201,7 +201,7 @@ def _finalize_cardinality_maps( """Build retained-value maps followed by optional hashing buckets.""" for column, config in cardinality_config.items(): retained = [] - if "min_freq" in config or "top_k" in config: + if "min_count" in config or "max_categories" in config: counts = value_counts.get(column, Counter()) if not counts: raise ValueError( @@ -213,11 +213,11 @@ def _finalize_cardinality_maps( for value, count in counts.items() if value not in SPECIAL_TOKEN_LABELS } - if "min_freq" in config: + if "min_count" in config: retained = [ value for value, count in user_counts.items() - if count >= int(config["min_freq"]) + if count >= int(config["min_count"]) ] else: retained = [ @@ -225,7 +225,7 @@ def _finalize_cardinality_maps( for value, _ in sorted( user_counts.items(), key=lambda item: (-item[1], _category_sort_key(item[0])), - )[: int(config["top_k"])] + )[: int(config["max_categories"])] ] if retained: id_maps[column] = create_id_map(pl.DataFrame({column: retained}), column) @@ -531,10 +531,8 @@ def _normalize_column_types( def _column_types_from_metadata( metadata: dict[str, Any], ) -> Optional[dict[str, str]]: - """Read current and historical metadata field names.""" - return _normalize_column_types( - metadata.get("column_data_types") or metadata.get("column_types") - ) + """Read the canonical storage dtype mapping.""" + return _normalize_column_types(metadata.get("column_data_types")) @beartype @@ -608,7 +606,11 @@ def _validate_declared_roles_against_metadata( @beartype def _dtype_is_numeric(dtype: Any) -> bool: - return dtype.is_numeric() if hasattr(dtype, "is_numeric") else False + try: + is_numeric = dtype.is_numeric + except AttributeError: + return False + return is_numeric() def _is_temporal_dtype_name(dtype_name: str) -> bool: @@ -669,98 +671,36 @@ def _apply_output_type_casting( return data.with_columns(casts) -@beartype -def _highest_ranked_type(types: list[str], order: tuple[str, ...]) -> str: - return max(types, key=lambda type_: order.index(type_)) - - -@beartype -def _smallest_float_covering_integer_range(integer_type: str) -> str: - integer_info = INTEGER_TYPE_INFO[integer_type] - largest_magnitude = max(abs(int(integer_info.min)), int(integer_info.max)) - for float_type in FLOAT_TYPE_ORDER: - if ( - largest_magnitude <= float(FLOAT_TYPE_INFO[float_type].max) - and largest_magnitude <= FLOAT_EXACT_INTEGER_LIMITS[float_type] - ): - return float_type - return "Float64" - - -@beartype -def _resolve_integer_sequence_type(integer_types: list[str]) -> Any: - if not integer_types: - raise ValueError("Cannot resolve an integer sequence type without integers") - - min_value = min(int(INTEGER_TYPE_INFO[type_].min) for type_ in integer_types) - max_value = max(int(INTEGER_TYPE_INFO[type_].max) for type_ in integer_types) - - if min_value >= 0 and all(type_.startswith("UInt") for type_ in integer_types): - for dtype_name in ("UInt8", "UInt16", "UInt32", "UInt64"): - info = INTEGER_TYPE_INFO[dtype_name] - if max_value <= int(info.max): - return polars_dtype_from_name(dtype_name) - - for dtype_name in ("Int8", "Int16", "Int32", "Int64"): - info = INTEGER_TYPE_INFO[dtype_name] - if min_value >= int(info.min) and max_value <= int(info.max): - return polars_dtype_from_name(dtype_name) - - raise ValueError(f"Cannot resolve a safe integer dtype for {integer_types}") - - @beartype def _resolve_unified_parquet_type(column_data_types: dict[str, str]) -> Any: - if not column_data_types: - raise ValueError("column_data_types cannot be empty") - - normalized_types = [ - ( - "Int64" - if _is_temporal_dtype_name(type_) - else canonicalize_polars_dtype_name(type_) - ) - for type_ in column_data_types.values() - ] - float_types = [type_ for type_ in normalized_types if is_float_dtype_name(type_)] - integer_types = [ - type_ for type_ in normalized_types if is_integer_dtype_name(type_) - ] - - if not float_types: - if len(set(integer_types)) > 1: + normalized = { + column: "Int64" + if _is_temporal_dtype_name(dtype) + else canonicalize_polars_dtype_name(dtype) + for column, dtype in column_data_types.items() + } + resolved = resolve_unified_polars_numeric_dtype(normalized) + types = set(normalized.values()) + floats = [dtype for dtype in types if is_float_dtype_name(dtype)] + if not floats: + if len(types) > 1: logger.warning( "Multiple integer column_data_types were specified for Parquet output; " "using a unified integer schema." ) - return _resolve_integer_sequence_type(integer_types) - - resolved_float = _highest_ranked_type(float_types, FLOAT_TYPE_ORDER) - if len(set(normalized_types)) > 1: - logger.warning( - "Multiple column_data_types were specified for Parquet output; " - f"using unified sequence dtype {resolved_float}." - ) - - if integer_types: - required_float = _highest_ranked_type( - [ - _smallest_float_covering_integer_range(integer_type) - for integer_type in integer_types - ], - FLOAT_TYPE_ORDER, - ) - if FLOAT_TYPE_ORDER.index(required_float) > FLOAT_TYPE_ORDER.index( - resolved_float - ): + else: + requested = max(floats, key=FLOAT_TYPE_ORDER.index) + if len(types) > 1: + logger.warning( + "Multiple column_data_types were specified for Parquet output; " + f"using unified sequence dtype {requested}." + ) + if str(resolved) != requested: logger.warning( "An integer column_type has a range exceeding " - f"{resolved_float}; upgrading unified Parquet sequence dtype " - f"to {required_float}." + f"{requested}; upgrading unified Parquet sequence dtype to {resolved}." ) - resolved_float = required_float - - return polars_dtype_from_name(resolved_float) + return resolved @beartype @@ -784,7 +724,19 @@ def preprocess(args: Any, args_config: dict[str, Any]) -> None: logger.info("--- Starting Preprocessing ---") config_path = args.config_path or "configs/preprocess.yaml" config = load_preprocessor_config(config_path, args_config) - Preprocessor(**config.dict()) + from sequifier.config.sequence_import_config import SequenceImportConfig + + if isinstance(config, SequenceImportConfig): + from sequifier.trajectory.export import import_sequences + + import_sequences(config) + else: + Preprocessor( + **config.model_dump(exclude={"run"}), + project_root=config.project_root, + seed=config.seed, + num_workers=config.num_workers, + ) logger.info("--- Preprocessing Complete ---") @@ -904,7 +856,7 @@ def _audit_folder_sequence_summaries( max_rows: Optional[int], split_column: Optional[str] = None, split_values: Optional[list[Any]] = None, - n_cores: int = 1, + num_workers: int = 1, ) -> tuple[list[str], dict[int, list[int]], Optional[list[int]]]: """Return selected folder files and global sequence coordinate summaries.""" summaries: dict[int, list[int]] = {} @@ -912,11 +864,11 @@ def _audit_folder_sequence_summaries( rows_read = 0 cutoffs = _normalized_split_cutoffs(split_values) cutoff_counts = [0] * (len(cutoffs) + 1) if cutoffs is not None else None - if max_rows is None and n_cores > 1 and len(file_paths) > 1: + if max_rows is None and num_workers > 1 and len(file_paths) > 1: jobs = [ (path, read_format, None, split_column, split_values) for path in file_paths ] - with _create_preprocess_pool(min(n_cores, len(jobs))) as pool: + with _create_preprocess_pool(min(num_workers, len(jobs))) as pool: results = pool.map(_audit_folder_file, jobs) files_and_results = zip(file_paths, results) else: @@ -1153,11 +1105,10 @@ def __init__( allow_sequence_splitting: bool, selected_columns: Optional[list[str]], split_ratios: Optional[list[float]], - window_length: Optional[int], window_stride: int, max_rows: Optional[int], seed: int, - n_cores: Optional[int], + num_workers: Optional[int], batches_per_file: int, process_by_file: bool, prediction_aligned_splits: list[int], @@ -1268,16 +1219,13 @@ def __init__( raise ValueError("Ratio-based split methods require split_ratios") self.split_ratios = split_ratios self.window_stride = window_stride - if window_lengths is None: - if window_length is None: - raise ValueError("Specify window_length or window_lengths") - resolved_window_lengths = [window_length] * self.n_splits - else: - resolved_window_lengths = window_lengths + if not window_lengths: + raise ValueError("window_lengths requires one width per split") + resolved_window_lengths = window_lengths self.window_lengths: list[int] = resolved_window_lengths self.split_layouts = [ StoredWindowLayout( - width, max_target_offset, CURRENT_STORED_WINDOW_LAYOUT_VERSION + width, max_target_offset, ContractVersions.STORED_WINDOW_LAYOUT ) for width in resolved_window_lengths ] @@ -1314,14 +1262,16 @@ def __init__( self.seed = seed np.random.seed(seed) - self.n_cores = n_cores or multiprocessing.cpu_count() + self.num_workers = num_workers or multiprocessing.cpu_count() self.continue_preprocessing = continue_preprocessing self.storage_layout = StoredWindowLayout( window_length=resolved_window_lengths[0], max_target_offset=max_target_offset, - version=CURRENT_STORED_WINDOW_LAYOUT_VERSION, + version=ContractVersions.STORED_WINDOW_LAYOUT, ) self.has_sample_positions = False + self.output_n_classes = {} + self.output_column_data_types = {} self.image_manifest = self._validate_image_sources() self._setup_directories() @@ -1471,6 +1421,7 @@ def __init__( with open(metadata_path, "r") as f: preexisting_metadata = json.load(f) + validate_persisted_metadata_versions(preexisting_metadata) self.metadata_fitted_on_all_data = preexisting_metadata.get( "normalize_on_all_data", True @@ -1594,7 +1545,7 @@ def __init__( self.data_name_root, data, schema, - self.n_cores, + self.num_workers, self.split_layouts, window_stride, data_columns, @@ -1664,7 +1615,7 @@ def __init__( max_rows, self.split_column if self.metadata_config_path else None, self.split_values if self.metadata_config_path else None, - self.n_cores, + self.num_workers, ) fragmented_sequence_ids = { sequence_id @@ -1691,6 +1642,7 @@ def __init__( with open(metadata_path, "r") as f: preexisting_metadata = json.load(f) + validate_persisted_metadata_versions(preexisting_metadata) self.metadata_fitted_on_all_data = preexisting_metadata.get( "normalize_on_all_data", True @@ -1724,7 +1676,7 @@ def __init__( if col_types is None: raise ValueError( "Metadata used with folder preprocessing must contain " - "'column_data_types' or the historical 'column_types' field." + "'column_data_types' field." ) col_types = { column: dtype @@ -1890,7 +1842,7 @@ def __init__( selected_columns, max_rows, schema, - self.n_cores, + self.num_workers, self.split_layouts, window_stride, data_columns, @@ -1970,7 +1922,7 @@ def _validate_image_sources(self) -> dict[str, Any]: self.static_data_path, ) _, summaries, _ = _audit_folder_sequence_summaries( - files, self.read_format, self.max_rows, n_cores=1 + files, self.read_format, self.max_rows, num_workers=1 ) if any(summary[0] != 0 for summary in summaries.values()): raise ValueError("Image sequences require itemPosition to start at zero") @@ -2457,10 +2409,10 @@ def consume(path: str, result: _FolderMetadataResult) -> None: } logger.info(f"Data path: {data_path}") - if max_rows is None and self.n_cores > 1 and len(files_to_process) > 1: + if max_rows is None and self.num_workers > 1 and len(files_to_process) > 1: jobs = (job_for(path, None) for path in files_to_process) with _create_preprocess_pool( - min(self.n_cores, len(files_to_process)) + min(self.num_workers, len(files_to_process)) ) as pool: for path, result in zip( files_to_process, pool.imap(_fit_folder_metadata_file, jobs) @@ -2566,7 +2518,7 @@ def _process_fragmented_folder( target_bucket_bytes = 256 * 1024 * 1024 bucket_count = min( 4096, - max(self.n_cores, math.ceil(total_input_bytes / target_bucket_bytes)), + max(self.num_workers, math.ceil(total_input_bytes / target_bucket_bytes)), ) bucket_dir = os.path.join( self.project_root, "data", self.target_dir, "fragment-buckets" @@ -2626,7 +2578,7 @@ def _process_fragmented_folder( split: [] for split in range(self.n_splits) } worker_pool = ( - _create_preprocess_pool(self.n_cores) if self.n_cores > 1 else None + _create_preprocess_pool(self.num_workers) if self.num_workers > 1 else None ) try: for bucket, paths in sorted(bucket_files.items()): @@ -2648,7 +2600,7 @@ def _process_fragmented_folder( bucket_name, data, schema, - self.n_cores, + self.num_workers, layout, window_stride, data_columns, @@ -2726,7 +2678,7 @@ def _process_batches_multiple_files( selected_columns: Optional[list[str]], max_rows: Optional[int], schema: Any, - n_cores: int, + num_workers: int, layout: StoredWindowLayout | list[StoredWindowLayout], window_stride: int, data_columns: list[str], @@ -2761,7 +2713,7 @@ def _process_batches_multiple_files( selected_columns=selected_columns, max_rows=max_rows, schema=schema, - n_cores=n_cores, + num_workers=num_workers, layout=layout, window_stride=window_stride, data_columns=data_columns, @@ -2800,7 +2752,7 @@ def _process_batches_multiple_files( ) else: assert process_by_file is False - worker_count = min(n_cores, len(file_paths)) + worker_count = min(num_workers, len(file_paths)) file_sizes = [os.path.getsize(path) for path in file_paths] file_sets = [] next_file = 0 @@ -2836,7 +2788,7 @@ def _process_batches_multiple_files( "selected_columns": selected_columns, "max_rows": max_rows, "schema": schema, - "n_cores": 1, + "num_workers": 1, "layout": layout, "window_stride": window_stride, "data_columns": data_columns, @@ -2857,7 +2809,7 @@ def _process_batches_multiple_files( "seed": self.seed, "normalize_real_columns": self.normalize_real_columns, "sequence_split_assignments": sequence_split_assignments, - "curriculum_column": getattr(self, "curriculum_column", None), + "curriculum_column": self.curriculum_column, "split_column": self.split_column, "split_values": self.split_values, "normalize_on_all_data": self.normalize_on_all_data, @@ -2871,7 +2823,7 @@ def _process_batches_multiple_files( + list(kwargs_2.values()) for process_id, file_set in enumerate(file_sets) ] - logger.info(f"_process_batches_multiple_files n_cores: {n_cores}") + logger.info(f"_process_batches_multiple_files num_workers: {num_workers}") logger.info(f"_process_batches_multiple_files {len(job_params) = }") with _create_preprocess_pool(len(job_params)) as pool: @@ -2941,9 +2893,11 @@ def _cleanup(self, write_format: str) -> None: @beartype def _layout_metadata(self) -> dict[str, Any]: + tensor_kind = dataset_tensor_kind(self.depth_layouts) return { "depth_layouts": self.depth_layouts.model_dump(mode="json"), - "tensor_payload_version": 2 if self.depth_layouts else 1, + "tensor_payload_kind": tensor_kind, + "tensor_payload_version": dataset_tensor_version(tensor_kind), "window_length": self.storage_layout.window_length, "window_lengths": self.window_lengths, "prediction_alignment_steps": self.prediction_alignment_steps, @@ -2966,7 +2920,7 @@ def _write_or_validate_resume_manifest( selected_columns_statistics: dict[str, dict[str, float]], ) -> None: manifest = { - "manifest_version": 1, + "manifest_version": ContractVersions.PREPROCESSING_MANIFEST, "preprocessing_config": { **self._layout_metadata(), "read_format": self.read_format, @@ -3003,11 +2957,11 @@ def _write_or_validate_resume_manifest( "mask_column": self.mask_column, **( {"curriculum_column": self.curriculum_column} - if getattr(self, "curriculum_column", None) is not None + if self.curriculum_column is not None else {} ), "use_precomputed_maps": self.use_precomputed_maps, - "cardinality_config": getattr(self, "cardinality_config", {}), + "cardinality_config": self.cardinality_config, "n_classes": n_classes, "id_maps": id_maps, "column_data_types": col_types, @@ -3039,31 +2993,13 @@ def _write_or_validate_resume_manifest( ) with open(manifest_path, "r") as f: previous_manifest = json.load(f) - previous_manifest.get("preprocessing_config", {}).setdefault( - "depth_layouts", {} - ) - previous_manifest.get("preprocessing_config", {}).setdefault( - "tensor_payload_version", 1 - ) - previous_manifest.get("preprocessing_config", {}).setdefault( - "normalize_on_all_data", True - ) - previous_manifest.get("preprocessing_config", {}).setdefault( - "categorical_columns", None + require_contract_version( + "preprocessing manifest", + previous_manifest.get("manifest_version"), + ContractVersions.PREPROCESSING_MANIFEST, ) - previous_manifest.get("preprocessing_config", {}).setdefault( - "real_columns", None - ) - previous_manifest.get("preprocessing_config", {}).setdefault( - "cardinality_config", {} - ) - previous_manifest.get("preprocessing_config", {}).setdefault( - "window_lengths", - [previous_manifest["preprocessing_config"]["window_length"]] - * self.n_splits, - ) - previous_manifest.get("preprocessing_config", {}).setdefault( - "prediction_alignment_steps", 1 + validate_persisted_metadata_versions( + previous_manifest["preprocessing_config"] ) if _stable_json_value(previous_manifest) != _stable_json_value(manifest): raise ValueError( @@ -3094,7 +3030,13 @@ def _restore_resume_manifest_state( ) with open(manifest_path, "r") as f: manifest = json.load(f) - previous = manifest.get("preprocessing_config", {}) + require_contract_version( + "preprocessing manifest", + manifest.get("manifest_version"), + ContractVersions.PREPROCESSING_MANIFEST, + ) + previous = manifest["preprocessing_config"] + validate_persisted_metadata_versions(previous) required = { "data_columns", "id_maps", @@ -3247,7 +3189,7 @@ def _create_metadata_for_folder( payload = load_pt_payload( file_path, layouts=self.depth_layouts, - n_classes=getattr(self, "output_n_classes", None), + n_classes=self.output_n_classes, ) sequences_dict = payload.sequences left_pad_lengths = payload.left_pad_lengths @@ -3338,8 +3280,8 @@ def _create_metadata_for_folder( metadata = { "split_index": split_index, - "n_classes": getattr(self, "output_n_classes", {}), - "column_data_types": getattr(self, "output_column_data_types", {}), + "n_classes": self.output_n_classes, + "column_data_types": self.output_column_data_types, "id_maps": self.output_id_maps, "selected_columns_statistics": self.output_selected_columns_statistics, "normalize_real_columns": self.normalize_real_columns, @@ -3349,7 +3291,7 @@ def _create_metadata_for_folder( **self._layout_metadata(), } metadata["window_length"] = self.window_lengths[split_index] - if getattr(self, "has_sample_positions", False): + if self.has_sample_positions: metadata["curriculum_column"] = self.curriculum_column metadata_path = directory / "metadata.json" @@ -3598,9 +3540,8 @@ def _apply_column_statistics( col_types[col] = "Float64" if normalize_real_columns: expressions.append( - ( - (pl.col(col) - selected_columns_statistics[col]["mean"]) - / (selected_columns_statistics[col]["std"] + 1e-9) + normalize_values( + pl.col(col), selected_columns_statistics[col] ).alias(col) ) @@ -3754,7 +3695,7 @@ def _get_column_statistics( if data_col not in precomputed_id_maps: cardinality = (cardinality_config or {}).get(data_col) if cardinality is not None: - if "min_freq" in cardinality or "top_k" in cardinality: + if "min_count" in cardinality or "max_categories" in cardinality: if categorical_value_counts is None: raise ValueError( "categorical_value_counts is required when fitting " @@ -3818,30 +3759,6 @@ def _get_column_statistics( return id_maps, selected_columns_statistics -@beartype -def _finite_mean_and_std(column: pl.Series, column_name: str) -> tuple[float, float]: - """Compute sample statistics without overflowing intermediate squares.""" - values = column.to_numpy().astype(np.float64, copy=False) - if values.size == 0: - raise ValueError(f"Cannot compute statistics for empty column {column_name!r}.") - if not np.isfinite(values).all(): - raise ValueError(f"Column {column_name!r} contains non-finite values.") - - scale = float(np.max(np.abs(values))) - if scale == 0.0: - return 0.0, 0.0 - - scaled_values = values / scale - mean = float(np.mean(scaled_values) * scale) - std = 0.0 if values.size <= 1 else float(np.std(scaled_values, ddof=1) * scale) - if not math.isfinite(mean) or not math.isfinite(std): - raise ValueError( - f"Column {column_name!r} has a numeric range too large for finite " - "normalization statistics." - ) - return mean, std - - @beartype def _validate_sequence_coordinates( data: pl.DataFrame, source: str, sort_output: bool = True @@ -4128,7 +4045,7 @@ def _process_batches_multiple_files_inner( selected_columns: Optional[list[str]], max_rows: Optional[int], schema: Any, - n_cores: int, + num_workers: int, layout: StoredWindowLayout | list[StoredWindowLayout], window_stride: int, data_columns: list[str], @@ -4170,7 +4087,7 @@ def _process_batches_multiple_files_inner( raise ValueError("No files found to process.") pad_width = len(str(n_files - 1)) n_rows_running_count = 0 - worker_pool = _create_preprocess_pool(n_cores) if n_cores > 1 else None + worker_pool = _create_preprocess_pool(num_workers) if num_workers > 1 else None for file_index, path in enumerate(file_paths): max_rows_inner = None if max_rows is None else max_rows - n_rows_running_count if max_rows_inner is None or max_rows_inner > 0: @@ -4245,7 +4162,7 @@ def _process_batches_multiple_files_inner( data_name_root_inner, data, schema, - n_cores, + num_workers, layout, window_stride, data_columns, @@ -4303,7 +4220,7 @@ def _process_batches_single_file( data_name_root: str, data: pl.DataFrame, schema: Any, - n_cores: Optional[int], + num_workers: Optional[int], layout: StoredWindowLayout | list[StoredWindowLayout], window_stride: int, data_columns: list[str], @@ -4332,26 +4249,26 @@ def _process_batches_single_file( raise ValueError( "split_column values must be non-decreasing within each sequence" ) - n_cores = n_cores or multiprocessing.cpu_count() + num_workers = num_workers or multiprocessing.cpu_count() sequence_count = len(_sequence_run_bounds(data)[0]) maximum_batches = data.height if allow_sequence_splitting else sequence_count - n_cores = min(n_cores, maximum_batches) + num_workers = min(num_workers, maximum_batches) if ( not normalize_on_all_data and allow_sequence_splitting and split_method == "within_sequence" - and n_cores > sequence_count + and num_workers > sequence_count ): raise ValueError( "Unsafe preprocessing configuration: normalize_on_all_data=False, " "allow_sequence_splitting=True, and split_method='within_sequence' " "would split sequences across worker batches " - f"(effective n_cores={n_cores}, sequences={sequence_count}). This " + f"(effective num_workers={num_workers}, sequences={sequence_count}). This " "would make split-0 metadata fitting disagree with the materialized " - "training split. Set allow_sequence_splitting=False, reduce n_cores " + "training split. Set allow_sequence_splitting=False, reduce num_workers " f"to at most {sequence_count}, or set normalize_on_all_data=True." ) - batch_limits = get_batch_limits(data, n_cores, allow_sequence_splitting) + batch_limits = get_batch_limits(data, num_workers, allow_sequence_splitting) valid_batch_limits = [(s, e) for s, e in batch_limits if (e - s) > 0] batches = [ ( @@ -4421,92 +4338,6 @@ def _create_preprocess_pool(processes: int) -> Any: os.environ[name] = value -@beartype -def get_combined_statistics( - n1: int, mean1: float, std1: float, n2: int, mean2: float, std2: float -) -> tuple[float, float]: - """Combine two mean/std summaries.""" - if n1 == 0: - return mean2, std2 - if n2 == 0: - return mean1, std1 - - scale = max(abs(mean1), abs(mean2), abs(std1), abs(std2)) - if not math.isfinite(scale): - raise ValueError("Cannot combine non-finite normalization statistics.") - if scale == 0.0: - return 0.0, 0.0 - - scaled_mean1 = mean1 / scale - scaled_mean2 = mean2 / scale - scaled_std1 = std1 / scale - scaled_std2 = std2 / scale - combined_mean_scaled = (n1 * scaled_mean1 + n2 * scaled_mean2) / (n1 + n2) - combined_mean = combined_mean_scaled * scale - - if n1 + n2 <= 1: - return combined_mean, 0.0 - - sum_of_squares1 = (n1 - 1) * scaled_std1**2 + n1 * ( - scaled_mean1 - combined_mean_scaled - ) ** 2 - sum_of_squares2 = (n2 - 1) * scaled_std2**2 + n2 * ( - scaled_mean2 - combined_mean_scaled - ) ** 2 - - combined_std = ( - math.sqrt((sum_of_squares1 + sum_of_squares2) / (n1 + n2 - 1)) * scale - ) - if not math.isfinite(combined_mean) or not math.isfinite(combined_std): - raise ValueError( - "Combined numeric range is too large for finite normalization statistics." - ) - - return combined_mean, combined_std - - -@beartype -def create_id_map(data: pl.DataFrame, column: str) -> dict[Union[str, int], int]: - """Map sorted user values to IDs after reserved tokens.""" - ids = sorted( - [int(x) if not isinstance(x, str) else x for x in np.unique(data[column])] - ) # type: ignore - - if isinstance(ids[0], str): - if SPECIAL_TOKEN_IDS.labels_by_id[SPECIAL_TOKEN_IDS.mask] in ids: - raise ValueError( - f"Found value '{SPECIAL_TOKEN_IDS.labels_by_id[SPECIAL_TOKEN_IDS.mask]}' in {column}, this is invalid" - ) - - for special_val in [ - SPECIAL_TOKEN_IDS.labels_by_id[SPECIAL_TOKEN_IDS.unknown], - SPECIAL_TOKEN_IDS.labels_by_id[SPECIAL_TOKEN_IDS.other], - ]: - if special_val in ids: - warnings.warn( - f"Found special value {special_val} in {column}, these will be combined with the sequifier-internal special value {special_val}" - ) - ids = [ - id_ - for id_ in ids - if id_ - not in [ - SPECIAL_TOKEN_IDS.labels_by_id[SPECIAL_TOKEN_IDS.unknown], - SPECIAL_TOKEN_IDS.labels_by_id[SPECIAL_TOKEN_IDS.other], - ] - ] - id_map = {id_: i + SPECIAL_TOKEN_IDS.user_start for i, id_ in enumerate(ids)} - id_map[SPECIAL_TOKEN_IDS.labels_by_id[SPECIAL_TOKEN_IDS.unknown]] = ( - SPECIAL_TOKEN_IDS.unknown - ) - id_map[SPECIAL_TOKEN_IDS.labels_by_id[SPECIAL_TOKEN_IDS.other]] = ( - SPECIAL_TOKEN_IDS.other - ) - else: - id_map = {id_: i + SPECIAL_TOKEN_IDS.user_start for i, id_ in enumerate(ids)} - return dict(id_map) - - @beartype def get_batch_limits( data: pl.DataFrame, n_batches: int, allow_sequence_splitting: bool diff --git a/src/sequifier/rl/__init__.py b/src/sequifier/rl/__init__.py new file mode 100644 index 000000000..bb67d4b93 --- /dev/null +++ b/src/sequifier/rl/__init__.py @@ -0,0 +1 @@ +"""Local categorical policy learning, independent of forecast alignment.""" diff --git a/src/sequifier/rl/action_plan.py b/src/sequifier/rl/action_plan.py new file mode 100644 index 000000000..9a4789664 --- /dev/null +++ b/src/sequifier/rl/action_plan.py @@ -0,0 +1,243 @@ +"""Compile the policy binding against the model's decoder branches.""" + +from collections.abc import Iterator, Mapping +from dataclasses import dataclass, field + +import torch +from torch import Tensor + +from sequifier.config.policy_config import ActionFieldConfig +from sequifier.io.actions import ( + action_component_key, + action_latent_key, + decode_action_value, + inactive_action, +) +from sequifier.io.preprocessing import normalize_values +from sequifier.io.values import DomainValue +from sequifier.model.decoders import AutoregressiveTransformerDecoderBranch + + +@dataclass +class ActionDecision: + """Batched executable values and the real-valued data needed to replay them.""" + + values: dict[str, Tensor] = field(default_factory=dict) + latents: dict[str, Tensor] = field(default_factory=dict) + components: dict[str, Tensor] = field(default_factory=dict) + + def cpu(self) -> "ActionDecision": + return ActionDecision( + {name: value.cpu() for name, value in self.values.items()}, + {name: value.cpu() for name, value in self.latents.items()}, + {name: value.cpu() for name, value in self.components.items()}, + ) + + +@dataclass(frozen=True) +class ActionField: + spec: ActionFieldConfig + inactive: DomainValue + activity: dict[str, tuple[DomainValue, ...]] + support: tuple[int, ...] + latent_key: str | None + component_key: str | None + + +class ActionExecutionPlan: + def __init__(self, actor, decoder): + interface = actor.interface + actions = actor.config.actions + dependencies = {name: set(field.active_when) for name, field in actions.items()} + self.independent = {} + self.prefix_steps = {} + self.has_autoregression = False + for branch_name, branch in decoder.branches.items(): + targets = tuple(name for name in branch.target_columns if name in actions) + if not targets: + continue + if isinstance(branch, AutoregressiveTransformerDecoderBranch): + if tuple(branch.target_columns[: len(targets)]) != targets: + raise ValueError( + "autoregressive actions must include all preceding decoder targets" + ) + self.has_autoregression |= len(targets) > 1 + previous = None + for name in targets: + self.prefix_steps[name] = (branch_name, previous) + if previous is not None: + dependencies[name].add(previous) + previous = name + else: + self.independent[branch_name] = targets + + pending = [name for name in interface.target_columns if name in actions] + if set(pending) != set(actions): + raise ValueError("policy actions must name existing model targets") + order = [] + resolved = set() + while pending: + ready = next( + (name for name in pending if dependencies[name] <= resolved), None + ) + if ready is None: + raise ValueError( + "action activity/decoder dependencies are cyclic or unknown" + ) + order.append(ready) + resolved.add(ready) + pending.remove(ready) + + self.order = tuple(order) + self.position = actor.config.decision_position + self.fields = {} + record_fields = set(actions) + for name in self.order: + spec = actions[name] + if spec.type != interface.target_column_types[name]: + raise ValueError(f"{name}: action type differs from the model target") + if spec.scale == "model" and name not in interface.prediction_distributions: + raise ValueError( + f"{name}: scale: model requires a model prediction_distributions entry" + ) + support = () + latent_key = component_key = None + if spec.type == "categorical": + decoder_indices = { + global_id: index + for index, global_id in enumerate( + interface.target_decoder_ids[name] + ) + } + support = tuple( + decoder_indices[i] for i in actor.action_decoder_ids(name) + ) + if spec.active_when and spec.inactive_value not in spec.values: + raise ValueError( + f"{name}: inactive categorical value must be executable" + ) + else: + latent_key = action_latent_key(name, actor.config) + component_key = action_component_key(name, actor.config) + record_fields.update((latent_key, component_key)) + + activity = {} + for predecessor, labels in spec.active_when.items(): + parent = actions[predecessor] + if parent.type == "categorical": + if any(label not in parent.values for label in labels): + raise ValueError( + f"{name}: activity condition uses an unknown category" + ) + labels = [parent.values.index(label) for label in labels] + activity[predecessor] = tuple(labels) + self.fields[name] = ActionField( + spec, + inactive_action(spec), + activity, + support, + latent_key, + component_key, + ) + self.record_fields = frozenset(record_fields) + self.action_names = tuple(actions) + self._indices = {} + + def from_record(self, actions: Mapping[str, Tensor]) -> ActionDecision: + """Translate a table or trajectory batch into the internal decision.""" + if actions.keys() - self.record_fields: + raise ValueError("joint action contains unknown fields") + return ActionDecision( + {name: actions[name] for name in self.action_names if name in actions}, + { + name: actions[field.latent_key] + for name, field in self.fields.items() + if field.latent_key is not None and field.latent_key in actions + }, + { + name: actions[field.component_key] + for name, field in self.fields.items() + if field.component_key is not None and field.component_key in actions + }, + ) + + def to_record(self, decision: ActionDecision) -> dict[str, Tensor]: + """Pack a decision in the existing flat action record format.""" + actions = {name: decision.values[name] for name in self.order} + for name in self.order: + field = self.fields[name] + if field.latent_key is not None and name in decision.latents: + actions[field.latent_key] = decision.latents[name] + if field.component_key is not None and name in decision.components: + actions[field.component_key] = decision.components[name] + return actions + + def domain_action( + self, decision: ActionDecision, index: int = 0 + ) -> dict[str, DomainValue]: + return { + name: decode_action_value( + decision.values[name][index].item(), self.fields[name].spec + ) + for name in self.action_names + } + + def outputs( + self, decoder, context, valid, statistics, decision: ActionDecision + ) -> Iterator[tuple[str, Tensor]]: + """Yield each decoder output using the values chosen at earlier steps.""" + support, _ = self.indices(context.device) + outputs = {} + for branch_name, targets in self.independent.items(): + branch = decoder.branches[branch_name] + outputs.update(branch.decode_targets(context, targets)) + prefixes = {} + for name in self.order: + if name not in self.prefix_steps: + yield name, outputs[name] + continue + branch_name, previous = self.prefix_steps[name] + branch = decoder.branches[branch_name] + if previous is None: + prefix = branch.start_prefix(context) + else: + teacher = self._teacher( + decision.values.get(previous), + support.get(previous), + valid, + statistics.get(previous), + ) + prefix = branch.append_prefix(prefixes[branch_name], previous, teacher) + prefixes[branch_name] = prefix + yield name, branch.decode_prefix(prefix, (name,))[name] + + @staticmethod + def _teacher(value, support, valid, stats): + if value is None: + dtype = torch.long if support is not None else torch.float32 + return valid.new_zeros(valid.shape, dtype=dtype) + if support is not None: + return support[value.long()].masked_fill(~valid, 0) + teacher = normalize_values(value, stats) if stats is not None else value + return teacher.float().expand(valid.shape[0]) + + def indices(self, device): + # The shared network can move independently of its policy view. + if device not in self._indices: + # Replay autograd can reuse indices first materialized in inference. + with torch.inference_mode(False): + support = { + name: torch.tensor(field.support, dtype=torch.long, device=device) + for name, field in self.fields.items() + if field.support + } + activity = { + name: { + parent: torch.tensor(values, device=device) + for parent, values in field.activity.items() + } + for name, field in self.fields.items() + if field.activity + } + self._indices[device] = support, activity + return self._indices[device] diff --git a/src/sequifier/rl/attachment.py b/src/sequifier/rl/attachment.py new file mode 100644 index 000000000..a44fcc0b2 --- /dev/null +++ b/src/sequifier/rl/attachment.py @@ -0,0 +1,137 @@ +"""Attach a policy task to a resolved model without rebuilding its interface.""" + +from sequifier.config.policy_config import ResolvedPolicyBinding +from sequifier.config.policy_inputs import PolicyInputLayout +from sequifier.model.execution_schema import ExecutionSchema + + +def resolve_binding(execution, interface_name, binding): + interface = execution.interface(interface_name) + schema = ExecutionSchema.from_interface(interface, execution.options.context_length) + if ( + -binding.decision_position + > execution.options.context_length - interface.decoder.support_length + 1 + ): + raise ValueError("decision_position lies outside the source decoder output") + values = binding.model_dump() + actions = dict(binding.actions) + if not actions: + raise ValueError( + "model attachment requires named actions with explicit real scales" + ) + for name, field in actions.items(): + if name not in interface.target_columns: + raise ValueError(f"{name}: policy action is not a model target") + if field.scale == "model" and name not in interface.prediction_distributions: + raise ValueError( + f"{name}: scale: model requires a model prediction_distributions entry" + ) + if field.type != interface.target_column_types[name]: + raise ValueError(f"{name}: action type differs from the model") + if field.type == "categorical" and not field.values: + inverse = {v: k for k, v in interface.id_maps.get(name, {}).items()} + actions[name] = field.model_copy( + update={ + "values": [ + inverse.get(i, i) for i in interface.target_decoder_ids[name] + ] + } + ) + static = [ + item.key + for item in schema.inputs + if item.role == "static_feature" or item.key == "__preconditioning_image" + ] + temporal = [ + item.key + for item in schema.inputs + if item.key not in static and item.key != "attention_valid_mask" + ] + conditioned = binding.return_conditioned + if conditioned and "__policy_return" not in temporal: + raise ValueError( + "return conditioning requires an explicit __policy_return model input" + ) + layout = PolicyInputLayout.select( + temporal, + static, + binding, + [k for k in interface.categorical_columns if k in temporal], + interface.n_classes, + interface.id_maps, + ) + values.update( + **layout.binding_values(), + actions=actions, + action_order=[name for name in interface.target_columns if name in actions], + return_conditioned=conditioned, + input_schema=schema.to_dict(), + input_statistics={ + name: stats + for name, stats in interface.selected_columns_statistics.items() + if name in set(temporal) | set(static) and interface.normalize_real_columns + }, + context_length=execution.options.context_length, + interface_name=interface_name, + ) + for column in binding.previous_action_inputs: + if column not in temporal: + raise ValueError(f"{column}: previous action binding is not a model input") + return ResolvedPolicyBinding.model_validate(values) + + +def apply_binding(execution, interface_name, policy): + """Attach task behavior without changing model-owned distribution parameters.""" + interface = execution.interface(interface_name) + for name, action in policy.actions.items(): + if action.scale == "model" and name not in interface.prediction_distributions: + raise ValueError( + f"{name}: scale: model requires a model prediction_distributions entry" + ) + interface.policy = policy + + +def attach_policy(network, execution, interface_name, binding): + from sequifier.rl.policy import Policy + + original_parameters = {name for name, _ in network.named_parameters()} + policy = resolve_binding(execution, interface_name, binding) + apply_binding(execution, interface_name, policy) + actor = Policy.from_network(network, execution, interface_name) + actor.transfer_report = attachment_report(actor, original_parameters) + return actor + + +def attachment_report(actor, original_parameters): + policy = actor.config + return { + "mode": "adapt" + if policy.previous_action_inputs + or any( + field.lower is not None or field.upper is not None + for field in policy.actions.values() + ) + or any( + field.type == "categorical" + and set(actor.action_decoder_ids(name)) + != set(actor.interface.target_decoder_ids[name]) + for name, field in policy.actions.items() + ) + else "attach", + "interface": actor.interface_name, + "reused_parameters": [ + name + for name, _ in actor.network.named_parameters() + if name in original_parameters + ], + "new_parameters": [ + name + for name, _ in actor.network.named_parameters() + if name not in original_parameters + ], + "source_preprocessing": "preserved", + "action_support": { + name: field.model_dump(mode="json") + for name, field in policy.actions.items() + }, + } diff --git a/src/sequifier/rl/batch_order.py b/src/sequifier/rl/batch_order.py new file mode 100644 index 000000000..cb916d328 --- /dev/null +++ b/src/sequifier/rl/batch_order.py @@ -0,0 +1,76 @@ +"""Resumable dataset permutations at complete accumulation boundaries.""" + +import torch + + +class BatchOrder: + def __init__(self, size, group_size, seed, *, sample=None, state=None): + if size < 1 or group_size < 1: + raise ValueError("batch order requires positive dataset and group sizes") + self.size, self.group_size, self.sample = size, group_size, sample + self.generator = torch.Generator().manual_seed(seed) + self.epoch, self.position, self.order = 0, 0, None + if state is not None: + self.epoch = state["epoch"] + self.position = state["position"] + self.order = state["order"] + self._validate() + self.generator.set_state(state["generator"]) + + @property + def transactions(self): + per_epoch = (self.size + self.group_size - 1) // self.group_size + return self.epoch * per_epoch + self.position // self.group_size + + def _validate(self): + if ( + not isinstance(self.epoch, int) + or isinstance(self.epoch, bool) + or self.epoch < 0 + ): + raise ValueError("checkpoint sampler epoch is invalid") + if ( + not isinstance(self.position, int) + or isinstance(self.position, bool) + or not 0 <= self.position < self.size + or self.position % self.group_size + ): + raise ValueError( + "checkpoint sampler is not a complete accumulation boundary" + ) + if self.order is None: + if self.position: + raise ValueError("checkpoint sampler has a position without an order") + elif ( + not isinstance(self.order, list) + or any( + not isinstance(index, int) or isinstance(index, bool) + for index in self.order + ) + or sorted(self.order) != list(range(self.size)) + ): + raise ValueError("checkpoint sampler order is not the dataset permutation") + elif not self.position: + raise ValueError("checkpoint sampler has an unconsumed order") + + def next(self): + if self.order is None: + self.order = ( + list(self.sample(self.generator)) + if self.sample is not None + else torch.randperm(self.size, generator=self.generator).tolist() + ) + group = self.order[self.position : self.position + self.group_size] + self.position += len(group) + if self.position == self.size: + self.epoch += 1 + self.position, self.order = 0, None + return group + + def state_dict(self): + return { + "epoch": self.epoch, + "position": self.position, + "order": self.order, + "generator": self.generator.get_state(), + } diff --git a/src/sequifier/rl/batching.py b/src/sequifier/rl/batching.py new file mode 100644 index 000000000..432fff644 --- /dev/null +++ b/src/sequifier/rl/batching.py @@ -0,0 +1,24 @@ +"""Batch mechanics without algorithm-specific weighting or sampling policies.""" + +import torch + + +def batches(values, size): + for start in range(0, len(values), size): + yield values[start : start + size] + + +def shuffled_groups(size, group_size, generator): + return batches(torch.randperm(size, generator=generator).tolist(), group_size) + + +class MetricTotals(dict): + def __init__(self, *names): + super().__init__(dict.fromkeys(names, 0.0)) + + def add(self, values, divisor=1): + for name in self: + self[name] += float(values[name].detach().sum()) / divisor + + def means(self, count): + return {name: total / count for name, total in self.items()} diff --git a/src/sequifier/rl/checkpoint.py b/src/sequifier/rl/checkpoint.py new file mode 100644 index 000000000..0f7727bba --- /dev/null +++ b/src/sequifier/rl/checkpoint.py @@ -0,0 +1,102 @@ +"""Policy views over shared portable models and execution checkpoints.""" + +from pathlib import Path + +import torch + +from sequifier.artifacts.model_artifact import ( + ModelArtifact, + build_model_artifact, + read_artifact, + restore_model_artifact, + save_model_artifact, +) +from sequifier.artifacts.phase import PhaseArtifacts +from sequifier.rl.policy import Policy + + +def phase_actor_path(directory, index, name): + return PhaseArtifacts.for_phase(directory, index, name).model + + +def validate_policy_weights(policy: Policy) -> None: + for name, value in policy.network.state_dict().items(): + if not torch.isfinite(value).all(): + raise ValueError(f"policy tensor {name!r} must be finite") + statistics = policy.interface.selected_columns_statistics + policy.set_normalization_statistics(statistics) + + +def save_policy( + policy: Policy, path: str | Path, *, metadata: dict | None = None +) -> None: + save_model_artifact(policy_model_artifact(policy, metadata=metadata), path) + + +def policy_model_artifact( + policy: Policy, + *, + metadata: dict | None = None, + weights: dict[str, torch.Tensor] | None = None, +) -> ModelArtifact: + validate_policy_weights(policy) + if weights is None: + weights = { + name: value.detach().cpu() + for name, value in policy.network.state_dict().items() + } + return build_model_artifact( + policy.network, + policy.execution_config, + provenance={ + **(metadata or {}), + "transfer": policy.transfer_report, + }, + state_dict=weights, + ) + + +def load_policy( + path: str | Path | ModelArtifact, + *, + device: str = "cpu", + interface_name: str | None = None, + binding=None, + model=None, +) -> Policy: + artifact = ( + path + if isinstance(path, ModelArtifact) + else read_artifact(path, expected_type="sequifier_model")[0] + ) + network, execution, _ = restore_model_artifact( + artifact, device=device, interface_name=interface_name + ) + if interface_name is None: + candidates = ( + list(execution.interface_names) + if binding is not None + else [ + name + for name, interface in execution.interfaces.items() + if interface.policy is not None + ] + ) + if len(candidates) != 1: + raise ValueError("Select one policy model_interface") + interface_name = candidates[0] + assert interface_name is not None + if model is not None: + from sequifier.artifacts.model_config import extend_prediction_distributions + + extend_prediction_distributions(execution, model) + if binding is not None: + from sequifier.rl.attachment import attach_policy + + policy = attach_policy(network, execution, interface_name, binding) + else: + policy = Policy.from_network(network, execution, interface_name) + policy.units = dict(policy.config.units) + validate_policy_weights(policy) + policy.eval() + return policy diff --git a/src/sequifier/rl/clipped_objective.py b/src/sequifier/rl/clipped_objective.py new file mode 100644 index 000000000..ffe6c9a19 --- /dev/null +++ b/src/sequifier/rl/clipped_objective.py @@ -0,0 +1,26 @@ +"""Elementwise clipped actor objective, independent of algorithm weighting.""" + +from dataclasses import dataclass + +import torch +from torch import Tensor + + +@dataclass(frozen=True) +class ClippedPolicyObjective: + loss: Tensor + ratio: Tensor + approximate_kl: Tensor + clip_fraction: Tensor + + +def clipped_policy_objective(log_probability, old_log_probability, advantages, epsilon): + log_ratio = log_probability.float() - old_log_probability.float() + ratio = log_ratio.exp() + clipped = ratio.clamp(1 - epsilon, 1 + epsilon) + return ClippedPolicyObjective( + loss=-torch.minimum(ratio * advantages, clipped * advantages), + ratio=ratio, + approximate_kl=ratio - 1 - log_ratio, + clip_fraction=((ratio - 1).abs() > epsilon).float(), + ) diff --git a/src/sequifier/rl/collection.py b/src/sequifier/rl/collection.py new file mode 100644 index 000000000..c0aed1fb8 --- /dev/null +++ b/src/sequifier/rl/collection.py @@ -0,0 +1,152 @@ +"""Independent seed streams and categorical online decision scoring.""" + +import hashlib +import math +from dataclasses import dataclass + +import torch + +from sequifier.contract_versions import ContractVersions +from sequifier.io.actions import action_record +from sequifier.rl.distributions import action_generator +from sequifier.rl.environment import validate_reset, validate_step +from sequifier.rl.history import PolicyHistory +from sequifier.rl.window_storage import tensor_storage_bytes +from sequifier.runtime.resource_accounting import allocation +from sequifier.trajectory.returns import remaining_return + + +def stream_seed(seed, stream, reset_count): + value = f"sequifier-policy-v{ContractVersions.POLICY_COLLECTION_SEED}:{seed}:{stream}:{reset_count}".encode() + return int.from_bytes(hashlib.blake2b(value, digest_size=8).digest(), "big") % ( + 2**63 + ) + + +@dataclass(frozen=True) +class PolicyDecision: + domain_action: dict + action: int | dict + log_probability: float + entropy: float + value: float + + +class PolicyCursor: + """Own causal history and action RNG for one live environment. + + Decision and execution are separate so collectors can score a reference and + account for issued actions before handing control to the environment. + """ + + def __init__( + self, + actor, + adapter, + units, + seed, + *, + strict_vocabulary=True, + defer_device_transfer=False, + ): + self.actor, self.adapter, self.units = actor, adapter, units + self.defer_device_transfer = defer_device_transfer + self.history = PolicyHistory(actor.config, strict_vocabulary=strict_vocabulary) + self.generator = action_generator(actor.device, seed) + self.desired_return = None + self.episode_return, self.episode_length, self.ended = 0.0, 0, True + self.last_result = None + self._inputs = None + self._window_storage = allocation("reconstructed_windows") + + @property + def inputs(self): + if self._inputs is None: + raise ValueError("policy cursor has no reconstructed inputs") + return self._inputs + + def refresh(self): + inputs = self.history.inputs() + moved = inputs if self.defer_device_transfer else inputs.to(self.actor.device) + if self._window_storage.accounting is not None: + seen = set() + size = tensor_storage_bytes(moved, seen) + self._window_storage.resize( + size + + tensor_storage_bytes(inputs, seen) + + tensor_storage_bytes(self._inputs, seen) + ) + self._window_storage.resize(size) + self._inputs = moved + + def close(self): + self._inputs = None + self.last_result = None + self.history.frames.clear() + self._window_storage.close() + + def reset(self, result, episode_id, desired_return=None): + result = validate_reset(result, self.units, compact_images=True) + if self.actor.config.return_conditioned and desired_return is None: + desired_return = self.actor.config.initial_desired_return + if desired_return is None: + raise ValueError( + "return-conditioned collection needs an initial desired return" + ) + self.history.reset( + result.observation, result.static_inputs, episode_id, desired_return + ) + self.desired_return = desired_return + self.episode_return, self.episode_length, self.ended = 0.0, 0, False + self.last_result = None + self.refresh() + + @torch.no_grad() + def decide(self, value=None, *, deterministic=False, compute_entropy=True): + """Score action and history-only V from exactly the same causal input.""" + actor = self.actor + actor.eval() + if value is not None: + value.eval() + distribution = actor.distribution(self.inputs) + features, metadata = distribution.features, distribution.metadata + selected = distribution.select_and_score(deterministic, self.generator) + estimate = ( + value.prepared_value(features, metadata) + if value is not None + else torch.zeros(1, device=actor.device) + ) + if not torch.isfinite(estimate).all(): + raise ValueError("non-finite collection value") + decision = selected.decision.cpu() + return PolicyDecision( + actor.action_plan.domain_action(decision), + action_record(actor.action_plan.to_record(decision)), + float(selected.log_prob[0]), + float(distribution.entropy(self.generator)[0]) if compute_entropy else 0.0, + float(estimate[0]), + ) + + def step(self, decision): + return self.accept_step(decision, self.adapter.step(decision.domain_action)) + + def accept_step(self, decision, result): + """Validate and commit exactly one simulator response.""" + result = validate_step(result, self.units, compact_images=True) + self.last_result = result + self.episode_return += float(result.reward) + self.episode_length += 1 + self.ended = result.terminated or result.truncated + if not math.isfinite(self.episode_return): + raise ValueError("non-finite collected episode return") + if self.actor.config.return_conditioned: + assert self.desired_return is not None + self.desired_return = remaining_return( + self.desired_return, result.reward, self.actor.config.returns.gamma + ) + if result.observation is not None: + self.history.advance( + decision.action, result.observation, self.desired_return + ) + self.refresh() + return result diff --git a/src/sequifier/rl/control.py b/src/sequifier/rl/control.py new file mode 100644 index 000000000..1494cd96a --- /dev/null +++ b/src/sequifier/rl/control.py @@ -0,0 +1,111 @@ +"""Minimal deterministic categorical control environment for local PPO examples. + +The observation alternates between -1 and +1. Matching its sign earns +1; +choosing the other action earns -1. An episode lasts eight actions by default. +A random policy has expected return zero and the optimal return is the horizon. +""" + +import random + +from sequifier.contract_versions import ContractVersions +from sequifier.rl.environment import ( + EnvironmentCapabilities, + OnlineEnvironmentAdapterBase, + OnlineResetResult, + OnlineStepResult, +) + + +class CategoricalControlAdapter(OnlineEnvironmentAdapterBase): + implementation_id = ( + f"sequifier.categorical-control.v{ContractVersions.CATEGORICAL_CONTROL}" + ) + capabilities = EnvironmentCapabilities( + seeded_reset=True, + exact_snapshot=True, + group_start="scenario", + reproducible_group_start=True, + group_start_randomness="independent", + ) + + def __init__(self, horizon=8): + if not isinstance(horizon, int) or isinstance(horizon, bool) or horizon <= 0: + raise ValueError("control horizon must be a positive integer") + self.horizon = horizon + self.step_id = 0 + self.cue = 1.0 + self.correct = 0 + self.ended = True + + def reset(self, seed): + self.cue = random.Random(seed).choice((-1.0, 1.0)) + self.step_id, self.correct = 0, 0 + self.ended = False + return OnlineResetResult({"cue": self.cue}) + + def reset_from_observation(self, initial, seed): + if ( + set(initial.observation) != {"cue"} + or initial.observation["cue"] not in {-1.0, 1.0} + or initial.static_inputs + or initial.units + ): + raise ValueError("control seed requires only a cue of -1 or +1") + self.reset(seed) + self.cue = float(initial.observation["cue"]) + return OnlineResetResult({"cue": self.cue}) + + def create_group_start(self, seed): + return {"cue": random.Random(seed).choice((-1.0, 1.0)), "horizon": self.horizon} + + def reset_group(self, token, rollout_seed): + if token.get("horizon") != self.horizon or token.get("cue") not in {-1.0, 1.0}: + raise ValueError("incompatible control group-start token") + self.step_id, self.correct, self.ended = 0, 0, False + self.cue = token["cue"] + return OnlineResetResult({"cue": self.cue}) + + def step(self, action): + if ( + self.ended + or set(action) != {"action"} + or action["action"] not in {"left", "right"} + ): + raise ValueError( + "control requires a live episode and one left/right action" + ) + correct = action["action"] == ("right" if self.cue > 0 else "left") + self.correct += int(correct) + self.step_id += 1 + self.cue = -self.cue + self.ended = self.step_id == self.horizon + return OnlineStepResult( + {"cue": self.cue}, + 1.0 if correct else -1.0, + terminated=self.ended, + success=self.correct == self.horizon if self.ended else None, + ) + + def snapshot_state(self): + return { + "horizon": self.horizon, + "step_id": self.step_id, + "cue": self.cue, + "correct": self.correct, + "ended": self.ended, + } + + def restore_state(self, state): + if ( + state["horizon"] != self.horizon + or not 0 <= state["step_id"] <= self.horizon + or state["cue"] not in {-1.0, 1.0} + or not 0 <= state["correct"] <= state["step_id"] + or not isinstance(state["ended"], bool) + ): + raise ValueError("incompatible control snapshot") + self.step_id, self.cue = state["step_id"], state["cue"] + self.correct, self.ended = state["correct"], state["ended"] + + def close(self): + pass diff --git a/src/sequifier/rl/distributions.py b/src/sequifier/rl/distributions.py new file mode 100644 index 000000000..23df7e77d --- /dev/null +++ b/src/sequifier/rl/distributions.py @@ -0,0 +1,98 @@ +"""The categorical action distribution used by learning, scoring, and execution.""" + +import torch +from torch import Tensor + + +def action_generator(device: torch.device, seed: int) -> torch.Generator: + return torch.Generator( + device=device if device.type == "cuda" else "cpu" + ).manual_seed(seed) + + +class CategoricalActionDistribution: + def __init__(self, logits: Tensor): + logits = logits.float() + if logits.ndim < 2 or logits.shape[-1] < 1 or not torch.isfinite(logits).all(): + raise ValueError( + "categorical policy logits must be finite with at least one action" + ) + self.logits = logits + self.distribution = torch.distributions.Categorical(logits=self.logits) + + def mode(self) -> Tensor: + return self.logits.argmax(dim=-1) + + def select(self, deterministic: bool, generator: torch.Generator) -> Tensor: + return self.mode() if deterministic else self.sample(generator) + + def sample( + self, + generator: torch.Generator + | list[torch.Generator] + | tuple[torch.Generator, ...] + | None = None, + ) -> Tensor: + """Keep probability computation batched, with optional per-row RNGs.""" + probabilities = self.distribution.probs + if isinstance(generator, (list, tuple)): + if len(generator) != probabilities.shape[0]: + raise ValueError("one action RNG is required per leading batch row") + if any(not isinstance(rng, torch.Generator) for rng in generator): + raise TypeError("batched action RNGs must be torch.Generator instances") + if not generator: + return torch.empty( + probabilities.shape[:-1], + device=probabilities.device, + dtype=torch.int64, + ) + sampled = torch.cat( + [ + torch.multinomial( + probabilities[index : index + 1] + .to(rng.device) + .reshape(-1, probabilities.shape[-1]), + 1, + generator=rng, + ).to(probabilities.device) + for index, rng in enumerate(generator) + ] + ) + return sampled.reshape(probabilities.shape[:-1]) + # MPS has no device-specific Generator; sample on its generator's device. + sampling_device = ( + generator.device if generator is not None else probabilities.device + ) + sampled = torch.multinomial( + probabilities.to(sampling_device).reshape(-1, probabilities.shape[-1]), + 1, + generator=generator, + ).to(probabilities.device) + return sampled.reshape(probabilities.shape[:-1]) + + def log_prob(self, actions: Tensor) -> Tensor: + if actions.dtype != torch.int64 or actions.shape != self.logits.shape[:-1]: + raise ValueError( + "categorical action IDs must be int64 and match the logits batch shape" + ) + if (actions < 0).any() or (actions >= self.logits.shape[-1]).any(): + raise ValueError("categorical action ID is out of range") + return self.distribution.log_prob(actions) + + def entropy(self) -> Tensor: + return self.distribution.entropy() + + +def masked_action_nll_loss( + distribution: CategoricalActionDistribution, actions: Tensor, valid_mask: Tensor +) -> Tensor: + if ( + valid_mask.dtype != torch.bool + or valid_mask.shape != actions.shape + or actions.shape != distribution.logits.shape[:-1] + ): + raise ValueError("action loss masks must be boolean and match actions/logits") + if not valid_mask.any(): + raise ValueError("action likelihood requires at least one labeled decision") + selected = CategoricalActionDistribution(distribution.logits[valid_mask]) + return -selected.log_prob(actions[valid_mask]).mean() diff --git a/src/sequifier/rl/dpo.py b/src/sequifier/rl/dpo.py new file mode 100644 index 000000000..72c7949a7 --- /dev/null +++ b/src/sequifier/rl/dpo.py @@ -0,0 +1,128 @@ +"""Canonical sigmoid DPO with bounded differentiable sequence scoring.""" + +import torch +from torch.nn import functional as F + +from sequifier.config.policy_plan import DPOPhaseConfig +from sequifier.io.preference_dataset import PreferenceData +from sequifier.rl.batch_order import BatchOrder +from sequifier.rl.run_state import PhaseContext, PhaseSession +from sequifier.rl.scoring import sequence_log_probabilities, sequence_score + + +def dpo_loss(chosen, rejected, reference_chosen, reference_rejected, beta): + margin = (chosen.float() - rejected.float()) - ( + reference_chosen.float() - reference_rejected.float() + ) + loss = -F.logsigmoid(beta * margin) + if not torch.isfinite(margin).all() or not torch.isfinite(loss).all(): + raise ValueError("non-finite DPO margin/loss") + return loss, margin + + +def pair_scores(actor, reference, dataset, pair, chunk_size): + scores = [] + for policy in (actor, reference): + for span in (pair.chosen, pair.rejected): + scores.append( + sequence_score( + policy, dataset.episode(span), span.start, span.end, chunk_size + ) + ) + return scores[0], scores[1], scores[2], scores[3] + + +@torch.no_grad() +def evaluate_dpo(actor, reference, dataset, phase): + totals = { + key: 0.0 + for key in ( + "dpo_loss", + "margin", + "preference_accuracy", + "chosen_length", + "rejected_length", + ) + } + for pair in dataset.pairs: + loss, margin = dpo_loss( + *pair_scores(actor, reference, dataset, pair, phase.scoring_chunk_size), + phase.beta, + ) + totals["dpo_loss"] += float(loss) + totals["margin"] += float(margin) + totals["preference_accuracy"] += float(margin > 0) + totals["chosen_length"] += pair.chosen.end - pair.chosen.start + totals["rejected_length"] += pair.rejected.end - pair.rejected.start + return {key: value / len(dataset) for key, value in totals.items()} + + +def run_dpo_phase(context: PhaseContext[DPOPhaseConfig, PreferenceData]): + actor, phase, data, reference = ( + context.actor.eval(), + context.phase, + context.data, + context.reference, + ) + if reference is None: + raise ValueError("DPO requires a frozen reference actor") + session = PhaseSession(context) + state = session.state + if not 0 <= session.transaction <= phase.iterations: + raise ValueError("DPO checkpoint exceeds authored update budget") + batches = BatchOrder( + len(data.training), + phase.batch_size * phase.accumulation_batches, + context.identity.sampling_seed, + state=state["batch_order"] if state else None, + ) + if ( + session.transaction != batches.transactions + or session.steps["actor"] != session.transaction + ): + raise ValueError("DPO sampler position disagrees with optimizer transaction") + + def evaluate(**record): + session.evaluate_validation( + lambda: evaluate_dpo(actor, reference, data.validation, phase), **record + ) + + if session.should_evaluate_initial(): + evaluate(initial=True) + for update in range(session.transaction + 1, phase.iterations + 1): + if session.stopping.triggered: + break + group = batches.next() + session.zero_grad() + loss_total = 0.0 + for index in group: + pair = data.training.pairs[index] + # The derivative of the scalar DPO margin is known. First score without + # graphs, then backpropagate each causal chunk with that exact derivative. + # This bounds activation memory even for arbitrarily long alternatives. + with torch.no_grad(): + scores = pair_scores( + actor, reference, data.training, pair, phase.scoring_chunk_size + ) + loss, margin = dpo_loss(*scores, phase.beta) + coefficient = ( + -phase.beta * torch.sigmoid(-phase.beta * margin) / len(group) + ) + loss_total += float(loss) / len(group) + for span, sign in ((pair.chosen, 1), (pair.rejected, -1)): + episode = data.training.episode(span) + for chunk in sequence_log_probabilities( + actor, episode, span.start, span.end, phase.scoring_chunk_size + ): + (sign * coefficient * chunk.sum()).backward() + session.step() + session.complete_update( + final=update == phase.iterations, + training={"training_dpo_loss": loss_total}, + evaluate=evaluate, + checkpoint={"batch_order": batches.state_dict()}, + ) + return session.finish( + {"optimizer_steps": session.steps["actor"]}, + {"data": data.contract, "reference_weight_hash": context.reference_identity}, + ) diff --git a/src/sequifier/rl/environment.py b/src/sequifier/rl/environment.py new file mode 100644 index 000000000..6eedf1f35 --- /dev/null +++ b/src/sequifier/rl/environment.py @@ -0,0 +1,278 @@ +"""Explicit capabilities for local PPO/GRPO environments and seeded rollout bridges.""" + +import json +from dataclasses import dataclass, field +from typing import Any, Literal, Mapping, Protocol + +import numpy as np + +from sequifier.contract_versions import ( + ONLINE_POLICY_PROTOCOL, + ContractVersions, + require_contract_version, +) +from sequifier.fingerprints import fingerprint +from sequifier.integration.factories import load_factory +from sequifier.io.values import ( + DomainValue, + numeric_scalar, + portable_fields, + portable_value, +) + + +@dataclass(frozen=True) +class OnlineResetResult: + observation: Mapping[str, Any] + static_inputs: Mapping[str, Any] = field(default_factory=dict) + units: Mapping[str, str] = field(default_factory=dict) + + +@dataclass(frozen=True) +class OnlineStepResult: + # None is permitted only at a true terminal, where no bootstrap is requested. + observation: Mapping[str, Any] | None + reward: float + terminated: bool = False + truncated: bool = False + units: Mapping[str, str] = field(default_factory=dict) + success: bool | None = None + + +@dataclass(frozen=True) +class EnvironmentCapabilities: + seeded_reset: bool = False + exact_snapshot: bool = False + group_start: Literal["snapshot", "scenario"] | None = None + reproducible_group_start: bool = False + # captured: the start token fixes simulator RNG state; reset_group must still + # use its rollout seed for independent future environment randomness. + group_start_randomness: Literal["captured", "independent"] = "independent" + + def __post_init__(self): + if any( + (not isinstance(flag, bool)) + for flag in ( + self.seeded_reset, + self.exact_snapshot, + self.reproducible_group_start, + ) + ): + raise ValueError("environment capability flags must be booleans") + if self.group_start not in { + None, + "snapshot", + "scenario", + } or self.group_start_randomness not in {"captured", "independent"}: + raise ValueError("unsupported group-start capability") + if self.reproducible_group_start and self.group_start is None: + raise ValueError( + "reproducible_group_start requires a declared group_start capability" + ) + + +class OnlineEnvironmentAdapterBase: + """Supply the protocol metadata for an online policy adapter.""" + + protocol = ONLINE_POLICY_PROTOCOL + protocol_version = ContractVersions.ONLINE_POLICY_ADAPTER + + +class OnlineEnvironmentAdapter(Protocol): + protocol: str + protocol_version: int + implementation_id: str + capabilities: EnvironmentCapabilities + + def reset(self, seed: int) -> OnlineResetResult: ... + def reset_from_observation( + self, initial: OnlineResetResult, seed: int + ) -> OnlineResetResult: ... + def step(self, action: Mapping[str, DomainValue]) -> OnlineStepResult: ... + def close(self) -> None: ... + def snapshot_state(self) -> dict: ... + def restore_state(self, state: dict) -> None: ... + def create_group_start(self, seed: int) -> object: ... + def reset_group(self, token: object, rollout_seed: int) -> OnlineResetResult: ... + + +def load_online_adapter(spec): + adapter = load_factory(spec.factory)(**spec.config) + try: + validate_online_adapter(adapter) + return adapter + except BaseException: + try: + close = adapter.close + except AttributeError: + close = None + if callable(close): + close() + raise + + +def validate_online_adapter(adapter): + try: + protocol = adapter.protocol + except AttributeError: + protocol = None + if protocol != ONLINE_POLICY_PROTOCOL: + raise ValueError(f"adapter.protocol must be {ONLINE_POLICY_PROTOCOL!r}") + try: + protocol_version = adapter.protocol_version + except AttributeError: + protocol_version = None + require_contract_version( + "online policy environment adapter protocol", + protocol_version, + ContractVersions.ONLINE_POLICY_ADAPTER, + ) + try: + implementation_id = adapter.implementation_id + except AttributeError: + implementation_id = None + if not isinstance(implementation_id, str) or not implementation_id.strip(): + raise ValueError("online adapter requires a nonempty implementation_id") + try: + capabilities = adapter.capabilities + except AttributeError: + capabilities = None + if not isinstance(capabilities, EnvironmentCapabilities): + raise ValueError("online adapter requires explicit EnvironmentCapabilities") + try: + methods = (adapter.reset, adapter.step, adapter.close) + except AttributeError as error: + raise ValueError(f"online adapter requires {error.name}") from error + for name, method in zip(("reset", "step", "close"), methods): + if not callable(method): + raise ValueError(f"online adapter requires {name}") + + +def require_exact_snapshot(adapter): + if adapter.capabilities.exact_snapshot is not True: + raise ValueError("exact PPO resume requires capabilities.exact_snapshot") + try: + methods = (adapter.snapshot_state, adapter.restore_state) + except AttributeError as error: + raise ValueError(f"exact snapshot adapter requires {error.name}") from error + for name, method in zip(("snapshot_state", "restore_state"), methods): + if not callable(method): + raise ValueError(f"exact snapshot adapter requires {name}") + + +def require_group_start(adapter, *, exact=False): + if adapter.capabilities.group_start not in {"snapshot", "scenario"}: + raise ValueError( + "GRPO requires explicit group_start: snapshot or scenario capability; a repeated seed is insufficient" + ) + try: + methods = (adapter.create_group_start, adapter.reset_group) + except AttributeError as error: + raise ValueError(f"group-start adapter requires {error.name}") from error + for name, method in zip(("create_group_start", "reset_group"), methods): + if not callable(method): + raise ValueError(f"group-start adapter requires {name}") + if exact and adapter.capabilities.reproducible_group_start is not True: + raise ValueError( + "exact GRPO resume requires documented reproducible_group_start capability" + ) + + +def validate_start_token(token): + try: + # Tokens are opaque to Sequifier, but must survive portable JSON round trips. + encoded = json.dumps(token, allow_nan=False, sort_keys=True) + decoded = json.loads(encoded) + if decoded != token: + raise ValueError("group-start token changes after JSON serialization") + return decoded + except (TypeError, ValueError) as exc: + raise ValueError( + "group-start token must be finite JSON-serializable content" + ) from exc + + +def _observation_fields(values, compact_images): + if not compact_images: + return portable_fields(values) + return { + key: ( + value.copy() + if isinstance(value, np.ndarray) + and ( + (key.startswith("__image:") and value.dtype == np.uint8) + or (key.startswith("__image_mask:") and value.dtype == np.bool_) + ) + else portable_value(value) + ) + for key, value in values.items() + } + + +def validate_reset(result, units, *, compact_images=False): + if not isinstance(result, OnlineResetResult) or dict(result.units) != units: + raise ValueError( + "online reset requires an initial observation with the frozen units" + ) + return OnlineResetResult( + _observation_fields(result.observation, compact_images), + portable_fields(result.static_inputs), + dict(result.units), + ) + + +def validate_step(result, units, *, compact_images=False): + if not isinstance(result, OnlineStepResult): + raise ValueError( + "online step must return OnlineStepResult with an immediate scalar reward" + ) + if not isinstance(result.terminated, bool) or not isinstance( + result.truncated, bool + ): + raise ValueError("online boundary flags must be booleans") + if dict(result.units) != units: + raise ValueError("online observation units changed") + numeric_scalar(result.reward, "online reward") + if result.observation is None and not result.terminated: + raise ValueError( + "bootstrap requires the genuine successor, never an autoreset observation" + ) + if result.success is not None and not isinstance(result.success, bool): + raise ValueError("adapter success must be boolean when defined") + return OnlineStepResult( + ( + _observation_fields(result.observation, compact_images) + if result.observation is not None + else None + ), + float(result.reward), + result.terminated, + result.truncated, + dict(result.units), + result.success, + ) + + +def require_seeded_reset(adapter): + try: + reset = adapter.reset_from_observation + except AttributeError: + reset = None + if not adapter.capabilities.seeded_reset or not callable(reset): + raise ValueError( + "policy rollout requires seeded_reset and reset_from_observation(initial, seed)" + ) + + +def adapter_identity(adapter): + from dataclasses import asdict + + return { + "module": type(adapter).__module__, + "class": type(adapter).__qualname__, + "protocol": adapter.protocol, + "implementation_fingerprint": fingerprint( + [adapter.protocol_version, adapter.implementation_id] + ), + "capabilities": asdict(adapter.capabilities), + } diff --git a/src/sequifier/rl/group_collection.py b/src/sequifier/rl/group_collection.py new file mode 100644 index 000000000..fa6e7ce26 --- /dev/null +++ b/src/sequifier/rl/group_collection.py @@ -0,0 +1,215 @@ +"""Collection budgets, trajectory ownership and sealing shared by GRPO schedulers.""" + +from dataclasses import dataclass, field, replace + +import torch + +from sequifier.fingerprints import fingerprint +from sequifier.io.image_feature_cache import IMAGE_CACHE_PREFIX +from sequifier.rl.collection import PolicyCursor, stream_seed +from sequifier.rl.environment import OnlineStepResult, validate_start_token +from sequifier.rl.grpo_buffer import GRPOTrajectory, SealedGroupBuffer +from sequifier.rl.history import _json_payload_bytes + + +class GroupCollectionError(RuntimeError): + def __init__(self, message, executed_actions): + super().__init__( + f"{message}; discarded {executed_actions} uncommitted environment actions" + ) + self.executed_actions = executed_actions + + +@dataclass +class GroupTrajectory: + group: int + sample: int + token: object + cursor: PolicyCursor + frames: list = field(default_factory=list) + actions: list = field(default_factory=list) + rewards: list = field(default_factory=list) + old_scores: list = field(default_factory=list) + reference_scores: list = field(default_factory=list) + + def append( + self, frame, decision, reference_score, result: OnlineStepResult, reward_scale + ): + reward = float(result.reward) * reward_scale + if not torch.isfinite( + torch.tensor( + [reward, reference_score, decision.log_probability], dtype=torch.float32 + ) + ).all(): + raise ValueError("non-finite GRPO reward or fixed action score") + self.frames.append(frame) + self.actions.append(decision.action) + self.rewards.append(reward) + self.old_scores.append(decision.log_probability) + self.reference_scores.append(reference_score) + + def seal(self, generation, result: OnlineStepResult): + score = float(torch.tensor(self.rewards, dtype=torch.float32).sum()) + if not torch.isfinite(torch.tensor(score)): + raise ValueError("non-finite FP32 GRPO trajectory score") + return GRPOTrajectory( + self.group, + self.sample, + self.token, + self.cursor.history.current.episode_id, + tuple(self.frames), + self.cursor.actor.config.context_length, + tuple(self.actions), + tuple(self.rewards), + tuple(self.old_scores), + tuple(self.reference_scores), + "terminated" + if result.terminated + else "truncated" + if result.truncated + else "max_steps", + score, + 0.0, + generation, + ) + + +class GroupCollection: + def __init__( + self, + context, + adapter, + count, + generation, + env_steps, + restart_stream, + *, + storage=None, + ): + self.context, self.adapter = context, adapter + self.phase, self.generation = context.phase, generation + bound = count * self.phase.group_size * self.phase.max_steps + if ( + self.phase.max_env_steps is not None + and env_steps + bound > self.phase.max_env_steps + ): + raise ValueError( + "complete GRPO groups exceed max_env_steps (conservative group_size * max_steps bound)" + ) + if ( + bound * (2048 + 128 * len(context.actor.interface.input_columns)) + > self.phase.max_buffer_bytes + ): + raise ValueError("complete GRPO groups exceed the minimum buffer estimate") + self.seed = context.identity.grpo_seed(restart_stream) + self.used, self.executed = 0, 0 + self.initial = {} + self.cursors = set() + self.storage = storage + self.window_bytes = 0 + + def __enter__(self): + return self + + def __exit__(self, kind, error, traceback): + for cursor in self.cursors: + cursor.close() + self.cursors.clear() + if error is not None and isinstance(error, Exception): + raise GroupCollectionError(str(error), self.executed) from error + + def account(self, size, *, window=False): + self.used += size + if window: + self.window_bytes += size + if self.storage is not None: + self.storage.resize(self.used - self.window_bytes) + if self.used > self.phase.max_buffer_bytes: + raise ValueError("GRPO collection exceeded max_buffer_bytes") + + def start(self, group): + token = validate_start_token( + self.adapter.create_group_start( + stream_seed(self.seed, "group-start", group) + ) + ) + self.account(_json_payload_bytes(token) * self.phase.group_size + 1024) + return token + + def trajectory(self, group, sample, token, *, deferred=False): + cursor = PolicyCursor( + self.context.actor, + self.adapter, + self.context.run.units, + stream_seed(self.seed, f"group-{group}-actions", sample), + defer_device_transfer=deferred, + ) + self.cursors.add(cursor) + return GroupTrajectory(group, sample, token, cursor) + + def seal_trajectory(self, trajectory, result): + sealed = trajectory.seal(self.generation, result) + trajectory.cursor.close() + self.cursors.remove(trajectory.cursor) + return sealed + + def environment_seed(self, group, sample): + return stream_seed(self.seed, f"group-{group}-environment", sample) + + def validate_initial(self, trajectory): + cursor = trajectory.cursor + inputs = cursor.inputs + static = set(cursor.actor.config.static_columns) + # reset() has one valid position. Include the full shape while avoiding + # expansion of context_length copies of padded image pixels. + identity = fingerprint( + { + "features": { + key: { + "shape": list(value.shape), + "values": (value if key in static else value[:, -1:]).tolist(), + } + for key, value in inputs.inputs.items() + }, + "metadata": { + key: value.tolist() + for key, value in inputs.metadata.items() + if not key.startswith(IMAGE_CACHE_PREFIX) + }, + "units": dict(cursor.units), + } + ) + if ( + trajectory.group in self.initial + and self.initial[trajectory.group] != identity + ): + raise ValueError( + "GRPO group members have different canonical initial histories or units" + ) + self.initial[trajectory.group] = identity + + def seal_group(self, trajectories): + if len(trajectories) != self.phase.group_size: + raise ValueError("GRPO cannot seal an incomplete group") + trajectories = sorted(trajectories, key=lambda value: value.sample_index) + scores = torch.tensor([t.score for t in trajectories], dtype=torch.float32) + mean, spread = scores.mean(), scores.std(unbiased=False) + if not torch.isfinite(mean) or not torch.isfinite(spread): + raise ValueError( + "non-finite FP32 GRPO group mean or population standard deviation" + ) + advantages = (scores - mean) / (spread + self.phase.advantage_epsilon) + if not torch.isfinite(advantages).all(): + raise ValueError("non-finite GRPO group advantages") + return tuple( + replace(t, advantage=float(a)) for t, a in zip(trajectories, advantages) + ) + + def finish(self, groups): + return SealedGroupBuffer( + groups, + self.generation, + self.phase.group_size, + self.context.reference_identity, + self.used, + ), self.executed diff --git a/src/sequifier/rl/grpo.py b/src/sequifier/rl/grpo.py new file mode 100644 index 000000000..1e1f700f3 --- /dev/null +++ b/src/sequifier/rl/grpo.py @@ -0,0 +1,213 @@ +"""Actor-only GRPO runtime; checkpoints contain no live or reusable group.""" + +import statistics +from contextlib import ExitStack +from pathlib import Path +from time import perf_counter + +import torch +from loguru import logger + +from sequifier.config.policy_plan import GRPOPhaseConfig +from sequifier.contract_versions import ContractVersions +from sequifier.io.atomic import write_json_atomic +from sequifier.model.frozen_image_cache import ( + frozen_image_cache_scope, + image_cache_statistics, + release_image_cache_entries, +) +from sequifier.rl.environment import ( + adapter_identity, + load_online_adapter, + require_group_start, +) +from sequifier.rl.grpo_collector import collect_groups +from sequifier.rl.grpo_update import update_grpo +from sequifier.rl.run_state import PhaseContext, PhaseSession, require_equal +from sequifier.runtime.resource_accounting import allocation +from sequifier.trajectory.archives import collection_archive + + +def run_grpo_phase(context: PhaseContext[GRPOPhaseConfig, None]): + actor, phase = context.actor.eval(), context.phase + session = PhaseSession(context) + state = session.state + groups, env_steps = state.get("groups", 0), state.get("env_steps", 0) + restart_stream = state.get("restart_stream", 0) + if ( + not 0 <= groups <= phase.total_groups + or session.transaction + != (groups + phase.groups_per_iteration - 1) // phase.groups_per_iteration + ): + raise ValueError("GRPO checkpoint is not a complete update/group boundary") + if groups != min( + session.transaction * phase.groups_per_iteration, phase.total_groups + ): + raise ValueError("GRPO checkpoint group counter is not a complete iteration") + if ( + not groups * phase.group_size + <= env_steps + <= groups * phase.group_size * phase.max_steps + ): + raise ValueError("GRPO environment clock exceeds authored bounds") + actor.grpo_policy_version = session.transaction + adapter = load_online_adapter(phase.environment) + resources = ExitStack() + workers = None + try: + require_group_start(adapter, exact=context.run.resume.mode == "exact") + identity = adapter_identity(adapter) + if state: + require_equal(state["adapter_identity"], identity, "grpo.adapter_identity") + if context.run.resume.mode != "exact" and groups < phase.total_groups: + restart_stream += 1 + session.record_continuation( + mode="new_groups", + restart_stream=restart_stream, + trajectory_equivalent=False, + ) + + session.restore_random() + # Register cleanup before collection or evaluation can fail. Both + # optimizations are disabled for configurations that omit their opt-ins. + resources.enter_context( + frozen_image_cache_scope( + actor, context.reference, phase.image_feature_cache_bytes + ) + ) + if phase.simulator_workers > 1 and groups < phase.total_groups: + from sequifier.rl.simulator_workers import ( + SimulatorWorkers, + require_parallel_group_start, + ) + + require_parallel_group_start(adapter) + workers = SimulatorWorkers( + phase.environment, + min( + phase.simulator_workers, + phase.groups_per_iteration * phase.group_size, + ), + phase.worker_timeout_seconds, + phase.worker_threads, + identity, + ) + resources.callback(workers.close) + + if session.should_evaluate_initial(): + session.evaluate_online(adapter, initial=True) + while groups < phase.total_groups and not session.stopping.triggered: + count = min(phase.groups_per_iteration, phase.total_groups - groups) + with allocation("rollout_storage") as storage: + collection_start = perf_counter() + buffer, executed = collect_groups( + context, + adapter, + groups, + count, + session.transaction, + env_steps, + restart_stream, + workers=workers, + storage=storage, + ) + collection_seconds = perf_counter() - collection_start + update_start = perf_counter() + record = update_grpo(session, buffer) + if actor.device.type == "cuda": + torch.cuda.synchronize(actor.device) + update_seconds = perf_counter() - update_start + env_steps += executed + groups += count + session.transaction += 1 + final = groups == phase.total_groups + if phase.archive_directory: + with allocation("rollout_archive") as archive: + archive.resize(buffer.estimated_bytes) + write_json_atomic( + Path(phase.archive_directory) + / f"groups-{session.transaction:08d}.json", + collection_archive( + "grpo", + { + "format": "sequifier.grpo-archive", + "version": ContractVersions.GRPO_ARCHIVE, + "reference_identity": context.reference_identity, + "generation": buffer.generation, + "trajectories": [ + t.as_dict() for t in buffer.trajectories + ], + }, + ), + ) + scores = [t.score for t in buffer.trajectories] + record.update( + { + "transaction": session.transaction, + "groups": groups, + "env_steps": env_steps, + "policy_version": actor.grpo_policy_version, + "collected_policy_version": buffer.generation, + "score_mean": statistics.fmean(scores), + "score_std": statistics.pstdev(scores), + "zero_variance_groups": sum( + len({t.score for t in group}) == 1 + for group in buffer.groups + ), + "lengths": [len(t.actions) for t in buffer.trajectories], + "cutoffs": [t.cutoff_reason for t in buffer.trajectories], + "start_tokens": [ + group[0].start_token for group in buffer.groups + ], + "buffer_bytes": buffer.estimated_bytes, + "simulator_workers": len(workers) if workers is not None else 1, + "rollout_inference_batch_size": phase.rollout_inference_batch_size, + "collection_seconds": collection_seconds, + "collection_env_steps_per_second": ( + executed / collection_seconds + ), + "update_seconds": update_seconds, + } + ) + del buffer, scores + evaluation_start = perf_counter() + if session.should_evaluate(final): + session.evaluate_online(adapter) + evaluation_seconds = perf_counter() - evaluation_start + record["evaluation_seconds"] = evaluation_seconds + record.update(image_cache_statistics(actor, context.reference)) + release_image_cache_entries(actor, context.reference) + session.record_training(final=final, **record) + logger.info( + "GRPO iteration {}: {} steps, collection {:.2f}s ({:.1f} steps/s), " + "update {:.2f}s, evaluation {:.2f}s", + session.transaction, + executed, + collection_seconds, + executed / collection_seconds, + update_seconds, + evaluation_seconds, + ) + session.checkpoint( + final, + groups=groups, + env_steps=env_steps, + adapter_identity=identity, + restart_stream=restart_stream, + ) + finally: + try: + resources.close() + finally: + adapter.close() + return session.finish( + { + "optimizer_steps": session.steps["actor"], + "groups": groups, + "env_steps": env_steps, + }, + { + "reference_weight_hash": context.reference_identity, + "adapter": identity, + }, + ) diff --git a/src/sequifier/rl/grpo_buffer.py b/src/sequifier/rl/grpo_buffer.py new file mode 100644 index 000000000..19faa933d --- /dev/null +++ b/src/sequifier/rl/grpo_buffer.py @@ -0,0 +1,112 @@ +"""Bounded complete groups with reconstructable causal frames and fixed scores.""" + +from dataclasses import dataclass + +import torch + +from sequifier.io.actions import action_batch +from sequifier.rl.history import history_inputs +from sequifier.rl.window_storage import collated_windows + + +@dataclass(frozen=True) +class GRPOTrajectory: + group_index: int + sample_index: int + start_token: object + episode_id: str + frames: tuple + context_length: int + actions: tuple[int | dict, ...] + rewards: tuple[float, ...] + old_log_probabilities: tuple[float, ...] + reference_log_probabilities: tuple[float, ...] + cutoff_reason: str + score: float + advantage: float + generation: int + + def inputs(self, spec, step): + return history_inputs( + spec, self.frames[max(0, step + 1 - spec.context_length) : step + 1] + ) + + def batches(self, actor, chunk_size): + for start in range(0, len(self.actions), chunk_size): + end = min(start + chunk_size, len(self.actions)) + with collated_windows( + (self.inputs(actor.config, t) for t in range(start, end)), + actor.device, + ) as inputs: + yield ( + inputs, + action_batch( + self.actions[start:end], actor.config, device=actor.device + ), + torch.tensor( + self.old_log_probabilities[start:end], + dtype=torch.float32, + device=actor.device, + ), + torch.tensor( + self.reference_log_probabilities[start:end], + dtype=torch.float32, + device=actor.device, + ), + ) + del inputs + + def as_dict(self): + return { + **self.__dict__, + "frames": [frame.state_dict() for frame in self.frames], + "decision_mask": [True] * len(self.actions), + "history_lengths": [ + min(i + 1, self.context_length) for i in range(len(self.actions)) + ], + } + + +class SealedGroupBuffer: + def __init__( + self, groups, generation, group_size, reference_identity, estimated_bytes + ): + if not groups or any(len(group) != group_size for group in groups): + raise ValueError("GRPO requires complete groups") + self.groups, self.generation = tuple(tuple(g) for g in groups), generation + self.reference_identity, self.estimated_bytes = ( + reference_identity, + estimated_bytes, + ) + self.trajectories = tuple( + trajectory for group in self.groups for trajectory in group + ) + self.consumed = False + if any(t.generation != generation or not t.actions for t in self.trajectories): + raise ValueError("GRPO group buffer has stale/empty trajectories") + + indices = [group[0].group_index for group in self.groups] + if len(set(indices)) != len(indices): + raise ValueError("GRPO buffer repeats a group identity") + for group in self.groups: + if {t.sample_index for t in group} != set(range(group_size)) or any( + t.group_index != group[0].group_index + or t.start_token != group[0].start_token + for t in group + ): + raise ValueError("GRPO group has inconsistent start/sample identities") + + def require_fresh(self, generation, reference_identity): + if ( + self.consumed + or generation != self.generation + or reference_identity != self.reference_identity + ): + raise ValueError( + "GRPO buffer is consumed, stale or has a different reference" + ) + + def consume(self): + if self.consumed: + raise ValueError("GRPO buffer has already been consumed") + self.consumed = True diff --git a/src/sequifier/rl/grpo_collector.py b/src/sequifier/rl/grpo_collector.py new file mode 100644 index 000000000..18c5246cd --- /dev/null +++ b/src/sequifier/rl/grpo_collector.py @@ -0,0 +1,86 @@ +"""Sequential same-start groups with independent action/environment RNG streams.""" + +import torch + +from sequifier.io.actions import action_batch +from sequifier.rl.environment import require_group_start, validate_start_token +from sequifier.rl.group_collection import ( # noqa: F401 + GroupCollection, + GroupCollectionError, +) +from sequifier.rl.history import _json_payload_bytes, frame_storage_bytes + + +@torch.no_grad() +def collect_groups( + context, + adapter, + start_group, + count, + generation, + env_steps, + restart_stream=0, + *, + workers=None, + storage=None, +): + if workers is not None: + from sequifier.rl.parallel_grpo_collector import collect_parallel_groups + + return collect_parallel_groups( + context, + adapter, + workers, + start_group, + count, + generation, + env_steps, + restart_stream, + storage=storage, + ) + actor, reference, phase = context.actor.eval(), context.reference, context.phase + require_group_start(adapter) + with GroupCollection( + context, adapter, count, generation, env_steps, restart_stream, storage=storage + ) as collection: + groups = [] + for group_index in range(start_group, start_group + count): + token = collection.start(group_index) + trajectories = [] + for sample in range(phase.group_size): + state = collection.trajectory(group_index, sample, token) + cursor = state.cursor + cursor.reset( + adapter.reset_group( + validate_start_token(token), + collection.environment_seed(group_index, sample), + ), + f"group-{group_index}-sample-{sample}", + ) + collection.validate_initial(state) + result = None + for _ in range(phase.max_steps): + frame = cursor.history.current + collection.account(frame_storage_bytes(frame) + 1024) + decision = cursor.decide(compute_entropy=False) + collection.account(4 * _json_payload_bytes(decision.action)) + reference_score = float( + reference.evaluate_actions( + cursor.inputs, + action_batch( + [decision.action], actor.config, device=actor.device + ), + )[0] + ) + collection.executed += 1 + result = cursor.step(decision) + state.append( + frame, decision, reference_score, result, phase.reward_scale + ) + if result.terminated or result.truncated: + break + if result is None: + raise ValueError("GRPO requires a positive trajectory horizon") + trajectories.append(collection.seal_trajectory(state, result)) + groups.append(collection.seal_group(trajectories)) + return collection.finish(groups) diff --git a/src/sequifier/rl/grpo_update.py b/src/sequifier/rl/grpo_update.py new file mode 100644 index 000000000..b1e455b53 --- /dev/null +++ b/src/sequifier/rl/grpo_update.py @@ -0,0 +1,115 @@ +"""Trajectory-weighted clipped GRPO with a sampled frozen-reference penalty.""" + +import torch + +from sequifier.rl.batching import MetricTotals, shuffled_groups +from sequifier.rl.clipped_objective import clipped_policy_objective + + +def grpo_objective( + log_probability, old_log_probability, reference_log_probability, advantage, phase +): + policy = clipped_policy_objective( + log_probability, old_log_probability, advantage, phase.clip_epsilon + ) + reference_delta = reference_log_probability.float() - log_probability.float() + penalty = torch.expm1(reference_delta) - reference_delta + loss = policy.loss + phase.reference_kl_coefficient * penalty + old_kl = policy.approximate_kl + if any(not torch.isfinite(t).all() for t in (policy.ratio, penalty, loss, old_kl)): + raise ValueError( + "non-finite GRPO ratio, sampled reference penalty or objective" + ) + return loss, penalty, old_kl, policy.clip_fraction + + +@torch.no_grad() +def validate_old_scores(actor, buffer, chunk_size): + for trajectory in buffer.trajectories: + for inputs, actions, old, _ in trajectory.batches(actor, chunk_size): + if not torch.allclose( + actor.evaluate_actions(inputs, actions).float(), + old, + atol=1e-5, + rtol=1e-4, + ): + raise ValueError( + "GRPO reconstructed old-policy scores differ before first update" + ) + del inputs, actions, old, _ + + +def update_grpo(session, buffer): + context = session.context + actor, phase = context.actor.eval(), context.phase + generation = session.transaction + buffer.require_fresh(generation, context.reference_identity) + if actor.grpo_policy_version != generation: + raise ValueError("GRPO collection generation differs from actor") + validate_old_scores(actor, buffer, phase.scoring_chunk_size) + trajectories = buffer.trajectories + totals = MetricTotals( + "loss", + "sampled_reference_penalty", + "old_policy_approximate_kl", + "clip_fraction", + ) + scored, stopped = 0, False + size = phase.batch_size * phase.accumulation_batches + epoch = -1 + for epoch in range(phase.optimization_epochs): + for indices in shuffled_groups(len(trajectories), size, session.sampler): + if phase.old_policy_target_kl is not None: + old_kl = 0.0 + with torch.no_grad(): + for i in indices: + trajectory = trajectories[i] + for inputs, actions, old, ref in trajectory.batches( + actor, phase.scoring_chunk_size + ): + _, _, kl, _ = grpo_objective( + actor.evaluate_actions(inputs, actions), + old, + ref, + trajectory.advantage, + phase, + ) + old_kl += ( + float(kl.sum()) / len(trajectory.actions) / len(indices) + ) + del inputs, actions, old, ref + if old_kl > phase.old_policy_target_kl: + stopped = True + break + session.zero_grad() + for i in indices: + trajectory = trajectories[i] + for inputs, actions, old, ref in trajectory.batches( + actor, phase.scoring_chunk_size + ): + values = grpo_objective( + actor.evaluate_actions(inputs, actions), + old, + ref, + trajectory.advantage, + phase, + ) + ( + values[0].sum() / len(trajectory.actions) / len(indices) + ).backward() + totals.add(dict(zip(totals, values)), len(trajectory.actions)) + del inputs, actions, old, ref + scored += 1 + session.step() + if stopped: + break + if not scored: + raise ValueError("GRPO old-policy KL stopped all optimizer transactions") + buffer.consume() + actor.grpo_policy_version = generation + 1 + return { + **totals.means(scored), + "optimizer_steps": session.steps["actor"], + "optimization_epochs": epoch + 1, + "old_policy_kl_stopped": stopped, + } diff --git a/src/sequifier/rl/history.py b/src/sequifier/rl/history.py new file mode 100644 index 000000000..b002b0f42 --- /dev/null +++ b/src/sequifier/rl/history.py @@ -0,0 +1,258 @@ +"""Bounded causal state shared by online collection and policy execution.""" + +import json +from collections import deque +from copy import deepcopy +from dataclasses import dataclass +from functools import cached_property + +import numpy as np +import torch + +from sequifier.io.image_feature_cache import ( + IMAGE_CACHE_PREFIX, + image_feature_cache_enabled, + new_image_identity, +) +from sequifier.io.policy_window import PolicyWindowView +from sequifier.io.values import portable_fields + + +@dataclass(frozen=True) +class PolicyFrame: + observation: tuple + previous_action: int | dict | None + static_inputs: tuple + episode_id: str + step_id: int + desired_return: float | None = None + + @cached_property + def image_identities(self): + # Transient IDs are deliberately excluded from state_dict and archives. + return { + key.removeprefix("__image:"): torch.tensor( + [new_image_identity() for _ in value], dtype=torch.int64 + ) + for key, value in self.observation + if key.startswith("__image:") + } + + def _values(self): + return { + "observation": dict(self.observation), + "previous_action": self.previous_action, + "static_inputs": dict(self.static_inputs), + "episode_id": self.episode_id, + "step_id": self.step_id, + "desired_return": self.desired_return, + } + + def state_dict(self): + # Materialize portable lists only at checkpoint/archive boundaries. + return portable_fields(self._values()) + + @cached_property + def storage_bytes(self): + return 1024 + 4 * _json_payload_bytes(self._values()) + + @classmethod + def from_state_dict(cls, values): + return cls( + tuple(values["observation"].items()), + values["previous_action"], + tuple(values["static_inputs"].items()), + values["episode_id"], + values["step_id"], + values["desired_return"], + ) + + +def _json_payload_bytes(value): + """Size of default json.dumps(...).encode('utf-8') without expanding pixels.""" + if isinstance(value, np.ndarray) and value.ndim and value.size: + if value.dtype == np.uint8: + digits = ( + value.size + + np.count_nonzero(value >= 10) + + np.count_nonzero(value >= 100) + ) + elif value.dtype == np.bool_: + digits = 5 * value.size - np.count_nonzero(value) + else: + return _json_payload_bytes(value.tolist()) + # Each list contributes brackets; separators contribute two bytes. + parents = sum(int(np.prod(value.shape[:axis])) for axis in range(1, value.ndim)) + return int(digits + 2 * value.size + 2 * parents) + if isinstance(value, np.ndarray): + return _json_payload_bytes(value.tolist()) + if isinstance(value, dict): + return ( + 2 + + max(0, len(value) - 1) * 2 + + sum( + len(json.dumps({key: None}, allow_nan=False).encode("utf-8")) + - 6 + + _json_payload_bytes(item) + for key, item in value.items() + ) + ) + if isinstance(value, (list, tuple)): + return ( + 2 + + max(0, len(value) - 1) * 2 + + sum(_json_payload_bytes(item) for item in value) + ) + return len(json.dumps(value, allow_nan=False).encode("utf-8")) + + +def frame_storage_bytes(frame): + """Conservative host-object allowance plus the complete domain payload.""" + return frame.storage_bytes + + +def history_inputs(spec, frames): + frames = list(frames) + inputs = PolicyWindowView(spec).inputs( + [dict(frame.observation) for frame in frames], + [frame.previous_action for frame in frames], + [frame.desired_return for frame in frames] if spec.return_conditioned else None, + dict(frames[-1].static_inputs), + ) + if image_feature_cache_enabled(): + for view in frames[-1].image_identities: + identities = torch.stack([frame.image_identities[view] for frame in frames]) + padding = identities.new_zeros( + spec.context_length - len(frames), *identities.shape[1:] + ) + inputs.metadata[IMAGE_CACHE_PREFIX + view] = torch.cat( + [padding, identities] + )[None] + return inputs + + +class PolicyHistory: + def __init__(self, spec, *, strict_vocabulary=False): + self.spec = spec + self.strict_vocabulary = strict_vocabulary + self.frames = deque(maxlen=spec.context_length) + self.reset_count = 0 + + @property + def attention_valid_mask(self): + return (False,) * (self.spec.context_length - len(self.frames)) + (True,) * len( + self.frames + ) + + @property + def current(self): + if not self.frames: + raise ValueError("policy history has not been reset") + return self.frames[-1] + + def _frame( + self, observation, previous_action, static, episode_id, step_id, desired_return + ): + if ( + not isinstance(episode_id, str) + or not episode_id + or not isinstance(step_id, int) + or isinstance(step_id, bool) + or step_id < 0 + ): + raise ValueError( + "policy frame requires an episode ID and nonnegative integer step" + ) + if set(observation) != set(self.spec.observation_columns) or set(static) != set( + self.spec.static_columns + ): + raise ValueError( + "environment payload does not match the model input binding" + ) + owned = {} + for key, value in observation.items(): + if key.startswith("__image:") or key.startswith("__image_mask:"): + dtype = torch.uint8 if key.startswith("__image:") else torch.bool + # Preserve the materializer's dtype conversion and take one + # owned compact copy, instead of retaining nested pixel lists. + owned[key] = torch.as_tensor(value, dtype=dtype).cpu().numpy().copy() + else: + owned[key] = deepcopy(value) + frame = PolicyFrame( + tuple(owned.items()), + deepcopy(previous_action), + tuple(deepcopy(static).items()), + episode_id, + step_id, + desired_return, + ) + history_inputs(self.spec, [frame]) + return frame + + def reset(self, observation, static_inputs, episode_id, desired_return=None): + frame = self._frame( + observation, + None, + static_inputs, + episode_id, + 0, + desired_return, + ) + self.frames.clear() + self.frames.append(frame) + self.reset_count += 1 + + def advance(self, action, successor, desired_return=None): + current = self.current + frame = self._frame( + successor, + action, + dict(current.static_inputs), + current.episode_id, + current.step_id + 1, + desired_return, + ) + self.frames.append(frame) + + def inputs(self): + return history_inputs(self.spec, self.frames) + + def state_dict(self): + return { + "frames": [frame.state_dict() for frame in self.frames], + "reset_count": self.reset_count, + "attention_valid_mask": self.attention_valid_mask, + } + + def load_state_dict(self, values): + frames = values["frames"] + if not 0 < len(frames) <= self.spec.context_length: + raise ValueError("invalid checkpoint history length") + restored = [] + for values_frame in frames: + frame = PolicyFrame.from_state_dict(values_frame) + restored.append( + self._frame( + dict(frame.observation), + frame.previous_action, + dict(frame.static_inputs), + frame.episode_id, + frame.step_id, + frame.desired_return, + ) + ) + if any( + right.episode_id != left.episode_id + or right.step_id != left.step_id + 1 + or right.static_inputs != left.static_inputs + for left, right in zip(restored, restored[1:]) + ): + raise ValueError("checkpoint causal history is discontinuous") + self.frames.clear() + self.frames.extend(restored) + self.reset_count = int(values["reset_count"]) + if ( + self.reset_count < 1 + or tuple(values["attention_valid_mask"]) != self.attention_valid_mask + ): + raise ValueError("checkpoint history mask or reset count is invalid") diff --git a/src/sequifier/rl/inference.py b/src/sequifier/rl/inference.py new file mode 100644 index 000000000..6e66308c8 --- /dev/null +++ b/src/sequifier/rl/inference.py @@ -0,0 +1,282 @@ +"""CLI policy prediction and environment execution using the shared model lifecycle.""" + +from itertools import islice +from pathlib import Path +from typing import cast + +import polars as pl +import torch +from loguru import logger + +from sequifier.config.policy_infer_config import ResolvedPolicyInferenceConfig +from sequifier.helpers import configure_determinism, normalize_path, write_data +from sequifier.io.actions import encode_action +from sequifier.io.batch import SequifierInputs, collate_model_inputs +from sequifier.io.policy_window import PolicyWindowView +from sequifier.io.sequence import ordered_sequence_rows +from sequifier.rl.checkpoint import load_policy +from sequifier.rl.distributions import action_generator +from sequifier.rl.environment import load_online_adapter +from sequifier.rl.runtime import collect_policy_rollout +from sequifier.rollout.history import numeric_scalar +from sequifier.trajectory.schema import SeedHistory +from sequifier.trajectory.store import save_trajectory + + +def infer_policy(config: ResolvedPolicyInferenceConfig) -> None: + configure_determinism(config.seed, config.enforce_determinism) + policy = load_policy( + config._artifacts.get(config.model_paths[0]) + or normalize_path(config.model_paths[0], config.project_root), + device=config.device, + interface_name=config.model_interface, + ) + spec = policy.config + path = normalize_path(config.data_path, config.project_root) + data = pl.read_csv(path) if config.read_format == "csv" else pl.read_parquet(path) + if not len(data): + raise ValueError("policy inference input must be nonempty") + required = set(spec.observation_columns + spec.static_columns) + if spec.return_conditioned and config.desired_return is None: + required.add(config.desired_return_column) + if not spec.return_conditioned and config.desired_return is not None: + raise ValueError("behavior cloning has no desired-return input") + if required - set(data.columns): + raise ValueError( + f"policy inference is missing columns: {sorted(required - set(data.columns))}" + ) + + if config.rollout is not None: + infer_rollout(policy, data, config) + return + predictions = predict_policy(policy, data, config) + if config.output_path is None: + from sequifier.infer import inference_output_path + + destination = inference_output_path( + config.project_root, + config.write_format, + "predictions", + Path(config.model_paths[0]).stem, + 0, + config=config, + ) + else: + destination = normalize_path(config.output_path, config.project_root) + config.validate_output_destination(destination, label="output_path") + Path(destination).parent.mkdir(parents=True, exist_ok=True) + write_data(predictions, destination, config.write_format) + logger.info(f"Wrote {len(predictions)} policy predictions to '{destination}'") + + +def desired_return(config, spec, row): + return ( + numeric_scalar( + config.desired_return + if config.desired_return is not None + else row[config.desired_return_column], + "desired_return", + ) + if spec.return_conditioned + else None + ) + + +def infer_rollout(policy, data, config): + spec = policy.config + rollout = config.rollout + if rollout.seed_row_index >= len(data): + raise ValueError("policy rollout seed_row_index is outside the input rows") + row = data.row(rollout.seed_row_index, named=True) + seed = SeedHistory( + observations={ + column: [0.0] * (spec.context_length - 1) + [row[column]] + for column in spec.observation_columns + }, + attention_valid_mask=[False] * (spec.context_length - 1) + [True], + static_inputs={column: row[column] for column in spec.static_columns}, + ) + budget = desired_return(config, spec, row) + trajectory = collect_policy_rollout( + policy, + load_online_adapter(rollout.environment), + seed, + rollout, + rollout.max_steps, + desired_return=budget, + action_selection=config.action_selection, + random_seed=config.seed, + ) + trajectory.metadata.update( + { + "model_path": config.model_paths[0], + "model_interface": config.model_interface, + "seed_row_index": rollout.seed_row_index, + "enforce_determinism": config.enforce_determinism, + "seed": config.seed, + } + ) + destination = normalize_path(rollout.trajectory_path, config.project_root) + config.validate_output_destination(destination, label="trajectory_path") + save_trajectory(trajectory, destination) + logger.info( + f"Wrote {len(trajectory.transitions)} policy transitions to '{destination}'" + ) + + +def policy_windows(data, spec, config): + # Rows without a sequence ID are independent single-observation decisions. + # Within sequences, recorded actions determine the causal previous-action input. + yield from joint_policy_windows(data, spec, config) + return + + +@torch.no_grad() +def predict_policy(policy, data, config): + spec = policy.config + if config.output_path is not None and config.output_probabilities: + from sequifier.infer import validate_probability_destinations + + validate_probability_destinations( + config, + Path(config.model_paths[0]).stem, + {name: field.type for name, field in spec.actions.items()}, + 1, + ) + row_key = policy_row_key(data, spec) + windows = policy_windows(data, spec, config) + generator = action_generator(policy.device, config.seed) + outputs = [] + probability_rows = { + name: [] for name, field in spec.actions.items() if field.type == "categorical" + } + while batch := list(islice(windows, config.inference_batch_size)): + rows, histories, identities = zip(*batch) + inputs = cast(SequifierInputs, collate_model_inputs(histories)).to( + policy.device + ) + distribution = policy.distribution(inputs) + deterministic = config.action_selection == "deterministic" + if config.output_probabilities: + selected = distribution.select_and_score( + deterministic, generator, include_probabilities=True + ) + sampled = selected.decision + scores, probabilities = selected.log_prob, selected.probabilities + else: + sampled = distribution.select_decision(deterministic, generator) + scores, probabilities = None, {} + actions = distribution.domain_actions(sampled) + for index, (row_index, action) in enumerate(zip(rows, actions)): + output = { + row_key: row_index, + **identities[index], + **action, + } + if scores is not None: + output["action_log_probability"] = float(scores[index]) + for name, values in probabilities.items(): + probability_rows[name].append( + { + row_key: row_index, + **{ + str(label): float(values[index][action_id]) + for action_id, label in enumerate(spec.actions[name].values) + }, + } + ) + outputs.append(output) + predictions = pl.DataFrame(outputs).sort(row_key).drop(row_key) + if config.output_probabilities: + from sequifier.infer import inference_output_path + + for name, records in probability_rows.items(): + if not records: + continue + destination = inference_output_path( + config.project_root, + config.write_format, + "probabilities", + Path(config.model_paths[0]).stem, + 0, + name, + config=config, + ) + write_data( + pl.DataFrame(records).sort(row_key).drop(row_key), + destination, + config.write_format, + ) + return predictions + + +def policy_row_key(data, spec): + reserved = ( + set(data.columns) + | set(spec.actions) + | {str(value) for field in spec.actions.values() for value in field.values} + ) + key = "__policy_row" + while key in reserved: + key += "_" + return key + + +def joint_policy_windows(data, spec, config): + row_key = policy_row_key(data, spec) + indexed = data.with_row_index(row_key) + groups = ( + indexed.partition_by(config.sequence_id_column, maintain_order=True) + if config.sequence_id_column in data.columns + else [indexed.slice(i, 1) for i in range(len(indexed))] + ) + view = PolicyWindowView(spec) + for group_index, group in enumerate(groups): + if config.item_position_column not in group.columns: + group = group.with_row_index(config.item_position_column) + group = ordered_sequence_rows(group, config.item_position_column) + rows = group.to_dicts() + previous, observations, budgets = [], [], [] + for index, row in enumerate(rows): + if spec.previous_action_inputs: + if index: + prior = {name: rows[index - 1][name] for name in spec.actions} + previous.append(encode_action(prior, spec)) + elif row.get(config.item_position_column, 0) != 0: + if not config.previous_action_columns: + raise ValueError( + "fragment inference requires previous_action_columns" + ) + previous.append( + encode_action( + { + name: row[column] + for name, column in config.previous_action_columns.items() + }, + spec, + ) + ) + else: + previous.append(None) + else: + previous.append(None) + observations.append({name: row[name] for name in spec.observation_columns}) + if spec.return_conditioned: + budgets.append(desired_return(config, spec, row)) + yield ( + row[row_key], + view.inputs( + observations, + previous, + budgets if spec.return_conditioned else None, + {name: row[name] for name in spec.static_columns}, + ), + { + "sequenceId": row.get(config.sequence_id_column, group_index), + "subsequenceId": index, + "windowStartOffset": rows[max(0, index + 1 - spec.context_length)][ + config.item_position_column + ], + "itemPosition": row[config.item_position_column], + }, + ) diff --git a/src/sequifier/rl/iql.py b/src/sequifier/rl/iql.py new file mode 100644 index 000000000..0146683d6 --- /dev/null +++ b/src/sequifier/rl/iql.py @@ -0,0 +1,158 @@ +"""Fixed-dataset categorical IQL as a phase-local actor/value/twin-Q runtime.""" + +import math +from typing import cast + +import torch +from torch import nn + +from sequifier.config.policy_plan import IQLPhaseConfig +from sequifier.io.iql_dataset import IQLTransitionDataset +from sequifier.rl.offline_data import OfflineData +from sequifier.rl.run_state import PhaseContext, PhaseSession +from sequifier.rl.window_storage import collated_windows + + +def _select(inputs, mask): + return type(inputs)( + {name: value[mask] for name, value in inputs.inputs.items()}, + {name: value[mask] for name, value in inputs.metadata.items()}, + ) + + +def _polyak(target, source, rate): + with torch.no_grad(): + for target_parameter, parameter in zip( + target.parameters(), source.parameters() + ): + target_parameter.lerp_(parameter, rate) + for target_buffer, buffer in zip(target.buffers(), source.buffers()): + target_buffer.copy_(buffer) + + +def _evaluate( + actor, value, q1, q2, dataset: IQLTransitionDataset, batch_size, gamma, device +): + totals = {"action_nll": 0.0, "bellman_residual": 0.0, "value": 0.0, "q_min": 0.0} + with torch.no_grad(): + for start in range(0, len(dataset), batch_size): + indices = range(start, min(start + batch_size, len(dataset))) + with collated_windows((dataset[i] for i in indices), device) as batch: + action = batch.action + v = value(batch.current) + q = torch.minimum(q1(batch.current, action), q2(batch.current, action)) + nll = -actor.evaluate_actions(batch.current, action) + successor_value = torch.zeros_like(v) + if batch.bootstrap.any(): + successor_value[batch.bootstrap] = value( + _select(batch.successor, batch.bootstrap) + ) + totals["action_nll"] += float(nll.sum()) + totals["bellman_residual"] += float( + (q - batch.reward - gamma * successor_value).abs().sum() + ) + totals["value"] += float(v.sum()) + totals["q_min"] += float(q.sum()) + del batch, action + result = {key: value / len(dataset) for key, value in totals.items()} + if not all(math.isfinite(value) for value in result.values()): + raise ValueError("non-finite IQL held-out likelihood or critic diagnostics") + return result + + +def run_iql_phase(context: PhaseContext[IQLPhaseConfig, OfflineData]): + actor, phase = context.actor.eval(), context.phase + data = context.data.prepare_transitions() + session = PhaseSession(context) + value = session.modules["value"] + q1, q2 = cast(nn.ModuleList, session.modules["critic"]) + targets = session.networks["target_q1"], session.networks["target_q2"] + if not 0 <= session.transaction <= phase.iterations: + raise ValueError("IQL checkpoint exceeds phase update budget") + if any(steps != session.transaction for steps in session.steps.values()): + raise ValueError("IQL optimizer clocks disagree with committed updates") + + def evaluate(**record): + session.evaluate_validation( + lambda: _evaluate( + actor, + value, + q1, + q2, + data[1], + phase.evaluation.batch_size, + phase.gamma, + actor.device, + ), + **record, + ) + + if session.should_evaluate_initial(): + evaluate(initial=True) + for update in range(session.transaction + 1, phase.iterations + 1): + if session.stopping.triggered: + break + indices = torch.randint( + len(data[0]), (phase.batch_size,), generator=session.sampler + ) + with collated_windows( + (data[0][int(i)] for i in indices), actor.device + ) as batch: + with torch.no_grad(): + qbar = torch.minimum( + targets[0](batch.current, batch.action), + targets[1](batch.current, batch.action), + ) + old_v = value(batch.current) + advantage = qbar - old_v + weight = torch.exp( + (phase.inverse_temperature * advantage).clamp( + max=math.log(phase.max_advantage_weight) + ) + ) + successor_value = torch.zeros_like(old_v) + if batch.bootstrap.any(): + successor_value[batch.bootstrap] = value( + _select(batch.successor, batch.bootstrap) + ) + target = batch.reward + phase.gamma * successor_value + delta = qbar - value(batch.current) + v_loss = ( + torch.where(delta < 0, 1 - phase.expectile, phase.expectile) + * delta.square() + ).mean() + q_loss = ( + (q1(batch.current, batch.action) - target).square() + + (q2(batch.current, batch.action) - target).square() + ).mean() + actor_loss = -( + weight * actor.evaluate_actions(batch.current, batch.action) + ).mean() + losses = {"value": v_loss, "critic": q_loss, "actor": actor_loss} + session.zero_grad() + for role, loss in losses.items(): + if not torch.isfinite(loss): + raise ValueError(f"non-finite IQL {role} objective") + loss.backward() + del batch + session.step() + _polyak(targets[0], q1, phase.target_update_rate) + _polyak(targets[1], q2, phase.target_update_rate) + session.complete_update( + final=update == phase.iterations, + training={ + f"{role}_loss": float(loss.detach()) for role, loss in losses.items() + }, + evaluate=evaluate, + evaluation_record=lambda: { + "advantage_mean": float(advantage.mean()), + "weight_mean": float(weight.mean()), + "weight_cap_fraction": float( + (weight >= phase.max_advantage_weight).float().mean() + ), + }, + ) + return session.finish( + session.optimizer_clocks(), + {"data": context.data.contract}, + ) diff --git a/src/sequifier/rl/joint_distribution.py b/src/sequifier/rl/joint_distribution.py new file mode 100644 index 000000000..09581a422 --- /dev/null +++ b/src/sequifier/rl/joint_distribution.py @@ -0,0 +1,310 @@ +"""Joint sampling and recorded-prefix scoring over existing model decoders.""" + +from collections.abc import Mapping +from dataclasses import dataclass + +import torch +from torch import Tensor + +from sequifier.io.preprocessing import denormalize_values +from sequifier.model.prediction_distribution import RealPredictionDistribution +from sequifier.rl.action_plan import ActionDecision, ActionExecutionPlan +from sequifier.rl.distributions import CategoricalActionDistribution + + +@dataclass(frozen=True) +class ScoredSelection: + decision: ActionDecision + log_prob: Tensor + probabilities: dict[str, Tensor] + _plan: ActionExecutionPlan + + @property + def actions(self) -> dict[str, Tensor]: + return self._plan.to_record(self.decision) + + +class JointActionDistribution: + """The model owns all weights; this object owns one encoded input window.""" + + def __init__(self, actor, inputs, *, _encoded=None): + self.actor = actor + if _encoded is None: + self.features, self.metadata = actor.model_inputs(inputs) + self.representation = actor.network.encode( + self.features, self.metadata, interface_name=actor.interface_name + ) + else: + self.features, self.metadata, self.representation = _encoded + self.plan = actor.action_plan + self.route = actor.network.resolve_interface(actor.interface_name) + self.decoder = self.route.decoder + self.batch = self.representation.shape[0] + end = self.representation.shape[1] + self.plan.position + 1 + start = end - self.route.decoding_support + if start < 0: + raise ValueError("decision_position lies outside the source decoder output") + self.context = self.route.decoder_input(self.representation[:, start:end])[:, 0] + self.valid = self.metadata["attention_valid_mask"][:, self.plan.position] + self.support, self.activity = self.plan.indices(self.context.device) + interface = actor.interface + self.statistics = ( + interface.selected_columns_statistics + if interface.normalize_real_columns + else {} + ) + + def row(self, index): + """View one encoded batch row without re-encoding or sharing action RNG.""" + if not 0 <= index < self.batch: + raise IndexError("joint-distribution row is outside the batch") + return type(self)( + self.actor, + None, + _encoded=( + {key: value[index : index + 1] for key, value in self.features.items()}, + {key: value[index : index + 1] for key, value in self.metadata.items()}, + self.representation[index : index + 1], + ), + ) + + def _distributions(self, decision: ActionDecision, *, categorical_only=False): + for name, output in self.plan.outputs( + self.decoder, self.context, self.valid, self.statistics, decision + ): + if not categorical_only or name in self.support: + yield name, self._field(name, output) + + def _field(self, name, output): + field = self.plan.fields[name].spec + output = self.decoder.resolve_target(name, output) + if field.type == "categorical": + return CategoricalActionDistribution( + output.index_select(-1, self.support[name]) + ) + location = output.squeeze(-1) + stats = self.statistics.get(name) + if stats is not None: + location = denormalize_values(location, stats) + scale = ( + self.route.prediction_scale(name) + if field.scale == "model" + else field.scale.fixed + ) + return RealPredictionDistribution(location, scale, field) + + def _active(self, name, decision: ActionDecision): + active = None + for predecessor, allowed in self.activity.get(name, {}).items(): + if predecessor not in decision.values: + raise ValueError( + "conditional activity requires a preceding action field" + ) + matches = torch.isin(decision.values[predecessor], allowed) + active = matches if active is None else active & matches + return active + + def _select(self, deterministic, generator): + decision, fields = ActionDecision(), {} + for name, distribution in self._distributions(decision): + value = ( + distribution.mode() if deterministic else distribution.sample(generator) + ) + field = self.plan.fields[name] + active = self._active(name, decision) + decision.values[name] = ( + value + if active is None + else torch.where(active, value, value.new_tensor(field.inactive)) + ) + if isinstance(distribution, RealPredictionDistribution): + decision.latents[name] = ( + distribution.location + if deterministic + else distribution.sampled_latent + ) + decision.components[name] = ( + torch.full_like(value, -1, dtype=torch.long) + if deterministic + else distribution.sampled_component + ) + fields[name] = distribution, active + return decision, fields + + def select_decision(self, deterministic, generator) -> ActionDecision: + decision, _ = self._select(deterministic, generator) + return decision + + def select(self, deterministic, generator): + return self.plan.to_record(self.select_decision(deterministic, generator)) + + def select_and_score( + self, deterministic, generator, *, include_probabilities=False + ): + """Score the selected action with the distributions used to select it.""" + decision, fields = self._select(deterministic, generator) + return ScoredSelection( + decision, + self._score(decision, fields), + self._probabilities(fields) if include_probabilities else {}, + self.plan, + ) + + def sample(self, generator=None): + return self.select(False, generator) + + def mode(self): + return self.select(True, None) + + def _score(self, decision: ActionDecision, fields): + score = self.context.new_zeros( + self.batch, + dtype=torch.float32 if self.context.device.type == "mps" else torch.float64, + ) + for name, (distribution, active) in fields.items(): + field = self.plan.fields[name] + value = decision.values[name] + if active is not None: + if ((value != field.inactive) & ~active).any(): + raise ValueError( + f"{name}: inactive action differs from its point mass" + ) + value = torch.where(active, value, distribution.mode().detach()) + if isinstance(distribution, RealPredictionDistribution): + latent = decision.latents.get(name) + component = decision.components.get(name) + if active is not None: + if latent is not None: + latent = torch.where( + active, latent, distribution.location.detach() + ) + if component is not None: + component = torch.where(active, component, -1) + log_prob = distribution.log_prob( + value, latent=latent, component=component + ) + else: + log_prob = distribution.log_prob(value) + score = score + ( + log_prob if active is None else torch.where(active, log_prob, 0.0) + ) + return score.float() + + def _decision( + self, actions: ActionDecision | Mapping[str, Tensor] + ) -> ActionDecision: + if isinstance(actions, ActionDecision): + return actions + if isinstance(actions, Mapping): + return self.plan.from_record(actions) + raise ValueError("joint likelihood requires named action fields") + + def log_prob(self, actions: ActionDecision | Mapping[str, Tensor]): + decision = self._decision(actions) + if self.plan.fields.keys() - decision.values.keys(): + raise ValueError("joint likelihood requires every action field") + if decision.values.keys() - self.plan.fields.keys(): + raise ValueError("joint action contains unknown fields") + for name in self.plan.order: + if decision.values[name].shape != (self.batch,): + raise ValueError(f"{name}: expected one action per decision") + return self._score( + decision, + { + name: (distribution, self._active(name, decision)) + for name, distribution in self._distributions(decision) + }, + ) + + def _probabilities(self, fields): + result = {} + for name in self.plan.action_names: + if name not in fields: + continue + distribution, active = fields[name] + if isinstance(distribution, CategoricalActionDistribution): + probabilities = distribution.distribution.probs + if active is not None: + point = torch.zeros_like(probabilities) + point[:, self.plan.fields[name].inactive] = 1 + probabilities = torch.where(active[:, None], probabilities, point) + result[name] = probabilities + return result + + def categorical_probabilities(self, actions): + """Per-field conditional probabilities at the supplied action prefix.""" + decision = self._decision(actions) + return self._probabilities( + { + name: (distribution, self._active(name, decision)) + for name, distribution in self._distributions( + decision, categorical_only=True + ) + } + ) + + @property + def entropy_is_exact(self): + return not self.plan.has_autoregression and all( + not field.active_when + and field.lower is None + and field.upper is None + and not field.atoms + for field in (action.spec for action in self.plan.fields.values()) + ) + + def entropy(self, generator=None): + if self.entropy_is_exact: + return ( + torch.stack( + [ + distribution.entropy() + for _, distribution in self._distributions(ActionDecision()) + ] + ) + .sum(dim=0) + .float() + ) + # Sampled actions are detached. The score-function surrogate includes + # the likelihood of the sampled prefixes. + logp = self.select_and_score(False, generator).log_prob + surrogate = -(logp.detach() + 1) * logp + return -logp.detach() + surrogate - surrogate.detach() + + def kl(self, reference, generator=None): + if self.actor.config.actions != reference.actor.config.actions: + raise ValueError( + "KL requires matching action measures and physical transforms" + ) + if not self.plan.has_autoregression and not self.activity: + current = dict(self._distributions(ActionDecision())) + frozen = dict(reference._distributions(ActionDecision())) + values = [] + for name, field in self.actor.config.actions.items(): + p, q = current[name], frozen[name] + if isinstance(p, CategoricalActionDistribution) and isinstance( + q, CategoricalActionDistribution + ): + values.append( + torch.distributions.kl_divergence( + p.distribution, q.distribution + ) + ) + else: + if not isinstance(p, RealPredictionDistribution) or not isinstance( + q, RealPredictionDistribution + ): + raise ValueError("reference distribution families differ") + values.append( + (1 - sum(field.atoms.values())) + * torch.distributions.kl_divergence(p.normal, q.normal) + ) + return torch.stack(values).sum(dim=0).float() + selected = self.select_and_score(False, generator) + delta = selected.log_prob - reference.log_prob(selected.decision) + surrogate = (delta.detach() + 1) * selected.log_prob + return delta.detach() + surrogate - surrogate.detach() + + def domain_actions(self, actions): + decision = self._decision(actions).cpu() + return [self.plan.domain_action(decision, i) for i in range(self.batch)] diff --git a/src/sequifier/rl/learner.py b/src/sequifier/rl/learner.py new file mode 100644 index 000000000..2766e817d --- /dev/null +++ b/src/sequifier/rl/learner.py @@ -0,0 +1,452 @@ +"""Native policy phases over prepared data and validated recovery state.""" + +from __future__ import annotations + +from dataclasses import replace +from pathlib import Path +from typing import TYPE_CHECKING + +import torch + +from sequifier.artifacts.dependencies import policy_dependencies +from sequifier.artifacts.phase import PhaseArtifacts +from sequifier.artifacts.reports import phase_report, publish_run_report, run_report +from sequifier.fingerprints import fingerprint +from sequifier.helpers import configure_determinism, configure_logger +from sequifier.rl.checkpoint import save_policy, validate_policy_weights +from sequifier.rl.preparation import ( + build_actor, + preflight_environments, + preflight_optimization, + resolve_normalization, + validate_data, + validate_declared_policy, + validate_output_paths, +) +from sequifier.rl.prepared_inputs import PreparedTrainingInputs +from sequifier.rl.references import PolicyReferences +from sequifier.rl.registry import BUILTIN_ALGORITHMS +from sequifier.rl.run_checkpoint import RunCheckpoint +from sequifier.rl.run_config import ResolvedPolicyRun +from sequifier.rl.run_state import ( + ActorRepresentation, + PhaseContext, + PolicyPhaseIdentity, + PolicyRunResult, + clone_weights, + require_equal, + weight_fingerprint, +) +from sequifier.rl.telemetry import TelemetryJournal +from sequifier.runtime.plan import ( + PhaseOutcome, + RunIdentity, + run_phases, + validate_continuation, +) +from sequifier.runtime.random_state import capture_rng_state, restore_rng_state + +if TYPE_CHECKING: + from sequifier.config.train_config import ResolvedSequifierConfig, SequifierConfig + from sequifier.training.coordinator import LearningRunResult + + +def _plan_contract(config): + from pydantic_core import to_jsonable_python + + from sequifier.artifacts.model_export import model_execution_config + + values = config.model_dump(mode="json") + if config.phase_offset: + values["phase_offset"] = config.phase_offset + values["execution_model"] = to_jsonable_python( + model_execution_config(config.execution) + ) + values["model"] = config.execution.model.model_dump(mode="json") + values.pop("resume") + values.pop("checkpoint_path", None) + if config.actor_initialization == "attach": + values.pop("actor_initialization") + for phase in values["phases"]: + for role in phase["optimization"].values(): + if role["scheduler"]["name"] == "OneCycleLR": + # total_steps is an assertion, not a separately configurable budget. + role["scheduler"]["arguments"].pop("total_steps", None) + return values + + +def _phase_report(result, checkpoint): + return phase_report( + model_path=result.provenance["actor_path"], + checkpoint_path=checkpoint, + **result.as_dict(), + ) + + +def _publish_phase_report(record, checkpoint): + result = record.report_result() + artifacts = PhaseArtifacts(Path(result.provenance["actor_path"])) + record.commit.publish(artifacts, _phase_report(result, checkpoint)) + + +def _initialize_history_actor(config): + from sequifier.artifacts.model_artifact import load_execution_model + from sequifier.model.history_transfer import ( + copy_history_weights, + history_parameter_ids, + validate_history_transfer, + ) + from sequifier.rl.models import validate_independent_networks + from sequifier.trajectory.sources import file_checksum + + if not config.initialize_from: + raise ValueError("history_only requires initialize_from") + actor = build_actor(config) + name = actor.interface_name + decoder = actor.network.resolve_interface(name).require_decoder() + history_ids = history_parameter_ids(actor.network, name) + if any(id(parameter) in history_ids for parameter in decoder.parameters()): + raise ValueError(f"{name}: history_only requires an untied action decoder") + checksum = file_checksum(config.initialize_from) + random = capture_rng_state(actor.device) + try: + source, execution, _ = load_execution_model( + config.initialize_from, device=config.device, interface_name=name + ) + finally: + restore_rng_state(actor.device, random) + require_equal( + checksum, file_checksum(config.initialize_from), "initialize_from.sha256" + ) + validate_history_transfer(execution, actor.execution_config, name) + validate_independent_networks(source, actor.network) + components = copy_history_weights(source, actor.network, name) + validate_policy_weights(actor) + actor.transfer_report = { + "mode": "history_only", + "interface": name, + "source_sha256": checksum, + "source_preprocessing": "preserved", + "transferred_components": components, + "fresh_components": [f"interfaces.{name}.decoder"], + } + return actor + + +def _initialize_actor(config): + if config.actor_initialization == "history_only": + return _initialize_history_actor(config) + if config.initialize_from: + from sequifier.artifacts.model_artifact import read_execution_artifact + from sequifier.artifacts.model_config import validate_execution_attachment + from sequifier.rl.attachment import attachment_report + + artifact = read_execution_artifact(config.initialize_from) + validate_execution_attachment(artifact.model_config.values, config.execution) + actor = build_actor(config, artifact=artifact) + actor.transfer_report = attachment_report(actor, artifact.model_state_dict) + return actor + return build_actor(config) + + +def run_policy_training( + config: ResolvedPolicyRun | ResolvedSequifierConfig | SequifierConfig, + *, + prepared: PreparedTrainingInputs | None = None, +) -> PolicyRunResult | LearningRunResult: + """Validate the whole run, execute phase-local lifecycles, and export its final actor.""" + if isinstance(config, ResolvedPolicyRun): + return _run_policy(config, prepared or PreparedTrainingInputs()) + + from sequifier.config.train_config import ResolvedSequifierConfig, SequifierConfig + from sequifier.config.training_resolution import resolve_training_run + from sequifier.training.coordinator import run_policy_plan + + if isinstance(config, SequifierConfig): + config = resolve_training_run(config) + if not isinstance(config, ResolvedSequifierConfig) or any( + phase.algorithm == "supervised" for phase in config.training_plan + ): + raise TypeError("policy training requires a resolved policy-only training plan") + return run_policy_plan(config) + + +def run_policy_group( + config: ResolvedPolicyRun, + identity: RunIdentity, + on_checkpoint, + *, + prepared: PreparedTrainingInputs, +): + """Run a group of policy phases with a shared actor and optimizer lifecycle.""" + _run_policy( + config, prepared, group=True, identity=identity, on_checkpoint=on_checkpoint + ) + + +def _run_policy(config, prepared, *, group=False, identity=None, on_checkpoint=None): + with prepared.run_scope(): + return _execute_policy(config, prepared, group, identity, on_checkpoint) + + +def _execute_policy(config, prepared, group, identity, on_checkpoint): + torch.set_float32_matmul_precision("highest") + configure_determinism(config.seed, config.enforce_determinism) + for index, phase in enumerate(config.phases): + validate_continuation( + config.phases[index - 1] if index else None, phase, resolved=True + ) + phase_identities = tuple( + PolicyPhaseIdentity.resolve(index, config.seed) + for index, _ in enumerate(config.phases, config.phase_offset) + ) + plan = _plan_contract(config) + inputs = prepared.resolve(config) + inputs.revalidate(inputs.identity, "run.source_identity") + checkpoint = Path(config.checkpoint_path) + recovery = RunCheckpoint.load( + checkpoint, + config, + plan, + run_id=identity.run_id if identity is not None else None, + ) + resuming = recovery is not None + dependencies = policy_dependencies( + config, + prepared, + inputs, + saved=recovery.data_identity if recovery is not None else None, + committed_reference_count=( + recovery.phase_index + int(recovery.phase_state is not None) + if recovery is not None + else 0 + ), + ) + data_identity = dependencies.identity + if recovery is not None: + recovery.validate_dependencies(config, data_identity) + validate_output_paths(config, dependencies) + configure_logger(config.project_root, config.model_name, 0) + adapter_contract = preflight_environments(config) + if recovery is not None: + require_equal( + adapter_contract, recovery.adapter_contract, "resume.environments" + ) + representation = recovery.representation + validate_declared_policy(config.policy, representation.policy, "resume.policy") + data = prepared.prepare(config, representation.policy, inputs) + actor = build_actor(config, representation.policy) + recovery.validate_actor(actor, config.units) + else: + data = prepared.prepare(config, config.policy, inputs) + actor = _initialize_actor(config) + representation = ActorRepresentation.from_actor(actor, config.units) + if config.normalization is not None: + expected_normalization = resolve_normalization( + actor.config, config.normalization + ) + require_equal( + expected_normalization, + { + key: value + for key, value in representation.values["statistics"].items() + if key in expected_normalization + }, + "run.normalization", + ) + validate_data(data, actor) + prepared.measure() + data_contract = [item.contract if item is not None else None for item in data] + if recovery is not None: + require_equal(data_contract, recovery.data_contract, "resume.partitions") + if recovery is None: + recovery = RunCheckpoint( + plan, + representation, + data_identity, + data_contract, + adapter_contract, + { + "path": config.initialize_from, + "transfer": actor.transfer_report, + "weight_hash": weight_fingerprint(actor.network.state_dict()), + }, + clone_weights(actor.network), + capture_rng_state(actor.device), + ) + from sequifier.rl.metrics import PolicyMetrics + + identity = identity or RunIdentity.start(recovery.run_id) + recovery.run_id = identity.run_id + metrics = PolicyMetrics(config, identity) + references = PolicyReferences(config, recovery, resuming=resuming) + saved_random = capture_rng_state(actor.device) + optimizations = preflight_optimization(config, actor, data) + restore_rng_state(actor.device, recovery.random if resuming else saved_random) + + def save_checkpoint(state=None): + recovery.save(checkpoint, actor.device, state) + if on_checkpoint is not None: + on_checkpoint( + checkpoint, [record.as_dict() for record in recovery.completed] + ) + + if not resuming: + # Persist the complete fitted representation before the first update. + save_checkpoint() + + def replay(): + for record in recovery.completed: + _publish_phase_report(record, checkpoint) + + def run_phase(transition): + position, phase = transition.index, transition.phase + phase_identity = phase_identities[position] + phase_index = phase_identity.index + current_state = recovery.phase_state + dependencies.revalidate() + incoming_weight_hash = weight_fingerprint(actor.network.state_dict()) + initialization = {} + if phase.uses_actor_history: + from sequifier.model.history_transfer import history_components + + initialization["history_initialization"] = { + "source": "phase_start_actor", + "source_weight_hash": incoming_weight_hash, + "transferred_components": list( + history_components(actor.network, actor.interface_name) + ), + "fresh_components": {"value": ["head"]}, + } + if phase.algorithm == "iql": + initialization["history_initialization"]["fresh_components"].update( + {name: ["action_embedding", "head"] for name in ("q1", "q2")} + ) + if phase_index == 0 and config.actor_initialization == "history_only": + initialization["actor_initialization"] = recovery.initialization + actor.eval() + reference, reference_identity = references.resolve(actor, phase_index) + + def commit(state): + representation.validate_actor( + actor, config.units, f"phases.{phase.name}.actor" + ) + save_checkpoint({**state, "telemetry_path": str(artifacts.telemetry)}) + + artifacts = PhaseArtifacts.for_phase( + config.phase_directory, phase_index, phase.name + ) + telemetry_position = ( + current_state["telemetry_position"] if current_state else None + ) + with TelemetryJournal.open( + artifacts.telemetry, telemetry_position + ) as telemetry: + context = PhaseContext( + config, + phase_identity, + phase, + actor, + data[position] if data else None, + reference, + reference_identity, + current_state, + commit, + optimizations[position], + telemetry, + metrics=metrics, + continuation=recovery.continuation + if transition.continues_optimizer + else None, + preserve_optimizer=transition.preserves_optimizer, + ) + execution = BUILTIN_ALGORITHMS[phase.algorithm].runtime(context) + telemetry_position = telemetry.commit() + representation.validate_actor( + execution.actor, config.units, f"phases.{phase.name}.selected_actor" + ) + result = execution.result + result = replace( + result, + provenance={ + **result.provenance, + "config_fingerprint": fingerprint(phase.model_dump(mode="json")), + "resolved_phase": phase.model_dump(mode="json"), + "representation_fingerprint": representation.identity, + "incoming_weight_hash": incoming_weight_hash, + **initialization, + **artifacts.describe(), + "resource_accounting": prepared.accounting.snapshot(), + "source_provenance": data_identity, + }, + ) + return PhaseOutcome( + (replace(execution, result=result), telemetry_position), result.stop_scope + ) + + def commit_phase(transition, outcome): + nonlocal actor + execution, position = outcome.result + actor = execution.actor + recovery.finish_phase(execution, position) + save_checkpoint() + return recovery.completed[-1] + + def publish(record): + _publish_phase_report(record, checkpoint) + if not group and record.result.index == config.phase_offset + last_data_phase: + release_phase_data() + + last_data_phase = max( + (index for index, part in enumerate(data) if part is not None), default=-1 + ) + + def release_phase_data(): + data.clear() + prepared.release_data() + + if not group and last_data_phase < recovery.phase_index: + release_phase_data() + completion_reason = run_phases( + config.phases, + run_phase, + commit_phase, + start=recovery.phase_index, + stopped=recovery.stopped, + replay=replay, + publish=publish, + optimizer_continues=lambda phase: phase.optimizer_state == "continue", + ) + results = () + if not group: + results = tuple(record.report_result() for record in recovery.completed) + data.clear() + prepared.close() + report = run_report( + model_path=config.model_path, + checkpoint_path=checkpoint, + phases=[_phase_report(result, checkpoint) for result in results], + run_id=recovery.run_id, + session_id=identity.session_id, + status="complete", + completion_reason=completion_reason, + exports=[config.model_path], + plan=plan, + plan_fingerprint=fingerprint(plan), + representation_fingerprint=representation.identity, + source_provenance=data_identity, + reference_provenance=recovery.reference_contract, + initialization=recovery.initialization, + resume=config.resume.model_dump(mode="json"), + resource_accounting=prepared.accounting.snapshot(), + ) + save_policy(actor.eval(), config.model_path, metadata=report) + publish_run_report(report, config.report_path) + return PolicyRunResult( + actor, + results, + representation, + config.model_path, + config.report_path, + config.checkpoint_path, + ) diff --git a/src/sequifier/rl/metrics.py b/src/sequifier/rl/metrics.py new file mode 100644 index 000000000..79c95fa5c --- /dev/null +++ b/src/sequifier/rl/metrics.py @@ -0,0 +1,89 @@ +"""Policy observations in the native supervised metric tables.""" + +import math +import time + +from sequifier.training.metrics import StructuredMetricWriters + + +class PolicyMetrics: + def __init__(self, config, identity): + self.writer = StructuredMetricWriters( + config.project_root, + config.model_name, + 0, + dataset_name=config.policy_interface, + ) + self.run_id = identity.run_id + self.session_id = identity.session_id + self.started = time.monotonic() + self.pending = None + + def record( + self, + session, + values, + *, + validation=False, + primary=None, + initial=False, + final=False, + ): + context = session.context + if not validation: + self.pending = (session, values) + if ( + not validation + and not final + and not session.stopping.triggered + and session.transaction % context.run.log_interval_iterations + ): + return + scalars = { + name: float(value) + for name, value in values.items() + if isinstance(value, (float, int)) + and not isinstance(value, bool) + and math.isfinite(value) + } + if not scalars: + return + if primary is None: + primary = next( + ( + name + for name in ( + "training_action_nll", + "actor_loss", + "training_dpo_loss", + "loss", + "policy_loss", + ) + if name in scalars + ), + next(iter(scalars)), + ) + self.writer.phase = context.phase.name + self.writer.algorithm = context.phase.algorithm + now = time.monotonic() + self.writer.write_observation( + validation=validation, + run_id=self.run_id, + session_id=self.session_id, + epoch=context.index + 1, + iteration=session.transaction, + batches_total=session.iteration_budget, + global_step=sum(session.steps.values()), + values=scalars, + primary=primary, + learning_rate=float(session.optimization["actor"][0].param_groups[0]["lr"]), + elapsed_seconds=now - self.started, + initial=initial, + ) + if not validation: + self.started = now + self.pending = None + + def flush(self, session): + if self.pending is not None and self.pending[0] is session: + self.record(session, self.pending[1], final=True) diff --git a/src/sequifier/rl/models.py b/src/sequifier/rl/models.py new file mode 100644 index 000000000..4435cbcc6 --- /dev/null +++ b/src/sequifier/rl/models.py @@ -0,0 +1,196 @@ +"""Independent history value and state-action critics for offline IQL.""" + +from contextlib import nullcontext +from copy import deepcopy + +import torch +from torch import nn + +from sequifier.io.batch import SequifierInputs +from sequifier.io.policy_window import normalize_policy_features +from sequifier.model.dtypes import cast_floating_to_module_dtype +from sequifier.model.factory import build_history_encoder +from sequifier.model.history_transfer import copy_history_weights + + +def critic_history_options(actor, phase, *, restore=False): + restore = restore or phase.optimizer_state == "continue" + return { + "history_source": actor.network + if phase.history_initialization == "phase_start_actor" and not restore + else None, + "restore_history": restore, + } + + +def validate_independent_networks(*modules): + seen = set() + for module in modules: + storage = { + (tensor.device, tensor.untyped_storage().data_ptr()) + for tensor in (*module.parameters(), *module.buffers()) + if tensor.numel() + } + if seen & storage: + raise ValueError("policy networks share parameter or buffer storage") + seen.update(storage) + + +class ActionEncoder(nn.Module): + """Fuse typed categorical embeddings and standardized real projections.""" + + def __init__(self, fields, width, statistics): + super().__init__() + self.fields = {name: fields[name] for name in sorted(fields)} + self.statistics = statistics + self.encoders = nn.ModuleDict( + { + name: nn.Embedding(len(field.values), width) + if field.type == "categorical" + else nn.Linear(1, width) + for name, field in self.fields.items() + } + ) + self.fusion = nn.Linear(len(fields) * (width + 1), width) + + def forward(self, actions): + from sequifier.io.preprocessing import normalize_values + + pieces = [] + for name, field in self.fields.items(): + value = actions[name] + active = torch.ones_like(value, dtype=torch.bool) + for predecessor, labels in field.active_when.items(): + parent = self.fields[predecessor] + allowed = ( + [parent.values.index(label) for label in labels] + if parent.type == "categorical" + else labels + ) + active &= torch.isin( + actions[predecessor], torch.tensor(allowed, device=value.device) + ) + if field.type == "real": + value = value.float() + if name in self.statistics: + value = normalize_values(value, self.statistics[name]) + value = value[:, None] + encoded = self.encoders[name](value) + pieces.extend((encoded * active[:, None], active.float()[:, None])) + return self.fusion(torch.cat(pieces, dim=-1)) + + +class HistoryCritic(nn.Module): + def __init__( + self, + execution_config, + interface_name: str, + action_count: int | dict | None, + *, + history_source=None, + restore_history=False, + metadata_only=False, + ): + super().__init__() + device = torch.device("meta" if metadata_only else execution_config.device) + self.interface_name = interface_name + self.interface = execution_config.interface(interface_name) + with device if metadata_only else nullcontext(): + self.network = build_history_encoder( + execution_config, + interface_name, + device=device, + restored_history=restore_history or history_source is not None, + ) + width = self.network.model_dim + self.action_embedding = ( + ActionEncoder( + action_count, + width, + self.interface.selected_columns_statistics + if self.interface.normalize_real_columns + else {}, + ) + if isinstance(action_count, dict) + else nn.Embedding(action_count, width) + if action_count + else None + ) + self.head = nn.Sequential( + nn.Linear(width * (2 if action_count else 1), width), + nn.ReLU(), + nn.Linear(width, 1), + ) + self.to(device) + if history_source is not None: + copy_history_weights(history_source, self.network, interface_name) + + @classmethod + def ppo_value(cls, actor, **history): + value = cls(actor.execution_config, actor.interface_name, None, **history) + if history.get("history_source") is not None: + validate_independent_networks(actor.network, value) + return value + + def forward( + self, + history: SequifierInputs, + action: torch.Tensor | dict[str, torch.Tensor] | None = None, + ): + if (action is None) != (self.action_embedding is None): + raise ValueError("Q requires a current action and V forbids one") + features = normalize_policy_features( + history.inputs, history.metadata, self.interface + ) + return self.prepared_value(features, history.metadata, action) + + def prepared_value(self, features, metadata, action=None): + """Score the same normalized history used by the collecting actor.""" + representation = self.network.encode( + features, metadata, interface_name=self.interface_name + ) + state = representation[:, -1] + if self.action_embedding is not None: + if action is None: + raise ValueError("Q requires a current action") + if not isinstance(action, dict) and ( + action.dtype != torch.long + or action.ndim != 1 + or action.shape[0] != state.shape[0] + ): + raise ValueError("critic actions must be int64 [batch]") + state = torch.cat((state, self.action_embedding(action)), dim=-1) + return ( + self.head(cast_floating_to_module_dtype(state, self.head[0])) + .squeeze(-1) + .float() + ) + + +def make_iql_critics( + execution_config, interface_name: str, action_count: int | dict, **history +): + value = HistoryCritic(execution_config, interface_name, None, **history) + q1 = HistoryCritic(execution_config, interface_name, action_count, **history) + q2 = HistoryCritic(execution_config, interface_name, action_count, **history) + return value, q1, q2 + + +def make_iql_models( + execution_config, interface_name: str, action_count: int | dict, **history +): + value, q1, q2 = make_iql_critics( + execution_config, interface_name, action_count, **history + ) + targets = ( + deepcopy(q1).requires_grad_(False), + deepcopy(q2).requires_grad_(False), + ) + modules = (value, q1, q2, *targets) + if len( + {id(parameter) for module in modules for parameter in module.parameters()} + ) != sum(1 for module in modules for _ in module.parameters()): + raise ValueError("IQL networks share parameters") + if (source := history.get("history_source")) is not None: + validate_independent_networks(source, *modules) + return value, q1, q2, targets diff --git a/src/sequifier/rl/offline_data.py b/src/sequifier/rl/offline_data.py new file mode 100644 index 000000000..741bb81d5 --- /dev/null +++ b/src/sequifier/rl/offline_data.py @@ -0,0 +1,109 @@ +"""Offline phase partitioning and frozen-codec validation.""" + +from dataclasses import dataclass, field + +from sequifier.io.iql_dataset import IQLTransitionDataset +from sequifier.io.policy_dataset import EpisodeDecisions, episode_partition_indices + + +@dataclass +class OfflineData: + training: EpisodeDecisions + validation: EpisodeDecisions + transitions: tuple[IQLTransitionDataset, IQLTransitionDataset] | None = field( + default=None, init=False + ) + + def prepare_transitions(self): + if self.transitions is None: + self.transitions = ( + IQLTransitionDataset(self.training), + IQLTransitionDataset(self.validation), + ) + return self.transitions + + @property + def contract(self): + return [ + { + "episodes": part.episode_fingerprints, + "episode_ids": part.episode_ids, + "decisions": len(part), + } + for part in (self.training, self.validation) + ] + + def fitting_dataset(self): + return self.training + + def validate(self, actor): + for part in (self.training, self.validation): + part.validate_features() + part.validate_normalization(actor.interface.selected_columns_statistics) + if self.transitions is not None: + for transitions in self.transitions: + transitions.validate_normalization(actor) + + +def prepare_offline(run, phase, catalog, cache, membership): + # The cache belongs to one run: budgets and optimizers do not affect partitions. + key = ( + "offline", + tuple(ref.source for ref in phase.sources), + tuple(ref.source for ref in (phase.evaluation.sources or [])), + ) + if key in cache: + return cache[key] + corpus = catalog.dataset + training = corpus([ref.source for ref in phase.sources]) + if phase.evaluation.sources: + validation = corpus([ref.source for ref in phase.evaluation.sources]) + else: + if key not in membership: + membership[key] = episode_partition_indices( + training, + phase_validation_fraction(run, phase), + run.seed, + group_identical=True, + ) + training, validation = (training.subset(indices) for indices in membership[key]) + if not len(training) or not len(validation): + raise ValueError( + f"phases.{phase.name}: nonempty train and validation partitions required" + ) + for attribute, train_ids, validation_ids in ( + ("all_episode_ids", training.all_episode_ids, validation.all_episode_ids), + ("group_ids", training.group_ids, validation.group_ids), + ( + "demonstration_keys", + training.demonstration_keys, + validation.demonstration_keys, + ), + ): + if (set(train_ids) & set(validation_ids)) - {None}: + raise ValueError( + f"phases.{phase.name}: {attribute} crosses train/validation partitions" + ) + data = OfflineData(training, validation) + cache[key] = data + return data + + +def prepare_iql(run, phase, catalog, cache, membership): + data = prepare_offline(run, phase, catalog, cache, membership) + data.prepare_transitions() + return data + + +def phase_validation_fraction(run, phase): + """A combined leakage-safe split requires a single agreed fraction.""" + refs = [phase.preferences] if phase.algorithm == "dpo" else phase.sources + fractions = { + run.source_validation_fractions.get(ref.source, run.validation_fraction) + for ref in refs + } + if len(fractions) != 1: + raise ValueError( + f"{phase.name}: automatic splitting of combined datasets requires equal dataset split fractions; declare evaluation.sources for explicit partitions" + ) + return next(iter(fractions)) diff --git a/src/sequifier/rl/parallel_grpo_collector.py b/src/sequifier/rl/parallel_grpo_collector.py new file mode 100644 index 000000000..3b6ecf1f9 --- /dev/null +++ b/src/sequifier/rl/parallel_grpo_collector.py @@ -0,0 +1,164 @@ +"""Explicitly enabled synchronous GRPO collection across independent simulators.""" + +import math +from collections import deque + +import torch + +from sequifier.io.actions import action_record +from sequifier.rl.collection import PolicyDecision +from sequifier.rl.environment import validate_start_token +from sequifier.rl.group_collection import GroupCollection +from sequifier.rl.history import frame_storage_bytes +from sequifier.rl.simulator_workers import require_parallel_group_start +from sequifier.rl.window_storage import collated_windows + + +@torch.no_grad() +def _decide(actor, reference, cursors): + with collated_windows( + (cursor.inputs for cursor in cursors), actor.device, borrowed=True + ) as inputs: + distribution = actor.distribution(inputs) + generators = [cursor.generator for cursor in cursors] + # Decode and validate distributions across the full environment batch. + # Only random draws use individual generators; no per-row decoder calls. + selected = distribution.select_and_score(False, generators) + scores = ( + torch.stack( + [ + selected.log_prob, + reference.evaluate_actions(inputs, selected.decision), + ], + dim=1, + ) + .cpu() + .tolist() + ) + del inputs, distribution + host_decision = selected.decision.cpu() + host_actions = actor.action_plan.to_record(host_decision) + decisions = [] + for index, (old, ref) in enumerate(scores): + if not math.isfinite(old) or not math.isfinite(ref): + raise ValueError("non-finite GRPO policy/reference action score") + action = action_record(host_actions, index) + decisions.append( + ( + PolicyDecision( + actor.action_plan.domain_action(host_decision, index), + action, + old, + 0.0, + 0.0, + ), + ref, + ) + ) + return decisions + + +@torch.no_grad() +def collect_parallel_groups( + context, + adapter, + workers, + start_group, + count, + generation, + env_steps, + restart_stream, + *, + storage=None, +): + actor, reference, phase = ( + context.actor.eval(), + context.reference.eval(), + context.phase, + ) + require_parallel_group_start(adapter) + pending, active, completed = deque(), {}, {} + with GroupCollection( + context, adapter, count, generation, env_steps, restart_stream, storage=storage + ) as collection: + from sequifier.rl.history import _json_payload_bytes + + account = collection.account + for group in range(start_group, start_group + count): + token = collection.start(group) + completed[group] = [] + pending.extend((group, sample, token) for sample in range(phase.group_size)) + while pending or active: + resetting = [] + for slot in range(len(workers)): + if slot in active or not pending: + continue + group, sample, token = pending.popleft() + active[slot] = collection.trajectory( + group, sample, token, deferred=True + ) + workers.send( + slot, + "reset_group", + validate_start_token(token), + collection.environment_seed(group, sample), + ) + resetting.append(slot) + for slot in resetting: + state = active[slot] + with workers.response(slot) as response: + state.cursor.reset( + response, f"group-{state.group}-sample-{state.sample}" + ) + del response + frame = state.cursor.history.current + collection.validate_initial(state) + # Include live history before any environment step is issued. + account(frame_storage_bytes(frame) + 1024) + account( + sum( + value.numel() * value.element_size() + for value in ( + *state.cursor.inputs.inputs.values(), + *state.cursor.inputs.metadata.values(), + ) + ), + window=True, + ) + decisions = {} + slots = sorted(active) + for start in range(0, len(slots), phase.rollout_inference_batch_size): + chunk = slots[start : start + phase.rollout_inference_batch_size] + scored = _decide(actor, reference, [active[s].cursor for s in chunk]) + decisions.update(zip(chunk, scored)) + for decision, _ in decisions.values(): + account(4 * _json_payload_bytes(decision.action)) + for slot, (decision, _) in decisions.items(): + # Count attempted actions conservatively before sending: a failed + # send must never hide a potentially executed simulator action. + collection.executed += 1 + workers.send(slot, "step", decision.domain_action) + for slot, (decision, ref) in decisions.items(): + state = active[slot] + frame = state.cursor.history.current + with workers.response(slot) as response: + result = state.cursor.accept_step(decision, response) + del response + state.append(frame, decision, ref, result, phase.reward_scale) + # Include the successor even on the last step of a trajectory. + if result.observation is not None: + account( + frame_storage_bytes(state.cursor.history.current) + 1024 + ) + if ( + result.terminated + or result.truncated + or len(state.actions) == phase.max_steps + ): + completed[state.group].append( + collection.seal_trajectory(state, result) + ) + del active[slot] + return collection.finish( + [collection.seal_group(group) for group in completed.values()] + ) diff --git a/src/sequifier/rl/phase_optimization.py b/src/sequifier/rl/phase_optimization.py new file mode 100644 index 000000000..42a0a508a --- /dev/null +++ b/src/sequifier/rl/phase_optimization.py @@ -0,0 +1,83 @@ +"""Resolve policy optimizer roles and masks before executing a phase.""" + +from dataclasses import dataclass + +from torch import nn + +from sequifier.config.policy_optimization import OptimizationConfig +from sequifier.rl.models import HistoryCritic, critic_history_options, make_iql_models + + +@dataclass(frozen=True) +class PhaseRoles: + trainable: dict[str, nn.Module] + frozen: dict[str, nn.Module] + + @classmethod + def build(cls, actor, phase, *, metadata_only=False, restoring=False): + trainable, frozen = {"actor": actor.network}, {} + if phase.algorithm in {"ppo", "iql"}: + history = ( + {"metadata_only": True} + if metadata_only + else critic_history_options(actor, phase, restore=restoring) + ) + if phase.algorithm == "ppo": + trainable["value"] = HistoryCritic.ppo_value(actor, **history).eval() + else: + value, q1, q2, targets = make_iql_models( + actor.execution_config, + actor.interface_name, + actor.config.actions, + **history, + ) + trainable.update( + value=value.eval(), critic=nn.ModuleList([q1, q2]).eval() + ) + frozen = { + f"target_q{i}": target.eval() for i, target in enumerate(targets, 1) + } + return cls(trainable, frozen) + + +@dataclass(frozen=True) +class PhaseOptimization: + step_budget: int | None + configs: dict[str, OptimizationConfig] + masks: dict[str, list[str]] + + @classmethod + def resolve(cls, phase, roles, training_size): + budget = phase.optimizer_step_budget(training_size) + masks = {} + for role, module in roles.trainable.items(): + frozen = phase.trainability.policy(role).resolve(module) + masks[role] = sorted( + name + for name, parameter in module.named_parameters() + if parameter.requires_grad and id(parameter) not in frozen + ) + if not masks[role]: + raise ValueError(f"{phase.name}: {role} leaves no trainable parameters") + configs = { + role: config.resolve_budget(budget) + for role, config in phase.optimization_configs().items() + } + return cls(budget, configs, masks) + + def build(self, roles): + if roles.trainable.keys() != self.configs.keys(): + raise ValueError("phase optimization roles disagree with trainable modules") + for role, module in roles.trainable.items(): + parameters = dict(module.named_parameters()) + mask = set(self.masks[role]) + if mask - parameters.keys(): + raise ValueError( + f"{role}: trainable parameter names disagree with preflight" + ) + for name, parameter in parameters.items(): + parameter.requires_grad_(name in mask) + return { + role: self.configs[role].build(module, step_budget=self.step_budget) + for role, module in roles.trainable.items() + } diff --git a/src/sequifier/rl/policy.py b/src/sequifier/rl/policy.py new file mode 100644 index 000000000..facdedb70 --- /dev/null +++ b/src/sequifier/rl/policy.py @@ -0,0 +1,200 @@ +"""Policy alignment adapter over Sequifier's configurable model system.""" + +from __future__ import annotations + +from collections.abc import Mapping +from typing import TYPE_CHECKING + +import torch +from torch import Tensor, nn + +from sequifier.config.policy_config import ResolvedPolicyBinding +from sequifier.io.batch import SequifierInputs +from sequifier.io.policy_window import normalize_policy_features +from sequifier.io.preprocessing import normalization_statistics +from sequifier.model.execution import ModelExecution +from sequifier.model.execution_schema import ExecutionSchema +from sequifier.model.prediction_distribution import PredictionScales +from sequifier.rl.action_plan import ActionDecision, ActionExecutionPlan +from sequifier.rl.joint_distribution import JointActionDistribution + +if TYPE_CHECKING: + from sequifier.model.network import ComposableTransformerNetwork + + +class Policy(nn.Module): + """A convenience view; the network owns all trainable weights and execution.""" + + config: ResolvedPolicyBinding + execution_config: ModelExecution + interface_name: str + network: ComposableTransformerNetwork + units: dict[str, str] + transfer_report: dict | None + ppo_policy_version: int + grpo_policy_version: int + + def __init__(self): + super().__init__() + self.units = {} + self.transfer_report = None + + @classmethod + def from_network(cls, network, execution_config, interface_name: str): + interface = execution_config.interface(interface_name) + if interface.policy is None: + raise ValueError( + f"Model interface {interface_name!r} has no policy alignment" + ) + policy = cls() + policy.config = interface.policy + policy.units = dict(interface.policy.units) + policy.execution_config = execution_config + policy.interface_name = interface_name + policy.network = network + policy._resolve_actions() + return policy + + @property + def interface(self): + return self.execution_config.interface(self.interface_name) + + @property + def device(self) -> torch.device: + return next(self.network.parameters()).device + + @property + def backbone(self): + return self.network.backbone + + def _resolve_actions(self): + route = self.network.resolve_interface(self.interface_name) + self.action_plan = ActionExecutionPlan(self, route.require_decoder()) + self.action_order = self.action_plan.order + self.autoregressive_actions = self.action_plan.has_autoregression + if route.prediction_scales is None: + route.prediction_scales = PredictionScales( + self.interface.prediction_distributions + ).to(self.device) + else: + route.prediction_scales.ensure_fields( + self.interface.prediction_distributions + ) + route.prediction_scales.to(self.device) + + def action_decoder_ids(self, name): + field = self.config.actions[name] + mapping = self.interface.id_maps.get(name, {}) + support = set(self.interface.target_decoder_ids.get(name, ())) + ids = [] + for label in field.values: + key = ( + label + if isinstance(label, (str, int)) + else int(label) + if label.is_integer() + else str(label) + ) + global_id = mapping.get(key, mapping.get(str(label), mapping.get(str(key)))) + if global_id is None: + if isinstance(label, int) and label in support: + global_id = label + else: + raise ValueError( + f"{name}: action label {label!r} is absent from the model codec" + ) + if global_id not in support: + raise ValueError(f"{name}: action is absent from decoder support") + ids.append(global_id) + if not ids or len(ids) != len(set(ids)): + raise ValueError( + f"{name}: categorical actions need distinct executable decoder IDs" + ) + return ids + + def set_normalization_statistics( + self, statistics: Mapping[str, Mapping[str, float]] + ) -> None: + self.interface.selected_columns_statistics = ( + normalization_statistics(statistics) + if self.interface.normalize_real_columns + else {name: dict(values) for name, values in statistics.items()} + ) + return + + def model_inputs( + self, inputs: SequifierInputs + ) -> tuple[dict[str, Tensor], dict[str, Tensor]]: + """Encode raw named histories without mutating stored data or masks.""" + self.validate_inputs(inputs) + features = normalize_policy_features( + inputs.inputs, inputs.metadata, self.interface + ) + if any( + t.is_floating_point() and not torch.isfinite(t).all() + for t in features.values() + ): + raise ValueError("frozen input transforms produced non-finite values") + return features, inputs.metadata + + def forward(self, inputs: SequifierInputs) -> Tensor | dict[str, Tensor]: + return self.prepared_logits(*self.model_inputs(inputs)) + + def prepared_logits(self, features, metadata) -> Tensor | dict[str, Tensor]: + """Execute features already checked and normalized by model_inputs.""" + outputs = self.network( + features, metadata, interface_name=self.interface_name + ).logits + return { + name: value.masked_fill( + ~metadata["attention_valid_mask"][:, -value.shape[1] :, None], 0.0 + ) + for name, value in outputs.items() + } + + def distribution(self, inputs: SequifierInputs) -> JointActionDistribution: + return JointActionDistribution(self, inputs) + + def act(self, inputs: SequifierInputs, *, deterministic: bool, generator): + selected = self.distribution(inputs).select_and_score(deterministic, generator) + return selected.actions, selected.log_prob + + def evaluate_actions( + self, + inputs: SequifierInputs, + actions: ActionDecision | Mapping[str, Tensor], + ) -> Tensor: + return self.distribution(inputs).log_prob(actions) + + def validate_inputs(self, inputs: SequifierInputs) -> None: + schema = ExecutionSchema.from_interface( + self.interface, self.config.context_length + ) + expected = {item.key for item in schema.inputs if item.role != "metadata"} + if set(inputs.inputs) != expected: + raise ValueError( + "policy inputs must match the persisted named feature schema" + ) + mask = inputs.attention_valid_mask + if ( + mask.ndim != 2 + or mask.dtype != torch.bool + or mask.shape[1] != self.config.context_length + or mask.shape[0] < 1 + ): + raise ValueError( + "policy attention masks must be bool [batch, context_length]" + ) + if not mask[:, -1].all() or (mask[:, :-1] & ~mask[:, 1:]).any(): + raise ValueError( + "policy histories require left padding followed by valid positions" + ) + ExecutionSchema.from_interface( + self.interface, self.config.context_length + ).validate(inputs.inputs, inputs.metadata) + if any( + value.device != self.device + for value in [*inputs.inputs.values(), *inputs.metadata.values()] + ): + raise ValueError("policy features and masks must be on the model device") + return diff --git a/src/sequifier/rl/ppo.py b/src/sequifier/rl/ppo.py new file mode 100644 index 000000000..5e1489b07 --- /dev/null +++ b/src/sequifier/rl/ppo.py @@ -0,0 +1,120 @@ +"""Phase-local PPO lifecycle over the shared collector and clipped actor/V update.""" + +from copy import deepcopy +from pathlib import Path + +from sequifier.config.policy_plan import PPOPhaseConfig +from sequifier.rl.environment import ( + adapter_identity, + load_online_adapter, + require_exact_snapshot, +) +from sequifier.rl.ppo_collector import PPOCollector +from sequifier.rl.ppo_update import update_ppo +from sequifier.rl.run_state import PhaseContext, PhaseSession, require_equal +from sequifier.rl.snapshots import snapshots_equal +from sequifier.runtime.resource_accounting import allocation + + +def run_ppo_phase(context: PhaseContext[PPOPhaseConfig, None]): + actor, phase = context.actor.eval(), context.phase + session = PhaseSession(context) + value = session.modules["value"] + state = session.state + env_steps = state.get("env_steps", 0) + if ( + env_steps + != min(session.transaction * phase.rollout_env_steps, phase.total_env_steps) + or not 0 <= env_steps <= phase.total_env_steps + ): + raise ValueError("PPO checkpoint is not a complete collection/update boundary") + actor.ppo_policy_version = session.transaction + adapter = load_online_adapter(phase.environment) + collector = None + try: + if context.run.resume.mode == "exact": + require_exact_snapshot(adapter) + identity = adapter_identity(adapter) + collector = PPOCollector( + actor, value, adapter, phase, context.identity.sampling_seed + ) + if state: + require_equal(state["adapter_identity"], identity, "ppo.adapter_identity") + exact = context.run.resume.mode == "exact" + if exact: + if state["environment_state"] is None: + raise ValueError( + "exact PPO resume requires an embedded environment snapshot" + ) + adapter.restore_state(deepcopy(state["environment_state"])) + if not snapshots_equal( + adapter.snapshot_state(), state["environment_state"] + ): + raise ValueError("PPO adapter did not restore its snapshot exactly") + collector.load_state_dict(state["collector"], exact=exact) + session.record_continuation( + mode=context.run.resume.mode, trajectory_equivalent=exact + ) + session.restore_random() + if session.should_evaluate_initial(): + session.evaluate_online(adapter, initial=True) + while env_steps < phase.total_env_steps and not session.stopping.triggered: + horizon = min(phase.rollout_env_steps, phase.total_env_steps - env_steps) + with allocation("rollout_storage") as storage: + buffer, episodes = collector.collect( + horizon, session.transaction, storage=storage + ) + if phase.archive_directory: + with allocation("rollout_archive") as archive: + archive.resize(buffer.estimated_bytes) + buffer.archive( + Path(phase.archive_directory) + / f"collection-{session.transaction + 1:08d}.json", + include_scores=phase.archive_scores, + ) + record = update_ppo(session, value, buffer) + del buffer + env_steps += horizon + session.transaction += 1 + final = env_steps == phase.total_env_steps + if session.should_evaluate(final): + session.evaluate_online(adapter, env_steps=env_steps) + session.record_training( + final=final, + **{ + **record, + "transaction": session.transaction, + "env_steps": env_steps, + "policy_version": actor.ppo_policy_version, + "episodes": episodes, + }, + ) + if session.should_checkpoint(final): + environment_state = None + if adapter.capabilities.exact_snapshot: + require_exact_snapshot(adapter) + environment_state = adapter.snapshot_state() + if not isinstance(environment_state, dict) or not snapshots_equal( + environment_state, environment_state + ): + raise ValueError( + "PPO snapshot must be a finite portable state dictionary" + ) + session.checkpoint( + final, + env_steps=env_steps, + collector=collector.state_dict(), + environment_state=environment_state, + adapter_identity=identity, + ) + finally: + if collector is not None: + collector.cursor.close() + adapter.close() + return session.finish( + { + "env_steps": env_steps, + **session.optimizer_clocks(), + }, + {"adapter": identity}, + ) diff --git a/src/sequifier/rl/ppo_buffer.py b/src/sequifier/rl/ppo_buffer.py new file mode 100644 index 000000000..c294977de --- /dev/null +++ b/src/sequifier/rl/ppo_buffer.py @@ -0,0 +1,296 @@ +"""Compact, single-generation on-policy storage and fixed FP32 GAE targets.""" + +import json +from contextlib import contextmanager +from dataclasses import asdict, dataclass + +import torch + +from sequifier.config.policy_config import ResolvedPolicyBinding +from sequifier.contract_versions import ContractVersions +from sequifier.io.actions import action_batch, decode_action +from sequifier.io.atomic import write_json_atomic +from sequifier.rl.history import frame_storage_bytes, history_inputs +from sequifier.rl.window_storage import collated_windows + + +@dataclass(frozen=True) +class PPODecision: + frame: int + successor: int | None + action: int | dict + domain_action: str | float | int | dict + old_log_probability: float + old_value: float + entropy: float + reward: float + raw_reward: float + successor_value: float + terminated: bool + truncated: bool + bootstrap: bool + episode_id: str + step_id: int + policy_version: int + + +@dataclass(frozen=True) +class PPORolloutBuffer: + """Sealed tuples retain each frame once, including the pre-buffer context. + + Access reconstructs complete ordered windows. No optimization tensor is + shared with the fixed old scores or targets held by this buffer. + """ + + spec: ResolvedPolicyBinding + policy_version: int + frames: tuple + decisions: tuple[PPODecision, ...] + advantages: tuple[float, ...] + targets: tuple[float, ...] + estimated_bytes: int + + def __len__(self): + return len(self.decisions) + + def require_version(self, version): + if version != self.policy_version or any( + step.policy_version != version for step in self.decisions + ): + raise ValueError("PPO cannot update from another policy generation") + + def window(self, reference): + frame = self.frames[reference] + start = reference + while ( + start > 0 + and reference - start + 1 < self.spec.context_length + and self.frames[start - 1].episode_id == frame.episode_id + ): + if self.frames[start - 1].step_id + 1 != self.frames[start].step_id: + raise ValueError("PPO frame history is discontinuous") + start -= 1 + return history_inputs(self.spec, self.frames[start : reference + 1]) + + def batch(self, indices, device): + with self.batch_scope(indices, device) as batch: + return batch + + @contextmanager + def batch_scope(self, indices, device): + indices = [int(index) for index in indices] + with collated_windows( + (self.window(self.decisions[index].frame) for index in indices), device + ) as inputs: + yield ( + inputs, + action_batch( + [self.decisions[index].action for index in indices], + self.spec, + device=device, + ), + torch.tensor( + [self.decisions[index].old_log_probability for index in indices], + dtype=torch.float32, + device=device, + ), + torch.tensor( + [self.decisions[index].old_value for index in indices], + dtype=torch.float32, + device=device, + ), + torch.tensor( + [self.advantages[index] for index in indices], + dtype=torch.float32, + device=device, + ), + torch.tensor( + [self.targets[index] for index in indices], + dtype=torch.float32, + device=device, + ), + ) + + def archive(self, path, *, include_scores): + """A separate versioned collection record, never an offline update source.""" + decisions = [] + for index, step in enumerate(self.decisions): + record = asdict(step) + boundary = step.terminated or step.truncated + cut = index == len(self) - 1 and not boundary + record["cutoff_reason"] = ( + "terminated" + if step.terminated + else "truncated" + if step.truncated + else "collection_cut" + if cut + else None + ) + if not include_scores: + for key in ( + "old_log_probability", + "old_value", + "entropy", + "successor_value", + ): + record.pop(key) + decisions.append(record) + payload = { + "format": "sequifier.ppo-collection", + "version": ContractVersions.PPO_COLLECTION, + "policy_version": self.policy_version, + "policy": self.spec.model_dump(mode="json"), + "frames": [frame.state_dict() for frame in self.frames], + "decisions": decisions, + } + from sequifier.trajectory.archives import collection_archive + + write_json_atomic(path, collection_archive("ppo", payload)) + + +def load_collection_record(path): + """Read and validate a current PPO collection record.""" + from sequifier.trajectory.archives import read_collection_archive + + payload = read_collection_archive(path, "ppo") + from sequifier.config.policy_config import ResolvedPolicyBinding + from sequifier.rl.history import PolicyFrame + + spec = ResolvedPolicyBinding.model_validate(payload["policy"]) + frames = [PolicyFrame.from_state_dict(frame) for frame in payload["frames"]] + for frame in frames: + history_inputs(spec, [frame]) + for step in payload["decisions"]: + frame = frames[step["frame"]] + if ( + step["policy_version"] != payload["policy_version"] + or (step["episode_id"], step["step_id"]) + != (frame.episode_id, frame.step_id) + or step["domain_action"] != (decode_action(step["action"], spec)) + or step["bootstrap"] != (not step["terminated"]) + ): + raise ValueError("collection record violates the decision contract") + if step["successor"] is None: + if step["bootstrap"]: + raise ValueError("collection record lacks a bootstrap successor") + else: + successor = frames[step["successor"]] + if ( + successor.episode_id != frame.episode_id + or successor.step_id != frame.step_id + 1 + or successor.previous_action != step["action"] + ): + raise ValueError("collection record successor is discontinuous") + if not torch.isfinite(torch.tensor(step["reward"], dtype=torch.float32)): + raise ValueError("collection record reward must be finite FP32") + return payload + + +def compute_gae(decisions, gamma, gae_lambda, normalize): + """Boundary-aware FP32 targets; normalization happens once per iteration.""" + advantages = torch.zeros(len(decisions), dtype=torch.float32) + following_advantage = torch.zeros((), dtype=torch.float32) + for index in range(len(decisions) - 1, -1, -1): + step = decisions[index] + continues = ( + index + 1 < len(decisions) + and not (step.terminated or step.truncated) + and decisions[index + 1].episode_id == step.episode_id + and decisions[index + 1].step_id == step.step_id + 1 + ) + delta = ( + torch.tensor(step.reward, dtype=torch.float32) + + gamma + * float(step.bootstrap) + * torch.tensor(step.successor_value, dtype=torch.float32) + - torch.tensor(step.old_value, dtype=torch.float32) + ) + following_advantage = ( + delta + gamma * gae_lambda * float(continues) * following_advantage + ) + advantages[index] = following_advantage + targets = advantages + torch.tensor( + [step.old_value for step in decisions], dtype=torch.float32 + ) + if normalize and len(decisions) > 1: + advantages = (advantages - advantages.mean()) / advantages.std( + unbiased=False + ).clamp_min(1e-8) + if not torch.isfinite(advantages).all() or not torch.isfinite(targets).all(): + raise ValueError("non-finite PPO advantage or target") + return tuple(advantages.tolist()), tuple(targets.tolist()) + + +class PPOBufferBuilder: + def __init__( + self, spec, version, horizon, initial_frames, max_bytes, *, storage=None + ): + self.spec = spec.model_copy(deep=True) + self.version = version + self.horizon = horizon + self.frames = list(initial_frames) + self.decisions = [] + self.max_bytes = max_bytes + self.bytes_per_frame = max( + [frame_storage_bytes(frame) for frame in self.frames] + [1024] + ) + self.estimated_bytes = ( + 2 * horizon + spec.context_length + ) * self.bytes_per_frame + horizon * 1024 + if self.estimated_bytes > max_bytes: + raise ValueError( + f"PPO buffer estimate {self.estimated_bytes} exceeds max_buffer_bytes={max_bytes}" + ) + self.used_bytes = sum(frame_storage_bytes(frame) for frame in self.frames) + self.sealed = False + self.decision_bytes = 0 + self.storage = storage + self._account() + + def _account(self): + if self.storage is not None: + self.storage.resize( + max(self.estimated_bytes, self.used_bytes + self.decision_bytes) + ) + + def append_frame(self, frame): + if self.sealed: + raise ValueError("PPO buffer is sealed") + cost = frame_storage_bytes(frame) + if self.used_bytes + cost + self.decision_bytes > self.max_bytes: + raise ValueError("PPO history exceeded its declared memory budget") + if len(self.frames) >= 2 * self.horizon + self.spec.context_length: + raise ValueError("PPO history exceeded its bounded frame capacity") + self.frames.append(frame) + self.used_bytes += cost + self._account() + return len(self.frames) - 1 + + def append(self, decision): + if self.sealed or len(self.decisions) >= self.horizon: + raise ValueError("PPO buffer is sealed or full") + if decision.policy_version != self.version: + raise ValueError("policy generation changed during collection") + self.decision_bytes += 1024 + 4 * len( + json.dumps(asdict(decision), allow_nan=False).encode() + ) + self.decisions.append(decision) + self._account() + if self.used_bytes + self.decision_bytes > self.max_bytes: + raise ValueError("PPO decisions exceeded the memory budget") + + def seal(self, gamma, gae_lambda, normalize): + if self.sealed or len(self.decisions) != self.horizon: + raise ValueError("PPO must seal exactly one complete collection horizon") + self.sealed = True + advantages, targets = compute_gae(self.decisions, gamma, gae_lambda, normalize) + return PPORolloutBuffer( + self.spec, + self.version, + tuple(self.frames), + tuple(self.decisions), + advantages, + targets, + max(self.estimated_bytes, self.used_bytes + self.decision_bytes), + ) diff --git a/src/sequifier/rl/ppo_collector.py b/src/sequifier/rl/ppo_collector.py new file mode 100644 index 000000000..2f298d528 --- /dev/null +++ b/src/sequifier/rl/ppo_collector.py @@ -0,0 +1,164 @@ +"""One persistent environment slot; horizons cut history without resetting it.""" + +import torch + +from sequifier.rl.collection import PolicyCursor, stream_seed +from sequifier.rl.ppo_buffer import PPOBufferBuilder, PPODecision + + +class PPOCollector: + def __init__(self, actor, value, adapter, phase, seed): + self.actor = actor + self.value = value + self.adapter = adapter + self.phase = phase + self.seed = seed + self.cursor = PolicyCursor( + actor, + adapter, + phase.environment.units, + stream_seed(seed, "training-actions", 0), + ) + + @property + def reset_count(self): + return self.cursor.history.reset_count + + def reset(self): + self.cursor.reset( + self.adapter.reset( + stream_seed(self.seed, "training-reset", self.reset_count) + ), + f"train-{self.reset_count}", + ) + + def collect(self, horizon, policy_version, *, storage=None): + if self.actor.ppo_policy_version != policy_version: + raise ValueError("collection policy generation disagrees with the actor") + self.actor.eval() + self.value.eval() + # Preflight before the first reset where possible; exact domain size is + # checked again after reset and on each insertion. + minimum = (2 * horizon + self.actor.config.context_length) * ( + 1024 + 128 * len(self.actor.interface.input_columns) + ) + horizon * 1024 + if minimum > self.phase.max_buffer_bytes: + raise ValueError( + f"PPO buffer requires approximately {minimum} bytes before collection" + ) + if self.cursor.ended: + self.reset() + builder = PPOBufferBuilder( + self.actor.config, + policy_version, + horizon, + self.cursor.history.frames, + self.phase.max_buffer_bytes, + storage=storage, + ) + current_reference = len(builder.frames) - 1 + completed = [] + with torch.no_grad(): + for _ in range(horizon): + if self.cursor.ended: + self.reset() + current_reference = builder.append_frame( + self.cursor.history.current + ) + if current_reference is None: + raise ValueError( + "live PPO decision lacks its observation reference" + ) + current = self.cursor.history.current + decision = self.cursor.decide(self.value) + result = self.cursor.step(decision) + successor_reference = None + successor_value = 0.0 + if result.observation is not None: + features, metadata = self.actor.model_inputs(self.cursor.inputs) + if not result.terminated: + successor_value = float( + self.value.prepared_value(features, metadata)[0] + ) + if not torch.isfinite( + torch.tensor(successor_value, dtype=torch.float32) + ): + raise ValueError("non-finite PPO successor value") + successor_reference = builder.append_frame( + self.cursor.history.current + ) + reward = float(result.reward) * self.phase.reward_scale + if not torch.isfinite( + torch.tensor([reward, successor_value], dtype=torch.float32) + ).all(): + raise ValueError("reward or successor V exceeds finite FP32 range") + builder.append( + PPODecision( + current_reference, + successor_reference, + decision.action, + decision.domain_action, + decision.log_probability, + decision.value, + decision.entropy, + reward, + float(result.reward), + successor_value, + result.terminated, + result.truncated, + not result.terminated, + current.episode_id, + current.step_id, + policy_version, + ) + ) + current_reference = successor_reference + if self.cursor.ended: + completed.append( + { + "episode_id": current.episode_id, + "return": self.cursor.episode_return, + "length": self.cursor.episode_length, + "terminated": result.terminated, + "truncated": result.truncated, + "success": result.success, + } + ) + return builder.seal( + self.phase.gamma, self.phase.gae_lambda, self.phase.normalize_advantages + ), completed + + def state_dict(self): + return { + "history": self.cursor.history.state_dict(), + "reset_count": self.reset_count, + "generator": self.cursor.generator.get_state(), + "ended": self.cursor.ended, + "episode_return": self.cursor.episode_return, + "episode_length": self.cursor.episode_length, + } + + def load_state_dict(self, state, *, exact): + reset_count = int(state["reset_count"]) + if reset_count < 1: + raise ValueError("PPO checkpoint has an invalid reset count") + self.cursor.generator.set_state(state["generator"]) + if exact: + self.cursor.history.load_state_dict(state["history"]) + self.cursor.desired_return = self.cursor.history.current.desired_return + self.cursor.refresh() + if ( + self.reset_count != reset_count + or self.cursor.history.current.episode_id != f"train-{reset_count - 1}" + ): + raise ValueError( + "PPO checkpoint episode identity disagrees with reset count" + ) + self.cursor.ended = bool(state["ended"]) + self.cursor.episode_return = float(state["episode_return"]) + self.cursor.episode_length = int(state["episode_length"]) + else: + self.cursor.history.reset_count = reset_count + self.cursor.ended = True + self.cursor.episode_return = 0.0 + self.cursor.episode_length = 0 diff --git a/src/sequifier/rl/ppo_evaluation.py b/src/sequifier/rl/ppo_evaluation.py new file mode 100644 index 000000000..bac4697a0 --- /dev/null +++ b/src/sequifier/rl/ppo_evaluation.py @@ -0,0 +1,158 @@ +"""Bounded local policy evaluation with independent seeds, adapters and RNG.""" + +import signal +import statistics +import threading +import time +from contextlib import contextmanager + +from sequifier.rl.collection import PolicyCursor, stream_seed +from sequifier.rl.environment import load_online_adapter +from sequifier.runtime.random_state import capture_rng_state, restore_rng_state + + +class EvaluationTimeout(TimeoutError): + pass + + +def require_evaluation_timer(): + if ( + threading.current_thread() is not threading.main_thread() + or "setitimer" not in vars(signal) + ): + raise ValueError( + "bounded local PPO evaluation requires the main thread and a POSIX interval timer" + ) + + +@contextmanager +def _deadline(seconds): + require_evaluation_timer() + previous_handler = signal.getsignal(signal.SIGALRM) + previous_timer = signal.getitimer(signal.ITIMER_REAL) + started = time.monotonic() + + def expired(signum, frame): + raise EvaluationTimeout("PPO evaluation exceeded timeout_seconds") + + signal.signal(signal.SIGALRM, expired) + try: + signal.setitimer(signal.ITIMER_REAL, seconds) + yield + finally: + signal.setitimer(signal.ITIMER_REAL, 0) + signal.signal(signal.SIGALRM, previous_handler) + if previous_timer[0] > 0: + signal.setitimer( + signal.ITIMER_REAL, + max(1e-6, previous_timer[0] - (time.monotonic() - started)), + previous_timer[1], + ) + + +def evaluate_policy(actor, evaluation, *, forbidden_adapter=None): + """Evaluate synchronously with stochastic layers disabled; completed returns exclude censored episodes. + + The timeout covers adapter creation/reset/steps for the entire evaluation. + Adapters must release resources promptly from close(). + """ + require_evaluation_timer() + if evaluation.environment.units != actor.units: + raise ValueError( + "evaluation environment units differ from the portable actor representation" + ) + saved_random = capture_rng_state(actor.device) + training_modes = {module: module.training for module in actor.modules()} + started = time.monotonic() + adapter = None + episodes = [] + timed_out = False + try: + actor.eval() + with _deadline(evaluation.timeout_seconds): + adapter = load_online_adapter(evaluation.environment) + if adapter is forbidden_adapter: + adapter = None # The training owner must retain and close it. + raise ValueError( + "evaluation factory returned the live training adapter" + ) + for index, seed in enumerate(evaluation.seeds): + cursor = PolicyCursor( + actor, + adapter, + evaluation.environment.units, + stream_seed(seed, "evaluation-actions", 0), + ) + completed_episode = False + try: + cursor.reset( + adapter.reset(stream_seed(seed, "evaluation-reset", 0)), + f"evaluation-{index}", + ) + for _ in range(evaluation.max_episode_steps): + step = cursor.step( + cursor.decide( + deterministic=evaluation.action_selection + == "deterministic" + ) + ) + if cursor.ended: + completed_episode = step.terminated + break + finally: + episodes.append( + { + "seed": seed, + "return": cursor.episode_return, + "length": cursor.episode_length, + "completed": completed_episode, + "censored": not completed_episode, + "success": cursor.last_result.success + if completed_episode and cursor.last_result is not None + else None, + } + ) + cursor.close() + except EvaluationTimeout: + timed_out = True + finally: + try: + if adapter is not None: + adapter.close() + finally: + for module, training in training_modes.items(): + module.training = training + restore_rng_state(actor.device, saved_random) + completed = [episode for episode in episodes if episode["completed"]] + returns = [episode["return"] for episode in completed] + lengths = [episode["length"] for episode in completed] + successes = [episode["success"] for episode in completed] + result = { + "mean_completed_return": statistics.fmean(returns) if returns else None, + "return_std": statistics.pstdev(returns) if returns else None, + "return_min": min(returns) if returns else None, + "return_max": max(returns) if returns else None, + "mean_completed_length": statistics.fmean(lengths) if lengths else None, + "mean_length": statistics.fmean(episode["length"] for episode in episodes) + if episodes + else None, + "completed_count": len(completed), + "censored_count": len(episodes) - len(completed), + "unattempted_count": len(evaluation.seeds) - len(episodes), + "timed_out": timed_out, + "episodes": episodes, + "seconds": time.monotonic() - started, + "action_selection": evaluation.action_selection, + } + if successes and all(success is not None for success in successes): + result["success_rate"] = statistics.fmean(successes) + if evaluation.monitor.metric == "score_mean": + if timed_out or len(episodes) != len(evaluation.seeds): + raise ValueError("GRPO evaluation requires every finite-horizon score") + result["score_mean"] = statistics.fmean( + episode["return"] for episode in episodes + ) + result["selection_metric"] = evaluation.monitor.metric + result["selection_value"] = result.get(evaluation.monitor.metric) + result["selection_available"] = result["selection_value"] is not None + return result diff --git a/src/sequifier/rl/ppo_update.py b/src/sequifier/rl/ppo_update.py new file mode 100644 index 000000000..822328915 --- /dev/null +++ b/src/sequifier/rl/ppo_update.py @@ -0,0 +1,181 @@ +"""PPO objectives and sample-weighted independent actor/value optimization.""" + +from typing import Any + +import torch + +from sequifier.rl.batching import MetricTotals, batches, shuffled_groups +from sequifier.rl.clipped_objective import clipped_policy_objective + + +def ppo_objectives( + log_probability, + old_log_probability, + advantages, + prediction, + old_value, + target, + entropy, + phase, +): + policy = clipped_policy_objective( + log_probability, old_log_probability, advantages, phase.clip_epsilon + ) + policy_loss, ratio = policy.loss, policy.ratio + value_error = (prediction.float() - target).square() + if phase.value_clip_delta is not None: + clipped_value = old_value + (prediction.float() - old_value).clamp( + -phase.value_clip_delta, phase.value_clip_delta + ) + value_error = torch.maximum(value_error, (clipped_value - target).square()) + value_loss = 0.5 * value_error + approximate_kl = policy.approximate_kl + tensors = (ratio, policy_loss, value_loss, entropy, approximate_kl) + if any(not torch.isfinite(tensor).all() for tensor in tensors): + raise ValueError("non-finite PPO probability ratio or objective") + return ( + policy_loss, + value_loss, + { + "policy_loss": policy_loss, + "value_loss": value_loss, + "entropy": entropy, + "approximate_kl": approximate_kl, + "clip_fraction": policy.clip_fraction, + "ratio": ratio, + }, + ) + + +@torch.no_grad() +def validate_collection_scores(actor, value, buffer, batch_size): + """Check both identity ratio and V before the first optimizer transaction.""" + actor.eval() + value.eval() + for start in range(0, len(buffer), batch_size): + indices = range(start, min(start + batch_size, len(buffer))) + with buffer.batch_scope(indices, actor.device) as batch: + history, actions, old_logp, old_v, *_ = batch + if not torch.allclose( + actor.evaluate_actions(history, actions), old_logp, atol=1e-5, rtol=1e-4 + ): + raise ValueError( + "PPO reconstructed action scores differ from collection" + ) + if not torch.allclose(value(history), old_v, atol=1e-5, rtol=1e-4): + raise ValueError("PPO reconstructed values differ from collection") + del batch, history + + +def update_ppo(session, value, buffer): + actor, phase = session.context.actor, session.context.phase + policy_version, clocks = session.transaction, session.steps + buffer.require_version(policy_version) + if actor.ppo_policy_version != policy_version: + raise ValueError( + "PPO buffer has already been consumed or belongs to a stale actor" + ) + actor.eval() # Keep dropout off while retaining autograd. + value.eval() + validate_collection_scores(actor, value, buffer, phase.batch_size) + totals = MetricTotals( + "policy_loss", "value_loss", "entropy", "approximate_kl", "clip_fraction" + ) + ratio_chunks = [] + count = 0 + norms = {"actor": [], "value": []} + stopped = False + group_size = phase.batch_size * phase.accumulation_batches + epochs = 0 + for epoch in range(phase.optimization_epochs): + for group in shuffled_groups(len(buffer), group_size, session.sampler): + minibatches = list(batches(group, phase.batch_size)) + if phase.old_policy_target_kl is not None: + kl_sum = 0.0 + with torch.no_grad(): + for indices in minibatches: + with buffer.batch_scope(indices, actor.device) as batch: + history, actions, old_logp, _, _, _ = batch + log_ratio = ( + actor.evaluate_actions(history, actions) - old_logp + ) + kl = log_ratio.exp() - 1 - log_ratio + if not torch.isfinite(kl).all(): + raise ValueError("non-finite PPO KL") + kl_sum += float(kl.sum()) + del batch, history + if kl_sum / len(group) > phase.old_policy_target_kl: + stopped = True + break + session.zero_grad() + for indices in minibatches: + with buffer.batch_scope(indices, actor.device) as batch: + history, actions, old_logp, old_v, advantages, target = batch + distribution = actor.distribution(history) + policy_loss, value_loss, metrics = ppo_objectives( + distribution.log_prob(actions), + old_logp, + advantages, + value(history), + old_v, + target, + distribution.entropy(), + phase, + ) + actor_loss = ( + policy_loss - phase.entropy_coefficient * metrics["entropy"] + ).sum() / len(group) + critic_loss = ( + phase.value_coefficient * value_loss.sum() / len(group) + ) + if not torch.isfinite(actor_loss) or not torch.isfinite( + critic_loss + ): + raise ValueError("non-finite weighted PPO loss") + actor_loss.backward() + critic_loss.backward() + totals.add(metrics) + ratio_chunks.append(metrics["ratio"].detach().cpu()) + count += len(indices) + del batch, history, distribution + for role, norm in session.step().items(): + norms[role].append(norm) + epochs = epoch + 1 + if stopped: + break + if not count: + raise ValueError( + "PPO KL stopped all updates before an identity-ratio transaction" + ) + ratios = torch.cat(ratio_chunks) + targets = torch.tensor(buffer.targets, dtype=torch.float32) + with torch.no_grad(): + predictions = [] + for start in range(0, len(buffer), phase.batch_size): + indices = range(start, min(start + phase.batch_size, len(buffer))) + with buffer.batch_scope(indices, actor.device) as batch: + predictions.append(value(batch[0]).cpu()) + del batch + residual_variance = (targets - torch.cat(predictions)).var(unbiased=False) + target_variance = targets.var(unbiased=False) + result: dict[str, Any] = totals.means(count) + result.update( + { + "actor_optimizer_step": clocks["actor"], + "value_optimizer_step": clocks["value"], + "actor_grad_norm": sum(norms["actor"]) / len(norms["actor"]), + "value_grad_norm": sum(norms["value"]) / len(norms["value"]), + "ratio_min": float(ratios.min()), + "ratio_max": float(ratios.max()), + "ratio_p01": float(torch.quantile(ratios, 0.01)), + "ratio_p99": float(torch.quantile(ratios, 0.99)), + "explained_variance": float(1 - residual_variance / target_variance) + if target_variance > 1e-12 + else None, + "optimization_epochs": epochs, + "scored_decisions": count, + "kl_stopped": stopped, + } + ) + actor.ppo_policy_version = policy_version + 1 + return result diff --git a/src/sequifier/rl/preparation.py b/src/sequifier/rl/preparation.py new file mode 100644 index 000000000..0fa6772a0 --- /dev/null +++ b/src/sequifier/rl/preparation.py @@ -0,0 +1,199 @@ +"""Read-only policy run preparation and whole-plan preflight validation.""" + +from dataclasses import replace +from pathlib import Path + +import torch + +from sequifier.io.output_paths import ( + require_directory_destinations, + require_file_destinations, + unique_paths, + validate_path_conflicts, +) +from sequifier.model.factory import build_transformer_network, validate_hash_contracts +from sequifier.rl.checkpoint import phase_actor_path, validate_policy_weights +from sequifier.rl.environment import ( + adapter_identity, + load_online_adapter, + require_exact_snapshot, + require_group_start, +) +from sequifier.rl.policy import Policy +from sequifier.rl.ppo_evaluation import require_evaluation_timer +from sequifier.rl.prepared_inputs import PreparedTrainingInputs, RunInputs # noqa: F401 +from sequifier.rl.run_state import require_equal +from sequifier.runtime.random_state import capture_rng_state, restore_rng_state + + +def validate_output_paths(config, dependencies): + """Keep every output and archive separate from inputs and other outputs.""" + protected = dependencies.protected_paths + outputs = [ + Path(config.model_path), + Path(config.report_path), + Path(config.checkpoint_path), + ] + for i, phase in enumerate(config.phases, config.phase_offset): + actor = phase_actor_path(config.phase_directory, i, phase.name) + outputs += [actor, actor.with_name("report.json")] + message = "policy outputs and phase archives must have distinct paths" + outputs = unique_paths(outputs, message) + archives = unique_paths( + [ + Path(archive).resolve() + for phase in config.phases + if (archive := phase.rollout_archive_directory) + ], + message, + ) + require_file_destinations(outputs) + validate_path_conflicts(outputs, protected, label="policy output") + from sequifier.io.output_paths import validate_training_destinations + + validate_training_destinations( + config, config.checkpoint_path, protected, phase_offset=config.phase_offset + ) + require_directory_destinations(archives) + validate_path_conflicts(archives, protected | set(outputs), label="archive") + + +def resolve_normalization(policy, declaration): + real = ( + set(policy.observation_columns + policy.static_columns) + - policy.categorical_n_classes.keys() + - set(policy.categorical_columns) + ) + if declaration is None: + if real: + raise ValueError( + "online-first policy runs require frozen normalization or initialize_from" + ) + return {} + if declaration.source == "identity": + return {column: {"mean": 0.0, "std": 1.0} for column in sorted(real)} + require_equal( + sorted(real), sorted(declaration.statistics), "normalization.statistics fields" + ) + return declaration.statistics + + +def validate_declared_policy(declared, fitted, field): + expected, actual = declared.model_dump(mode="json"), fitted.model_dump(mode="json") + # Unspecified categorical codecs may be fitted once, or supplied by the incoming artifact. + for key in ("id_maps", "observation_n_classes", "static_n_classes"): + for column, value in expected.pop(key).items(): + require_equal(value, actual[key].get(column), f"{field}.{key}.{column}") + actual.pop(key) + require_equal(expected, actual, field) + + +def build_actor(config, binding=None, *, artifact=None): + execution = config.execution + name = config.policy_interface + # A runner owns its mutable policy view. Recovery supplies its saved binding. + interface = execution.interface(name).model_copy( + update={"policy": binding or config.policy} + ) + # Exact recovery depends on the selected route being built first. + interfaces = {name: interface} + interfaces.update( + (other, value) for other, value in execution.interfaces.items() if other != name + ) + execution = replace(execution, interfaces=interfaces) + network = build_transformer_network( + execution, + device=torch.device(config.device), + initialize=artifact is None, + restored_history_interface=name + if config.actor_initialization == "history_only" + else None, + ).network + if artifact is not None: + from sequifier.artifacts.model_config import load_attached_model_weights + + load_attached_model_weights(network, artifact) + validate_hash_contracts(network, execution) + actor = Policy.from_network(network, execution, name).eval() + actor.units = dict(config.units) + validate_policy_weights(actor) + return actor + + +def validate_data(data, actor): + validated = set() + for item in data: + if item is None or id(item) in validated: + continue + validated.add(id(item)) + item.validate(actor) + + +def preflight_environments(config): + # Adapter probes must not change the seeded initialization or phase RNG. + device = torch.device(config.device) + saved_random = capture_rng_state(device) + try: + identities = {} + for index, phase in enumerate(config.phases, config.phase_offset): + training_environment = phase.training_environment + if training_environment is None: + continue + require_evaluation_timer() + for role, spec in ( + ("training", training_environment), + ("evaluation", phase.evaluation_environment), + ): + adapter = load_online_adapter(spec) + try: + if role == "training": + if phase.algorithm == "grpo": + require_group_start( + adapter, exact=config.resume.mode == "exact" + ) + if phase.simulator_workers > 1: + from sequifier.rl.simulator_workers import ( + require_parallel_group_start, + ) + + require_parallel_group_start(adapter) + elif config.resume.mode == "exact": + require_exact_snapshot(adapter) + identities[f"{index}.{role}"] = adapter_identity(adapter) + finally: + adapter.close() + return identities + finally: + restore_rng_state(device, saved_random) + + +def preflight_optimization(config, actor, data): + """Resolve constructors and authored scheduler budgets before creating outputs.""" + from sequifier.rl.phase_optimization import PhaseOptimization, PhaseRoles + + baseline = { + name: parameter.requires_grad + for name, parameter in actor.network.named_parameters() + } + + def restore_trainability(): + for name, parameter in actor.network.named_parameters(): + parameter.requires_grad_(baseline[name]) + + plans = [] + try: + for phase, part in zip(config.phases, data): + restore_trainability() + roles = PhaseRoles.build(actor, phase, metadata_only=True) + plan = PhaseOptimization.resolve( + phase, roles, len(part.training) if part is not None else None + ) + if phase.optimizer_state == "continue" and plan.masks != plans[-1].masks: + raise ValueError( + "optimizer continuation requires identical resolved parameter masks" + ) + plan.build(roles) + plans.append(plan) + finally: + restore_trainability() + return plans diff --git a/src/sequifier/rl/prepared_inputs.py b/src/sequifier/rl/prepared_inputs.py new file mode 100644 index 000000000..d0c722a11 --- /dev/null +++ b/src/sequifier/rl/prepared_inputs.py @@ -0,0 +1,305 @@ +"""Run-owned source identities, episode catalogs and policy partitions.""" + +from contextlib import contextmanager +from copy import deepcopy +from dataclasses import dataclass, field +from pathlib import Path + +from sequifier.config.policy_config import PolicyBinding +from sequifier.config.policy_inputs import PolicyFittingSpec +from sequifier.fingerprints import fingerprint +from sequifier.io.episode_cache import EpisodeCache +from sequifier.io.memory_budget import resident_bytes +from sequifier.runtime.resource_accounting import ResourceAccounting +from sequifier.trajectory.sources import ( + ResolvedSequenceSource, + file_checksum, + resolve_source, +) + + +@dataclass(frozen=True) +class RunInputs: + sources: dict[str, ResolvedSequenceSource] + checksums: dict[Path, str] + _identity: dict[str, str] = field(repr=False) + + @classmethod + def from_sources(cls, sources, checksums): + identity = {str(path.resolve()): value for path, value in checksums.items()} + return cls(sources, checksums, identity) + + @classmethod + def resolve(cls, config): + prepared = config._prepared_inputs or PreparedTrainingInputs() + return prepared.resolve(config) + + @property + def identity(self): + return dict(self._identity) + + def revalidate(self, expected, field): + """Rehash every dependency, including manifests and retargeted links.""" + from sequifier.rl.run_state import require_equal + + require_equal( + expected, + {str(path.resolve()): file_checksum(path) for path in self.checksums}, + field, + ) + + +def _data_policy_contract(policy): + """Episode interpretation and eligibility, independent of fitted feature codecs.""" + binding = policy.binding if isinstance(policy, PolicyFittingSpec) else policy + values = binding.model_dump(mode="json") + return { + "binding": {key: values[key] for key in PolicyBinding.model_fields}, + "context_length": policy.context_length, + "interface_name": policy.interface_name, + "observation_columns": policy.observation_columns, + "static_columns": policy.static_columns, + "action_columns": policy.action_columns, + "input_encoding": policy.input_encoding, + } + + +class PreparedTrainingInputs: + """Private, in-memory preparation shared from metadata fitting through execution. + + Source interpretation keys episode records and split membership. Each + execution codec owns its views and caches for the run. Fitting views are + released as soon as metadata has been fitted. + """ + + def __init__(self): + self._sources = {} + self._checksums = {} + self._inputs = {} + self._episode_records = {} + self._memberships = {} + self._partitions = {} + self._catalogs = {} + self._preference_manifests = {} + self._episode_caches = [] + self.accounting = ResourceAccounting() + self._prepared_storage = self.accounting.allocation("prepared_data") + self._running = False + + def episode_cache(self, capacity): + cache = EpisodeCache(capacity, self.accounting.allocation("episode_caches")) + self._episode_caches.append(cache) + return cache + + def measure(self): + roots = ( + self._inputs, + self._sources, + self._checksums, + self._episode_records, + self._memberships, + self._partitions, + self._catalogs, + self._preference_manifests, + ) + seen = {id(self), *(id(cache) for cache in self._episode_caches)} + self._prepared_storage.resize(resident_bytes(roots, seen)) + + def _clear_data(self): + for cache in self._episode_caches: + cache.clear() + self._episode_caches.clear() + self._episode_records.clear() + self._memberships.clear() + self._partitions.clear() + self._catalogs.clear() + self._preference_manifests.clear() + + def release_data(self): + self._clear_data() + self.measure() + + def close(self): + self._clear_data() + self._inputs.clear() + self._sources.clear() + # Keep the resolved config's checksum baseline for subsequent runs. + self._prepared_storage.resize(0) + + @contextmanager + def run_scope(self): + if self._running: + yield + return + self._running = True + try: + with self.accounting.activate(): + yield + finally: + self.close() + self._running = False + self.accounting = ResourceAccounting() + self._prepared_storage = self.accounting.allocation("prepared_data") + + @classmethod + def for_run(cls, config): + if config._prepared_inputs is None: + config._prepared_inputs = cls() + return config._prepared_inputs + + def _phase_dependencies(self, config, phase, inputs): + if phase.algorithm == "dpo": + from sequifier.trajectory.preferences import load_preferences + + preferences = { + ref.source: config.preferences[ref.source] + for ref in [phase.preferences, *(phase.evaluation.sources or [])] + } + names = set() + for path in preferences.values(): + if path not in self._preference_manifests: + self._preference_manifests[path] = load_preferences(path).pairs + for pair in self._preference_manifests[path]: + names.update((pair.chosen.source, pair.rejected.source)) + if names - inputs.sources.keys(): + raise ValueError( + f"preferences reference unknown sources: {sorted(names - inputs.sources.keys())}" + ) + else: + preferences = {} + names = { + ref.source + for ref in [*phase.sources, *(phase.evaluation.sources or [])] + } + sources = {name: inputs.sources[name] for name in sorted(names)} + checksums = { + str(path): value + for source in sources.values() + for path, value in source.checksums.items() + } + checksums.update( + {path: inputs.checksums[Path(path)] for path in preferences.values()} + ) + return sources, preferences, checksums + + def phase_checksums(self, config, phase, inputs): + from sequifier.rl.registry import BUILTIN_ALGORITHMS + + if BUILTIN_ALGORITHMS[phase.algorithm].prepare_data is None: + return {} + return self._phase_dependencies(config, phase, inputs)[2] + + def resolve(self, config): + declarations = { + name: { + **source.model_dump(mode="json"), + "split_index": source.selected_split_index, + } + for name, source in config.sources.items() + } + key = fingerprint({"sources": declarations, "preferences": config.preferences}) + if key not in self._inputs: + sources, checksums = {}, {} + for name, source in config.sources.items(): + source_key = fingerprint(declarations[name]) + if source_key not in self._sources: + self._sources[source_key] = resolve_source(source) + resolved = self._sources[source_key] + sources[name] = resolved + for path, checksum in resolved.checksums.items(): + if path in self._checksums and self._checksums[path] != checksum: + raise ValueError(f"source changed during preparation: {path}") + self._checksums[path] = checksum + checksums[path] = checksum + for path in config.preferences.values(): + path = Path(path) + if path not in self._checksums: + self._checksums[path] = file_checksum(path) + checksums[path] = self._checksums[path] + self._inputs[key] = RunInputs.from_sources(sources, checksums) + return self._inputs[key] + + @contextmanager + def fitting_phase(self, config, phase, policy, inputs): + start = len(self._episode_caches) + try: + yield self._prepare_phase(config, phase, policy, inputs, fitting=True) + finally: + for cache in self._episode_caches[start:]: + cache.clear() + del self._episode_caches[start:] + self.measure() + + def _prepare_phase(self, config, phase, policy, inputs, *, fitting=False): + from sequifier.io.policy_dataset import PolicyDataCatalog + from sequifier.rl.offline_data import prepare_offline + from sequifier.rl.registry import BUILTIN_ALGORITHMS + + prepare = BUILTIN_ALGORITHMS[phase.algorithm].prepare_data + if prepare is None: + return None + if fitting and phase.algorithm == "iql": + prepare = prepare_offline + sources, preferences, checksums = self._phase_dependencies( + config, phase, inputs + ) + catalog_key = fingerprint( + { + "sources": { + name: { + "source": source.source.model_dump(mode="json"), + "split_index": source.source.selected_split_index, + "paths": [str(path) for path in source.paths], + "kind": source.kind, + "checksums": { + str(path): value for path, value in source.checksums.items() + }, + } + for name, source in sources.items() + }, + "policy": _data_policy_contract(policy), + "episode_cache_size": config.episode_cache_size, + } + ) + partition_key = fingerprint( + { + "catalog": catalog_key, + "preferences": preferences, + "identity": checksums, + "validation_fraction": config.validation_fraction, + "source_validation_fractions": config.source_validation_fractions, + "seed": config.seed, + } + ) + if fitting: + catalog_views, partitions = {}, {} + view_key = catalog_key + else: + codec_key = fingerprint(policy.model_dump(mode="json")) + catalog_views = self._catalogs + view_key = (catalog_key, codec_key) + partitions = self._partitions.setdefault((partition_key, codec_key), {}) + if view_key not in catalog_views: + catalog_views[view_key] = PolicyDataCatalog( + sources, + deepcopy(policy), + config.episode_cache_size, + preference_manifests=self._preference_manifests, + episode_records=self._episode_records.setdefault(catalog_key, {}), + cache_owner=self, + ) + return prepare( + config, + phase, + catalog_views[view_key], + partitions, + self._memberships.setdefault(partition_key, {}), + ) + + def prepare(self, config, policy, inputs): + data = [ + self._prepare_phase(config, phase, policy, inputs) + for phase in config.phases + ] + self.measure() + return data diff --git a/src/sequifier/rl/references.py b/src/sequifier/rl/references.py new file mode 100644 index 000000000..bb92b7dc6 --- /dev/null +++ b/src/sequifier/rl/references.py @@ -0,0 +1,121 @@ +"""Own reference resolution, validation and frozen weight reuse.""" + +from copy import deepcopy +from pathlib import Path + +from sequifier.rl.checkpoint import load_policy +from sequifier.rl.run_state import require_equal, weight_fingerprint +from sequifier.trajectory.sources import file_checksum + + +class PolicyReferences: + def __init__(self, config, recovery, *, resuming): + self.config, self.recovery = config, recovery + self.artifacts, self.checksums = {}, {} + if ( + resuming + and config.initialize_from + and Path(config.initialize_from).is_file() + ): + if config.actor_initialization == "history_only": + require_equal( + recovery.initialization["transfer"]["source_sha256"], + file_checksum(config.initialize_from), + "resume.initialize_from.sha256", + ) + else: + _, identity = self._artifact(config.initialize_from) + require_equal( + recovery.initialization["weight_hash"], + identity, + "resume.initialize_from.weights", + ) + for index, phase in enumerate(config.phases): + reference = phase.reference_config + if reference is None or reference.source != "artifact": + continue + if Path(reference.path).is_file(): + _, identity = self._artifact(reference.path) + expected = recovery.reference_contract.setdefault( + str(config.phase_offset + index), identity + ) + require_equal( + expected, identity, f"phases.{phase.name}.reference.weights" + ) + elif index > recovery.phase_index or ( + index == recovery.phase_index and recovery.phase_state is None + ): + raise FileNotFoundError( + f"reference artifact required before phase start: {reference.path}" + ) + # Completed references and initialization are no longer needed in memory. + pending = { + reference.path + for phase in config.phases[recovery.phase_index :] + if (reference := phase.reference_config) is not None + and reference.source == "artifact" + } + self.artifacts = { + path: value for path, value in self.artifacts.items() if path in pending + } + + def _artifact(self, path): + checksum = file_checksum(path) + if path not in self.artifacts or self.checksums[path] != checksum: + source = load_policy( + path, + device="cpu", + interface_name=self.config.policy_interface, + binding=self.config.policy, + model=self.config.execution.model, + ) + self.recovery.representation.validate_actor( + source, self.config.units, f"reference.{path}" + ) + weights = source.network.state_dict() + self.artifacts[path] = weights, weight_fingerprint(weights) + self.checksums[path] = checksum + return self.artifacts[path] + + def resolve(self, actor, index): + recovery = self.recovery + declaration = self.config.phases[recovery.phase_index].reference_config + if declaration is None: + return None, None + if recovery.phase_state is not None: + weights, identity = ( + recovery.phase_state["reference"], + recovery.phase_state["reference_identity"], + ) + elif declaration.source == "artifact": + weights, identity = self._artifact(declaration.path) + require_equal( + recovery.reference_contract[str(index)], + identity, + "reference.artifact.weights", + ) + else: + weights = recovery.incoming_actor + identity = weight_fingerprint(weights) + reference = deepcopy(actor).eval().requires_grad_(False) + reference.network.load_state_dict(weights, strict=True) + return reference, identity + + +def validate_reference_state(phase, recovery, index): + state = recovery.phase_state + if state is not None: + reference = phase.reference_config + if (reference is None) != (state["reference"] is None): + raise ValueError("checkpoint reference disagrees with the phase") + if reference is not None: + identity = weight_fingerprint(state["reference"]) + require_equal( + identity, state["reference_identity"], "resume.embedded_reference" + ) + expected = ( + weight_fingerprint(recovery.incoming_actor) + if reference.source == "phase_start" + else recovery.reference_contract[str(index)] + ) + require_equal(expected, identity, "resume.reference.weights") diff --git a/src/sequifier/rl/registry.py b/src/sequifier/rl/registry.py new file mode 100644 index 000000000..8ccb0e47e --- /dev/null +++ b/src/sequifier/rl/registry.py @@ -0,0 +1,28 @@ +"""Dispatch built-in phases to their runtimes and data preparation functions.""" + +from dataclasses import dataclass +from typing import Callable + +from sequifier.io.preference_dataset import prepare_preferences +from sequifier.io.prepared_policy_data import PreparedPolicyData +from sequifier.rl.dpo import run_dpo_phase +from sequifier.rl.grpo import run_grpo_phase +from sequifier.rl.iql import run_iql_phase +from sequifier.rl.offline_data import prepare_iql, prepare_offline +from sequifier.rl.ppo import run_ppo_phase +from sequifier.rl.training import run_imitation_phase + + +@dataclass(frozen=True) +class BuiltinAlgorithm: + runtime: Callable + prepare_data: Callable[..., PreparedPolicyData] | None = None + + +BUILTIN_ALGORITHMS = { + "policy_imitation": BuiltinAlgorithm(run_imitation_phase, prepare_offline), + "iql": BuiltinAlgorithm(run_iql_phase, prepare_iql), + "ppo": BuiltinAlgorithm(run_ppo_phase), + "dpo": BuiltinAlgorithm(run_dpo_phase, prepare_preferences), + "grpo": BuiltinAlgorithm(run_grpo_phase), +} diff --git a/src/sequifier/rl/run_checkpoint.py b/src/sequifier/rl/run_checkpoint.py new file mode 100644 index 000000000..eea3eb055 --- /dev/null +++ b/src/sequifier/rl/run_checkpoint.py @@ -0,0 +1,346 @@ +"""Policy recovery with artifact-backed, authoritative committed phase boundaries.""" + +import math +from dataclasses import dataclass, field, fields +from pathlib import Path + +import torch + +from sequifier.artifacts.checkpoint_envelope import ( + checkpoint_envelope, + phase_boundary, + recovery_payload, + require_phase_boundary, +) +from sequifier.artifacts.model_artifact import ( + ModelArtifact, + ModelArtifactMetadata, + ModelExecutionConfig, +) +from sequifier.artifacts.phase import PhaseArtifacts, PhaseCommit +from sequifier.contract_versions import ContractVersions, require_contract_version +from sequifier.io.atomic import atomic_save +from sequifier.rl.checkpoint import phase_actor_path, policy_model_artifact +from sequifier.rl.references import validate_reference_state +from sequifier.rl.run_state import ( + ActorRepresentation, + OptimizerContinuation, + PhaseExecution, + PolicyPhaseResult, + require_equal, + weight_fingerprint, +) +from sequifier.runtime.plan import new_run_id +from sequifier.runtime.random_state import capture_rng_state + + +@dataclass(frozen=True) +class CompletedPhase: + commit: PhaseCommit + + @property + def result(self): + values = self.commit.result + return PolicyPhaseResult(**{**values, "metrics": tuple(values["metrics"])}) + + @classmethod + def from_dict(cls, values): + return cls(PhaseCommit.load(values)) + + def as_dict(self): + return self.commit.as_dict() + + def report_result(self): + values = self.commit.report_result() + return PolicyPhaseResult(**{**values, "metrics": tuple(values["metrics"])}) + + +@dataclass +class RunCheckpoint: + plan: dict + representation: ActorRepresentation + data_identity: dict + data_contract: list + adapter_contract: dict + initialization: dict + incoming_actor: dict + random: dict + reference_contract: dict = field(default_factory=dict) + completed: list[CompletedPhase] = field(default_factory=list) + phase_state: dict | None = None + continuation: OptimizerContinuation | None = None + stopped: bool = False + run_id: str = field(default_factory=new_run_id) + + @property + def phase_index(self): + return len(self.completed) + + @property + def next_phase_index(self): + return self.plan.get("phase_offset", 0) + self.phase_index + + @classmethod + def load(cls, path, config, plan, data_identity=None, *, run_id=None): + path = Path(path) + if config.resume.policy == "never": + return None + if not path.is_file(): + if config.resume.policy == "required": + raise FileNotFoundError(path) + return None + envelope = torch.load(path, map_location="cpu", weights_only=False) + payload = recovery_payload(envelope, "policy") + require_contract_version( + "policy recovery", + payload.get("format_version"), + ContractVersions.POLICY_RECOVERY, + ) + if payload.get("artifact_type") != "sequifier_policy_run": + raise ValueError("Expected a policy checkpoint recovery payload") + if run_id is not None and payload.get("run_id") != run_id: + if config.resume.policy == "required": + raise ValueError("Policy recovery belongs to another run") + return None + try: + saved_plan = payload["plan"] + require_equal(plan, saved_plan, "resume.plan") + completed = [ + CompletedPhase.from_dict(record) for record in payload["completed"] + ] + checkpoint = cls( + **{ + f.name: payload[f.name] + for f in fields(cls) + if f.name not in {"representation", "completed"} + }, + representation=ActorRepresentation(payload["representation"]), + completed=completed, + ) + require_equal( + checkpoint.representation.identity, + payload["representation_fingerprint"], + "resume.representation_fingerprint", + ) + if data_identity is not None: + checkpoint.validate_dependencies(config, data_identity) + checkpoint.validate(config) + require_phase_boundary(envelope, checkpoint.phase_boundary()) + return checkpoint + except (KeyError, TypeError, AttributeError, IndexError) as exc: + raise ValueError(f"malformed policy run checkpoint: {exc}") from exc + + def validate_dependencies(self, config, data_identity): + if config.actor_initialization == "history_only": + path = str((Path(config.project_root) / config.initialize_from).resolve()) + require_equal( + self.initialization["transfer"]["source_sha256"], + data_identity[path], + "resume.initialize_from.sha256", + ) + require_equal(data_identity, self.data_identity, "resume.data") + + def validate(self, config): + if self.phase_index > len(config.phases): + raise ValueError("checkpoint completed handoffs disagree with the plan") + validate_finite_state(self.incoming_actor, "incoming_actor") + # Parsing the owned representation also validates the persisted policy schema. + self.representation.policy + for position, record in enumerate(self.completed): + result = record.result + phase = config.phases[position] + index = config.phase_offset + position + require_equal( + (index, phase.name, phase.algorithm, phase.handoff), + (result.index, result.name, result.algorithm, result.handoff), + "resume.completed_phase", + ) + path = phase_actor_path(config.phase_directory, index, phase.name) + require_equal( + str(path), result.provenance["actor_path"], "resume.actor_path" + ) + require_equal( + str(path.with_name("report.json")), + result.provenance["report_path"], + "resume.report_path", + ) + if sidecar := record.commit.sidecars.get("telemetry"): + require_equal( + str(PhaseArtifacts(path).telemetry), + sidecar["path"], + "resume.telemetry.path", + ) + validate_finite_state(record.commit.result, "completed_phase") + incoming_identity = weight_fingerprint(self.incoming_actor) + require_equal( + self.completed[-1].result.selected_weight_hash + if self.completed + else self.initialization["weight_hash"], + incoming_identity, + "resume.incoming_actor", + ) + state = self.phase_state + if state is None: + return + if self.phase_index == len(config.phases): + raise ValueError("completed run cannot have a live phase state") + require_contract_version( + "policy phase state", + state.get("version"), + ContractVersions.POLICY_PHASE_STATE, + ) + artifacts = PhaseArtifacts.for_phase( + config.phase_directory, + self.next_phase_index, + config.phases[self.phase_index].name, + ) + require_equal( + str(artifacts.telemetry), state["telemetry_path"], "resume.telemetry.path" + ) + if not isinstance(state["telemetry_position"], str): + raise ValueError("phase checkpoint requires a committed telemetry position") + transaction = state["transaction"] + if ( + not isinstance(transaction, int) or isinstance(transaction, bool) + ) or transaction < 1: + raise ValueError( + "phase checkpoint must follow a completed optimizer transaction" + ) + selection = state["selection"] + best = selection["best_transaction"] + if best is not None and ( + (not isinstance(best, int) or isinstance(best, bool)) + or not 0 <= best <= transaction + ): + raise ValueError( + "checkpoint selected transaction is outside committed progress" + ) + if (selection["best_metric"] is None) != ( + selection["best_weights"] is None + ) or (best is None) != (selection["best_weights"] is None): + raise ValueError( + "checkpoint best actor, metric and transaction must be recorded together" + ) + validate_reference_state( + config.phases[self.phase_index], + self, + self.next_phase_index, + ) + validate_finite_state(state, "phase") + + def validate_actor(self, actor, units): + self.representation.validate_actor(actor, units, "resume.representation") + expected = actor.network.state_dict() + weights = [self.incoming_actor] + if self.phase_state is not None: + state = self.phase_state + weights += [ + state["networks"]["actor"], + state["reference"], + state["selection"]["best_weights"], + ] + for snapshot in weights: + if snapshot is None: + continue + if snapshot.keys() != expected.keys() or any( + snapshot[k].shape != v.shape or snapshot[k].dtype != v.dtype + for k, v in expected.items() + ): + raise ValueError( + "checkpoint actor tensors disagree with the resolved model" + ) + actor.network.load_state_dict(self.incoming_actor, strict=True) + + def phase_boundary(self): + state = self.phase_state + active = None + if state is not None: + active = { + "index": self.next_phase_index, + "transaction": state["transaction"], + "sidecars": { + "telemetry": { + "path": state["telemetry_path"], + "position": state["telemetry_position"], + } + }, + } + return phase_boundary( + self.run_id, + selected=self.completed[-1].commit.model_identity + if self.completed + else None, + active=active, + ) + + def save(self, path, device, state=None): + """Atomically commit the run and its algorithm-owned recovery state.""" + self.random = capture_rng_state(device) + self.phase_state = state + model = ModelArtifact( + ContractVersions.MODEL_ARTIFACT, + ModelExecutionConfig(self.representation.values["execution"]), + state["networks"]["actor"] if state else self.incoming_actor, + ModelArtifactMetadata(), + ) + atomic_save( + path, + checkpoint_envelope( + "policy", + { + "artifact_type": "sequifier_policy_run", + "format_version": ContractVersions.POLICY_RECOVERY, + **{ + f.name: vars(self)[f.name] + for f in fields(self) + if f.name not in {"representation", "completed"} + }, + "representation": self.representation.canonical, + "representation_fingerprint": self.representation.identity, + "completed": [record.as_dict() for record in self.completed], + }, + model, + boundary=self.phase_boundary(), + ), + ) + + def finish_phase(self, execution: PhaseExecution, telemetry_position): + actor, result = execution.actor, execution.result + require_equal(self.next_phase_index, result.index, "handoff.phase_index") + require_equal( + result.selected_weight_hash, + weight_fingerprint(execution.weights), + "handoff.selected_actor", + ) + artifacts = PhaseArtifacts(Path(result.provenance["actor_path"])) + result_data = result.as_dict() + commit = PhaseCommit.stage( + artifacts, + result_data, + model=policy_model_artifact( + actor, metadata=result_data, weights=execution.weights + ), + telemetry={"path": str(artifacts.telemetry), "position": telemetry_position} + if telemetry_position is not None + else None, + ) + self.completed.append(CompletedPhase(commit)) + self.stopped = result.stop_scope == "end_plan" + self.incoming_actor = execution.weights + self.continuation = execution.continuation + self.phase_state = None + + +def validate_finite_state(value, field): + """Check algorithm-owned optimizer/network trees before restoring them.""" + if isinstance(value, torch.Tensor): + if not torch.isfinite(value).all(): + raise ValueError(f"non-finite checkpoint tensor: {field}") + elif isinstance(value, dict): + for key, item in value.items(): + validate_finite_state(item, f"{field}.{key}") + elif isinstance(value, (tuple, list)): + for index, item in enumerate(value): + validate_finite_state(item, f"{field}[{index}]") + elif isinstance(value, float) and not math.isfinite(value): + raise ValueError(f"non-finite checkpoint value: {field}") diff --git a/src/sequifier/rl/run_config.py b/src/sequifier/rl/run_config.py new file mode 100644 index 000000000..9fba87dd8 --- /dev/null +++ b/src/sequifier/rl/run_config.py @@ -0,0 +1,46 @@ +"""Resolved inputs for the native policy runner; never an authored config.""" + +from typing import Literal + +from pydantic import BaseModel, ConfigDict, Field + +from sequifier.config.components import ResumeConfig +from sequifier.config.normalization_config import FrozenNormalizationConfig +from sequifier.config.policy_config import ResolvedPolicyBinding +from sequifier.config.policy_plan import PolicyPhase +from sequifier.config.policy_train_config import SequenceSourceConfig +from sequifier.model.execution import ModelExecution + + +class ResolvedPolicyRun(BaseModel): + model_config = ConfigDict(extra="forbid", arbitrary_types_allowed=True) + project_root: str + model_name: str + device: str + seed: int + phase_offset: int = Field(default=0, ge=0, exclude=True) + enforce_determinism: bool = False + policy: ResolvedPolicyBinding + execution: ModelExecution = Field(exclude=True) + policy_interface: str + initialize_from: str | None = None + metadata_paths: list[str] = Field(default_factory=list) + actor_initialization: Literal["attach", "history_only"] = "attach" + normalization: FrozenNormalizationConfig | None = None + sources: dict[str, SequenceSourceConfig] + preferences: dict[str, str] + units: dict[str, str] + phases: list[PolicyPhase] + latest_checkpoint_interval_minutes: float | None = None + log_interval_iterations: int = Field(default=10, gt=0) + source_validation_fractions: dict[str, float] = Field(default_factory=dict) + validation_fraction: float + episode_cache_size: int + resume: ResumeConfig + embedding_layer_names: list[str] = Field( + default_factory=lambda: ["backbone.final_norm"] + ) + model_path: str + report_path: str + checkpoint_path: str + phase_directory: str diff --git a/src/sequifier/rl/run_state.py b/src/sequifier/rl/run_state.py new file mode 100644 index 000000000..d5c6e86b0 --- /dev/null +++ b/src/sequifier/rl/run_state.py @@ -0,0 +1,610 @@ +"""Shared policy transactions, representation checks and phase result contracts.""" + +import hashlib +import json +import math +import time +from dataclasses import asdict, dataclass, replace +from typing import TYPE_CHECKING, Callable, Generic, TypedDict, TypeVar + +import torch + +from sequifier.config.policy_config import ResolvedPolicyBinding +from sequifier.config.policy_plan import PhaseConfig +from sequifier.contract_versions import ContractVersions +from sequifier.rl.phase_optimization import PhaseOptimization, PhaseRoles +from sequifier.rl.policy import Policy +from sequifier.rl.telemetry import TelemetryJournal +from sequifier.runtime.plan import metric_improved, select_handoff, stop_scope +from sequifier.runtime.random_state import capture_rng_state, restore_rng_state +from sequifier.runtime.stopping import StoppingState + +if TYPE_CHECKING: + from sequifier.rl.metrics import PolicyMetrics + from sequifier.rl.run_config import ResolvedPolicyRun + +PhaseType = TypeVar("PhaseType", bound=PhaseConfig) +DataType = TypeVar("DataType") + + +def clone_weights(module): + return { + name: tensor.detach().cpu().clone() + for name, tensor in module.state_dict().items() + } + + +def weight_fingerprint(weights): + digest = hashlib.sha256() + for name, tensor in sorted(weights.items()): + value = tensor.detach().cpu().contiguous() + digest.update(name.encode()) + digest.update(str((value.dtype, tuple(value.shape))).encode()) + digest.update(value.reshape(-1).view(torch.uint8).numpy().tobytes()) + return digest.hexdigest() + + +def require_equal(expected, actual, field): + """Locate the first mismatching contract field rather than a generic failure.""" + if isinstance(expected, dict) and isinstance(actual, dict): + if expected.keys() != actual.keys(): + raise ValueError( + f"{field}: expected fields {sorted(expected)}, received {sorted(actual)}" + ) + for key in expected: + require_equal(expected[key], actual[key], f"{field}.{key}") + elif expected != actual: + raise ValueError(f"{field}: expected {expected!r}, received {actual!r}") + + +@dataclass(frozen=True) +class ActorRepresentation: + # Canonical JSON provides an immutable owned value, even for nested schemas. + canonical: str + + @classmethod + def from_actor(cls, actor, units): + from pydantic_core import to_jsonable_python + + from sequifier.artifacts.model_export import model_execution_config + + try: + units = actor.units + except AttributeError: + pass + model = actor.execution_config.model.model_dump(mode="json") + model["backbone"]["repository"] = None + return cls( + json.dumps( + { + "execution": to_jsonable_python( + model_execution_config(actor.execution_config) + ), + "policy": actor.config.model_dump(mode="json"), + "model": model, + "statistics": actor.interface.selected_columns_statistics, + "units": dict(units), + }, + sort_keys=True, + allow_nan=False, + ) + ) + + @property + def values(self): + return json.loads(self.canonical) + + @property + def policy(self): + return ResolvedPolicyBinding.model_validate(self.values["policy"]) + + @property + def identity(self): + return hashlib.sha256(self.canonical.encode()).hexdigest() + + def validate_actor(self, actor, units, field="actor"): + actual = type(self).from_actor(actor, units).values + if "execution" not in self.values: + actual.pop("execution") + require_equal(self.values, actual, field) + + +@dataclass(frozen=True) +class PolicyPhaseResult: + index: int + name: str + algorithm: str + monitor: str + direction: str + handoff: str + selected_metric: float | None + selected_transaction: int + selected_weight_hash: str + clocks: dict + metrics: tuple[dict, ...] + provenance: dict + stop_scope: str | None = None + + def as_dict(self): + from dataclasses import asdict + + return asdict(self) + + +@dataclass(frozen=True) +class PolicyRunResult: + policy: Policy + phases: tuple[PolicyPhaseResult, ...] + representation: ActorRepresentation + model_path: str + report_path: str + checkpoint_path: str + + +class OptimizerContinuation(TypedDict): + algorithm: str + networks: dict[str, dict[str, torch.Tensor]] + optimization: dict[str, dict] + trainable_parameters: dict[str, list[str]] + + +@dataclass(frozen=True) +class PhaseExecution: + actor: Policy + weights: dict[str, torch.Tensor] + result: PolicyPhaseResult + continuation: OptimizerContinuation | None + + +@dataclass(frozen=True) +class PolicyPhaseIdentity: + index: int + run_seed: int + sampling_seed: int + + @classmethod + def resolve(cls, index: int, run_seed: int): + return cls(index, run_seed, run_seed + index) + + def grpo_seed(self, restart): + from sequifier.rl.collection import stream_seed + + return stream_seed( + self.run_seed, f"grpo-phase-{self.index}-restart-{restart}", 0 + ) + + +@dataclass(frozen=True) +class PhaseContext(Generic[PhaseType, DataType]): + run: "ResolvedPolicyRun" + identity: PolicyPhaseIdentity + phase: PhaseType + actor: Policy + data: DataType + reference: Policy | None + reference_identity: str | None + resume_state: dict | None + commit: Callable[[dict], None] + optimization: PhaseOptimization + telemetry: TelemetryJournal + metrics: "PolicyMetrics | None" = None + continuation: OptimizerContinuation | None = None + preserve_optimizer: bool = False + + @property + def index(self): + return self.identity.index + + +class Selection: + """Strict comparisons retain the earliest finite best on ties.""" + + def __init__(self, phase, monitor, direction, state=None): + self.phase, self.monitor, self.direction = phase, monitor, direction + self.best_metric, self.best_weights, self.best_transaction = None, None, None + self.last_metric = None + if state: + self.best_metric = state["best_metric"] + self.best_weights = state["best_weights"] + self.best_transaction = state["best_transaction"] + self.last_metric = state["last_metric"] + + def observe(self, actor, metric, transaction): + if metric is not None and not math.isfinite(metric): + raise ValueError(f"non-finite selection metric {self.monitor}") + self.last_metric = metric + if metric is None: + return + if metric_improved(metric, self.best_metric, self.direction): + self.best_metric = float(metric) + self.best_weights = clone_weights(actor.network) + self.best_transaction = transaction + + def state_dict(self): + return { + "best_metric": self.best_metric, + "best_weights": self.best_weights, + "best_transaction": self.best_transaction, + "last_metric": self.last_metric, + } + + def finish(self, context, transaction: int, clocks, provenance): + actor = context.actor.eval() + selected, metric = select_handoff( + self.phase.handoff, + (transaction, self.last_metric), + (self.best_transaction, self.best_metric), + ) + if selected is None: + raise ValueError(f"{self.phase.name}: no selected transaction") + if self.phase.handoff == "best": + if self.best_weights is None: + raise ValueError( + f"phases.{self.phase.name}: handoff best requires a finite {self.monitor}" + ) + weights = self.best_weights + actor.network.load_state_dict(weights, strict=True) + else: + weights = clone_weights(actor.network) + result = PolicyPhaseResult( + context.index, + self.phase.name, + self.phase.algorithm, + self.monitor, + self.direction, + self.phase.handoff, + metric, + selected, + weight_fingerprint(weights), + clocks, + (), + provenance, + ) + return actor, weights, result + + +class PhaseSession: + """Own common progress and optimization; algorithms own their update loops.""" + + def __init__(self, context): + self.context = context + roles = PhaseRoles.build( + context.actor, context.phase, restoring=context.resume_state is not None + ) + self.modules = roles.trainable + self.networks = {**self.modules, **roles.frozen} + self.mask_ids = context.optimization.masks + self.configs = context.optimization.configs + self.optimization = context.optimization.build(roles) + if context.phase.optimizer_state == "continue" and context.resume_state is None: + continuation = context.continuation + if continuation is None: + raise ValueError( + "continued phase requires the preceding optimizer state" + ) + require_equal( + context.phase.algorithm, + continuation["algorithm"], + "optimization.algorithm", + ) + self._restore_networks_and_optimizers(continuation, "optimization") + self.steps = dict.fromkeys(self.modules, 0) + self.sampler = torch.Generator().manual_seed(context.identity.sampling_seed) + self.state = context.resume_state or {} + self.transaction = self.state.get("transaction", 0) + self.iteration_budget = context.phase.iteration_budget( + context.optimization.step_budget + ) + self.stopping = StoppingState(**self.state.get("stopping", {})) + self.initial_evaluated = self.state.get("initial_evaluated", False) + self._count_stopping = True + self.last_checkpoint_time = time.monotonic() + self.last_snapshot_time = self.last_checkpoint_time + self.telemetry = context.telemetry + selection = context.phase.evaluation.monitor + monitor = selection.metric + self.selection = Selection( + context.phase, monitor, selection.direction, self.state.get("selection") + ) + if self.state: + self._restore_networks_and_optimizers(self.state, "phase") + for role, saved in self.state["optimization"].items(): + steps = saved["steps"] + budget = context.optimization.step_budget + if ( + (not isinstance(steps, int)) + or steps < self.transaction + or (budget is not None and steps > budget) + ): + raise ValueError( + f"{role} optimizer clock exceeds committed progress/budget" + ) + self.steps[role] = steps + self.sampler.set_state(self.state["sampler"]) + self.restore_random() + + def _restore_networks_and_optimizers(self, state, field): + require_equal( + sorted(self.networks), sorted(state["networks"]), f"{field}.networks" + ) + require_equal( + sorted(self.mask_ids), + sorted(state["trainable_parameters"]), + f"{field}.trainability roles", + ) + require_equal( + sorted(self.optimization), + sorted(state["optimization"]), + f"{field}.optimization", + ) + for role, module in self.networks.items(): + module.load_state_dict(state["networks"][role], strict=True) + if role in self.optimization: + require_equal( + self.mask_ids[role], + state["trainable_parameters"][role], + f"{field}.trainability.{role}", + ) + optimizer, scheduler = self.optimization[role] + saved = state["optimization"][role] + optimizer.load_state_dict(saved["optimizer"]) + scheduler.load_state_dict(saved["scheduler"]) + + def restore_random(self): + if self.state: + restore_rng_state(self.context.actor.device, self.state["random"]) + + def zero_grad(self): + for optimizer, _ in self.optimization.values(): + optimizer.zero_grad(set_to_none=True) + + def step(self): + # Check every gradient before mutating any participating network. + norms = { + role: float( + torch.nn.utils.clip_grad_norm_( + module.parameters(), + self.configs[role].gradient_clip_norm or float("inf"), + error_if_nonfinite=True, + ) + ) + for role, module in self.modules.items() + } + for role, (optimizer, scheduler) in self.optimization.items(): + optimizer.step() + if any( + not torch.isfinite(p).all() for p in self.modules[role].parameters() + ): + raise ValueError( + f"{role} optimizer produced non-finite parameters; transaction was not committed" + ) + scheduler.step() + self.steps[role] += 1 + self.zero_grad() + return norms + + def snapshot_due(self): + minutes = self.context.phase.checkpoints.snapshot_interval_minutes + return ( + minutes is not None + and time.monotonic() - self.last_snapshot_time >= minutes * 60 + ) + + def should_evaluate_initial(self): + due = self.context.phase.evaluation.before_phase and not self.initial_evaluated + if due: + self.initial_evaluated = True + self._count_stopping = False + return due + + def should_evaluate(self, final): + scheduled = ( + final + or self.transaction % self.context.phase.evaluation_interval_iterations == 0 + ) + self._count_stopping = scheduled + return scheduled or ( + self.snapshot_due() and self.context.phase.evaluation.on_snapshot + ) + + def observe(self, metric, **record): + if self.context.metrics is not None: + values = {**record, self.selection.monitor: metric} + self.context.metrics.record( + self, + values, + validation=True, + primary=self.selection.monitor, + initial=self.transaction == 0, + ) + self.selection.observe(self.context.actor, metric, self.transaction) + if self._count_stopping: + self.stopping.observe( + self.context.phase.stopping, + metric, + self.selection.direction, + self.transaction, + ) + + def record_evaluation(self, metric, **record): + self.observe(metric, **record) + self.telemetry.append("metrics", {"transaction": self.transaction, **record}) + + def record_training(self, *, final=False, **record): + self.telemetry.append("metrics", {"transaction": self.transaction, **record}) + if self.context.metrics is not None: + self.context.metrics.record(self, record, final=final) + + def evaluate_validation(self, evaluator, **record): + values = evaluator() + key = self.selection.monitor.removeprefix("validation_") + self.record_evaluation( + values[key], + **record, + **{f"validation_{name}": value for name, value in values.items()}, + ) + + def complete_update( + self, + *, + final, + training, + evaluate, + checkpoint=None, + evaluation_record: dict | Callable[[], dict] | None = None, + ): + """Commit an offline update after all participating optimizers have stepped.""" + self.transaction += 1 + self.record_training(final=final, **training) + if self.should_evaluate(final): + extra = ( + evaluation_record() + if callable(evaluation_record) + else evaluation_record + ) + evaluate(**training, **(extra or {})) + self.checkpoint(final, **(checkpoint or {})) + + def optimizer_clocks(self): + return {f"{role}_optimizer_steps": steps for role, steps in self.steps.items()} + + def record_continuation(self, **record): + self.telemetry.append( + "continuations", {"transaction": self.transaction, **record} + ) + + def evaluate_online(self, adapter, *, initial=False, **progress): + from sequifier.rl.ppo_evaluation import evaluate_policy + + phase = self.context.phase + result = evaluate_policy( + self.context.actor, phase.evaluation, forbidden_adapter=adapter + ) + if ( + phase.algorithm == "grpo" + and phase.evaluation.monitor.metric == "score_mean" + ): + score = float( + torch.tensor( + result["score_mean"] * phase.reward_scale, dtype=torch.float32 + ) + ) + if not math.isfinite(score): + raise ValueError("GRPO evaluation score exceeds finite FP32 range") + result["score_mean"] = score + result["selection_value"] = score + self.observe(result["selection_value"], **result) + if initial and phase.algorithm == "ppo": + progress["initial"] = True + self.telemetry.append( + "evaluations", {**result, "transaction": self.transaction, **progress} + ) + + def optimization_state(self, *, include_steps=False): + return { + role: { + "optimizer": optimizer.state_dict(), + "scheduler": scheduler.state_dict(), + **({"steps": self.steps[role]} if include_steps else {}), + } + for role, (optimizer, scheduler) in self.optimization.items() + } + + def should_checkpoint(self, final): + return ( + final + or self.stopping.triggered + or self.snapshot_due() + or ( + self.context.run.latest_checkpoint_interval_minutes is not None + and time.monotonic() - self.last_checkpoint_time + >= self.context.run.latest_checkpoint_interval_minutes * 60 + ) + or self.transaction % self.context.phase.checkpoint_interval_iterations == 0 + ) + + def checkpoint(self, final, **extra): + if not self.should_checkpoint(final): + return + self.context.commit( + { + **extra, + "version": ContractVersions.POLICY_PHASE_STATE, + "networks": { + name: clone_weights(module) + for name, module in self.networks.items() + }, + "optimization": self.optimization_state(include_steps=True), + "sampler": self.sampler.get_state(), + "selection": self.selection.state_dict(), + "stopping": asdict(self.stopping), + "initial_evaluated": self.initial_evaluated, + "trainable_parameters": self.mask_ids, + "transaction": self.transaction, + "telemetry_position": self.telemetry.commit(), + "reference": clone_weights(self.context.reference.network) + if self.context.reference is not None + else None, + "reference_identity": self.context.reference_identity, + "random": capture_rng_state(self.context.actor.device), + } + ) + + retained = ( + final + or self.stopping.triggered + or self.snapshot_due() + or self.transaction % self.context.phase.checkpoint_interval_iterations == 0 + ) + if retained: + from pathlib import Path + from shutil import copyfile + + from sequifier.io.atomic import atomic_path + + destination = ( + Path(self.context.run.phase_directory) + / f"{self.context.index:03d}-{self.context.phase.name}" + / "checkpoints" + / f"transaction-{self.transaction:08d}.pt" + ) + with atomic_path(destination, durable=True) as temporary: + copyfile(self.context.run.checkpoint_path, temporary) + self.last_checkpoint_time = time.monotonic() + if self.snapshot_due(): + self.last_snapshot_time = self.last_checkpoint_time + + def finish(self, clocks, provenance): + if self.context.metrics is not None: + self.context.metrics.flush(self) + actor, weights, result = self.selection.finish( + self.context, + self.transaction, + clocks, + { + **provenance, + "trainable_parameters": self.mask_ids, + "resolved_seeds": { + "run": self.context.identity.run_seed, + "phase_sampling": self.context.identity.sampling_seed, + }, + }, + ) + scope = stop_scope(self.context.phase.stopping, self.stopping.triggered) + continuation: OptimizerContinuation | None = None + if self.context.preserve_optimizer and scope != "end_plan": + continuation = { + "algorithm": self.context.phase.algorithm, + "networks": { + name: weights if name == "actor" else clone_weights(module) + for name, module in self.networks.items() + }, + "optimization": self.optimization_state(), + "trainable_parameters": self.mask_ids, + } + return PhaseExecution( + actor, + weights, + replace(result, stop_scope=scope), + continuation, + ) diff --git a/src/sequifier/rl/runtime.py b/src/sequifier/rl/runtime.py new file mode 100644 index 000000000..ed032d05c --- /dev/null +++ b/src/sequifier/rl/runtime.py @@ -0,0 +1,159 @@ +"""Policy rollout through the same environment and causal history used in training.""" + +from copy import deepcopy +from typing import Literal +from uuid import uuid4 + +from sequifier.config.policy_infer_config import PolicyRolloutConfig +from sequifier.config.rollout_config import InputBinding +from sequifier.io.actions import action_component_key, action_latent_key +from sequifier.io.values import numeric_scalar, portable_fields +from sequifier.rl.collection import PolicyCursor +from sequifier.rl.environment import ( + OnlineEnvironmentAdapter, + OnlineResetResult, + require_seeded_reset, + validate_online_adapter, + validate_reset, +) +from sequifier.rl.policy import Policy +from sequifier.trajectory.recording import TrajectoryRecorder, close_adapter +from sequifier.trajectory.schema import SeedHistory, Trajectory + + +def collect_policy_rollout( + policy: Policy, + adapter: OnlineEnvironmentAdapter, + seed: SeedHistory, + config: PolicyRolloutConfig, + max_steps: int, + *, + desired_return: float | None = None, + action_selection: Literal["deterministic", "stochastic"] = "deterministic", + random_seed: int = 1010, +) -> Trajectory: + """Execute a supplied initial observation, preserving actual episode boundaries.""" + was_training, failed = policy.training, False + try: + validate_online_adapter(adapter) + require_seeded_reset(adapter) + if (not isinstance(max_steps, int)) or max_steps < 1: + raise ValueError("policy collection budget must be a positive integer") + if action_selection not in {"deterministic", "stochastic"}: + raise ValueError("action_selection must be deterministic or stochastic") + spec, units = policy.config, config.environment.units + if units != policy.units: + raise ValueError("rollout environment units differ from the policy") + seed = SeedHistory.model_validate(seed.model_dump()) + if ( + sum(seed.attention_valid_mask) != 1 + or len(seed.attention_valid_mask) != spec.context_length + ): + raise ValueError( + "policy seed requires one current observation in its context window" + ) + if not spec.return_conditioned and desired_return is not None: + raise ValueError("behavior cloning has no desired-return input") + if spec.return_conditioned and desired_return is None: + desired_return = spec.initial_desired_return + returns = ( + [numeric_scalar(desired_return, "desired_return")] + if spec.return_conditioned + else [] + ) + if ( + spec.return_conditioned + and spec.returns.finite_horizon is not None + and config.max_episode_steps != spec.returns.finite_horizon + ): + raise ValueError( + "policy max_episode_steps must equal its finite return horizon" + ) + initial = OnlineResetResult( + {column: values[-1] for column, values in seed.observations.items()}, + seed.static_inputs, + units, + ) + episode_id = str(uuid4()) + policy.eval() + cursor = PolicyCursor( + policy, adapter, units, random_seed, strict_vocabulary=False + ) + cursor.reset(initial, episode_id, returns[0] if returns else None) + reset = validate_reset( + adapter.reset_from_observation(deepcopy(initial), random_seed), units + ) + if dict(reset.observation) != dict(initial.observation) or dict( + reset.static_inputs + ) != dict(initial.static_inputs): + raise ValueError( + "seeded reset must preserve the supplied observation and static inputs" + ) + recorder = TrajectoryRecorder(episode_id, config.max_episode_steps) + previous_actions, log_probabilities = [], [] + observation = dict(initial.observation) + for _ in range(max_steps): + current = cursor.history.current + decision = cursor.decide(deterministic=action_selection == "deterministic") + result = cursor.step(decision) + + previous_actions.append(current.previous_action) + log_probabilities.append(decision.log_probability) + if spec.return_conditioned: + assert cursor.desired_return is not None + returns.append(cursor.desired_return) + successor = ( + portable_fields(result.observation) + if result.observation is not None + else None + ) + action_metadata = ( + decision.action if isinstance(decision.action, dict) else {} + ) + transition = recorder.append( + observation=observation, + predictions=decision.domain_action, + action_latents={ + name: action_metadata[action_latent_key(name, spec)] + for name, field in spec.actions.items() + if field.type == "real" + }, + action_components={ + name: action_metadata[action_component_key(name, spec)] + for name, field in spec.actions.items() + if field.type == "real" + }, + external_inputs=successor or {}, + reward=result.reward, + successor_observation=successor, + terminated=result.terminated, + truncated=result.truncated, + ) + if transition.terminated or transition.truncated: + break + observation = successor + return recorder.finish( + format="sequifier.policy-trajectory", + seed=seed, + bindings=[ + InputBinding(input=name, source="environment", key=name) + for name in spec.observation_columns + ], + prediction_columns=spec.action_columns, + metadata={ + "policy_execution": { + "config": spec.model_dump(mode="json"), + "initial_desired_return": desired_return, + "remaining_returns": returns, + "previous_action_ids": previous_actions, + "action_log_probabilities": log_probabilities, + "action_selection": action_selection, + }, + }, + ) + except BaseException: + failed = True + raise + finally: + policy.train(was_training) + close_adapter(adapter, failed) diff --git a/src/sequifier/rl/scoring.py b/src/sequifier/rl/scoring.py new file mode 100644 index 000000000..7f800b55d --- /dev/null +++ b/src/sequifier/rl/scoring.py @@ -0,0 +1,40 @@ +"""Bounded differentiable scores for causal action sequences.""" + +import torch + +from sequifier.io.actions import action_batch +from sequifier.io.policy_window import PolicyWindowView +from sequifier.rl.window_storage import collated_windows + + +def sequence_log_probabilities(actor, episode, start, end, chunk_size): + """One FP32 score per executable decision; context/padding never contribute.""" + actor.eval() + view = PolicyWindowView(actor.config) + for offset in range(start, end, chunk_size): + positions = range(offset, min(offset + chunk_size, end)) + with collated_windows( + (view.episode_inputs(episode, t, None) for t in positions), actor.device + ) as inputs: + actions = action_batch( + [episode.actions[t] for t in positions], + actor.config, + device=actor.device, + encoded=False, + ) + scores = actor.evaluate_actions(inputs, actions).float() + del inputs + if not torch.isfinite(scores).all(): + raise ValueError("non-finite sequence action log likelihood") + yield scores + + +def sequence_score(actor, episode, start, end, chunk_size): + return torch.stack( + [ + scores.sum() + for scores in sequence_log_probabilities( + actor, episode, start, end, chunk_size + ) + ] + ).sum() diff --git a/src/sequifier/rl/simulator_workers.py b/src/sequifier/rl/simulator_workers.py new file mode 100644 index 000000000..95a9cb560 --- /dev/null +++ b/src/sequifier/rl/simulator_workers.py @@ -0,0 +1,193 @@ +"""Opt-in spawned simulators with explicit requests, deadlines and bounded cleanup.""" + +import multiprocessing +import pickle +import time +import traceback +from contextlib import contextmanager +from multiprocessing.reduction import ForkingPickler + +from sequifier.io.memory_budget import resident_bytes +from sequifier.rl.environment import ( + adapter_identity, + load_online_adapter, + require_group_start, +) +from sequifier.runtime.resource_accounting import allocation + + +def require_parallel_group_start(adapter): + require_group_start(adapter) + try: + supported = adapter.supports_parallel_group_start + except AttributeError: + supported = False + if supported is not True: + raise ValueError( + "parallel GRPO requires adapter.supports_parallel_group_start = True; " + "its portable group-start tokens must work in independent adapter instances" + ) + + +def _serve(connection, spec, threads): + adapter = None + request = 0 + try: + import torch + + torch.set_num_threads(threads) + adapter = load_online_adapter(spec) + require_parallel_group_start(adapter) + connection.send((request, True, adapter_identity(adapter))) + operations = {"reset_group": adapter.reset_group, "step": adapter.step} + while True: + request, method, args = connection.recv() + if method == "close": + break + if method not in operations: + raise ValueError(f"unsupported simulator operation: {method}") + connection.send((request, True, operations[method](*args))) + except (EOFError, BrokenPipeError): + pass + except BaseException: + try: + connection.send((request, False, traceback.format_exc()[-32768:])) + except (OSError, EOFError): + pass + finally: + try: + if adapter is not None: + adapter.close() + finally: + connection.close() + + +class SimulatorWorkers: + """One in-flight request per worker; never retries an issued environment action.""" + + def __init__(self, spec, count, timeout, threads, expected_identity): + self.timeout = timeout + self.connections, self.processes = [], [] + self.pending, self.sequence = {}, 0 + self.requests = {} + self.closed = False + try: + context = multiprocessing.get_context("spawn") + for index in range(count): + parent, child = context.Pipe() + process = context.Process( + target=_serve, + args=(child, spec, threads), + name=f"sequifier-grpo-simulator-{index}", + ) + try: + process.start() + except BaseException: + parent.close() + child.close() + raise + child.close() + self.connections.append(parent) + self.processes.append(process) + self.pending[index] = (0, "startup", time.monotonic() + timeout) + for index in range(count): + if self.receive(index) != expected_identity: + raise ValueError( + f"simulator {index} adapter identity differs from parent" + ) + except BaseException: + self.close() + raise + + def __len__(self): + return len(self.connections) + + def send(self, index, method, *args): + if self.closed or index in self.pending: + raise RuntimeError( + "simulator is closed or already has an in-flight request" + ) + if method not in {"reset_group", "step"}: + raise ValueError(f"unsupported simulator operation: {method}") + self.sequence += 1 + self.pending[index] = (self.sequence, method, time.monotonic() + self.timeout) + try: + payload = ForkingPickler.dumps((self.sequence, method, args)) + storage = self.requests[index] = allocation("worker_payloads") + storage.resize(len(payload)) + self.connections[index].send_bytes(payload) + except (OSError, EOFError) as exc: + raise RuntimeError( + f"simulator {index} failed while sending {method}" + ) from exc + + def receive(self, index): + with self.response(index) as result: + return result + + @contextmanager + def response(self, index): + if self.closed or index not in self.pending: + raise RuntimeError("simulator has no pending response") + request, method, deadline = self.pending[index] + connection = self.connections[index] + with allocation("worker_payloads") as storage: + try: + if not connection.poll(max(0.0, deadline - time.monotonic())): + raise TimeoutError(f"simulator {index} timed out during {method}") + payload = connection.recv_bytes() + storage.resize(len(payload)) + received, ok, result = pickle.loads(payload) + except (OSError, EOFError) as exc: + raise RuntimeError(f"simulator {index} exited during {method}") from exc + if received != request: + raise RuntimeError( + f"simulator {index} returned an out-of-order response" + ) + del self.pending[index] + request_storage = self.requests.pop(index, None) + if request_storage is not None: + request_storage.close() + if not ok: + raise RuntimeError( + f"simulator {index} failed during {method}:\n{result}" + ) + if storage.accounting is not None: + decoded_size = resident_bytes(result) + storage.resize(len(payload) + decoded_size) + storage.resize(decoded_size) + del payload + yield result + + def close(self): + if self.closed: + return + self.closed = True + # Do not queue close behind a blocked, incomplete response. Such workers + # are terminated below; no pending collection is ever checkpointed. + for index, connection in enumerate(self.connections): + if index not in self.pending: + try: + connection.send((-1, "close", ())) + except (OSError, EOFError): + pass + deadline = time.monotonic() + 2.0 + for process in self.processes: + process.join(timeout=max(0.0, deadline - time.monotonic())) + for process in self.processes: + if process.is_alive(): + process.terminate() + deadline = time.monotonic() + 2.0 + for process in self.processes: + process.join(timeout=max(0.0, deadline - time.monotonic())) + for process in self.processes: + if process.is_alive(): + process.kill() + for process in self.processes: + process.join(timeout=0.1) + for connection in self.connections: + connection.close() + self.pending.clear() + for storage in self.requests.values(): + storage.close() + self.requests.clear() diff --git a/src/sequifier/rl/snapshots.py b/src/sequifier/rl/snapshots.py new file mode 100644 index 000000000..d85d97a29 --- /dev/null +++ b/src/sequifier/rl/snapshots.py @@ -0,0 +1,44 @@ +"""Portable environment snapshot comparison.""" + +import math + +import numpy as np +import torch + + +def snapshots_equal(left, right): + """Exact adapters use torch-serializable scalar, tensor and container state.""" + if type(left) is not type(right): + return False + if isinstance(left, torch.Tensor): + return ( + left.dtype == right.dtype + and bool(torch.isfinite(left).all()) + and torch.equal(left.cpu(), right.cpu()) + ) + if isinstance(left, np.ndarray): + finite = ( + np.isfinite(left).all() + if left.dtype.kind in "biufc" + else left.dtype.kind in "US" + ) + return ( + left.dtype == right.dtype and bool(finite) and np.array_equal(left, right) + ) + if isinstance(left, np.generic): + return snapshots_equal(left.item(), right.item()) + if isinstance(left, dict): + return left.keys() == right.keys() and all( + snapshots_equal(left[key], right[key]) for key in left + ) + if isinstance(left, (list, tuple)): + return len(left) == len(right) and all( + snapshots_equal(a, b) for a, b in zip(left, right) + ) + if isinstance(left, float): + return math.isfinite(left) and left == right + if isinstance(left, (str, int, bool, type(None))): + return left == right + raise ValueError( + "exact adapter snapshots must use scalar, tensor, NumPy array, dict, list, or tuple values" + ) diff --git a/src/sequifier/rl/telemetry.py b/src/sequifier/rl/telemetry.py new file mode 100644 index 000000000..51ccbd151 --- /dev/null +++ b/src/sequifier/rl/telemetry.py @@ -0,0 +1,87 @@ +"""Append-only policy telemetry with checkpoint-addressable histories.""" + +import json +import sqlite3 +from contextlib import closing, contextmanager +from uuid import uuid4 + + +def _connect(path, mode): + if mode != "rwc" and not path.is_file(): + raise FileNotFoundError(f"Missing policy telemetry journal: {path}") + return closing(sqlite3.connect(f"{path.resolve().as_uri()}?mode={mode}", uri=True)) + + +class TelemetryJournal: + def __init__(self, connection, position): + self.connection = connection + self.position = position + + @classmethod + @contextmanager + def open(cls, path, position=None): + if position is None: + path.parent.mkdir(parents=True, exist_ok=True) + with _connect(path, "rw" if position is not None else "rwc") as connection: + connection.execute("PRAGMA foreign_keys = ON") + connection.execute("PRAGMA synchronous = EXTRA") + connection.execute( + """CREATE TABLE IF NOT EXISTS records ( + id TEXT PRIMARY KEY, + previous TEXT REFERENCES records(id), + kind TEXT NOT NULL, + payload TEXT NOT NULL + )""" + ) + if ( + position is not None + and not connection.execute( + "SELECT 1 FROM records WHERE id = ?", (position,) + ).fetchone() + ): + raise ValueError(f"Missing committed telemetry position in {path}") + yield cls(connection, position) + + def append(self, kind, record): + position = uuid4().hex + self.connection.execute( + "INSERT INTO records VALUES (?, ?, ?, ?)", + ( + position, + self.position, + kind, + json.dumps(record, allow_nan=False, separators=(",", ":")), + ), + ) + self.position = position + + def commit(self): + # Publish the journal prefix before the checkpoint that references it. + self.connection.commit() + return self.position + + @staticmethod + def read(path, position): + if position is None: + return {} + # Following parent links leaves older snapshots readable after a resumed + # run adds a different suffix to their history. + # Allow SQLite to roll back an interrupted writer before reading. + with _connect(path, "rw") as connection: + rows = connection.execute( + """WITH RECURSIVE history AS ( + SELECT rowid, previous FROM records WHERE id = ? + UNION ALL + SELECT r.rowid, r.previous + FROM records r JOIN history h ON r.id = h.previous + ) + SELECT kind, payload FROM records + WHERE rowid IN (SELECT rowid FROM history) ORDER BY rowid""", + (position,), + ) + history = {} + for kind, payload in rows: + history.setdefault(kind, []).append(json.loads(payload)) + if not history: + raise ValueError(f"Missing committed telemetry position in {path}") + return history diff --git a/src/sequifier/rl/training.py b/src/sequifier/rl/training.py new file mode 100644 index 000000000..7e89b3ac4 --- /dev/null +++ b/src/sequifier/rl/training.py @@ -0,0 +1,96 @@ +"""Phase-local categorical behavior cloning, including return conditioning.""" + +import torch + +from sequifier.config.policy_plan import PolicyImitationPhaseConfig +from sequifier.io.policy_dataset import imitation_example +from sequifier.rl.batch_order import BatchOrder +from sequifier.rl.batching import batches as minibatches +from sequifier.rl.offline_data import OfflineData +from sequifier.rl.run_state import PhaseContext, PhaseSession +from sequifier.rl.window_storage import collated_windows + + +def action_nll(actor, batch): + return _action_nll(actor, batch.to(actor.device)) + + +def _action_nll(actor, batch): + actions = {name: value[:, -1] for name, value in batch.targets.items()} + return -actor.evaluate_actions(batch.model_inputs, actions).float() + + +@torch.no_grad() +def evaluate_imitation(actor, dataset, batch_size): + actor.eval() + total = 0.0 + for start in range(0, len(dataset), batch_size): + indices = range(start, min(start + batch_size, len(dataset))) + with collated_windows( + (imitation_example(dataset, i) for i in indices), actor.device + ) as batch: + total += float(_action_nll(actor, batch).sum()) + del batch + return total / len(dataset) + + +def run_imitation_phase(context: PhaseContext[PolicyImitationPhaseConfig, OfflineData]): + actor, phase, data = context.actor.eval(), context.phase, context.data + session = PhaseSession(context) + state = session.state + training = data.training + batches = BatchOrder( + len(training), + phase.batch_size * phase.accumulation_batches, + context.identity.sampling_seed, + sample=training.sampler, + state=state["batch_order"] if state else None, + ) + if ( + batches.epoch > phase.epochs + or session.transaction != batches.transactions + or session.steps["actor"] != session.transaction + ): + raise ValueError("imitation checkpoint epoch/transaction clocks disagree") + if batches.epoch == phase.epochs and batches.order is not None: + raise ValueError("completed imitation checkpoint has a live sampler order") + + def evaluate(**record): + score = evaluate_imitation(actor, data.validation, phase.evaluation.batch_size) + session.record_evaluation( + score, + epochs_completed=batches.epoch, + **record, + **{phase.evaluation.monitor.metric: score}, + ) + + if session.should_evaluate_initial(): + session.record_evaluation( + evaluate_imitation(actor, data.validation, phase.evaluation.batch_size), + initial=True, + ) + while batches.epoch < phase.epochs and not session.stopping.triggered: + group = batches.next() + session.zero_grad() + total = 0.0 + for indices in minibatches(group, phase.batch_size): + with collated_windows( + (imitation_example(training, i) for i in indices), actor.device + ) as batch: + loss = _action_nll(actor, batch).sum() / len(group) + if not torch.isfinite(loss): + raise ValueError("non-finite imitation action likelihood") + loss.backward() + total += float(loss.detach()) + del batch + session.step() + session.complete_update( + final=batches.epoch == phase.epochs, + training={"training_action_nll": total}, + evaluate=evaluate, + checkpoint={"batch_order": batches.state_dict()}, + ) + return session.finish( + {"optimizer_steps": session.steps["actor"], "epochs": batches.epoch}, + {"data": data.contract}, + ) diff --git a/src/sequifier/rl/window_storage.py b/src/sequifier/rl/window_storage.py new file mode 100644 index 000000000..36cdb3dcb --- /dev/null +++ b/src/sequifier/rl/window_storage.py @@ -0,0 +1,59 @@ +"""Account for reconstructed windows through collation, transfer and scoring.""" + +from collections.abc import Iterable, Iterator +from contextlib import contextmanager +from dataclasses import fields +from typing import cast + +import torch + +from sequifier.io.tensors import Record, TensorRecord, collate_tensor_records +from sequifier.runtime.resource_accounting import allocation + + +def tensor_storage_bytes(value, seen=None): + if seen is None: + seen = set() + if value is None: + return 0 + if isinstance(value, dict): + return sum(tensor_storage_bytes(item, seen) for item in value.values()) + if isinstance(value, TensorRecord): + return sum( + tensor_storage_bytes(vars(value)[field.name], seen) + for field in fields(value) + ) + storage = value.untyped_storage() + identity = storage._cdata + if identity in seen: + return 0 + seen.add(identity) + return storage.nbytes() + + +@contextmanager +def collated_windows( + windows: Iterable[Record], device: str | torch.device, *, borrowed: bool = False +) -> Iterator[Record]: + with allocation("reconstructed_windows") as batch_storage: + with allocation("reconstructed_windows") as rows_storage: + rows, seen, size = [], set(), 0 + for window in windows: + rows.append(window) + if not borrowed and rows_storage.accounting is not None: + size += tensor_storage_bytes(window, seen) + rows_storage.resize(size) + del window + batch = cast(Record, collate_tensor_records(rows)) + if batch_storage.accounting is not None: + batch_storage.resize(tensor_storage_bytes(batch)) + rows.clear() + + moved = batch.to(device) + if batch_storage.accounting is not None: + seen = set() + moved_size = tensor_storage_bytes(moved, seen) + batch_storage.resize(moved_size + tensor_storage_bytes(batch, seen)) + batch_storage.resize(moved_size) + del batch + yield moved diff --git a/src/sequifier/rl_api.py b/src/sequifier/rl_api.py new file mode 100644 index 000000000..6cf4b8f9a --- /dev/null +++ b/src/sequifier/rl_api.py @@ -0,0 +1,137 @@ +"""Local categorical policy learning, inference, and environment contracts.""" + +from sequifier.config.normalization_config import FrozenNormalizationConfig +from sequifier.config.parameter_selection import ( + ParameterSelector, + RoleTrainability, + TrainabilityConfig, +) +from sequifier.config.phase_controls import Cadence, StoppingConfig +from sequifier.config.policy_config import ( + ActionFieldConfig, + PolicyBinding, + ResolvedPolicyBinding, + ReturnDefinition, +) +from sequifier.config.policy_infer_config import ( + PolicyInferenceConfig, + PolicyRolloutConfig, +) +from sequifier.config.policy_optimization import OptimizationConfig +from sequifier.config.policy_plan import ( + DPOPhaseConfig, + GRPOEvaluationConfig, + GRPOPhaseConfig, + IQLPhaseConfig, + OnlineEnvironmentConfig, + OnlineEvaluationConfig, + PolicyImitationPhaseConfig, + PPOPhaseConfig, + ReferenceConfig, +) +from sequifier.config.policy_train_config import ( + PolicyTableColumns, + SequenceSourceConfig, +) +from sequifier.config.sequence_import_config import SequenceImportConfig +from sequifier.config.train_config import ResolvedSequifierConfig, SequifierConfig +from sequifier.io.batch import SequifierBatch, SequifierInputs, collate_sequence_batches +from sequifier.io.policy_dataset import PolicySequenceDataset, split_by_episode +from sequifier.io.policy_window import PolicyWindowView, action_index +from sequifier.io.sequence import SequenceEpisode, TransitionColumns +from sequifier.rl.checkpoint import load_policy, save_policy +from sequifier.rl.distributions import ( + CategoricalActionDistribution, + masked_action_nll_loss, +) +from sequifier.rl.environment import ( + EnvironmentCapabilities, + OnlineEnvironmentAdapter, + OnlineEnvironmentAdapterBase, + OnlineResetResult, + OnlineStepResult, +) +from sequifier.rl.learner import run_policy_training +from sequifier.rl.policy import Policy +from sequifier.rl.ppo_buffer import load_collection_record +from sequifier.rl.ppo_evaluation import evaluate_policy +from sequifier.rl.run_state import ( + ActorRepresentation, + PolicyPhaseResult, + PolicyRunResult, +) +from sequifier.rl.runtime import collect_policy_rollout +from sequifier.trajectory.export import import_sequences +from sequifier.trajectory.importers import source_episodes +from sequifier.trajectory.preferences import ( + PreferenceManifest, + PreferencePair, + PreferenceSpan, + load_preferences, +) +from sequifier.trajectory.returns import remaining_return, return_to_go + +__all__ = [ + "ParameterSelector", + "RoleTrainability", + "TrainabilityConfig", + "Cadence", + "StoppingConfig", + "ActorRepresentation", + "CategoricalActionDistribution", + "Policy", + "ResolvedPolicyBinding", + "DPOPhaseConfig", + "EnvironmentCapabilities", + "FrozenNormalizationConfig", + "GRPOPhaseConfig", + "GRPOEvaluationConfig", + "IQLPhaseConfig", + "OnlineEnvironmentAdapter", + "OnlineEnvironmentAdapterBase", + "OnlineEnvironmentConfig", + "OnlineEvaluationConfig", + "OnlineResetResult", + "OnlineStepResult", + "PolicyImitationPhaseConfig", + "PolicyRolloutConfig", + "PolicyInferenceConfig", + "OptimizationConfig", + "PolicyPhaseResult", + "SequifierConfig", + "ResolvedSequifierConfig", + "PolicyRunResult", + "ResolvedPolicyBinding", + "PolicySequenceDataset", + "PolicyTableColumns", + "PolicyWindowView", + "PPOPhaseConfig", + "PreferenceManifest", + "PreferencePair", + "PreferenceSpan", + "ReferenceConfig", + "ReturnDefinition", + "SequenceEpisode", + "SequenceImportConfig", + "SequenceSourceConfig", + "SequifierBatch", + "SequifierInputs", + "TransitionColumns", + "action_index", + "collate_sequence_batches", + "collect_policy_rollout", + "evaluate_policy", + "import_sequences", + "load_collection_record", + "load_policy", + "load_preferences", + "masked_action_nll_loss", + "remaining_return", + "return_to_go", + "run_policy_training", + "save_policy", + "source_episodes", + "split_by_episode", +] + +__all__ += ["ActionFieldConfig", "Policy", "PolicyBinding"] diff --git a/src/sequifier/rollout/__init__.py b/src/sequifier/rollout/__init__.py new file mode 100644 index 000000000..67c28b70e --- /dev/null +++ b/src/sequifier/rollout/__init__.py @@ -0,0 +1 @@ +"""Local numeric forecast collection.""" diff --git a/src/sequifier/rollout/contracts.py b/src/sequifier/rollout/contracts.py new file mode 100644 index 000000000..90ed4e620 --- /dev/null +++ b/src/sequifier/rollout/contracts.py @@ -0,0 +1,55 @@ +"""Adapter values are domain labels or finite scalars in original units.""" + +from __future__ import annotations + +from dataclasses import dataclass +from typing import Mapping, Protocol + +from sequifier.contract_versions import FORECAST_ROLLOUT_PROTOCOL, ContractVersions +from sequifier.io.values import DomainValue + + +@dataclass(frozen=True) +class ResetRequest: + episode_id: str + seed_history: Mapping[str, tuple[DomainValue, ...]] + attention_valid_mask: tuple[bool, ...] + static_inputs: Mapping[str, DomainValue] + seed: int + + +@dataclass(frozen=True) +class StepRequest: + episode_id: str + step_id: int + observation: Mapping[str, DomainValue] + predictions: Mapping[str, DomainValue] + static_inputs: Mapping[str, DomainValue] + + +@dataclass(frozen=True) +class StepResult: + external_inputs: Mapping[str, DomainValue] + reward: float + terminated: bool = False + truncated: bool = False + + +class ForecastRolloutAdapterBase: + """Supply the protocol metadata for a numeric forecast adapter.""" + + protocol = FORECAST_ROLLOUT_PROTOCOL + protocol_version = ContractVersions.FORECAST_ROLLOUT_ADAPTER + + +class EnvironmentAdapter(Protocol): + protocol: str + protocol_version: int + + def reset(self, request: ResetRequest) -> None: + """Initialize from the supplied history; reset must not replace the seed.""" + ... + + def step(self, request: StepRequest) -> StepResult: ... + + def close(self) -> None: ... diff --git a/src/sequifier/rollout/coordinator.py b/src/sequifier/rollout/coordinator.py new file mode 100644 index 000000000..2c5ba3ad9 --- /dev/null +++ b/src/sequifier/rollout/coordinator.py @@ -0,0 +1,148 @@ +"""Shared prediction/execution ordering and single-environment collection.""" + +from collections.abc import Callable +from copy import deepcopy +from uuid import uuid4 + +from sequifier.config.rollout_config import RolloutConfig +from sequifier.io.values import DomainValue +from sequifier.rollout.contracts import ( + EnvironmentAdapter, + ResetRequest, + StepRequest, + StepResult, +) +from sequifier.rollout.factories import validate_adapter +from sequifier.rollout.history import ( + advance_history, + numeric_fields, + numeric_scalar, + validate_seed, +) +from sequifier.trajectory.recording import TrajectoryRecorder, close_adapter +from sequifier.trajectory.schema import SeedHistory, Trajectory + + +def collect_rollout( + predictor: Callable, + adapter: EnvironmentAdapter, + seed: SeedHistory, + config: RolloutConfig, + prediction_columns: list[str], + max_steps: int, + *, + random_seed: int = 1010, + metadata: dict | None = None, + history_validator: Callable | None = None, +) -> Trajectory: + """Collect one episode fragment. This call owns and always closes the adapter. + + Predictor receives a copy of the current SeedHistory in original units and + returns named scalar forecasts. Conversion to model units belongs in that + predictor wrapper, using fixed training metadata. + """ + failed = False + try: + if (not isinstance(max_steps, int)) or max_steps < 1: + raise ValueError("rollout collection budget must be a positive integer") + validate_adapter(adapter) + history = validate_seed(seed) + for column, values in history.observations.items(): + for value in values: + numeric_scalar(value, f"forecast seed.{column}") + numeric_fields( + history.static_inputs, + set(history.static_inputs), + "forecast static inputs", + ) + initial_history = history.model_copy(deep=True) + if history_validator is not None: + history_validator(history.model_copy(deep=True)) + if not prediction_columns or len(prediction_columns) != len( + set(prediction_columns) + ): + raise ValueError("prediction_columns must be nonempty and unique") + config.validate_bindings(list(history.observations), prediction_columns) + episode_id = str(uuid4()) + reset_request = ResetRequest( + episode_id, + {key: tuple(values) for key, values in history.observations.items()}, + tuple(history.attention_valid_mask), + dict(history.static_inputs), + random_seed, + ) + adapter.reset(deepcopy(reset_request)) + recorder = TrajectoryRecorder(episode_id, config.max_episode_steps) + external_keys = { + binding.key + for binding in config.bindings + if binding.source == "environment" + } + + for step_id in range(max_steps): + predictions = numeric_fields( + predictor(history.model_copy(deep=True)), + set(prediction_columns), + "predictions", + ) + observation = { + key: values[-1] for key, values in history.observations.items() + } + result = adapter.step( + StepRequest( + episode_id, + step_id, + dict(observation), + dict(predictions), + dict(history.static_inputs), + ) + ) + if not isinstance(result, StepResult): + raise ValueError("adapter.step must return StepResult") + if not isinstance(result.terminated, bool) or not isinstance( + result.truncated, bool + ): + raise ValueError( + "adapter termination and truncation flags must be separate booleans" + ) + external: dict[str, DomainValue] = { + key: value + for key, value in numeric_fields( + result.external_inputs, external_keys, "adapter.external_inputs" + ).items() + } + reward = numeric_scalar(result.reward, "adapter.reward") + successor = { + binding.input: ( + predictions if binding.source == "prediction" else external + )[binding.key] + for binding in config.bindings + } + + next_history = advance_history(history, successor) + if history_validator is not None: + history_validator(next_history.model_copy(deep=True)) + transition = recorder.append( + observation=observation, + predictions=predictions, + external_inputs=external, + reward=reward, + successor_observation=successor, + terminated=result.terminated, + truncated=result.truncated, + ) + history = next_history + if transition.terminated or transition.truncated: + break + return recorder.finish( + format="sequifier.numeric-forecast-trajectory", + seed=initial_history, + bindings=config.bindings, + prediction_columns=prediction_columns, + metadata=metadata or {}, + ) + except BaseException: + failed = True + raise + finally: + close_adapter(adapter, failed) diff --git a/src/sequifier/rollout/factories.py b/src/sequifier/rollout/factories.py new file mode 100644 index 000000000..9b04e3b2e --- /dev/null +++ b/src/sequifier/rollout/factories.py @@ -0,0 +1,47 @@ +"""Persistent adapter loading; factories receive their config as keyword arguments.""" + +from typing import cast + +from sequifier.config.rollout_config import AdapterConfig +from sequifier.contract_versions import ( + FORECAST_ROLLOUT_PROTOCOL, + ContractVersions, + require_contract_version, +) +from sequifier.integration.factories import load_factory +from sequifier.rollout.contracts import EnvironmentAdapter + + +def validate_adapter(adapter): + try: + protocol = adapter.protocol + except AttributeError: + protocol = None + if protocol != FORECAST_ROLLOUT_PROTOCOL: + raise ValueError(f"adapter.protocol must be {FORECAST_ROLLOUT_PROTOCOL!r}") + try: + protocol_version = adapter.protocol_version + except AttributeError: + protocol_version = None + require_contract_version( + "forecast rollout adapter protocol", + protocol_version, + ContractVersions.FORECAST_ROLLOUT_ADAPTER, + ) + try: + methods = (adapter.reset, adapter.step, adapter.close) + except AttributeError as error: + raise ValueError( + f"rollout adapter requires a callable {error.name} method" + ) from error + for name, method in zip(("reset", "step", "close"), methods): + if not callable(method): + raise ValueError(f"rollout adapter requires a callable {name} method") + + +def load_adapter(spec: AdapterConfig) -> EnvironmentAdapter: + factory = load_factory(spec.factory) + if not callable(factory): + raise ValueError(f"Adapter factory {spec.factory!r} is not callable") + # Runtime validation happens inside collect_rollout's cleanup boundary. + return cast(EnvironmentAdapter, factory(**spec.config)) diff --git a/src/sequifier/rollout/history.py b/src/sequifier/rollout/history.py new file mode 100644 index 000000000..cbc90cbc5 --- /dev/null +++ b/src/sequifier/rollout/history.py @@ -0,0 +1,25 @@ +"""Validated seed windows and atomic numeric feature/mask advancement.""" + +from sequifier.io.values import ( # noqa: F401 + DomainValue, + numeric_fields, + numeric_scalar, +) +from sequifier.trajectory.schema import SeedHistory + + +def validate_seed(seed: SeedHistory) -> SeedHistory: + return SeedHistory.model_validate(seed.model_dump()) + + +def advance_history( + history: SeedHistory, successor: dict[str, DomainValue] +) -> SeedHistory: + return SeedHistory( + observations={ + key: values[1:] + [successor[key]] + for key, values in history.observations.items() + }, + attention_valid_mask=history.attention_valid_mask[1:] + [True], + static_inputs=dict(history.static_inputs), + ) diff --git a/src/sequifier/rollout/preprocessing.py b/src/sequifier/rollout/preprocessing.py new file mode 100644 index 000000000..44957560e --- /dev/null +++ b/src/sequifier/rollout/preprocessing.py @@ -0,0 +1,106 @@ +"""Fixed numeric training transforms; never fit on seeds or adapter values.""" + +import numpy as np + +from sequifier.io.preprocessing import ( + NORMALIZATION_EPSILON, + denormalize_values, + normalization_statistics, + normalize_values, +) +from sequifier.rollout.history import validate_seed +from sequifier.trajectory.schema import SeedHistory + + +class FixedNumericPreprocessing: + def __init__(self, columns, statistics, normalize, dtypes): + self.normalize = normalize + self.statistics = ( + normalization_statistics( + {column: statistics.get(column, {}) for column in columns} + ) + if normalize + else {} + ) + self.dtypes = {column: np.dtype(dtypes[column]) for column in columns} + for column in columns: + if self.dtypes[column].kind != "f": + raise ValueError( + f"rollout field {column!r} must have a floating numeric dtype" + ) + + def decode(self, column, values): + values = np.asarray(values, dtype=np.float64) + if self.normalize: + values = denormalize_values(values, self.statistics[column]) + if not np.isfinite(values).all(): + raise ValueError(f"rollout field {column!r} decoded to non-finite values") + return values + + def encode(self, column, values): + values = np.asarray(values, dtype=np.float64) + if self.normalize: + values = normalize_values(values, self.statistics[column]) + with np.errstate(over="ignore", invalid="ignore"): + values = values.astype(self.dtypes[column]) + if not np.isfinite(values).all(): + raise ValueError( + f"rollout field {column!r} is incompatible with its model dtype/scale" + ) + return values + + def seed_from_model(self, inputs, mask, input_columns, static_columns): + if mask.ndim != 2 or mask.shape[0] != 1 or mask.dtype != np.bool_: + raise ValueError( + "rollout seed requires a boolean attention mask for one window" + ) + observations, static = {}, {} + for column in input_columns: + values = inputs[column] + if values.shape != mask.shape: + raise ValueError( + f"rollout seed {column!r} must be a scalar feature window matching its mask" + ) + # Padding is represented by neutral domain values, never decoded as observations. + decoded = np.zeros(values.shape, dtype=np.float64) + decoded[mask] = self.decode(column, values[mask]) + observations[column] = decoded[0].tolist() + for column in static_columns: + if inputs[column].shape != (1,): + raise ValueError( + f"rollout static input {column!r} must have shape (1,)" + ) + static[column] = float(self.decode(column, inputs[column])[0]) + return validate_seed( + SeedHistory( + observations=observations, + attention_valid_mask=mask[0].tolist(), + static_inputs=static, + ) + ) + + def model_inputs(self, history: SeedHistory): + mask = np.asarray([history.attention_valid_mask], dtype=np.bool_) + features = {} + for column, values in history.observations.items(): + encoded = np.zeros(mask.shape, dtype=self.dtypes[column]) + # False-mask padding is not a physical value to normalize. + encoded[mask] = self.encode(column, np.asarray([values])[mask]) + features[column] = encoded + features.update( + { + column: self.encode(column, [value]) + for column, value in history.static_inputs.items() + } + ) + return features, {"attention_valid_mask": mask} + + def to_dict(self): + return { + "normalize_real_columns": self.normalize, + "selected_columns_statistics": self.statistics, + "model_dtypes": { + column: dtype.name for column, dtype in self.dtypes.items() + }, + "epsilon": NORMALIZATION_EPSILON, + } diff --git a/src/sequifier/rollout_api.py b/src/sequifier/rollout_api.py new file mode 100644 index 000000000..1e1798b9b --- /dev/null +++ b/src/sequifier/rollout_api.py @@ -0,0 +1,35 @@ +"""Public API for local numeric forecast adapters and trajectory collection.""" + +from sequifier.config.rollout_config import AdapterConfig, InputBinding, RolloutConfig +from sequifier.rollout.contracts import ( + EnvironmentAdapter, + ForecastRolloutAdapterBase, + ResetRequest, + StepRequest, + StepResult, +) +from sequifier.rollout.coordinator import collect_rollout +from sequifier.trajectory import ( + Trajectory, + load_trajectory, + save_trajectory, + validate_trajectory, +) +from sequifier.trajectory.schema import SeedHistory + +__all__ = [ + "AdapterConfig", + "InputBinding", + "RolloutConfig", + "EnvironmentAdapter", + "ForecastRolloutAdapterBase", + "ResetRequest", + "StepRequest", + "StepResult", + "SeedHistory", + "Trajectory", + "collect_rollout", + "load_trajectory", + "save_trajectory", + "validate_trajectory", +] diff --git a/src/sequifier/runtime/builder.py b/src/sequifier/runtime/builder.py index 6a8cc732d..ee94a1cc3 100644 --- a/src/sequifier/runtime/builder.py +++ b/src/sequifier/runtime/builder.py @@ -2,28 +2,34 @@ from __future__ import annotations -import uuid from collections.abc import Callable from dataclasses import dataclass from pathlib import Path -from typing import Any +from typing import Any, cast import torch from loguru import logger from sequifier.artifacts.backbone_repository import load_revision, select_revision +from sequifier.artifacts.dependencies import ( + DependencyManifest, + add_backbone_revision, + supervised_dependencies, +) from sequifier.artifacts.run_checkpoint import RunCheckpointStore, select_run_checkpoint from sequifier.config.optimizer_config import OptimizerPlan from sequifier.evaluation.service import EvaluationService from sequifier.export.service import ExportService -from sequifier.helpers import get_torch_dtype from sequifier.integration.callbacks import IntegrationManager +from sequifier.io.memory_budget import plan_data_loading from sequifier.model.factory import build_transformer_network, compile_unique_layers from sequifier.model.parameter_catalog import ( ParameterCatalog, semantic_optimizer_groups, ) from sequifier.runtime.context import ExecutionEnvironment, RunContext +from sequifier.runtime.plan import RunIdentity +from sequifier.runtime.precision import PrecisionPolicy from sequifier.runtime.random_state import RandomStateManager from sequifier.training.checkpoint_service import CheckpointRestorer, CheckpointService from sequifier.training.distributed_strategy import ( @@ -35,23 +41,30 @@ from sequifier.training.loader_state import LoaderStateService from sequifier.training.loss import LossService from sequifier.training.metrics_service import MetricsService -from sequifier.training.optimization import OptimizationRuntime +from sequifier.training.optimization import GradientPolicy, OptimizationRuntime +from sequifier.training.plan import SupervisedPhaseView, SupervisedRunPlan from sequifier.training.runtime import ( - DatasetFreezingPlan, + DatasetMetrics, DatasetRuntimeRegistry, + PhaseTrainabilityPlan, + build_dataset_parts, build_dataset_runtimes, + validate_phase_learning, ) +from sequifier.training.scheduler_budgets import resolve_scheduler_budgets from sequifier.training.state import RunState @dataclass class TrainingRun: config: Any + plan: SupervisedRunPlan network: Any callable_network: Any datasets: DatasetRuntimeRegistry optimization: OptimizationRuntime - optimization_factory: Callable[[int], OptimizationRuntime] + dataset_factory: Callable[[SupervisedPhaseView], DatasetRuntimeRegistry] + optimization_factory: Callable[[SupervisedPhaseView], OptimizationRuntime] optimization_phase_index: int state: RunState distributed: DistributedStrategy @@ -64,14 +77,28 @@ class TrainingRun: loss: LossService loader_state: LoaderStateService context: RunContext + dependencies: DependencyManifest + precision: PrecisionPolicy - def start_phase(self, phase_index: int) -> None: - """Start a phase with fresh optimizer, scheduler, and scaler state.""" - + def start_phase( + self, + view: SupervisedPhaseView, + datasets: DatasetRuntimeRegistry, + *, + continue_optimizer: bool, + ) -> None: + """Enter the phase with its resolved reset or continuation policy.""" + phase = view.spec() optimizer_step = self.optimization.optimizer_step - self.optimization = self.optimization_factory(phase_index) + if not continue_optimizer: + self.optimization = self.optimization_factory(view) + self.optimization.gradient_policy = GradientPolicy( + phase.accumulation_batches, + phase.optimization["model"].gradient_clip_norm, + ) self.optimization.optimizer_step = optimizer_step - self.optimization_phase_index = phase_index + self.optimization_phase_index = view.index + self.datasets = datasets class RunBuilder: @@ -79,29 +106,28 @@ def __init__(self, *, semantic_optimizer_grouping: bool = False) -> None: self.semantic_optimizer_grouping = semantic_optimizer_grouping def _strategy( - self, config: Any, execution: ExecutionEnvironment + self, config: Any, execution: ExecutionEnvironment, precision: PrecisionPolicy ) -> DistributedStrategy: if not execution.distributed: return LocalStrategy(device=execution.device) - if config.global_training.data_parallelism == "fsdp": - dtype = None - if config.global_training.layer_autocast: - layer_types = config.global_training.layer_type_dtypes or {} - dtype = get_torch_dtype(layer_types.get("linear", "bfloat16")) + if config.runtime_settings.data_parallelism == "fsdp": return FullyShardedStrategy( rank=execution.rank, local_rank=execution.local_rank, world_size=execution.world_size, device=execution.device, - cpu_offload=bool(config.global_training.fsdp_cpu_offload), - mixed_precision_dtype=dtype, + cpu_offload=bool(config.runtime_settings.fsdp_cpu_offload), + mixed_precision_dtype=precision.autocast_dtype, ) return DistributedDataParallelStrategy( rank=execution.rank, local_rank=execution.local_rank, world_size=execution.world_size, device=execution.device, - find_unused_parameters=len(config.dataset_training) > 1, + find_unused_parameters=max( + len(config.dataset_training), len(config.interfaces) + ) + > 1, ) def _warm_up(self, run: TrainingRun) -> None: @@ -122,14 +148,14 @@ def _warm_up(self, run: TrainingRun) -> None: try: for dataset in run.datasets.values(): schema = ExecutionSchema.from_interface( - dataset.config.interface, - run.config.global_training.context_length, + dataset.interface, + run.config.runtime_settings.context_length, ) features, metadata = schema.bind( - schema.example_inputs(batch_size, run.distributed.device) + schema.example_inputs(batch_size, str(run.distributed.device)) ) schema.validate(features, metadata) - with torch.no_grad(): + with torch.no_grad(), run.precision.autocast(): run.callable_network( features, metadata, interface_name=dataset.interface_name ) @@ -139,30 +165,96 @@ def _warm_up(self, run: TrainingRun) -> None: def _construct_loaders(self, run: TrainingRun) -> None: """Construct every configured loader before final RNG restoration.""" + phase = run.plan.phases[run.state.phase_index] for dataset in run.datasets.values(): for part_name, part in dataset.parts.items(): - part.loader("training") + part.loader(phase, "training") if dataset.config.parts[part_name].validation_data_path is not None: - part.loader("validation") + part.loader(phase, "validation") def build( self, config: Any, execution: ExecutionEnvironment, integrations: IntegrationManager, + *, + initial_artifact=None, + phase_offset: int = 0, + identity: RunIdentity | None = None, + on_checkpoint=None, + backbone_revision=None, + dependencies: DependencyManifest | None = None, ) -> TrainingRun: - strategy = self._strategy(config, execution) + from sequifier.artifacts.run_checkpoint import ( + checkpoint_path as output_checkpoint_path, + ) + from sequifier.io.output_paths import validate_training_destinations + + config = config.model_copy() + precision = PrecisionPolicy.from_settings( + config.runtime_settings, execution.device + ) + memory_budget = plan_data_loading(config) + strategy = self._strategy(config, execution, precision) random_manager = RandomStateManager(execution.device) loader_state = LoaderStateService() restorer = CheckpointRestorer() checkpoint_path = select_run_checkpoint(config) loaded = restorer.load(checkpoint_path) if checkpoint_path is not None else None + if ( + loaded is not None + and identity is not None + and loaded.checkpoint.run_state["run_id"] != identity.run_id + ): + if config.runtime_settings.resume.policy == "required": + raise ValueError("Supervised recovery belongs to another run") + loaded = None + revision = None + dependencies = dependencies or supervised_dependencies(config, initial_artifact) + if initial_artifact is None and config.initialize_from is None: + if loaded is None: + revision = ( + select_revision(config.model.backbone, config.project_root) + if backbone_revision is None + else backbone_revision or None + ) + revision_id = ( + loaded.checkpoint.run_state["backbone_parent_revision_id"] + if loaded is not None + else revision["revision_id"] + if revision is not None + else None + ) + add_backbone_revision(dependencies, config, revision_id) + validate_training_destinations( + config, + output_checkpoint_path(config), + dependencies.protected_paths, + phase_offset=phase_offset, + ) if loaded is not None: - restorer.validate_for_restore(loaded, config) + restorer.validate_for_restore(loaded, config, dependencies) + # Optimizer state uses the parameter order of the saved network. + interfaces = { + name: config.interfaces[name] + for name in loaded.checkpoint.model.model_config.values["interfaces"] + } + config = config.model_copy(update={"interfaces": interfaces}) state = RunState.from_state_dict(loaded.checkpoint.run_state) - state.session_id = uuid.uuid4().hex else: - state = RunState() + state = RunState(phase_offset=phase_offset) + if state.phase_offset != phase_offset: + raise ValueError("Supervised recovery has a different phase offset") + if identity is None: + identity = strategy.gather_objects( + RunIdentity.start(state.run_id) if strategy.rank == 0 else None + )[0] + if not isinstance(identity, RunIdentity): + raise RuntimeError("Rank zero did not provide a run identity") + state.run_id, state.session_id = identity.run_id, identity.session_id + + plan = SupervisedRunPlan.from_config(config) + validate_phase_learning(config, plan, execution.device) from sequifier.export.preflight import ensure_export_preflight @@ -174,21 +266,65 @@ def build( logger=logger, ) network = built.network - if loaded is None: - revision = select_revision(config.model.backbone, config.project_root) - if revision is not None: - load_revision(network.backbone, revision) - state.backbone_parent_revision_id = revision["revision_id"] + if loaded is None and initial_artifact is not None: + from sequifier.artifacts.model_artifact import read_execution_artifact + from sequifier.model.execution import ModelExecution + + artifact = read_execution_artifact(initial_artifact) + from sequifier.artifacts.model_config import validate_execution_attachment - freezing_plan = DatasetFreezingPlan.resolve(config, network) - freezing_plan.apply_permanent(network) + validate_execution_attachment( + artifact.model_config.values, ModelExecution.from_config(config) + ) + from sequifier.artifacts.model_config import load_attached_model_weights + + load_attached_model_weights(network, artifact) + if revision is not None: + load_revision(network.backbone, revision) + state.backbone_parent_revision_id = revision["revision_id"] + + freezing_plans = [] + planned_phases = [] + permanent_masks = [] + for view in plan.phases: + phase = view.spec() + freezing_plan = PhaseTrainabilityPlan.resolve(config, network, phase) + masks = freezing_plan.parameter_ids_by_dataset + if phase.optimizer_state == "continue" and ( + not freezing_plans or freezing_plan != freezing_plans[-1] + ): + raise ValueError( + "optimizer continuation requires identical resolved parameter masks" + ) + freezing_plans.append(freezing_plan) + permanent_masks.extend(set(selected) for selected in masks.values()) + summary = { + name: sorted(masks[name]) + for name in dict.fromkeys(source.dataset for source in phase.sources) + } + planned_phases.append( + view.with_spec( + phase.model_copy(update={"trainability_summary": summary}) + ) + ) + plan = SupervisedRunPlan(tuple(planned_phases)) + # Compatibility: avoid building backward graphs for parameters that + # every dataset in every phase leaves frozen. + if permanent_masks: + catalog = ParameterCatalog(network) + for parameter_id in set.intersection(*permanent_masks): + catalog.parameter(parameter_id).requires_grad_(False) compile_before_ddp = ( execution.distributed - and config.global_training.data_parallelism == "ddp" - and config.global_training.torch_compile == "outer" + and config.runtime_settings.data_parallelism == "ddp" + and config.runtime_settings.torch_compile == "outer" + ) + network_to_prepare = ( + cast(torch.nn.Module, torch.compile(network)) + if compile_before_ddp + else network ) - network_to_prepare = torch.compile(network) if compile_before_ddp else network prepared = strategy.prepare_network(network_to_prepare) revisions = strategy.gather_objects(state.backbone_parent_revision_id) if any(revision != revisions[0] for revision in revisions[1:]): @@ -198,56 +334,60 @@ def build( if loaded is not None: restorer.restore_model(loaded, network, strategy) - datasets = build_dataset_runtimes( - config, - network, - execution.device, - objectives=built.objectives, - runtime_metadata=built.runtime_metadata, - freezing_plan=freezing_plan, - ) - parameters: Any = tuple(strategy.prepare_optimizer_parameters(network)) - optimizer_plan = isinstance(config.global_training.optimizer, OptimizerPlan) - catalog = ( - ParameterCatalog(network) - if optimizer_plan or self.semantic_optimizer_grouping - else None - ) - if self.semantic_optimizer_grouping and not optimizer_plan: - assert catalog is not None - parameters = semantic_optimizer_groups( - catalog, - parameters={id(parameter) for parameter in parameters}, + parts = build_dataset_parts(config, plan, memory_budget) + dataset_metrics = {name: DatasetMetrics() for name in parts} + plan = resolve_scheduler_budgets(plan, config, parts) + active_phase = plan.phases[state.phase_index] + + def dataset_factory(view: SupervisedPhaseView) -> DatasetRuntimeRegistry: + return build_dataset_runtimes( + config, + view, + execution.device, + parts=parts, + metrics=dataset_metrics, + runtime_metadata=built.runtime_metadata, + masks=freezing_plans[view.index].bind(network), ) - reset_each_phase = config.global_training.reset_optimization_on_phase - total_epochs = sum(phase.epochs for phase in config.training_plan) + datasets = dataset_factory(active_phase) + parameters: Any = tuple(strategy.prepare_optimizer_parameters(network)) + catalog = ParameterCatalog(network) - def optimization_factory(phase_index: int) -> OptimizationRuntime: + def optimization_factory(view: SupervisedPhaseView) -> OptimizationRuntime: + phase = view.spec() + role = phase.optimization["model"] + selected_parameters = parameters + if self.semantic_optimizer_grouping and not isinstance( + role.optimizer, OptimizerPlan + ): + selected_parameters = semantic_optimizer_groups( + catalog, parameters={id(parameter) for parameter in parameters} + ) return OptimizationRuntime.create( - config.global_training, + role, str(execution.device), - parameters, + selected_parameters, parameter_catalog=catalog, - phase_epochs=( - config.training_plan[phase_index].epochs - if reset_each_phase - else total_epochs - ), + accumulation_batches=phase.accumulation_batches, + layer_type_dtypes=config.runtime_settings.layer_type_dtypes, + autocast_dtype=precision.autocast_dtype, ) - optimization = optimization_factory(state.phase_index) + optimization = optimization_factory(active_phase) if loaded is not None: restorer.restore_optimization(loaded, optimization, network, strategy) callable_network = prepared.callable_network - if config.global_training.torch_compile == "inner": + if config.runtime_settings.torch_compile == "inner": from sequifier.model.encoder_stack import TransformerEncoderStack for stack in list(network.modules()): if isinstance(stack, TransformerEncoderStack): compile_unique_layers(stack.layers) - elif config.global_training.torch_compile == "outer" and not compile_before_ddp: + elif ( + config.runtime_settings.torch_compile == "outer" and not compile_before_ddp + ): callable_network = torch.compile(callable_network) context = RunContext( @@ -262,9 +402,11 @@ def optimization_factory(phase_index: int) -> OptimizationRuntime: loss_service = LossService() run = TrainingRun( config=config, + plan=plan, network=network, callable_network=callable_network, datasets=datasets, + dataset_factory=dataset_factory, optimization=optimization, optimization_factory=optimization_factory, optimization_phase_index=state.phase_index, @@ -272,22 +414,26 @@ def optimization_factory(phase_index: int) -> OptimizationRuntime: distributed=strategy, random=random_manager, integrations=integrations, - evaluation=EvaluationService(loss_service), + evaluation=EvaluationService(loss_service, precision=precision), exporting=ExportService(config, execution.rank), checkpoints=CheckpointService( RunCheckpointStore(config, config.model_name), strategy, random_manager, loader_state, + on_checkpoint=on_checkpoint, ), metrics=MetricsService(context, config), loss=loss_service, loader_state=loader_state, context=context, + dependencies=dependencies, + precision=precision, ) self._construct_loaders(run) if loaded is not None: restorer.restore_runtime(loaded, run) + run.state.run_id = context.run_id run.state.session_id = context.session_id self._warm_up(run) if loaded is not None: diff --git a/src/sequifier/runtime/plan.py b/src/sequifier/runtime/plan.py new file mode 100644 index 000000000..e84904b16 --- /dev/null +++ b/src/sequifier/runtime/plan.py @@ -0,0 +1,128 @@ +"""Common phase ordering, commit boundaries, and plan stopping.""" + +from collections.abc import Callable, Sequence +from dataclasses import dataclass +from typing import Generic, TypeVar +from uuid import uuid4 + +Phase = TypeVar("Phase") +Result = TypeVar("Result") +Commit = TypeVar("Commit") + + +def new_run_id() -> str: + return uuid4().hex + + +@dataclass(frozen=True) +class RunIdentity: + run_id: str + session_id: str + + @classmethod + def start(cls, run_id: str | None = None): + return cls(run_id or new_run_id(), new_run_id()) + + +@dataclass(frozen=True) +class PhaseOutcome(Generic[Result]): + result: Result + stop_scope: str | None = None + completion_reason: str | None = None + + +def stop_scope(stopping, triggered: bool) -> str | None: + return stopping.on_trigger if triggered and stopping is not None else None + + +def metric_improved(value, best, direction, min_delta=0) -> bool: + if best is None: + return True + return value < best - min_delta if direction == "min" else value > best + min_delta + + +def select_handoff(handoff: str, last: Result, best: Result | None) -> Result: + return best if handoff == "best" and best is not None else last + + +@dataclass(frozen=True) +class PhaseTransition(Generic[Phase]): + index: int + phase: Phase + previous: Phase | None + continues_optimizer: bool = False + preserves_optimizer: bool = False + + +def validate_continuation(previous, phase, *, resolved=False): + if phase.optimizer_state != "continue": + return + if previous is None: + raise ValueError("the first phase cannot continue optimizer state") + if previous.algorithm != phase.algorithm or previous.handoff != "last": + raise ValueError( + "optimizer continuation requires the same algorithm and a last handoff" + ) + if ( + resolved + and phase.algorithm != "supervised" + and previous.model_interface != phase.model_interface + ): + raise ValueError("optimizer continuation cannot change model_interface") + + +def continuation_groups(phases): + """A reset starts a new portable execution boundary.""" + start = 0 + for index, phase in enumerate(phases[1:], 1): + if phase.optimizer_state != "continue": + yield start, phases[start:index] + start = index + yield start, phases[start:] + + +def run_phases( + phases: Sequence[Phase], + run_phase: Callable[[PhaseTransition[Phase]], PhaseOutcome[Result]], + commit_phase: Callable[[PhaseTransition[Phase], PhaseOutcome[Result]], Commit], + *, + start: int = 0, + stopped: bool = False, + completion_reason: str = "completed", + replay: Callable[[], None] | None = None, + publish: Callable[[Commit], None] | None = None, + terminal_reason: str | None = None, + optimizer_continues: Callable[[Phase], bool] | None = None, +) -> str: + """Replay committed publications, then advance only after a durable commit.""" + if replay is not None: + replay() + if stopped: + return "early_stopping" + if terminal_reason is not None: + return terminal_reason + for index in range(start, len(phases)): + transition = PhaseTransition( + index, + phases[index], + phases[index - 1] if index else None, + continues_optimizer=( + index > 0 + and optimizer_continues is not None + and optimizer_continues(phases[index]) + ), + preserves_optimizer=( + index + 1 < len(phases) + and optimizer_continues is not None + and optimizer_continues(phases[index + 1]) + ), + ) + outcome = run_phase(transition) + commit = commit_phase(transition, outcome) + if publish is not None: + publish(commit) + if outcome.stop_scope == "end_plan": + return "early_stopping" + if outcome.completion_reason is not None: + return outcome.completion_reason + return completion_reason diff --git a/src/sequifier/runtime/precision.py b/src/sequifier/runtime/precision.py new file mode 100644 index 000000000..65cac3915 --- /dev/null +++ b/src/sequifier/runtime/precision.py @@ -0,0 +1,63 @@ +"""A process-local precision policy shared by forward, validation and warm-up.""" + +from contextlib import nullcontext +from dataclasses import dataclass +from typing import Any + +import torch + + +@dataclass(frozen=True) +class PrecisionPolicy: + device_type: str + autocast_dtype: torch.dtype | None = None + sharded: bool = False + + @classmethod + def from_settings(cls, settings: Any, device: torch.device) -> "PrecisionPolicy": + torch.set_float32_matmul_precision(settings.float32_matmul_precision) + sharded = settings.distributed and settings.data_parallelism == "fsdp" + if not settings.layer_autocast: + return cls(device.type, sharded=sharded) + if not torch.amp.autocast_mode.is_autocast_available(device.type): + raise ValueError(f"layer_autocast is unavailable on {device.type}") + configured = (settings.layer_type_dtypes or {}).get("linear") + dtype = { + "float16": torch.float16, + "bfloat16": torch.bfloat16, + }.get(configured or "") + if dtype is None: + dtype = ( + torch.bfloat16 + if sharded or device.type == "mps" + else torch.get_autocast_dtype(device.type) + ) + if dtype not in {torch.float16, torch.bfloat16}: + raise ValueError(f"layer_autocast requires FP16 or BF16, found {dtype}") + if device.type == "cuda": + if not torch.cuda.is_available(): + raise ValueError( + "CUDA layer_autocast requires an available CUDA device" + ) + with torch.cuda.device(device): + if dtype == torch.bfloat16 and not torch.cuda.is_bf16_supported(): + raise ValueError( + "BF16 layer_autocast is unsupported by this CUDA device" + ) + if device.type == "mps" and ( + not torch.backends.mps.is_available() + or not torch.backends.mps.is_macos_or_newer(14, 0) + ): + raise ValueError("BF16 layer_autocast requires MPS on macOS 14 or newer") + if dtype == torch.float16 and device.type not in {"cpu", "cuda"}: + # Native recovery owns a GradScaler; do not silently run unscaled AMP. + raise ValueError( + f"FP16 layer_autocast on {device.type} has no supported native " + "gradient scaler; select BF16 linear precision where supported" + ) + return cls(device.type, dtype, sharded) + + def autocast(self): + if self.autocast_dtype is None or self.sharded: + return nullcontext() + return torch.autocast(self.device_type, dtype=self.autocast_dtype) diff --git a/src/sequifier/runtime/random_state.py b/src/sequifier/runtime/random_state.py index 7a97128cd..5827d4198 100644 --- a/src/sequifier/runtime/random_state.py +++ b/src/sequifier/runtime/random_state.py @@ -19,6 +19,7 @@ class RandomState: torch_cpu: torch.Tensor cuda: tuple[torch.Tensor, ...] | None mps: torch.Tensor | None = None + all_devices: bool = False def state_dict(self) -> dict[str, Any]: return { @@ -27,6 +28,7 @@ def state_dict(self) -> dict[str, Any]: "torch_cpu": self.torch_cpu, "cuda": self.cuda, "mps": self.mps, + "all_devices": self.all_devices, } @classmethod @@ -38,21 +40,25 @@ class RandomStateManager: def __init__(self, device: torch.device) -> None: self.device = device - def capture_local(self) -> RandomState: + def capture_local(self, *, all_devices: bool = False) -> RandomState: return RandomState( python=random.getstate(), numpy=np.random.get_state(), torch_cpu=torch.get_rng_state(), cuda=( - (torch.cuda.get_rng_state(self.device),) + tuple(torch.cuda.get_rng_state_all()) + if all_devices and torch.cuda.is_available() + else (torch.cuda.get_rng_state(self.device),) if self.device.type == "cuda" and torch.cuda.is_available() else None ), mps=( torch.mps.get_rng_state() - if self.device.type == "mps" and torch.backends.mps.is_available() + if (all_devices or self.device.type == "mps") + and torch.backends.mps.is_available() else None ), + all_devices=all_devices, ) def gather(self, distributed: Any) -> DistributedRandomState: @@ -69,10 +75,20 @@ def select_for_rank(self, state: DistributedRandomState, rank: int) -> RandomSta return RandomState.from_state_dict(state.states[rank]) def restore(self, state: RandomState) -> None: + if state.all_devices: + if state.cuda is not None and ( + not torch.cuda.is_available() + or len(state.cuda) != torch.cuda.device_count() + ): + raise ValueError("checkpoint CUDA RNG topology differs") + if state.mps is not None and not torch.backends.mps.is_available(): + raise ValueError("checkpoint requires the saved MPS RNG") random.setstate(state.python) np.random.set_state(state.numpy) torch.set_rng_state(state.torch_cpu) - if ( + if state.all_devices and state.cuda is not None: + torch.cuda.set_rng_state_all(list(state.cuda)) + elif ( state.cuda is not None and self.device.type == "cuda" and torch.cuda.is_available() @@ -86,7 +102,15 @@ def restore(self, state: RandomState) -> None: torch.cuda.set_rng_state(cuda_state, self.device) if ( state.mps is not None - and self.device.type == "mps" + and (state.all_devices or self.device.type == "mps") and torch.backends.mps.is_available() ): torch.mps.set_rng_state(state.mps) + + +def capture_rng_state(device): + return RandomStateManager(device).capture_local(all_devices=True).state_dict() + + +def restore_rng_state(device, state): + RandomStateManager(device).restore(RandomState.from_state_dict(state)) diff --git a/src/sequifier/runtime/resource_accounting.py b/src/sequifier/runtime/resource_accounting.py new file mode 100644 index 000000000..ca3cd9f18 --- /dev/null +++ b/src/sequifier/runtime/resource_accounting.py @@ -0,0 +1,72 @@ +"""Run-local estimates for policy data and working payloads.""" + +from __future__ import annotations + +from collections import Counter +from contextlib import contextmanager +from contextvars import ContextVar + +_active: ContextVar[ResourceAccounting | None] = ContextVar( + "policy_resource_accounting", default=None +) + + +class ResourceAccounting: + def __init__(self): + self.current: Counter[str] = Counter() + self.peaks: Counter[str] = Counter() + self.total = 0 + self.peak_total = 0 + + @contextmanager + def activate(self): + token = _active.set(self) + try: + yield + finally: + _active.reset(token) + + def allocation(self, category): + return Allocation(self, category) + + def snapshot(self): + return { + "current_bytes": {key: self.current[key] for key in sorted(self.peaks)}, + "peak_bytes": dict(sorted(self.peaks.items())), + "current_total_bytes": self.total, + "peak_total_bytes": self.peak_total, + } + + +class Allocation: + def __init__(self, accounting, category): + self.accounting = accounting + self.category = category + self.size = 0 + + def resize(self, size): + if self.accounting is None or size == self.size: + return + accounting, category = self.accounting, self.category + delta = size - self.size + self.size = size + accounting.current[category] += delta + accounting.total += delta + accounting.peaks[category] = max( + accounting.peaks[category], accounting.current[category] + ) + accounting.peak_total = max(accounting.peak_total, accounting.total) + + def close(self): + self.resize(0) + self.accounting = None + + def __enter__(self): + return self + + def __exit__(self, *_): + self.close() + + +def allocation(category): + return Allocation(_active.get(), category) diff --git a/src/sequifier/runtime/stopping.py b/src/sequifier/runtime/stopping.py new file mode 100644 index 000000000..b6701a961 --- /dev/null +++ b/src/sequifier/runtime/stopping.py @@ -0,0 +1,28 @@ +"""Serializable observation-based patience shared by native learning runners.""" + +import math +from dataclasses import dataclass + +from sequifier.runtime.plan import metric_improved + + +@dataclass +class StoppingState: + best: float | None = None + without_improvement: int = 0 + last_observation: int | None = None + triggered: bool = False + + def observe(self, config, value, direction, observation): + if config is None or value is None or not math.isfinite(value): + return self.triggered + if self.last_observation is not None and observation <= self.last_observation: + return self.triggered + self.last_observation = observation + if metric_improved(value, self.best, direction, config.min_delta): + self.best = float(value) + self.without_improvement = 0 + else: + self.without_improvement += 1 + self.triggered = self.without_improvement >= config.patience + return self.triggered diff --git a/src/sequifier/sequifier.py b/src/sequifier/sequifier.py index e7ae92041..4cb2816d3 100644 --- a/src/sequifier/sequifier.py +++ b/src/sequifier/sequifier.py @@ -1,7 +1,7 @@ -from argparse import ArgumentParser +from argparse import ArgumentParser, ArgumentTypeError from typing import Any -import numpy as np +import yaml from sequifier.docs import docs from sequifier.hyperparameter_search import hyperparameter_search @@ -13,6 +13,29 @@ from sequifier.visualize_training import visualize_training +class ConfigArgumentParser(ArgumentParser): + """Parse only explicitly declared command options.""" + + def __init__(self, *args: Any, **kwargs: Any): + kwargs.setdefault("allow_abbrev", False) + super().__init__(*args, **kwargs) + + +def parse_seed(value: str) -> int | str: + if value == "random": + return value + try: + return int(value) + except ValueError as error: + raise ArgumentTypeError("seed must be an integer or random") from error + + +def parse_boolean(value: str) -> bool: + if value not in {"true", "false"}: + raise ArgumentTypeError("boolean overrides must be true or false") + return value == "true" + + @beartype def build_args_config(args: Any) -> dict[str, Any]: """Build config overrides from parsed CLI args.""" @@ -20,15 +43,8 @@ def build_args_config(args: Any) -> dict[str, Any]: args_config = { k: v for k, v in vars(args).items() - if v is not None and k not in ["randomize", "command", "config_path"] + if v is not None and k not in ["command", "config_path"] } - if args.command not in ["make", "visualize-training"]: - if args.randomize: - seed = np.random.choice(np.arange(int(1e9))) - args_config["seed"] = seed - elif args.command in ["train", "infer"] and args.seed: - args_config["seed"] = args.seed - if "selected_columns" in args_config: if ( len(args_config["selected_columns"]) == 1 @@ -49,7 +65,7 @@ def build_args_config(args: Any) -> dict[str, Any]: @beartype def setup_parser() -> ArgumentParser: """Build the sequifier CLI parser.""" - parser = ArgumentParser() + parser = ConfigArgumentParser() subparsers = parser.add_subparsers(dest="command", help="Available commands") parser_make = subparsers.add_parser("make", help="Set up sequifier project") @@ -58,15 +74,17 @@ def setup_parser() -> ArgumentParser: parser_docs.add_argument( "topic", nargs="?", - choices=("train", "preprocess", "infer"), + choices=("train", "preprocess", "infer", "policy"), help="Show documentation for one command (default: all documentation)", ) parser_preprocess = subparsers.add_parser( "preprocess", help="Run the preprocessing step" ) - parser_train = subparsers.add_parser("train", help="Run the training step") + parser_train = subparsers.add_parser( + "train", + help="Train supervised models or ordered imitation/IQL/PPO/DPO/GRPO policy phases", + ) parser_infer = subparsers.add_parser("infer", help="Run the inference step") - parser_hyperparameter_search = subparsers.add_parser( "hyperparameter-search", help="Run hyperparamter search" ) @@ -87,39 +105,43 @@ def setup_parser() -> ArgumentParser: ) for subparser in [parser_preprocess, parser_train, parser_infer]: - subparser.add_argument("-r", "--randomize", action="store_true") + subparser.add_argument( + "--seed", type=parse_seed, help="Integer run seed, or random" + ) parser_preprocess.add_argument( - "-dp", - "--data-path", - dest="preprocessing_data_path", + "--preprocessing-data-path", type=str, ) - parser_infer.add_argument("-dp", "--data-path", type=str) + parser_infer.add_argument("--data-path", type=str) parser_infer.add_argument( - "-ic", "--input-columns", type=str, nargs="+", help="Space-separated list of input columns (e.g. --input-columns col1 col2)", ) - parser_infer.add_argument("-mc", "--metadata-config-path", type=str) + parser_infer.add_argument("--metadata-config-path", type=str) for subparser in [parser_train, parser_infer, parser_hyperparameter_search]: - subparser.add_argument("-sm", "--skip-metadata", action="store_true") + subparser.add_argument("--skip-metadata", action="store_true") parser_preprocess.add_argument( - "-sc", "--selected-columns", type=str, nargs="+", help="Space-separated list of selected columns", ) - parser_train.add_argument("-mn", "--model-name", type=str) - parser_train.add_argument("-s", "--seed", type=int) + parser_train.add_argument("--model-name", type=str) + for subparser in (parser_train, parser_infer, parser_preprocess): + subparser.add_argument("--run", type=yaml.safe_load) + parser_train.add_argument("--training-plan", type=yaml.safe_load) - parser_infer.add_argument("-mp", "--model-path", type=str) - parser_infer.add_argument("-s", "--seed", type=int) + parser_infer.add_argument("--model-paths", type=str, nargs="+") + parser_infer.add_argument("--enforce-determinism", type=parse_boolean, default=None) + parser_infer.add_argument( + "--action-selection", choices=("deterministic", "stochastic") + ) + parser_infer.add_argument("--task", choices=("forecast", "policy", "embedding")) parser_infer.add_argument("--dataset", type=str) parser_infer.add_argument("--part", type=str) parser_infer.add_argument("--model-interface", type=str) diff --git a/src/sequifier/templates/infer-policy.yaml b/src/sequifier/templates/infer-policy.yaml new file mode 100644 index 000000000..443ce1e31 --- /dev/null +++ b/src/sequifier/templates/infer-policy.yaml @@ -0,0 +1,14 @@ +data_path: PLEASE_FILL +read_format: csv +write_format: csv +desired_return: 10.0 +action_selection: deterministic +task: policy +model_paths: +- models/policy.pt +run: + project_root: . + execution: + device: cpu + reproducibility: + enforce_determinism: true diff --git a/src/sequifier/templates/train-dpo.yaml b/src/sequifier/templates/train-dpo.yaml new file mode 100644 index 000000000..789acc50c --- /dev/null +++ b/src/sequifier/templates/train-dpo.yaml @@ -0,0 +1,97 @@ +model: + backbone: + architecture: + model_dim: 32 + num_layers: 1 + max_context_length: 4 + attention: + num_heads: 4 + feed_forward: + hidden_dim: 64 + position_encoding_scope: global + dropout: 0.0 + interfaces: + actor: + input_columns: + - cue + - __policy_previous_action + target_columns: + - action + ingestion: + type: embedding + output_dim: 8 + feature_embedding_dims: + cue: 2 + __policy_previous_action: 6 + decoder: + type: linear + prediction_length: 1 + support_length: 1 + categorical_decoder_special_tokens: + action: [] + input_contract: + normalization: + source: identity + execution: + context_length: 4 +dataset_training: + decisions: + parts: + default: + type: sequence + paths: + - data/control-decisions.csv + model_interface: actor + split: + validation_fraction: 0.2 + rankings: + parts: + default: + type: preferences + manifest_path: data/control-preferences.json + split: + validation_fraction: 0.2 +training_plan: + phases: + - name: preferences + algorithm: dpo + preferences: + source: rankings.default + beta: 0.1 + iterations: 100 + model_selection: + handoff: last + checkpoints: + cadence: + iterations: 1 + evaluation: + cadence: + iterations: 1 + batch_size: 1 + before_phase: false + on_snapshot: false +policy_bindings: + actor: + previous_action_inputs: + __policy_previous_action: action + initial_inputs: + __policy_previous_action: start + return_conditioned: false + actions: + action: + type: categorical + values: + - left + - right +run: + project_root: . + model_name: dpo-control + reproducibility: + seed: 1010 + enforce_determinism: false + execution: + device: cpu + recovery: {} + logging: {} + outputs: {} + export: {} diff --git a/src/sequifier/templates/train-grpo.yaml b/src/sequifier/templates/train-grpo.yaml new file mode 100644 index 000000000..dc3d51515 --- /dev/null +++ b/src/sequifier/templates/train-grpo.yaml @@ -0,0 +1,98 @@ +model: + backbone: + architecture: + model_dim: 32 + num_layers: 1 + max_context_length: 4 + attention: + num_heads: 4 + feed_forward: + hidden_dim: 64 + position_encoding_scope: global + dropout: 0.0 + interfaces: + actor: + input_columns: + - cue + - __policy_previous_action + target_columns: + - action + ingestion: + type: embedding + output_dim: 8 + feature_embedding_dims: + cue: 2 + __policy_previous_action: 6 + decoder: + type: linear + prediction_length: 1 + support_length: 1 + categorical_decoder_special_tokens: + action: [] + input_contract: + normalization: + source: identity + execution: + context_length: 4 +training_plan: + phases: + - name: group_improvement + algorithm: grpo + total_groups: 32 + groups_per_iteration: 2 + group_size: 4 + max_steps: 8 + environment: + factory: sequifier.rl.control:CategoricalControlAdapter + config: + horizon: 8 + evaluation: + environment: + factory: sequifier.rl.control:CategoricalControlAdapter + config: + horizon: 8 + seeds: + - 17 + - 29 + - 43 + - 71 + max_episode_steps: 8 + timeout_seconds: 30 + monitor: + metric: score_mean + direction: max + cadence: + iterations: 1 + batch_size: 1 + before_phase: false + on_snapshot: false + model_selection: + handoff: last + checkpoints: + cadence: + iterations: 1 +policy_bindings: + actor: + previous_action_inputs: + __policy_previous_action: action + initial_inputs: + __policy_previous_action: start + return_conditioned: false + actions: + action: + type: categorical + values: + - left + - right +run: + project_root: . + model_name: grpo-control + reproducibility: + seed: 1010 + enforce_determinism: false + execution: + device: cpu + recovery: {} + logging: {} + outputs: {} + export: {} diff --git a/src/sequifier/templates/train-iql.yaml b/src/sequifier/templates/train-iql.yaml new file mode 100644 index 000000000..68cae4946 --- /dev/null +++ b/src/sequifier/templates/train-iql.yaml @@ -0,0 +1,88 @@ +model: + backbone: + architecture: + model_dim: 32 + num_layers: 2 + max_context_length: 16 + attention: + num_heads: 4 + feed_forward: + hidden_dim: 128 + position_encoding_scope: global + interfaces: + actor: + input_columns: + - OBSERVATION_COLUMN + - __policy_previous_action + target_columns: + - action + ingestion: + type: embedding + output_dim: 8 + feature_embedding_dims: + OBSERVATION_COLUMN: 2 + __policy_previous_action: 6 + decoder: + type: linear + prediction_length: 1 + support_length: 1 + categorical_decoder_special_tokens: + action: [] + execution: + context_length: 16 +dataset_training: + demonstrations: + parts: + recorded: + type: sequence + paths: + - PLEASE_FILL.csv + model_interface: actor + split: + validation_fraction: 0.2 +training_plan: + phases: + - name: offline_improvement + algorithm: iql + sources: + - source: demonstrations.recorded + batch_size: 64 + iterations: 10000 + model_selection: + handoff: last + checkpoints: + cadence: + iterations: 100 + evaluation: + cadence: + iterations: 100 + batch_size: 64 + before_phase: false + on_snapshot: false +policy_bindings: + actor: + previous_action_inputs: + __policy_previous_action: action + initial_inputs: + __policy_previous_action: start + return_conditioned: false + actions: + action: + type: categorical + values: + - 0.0 + - 1.0 +run: + project_root: . + model_name: offline-iql + reproducibility: + seed: 1010 + enforce_determinism: false + execution: + device: cpu + recovery: + resume: + policy: if_exists + logging: {} + outputs: {} + export: {} diff --git a/src/sequifier/templates/train-policy-from-supervised.yaml b/src/sequifier/templates/train-policy-from-supervised.yaml new file mode 100644 index 000000000..afd853bce --- /dev/null +++ b/src/sequifier/templates/train-policy-from-supervised.yaml @@ -0,0 +1,56 @@ +initialize_from: models/supervised.pt +dataset_training: + demonstrations: + parts: + default: + type: sequence + paths: + - data/demonstrations.parquet + format: table + split: + validation_fraction: 0.2 +training_plan: + phases: + - name: imitation + algorithm: policy_imitation + sources: + - source: demonstrations.default + epochs: 5 + model_selection: + handoff: last + checkpoints: + cadence: + iterations: 1 + evaluation: + cadence: + iterations: 1 + batch_size: 32 + before_phase: false + on_snapshot: false +policy_bindings: + default: + actions: + mode: + type: categorical + values: + - 'off' + - 'on' + quantity: + type: real + scale: + fixed: 0.5 +run: + project_root: . + model_name: posttrained + reproducibility: + seed: 1010 + enforce_determinism: false + execution: + device: cpu + recovery: + resume: + policy: if_exists + mode: reset_environments + logging: {} + outputs: {} + export: {} diff --git a/src/sequifier/templates/train-policy.yaml b/src/sequifier/templates/train-policy.yaml new file mode 100644 index 000000000..3e1ebe1d2 --- /dev/null +++ b/src/sequifier/templates/train-policy.yaml @@ -0,0 +1,97 @@ +model: + backbone: + architecture: + model_dim: 32 + num_layers: 2 + max_context_length: 4 + attention: + num_heads: 4 + feed_forward: + activation: swiglu + hidden_dim: 128 + position_encoding_scope: global + dropout: 0.0 + interfaces: + actor: + input_columns: + - OBSERVATION_COLUMN + - __policy_previous_action + - __policy_return + target_columns: + - action + ingestion: + type: embedding + output_dim: 8 + feature_embedding_dims: + OBSERVATION_COLUMN: 1 + __policy_previous_action: 6 + __policy_return: 1 + decoder: + type: linear + prediction_length: 1 + support_length: 1 + categorical_decoder_special_tokens: + action: [] + execution: + context_length: 4 +dataset_training: + demonstrations: + parts: + recorded: + type: sequence + paths: + - PLEASE_FILL.csv + model_interface: actor + split: + validation_fraction: 0.2 +training_plan: + phases: + - algorithm: policy_imitation + epochs: 80 + sources: + - source: demonstrations.recorded + name: policy_imitation + batch_size: 64 + optimization: + actor: + learning_rate: 0.005 + model_selection: + handoff: last + checkpoints: + cadence: + iterations: 1 + evaluation: + cadence: + iterations: 1 + batch_size: 64 + before_phase: false + on_snapshot: false +policy_bindings: + actor: + previous_action_inputs: + __policy_previous_action: action + initial_inputs: + __policy_previous_action: start + return_conditioned: true + actions: + action: + type: categorical + values: + - 0.0 + - 1.0 +run: + project_root: . + model_name: policy + reproducibility: + seed: 1010 + enforce_determinism: false + execution: + device: cpu + data_loading: + episode_cache_size: 8 + recovery: + resume: + policy: never + logging: {} + outputs: {} + export: {} diff --git a/src/sequifier/templates/train-ppo.yaml b/src/sequifier/templates/train-ppo.yaml new file mode 100644 index 000000000..5116e4058 --- /dev/null +++ b/src/sequifier/templates/train-ppo.yaml @@ -0,0 +1,102 @@ +model: + backbone: + architecture: + model_dim: 32 + num_layers: 1 + max_context_length: 4 + attention: + num_heads: 4 + feed_forward: + hidden_dim: 64 + position_encoding_scope: global + dropout: 0.0 + interfaces: + actor: + input_columns: + - cue + - __policy_previous_action + target_columns: + - action + ingestion: + type: embedding + output_dim: 8 + feature_embedding_dims: + cue: 2 + __policy_previous_action: 6 + decoder: + type: linear + prediction_length: 1 + support_length: 1 + categorical_decoder_special_tokens: + action: [] + input_contract: + normalization: + source: identity + execution: + context_length: 4 +training_plan: + phases: + - name: online_control + algorithm: ppo + environment: + factory: sequifier.rl.control:CategoricalControlAdapter + config: + horizon: 8 + total_env_steps: 8192 + rollout_env_steps: 256 + optimization_epochs: 4 + accumulation_batches: 1 + gamma: 0.99 + gae_lambda: 0.95 + clip_epsilon: 0.2 + evaluation: + environment: + factory: sequifier.rl.control:CategoricalControlAdapter + config: + horizon: 8 + seeds: + - 17 + - 29 + - 43 + - 71 + max_episode_steps: 8 + timeout_seconds: 30 + cadence: + iterations: 2 + batch_size: 1 + before_phase: false + on_snapshot: false + batch_size: 64 + model_selection: + handoff: last + checkpoints: + cadence: + iterations: 2 +policy_bindings: + actor: + previous_action_inputs: + __policy_previous_action: action + initial_inputs: + __policy_previous_action: start + return_conditioned: false + actions: + action: + type: categorical + values: + - left + - right +run: + project_root: . + model_name: online-ppo + reproducibility: + seed: 1010 + enforce_determinism: false + execution: + device: cpu + recovery: + resume: + policy: if_exists + mode: exact + logging: {} + outputs: {} + export: {} diff --git a/src/sequifier/templates/train-supervised-to-policy.yaml b/src/sequifier/templates/train-supervised-to-policy.yaml new file mode 100644 index 000000000..ee0ab7c63 --- /dev/null +++ b/src/sequifier/templates/train-supervised-to-policy.yaml @@ -0,0 +1,137 @@ +model: + backbone: + architecture: + model_dim: 32 + num_layers: 2 + max_context_length: 4 + attention: + num_heads: 4 + feed_forward: + activation: swiglu + hidden_dim: 128 + position_encoding_scope: global + dropout: 0.0 + interfaces: + actor: + input_columns: + - OBSERVATION_COLUMN + - __policy_previous_action + - __policy_return + target_columns: + - action + ingestion: + type: embedding + output_dim: 8 + feature_embedding_dims: + OBSERVATION_COLUMN: 1 + __policy_previous_action: 6 + __policy_return: 1 + decoder: + type: linear + prediction_length: 1 + support_length: 1 + categorical_decoder_special_tokens: + action: [] + execution: + context_length: 4 + training_objective: causal +dataset_training: + demonstrations: + parts: + recorded: + type: sequence + paths: + - PLEASE_FILL.csv + model_interface: actor + split: + validation_fraction: 0.2 + observations: + model_interface: actor + parts: + train: + metadata_config_path: configs/metadata_configs/observations.json + read_format: parquet + split: + validation_fraction: 0.2 +training_plan: + phases: + - name: supervised + algorithm: supervised + epochs: 10 + mode: sequential + sources: + - source: observations.train + batch_size: 64 + accumulation_batches: 1 + optimization: + model: + optimizer: + name: Adam + arguments: {} + learning_rate: 0.001 + scheduler: + name: StepLR + arguments: + step_size: 1 + gamma: 0.99 + scheduler_step_on: epoch + gradient_clip_norm: 1.0 + model_selection: + handoff: last + checkpoints: + cadence: + epochs: 1 + learning: + datasets: + observations: + criterion: + action: CrossEntropyLoss + - algorithm: policy_imitation + epochs: 80 + sources: + - source: demonstrations.recorded + name: policy_imitation + batch_size: 64 + optimization: + actor: + learning_rate: 0.005 + model_selection: + handoff: last + checkpoints: + cadence: + iterations: 1 + evaluation: + cadence: + iterations: 1 + batch_size: 64 + before_phase: false + on_snapshot: false +policy_bindings: + actor: + previous_action_inputs: + __policy_previous_action: action + initial_inputs: + __policy_previous_action: start + return_conditioned: true + actions: + action: + type: categorical + values: + - 0.0 + - 1.0 +run: + project_root: . + model_name: supervised-to-policy + reproducibility: + seed: 1010 + enforce_determinism: false + execution: + device: cpu + data_loading: + episode_cache_size: 8 + recovery: + resume: + policy: never + logging: {} + outputs: {} + export: {} diff --git a/src/sequifier/templates/train-supervised.yaml b/src/sequifier/templates/train-supervised.yaml new file mode 100644 index 000000000..4778a3088 --- /dev/null +++ b/src/sequifier/templates/train-supervised.yaml @@ -0,0 +1,98 @@ +model: + backbone: + architecture: + model_dim: 128 + num_layers: 3 + attention: + num_heads: 16 + feed_forward: + hidden_dim: 128 + dropout: 0.2 + interfaces: + default: + input_columns: + - EXAMPLE_INPUT_COLUMN_NAME + target_columns: + - EXAMPLE_TARGET_COLUMN_NAME + ingestion: + type: embedding + output_dim: 128 + decoder: + type: linear + prediction_length: 1 + execution: + context_length: 48 + training_objective: causal +training_plan: + phases: + - name: supervised + algorithm: supervised + epochs: 10 + mode: sequential + sources: + - source: default.default + batch_size: 10 + accumulation_batches: 1 + optimization: + model: + learning_rate: 0.0001 + optimizer: + name: AdamW + arguments: {} + scheduler: + name: StepLR + arguments: + step_size: 1 + gamma: 0.99 + scheduler_step_on: epoch + evaluation: + sources: + - source: default.default + cadence: + epochs: 1 + batch_size: 10 + before_phase: true + on_snapshot: true + model_selection: + handoff: last + checkpoints: + cadence: + epochs: 1 + learning: + datasets: + default: + criterion: + EXAMPLE_TARGET_COLUMN_NAME: MSELoss +dataset_training: + default: + parts: + default: + metadata_config_path: PLEASE FILL + read_format: parquet + split: + validation_fraction: 0.2 +run: + project_root: . + model_name: supervised + reproducibility: + seed: 1010 + enforce_determinism: false + execution: + device: cpu + distribution: + strategy: local + world_size: 1 + backend: nccl + precision: {} + data_loading: {} + compilation: + torch_compile: outer + recovery: {} + logging: {} + outputs: {} + export: + generative_model: true + embedding_model: false + onnx: true + pt: false + batch_size: 10 diff --git a/src/sequifier/train.py b/src/sequifier/train.py index 033f06286..f31bbf5eb 100644 --- a/src/sequifier/train.py +++ b/src/sequifier/train.py @@ -16,8 +16,8 @@ from torch import Tensor, nn # noqa: E402 from sequifier.artifacts.model_artifact import ( # noqa: E402 - load_model_artifact, - load_weights_from_run_checkpoint, + ModelArtifact, + load_execution_model, ) from sequifier.config.train_config import ( # noqa: E402 ResolvedSequifierConfig as TrainModel, @@ -34,6 +34,7 @@ from sequifier.model.tracing import CaptureRequest # noqa: E402 from sequifier.runtime.builder import RunBuilder # noqa: E402 from sequifier.runtime.context import ExecutionEnvironment # noqa: E402 +from sequifier.training.coordinator import coordinate_training # noqa: E402 from sequifier.training.engine import TrainingEngine # noqa: E402 from sequifier.typechecking import beartype # noqa: E402 @@ -46,14 +47,11 @@ class LoadedInferenceModel: config: Any interface_name: str embedding: bool + autoregressive_targets: frozenset[str] = frozenset() @property def interface(self) -> Any: - return next( - dataset.interface - for dataset in self.config.dataset_training.values() - if dataset.model_interface == self.interface_name - ) + return self.config.interface(self.interface_name) @property def categorical_columns(self) -> list[str]: @@ -87,47 +85,59 @@ def train_worker( ) -> None: """Initialize one execution environment and run the composed runtime.""" + from sequifier.artifacts.dependencies import supervised_dependencies + from sequifier.artifacts.run_checkpoint import checkpoint_path + from sequifier.io.output_paths import validate_training_destinations + + dependencies = supervised_dependencies(config) + validate_training_destinations( + config, checkpoint_path(config), dependencies.protected_paths + ) configure_logger( config.project_root, config.model_name, global_rank, dataset_names=tuple(config.dataset_training), - rank_specific=config.global_training.distributed, + rank_specific=config.runtime_settings.distributed, ) initialized_distributed = False - if config.global_training.distributed: + if config.runtime_settings.distributed: if config.device.startswith("cuda"): torch.cuda.set_device(local_rank) setup_distributed_env( global_rank, local_rank, world_size, - config.global_training.backend, + config.runtime_settings.backend, ) initialized_distributed = True - configure_determinism(config.seed, config.global_training.enforce_determinism) + configure_determinism(config.seed, config.runtime_settings.enforce_determinism) execution = ExecutionEnvironment( rank=global_rank, local_rank=local_rank, world_size=world_size, device=_execution_device(config, local_rank), - distributed=config.global_training.distributed, + distributed=config.runtime_settings.distributed, ) integrations = IntegrationManager( specs=integration_specs, instances=integration_instances, rank=global_rank, world_size=world_size, - distributed=config.global_training.distributed, + distributed=config.runtime_settings.distributed, ) integrations.validate_execution( - torch_compile=config.global_training.torch_compile, - data_parallelism=config.global_training.data_parallelism, + torch_compile=config.runtime_settings.torch_compile, + data_parallelism=config.runtime_settings.data_parallelism, ) run = None try: run = RunBuilder(semantic_optimizer_grouping=semantic_optimizer_grouping).build( - config, execution, integrations + config, + execution, + integrations, + initial_artifact=config.initialize_from, + dependencies=dependencies, ) loguru_logger.info( f"--- Starting Training for model: {run.context.model_name} | " @@ -167,12 +177,27 @@ def run_training( integration_specs: tuple[IntegrationSpec, ...] = (), integration_instances: tuple[Any, ...] = (), semantic_optimizer_grouping: bool = False, -) -> None: +) -> Any: """Launch canonical service-composed training locally or across workers.""" - if not isinstance(config, TrainModel): - raise TypeError("Training requires a canonical resolved config.") - spec = config.global_training + return coordinate_training( + config, + launch_supervised=launch_supervised, + integration_specs=integration_specs, + integration_instances=integration_instances, + semantic_optimizer_grouping=semantic_optimizer_grouping, + ) + + +def launch_supervised( + config: TrainModel, + *, + integration_specs: tuple[IntegrationSpec, ...], + integration_instances: tuple[Any, ...], + semantic_optimizer_grouping: bool, +) -> None: + """Launch the native supervised runtime in its execution environment.""" + spec = config.runtime_settings if spec.distributed and integration_instances: raise ValueError( "Distributed runs require IntegrationSpec; direct instances cannot " @@ -227,44 +252,23 @@ def load_inference_model( args_config: dict[str, Any], device: str, infer_with_dropout: bool, + artifact: ModelArtifact | None = None, ) -> LoadedInferenceModel: """Load only the new portable-model or exact-run artifact formats.""" del training_config_path - payload = torch.load(model_path, map_location="cpu", weights_only=False) - artifact_type = payload.get("artifact_type") if isinstance(payload, dict) else None interface_name = args_config.get("model_interface") or args_config.get("dataset") - if artifact_type == "sequifier_model": - network, config, _ = load_model_artifact( - model_path, device=device, interface_name=interface_name - ) - elif artifact_type == "sequifier_run_checkpoint": - network, config, _ = load_weights_from_run_checkpoint( - model_path, device=device, interface_name=interface_name - ) - else: - raise ValueError( - "Unsupported PyTorch artifact. Sequifier accepts only the current " - "portable model or run-checkpoint format." - ) + network, config, _ = load_execution_model( + artifact if artifact is not None else model_path, + device=device, + interface_name=interface_name, + ) if interface_name is None: if len(network.interfaces) != 1: raise ValueError( "model_interface is required for a multi-interface artifact." ) interface_name = next(iter(network.interfaces)) - if interface_name not in network.interfaces: - mapped = next( - ( - dataset.model_interface - for name, dataset in config.dataset_training.items() - if name == interface_name - ), - None, - ) - if mapped is None: - raise ValueError(f"Unknown model interface {interface_name!r}.") - interface_name = mapped network.eval() if infer_with_dropout: for module in network.modules(): @@ -272,6 +276,19 @@ def load_inference_model( if isinstance(module, (nn.Dropout, nn.MultiheadAttention, SelfAttention)): module.train() + from sequifier.model.decoders import AutoregressiveTransformerDecoderBranch + + route = network.resolve_interface(interface_name) + autoregressive_targets = ( + frozenset( + target + for branch in route.decoder.branches.values() + if isinstance(branch, AutoregressiveTransformerDecoderBranch) + for target in branch.target_columns + ) + if route.decoder is not None + else frozenset() + ) if not device.startswith("mps"): network = torch.compile(network) if model_type not in {"generative", "embedding"}: @@ -281,6 +298,7 @@ def load_inference_model( config=config, interface_name=interface_name, embedding=model_type == "embedding", + autoregressive_targets=autoregressive_targets, ) @@ -292,7 +310,7 @@ def _tensor_batches( column_data_types: dict[str, torch.dtype], ): categorical = set(model.categorical_columns) - layer_types = model.config.global_training.layer_type_dtypes or {} + layer_types = model.config.options.layer_type_dtypes or {} reference_dtype = get_torch_dtype(layer_types.get("linear", "float32")) for index, features in enumerate(x): values = { diff --git a/src/sequifier/training/checkpoint_service.py b/src/sequifier/training/checkpoint_service.py index ef706314a..67705f34f 100644 --- a/src/sequifier/training/checkpoint_service.py +++ b/src/sequifier/training/checkpoint_service.py @@ -6,15 +6,16 @@ from pathlib import Path from typing import Any, Mapping +from sequifier.artifacts.dependencies import DependencyManifest from sequifier.artifacts.model_artifact import build_model_artifact from sequifier.artifacts.model_export import model_execution_config from sequifier.artifacts.run_checkpoint import ( - RUN_CHECKPOINT_FORMAT_VERSION, IntegrationState, RunCheckpoint, RunCheckpointStore, load_run_checkpoint, ) +from sequifier.contract_versions import ContractVersions from sequifier.integration.contexts import CheckpointSaved, CheckpointSaving from sequifier.training.loader_state import LoaderStateService @@ -38,9 +39,13 @@ def validate(self, checkpoint: RunCheckpoint) -> None: checkpoint.validate() def validate_for_restore( - self, checkpoint: RunCheckpoint, current_config: Any + self, + checkpoint: RunCheckpoint, + current_config: Any, + dependencies: DependencyManifest, ) -> None: self.validate(checkpoint) + dependencies.validate(checkpoint.dependencies) saved = self._resume_settings( checkpoint.training_config, model_config=checkpoint.model.model_config.values, @@ -63,8 +68,6 @@ def _resume_settings( execution = copy.deepcopy(dict(model_config or model_execution_config(config))) execution.pop("embedding_layer_names", None) - # Initialization is strict run provenance, compared from resolved training - # configs rather than older portable artifacts that omitted it. def strip_provenance(value: Any) -> Any: if isinstance(value, dict): return { @@ -84,23 +87,29 @@ def strip_provenance(value: Any) -> Any: execution = strip_provenance(execution) for interface in execution.get("interfaces", {}).values(): - # Paths locate data for inference; an exact resume may use a new - # project directory containing the same splits. + # The loader paths are compared with their owning dataset parts. interface.pop("split_paths", None) - interface.setdefault("depth_layouts", {}) - interface.setdefault("tensor_payload_version", 1) - training = config.global_training + training = config.runtime_settings datasets = { name: { "model_interface": dataset.model_interface, "parts": list(dataset.parts), - "criterion": dataset.criterion, - "class_weights": dataset.class_weights, - "loss_weights": dataset.loss_weights, - "freeze": dataset.freeze.model_dump(mode="python"), + "part_inputs": { + name: part.model_dump(mode="python", exclude={"metadata"}) + for name, part in dataset.parts.items() + }, } for name, dataset in config.dataset_training.items() } + phases = [phase.model_dump(mode="python") for phase in config.training_plan] + for phase in phases: + phase.pop("trainability_summary", None) + for source in phase["sources"] + phase["evaluation_sources"]: + source.pop("parts", None) + for role in phase["optimization"].values(): + if role["scheduler"]["name"] == "OneCycleLR": + # Recomputed from the phase/data budget before optimizer construction. + role["scheduler"]["arguments"].pop("total_steps", None) return { "model": execution, "initialization_provenance": { @@ -112,17 +121,14 @@ def strip_provenance(value: Any) -> Any: for name, interface in config.model.interfaces.items() }, }, + "owned_execution": config.ownership.get("run", {}).get("execution", {}), + "normalization_by_interface": { + name: value.model_dump(mode="python") + for name, value in config.normalization_by_interface.items() + }, + "source_validation_fractions": config.source_validation_fractions, "resume_training": { "seed": config.seed, - "read_format": training.read_format, - "batch_size": training.batch_size, - "accumulation_steps": training.accumulation_steps, - "learning_rate": training.learning_rate, - "optimizer": training.optimizer.model_dump(mode="python"), - "scheduler": training.scheduler.model_dump(mode="python"), - "scheduler_step_on": training.scheduler_step_on, - "reset_optimization_on_phase": (training.reset_optimization_on_phase), - "gradient_clip": training.gradient_clip, "distributed": training.distributed, "data_parallelism": training.data_parallelism, "world_size": training.world_size, @@ -130,9 +136,7 @@ def strip_provenance(value: Any) -> Any: "load_full_data_to_ram": training.load_full_data_to_ram, }, "datasets": datasets, - "training_plan": [ - phase.model_dump(mode="python") for phase in config.training_plan - ], + "training_plan": phases, } @classmethod @@ -184,7 +188,9 @@ def __init__( random_state: Any, loader_state: LoaderStateService, compatibility: CheckpointCompatibility | None = None, + on_checkpoint=None, ) -> None: + self.on_checkpoint = on_checkpoint self.store = store self.distributed = distributed self.random_state = random_state @@ -192,7 +198,9 @@ def __init__( self.compatibility = compatibility or CheckpointCompatibility() def save(self, request: CheckpointRequest, run: Any) -> Path | None: - path = self.store.path_for(request.suffix) + path = self.store.path_for( + request.suffix, run.state.phase_index, phase_offset=run.state.phase_offset + ) access = run.optimization.access(run.network) run.integrations.emit( CheckpointSaving(access=access, identity=request.identity, path=path) @@ -208,7 +216,7 @@ def save(self, request: CheckpointRequest, run: Any) -> Path | None: if self.distributed.rank != 0: return None checkpoint = RunCheckpoint( - format_version=RUN_CHECKPOINT_FORMAT_VERSION, + format_version=ContractVersions.SUPERVISED_RECOVERY, model=build_model_artifact( run.network, run.config, @@ -223,10 +231,15 @@ def save(self, request: CheckpointRequest, run: Any) -> Path | None: random_state=random_state, loader_state=loader_state, integration_state=IntegrationState(values=integration_state), - training_config=run.config, + dependencies=run.dependencies.identity, + training_config=run.config.model_copy( + update={"training_plan": [phase.spec() for phase in run.plan.phases]} + ), ) self.compatibility.validate(checkpoint) written = self.store.save(checkpoint, request.suffix) + if self.on_checkpoint is not None: + self.on_checkpoint(self.store.latest_path, run.state.phase_commits) run.integrations.emit( CheckpointSaved(access=access, identity=request.identity, path=written) ) @@ -243,9 +256,14 @@ def load(self, path: Path) -> LoadedRunCheckpoint: return LoadedRunCheckpoint(checkpoint, path) def validate_for_restore( - self, loaded: LoadedRunCheckpoint, current_config: Any + self, + loaded: LoadedRunCheckpoint, + current_config: Any, + dependencies: DependencyManifest, ) -> None: - self.compatibility.validate_for_restore(loaded.checkpoint, current_config) + self.compatibility.validate_for_restore( + loaded.checkpoint, current_config, dependencies + ) def restore_model( self, loaded: LoadedRunCheckpoint, network: Any, distributed: Any diff --git a/src/sequifier/training/coordinator.py b/src/sequifier/training/coordinator.py new file mode 100644 index 000000000..e2e0bfdd7 --- /dev/null +++ b/src/sequifier/training/coordinator.py @@ -0,0 +1,468 @@ +"""Coordinate native training and commit portable handoffs.""" + +from dataclasses import dataclass +from os import link +from pathlib import Path +from shutil import copyfile +from uuid import uuid4 + +import torch + +from sequifier.artifacts.backbone_repository import select_revision +from sequifier.artifacts.checkpoint_envelope import ( + checkpoint_envelope, + phase_boundary, + recovery_payload, + require_phase_boundary, +) +from sequifier.artifacts.dependencies import ( + DependencyManifest, + add_backbone_revision, + add_initialization, + policy_dependencies, + supervised_dependencies, +) +from sequifier.artifacts.model_artifact import ( + read_execution_artifact, + save_model_artifact, +) +from sequifier.artifacts.phase import PhaseArtifacts, PhaseCommit, verify_artifact +from sequifier.artifacts.reports import phase_report, publish_run_report, run_report +from sequifier.artifacts.run_checkpoint import checkpoint_path, training_plan_identity +from sequifier.config.components import ResumeConfig +from sequifier.contract_versions import ContractVersions, require_contract_version +from sequifier.io.atomic import atomic_path, atomic_save +from sequifier.io.output_paths import validate_training_destinations +from sequifier.runtime.plan import ( + PhaseOutcome, + RunIdentity, + continuation_groups, + new_run_id, + run_phases, +) +from sequifier.training.phase_runners import ( + PortableGroup, + policy_group_config, + run_portable_policy_group, + run_supervised_group, + supervised_group_config, +) +from sequifier.trajectory.sources import file_checksum + + +@dataclass(frozen=True) +class LearningRunResult: + model_path: str + checkpoint_path: str + report_path: str + phases: tuple[dict, ...] + + +def coordinate_training( + config, + *, + launch_supervised, + integration_specs, + integration_instances, + semantic_optimizer_grouping, +): + """Choose native execution and recovery services for the resolved plan.""" + from sequifier.config.train_config import ResolvedSequifierConfig + + if not isinstance(config, ResolvedSequifierConfig): + raise TypeError("Training requires a canonical resolved config.") + if config.publication.mode == "supervised": + return launch_supervised( + config, + integration_specs=integration_specs, + integration_instances=integration_instances, + semantic_optimizer_grouping=semantic_optimizer_grouping, + ) + if integration_specs or integration_instances or semantic_optimizer_grouping: + raise ValueError( + "policy phases do not support custom integrations or parameter grouping" + ) + return run_policy_plan(config) + + +def run_policy_plan(config): + """Use native policy recovery when the plan shares an actor representation.""" + from sequifier.config.training_resolution import policy_run_config + from sequifier.rl.learner import run_policy_training + from sequifier.rl.prepared_inputs import PreparedTrainingInputs + + if config.publication.mode == "policy": + return run_policy_training( + policy_run_config(config, config.training_plan), + prepared=PreparedTrainingInputs.for_run(config), + ) + return run_learning_plan(config) + + +def _snapshot_recovery(source, destination): + """Pin an atomic native checkpoint without depending on its rolling path.""" + with atomic_path(destination, durable=True) as temporary: + try: + link(source, temporary) + except OSError: + copyfile(source, temporary) + + +def _portable_commit(config, index, phase, checkpoint, values): + commit = PhaseCommit(**values) + if commit.result["index"] != index: + raise ValueError("native phase commit disagrees with the original phase index") + artifacts = PhaseArtifacts.for_phase(config.phase_directory, index, phase.name) + details = dict(commit.result) + for key in ( + "index", + "name", + "algorithm", + "model_path", + "checkpoint_path", + "artifact_type", + "format_version", + ): + details.pop(key, None) + details["provenance"] = {**details["provenance"], **artifacts.describe()} + if phase.algorithm == "supervised": + details["optimizer_steps"] = details["clocks"]["optimizer_steps"] + return commit.with_result( + phase_report( + index=index, + name=phase.name, + algorithm=phase.algorithm, + model_path=artifacts.model, + checkpoint_path=checkpoint, + **details, + ) + ) + + +def run_learning_plan(config) -> LearningRunResult: + from sequifier.rl.prepared_inputs import PreparedTrainingInputs + + prepared = PreparedTrainingInputs.for_run(config) + with prepared.run_scope(): + return _run_learning_plan(config, prepared) + + +def _prepare_plan(config, prepared, groups, revision, state): + from sequifier.rl.preparation import preflight_environments, validate_output_paths + + manifest = DependencyManifest() + add_initialization(manifest, config) + add_backbone_revision( + manifest, config, revision["revision_id"] if revision else None + ) + last_data_phase = -1 + supervised_runs = [] + committed = len(state["completed"]) + active = state["active"] + for index, phases in groups: + phase = phases[0] + if phase.algorithm == "supervised": + native = supervised_group_config(config, phases, index, False) + manifest.merge(supervised_dependencies(native)) + supervised_runs.append((index, native)) + continue + artifacts = PhaseArtifacts.for_phase(config.phase_directory, index, phase.name) + native = policy_group_config( + config, phases, index, None, artifacts.model, False + ) + inputs = prepared.resolve(native) + inputs.revalidate(inputs.identity, f"phases.{phase.name}.source_identity") + # Preflight can defer the active reference to the native checkpoint. + # The policy runner verifies that its weights were actually embedded. + can_recover_active = ( + active is not None + and active["index"] == index + and active["recovery"] is not None + ) + dependencies = policy_dependencies( + native, + prepared, + inputs, + saved=state.get("dependencies"), + committed_reference_count=max( + 0, committed - index + int(can_recover_active) + ), + ) + validate_output_paths(native, dependencies) + manifest.merge(dependencies) + preflight_environments(native) + for offset, data in enumerate(prepared.prepare(native, native.policy, inputs)): + if data is not None: + last_data_phase = index + offset + for index, native in supervised_runs: + validate_training_destinations( + native, + checkpoint_path(native), + manifest.protected_paths, + phase_offset=index, + ) + return manifest, last_data_phase + + +def _run_learning_plan(config, prepared): + phases = config.training_plan + groups = [(index, tuple(group)) for index, group in continuation_groups(phases)] + resume = config.runtime_settings.resume or ResumeConfig() + checkpoint = checkpoint_path(config) + plan_identity = training_plan_identity(config) + state = { + "artifact_type": "sequifier_learning_run", + "format_version": ContractVersions.LEARNING_RUN_STATE, + "plan": plan_identity, + "completed": [], + "active": None, + "run_id": new_run_id(), + } + envelope = None + resuming = resume.policy != "never" and checkpoint.is_file() + if resuming: + envelope = torch.load(checkpoint, map_location="cpu", weights_only=False) + state = recovery_payload(envelope, "learning") + require_contract_version( + "learning run coordinator state", + state.get("format_version"), + ContractVersions.LEARNING_RUN_STATE, + ) + if ( + state.get("artifact_type") != "sequifier_learning_run" + or state.get("plan") != plan_identity + ): + raise ValueError("learning run checkpoint differs from the requested plan") + elif resume.policy == "required": + raise FileNotFoundError(checkpoint) + state["seed"] = config.seed + + revision = None + if ( + phases[0].algorithm == "supervised" + and not config.initialize_from + and (not resuming or state["backbone_revision"] is not None) + ): + revision = select_revision( + config.model.backbone, + config.project_root, + revision_id=state.get("backbone_revision"), + ) + state["backbone_revision"] = revision["revision_id"] if revision else None + identity = RunIdentity.start(state["run_id"]) + completed = [PhaseCommit.load(values) for values in state["completed"]] + if len(completed) > len(phases) or any( + (commit.result["index"], commit.result["name"], commit.result["algorithm"]) + != (index, phase.name, phase.algorithm) + for index, (commit, phase) in enumerate(zip(completed, phases)) + ): + raise ValueError("learning run checkpoint differs from the phase plan") + active = state["active"] + if active is not None: + ends = {start: start + len(group) for start, group in groups} + start = active["index"] + if ( + active["run_id"] != identity.run_id + or start not in ends + or not start <= len(completed) < ends[start] + or active["phase_index"] != len(completed) + or (active["recovery"] is None and start != len(completed)) + ): + raise ValueError( + "learning run checkpoint has an invalid active phase group" + ) + manifest, last_data_phase = _prepare_plan(config, prepared, groups, revision, state) + if resuming: + manifest.validate(state["dependencies"]) + state["dependencies"] = manifest.identity + validate_training_destinations(config, checkpoint, manifest.protected_paths) + + if last_data_phase < len(completed): + prepared.release_data() + + def artifact_before(index): + return ( + completed[index - 1].model_path if index else config.initialize_from or None + ) + + initial = config.initialize_from + initial_identity = ( + {"path": initial, "sha256": manifest.identity[str(Path(initial).resolve())]} + if initial + else None + ) + + def boundary(): + return phase_boundary( + identity.run_id, + selected=completed[-1].model_identity if completed else initial_identity, + active=state["active"], + ) + + if envelope is not None: + require_phase_boundary(envelope, boundary()) + + selected_path, selected_model = None, None + + def save_boundary(): + nonlocal selected_path, selected_model + selected = artifact_before(len(completed)) + if selected != selected_path: + selected_model = ( + read_execution_artifact(selected) if selected is not None else None + ) + selected_path = selected + atomic_save( + checkpoint, + checkpoint_envelope( + "learning", + state, + selected_model, + boundary=boundary(), + ), + ) + + def publish(commit): + report = commit.report_result() + commit.publish(PhaseArtifacts(Path(report["model_path"])), report) + + def replay(): + for commit in completed: + publish(commit) + + def run_group(transition): + index, group = transition.phase + first = group[0] + base = PhaseArtifacts.for_phase(config.phase_directory, index, first.name) + native_checkpoint = base.model.with_name("run.pt") + manifest.revalidate() + if index: + incoming = completed[index - 1] + verify_artifact( + incoming.model_path, + incoming.model_sha256, + "Selected phase model changed", + ) + active = state["active"] + if active is None: + active = { + "index": index, + "phase_index": index, + "run_id": identity.run_id, + "recovery": None, + } + state["active"] = active + save_boundary() + prior = active["recovery"] + if prior is not None: + verify_artifact( + prior["path"], prior["sha256"], "Active phase recovery changed" + ) + _snapshot_recovery(prior["path"], native_checkpoint) + + def save_progress(path, records): + previous = active["recovery"] + first_new = len(completed) - index + new = [ + _portable_commit( + config, index + offset, group[offset], native_checkpoint, values + ) + for offset, values in enumerate(records[first_new:], first_new) + ] + finished = len(records) == len(group) or ( + new and new[-1].result["stop_scope"] == "end_plan" + ) + if finished: + state["active"] = None + else: + recovery_path = ( + base.model.parent / "commits" / f"{uuid4().hex}.recovery.pt" + ) + _snapshot_recovery(path, recovery_path) + active["phase_index"] = len(completed) + len(new) + active["recovery"] = { + "path": str(recovery_path), + "sha256": file_checksum(recovery_path), + } + completed.extend(new) + state["completed"].extend(commit.as_dict() for commit in new) + save_boundary() + for commit in new: + publish(commit) + if previous is not None: + Path(previous["path"]).unlink(missing_ok=True) + + request = PortableGroup( + index, + group, + artifact_before(index), + prior is not None, + identity, + save_progress, + ) + if first.algorithm == "supervised": + run_supervised_group(config, request, revision if revision else {}) + else: + run_portable_policy_group(config, request, prepared) + stop = completed[-1].result["stop_scope"] + if stop != "end_plan" and len(completed) != index + len(group): + raise ValueError("native phase group returned before committing its phases") + return PhaseOutcome(None, stop) + + def finish_group(transition, outcome): + if transition.phase[0] <= last_data_phase < len(completed): + prepared.release_data() + + next_index = state["active"]["index"] if state["active"] else len(completed) + start = next( + (offset for offset, (index, _) in enumerate(groups) if index == next_index), + len(groups), + ) + stopped = bool(completed and completed[-1].result["stop_scope"] == "end_plan") + if start == len(groups) and next_index != len(phases) and not stopped: + raise ValueError("learning run checkpoint is between inconsistent phase groups") + completion_reason = run_phases( + groups, + run_group, + finish_group, + start=start, + stopped=stopped and state["active"] is None, + replay=replay, + ) + final = read_execution_artifact(completed[-1].model_path) + save_model_artifact(final, config.model_path) + reports = [commit.report_result() for commit in completed] + initialization = reports[0]["provenance"].get( + "actor_initialization", {"path": config.initialize_from} + ) + prepared.close() + publish_run_report( + run_report( + model_path=config.model_path, + checkpoint_path=checkpoint, + phases=reports, + plan_fingerprint=plan_identity, + run_id=identity.run_id, + session_id=identity.session_id, + source_provenance=manifest.identity, + initialization=initialization, + resume=resume.model_dump(mode="json"), + exports=[config.model_path], + status="complete", + completion_reason=completion_reason, + resource_accounting=prepared.accounting.snapshot(), + ), + config.report_path, + ) + return LearningRunResult( + config.model_path, + str(checkpoint), + config.report_path, + tuple( + { + **{key: report[key] for key in ("name", "algorithm", "model_path")}, + "report": report, + } + for report in reports + ), + ) diff --git a/src/sequifier/training/distributed_strategy.py b/src/sequifier/training/distributed_strategy.py index 61e356b4d..482c86886 100644 --- a/src/sequifier/training/distributed_strategy.py +++ b/src/sequifier/training/distributed_strategy.py @@ -155,8 +155,14 @@ def prepare_network(self, network: nn.Module) -> PreparedNetwork: if self.cpu_offload: kwargs["offload_policy"] = CPUOffloadPolicy() seen: set[int] = set() - backbone = getattr(network, "backbone", None) - for layer in getattr(backbone, "layers", ()): + from sequifier.model.network import ComposableTransformerNetwork + + layers = ( + network.backbone.layers + if isinstance(network, ComposableTransformerNetwork) + else () + ) + for layer in layers: if id(layer) not in seen: fully_shard(layer, **kwargs) seen.add(id(layer)) diff --git a/src/sequifier/training/engine.py b/src/sequifier/training/engine.py index 252b2db64..a6fef712a 100644 --- a/src/sequifier/training/engine.py +++ b/src/sequifier/training/engine.py @@ -2,20 +2,28 @@ from __future__ import annotations +import math import os import time from contextlib import nullcontext from copy import deepcopy -from dataclasses import dataclass +from dataclasses import asdict, dataclass from pathlib import Path from typing import Any import torch from sequifier.artifacts.backbone_repository import publish_revision -from sequifier.artifacts.manifests import write_manifest +from sequifier.artifacts.model_artifact import ( + build_model_artifact, + read_execution_artifact, + save_model_artifact, +) +from sequifier.artifacts.phase import PhaseArtifacts, PhaseCommit +from sequifier.artifacts.reports import phase_report, publish_run_report, run_report from sequifier.artifacts.run_checkpoint import LoaderState from sequifier.artifacts.state_dict import canonicalize_state_dict +from sequifier.evaluation.compatibility import compatible_supervised_phases from sequifier.evaluation.service import EvaluationContext, EvaluationResult from sequifier.export.service import ExportOptions from sequifier.integration.contexts import ( @@ -28,11 +36,24 @@ ValidationCompleted, ) from sequifier.logging_paths import model_log_directory +from sequifier.runtime.plan import ( + PhaseOutcome, + metric_improved, + run_phases, + select_handoff, + stop_scope, +) from sequifier.runtime.random_state import RandomState +from sequifier.runtime.stopping import StoppingState from sequifier.training.checkpoint_service import CheckpointRequest from sequifier.training.metrics_service import TrainingMetrics from sequifier.training.optimization import OptimizationBoundaryState, UpdatePolicy -from sequifier.training.runtime import SourceScheduler, build_source_runtime +from sequifier.training.plan import SupervisedPhaseView +from sequifier.training.runtime import ( + DatasetRuntimeRegistry, + SourceScheduler, + build_source_runtime, +) from sequifier.training.state import RunStateSnapshot @@ -88,7 +109,7 @@ def _identity( global_batch_step=run.state.global_batch_step + 1, optimizer_step=run.optimization.optimizer_step, accumulation_index=accumulation_index, - accumulation_steps=run.optimization.gradient_policy.accumulation_steps, + accumulation_batches=run.optimization.gradient_policy.accumulation_batches, rank=run.distributed.rank, world_size=run.distributed.world_size, ) @@ -107,7 +128,7 @@ def _evaluation_context( return EvaluationContext( run_id=run.state.run_id, session_id=run.state.session_id, - phase_index=phase_index, + phase_index=run.state.phase_offset + phase_index, phase_epoch=phase_epoch, epoch=epoch, training_batch=batch, @@ -120,10 +141,14 @@ def _evaluation_context( kind=kind, ) - def _evaluate(self, run: Any, context: EvaluationContext) -> EvaluationResult: - result = run.evaluation.evaluate( - run.network, run.config.evaluation_sources, run.datasets, context - ) + def _evaluate( + self, + run: Any, + phase: SupervisedPhaseView, + datasets: DatasetRuntimeRegistry, + context: EvaluationContext, + ) -> EvaluationResult: + result = run.evaluation.evaluate(run.network, phase, datasets, context) run.metrics.record_evaluation( result, context=context, @@ -160,25 +185,37 @@ def _check_pruning(self, run: Any) -> None: if should_prune: raise SystemExit(143) - def _update_monitor(self, run: Any, result: EvaluationResult) -> tuple[float, bool]: - monitor = run.config.evaluation_monitor + def _update_monitor( + self, + run: Any, + result: EvaluationResult, + phase: Any, + *, + count_patience: bool = True, + ) -> tuple[float, bool]: + monitor = phase.evaluation_monitor if monitor is None or monitor.source not in result.sources: return float("nan"), False value = result.sources[monitor.source].total_loss - if monitor.mode == "max" and run.state.best_validation_loss == float("inf"): + if monitor.direction == "max" and run.state.best_validation_loss == float( + "inf" + ): run.state.best_validation_loss = float("-inf") - improved = ( - value < run.state.best_validation_loss - if monitor.mode == "min" - else value > run.state.best_validation_loss + if not math.isfinite(value): + return value, False + run.state.last_validation_metric = value + improved = metric_improved( + value, run.state.best_validation_loss, monitor.direction ) if improved: run.state.best_validation_loss = value - run.state.epochs_without_improvement = 0 + run.state.best_optimizer_step = run.state.optimizer_step + if count_patience: + run.state.epochs_without_improvement = 0 state = run.distributed.capture_model_state(run.network) if run.distributed.rank == 0: run.state.best_model_state_dict = state - else: + elif count_patience: run.state.epochs_without_improvement += 1 return value, improved @@ -206,29 +243,99 @@ def _manifest( ) -> None: if run.distributed.rank != 0: return - path = run.checkpoints.store.latest_path.parent / "manifest.json" - write_manifest( - path, - { - "artifact_type": "sequifier_run_manifest", - "format_version": 2, - "run_id": run.state.run_id, - "session_id": run.state.session_id, - "status": status, - "completion_reason": result.completion_reason, - "source_epoch": run.state.epoch, - "exports": [str(path) for path in result.exports], - "backbone_parent_revision_id": run.state.backbone_parent_revision_id, - "backbone_publication": result.publication, + publish_run_report( + run_report( + model_path=run.config.model_path, + checkpoint_path=run.checkpoints.store.latest_path, + phases=[record["result"] for record in run.state.phase_commits], + source_provenance=run.dependencies.identity, + run_id=run.state.run_id, + session_id=run.state.session_id, + status=status, + completion_reason=result.completion_reason, + source_epoch=run.state.epoch, + exports=[str(path) for path in result.exports] + + [run.config.model_path], + backbone_parent_revision_id=run.state.backbone_parent_revision_id, + backbone_publication=result.publication, + ), + run.config.report_path, + ) + + def _stage_phase(self, run, index, view, outcome): + """Stage the selected model for the next checkpoint publication.""" + from sequifier.rl.run_state import weight_fingerprint + + phase = view.spec() + last = run.distributed.capture_model_state(run.network) + if run.distributed.rank != 0: + return + selected = select_handoff(phase.handoff, last, run.state.best_model_state_dict) + artifacts = PhaseArtifacts.for_phase( + run.config.phase_directory, index, phase.name + ) + metric = run.state.best_validation_loss + report = phase_report( + index=index, + name=phase.name, + algorithm="supervised", + resolved_seed=view.seed, + model_path=artifacts.model, + checkpoint_path=run.checkpoints.store.latest_path, + handoff=phase.handoff, + monitor=phase.evaluation_monitor.metric + if phase.evaluation_monitor + else None, + direction=phase.evaluation_monitor.direction + if phase.evaluation_monitor + else None, + selected_metric=metric + if math.isfinite(metric) and phase.handoff == "best" + else run.state.last_validation_metric, + selected_transaction=run.state.best_optimizer_step + if phase.handoff == "best" + else run.state.optimizer_step, + selected_weight_hash=weight_fingerprint(selected), + clocks={ + "epochs": run.state.phase_epoch, + "optimizer_steps": run.state.optimizer_step, }, + stop_scope=stop_scope( + phase.stopping, run.state.stopping_state.get("triggered") + ), + provenance={"source_provenance": run.dependencies.identity}, + completion_reason=( + outcome.completion_reason + or ( + "early_stopping" + if run.state.stopping_state.get("triggered") + else "completed" + ) + ), + ) + return PhaseCommit.stage( + artifacts, + report, + model=build_model_artifact(run.network, run.config, state_dict=selected), ) def run(self, run: Any) -> RunResult: + return self._run(run, group=False) + + def run_group(self, run: Any) -> None: + """Run a native group with its supplied checkpoint publisher.""" + self._run(run, group=True) + + def _run(self, run: Any, *, group: bool) -> RunResult: config = run.config state = run.state + phases = run.plan.phases time_checkpoint_intervals_enabled = bool( - config.global_training.save_latest_interval_minutes is not None - or config.global_training.save_interval_minutes is not None + config.runtime_settings.latest_checkpoint_interval_minutes is not None + or any( + view.spec().checkpoints.snapshot_interval_minutes is not None + for view in phases + ) ) defer_ddp_gradient_sync = bool( isinstance( @@ -236,7 +343,7 @@ def run(self, run: Any) -> RunResult: torch.nn.parallel.DistributedDataParallel, ) and len(run.datasets) == 1 - and run.optimization.gradient_policy.accumulation_steps > 1 + and run.optimization.gradient_policy.accumulation_batches > 1 and not time_checkpoint_intervals_enabled ) accumulation_count = 0 @@ -247,7 +354,7 @@ def run(self, run: Any) -> RunResult: metric_windows: dict[str, dict[str, Any]] = {} last_latest = time.monotonic() last_snapshot = time.monotonic() - last_snapshot_step = state.global_batch_step + last_snapshot_step = state.last_snapshot_batch_step access = run.optimization.access(run.network) run.integrations.emit(ModelReady(access=access)) run.optimization.optimizer.zero_grad(set_to_none=True) @@ -323,132 +430,168 @@ def flush_metrics( ) current_batch = sum(state.iterator_positions.values()) current_batches_total = 1 - try: + + def run_phase(transition): + nonlocal access, active_dataset, accumulation_count, last_identity + nonlocal boundary, boundary_identity, batch_in_progress, batch_committed + nonlocal last_snapshot, last_snapshot_step, last_latest + nonlocal current_epoch, current_batch, current_batches_total + nonlocal stop_requested, completion_reason + phase_index, view = transition.index, transition.phase + phase = view.spec() + if ( + phase_index == state.phase_index + and state.phase_epoch_complete + and ( + state.phase_epoch >= phase.epochs + or state.stopping_state.get("triggered") + ) + ): + return PhaseOutcome( + None, + stop_scope(phase.stopping, state.stopping_state.get("triggered")), + ) + run.dependencies.revalidate() + run.metrics.set_phase(phase) + if phase_index != state.phase_index: + previous = transition.previous.spec() + if previous.handoff == "best": + selected = run.distributed.gather_objects( + state.best_model_state_dict + )[0] + if selected is None: + raise ValueError( + f"{previous.name}: best handoff has no finite evaluation" + ) + run.distributed.restore_model_state(run.network, selected) + if not compatible_supervised_phases( + config, phase_index - 1, phase_index + ): + state.best_validation_loss = float("inf") + state.best_model_state_dict = None + state.epochs_without_improvement = 0 + state.stopping_state = {} + state.last_validation_metric = None + last_snapshot = time.monotonic() + last_snapshot_step = state.global_batch_step + state.last_snapshot_batch_step = last_snapshot_step + phase_stopping = StoppingState(**state.stopping_state) + phase_stop = phase_stopping.triggered + datasets = ( + run.datasets + if run.optimization_phase_index == phase_index + else run.dataset_factory(view) + ) if ( - state.global_batch_step == 0 - and config.evaluation_sources - and config.global_training.calculate_validation_loss_on_initialization + phase.evaluation + and phase.evaluation.before_phase + and phase_index not in state.initial_evaluated_phases ): - first_sources = [ - build_source_runtime(source, run.datasets) - for source in config.training_plan[0].sources - ] self._evaluate( run, + view, + datasets, self._evaluation_context( run, kind="initial", - phase_index=0, + phase_index=phase_index, phase_epoch=0, - epoch=0, + epoch=state.epoch, batch=0, - batches_total=sum( - source.num_batches() for source in first_sources - ), + batches_total=1, ), ) - for phase_index, phase in enumerate(config.training_plan): - if phase_index < state.phase_index: - continue - sources = [ - build_source_runtime(source, run.datasets) - for source in phase.sources - ] - scheduler = SourceScheduler( - phase, - sources, - seed=config.seed + phase_index, - phase_index=phase_index, + state.initial_evaluated_phases.append(phase_index) + sources = [ + build_source_runtime(source, datasets, view) for source in phase.sources + ] + scheduler = SourceScheduler(view, sources) + if phase_index == state.phase_index and state.source_scheduler_state: + scheduler.load_state_dict(state.source_scheduler_state) + start_epoch = ( + state.phase_epoch + (1 if state.phase_epoch_complete else 0) + if phase_index == state.phase_index + else 1 + ) + for phase_epoch in range(max(1, start_epoch), phase.epochs + 1): + resuming = ( + state.phase_index == phase_index + and state.phase_epoch == phase_epoch + and not state.phase_epoch_complete + and bool(state.source_scheduler_state) ) - if phase_index == state.phase_index and state.source_scheduler_state: - scheduler.load_state_dict(state.source_scheduler_state) - start_epoch = ( - state.phase_epoch + (1 if state.phase_epoch_complete else 0) - if phase_index == state.phase_index - else 1 + current_epoch = state.epoch if resuming else state.epoch + 1 + state.phase_index = phase_index + state.phase_epoch = phase_epoch + state.phase_epoch_complete = False + if run.optimization_phase_index != phase_index: + run.start_phase( + view, + datasets, + continue_optimizer=transition.continues_optimizer, + ) + state.optimizer_step = run.optimization.optimizer_step + state.epoch = current_epoch + state.batch = 0 + state.accumulation_index = 0 + state.iterator_positions = {} + state.source_scheduler_state = scheduler.state_dict() + access = run.optimization.access(run.network) + run.optimization.optimizer.zero_grad(set_to_none=True) + boundary = capture_runtime_boundary(run) + boundary_identity = last_identity + run.callable_network.train() + current_batches_total = sum(source.num_batches() for source in sources) + current_batch = ( + sum(state.iterator_positions.values()) if resuming else 0 ) - for phase_epoch in range(max(1, start_epoch), phase.epochs + 1): - resuming = ( - state.phase_index == phase_index - and state.phase_epoch == phase_epoch - and not state.phase_epoch_complete - and bool(state.source_scheduler_state) + for runtime_batch in scheduler.iter_epoch(phase_epoch): + batch_in_progress = True + batch_committed = False + self._check_pruning(run) + current_batch += 1 + dataset = run.datasets.resolve(runtime_batch.dataset) + if active_dataset is not None and active_dataset != dataset.name: + flush() + active_dataset = dataset.name + started = time.perf_counter() + prepared = run.loss.prepare_batch( + runtime_batch.batch, dataset, run.distributed.device ) - current_epoch = state.epoch if resuming else state.epoch + 1 - state.phase_index = phase_index - state.phase_epoch = phase_epoch - state.phase_epoch_complete = False - if ( - config.global_training.reset_optimization_on_phase - and run.optimization_phase_index != phase_index - ): - run.start_phase(phase_index) - state.optimizer_step = run.optimization.optimizer_step - state.epoch = current_epoch - state.batch = 0 - state.accumulation_index = 0 - state.iterator_positions = {} - state.source_scheduler_state = scheduler.state_dict() - access = run.optimization.access(run.network) - run.optimization.optimizer.zero_grad(set_to_none=True) - boundary = capture_runtime_boundary(run) - boundary_identity = last_identity - run.callable_network.train() - current_batches_total = sum( - source.num_batches() for source in sources + dataset.metrics.training_batches += 1 + identity = self._identity( + run, current_epoch, current_batch, accumulation_count ) - current_batch = ( - sum(state.iterator_positions.values()) if resuming else 0 + prepared_event = BatchPrepared( + access=access, + identity=identity, + inputs=prepared.features, + targets=prepared.targets, + metadata=prepared.metadata, ) - for runtime_batch in scheduler.iter_epoch(phase_epoch): - batch_in_progress = True - batch_committed = False - self._check_pruning(run) - current_batch += 1 - dataset = run.datasets.resolve(runtime_batch.dataset) - if ( - active_dataset is not None - and active_dataset != dataset.name - ): - flush() - active_dataset = dataset.name - started = time.perf_counter() - prepared = run.loss.prepare_batch( - runtime_batch.batch, dataset, run.distributed.device - ) - dataset.metrics.training_batches += 1 - identity = self._identity( - run, current_epoch, current_batch, accumulation_count - ) - prepared_event = BatchPrepared( - access=access, - identity=identity, - inputs=prepared.features, - targets=prepared.targets, - metadata=prepared.metadata, - ) - if run.integrations.enabled: - run.integrations.emit(prepared_event) - save_interval_batches = ( - config.global_training.save_interval_batches - ) - batch_snapshot_due = bool( - save_interval_batches is not None - and identity.global_batch_step - last_snapshot_step - >= save_interval_batches - ) - gradient_sync_due = bool( - accumulation_count + 1 - >= run.optimization.gradient_policy.accumulation_steps - or current_batch >= current_batches_total - or batch_snapshot_due - ) - gradient_sync_context = ( - run.callable_network.no_sync() - if defer_ddp_gradient_sync and not gradient_sync_due - else nullcontext() - ) - with gradient_sync_context: + if run.integrations.enabled: + run.integrations.emit(prepared_event) + snapshot_interval_batches = ( + phase.checkpoints.snapshot_interval_batches + ) + batch_snapshot_due = bool( + snapshot_interval_batches is not None + and identity.global_batch_step - last_snapshot_step + >= snapshot_interval_batches + ) + gradient_sync_due = bool( + accumulation_count + 1 + >= run.optimization.gradient_policy.accumulation_batches + or current_batch >= current_batches_total + or batch_snapshot_due + ) + gradient_sync_context = ( + run.callable_network.no_sync() + if defer_ddp_gradient_sync and not gradient_sync_due + else nullcontext() + ) + with gradient_sync_context: + with run.precision.autocast(): trace = run.integrations.forward_trace(prepared_event) output = run.callable_network( prepared.features, @@ -481,191 +624,267 @@ def flush_metrics( backward_loss=loss.backward_loss, ) ) - run.optimization.accumulate( - loss.backward_loss, - identity, - run.integrations, - run.network, - ) - accumulation_count += 1 - last_identity = identity - state.epoch = current_epoch - state.batch = current_batch - state.global_batch_step = identity.global_batch_step - state.accumulation_index = accumulation_count - 1 - state.source_scheduler_state = scheduler.state_dict() - position_key = f"{runtime_batch.dataset}.{runtime_batch.part}" - state.iterator_positions[position_key] = ( - state.iterator_positions.get(position_key, 0) + 1 + run.optimization.accumulate( + loss.backward_loss, + identity, + run.integrations, + run.network, ) - dataset.parts[runtime_batch.part].iterator_positions[ - "training" - ] = state.iterator_positions[position_key] - batch_committed = True - batch_in_progress = False - window = metric_windows.setdefault( - dataset.name, - { - "sums": { - name: value.new_zeros(()) - for name, value in loss.accounting_sums.items() - }, - "count": loss.accounting_count.new_zeros(()).to( - next(iter(loss.accounting_sums.values())).dtype - ), - "batches": 0, - "seconds": 0.0, - "part": runtime_batch.part, + accumulation_count += 1 + last_identity = identity + state.epoch = current_epoch + state.batch = current_batch + state.global_batch_step = identity.global_batch_step + state.accumulation_index = accumulation_count - 1 + state.source_scheduler_state = scheduler.state_dict() + position_key = f"{runtime_batch.dataset}.{runtime_batch.part}" + state.iterator_positions[position_key] = ( + state.iterator_positions.get(position_key, 0) + 1 + ) + dataset.parts[runtime_batch.part].iterator_positions["training"] = ( + state.iterator_positions[position_key] + ) + batch_committed = True + batch_in_progress = False + window = metric_windows.setdefault( + dataset.name, + { + "sums": { + name: value.new_zeros(()) + for name, value in loss.accounting_sums.items() }, - ) - for name, value in loss.accounting_sums.items(): - window["sums"][name] += value - window["count"] += loss.accounting_count.to( - window["count"].dtype - ) - window["batches"] += 1 - window["seconds"] += time.perf_counter() - started - if window["part"] != runtime_batch.part: - window["part"] = None - if window["batches"] >= config.global_training.log_interval: - flush_metrics( - dataset.name, - current_epoch, - current_batch, - current_batches_total, - ) - if ( - accumulation_count - >= run.optimization.gradient_policy.accumulation_steps - ): - flush() - now = time.monotonic() - snapshot_due = bool( - config.global_training.save_interval_batches is not None - and state.global_batch_step - last_snapshot_step - >= config.global_training.save_interval_batches - ) or bool( - config.global_training.save_interval_minutes is not None - and now - last_snapshot - >= config.global_training.save_interval_minutes * 60 - ) - latest_due = bool( - config.global_training.save_latest_interval_minutes - is not None - and now - last_latest - >= config.global_training.save_latest_interval_minutes * 60 - ) - if ( - run.distributed.world_size > 1 - and time_checkpoint_intervals_enabled - ): - checkpoint_due = torch.tensor( - [ - int(latest_due) if run.distributed.rank == 0 else 0, - int(snapshot_due) - if run.distributed.rank == 0 - else 0, - ], - device=run.distributed.device, - dtype=torch.int32, - ) - torch.distributed.broadcast(checkpoint_due, src=0) - latest_due = bool(checkpoint_due[0].item()) - snapshot_due = bool(checkpoint_due[1].item()) - if latest_due or snapshot_due: - flush() - if latest_due: - run.checkpoints.save( - CheckpointRequest("latest", identity), run - ) - last_latest = time.monotonic() - if snapshot_due: - if ( - config.global_training.save_interval_val_loss - and config.evaluation_sources - ): - self._evaluate( - run, - self._evaluation_context( - run, - kind="interval", - phase_index=phase_index, - phase_epoch=phase_epoch, - epoch=current_epoch, - batch=current_batch, - batches_total=current_batches_total, - ), - ) - run.callable_network.train() - run.checkpoints.save( - CheckpointRequest( - f"epoch-{current_epoch}-batch-{current_batch}", - identity, - ), - run, - ) - last_snapshot = time.monotonic() - last_snapshot_step = state.global_batch_step - if stop_requested: - completion_reason = "integration_requested_stop" - break - flush() - for dataset_name in tuple(metric_windows): + "count": loss.accounting_count.new_zeros(()).to( + next(iter(loss.accounting_sums.values())).dtype + ), + "batches": 0, + "seconds": 0.0, + "part": runtime_batch.part, + }, + ) + for name, value in loss.accounting_sums.items(): + window["sums"][name] += value + window["count"] += loss.accounting_count.to(window["count"].dtype) + window["batches"] += 1 + window["seconds"] += time.perf_counter() - started + if window["part"] != runtime_batch.part: + window["part"] = None + if ( + window["batches"] + >= config.runtime_settings.log_interval_batches + ): flush_metrics( - dataset_name, + dataset.name, current_epoch, current_batch, current_batches_total, ) - state.phase_epoch_complete = True - evaluation = ( - self._evaluate( - run, - self._evaluation_context( - run, - kind="epoch_end", - phase_index=phase_index, - phase_epoch=phase_epoch, - epoch=current_epoch, - batch=current_batch, - batches_total=current_batches_total, - ), - ) - if config.evaluation_sources - else EvaluationResult() + if ( + accumulation_count + >= run.optimization.gradient_policy.accumulation_batches + ): + flush() + now = time.monotonic() + snapshot_due = bool( + phase.checkpoints.snapshot_interval_batches is not None + and state.global_batch_step - last_snapshot_step + >= phase.checkpoints.snapshot_interval_batches + ) or bool( + phase.checkpoints.snapshot_interval_minutes is not None + and now - last_snapshot + >= phase.checkpoints.snapshot_interval_minutes * 60 + ) + latest_due = bool( + config.runtime_settings.latest_checkpoint_interval_minutes + is not None + and now - last_latest + >= config.runtime_settings.latest_checkpoint_interval_minutes + * 60 ) - if run.optimization.scheduler_policy.step_on == "epoch": - monitor = config.evaluation_monitor - metric = ( - evaluation.sources[monitor.source].total_loss - if monitor is not None - else None + if ( + run.distributed.world_size > 1 + and time_checkpoint_intervals_enabled + ): + checkpoint_due = torch.tensor( + [ + int(latest_due) if run.distributed.rank == 0 else 0, + int(snapshot_due) if run.distributed.rank == 0 else 0, + ], + device=run.distributed.device, + dtype=torch.int32, ) - run.optimization.step_scheduler(metric) - state.iterator_positions = {} - state.source_scheduler_state = scheduler.state_dict() - self._update_monitor(run, evaluation) - boundary = capture_runtime_boundary(run) - boundary_identity = last_identity - if current_epoch % config.global_training.save_interval_epochs == 0: + torch.distributed.broadcast(checkpoint_due, src=0) + latest_due = bool(checkpoint_due[0].item()) + snapshot_due = bool(checkpoint_due[1].item()) + if latest_due or snapshot_due: + flush() + if latest_due: + run.checkpoints.save(CheckpointRequest("latest", identity), run) + last_latest = time.monotonic() + if snapshot_due: + if ( + phase.evaluation_sources + and phase.evaluation is not None + and phase.evaluation.on_snapshot + ): + self._evaluate( + run, + view, + datasets, + self._evaluation_context( + run, + kind="interval", + phase_index=phase_index, + phase_epoch=phase_epoch, + epoch=current_epoch, + batch=current_batch, + batches_total=current_batches_total, + ), + ) + run.callable_network.train() + state.last_snapshot_batch_step = state.global_batch_step run.checkpoints.save( - CheckpointRequest(f"epoch-{current_epoch}", last_identity), + CheckpointRequest( + f"epoch-{current_epoch}-batch-{current_batch}", + identity, + ), run, ) - patience = config.global_training.early_stopping_epochs - if ( - patience is not None - and config.evaluation_monitor is not None - and state.epochs_without_improvement >= patience - ): - stop_requested = True - completion_reason = "early_stopping" + last_snapshot = time.monotonic() + last_snapshot_step = state.global_batch_step if stop_requested: + completion_reason = "integration_requested_stop" break - if stop_requested: + flush() + for dataset_name in tuple(metric_windows): + flush_metrics( + dataset_name, + current_epoch, + current_batch, + current_batches_total, + ) + state.phase_epoch_complete = True + evaluation = ( + self._evaluate( + run, + view, + datasets, + self._evaluation_context( + run, + kind="epoch_end", + phase_index=phase_index, + phase_epoch=phase_epoch, + epoch=current_epoch, + batch=current_batch, + batches_total=current_batches_total, + ), + ) + if phase.evaluation_sources + and ( + phase_epoch % phase.evaluation_interval_epochs == 0 + or phase_epoch == phase.epochs + ) + else EvaluationResult() + ) + if run.optimization.scheduler_policy.step_on == "epoch": + monitor = phase.evaluation_monitor + metric = ( + evaluation.sources[monitor.source].total_loss + if monitor is not None and monitor.source in evaluation.sources + else None + ) + run.optimization.step_scheduler(metric) + state.iterator_positions = {} + state.source_scheduler_state = scheduler.state_dict() + monitored, _ = self._update_monitor(run, evaluation, phase) + phase_stop = phase_stopping.observe( + phase.stopping, + monitored, + phase.evaluation_monitor.direction + if phase.evaluation_monitor + else "min", + phase_epoch, + ) + state.stopping_state = asdict(phase_stopping) + if phase_stop: + state.plan_stopped = ( + stop_scope(phase.stopping, phase_stop) == "end_plan" + ) + stop_requested = state.plan_stopped + completion_reason = "early_stopping" + boundary = capture_runtime_boundary(run) + boundary_identity = last_identity + if ( + phase_stop + or phase_epoch % phase.checkpoint_interval_epochs == 0 + or phase_epoch == phase.epochs + ): + run.checkpoints.save( + CheckpointRequest(f"epoch-{current_epoch}", last_identity), + run, + ) + if stop_requested or phase_stop: break + return PhaseOutcome( + None, + stop_scope(phase.stopping, phase_stopping.triggered), + completion_reason if stop_requested else None, + ) + + committing = False + + def publish(commit): + if run.distributed.rank == 0: + commit.publish( + PhaseArtifacts(Path(commit.result["model_path"])), commit.result + ) + + def replay(): + if run.distributed.rank == 0: + for values in state.phase_commits: + publish(PhaseCommit.load(values)) + + def commit_phase(transition, outcome): + nonlocal committing + committing = True + commit = self._stage_phase( + run, + state.phase_offset + transition.index, + transition.phase, + outcome, + ) + commit = run.distributed.gather_objects(commit)[0] + state.phase_commits.append(commit.as_dict()) + run.checkpoints.save(CheckpointRequest("latest", last_identity), run) + return commit + + def finish_publication(commit): + nonlocal committing, boundary + publish(commit) + boundary = capture_runtime_boundary(run) + committing = False + + try: + completion_reason = run_phases( + phases, + run_phase, + commit_phase, + start=len(state.phase_commits), + stopped=state.plan_stopped + and len(state.phase_commits) > state.phase_index, + completion_reason="normal_completion", + replay=replay, + publish=finish_publication, + optimizer_continues=lambda view: view.optimizer_state == "continue", + terminal_reason=( + "integration_requested_stop" + if state.phase_commits + and state.phase_commits[-1]["result"]["completion_reason"] + == "integration_requested_stop" + else None + ), + ) except BaseException: - if update_failed: + if committing or update_failed: run.optimization.optimizer.zero_grad(set_to_none=True) raise if batch_in_progress and not batch_committed: @@ -681,6 +900,14 @@ def flush_metrics( ) raise + if group: + return RunResult( + "early_stopping" + if state.stopping_state.get("triggered") + else completion_reason, + (), + {}, + ) run.distributed.barrier() last_state = run.distributed.capture_model_state(run.network) best_state = state.best_model_state_dict or last_state @@ -702,6 +929,11 @@ def flush_metrics( best_export.paths + last_export.paths, publication, ) + if run.distributed.rank == 0: + commit = PhaseCommit.load(state.phase_commits[-1]) + save_model_artifact( + read_execution_artifact(commit.model_path), config.model_path + ) self._manifest(run, result) run.integrations.emit( RunCompleted(access=access, completion_reason=completion_reason) diff --git a/src/sequifier/training/loss.py b/src/sequifier/training/loss.py index 9be8562a5..6d18edf40 100644 --- a/src/sequifier/training/loss.py +++ b/src/sequifier/training/loss.py @@ -41,7 +41,7 @@ def prepare_batch( *, eval_seed: int | None = None, ) -> PreparedBatch: - interface = dataset.config.interface + interface = dataset.interface if ( interface.depth_layouts or interface.static_conditioning is not None @@ -112,7 +112,7 @@ def calculate( dataset: DatasetRuntime, network: ComposableTransformerNetwork, ) -> LossResult: - interface = dataset.config.interface + interface = dataset.interface target_names = list(interface.target_columns) missing = set(target_names).difference(batch.targets) if missing: @@ -149,7 +149,7 @@ def calculate( for target in target_names: kind = interface.target_column_types[target] if ( - target in getattr(interface, "categorical_hashing", {}) + target in interface.categorical_hashing and target not in output.auxiliary_logits ): raise RuntimeError( @@ -172,9 +172,11 @@ def calculate( target_for_loss = lookup[safe_global_ids] excluded = target_for_loss < 0 if target in output.auxiliary_logits: - codec = network.resolve_interface( - dataset.interface_name - ).decoder.hash_codecs[target] + codec = ( + network.resolve_interface(dataset.interface_name) + .require_decoder() + .hash_codec(target) + ) if bool((excluded & flat_mask).any()): raise ValueError( f"Categorical target {target!r} contains excluded special " @@ -254,7 +256,31 @@ def calculate( ) target_for_loss = target_for_loss.masked_fill(excluded, 0) criterion = dataset.criteria[target] - if isinstance(criterion, AccurateCrossEntropyLoss): + if isinstance(criterion, torch.nn.GaussianNLLLoss): + from sequifier.io.preprocessing import denormalize_values + from sequifier.model.prediction_distribution import ( + RealPredictionDistribution, + ) + + field = interface.prediction_distributions.get(target) + if field is None or field.type != "real": + raise ValueError( + f"{target}: GaussianNLLLoss requires a real prediction_distributions entry" + ) + route = network.resolve_interface(dataset.interface_name) + location, observed = ( + logits_for_loss[flat_mask], + target_for_loss[flat_mask], + ) + statistics = interface.selected_columns_statistics.get(target) + if statistics is not None and interface.normalize_real_columns: + location = denormalize_values(location, statistics) + observed = denormalize_values(observed, statistics) + distribution = RealPredictionDistribution( + location, route.prediction_scale(target) + ) + sums[target] = -distribution.log_prob(observed).sum() + elif isinstance(criterion, AccurateCrossEntropyLoss): sums[target] = criterion( logits_for_loss[flat_mask], target_for_loss[flat_mask] ).sum() @@ -302,7 +328,7 @@ def finalize_accounting( *, allow_empty: bool = False, ) -> tuple[Tensor, dict[str, Tensor]]: - targets = list(dataset.config.interface.target_columns) + targets = list(dataset.interface.target_columns) reduced_sums, reduced_count = self.reduce_accounting(sums, count, targets) return self.finalize_reduced_accounting( reduced_sums, @@ -339,7 +365,7 @@ def finalize_reduced_accounting( allow_empty: bool = False, ) -> tuple[Tensor, dict[str, Tensor]]: """Apply the dataset's current loss weights to reduced accounting sums.""" - logical_targets = list(dataset.config.interface.target_columns) + logical_targets = list(dataset.interface.target_columns) targets = logical_targets + sorted(set(sums) - set(logical_targets)) if count.item() == 0: if not allow_empty: @@ -353,9 +379,7 @@ def finalize_reduced_accounting( else: logical = target[len("@hash/") :].rsplit("/", 1)[0] width_count = len( - dataset.config.interface.categorical_hash_contracts[logical][ - "widths" - ] + dataset.interface.categorical_hash_contracts[logical]["widths"] ) weight = ( float((dataset.loss_weights or {}).get(logical, 1.0)) / width_count diff --git a/src/sequifier/training/metrics.py b/src/sequifier/training/metrics.py index cf54636e2..316c29683 100644 --- a/src/sequifier/training/metrics.py +++ b/src/sequifier/training/metrics.py @@ -12,16 +12,19 @@ import numpy as np from loguru import logger +from torch import Tensor +from sequifier.contract_versions import ContractVersions, require_contract_version from sequifier.logging_paths import dataset_artifact_prefix from sequifier.typechecking import beartype -METRICS_SCHEMA_VERSION = 1 -FULL_METRICS_SCHEMA_VERSION = 2 TOTAL_TARGET = "__total__" COMMON_FIELDS = [ "schema_version", + "phase", + "algorithm", + "iteration", "run_id", "session_id", "timestamp_utc", @@ -40,6 +43,8 @@ ] TRAINING_FIELDS = FULL_COMMON_FIELDS + [ + "metric", + "is_primary", "window_batches", "target", "value", @@ -48,6 +53,7 @@ ] TRAINING_CONDENSED_FIELDS = [ + "schema_version", "run_id", "epoch", "batch", @@ -58,6 +64,7 @@ ] VALIDATION_FIELDS = FULL_COMMON_FIELDS + [ + "is_primary", "evaluation_id", "evaluation_kind", "metric", @@ -68,6 +75,7 @@ ] VALIDATION_CONDENSED_FIELDS = [ + "schema_version", "run_id", "epoch", "batch", @@ -101,8 +109,9 @@ def _timestamp_utc() -> str: @beartype def _metric_value(value: Any) -> float: """Convert scalar tensor/NumPy/Python values without display rounding.""" - item = getattr(value, "item", None) - scalar: Any = item() if callable(item) else value + scalar: Any = ( + value.item() if isinstance(value, (Tensor, np.generic, np.ndarray)) else value + ) return float(scalar) @@ -116,6 +125,19 @@ def _format_number(number: float) -> str: return f"{adjusted:5.2f}e{order_of_magnitude}" +def validate_metrics_csv_header(fields, path): + if fields is None or "schema_version" not in fields: + raise ValueError( + f"Unsupported metrics schema in {path}: missing schema_version" + ) + + +def validate_metrics_csv_row(row, path): + value = row.get("schema_version") + if value != str(ContractVersions.METRICS_SCHEMA): + raise ValueError(f"Unsupported metrics schema in {path}") + + class _CsvAppender: """Validate and append complete row groups to one CSV file.""" @@ -130,7 +152,12 @@ def __init__(self, path: Path, fields: list[str]) -> None: def _ensure_header(self) -> None: if self.path.exists() and self.path.stat().st_size > 0: with self.path.open("r", encoding="utf-8", newline="") as file: - existing = next(csv.reader(file), None) + reader = csv.DictReader(file) + existing = reader.fieldnames + if "schema_version" in self.fields: + validate_metrics_csv_header(existing, self.path) + for row in reader: + validate_metrics_csv_row(row, self.path) if existing != self.fields: raise ValueError( f"Structured metric schema mismatch in {self.path}: " @@ -144,12 +171,29 @@ def _ensure_header(self) -> None: @beartype def append(self, rows: Iterable[Mapping[str, Any]], *, durable: bool) -> None: - row_group = list(rows) + row_group = [dict(row) for row in rows] + if "schema_version" in self.fields: + for row in row_group: + version = row.setdefault( + "schema_version", ContractVersions.METRICS_SCHEMA + ) + require_contract_version( + "metrics schema", version, ContractVersions.METRICS_SCHEMA + ) if not row_group: return with self.path.open("a", encoding="utf-8", newline="") as file: writer = csv.DictWriter(file, fieldnames=self.fields, extrasaction="raise") - writer.writerows(row_group) + for row in row_group: + if "metric" in self.fields: + row.setdefault("metric", "loss") + if "is_primary" in self.fields: + row.setdefault( + "is_primary", + row.get("metric") == "loss" + and row.get("target") == TOTAL_TARGET, + ) + writer.writerow(row) file.flush() if durable: os.fsync(file.fileno()) @@ -177,6 +221,9 @@ def __init__( dataset_count=dataset_count, ) self.model_name = model_name + self.phase = "" + self.algorithm = "supervised" + self.iteration = 0 self.dataset_name = dataset_name self.rank = rank self.class_share_columns = tuple(class_share_columns) @@ -219,7 +266,10 @@ def _common( part: str | None = None, ) -> dict[str, Any]: return { - "schema_version": METRICS_SCHEMA_VERSION, + "schema_version": ContractVersions.METRICS_SCHEMA, + "phase": self.phase, + "algorithm": self.algorithm, + "iteration": self.iteration, "run_id": run_id, "session_id": session_id, "timestamp_utc": timestamp, @@ -268,7 +318,6 @@ def write_training( for field, value in common.items() if field in FULL_COMMON_FIELDS } - full_common["schema_version"] = FULL_METRICS_SCHEMA_VERSION measurements = [(TOTAL_TARGET, total_loss), *target_losses.items()] rows = [ { @@ -346,7 +395,6 @@ def write_validation( for field, value in common.items() if field in FULL_COMMON_FIELDS }, - "schema_version": FULL_METRICS_SCHEMA_VERSION, "evaluation_id": evaluation_id, "evaluation_kind": evaluation_kind, "learning_rate": float(learning_rate), @@ -447,11 +495,100 @@ def write_validation( ) if class_rows and not self.class_share_columns: raise ValueError( - "Class-share rows were provided without configured class-share " - "columns." + "Class-share rows were provided without configured class-share columns." ) if class_rows and self._class_shares is None: self._class_shares = _CsvAppender(self.class_share_path, CLASS_SHARE_FIELDS) if self._class_shares is not None: self._class_shares.append(class_rows, durable=True) return evaluation_id + + def write_observation( + self, + *, + validation, + run_id, + session_id, + epoch, + iteration, + batches_total, + global_step, + values, + primary, + learning_rate, + elapsed_seconds=0.0, + initial=False, + ): + """Use the supervised tables for named algorithm-specific scalar metrics.""" + self.iteration = iteration + common = self._common( + run_id=run_id, + session_id=session_id, + timestamp=_timestamp_utc(), + epoch=epoch, + batch=iteration, + batches_total=max(1, batches_total), + global_step=global_step, + ) + common = { + key: value for key, value in common.items() if key in FULL_COMMON_FIELDS + } + evaluation_id = uuid.uuid4().hex + rows = [] + for name, value in values.items(): + row = { + **common, + "metric": name, + "target": TOTAL_TARGET, + "value": float(value), + "is_primary": name == primary, + "learning_rate": learning_rate, + } + if validation: + row.update( + evaluation_id=evaluation_id, + evaluation_kind="initial" if initial else "iteration_end", + elapsed_seconds=elapsed_seconds, + ) + else: + row.update(window_batches=1, seconds_per_batch=elapsed_seconds) + rows.append(row) + full = self._validation_full if validation else self._training_full + condensed = self._validation if validation else self._training + if full is None: + full = _CsvAppender( + self.validation_full_path if validation else self.training_full_path, + VALIDATION_FIELDS if validation else TRAINING_FIELDS, + ) + if validation: + self._validation_full = full + else: + self._training_full = full + if condensed is None: + condensed = _CsvAppender( + self.validation_path if validation else self.training_path, + VALIDATION_CONDENSED_FIELDS + if validation + else TRAINING_CONDENSED_FIELDS, + ) + if validation: + self._validation = condensed + else: + self._training = condensed + full.append(rows, durable=validation) + fields = ( + VALIDATION_CONDENSED_FIELDS if validation else TRAINING_CONDENSED_FIELDS + ) + condensed.append( + [{key: row.get(key, "") for key in fields} for row in rows], + durable=validation, + ) + if primary in values: + logger.info( + "{} {} | Iteration {} | {}: {}", + self.phase, + "Validation" if validation else "Training", + iteration, + primary, + values[primary], + ) diff --git a/src/sequifier/training/metrics_service.py b/src/sequifier/training/metrics_service.py index f920f00c0..600a679ff 100644 --- a/src/sequifier/training/metrics_service.py +++ b/src/sequifier/training/metrics_service.py @@ -35,7 +35,17 @@ def __init__(self, context: Any, config: Any) -> None: config.project_root, config.model_name, context.rank, - class_share_columns=dataset.class_share_log_columns, + class_share_columns=list( + dict.fromkeys( + column + for phase in config.training_plan + for column in ( + phase.learning.datasets[name].class_share_log_columns + if name in phase.learning.datasets + else [] + ) + ) + ), dataset_name=name, dataset_count=dataset_count, validation_enabled=name in evaluated, @@ -46,6 +56,11 @@ def __init__(self, context: Any, config: Any) -> None: else {} ) + def set_phase(self, phase): + for writer in self.writers.values(): + writer.phase = phase.name + writer.algorithm = phase.algorithm + def record_training(self, result: TrainingMetrics) -> None: if self.context.rank != 0: return diff --git a/src/sequifier/training/optimization.py b/src/sequifier/training/optimization.py index e740fca3f..076a7e7ca 100644 --- a/src/sequifier/training/optimization.py +++ b/src/sequifier/training/optimization.py @@ -2,15 +2,14 @@ from __future__ import annotations -import inspect from copy import deepcopy from dataclasses import dataclass from typing import Any, Iterable -from torch import float16, nn +from torch import dtype, float16, nn from torch.amp.grad_scaler import GradScaler from torch.optim import Optimizer -from torch.optim.lr_scheduler import ReduceLROnPlateau +from torch.optim.lr_scheduler import OneCycleLR, ReduceLROnPlateau from sequifier.artifacts.run_checkpoint import OptimizationState from sequifier.config.optimizer_config import ( @@ -33,6 +32,18 @@ from sequifier.optimizers.plan import CompositeOptimizer, route_optimizer_parameters +def create_optimizer(spec, parameters, learning_rate, *, parameter_catalog=None): + """Build an optimizer with the same ownership rules for every learner.""" + if isinstance(spec, OptimizerPlan): + if parameter_catalog is None: + raise ValueError("An optimizer plan requires a parameter catalog.") + return CompositeOptimizer( + route_optimizer_parameters(spec, parameter_catalog, parameters) + ) + optimizer_class: Any = get_optimizer_class(spec.name) + return optimizer_class(parameters, lr=learning_rate, **spec.arguments) + + @dataclass(frozen=True) class SchedulerPolicy: step_on: str @@ -40,7 +51,7 @@ class SchedulerPolicy: @dataclass(frozen=True) class GradientPolicy: - accumulation_steps: int + accumulation_batches: int clip_norm: float | None @@ -85,35 +96,22 @@ def create( parameters: Iterable[nn.Parameter] | list[dict[str, Any]], *, parameter_catalog: ParameterCatalog | None = None, - phase_epochs: int | None = None, + accumulation_batches: int = 1, + layer_type_dtypes: dict[str, str] | None = None, + autocast_dtype: dtype | None = None, ) -> "OptimizationRuntime": - if isinstance(training.optimizer, OptimizerPlan): - if parameter_catalog is None: - raise ValueError("An optimizer plan requires a parameter catalog.") - optimizer = CompositeOptimizer( - route_optimizer_parameters( - training.optimizer, parameter_catalog, parameters - ) - ) - else: - optimizer_class = get_optimizer_class(training.optimizer.name) - optimizer = optimizer_class( - parameters, - lr=training.learning_rate, - **training.optimizer.arguments, - ) + optimizer = create_optimizer( + training.optimizer, + parameters, + training.learning_rate, + parameter_catalog=parameter_catalog, + ) scheduler_class = get_scheduler_class(training.scheduler.name) scheduler_arguments = dict(training.scheduler.arguments) if isinstance(training.optimizer, OptimizerPlan): scheduler_arguments = resolve_plan_scheduler_arguments( training.optimizer, training.scheduler ) - if ( - training.scheduler_step_on == "epoch" - and phase_epochs is not None - and "total_steps" in inspect.signature(scheduler_class).parameters - ): - scheduler_arguments["total_steps"] = phase_epochs scheduler = scheduler_class(optimizer, **scheduler_arguments) # GradScaler cannot unscale FP16 optimizer gradients. Explicitly pre-cast # FP16 parameters use ordinary, unscaled backward/optimizer steps. @@ -122,9 +120,9 @@ def create( for group in optimizer.param_groups for parameter in group["params"] ) - use_scaler = not has_fp16_parameters and bool( - training.layer_type_dtypes - and "float16" in training.layer_type_dtypes.values() + use_scaler = not has_fp16_parameters and ( + autocast_dtype == float16 + or bool(layer_type_dtypes and "float16" in layer_type_dtypes.values()) ) return cls( optimizer=optimizer, @@ -132,8 +130,8 @@ def create( scaler=GradScaler(device=device.split(":")[0], enabled=use_scaler), scheduler_policy=SchedulerPolicy(training.scheduler_step_on), gradient_policy=GradientPolicy( - accumulation_steps=training.accumulation_steps or 1, - clip_norm=training.gradient_clip, + accumulation_batches=accumulation_batches, + clip_norm=training.gradient_clip_norm, ), ) @@ -242,7 +240,7 @@ def complete_step( overflow = has_gradients and not skip and not applied if applied: self.optimizer_step += 1 - if self.scheduler_policy.step_on == "batch": + if self.scheduler_policy.step_on == "optimizer_step": if directive is None or not directive.skip_scheduler_step: self.step_scheduler() elif directive is not None and directive.skip_scheduler_step: @@ -257,7 +255,7 @@ def complete_step( global_batch_step=identity.global_batch_step, optimizer_step=self.optimizer_step, accumulation_index=identity.accumulation_index, - accumulation_steps=identity.accumulation_steps, + accumulation_batches=identity.accumulation_batches, rank=identity.rank, world_size=identity.world_size, ), @@ -275,7 +273,7 @@ def step_scheduler(self, metric: float | None = None) -> bool: self.skip_next_scheduler_step = False return False if ( - hasattr(self.scheduler, "total_steps") + isinstance(self.scheduler, OneCycleLR) and self.scheduler.last_epoch >= self.scheduler.total_steps ): return False diff --git a/src/sequifier/training/phase_runners.py b/src/sequifier/training/phase_runners.py new file mode 100644 index 000000000..87a1c44fa --- /dev/null +++ b/src/sequifier/training/phase_runners.py @@ -0,0 +1,132 @@ +"""Adapt resolved phases to native execution and exact recovery.""" + +from collections.abc import Callable +from dataclasses import dataclass + +import torch + +from sequifier.artifacts.phase import PhaseArtifacts +from sequifier.runtime.plan import RunIdentity + + +@dataclass(frozen=True) +class PortableGroup: + index: int + phases: tuple + incoming: str | None + resuming: bool + identity: RunIdentity + on_checkpoint: Callable + + +def supervised_group_config(config, phases, index, resume): + phase = phases[0] + from sequifier.config.components import ResumeConfig + + native = PhaseArtifacts.for_phase( + config.phase_directory, index, phase.name + ).model.parent + resume = ResumeConfig( + policy="required" if resume else "never", + checkpoint_path=str(native / "run.pt"), + ) + names = { + ref.dataset + for item in phases + for ref in [*item.sources, *item.evaluation_sources] + } + datasets = { + name: dataset + for name, dataset in config.dataset_training.items() + if name in names + } + resolved = config.model_copy( + update={ + "dataset_training": datasets, + "training_plan": list(phases), + "runtime_settings": config.runtime_settings.model_copy( + update={"resume": resume} + ), + "seed": phase.resolved_seed + if phase.resolved_seed is not None + else config.seed + index, + "model_name": f"{config.model_name}-phase-{index:03d}-{phase.name}", + "initialize_from": None, + "actor_initialization": "attach", + "curriculum_training": bool(phase.curriculum), + "curriculum_column": phase.curriculum.column if phase.curriculum else None, + "model_path": str(native / "native/model.pt"), + "report_path": str(native / "native/report.json"), + "phase_directory": str(native / "native/phases"), + } + ) + return resolved + + +def run_supervised_group(config, request: PortableGroup, backbone_revision): + from sequifier.helpers import configure_determinism, configure_logger + from sequifier.integration import IntegrationManager + from sequifier.runtime.builder import RunBuilder + from sequifier.runtime.context import ExecutionEnvironment + from sequifier.training.engine import TrainingEngine + + resolved = supervised_group_config( + config, request.phases, request.index, request.resuming + ) + torch.set_float32_matmul_precision( + resolved.runtime_settings.float32_matmul_precision + ) + configure_logger(config.project_root, resolved.model_name, 0) + configure_determinism(resolved.seed, resolved.runtime_settings.enforce_determinism) + environment = ExecutionEnvironment( + rank=0, + local_rank=0, + world_size=1, + device=torch.device(config.device), + distributed=False, + ) + integrations = IntegrationManager(rank=0, world_size=1, distributed=False) + run = RunBuilder().build( + resolved, + environment, + integrations, + initial_artifact=request.incoming, + phase_offset=request.index, + identity=request.identity, + on_checkpoint=request.on_checkpoint, + backbone_revision=backbone_revision, + ) + try: + TrainingEngine().run_group(run) + finally: + run.distributed.finalize() + + +def policy_group_config(config, phases, index, incoming, destination, resume): + from sequifier.config.training_resolution import policy_run_config + + return policy_run_config( + config, + phases, + incoming=incoming, + index=index, + destination=destination, + resuming=resume, + ) + + +def run_portable_policy_group(config, request: PortableGroup, prepared): + from sequifier.rl.learner import run_policy_group + + destination = PhaseArtifacts.for_phase( + config.phase_directory, request.index, request.phases[0].name + ).model + run = policy_group_config( + config, + request.phases, + request.index, + request.incoming, + destination, + request.resuming, + ) + run_policy_group(run, request.identity, request.on_checkpoint, prepared=prepared) diff --git a/src/sequifier/training/plan.py b/src/sequifier/training/plan.py new file mode 100644 index 000000000..04b50b0f9 --- /dev/null +++ b/src/sequifier/training/plan.py @@ -0,0 +1,106 @@ +"""Read-only phase views for native supervised execution.""" + +from dataclasses import dataclass +from typing import Literal + +from sequifier.config.train_config import ( + ResolvedDatasetTrainingSpec, + ResolvedSequifierConfig, + ResolvedTrainingPhase, + TrainingRuntimeSettings, +) + + +@dataclass(frozen=True) +class PhaseLoading: + training_batch_size: int + validation_batch_size: int + window_stride: int | None + curriculum_column: str | None + + def batch_size(self, split: Literal["training", "validation"]) -> int: + return ( + self.training_batch_size + if split == "training" + else self.validation_batch_size + ) + + +@dataclass(frozen=True) +class SupervisedPhaseView: + index: int + seed: int + loading: PhaseLoading + _spec: ResolvedTrainingPhase + + @classmethod + def from_spec( + cls, index: int, seed: int, spec: ResolvedTrainingPhase + ) -> "SupervisedPhaseView": + from sequifier.evaluation.compatibility import evaluation_batch_size + + return cls( + index=index, + seed=seed, + loading=PhaseLoading( + spec.batch_size, + evaluation_batch_size(spec), + spec.learning.window_stride, + spec.curriculum.column if spec.curriculum else None, + ), + _spec=spec.model_copy(deep=True), + ) + + @property + def optimizer_state(self): + return self._spec.optimizer_state + + def spec(self) -> ResolvedTrainingPhase: + """Give consumers their own copy of the configuration models.""" + return self._spec.model_copy(deep=True) + + def with_spec(self, spec: ResolvedTrainingPhase) -> "SupervisedPhaseView": + return self.from_spec(self.index, self.seed, spec) + + def objective_settings( + self, settings: TrainingRuntimeSettings + ) -> TrainingRuntimeSettings: + learning = self._spec.learning + return settings.model_copy( + update={ + "bert_spec": learning.bert_spec, + "next_occurrence_config": learning.next_occurrence_config, + "window_stride": self.loading.window_stride, + } + ).model_copy(deep=True) + + +@dataclass(frozen=True) +class SupervisedRunPlan: + phases: tuple[SupervisedPhaseView, ...] + + @classmethod + def from_config(cls, config: ResolvedSequifierConfig) -> "SupervisedRunPlan": + phases = [] + for index, spec in enumerate(config.training_plan): + if not isinstance(spec, ResolvedTrainingPhase): + raise ValueError( + "Native supervised training requires supervised phases" + ) + phases.append( + SupervisedPhaseView.from_spec( + index, + spec.resolved_seed + if spec.resolved_seed is not None + else config.seed + index, + spec, + ) + ) + return cls(tuple(phases)) + + +def dataset_learning_view( + dataset: ResolvedDatasetTrainingSpec, phase: ResolvedTrainingPhase +) -> ResolvedDatasetTrainingSpec: + learning = phase.learning.datasets.get(dataset.name) + return dataset.model_copy(update=learning.model_dump()) if learning else dataset diff --git a/src/sequifier/training/runtime.py b/src/sequifier/training/runtime.py index 40aa3c1d4..aa97b575f 100644 --- a/src/sequifier/training/runtime.py +++ b/src/sequifier/training/runtime.py @@ -3,7 +3,6 @@ from __future__ import annotations import random -import warnings from collections.abc import Iterator, Sequence from dataclasses import dataclass, field, replace from typing import Any, Literal, Optional, Protocol, runtime_checkable @@ -14,15 +13,19 @@ from torch.utils.data import DataLoader from sequifier.config.train_config import ( - DatasetFreezingSpecModel, ResolvedDatasetTrainingSpec, + ResolvedModelInterface, ResolvedSequifierConfig, ResolvedTrainingPhase, ResolvedTrainingSource, + SelectedObjectiveConfig, dataset_part_view, + interface_build_view, ) from sequifier.helpers import normalize_path from sequifier.io.batch import SequifierBatch +from sequifier.io.folder_dataset import FolderDataset +from sequifier.io.memory_budget import DataLoadingBudget from sequifier.io.sequifier_dataset_from_file import SequifierDatasetFromFile from sequifier.io.sequifier_dataset_from_folder_parquet import ( SequifierDatasetFromFolderParquet, @@ -36,9 +39,14 @@ ) from sequifier.model.network import ComposableTransformerNetwork from sequifier.model.parameter_catalog import ParameterCatalog -from sequifier.model.parameter_groups import semantic_parameter_groups -from sequifier.objectives import Objective +from sequifier.objectives import CausalObjective, Objective, create_objective from sequifier.training.accurate_cross_entropy import AccurateCrossEntropyLoss +from sequifier.training.plan import ( + PhaseLoading, + SupervisedPhaseView, + SupervisedRunPlan, + dataset_learning_view, +) from sequifier.typechecking import beartype @@ -49,23 +57,14 @@ class RuntimeBatch: batch: SequifierBatch -def _training_plan_uses_part( - config: ResolvedSequifierConfig, dataset_name: str, part_name: str -) -> bool: - """Return whether any training phase selects this dataset part.""" - return any( - source.dataset == dataset_name - and (source.part is None or source.part == part_name) - for phase in config.training_plan - for source in phase.sources - ) - - @dataclass class PartLoaderFactory: config: ResolvedSequifierConfig dataset_name: str part_name: str + memory_budget: DataLoadingBudget + curriculum_eligible: bool + _loading: PhaseLoading | None = field(default=None, init=False) loaders: dict[Literal["training", "validation"], DataLoader] = field( default_factory=dict, init=False, @@ -75,24 +74,24 @@ class PartLoaderFactory: init=False, ) - @beartype - def build(self, split: Literal["training", "validation"]) -> DataLoader: - global_spec = self.config.global_training - if split in self.loaders: - return self.loaders[split] - - dataset_config: Any = dataset_part_view( - self.config, self.dataset_name, self.part_name + def _build_dataset( + self, loading: PhaseLoading, split: Literal["training", "validation"] + ): + global_spec = self.config.runtime_settings + curriculum_column = ( + loading.curriculum_column + if split == "training" and self.curriculum_eligible + else None ) - if split != "training" or not _training_plan_uses_part( - self.config, self.dataset_name, self.part_name - ): - dataset_config = replace( - dataset_config, - curriculum_training=False, - curriculum_column=None, - ) - part = self.config.dataset_training[self.dataset_name].parts[self.part_name] + dataset_config = dataset_part_view( + self.config, + self.dataset_name, + self.part_name, + batch_size=loading.batch_size(split), + window_stride=loading.window_stride, + curriculum_column=curriculum_column, + ) + part = dataset_config.part path = ( part.training_data_path if split == "training" @@ -120,14 +119,14 @@ def build(self, split: Literal["training", "validation"]) -> DataLoader: ), ) dataset = SequifierDatasetFromFile(path, dataset_config, shuffle=shuffle) - elif global_spec.read_format == "pt": + elif dataset_config.read_format == "pt": cls = ( SequifierDatasetFromFolderPt if global_spec.load_full_data_to_ram else SequifierDatasetFromFolderPtLazy ) dataset = cls(path, dataset_config, shuffle=shuffle) - elif global_spec.read_format == "parquet": + elif dataset_config.read_format == "parquet": cls = ( SequifierDatasetFromFolderParquet if global_spec.load_full_data_to_ram @@ -136,8 +135,30 @@ def build(self, split: Literal["training", "validation"]) -> DataLoader: dataset = cls(path, dataset_config, shuffle=shuffle) else: raise ValueError( - f"Folder parts do not support read_format={global_spec.read_format!r}" + f"Folder parts do not support read_format={dataset_config.read_format!r}" ) + self.memory_budget.validate_dataset( + f"{self.dataset_name}.{self.part_name}.{split}", + dataset, + global_spec.num_workers, + ) + return dataset + + def count_training_batches(self, phase: SupervisedPhaseView) -> int: + # The dataset emits pre-batched items and the loader uses batch_size=None. + return len(self._build_dataset(phase.loading, "training")) + + @beartype + def build( + self, phase: SupervisedPhaseView, split: Literal["training", "validation"] + ) -> DataLoader: + if self._loading != phase.loading: + self.loaders.clear() + self._loading = phase.loading + if split in self.loaders: + return self.loaders[split] + dataset = self._build_dataset(phase.loading, split) + global_spec = self.config.runtime_settings generator = self.generators.get(split) if generator is None: generator = torch.Generator().manual_seed( @@ -192,11 +213,15 @@ def source_metadata(self) -> dict[str, Any]: "validation_data_path": part.validation_data_path, } - def loader(self, split: Literal["training", "validation"]) -> DataLoader: - return self.factory.build(split) + def loader( + self, phase: SupervisedPhaseView, split: Literal["training", "validation"] + ) -> DataLoader: + return self.factory.build(phase, split) - def iter_loader(self, split: Literal["training", "validation"]) -> Iterator[Any]: - loader = self.loader(split) + def iter_loader( + self, phase: SupervisedPhaseView, split: Literal["training", "validation"] + ) -> Iterator[Any]: + loader = self.loader(phase, split) generator = self.factory.generators[split] if split in self._restore_from_iterator_start: generator.set_state(self.iterator_start_states[split]) @@ -235,8 +260,12 @@ def load_loader_state_dict( str(key): value for key, value in state.get("iterator_start_states", {}).items() } + # Mid-epoch recovery replays training batches. Validation runs to completion + # before a checkpoint and starts its next iterator from the saved generator. self._restore_from_iterator_start = ( - set(self.iterator_start_states) if resume_from_iterator_start else set() + {"training"} & self.iterator_start_states.keys() + if resume_from_iterator_start + else set() ) @@ -251,6 +280,7 @@ class DatasetRuntime: name: str config: ResolvedDatasetTrainingSpec interface_name: str + interface: ResolvedModelInterface objective: Objective parts: dict[str, DatasetPartRuntime] criteria: dict[str, nn.Module] @@ -260,10 +290,6 @@ class DatasetRuntime: runtime_metadata: Any metrics: DatasetMetrics = field(default_factory=DatasetMetrics) - @property - def freeze_policy(self) -> DatasetFreezingSpecModel: - return self.config.freeze - class DatasetRuntimeRegistry: def __init__(self, datasets: dict[str, DatasetRuntime]) -> None: @@ -291,8 +317,8 @@ def items(self): @dataclass(frozen=True) -class DatasetFreezingPlan: - """Dataset freezing selections expressed as stable parameter catalog IDs.""" +class PhaseTrainabilityPlan: + """Phase-owned update masks expressed as stable parameter catalog IDs.""" parameter_ids_by_dataset: dict[str, frozenset[str]] @@ -302,24 +328,49 @@ def resolve( cls, config: ResolvedSequifierConfig, network: ComposableTransformerNetwork, - ) -> DatasetFreezingPlan: + phase: ResolvedTrainingPhase, + ) -> PhaseTrainabilityPlan: catalog = ParameterCatalog(network) parameter_id_by_identity = { id(catalog.parameter(descriptor.parameter_id)): descriptor.parameter_id for descriptor in catalog.descriptors() } parameter_ids_by_dataset = {} - for name, dataset in config.dataset_training.items(): - frozen = frozen_parameter_ids( - network, dataset.model_interface, dataset.freeze + active_datasets = {ref.dataset for ref in phase.sources} + phase_parameters = {id(p) for p in network.backbone.parameters()} + for name in active_datasets: + phase_parameters.update( + id(p) + for p in network.interfaces[ + config.dataset_training[name].model_interface + ].parameters() ) + default_mask = phase.trainability.policy("model").resolve( + network, eligible=phase_parameters + ) + for name, dataset in config.dataset_training.items(): route = network.interfaces[dataset.model_interface] active_parameter_ids = { id(parameter) for module in (network.backbone, route) for parameter in module.parameters() } - if frozen >= active_parameter_ids: + override = phase.trainability.by_dataset.get(name) + if name not in active_datasets: + frozen = frozenset() + elif override is not None and "model" in override.roles: + frozen = override.roles["model"].resolve( + network, eligible=active_parameter_ids + ) + else: + frozen = default_mask & active_parameter_ids + trainable = { + id(p) + for module in (network.backbone, route) + for p in module.parameters() + if p.requires_grad + } + if name in active_datasets and not trainable - frozen: raise ValueError(f"Dataset {name!r} leaves no trainable parameters") missing = frozen.difference(parameter_id_by_identity) if missing: @@ -331,22 +382,6 @@ def resolve( ) return cls(parameter_ids_by_dataset) - @property - def permanently_frozen_parameter_ids(self) -> frozenset[str]: - selections = tuple( - set(parameter_ids) - for parameter_ids in self.parameter_ids_by_dataset.values() - ) - if not selections: - return frozenset() - return frozenset(set.intersection(*selections)) - - @beartype - def apply_permanent(self, network: ComposableTransformerNetwork) -> None: - catalog = ParameterCatalog(network) - for parameter_id in self.permanently_frozen_parameter_ids: - catalog.parameter(parameter_id).requires_grad_(False) - @beartype def bind(self, network: ComposableTransformerNetwork) -> dict[str, frozenset[int]]: catalog = ParameterCatalog(network) @@ -363,33 +398,32 @@ class TrainingSourceRuntime: config: ResolvedTrainingSource dataset: DatasetRuntime part_names: tuple[str, ...] - loaders: dict[str, list[tuple[str, DataLoader]]] = field(default_factory=dict) + phase: SupervisedPhaseView + + @property + def dataset_config(self) -> ResolvedDatasetTrainingSpec: + return self.dataset.config @beartype def _loaders(self, split: Literal["training", "validation"]): - if split not in self.loaders: - self.loaders[split] = [ - ( - part_name, - self.dataset.parts[part_name].loader(split), - ) - for part_name in self.part_names - ] - return self.loaders[split] + return [ + (part_name, self.dataset.parts[part_name].loader(self.phase, split)) + for part_name in self.part_names + ] @beartype def set_epoch(self, epoch: int, split: Literal["training", "validation"]): for _, loader in self._loaders(split): - set_epoch = getattr(loader.dataset, "set_epoch", None) - if callable(set_epoch): - set_epoch(epoch) + dataset = loader.dataset + if isinstance(dataset, (SequifierDatasetFromFile, FolderDataset)): + dataset.set_epoch(epoch) @beartype def iter_batches( self, split: Literal["training", "validation"] = "training" ) -> Iterator[RuntimeBatch]: for part_name, _ in self._loaders(split): - for batch in self.dataset.parts[part_name].iter_loader(split): + for batch in self.dataset.parts[part_name].iter_loader(self.phase, split): if not isinstance(batch, SequifierBatch): raise TypeError( "Sequifier loaders must yield SequifierBatch, got " @@ -407,7 +441,9 @@ class ScheduledSource(Protocol): """Structural source contract consumed by the deterministic scheduler.""" config: ResolvedTrainingSource - dataset: DatasetRuntime + + @property + def dataset_config(self) -> ResolvedDatasetTrainingSpec: ... @beartype def set_epoch( @@ -420,120 +456,13 @@ def iter_batches( ) -> Iterator[Any]: ... -@beartype -def _policy_parameter_ids(module: nn.Module, policy: Any, usage: str) -> set[int]: - if policy.freeze is None and policy.freezing_except is None: - return set() - groups = semantic_parameter_groups(module) - configured = set( - policy.freeze if policy.freeze is not None else policy.freezing_except or [] - ) - matched = {group for group in configured if groups.get(group)} - unmatched = configured - matched - if unmatched and policy.freezing_except is not None: - raise ValueError( - f"{usage} freezing_except groups matched no parameters: " - f"{', '.join(sorted(unmatched))}" - ) - if unmatched: - warnings.warn( - f"{usage} freezing groups matched no parameters: {sorted(unmatched)}", - stacklevel=2, - ) - selected = { - id(parameter) for group in configured for parameter in groups.get(group, ()) - } - if policy.freeze is not None: - return selected - return {id(parameter) for parameter in module.parameters()} - selected - - -def _ingestion_policy_ids( - module, - policy, - usage, - static_encoder=None, - image_encoder=None, - prefix_projection=None, -): - decisions = {} - root = nn.Module() - root.add_module("ingestion", module) - if static_encoder is not None: - root.add_module("static_condition_encoder", static_encoder) - if image_encoder is not None: - root.add_module("preconditioning_image_encoder", image_encoder) - if prefix_projection is not None: - root.add_module("prefix_projection", prefix_projection) - root_frozen = _policy_parameter_ids(root, policy, usage) - - def visit(node, selection, inherited, path, *, is_root=False): - own = {id(p) for p in node.parameters()} - if is_root: - local = {identity: identity in root_frozen for identity in own} - elif selection is not None and ( - selection.freeze is not None or selection.freezing_except is not None - ): - frozen = _policy_parameter_ids(node, selection, path) - local = {identity: identity in frozen for identity in own} - else: - local = {identity: inherited.get(identity, False) for identity in own} - branches = getattr(node, "branches", {}) - child_policies = getattr(selection, "branches", {}) - unknown = set(child_policies) - set(branches) - if unknown: - raise ValueError(f"{path}: unknown freezing branches {sorted(unknown)}") - child_ids = {id(p) for child in branches.values() for p in child.parameters()} - for identity in own - child_ids: - decision = local[identity] - if identity in decisions and decisions[identity] != decision: - raise ValueError( - f"{path}: conflicting freezing decisions for a shared parameter" - ) - decisions[identity] = decision - for name, child in branches.items(): - visit(child, child_policies.get(name), local, f"{path}.branches.{name}") - - visit(module, policy, {}, usage, is_root=True) - frozen = {identity for identity, decision in decisions.items() if decision} - for extra in (static_encoder, image_encoder, prefix_projection): - if extra is not None: - frozen.update(root_frozen.intersection(id(p) for p in extra.parameters())) - return frozen - - -@beartype -def frozen_parameter_ids( - network: ComposableTransformerNetwork, - interface_name: str, - freeze: DatasetFreezingSpecModel, -) -> frozenset[int]: - route = network.interfaces[interface_name] - frozen = _policy_parameter_ids(network.backbone, freeze.backbone, "backbone") - frozen.update( - _ingestion_policy_ids( - route.ingestion, - freeze.ingestion, - "ingestion", - route.static_condition_encoder, - route.preconditioning_image_encoder, - route.prefix_projection, - ) - ) - if freeze.ingestion_adapter: - frozen.update( - id(parameter) for parameter in route.ingestion_adapter.parameters() - ) - frozen.update(_policy_parameter_ids(route.decoder, freeze.decoder, "decoder")) - return frozenset(frozen) - - @beartype def _criterion_modules( - dataset: ResolvedDatasetTrainingSpec, device: torch.device + dataset: ResolvedDatasetTrainingSpec, + interface: ResolvedModelInterface, + device: torch.device, ) -> dict[str, nn.Module]: modules = {} - interface = dataset.interface for target in interface.target_columns: if target in interface.categorical_hashing: if dataset.class_weights and target in dataset.class_weights: @@ -552,7 +481,7 @@ def _criterion_modules( criterion_class = ( AccurateCrossEntropyLoss if dataset.criterion[target] == "AccurateCrossEntropyLoss" - else getattr(torch.nn, dataset.criterion[target]) + else vars(torch.nn)[dataset.criterion[target]] ) kwargs: dict[str, Any] = {"reduction": "none"} if dataset.class_weights is not None and target in dataset.class_weights: @@ -569,62 +498,80 @@ def _criterion_modules( return modules -@beartype +def build_dataset_parts( + config: ResolvedSequifierConfig, + plan: SupervisedRunPlan, + memory_budget: DataLoadingBudget, +) -> dict[str, dict[str, DatasetPartRuntime]]: + training_parts = { + (source.dataset, part) + for phase in plan.phases + for source in phase.spec().sources + for part in source.parts + } + return { + name: { + part: DatasetPartRuntime( + part, + PartLoaderFactory( + config, name, part, memory_budget, (name, part) in training_parts + ), + ) + for part in dataset.parts + } + for name, dataset in config.dataset_training.items() + } + + def build_dataset_runtimes( config: ResolvedSequifierConfig, - network: ComposableTransformerNetwork, + phase: SupervisedPhaseView, device: torch.device, *, - objectives: dict[str, Objective], + parts: dict[str, dict[str, DatasetPartRuntime]], + metrics: dict[str, DatasetMetrics], runtime_metadata: Any, - freezing_plan: DatasetFreezingPlan | None = None, + masks: dict[str, frozenset[int]], ) -> DatasetRuntimeRegistry: - if freezing_plan is None: - freezing_plan = DatasetFreezingPlan.resolve(config, network) - freezing_plan.apply_permanent(network) - frozen_parameter_ids_by_dataset = freezing_plan.bind(network) - + spec = phase.spec() + settings = phase.objective_settings(config.runtime_settings) runtimes = {} - for name, dataset in config.dataset_training.items(): - frozen = frozen_parameter_ids_by_dataset[name] - route = network.interfaces[dataset.model_interface] - active_parameter_ids = { - id(parameter) - for module in (network.backbone, route) - for parameter in module.parameters() - } - if frozen >= active_parameter_ids: - raise ValueError(f"Dataset {name!r} leaves no trainable parameters") + for name, base in config.dataset_training.items(): + dataset = dataset_learning_view(base, spec) + interface = config.interface(dataset.model_interface) + objective_view = SelectedObjectiveConfig( + **vars(interface_build_view(config, interface)), runtime_settings=settings + ) + objective = ( + CausalObjective(objective_view) + if name not in spec.learning.datasets + and objective_view.training_objective == "next_occurrence" + else create_objective(objective_view) + ) runtimes[name] = DatasetRuntime( name=name, config=dataset, interface_name=dataset.model_interface, - objective=objectives[dataset.model_interface], - parts={ - part_name: DatasetPartRuntime( - part_name, PartLoaderFactory(config, name, part_name) - ) - for part_name in dataset.parts - }, - criteria=_criterion_modules(dataset, device), + interface=interface, + objective=objective, + parts=parts[name], + criteria=_criterion_modules(dataset, interface, device), loss_weights=dataset.loss_weights, class_weights=dataset.class_weights, - frozen_parameter_ids=frozen, + frozen_parameter_ids=masks[name], runtime_metadata=runtime_metadata.interfaces[dataset.model_interface], + metrics=metrics[name], ) return DatasetRuntimeRegistry(runtimes) -@beartype def build_source_runtime( source: ResolvedTrainingSource, datasets: DatasetRuntimeRegistry, + phase: SupervisedPhaseView, ) -> TrainingSourceRuntime: dataset = datasets.resolve(source.dataset) - part_names = ( - (source.part,) if source.part is not None else tuple(dataset.config.parts) - ) - return TrainingSourceRuntime(source, dataset, part_names) + return TrainingSourceRuntime(source, dataset, source.parts, phase) class SourceScheduler: @@ -633,16 +580,13 @@ class SourceScheduler: @beartype def __init__( self, - phase: ResolvedTrainingPhase, + phase: SupervisedPhaseView, sources: Sequence[ScheduledSource], - *, - seed: int, - phase_index: int = 0, ) -> None: - self.phase = phase + self.phase = phase.spec() self.sources = sources - self.phase_index = phase_index - self.rng = random.Random(seed) + self.phase_index = phase.index + self.rng = random.Random(phase.seed) self.round_robin_cursor = 0 self.epoch = 0 self.consumed_by_source = [0 for _ in sources] @@ -685,7 +629,9 @@ def _choose(self, active: list[int]) -> int: if self.phase.selection == "weighted_random": return self.rng.choices( active, - weights=[self.phase.sources[index].weight for index in active], + weights=[ + self.sources[index].config.selection_weight for index in active + ], k=1, )[0] for _ in range(len(self.sources)): @@ -733,7 +679,7 @@ def iter_epoch(self, epoch: int) -> Iterator[RuntimeBatch]: dataset_indices = { name: index for index, name in enumerate( - dict.fromkeys(source.dataset.config.name for source in self.sources) + dict.fromkeys(source.dataset_config.name for source in self.sources) ) } while active: @@ -747,12 +693,12 @@ def iter_epoch(self, epoch: int) -> Iterator[RuntimeBatch]: part_index = ( -1 if source.config.part is None - else list(source.dataset.config.parts).index(source.config.part) + else list(source.dataset_config.parts).index(source.config.part) ) record = ( self.phase_index, selected, - dataset_indices[source.dataset.config.name], + dataset_indices[source.dataset_config.name], part_index, source.config.batches_per_selection, ) @@ -774,13 +720,13 @@ def iter_epoch(self, epoch: int) -> Iterator[RuntimeBatch]: expected_part = ( -1 if selected_source.config.part is None - else list(selected_source.dataset.config.parts).index( + else list(selected_source.dataset_config.parts).index( selected_source.config.part ) ) if ( record[2] - != dataset_indices[selected_source.dataset.config.name] + != dataset_indices[selected_source.dataset_config.name] or record[3] != expected_part ): raise RuntimeError( @@ -804,3 +750,19 @@ def iter_epoch(self, epoch: int) -> Iterator[RuntimeBatch]: yield batch if self.pending_batches == 0: self.pending_source = None + + +def validate_phase_learning(config, plan: SupervisedRunPlan, device): + """Validate every phase's losses before native outputs are created.""" + for view in plan.phases: + phase = view.spec() + for name, learning in phase.learning.datasets.items(): + dataset = dataset_learning_view(config.dataset_training[name], phase) + interface = config.interface(dataset.model_interface) + learning.validate_targets(interface.target_columns, f"{phase.name}: {name}") + for column in dataset.class_share_log_columns: + if interface.target_column_types.get(column) != "categorical": + raise ValueError( + f"{phase.name}: class share columns must be categorical targets" + ) + _criterion_modules(dataset, interface, device) diff --git a/src/sequifier/training/scheduler_budgets.py b/src/sequifier/training/scheduler_budgets.py new file mode 100644 index 000000000..70bfe92ee --- /dev/null +++ b/src/sequifier/training/scheduler_budgets.py @@ -0,0 +1,115 @@ +"""Preflight scheduler budgets using the same source ordering as training.""" + +from dataclasses import dataclass + +import torch + +from sequifier.config.train_config import ( + ResolvedDatasetTrainingSpec, + ResolvedTrainingSource, +) +from sequifier.io.batch import SequifierBatch +from sequifier.training.plan import SupervisedRunPlan +from sequifier.training.runtime import RuntimeBatch, SourceScheduler + + +@dataclass +class _CountedSource: + """Supply batch identities to the scheduler without consuming training data.""" + + config: ResolvedTrainingSource + dataset_config: ResolvedDatasetTrainingSpec + batches: int + + def set_epoch(self, epoch, split): + pass + + def iter_batches(self, split="training"): + batch = RuntimeBatch( + self.dataset_config.name, + self.config.part or "", + SequifierBatch( + {}, + {}, + { + "attention_valid_mask": torch.empty(0, dtype=torch.bool), + "target_valid_mask": torch.empty(0, dtype=torch.bool), + }, + ), + ) + for _ in range(self.batches): + yield batch + + +def _phase_update_budget(config, parts, view): + phase = view.spec() + counts = {} + sources = [] + for source in phase.sources: + dataset = config.dataset_training[source.dataset] + batches = 0 + for part in source.parts: + key = (source.dataset, part) + if key not in counts: + counts[key] = parts[source.dataset][ + part + ].factory.count_training_batches(view) + batches += counts[key] + sources.append(_CountedSource(source, dataset, batches)) + scheduler = SourceScheduler(view, sources) + updates = 0 + for epoch in range(1, phase.epochs + 1): + active_dataset, accumulated = None, 0 + for batch in scheduler.iter_epoch(epoch): + if active_dataset != batch.dataset: + updates += int(accumulated > 0) + accumulated = 0 + active_dataset = batch.dataset + accumulated += 1 + if accumulated == phase.accumulation_batches: + updates += 1 + accumulated = 0 + updates += int(accumulated > 0) + return updates + + +def resolve_scheduler_budgets(plan, config, parts): + """Resolve applied-update budgets, including dataset-boundary flushes.""" + views = list(plan.phases) + phases = [view.spec() for view in views] + phase_budgets = {} + for index, phase in enumerate(phases): + role = phase.optimization["model"] + if ( + role.scheduler.name != "OneCycleLR" + or role.scheduler_step_on != "optimizer_step" + or role.scheduler.arguments.get("total_steps") is not None + ): + continue + start = index + while start > 0 and phases[start].optimizer_state == "continue": + start -= 1 + end = index + 1 + while end < len(phases) and phases[end].optimizer_state == "continue": + end += 1 + for selected in range(start, end): + if selected not in phase_budgets: + phase_budgets[selected] = _phase_update_budget( + config, parts, views[selected] + ) + budget = sum(phase_budgets[selected] for selected in range(start, end)) + if budget <= 0: + raise ValueError( + f"{phase.name}: OneCycleLR requires a positive optimizer-update budget" + ) + scheduler = role.scheduler.model_copy( + update={"arguments": {**role.scheduler.arguments, "total_steps": budget}} + ) + optimization = { + **phase.optimization, + "model": role.model_copy(update={"scheduler": scheduler}), + } + views[index] = views[index].with_spec( + phase.model_copy(update={"optimization": optimization}) + ) + return SupervisedRunPlan(tuple(views)) diff --git a/src/sequifier/training/state.py b/src/sequifier/training/state.py index ea0c9893b..8e3ae9685 100644 --- a/src/sequifier/training/state.py +++ b/src/sequifier/training/state.py @@ -2,13 +2,14 @@ from __future__ import annotations -import uuid from copy import deepcopy from dataclasses import asdict, dataclass, field from typing import Any, Mapping from torch import Tensor +from sequifier.runtime.plan import new_run_id + @dataclass(frozen=True) class RunStateSnapshot: @@ -17,9 +18,16 @@ class RunStateSnapshot: @dataclass class RunState: - run_id: str = field(default_factory=lambda: uuid.uuid4().hex) - session_id: str = field(default_factory=lambda: uuid.uuid4().hex) - phase_index: int = 0 + run_id: str = field(default_factory=new_run_id) + session_id: str = field(default_factory=new_run_id) + stopping_state: dict[str, Any] = field(default_factory=dict) + phase_commits: list[dict[str, Any]] = field(default_factory=list) + last_validation_metric: float | None = None + best_optimizer_step: int | None = None + plan_stopped: bool = False + initial_evaluated_phases: list[int] = field(default_factory=list) + phase_index: int = 0 # Position within the native plan. + phase_offset: int = 0 # Original index of the first phase in the plan. phase_epoch: int = 0 phase_epoch_complete: bool = False source_index: int = 0 @@ -27,6 +35,7 @@ class RunState: iterator_positions: dict[str, int] = field(default_factory=dict) epoch: int = 0 batch: int = 0 + last_snapshot_batch_step: int = 0 global_batch_step: int = 0 optimizer_step: int = 0 accumulation_index: int = 0 diff --git a/src/sequifier/training_api.py b/src/sequifier/training_api.py index 4e6d88874..691fc697f 100644 --- a/src/sequifier/training_api.py +++ b/src/sequifier/training_api.py @@ -1,7 +1,16 @@ """Stable update-aware primitives for training integrations.""" +from sequifier.config.ownership import RunSettings +from sequifier.config.parameter_selection import ( + ParameterSelector, + RoleTrainability, + TrainabilityConfig, +) +from sequifier.config.phase_controls import Cadence, StoppingConfig +from sequifier.config.train_config import ResolvedSequifierConfig, SequifierConfig from sequifier.integration.contexts import StepIdentity from sequifier.integration.controls import TrainingDirective +from sequifier.training.coordinator import LearningRunResult, run_learning_plan from sequifier.training.distributed_strategy import ( DistributedDataParallelStrategy, DistributedStrategy, @@ -12,6 +21,12 @@ from sequifier.training.state import RunState __all__ = [ + "RunSettings", + "ParameterSelector", + "RoleTrainability", + "TrainabilityConfig", + "Cadence", + "StoppingConfig", "DistributedDataParallelStrategy", "DistributedStrategy", "FullyShardedStrategy", @@ -21,4 +36,8 @@ "StepIdentity", "StepResult", "TrainingDirective", + "SequifierConfig", + "ResolvedSequifierConfig", + "LearningRunResult", + "run_learning_plan", ] diff --git a/src/sequifier/trajectory/__init__.py b/src/sequifier/trajectory/__init__.py new file mode 100644 index 000000000..e1d14b8e7 --- /dev/null +++ b/src/sequifier/trajectory/__init__.py @@ -0,0 +1,6 @@ +"""Versioned numeric trajectory artifacts.""" + +from sequifier.trajectory.schema import Trajectory, validate_trajectory +from sequifier.trajectory.store import load_trajectory, save_trajectory + +__all__ = ["Trajectory", "validate_trajectory", "load_trajectory", "save_trajectory"] diff --git a/src/sequifier/trajectory/archives.py b/src/sequifier/trajectory/archives.py new file mode 100644 index 000000000..a0f9d5dcf --- /dev/null +++ b/src/sequifier/trajectory/archives.py @@ -0,0 +1,47 @@ +"""Collection archive envelopes with algorithm-owned payloads.""" + +import json +from pathlib import Path + +from sequifier.contract_versions import ContractVersions, require_contract_version + + +def collection_archive(algorithm, payload): + return { + "format": "sequifier.collection-archive", + "version": ContractVersions.COLLECTION_ARCHIVE, + "algorithm": algorithm, + "collection": payload, + } + + +def read_collection_archive(path, algorithm=None): + payload = json.loads(Path(path).read_text(encoding="utf-8")) + require_contract_version( + "collection archive envelope", + payload.get("version") if isinstance(payload, dict) else None, + ContractVersions.COLLECTION_ARCHIVE, + ) + if payload.get("format") != "sequifier.collection-archive": + raise ValueError("Expected a Sequifier collection archive envelope") + actual = payload.get("algorithm") + if algorithm is not None and actual != algorithm: + raise ValueError(f"Expected {algorithm} collection archive") + contracts = { + "ppo": ("sequifier.ppo-collection", ContractVersions.PPO_COLLECTION), + "grpo": ("sequifier.grpo-archive", ContractVersions.GRPO_ARCHIVE), + } + collection = payload.get("collection") + if ( + not isinstance(actual, str) + or actual not in contracts + or not isinstance(collection, dict) + ): + raise ValueError( + "Collection archive requires a known algorithm and collection mapping" + ) + kind, version = contracts[actual] + require_contract_version(kind, collection.get("version"), version) + if collection.get("format") != kind: + raise ValueError(f"Expected {kind} collection payload") + return collection diff --git a/src/sequifier/trajectory/export.py b/src/sequifier/trajectory/export.py new file mode 100644 index 000000000..5771bed3d --- /dev/null +++ b/src/sequifier/trajectory/export.py @@ -0,0 +1,251 @@ +"""Atomically export canonical episodes as checksummed raw Parquet shards.""" + +import json +import os +import shutil +import sys +import tempfile +from pathlib import Path + +from sequifier.contract_versions import ContractVersions +from sequifier.io.actions import action_component_key, action_latent_key +from sequifier.io.sequence import TransitionColumns +from sequifier.trajectory.importers import source_episodes +from sequifier.trajectory.sources import file_checksum, source_paths +from sequifier.trajectory.tables import ( + ACTION_RECORD, + EPISODE_ID_COLUMN, + GROUP_COLUMN, + PREVIOUS_ACTION_RECORD, + REWARD_COLUMN, + SUCCESSOR_PREFIX, + TRANSITION_FLAGS, +) + +MAX_FLOAT64_INTEGER = int(sys.float_info.max) + + +def _is_integer(value: object) -> bool: + return isinstance(value, int) and not isinstance(value, bool) + + +def _float64_value(value, dtype, column, pa): + if value is None: + return value + if pa.types.is_float64(dtype) and _is_integer(value): + if abs(value) > MAX_FLOAT64_INTEGER: + raise ValueError(f"{column}: integer exceeds the finite float64 range") + converted = float(value) + if converted != value: + raise ValueError( + f"{column}: integer cannot be represented exactly as float64" + ) + return converted + if pa.types.is_list(dtype) or pa.types.is_large_list(dtype): + return [_float64_value(item, dtype.value_type, column, pa) for item in value] + if pa.types.is_struct(dtype): + return { + name: _float64_value(item, dtype.field(name).type, f"{column}.{name}", pa) + for name, item in value.items() + } + return value + + +def _checked_array(entries, dtype, column, pa): + dtype = dtype if dtype is not None else pa.infer_type(entries) + return pa.array( + [_float64_value(value, dtype, column, pa) for value in entries], type=dtype + ) + + +def _cast_table(table, schema, pa): + arrays = [] + for field in schema: + column = table[field.name] + arrays.append( + column + if column.type == field.type + else _checked_array(column.to_pylist(), field.type, field.name, pa) + ) + return pa.Table.from_arrays(arrays, schema=schema) + + +def _action_type(field, pa): + if field.type == "real": + return pa.float64() + if not field.values: + raise ValueError("categorical sequence actions require declared values") + if all(isinstance(label, str) for label in field.values): + return pa.string() + if any(isinstance(label, str) for label in field.values): + raise ValueError( + "categorical sequence actions cannot mix string and numeric labels" + ) + if all(_is_integer(label) for label in field.values): + if min(field.values) >= 0 and max(field.values) >= 2**63: + return pa.uint64() + return pa.int64() + return pa.float64() + + +def import_sequences(config) -> Path: + """Commit canonical columnar episodes as checksummed raw Parquet shards.""" + import pyarrow as pa + import pyarrow.parquet as pq + + source = config.resolved_source() + policy = config.resolved_policy() + destination = (Path(config.project_root) / config.output_path).resolve() + if destination.exists(): + raise ValueError( + "sequence import output_path must name a new dataset directory" + ) + destination.parent.mkdir(parents=True, exist_ok=True) + fields = policy.observation_columns + policy.static_columns + successor_columns = [SUCCESSOR_PREFIX + name for name in policy.observation_columns] + reserved = { + "sequenceId", + "itemPosition", + EPISODE_ID_COLUMN, + ACTION_RECORD, + PREVIOUS_ACTION_RECORD, + GROUP_COLUMN, + REWARD_COLUMN, + *TRANSITION_FLAGS, + *successor_columns, + } + if set(fields) & reserved: + raise ValueError( + "policy feature names collide with sequence import metadata columns" + ) + references = source_episodes(source, policy) + if not references: + raise ValueError("sequence import requires at least one episode") + action_type = pa.struct( + [ + pa.field(name, _action_type(field, pa)) + for name, field in policy.actions.items() + ] + + [ + pa.field(action_latent_key(name, policy), pa.float64()) + for name, field in policy.actions.items() + if field.type == "real" + ] + + [ + pa.field(action_component_key(name, policy), pa.int64()) + for name, field in policy.actions.items() + if field.type == "real" + ] + ) + types = { + REWARD_COLUMN: pa.float64(), + **{name: pa.bool_() for name in TRANSITION_FLAGS}, + ACTION_RECORD: action_type, + PREVIOUS_ACTION_RECORD: action_type, + GROUP_COLUMN: pa.string(), + } + stage = Path(tempfile.mkdtemp(prefix=".sequifier-import-", dir=destination.parent)) + tables, shards, seen, schema = [], [], set(), None + + def flush(): + nonlocal schema + if tables: + shard_schema = pa.unify_schemas( + [table.schema for table in tables], promote_options="permissive" + ) + table = pa.concat_tables( + [_cast_table(table, shard_schema, pa) for table in tables] + ) + schema = ( + shard_schema + if schema is None + else pa.unify_schemas( + [schema, shard_schema], promote_options="permissive" + ) + ) + name = f"part-{len(shards):05d}.parquet" + pq.write_table(table, stage / name) + shards.append( + { + "path": name, + "decisions": table.num_rows, + } + ) + tables.clear() + + try: + for index, reference in enumerate(references): + episode = reference.load() + if episode.episode_id in seen: + raise ValueError("duplicate episode IDs in sequence import") + seen.add(episode.episode_id) + length = len(episode.actions) + transitions = episode.transitions or TransitionColumns() + values = { + **episode.observations, + **{ + name: [value] * length + for name, value in episode.static_inputs.items() + }, + "sequenceId": [index] * length, + "itemPosition": episode.item_positions, + EPISODE_ID_COLUMN: [episode.episode_id] * length, + ACTION_RECORD: episode.actions, + PREVIOUS_ACTION_RECORD: ( + episode.previous_action, + *episode.actions[:-1], + ), + GROUP_COLUMN: [episode.group_id] * length, + REWARD_COLUMN: transitions.rewards or [None] * length, + "terminated": transitions.terminated or [None] * length, + "truncated": transitions.truncated or [None] * length, + "bootstrap_allowed": transitions.bootstrap_allowed or [None] * length, + } + episode_types = dict(types) + for name, column in zip( + policy.observation_columns, successor_columns, strict=True + ): + values[column] = [ + successor[name] if successor is not None else None + for successor in (transitions.successors or [None] * length) + ] + dtype = pa.infer_type([*values[name], *values[column]]) + episode_types[name] = episode_types[column] = dtype + arrays = { + name: _checked_array(entries, episode_types.get(name), name, pa) + for name, entries in values.items() + } + tables.append(pa.table(arrays)) + if (index + 1) % config.episodes_per_shard == 0: + flush() + flush() + # Promote earlier shards only when a later episode widened their schema. + for shard in shards: + path = stage / shard["path"] + if pq.read_schema(path) != schema: + table = _cast_table(pq.read_table(path), schema, pa) + pq.write_table(table, path) + shard["sha256"] = file_checksum(path) + manifest = { + "format": "sequifier.sequence-data", + "version": ContractVersions.SEQUENCE_DATA_MANIFEST, + "episodes": len(seen), + "decisions": sum(shard["decisions"] for shard in shards), + "policy_schema": policy.model_dump(mode="json"), + "shards": shards, + "provenance": { + "source": source.model_dump(mode="json"), + "files": [ + {"path": str(path), "sha256": file_checksum(path)} + for path in source_paths(source) + ], + }, + } + (stage / "manifest.json").write_text( + json.dumps(manifest, indent=2, allow_nan=False) + "\n", encoding="utf-8" + ) + os.rename(stage, destination) + return destination / "manifest.json" + finally: + if stage.exists(): + shutil.rmtree(stage) diff --git a/src/sequifier/trajectory/importers.py b/src/sequifier/trajectory/importers.py new file mode 100644 index 000000000..d17d0024c --- /dev/null +++ b/src/sequifier/trajectory/importers.py @@ -0,0 +1,98 @@ +"""Episode readers for resolved tables, prepared histories, and rollout logs.""" + +from dataclasses import dataclass +from pathlib import Path + +from sequifier.config.policy_inputs import PolicyEpisodeSpec +from sequifier.config.policy_train_config import SequenceSourceConfig +from sequifier.io.actions import action_component_key, action_latent_key +from sequifier.io.sequence import SequenceEpisode, TransitionColumns +from sequifier.trajectory.schema import Trajectory, validate_trajectory +from sequifier.trajectory.sources import ResolvedSequenceSource, resolve_source +from sequifier.trajectory.store import load_trajectory +from sequifier.trajectory.tables import table_episodes + + +def sequence_from_trajectory( + trajectory: Trajectory, policy: PolicyEpisodeSpec, action_semantics="policy" +) -> SequenceEpisode: + episode = validate_trajectory(trajectory) + if ( + episode.format + not in {"sequifier.scalar-policy-trajectory", "sequifier.policy-trajectory"} + and action_semantics != "executed_predictions" + ): + raise ValueError( + "forecast sources require explicit action_semantics='executed_predictions'" + ) + if episode.prediction_columns != policy.action_columns: + raise ValueError("recorded actions do not match the policy action column") + steps = episode.transitions + return SequenceEpisode( + episode_id=episode.episode_id, + item_positions=tuple(step.step_id for step in steps), + observations={ + column: tuple(step.observation[column] for step in steps) + for column in episode.seed.observations + }, + actions=tuple( + { + **step.predictions, + **{ + action_latent_key(name, policy): value + for name, value in step.action_latents.items() + }, + **{ + action_component_key(name, policy): value + for name, value in step.action_components.items() + }, + } + for step in steps + ), + static_inputs=episode.seed.static_inputs, + transitions=TransitionColumns( + rewards=tuple(step.reward for step in steps), + terminated=tuple(step.terminated for step in steps), + truncated=tuple(step.truncated for step in steps), + successors=tuple(step.successor_observation for step in steps), + bootstrap_allowed=tuple(step.bootstrap_allowed for step in steps), + ), + ) + + +@dataclass(frozen=True) +class TrajectoryReference: + path: Path + policy: PolicyEpisodeSpec + action_semantics: str = "policy" + namespace: str = "" + + def load(self) -> SequenceEpisode: + episode = sequence_from_trajectory( + load_trajectory(self.path), self.policy, self.action_semantics + ) + return episode.model_copy( + update={"episode_id": self.namespace + episode.episode_id} + ) + + +def source_episodes( + source: SequenceSourceConfig | ResolvedSequenceSource, policy: PolicyEpisodeSpec +): + resolved: ResolvedSequenceSource = ( + resolve_source(source) if isinstance(source, SequenceSourceConfig) else source + ) + source_config = resolved.source + if resolved.kind == "trajectory": + return [ + TrajectoryReference( + path, + policy, + source_config.action_semantics, + source_config.episode_namespace, + ) + for path in resolved.paths + ] + return table_episodes( + source_config, policy, resolved.paths, prepared=resolved.kind == "prepared" + ) diff --git a/src/sequifier/trajectory/preferences.py b/src/sequifier/trajectory/preferences.py new file mode 100644 index 000000000..0490284d4 --- /dev/null +++ b/src/sequifier/trajectory/preferences.py @@ -0,0 +1,51 @@ +"""Authored reward-free preference manifests over contiguous decision spans.""" + +import json +from pathlib import Path +from typing import Literal + +from pydantic import BaseModel, ConfigDict, Field, model_validator + + +class PreferenceSpan(BaseModel): + model_config = ConfigDict(extra="forbid", frozen=True) + source: str = Field(min_length=1) + episode_id: str = Field(min_length=1) + start: int = Field(default=0, ge=0, strict=True) + end: int = Field(gt=0, strict=True) + + @model_validator(mode="after") + def bounds(self): + if self.end <= self.start: + raise ValueError( + "preference spans require start < end (zero-based, end-exclusive)" + ) + return self + + +class PreferencePair(BaseModel): + model_config = ConfigDict(extra="forbid", frozen=True) + pair_id: str = Field(min_length=1) + context_id: str = Field(min_length=1) + group_id: str | None = None + chosen: PreferenceSpan + rejected: PreferenceSpan + + +class PreferenceManifest(BaseModel): + model_config = ConfigDict(extra="forbid", frozen=True) + format: Literal["sequifier.preference-pairs"] + pairs: list[PreferencePair] = Field(min_length=1) + + @model_validator(mode="after") + def unique_ids(self): + ids = [pair.pair_id for pair in self.pairs] + if len(ids) != len(set(ids)): + raise ValueError("preference pair IDs must be unique") + return self + + +def load_preferences(path): + return PreferenceManifest.model_validate( + json.loads(Path(path).read_text(encoding="utf-8")) + ) diff --git a/src/sequifier/trajectory/prepared.py b/src/sequifier/trajectory/prepared.py new file mode 100644 index 000000000..13391369f --- /dev/null +++ b/src/sequifier/trajectory/prepared.py @@ -0,0 +1,86 @@ +"""Reconstruct split-owned decisions from overlapping prepared histories.""" + +from pathlib import Path + +import polars as pl + +from sequifier.config.metadata import load_dataset_metadata +from sequifier.config.policy_train_config import SequenceSourceConfig + + +def prepared_table( + paths: tuple[Path, ...], source: SequenceSourceConfig +) -> pl.DataFrame: + """Recover unique positions from existing windows, validating overlaps. + + Split target bounds select decisions. A preceding action may be recovered + from stored context, but observations outside the split are not targets. + """ + if source.metadata_config_path is None: + raise ValueError("prepared histories require metadata_config_path") + from sequifier.config.depth_layout import depth_mask_metadata_key + from sequifier.helpers import PANDAS_TO_TORCH_TYPES, read_data + from sequifier.io.pt_payload import load_pt_payload, tabular_tensor_batch + + metadata = load_dataset_metadata(source.metadata_config_path) + split_index = source.selected_split_index + + rows = {} + for path in paths: + if path.suffix.lower() == ".pt": + batch = load_pt_payload( + path, layouts=metadata.depth_layouts, n_classes=metadata.n_classes + ) + else: + window_length = ( + metadata.width_for_split(split_index) + if split_index is not None + else metadata.window_length + ) + data = read_data(str(path), path.suffix.lower().lstrip(".")) + batch = tabular_tensor_batch( + data, + { + column: PANDAS_TO_TORCH_TYPES[dtype] + for column, dtype in metadata.column_data_types.items() + }, + sorted(set(data["inputCol"].to_list())), + window_length, + ) + sequences, ids, starts, pads = ( + batch.sequences, + batch.sequence_ids, + batch.start_item_positions, + batch.left_pad_lengths, + ) + lows, highs = batch.split_start_item_positions, batch.split_end_item_positions + for row in range(len(ids)): + for offset in range( + int(pads[row]), next(iter(sequences.values())).shape[1] + ): + position = int(starts[row]) + offset + key = (int(ids[row]), position) + values = { + column: tensor[row, offset].tolist() + for column, tensor in sequences.items() + } + values.update( + { + depth_mask_metadata_key(name): mask[row, offset].tolist() + for name, mask in batch.depth_valid_masks.items() + } + ) + record = { + "sequenceId": key[0], + "itemPosition": position, + **values, + "__decision": int(lows[row]) <= position < int(highs[row]), + } + if key in rows and rows[key] != record: + raise ValueError( + "prepared history windows disagree on overlapping positions or split bounds" + ) + rows[key] = record + if not rows: + raise ValueError("prepared source contains no valid positions") + return pl.DataFrame(list(rows.values())).sort(["sequenceId", "itemPosition"]) diff --git a/src/sequifier/trajectory/recording.py b/src/sequifier/trajectory/recording.py new file mode 100644 index 000000000..f5c7b9bfd --- /dev/null +++ b/src/sequifier/trajectory/recording.py @@ -0,0 +1,48 @@ +"""Shared trajectory assembly and adapter cleanup for local collectors.""" + +import math +import warnings + +from sequifier.trajectory.schema import Trajectory, Transition + + +class TrajectoryRecorder: + def __init__(self, episode_id, max_episode_steps): + self.episode_id, self.max_episode_steps = episode_id, max_episode_steps + self.transitions = [] + + def append(self, *, terminated, truncated, **values): + step = len(self.transitions) + transition = Transition( + episode_id=self.episode_id, + step_id=step, + terminated=terminated, + truncated=truncated + or ( + self.max_episode_steps is not None + and step + 1 >= self.max_episode_steps + ), + bootstrap_allowed=not terminated, + **values, + ) + self.transitions.append(transition) + return transition + + def finish(self, **values): + last = self.transitions[-1] + return Trajectory( + episode_id=self.episode_id, + transitions=self.transitions, + collection_cutoff=not (last.terminated or last.truncated), + total_return=math.fsum(step.reward for step in self.transitions), + **values, + ) + + +def close_adapter(adapter, failed): + try: + adapter.close() + except Exception as exc: + if not failed: + raise + warnings.warn(f"Adapter close also failed: {exc}", RuntimeWarning) diff --git a/src/sequifier/trajectory/returns.py b/src/sequifier/trajectory/returns.py new file mode 100644 index 000000000..bb78aad20 --- /dev/null +++ b/src/sequifier/trajectory/returns.py @@ -0,0 +1,34 @@ +"""Exact recorded-reward returns and matching online return-budget arithmetic.""" + +from collections.abc import Sequence + +from sequifier.io.values import numeric_scalar + + +def return_to_go(rewards: Sequence[float], gamma: float = 1.0) -> list[float]: + gamma = numeric_scalar(gamma, "gamma") + if not 0 < gamma <= 1: + raise ValueError("gamma must be in (0, 1]") + result = [0.0] * len(rewards) + remaining = 0.0 + for index in range(len(rewards) - 1, -1, -1): + remaining = numeric_scalar( + numeric_scalar(rewards[index], "reward") + gamma * remaining, "return-to-go" + ) + result[index] = remaining + return result + + +def remaining_return(desired_return: float, reward: float, gamma: float = 1.0) -> float: + """If G_t = r_t + gamma G_(t+1), then G_(t+1) = (G_t-r_t)/gamma.""" + gamma = numeric_scalar(gamma, "gamma") + if not 0 < gamma <= 1: + raise ValueError("gamma must be in (0, 1]") + return numeric_scalar( + ( + numeric_scalar(desired_return, "desired_return") + - numeric_scalar(reward, "reward") + ) + / gamma, + "successor desired_return", + ) diff --git a/src/sequifier/trajectory/schema.py b/src/sequifier/trajectory/schema.py new file mode 100644 index 000000000..75515a843 --- /dev/null +++ b/src/sequifier/trajectory/schema.py @@ -0,0 +1,246 @@ +"""Inspectable forecast and policy collection records.""" + +import json +import math +from typing import Any, Literal + +from pydantic import ( + BaseModel, + ConfigDict, + Field, + FiniteFloat, + JsonValue, + StrictInt, + TypeAdapter, + model_validator, +) + +from sequifier.config.rollout_config import InputBinding +from sequifier.io.values import DomainValue +from sequifier.trajectory.returns import remaining_return + + +class Record(BaseModel): + model_config = ConfigDict(extra="forbid", strict=True) + + +class SeedHistory(Record): + observations: dict[str, list[JsonValue]] + attention_valid_mask: list[bool] + static_inputs: dict[str, JsonValue] + + @model_validator(mode="after") + def validate_window(self): + mask = self.attention_valid_mask + if ( + not mask + or not mask[-1] + or any(mask[i] and not mask[i + 1] for i in range(len(mask) - 1)) + ): + raise ValueError("seed mask must be left padding followed by valid history") + if not self.observations or any( + len(values) != len(mask) for values in self.observations.values() + ): + raise ValueError( + "seed feature histories must be nonempty and match the attention mask length" + ) + if set(self.observations) & set(self.static_inputs): + raise ValueError("static inputs must be separate from changing inputs") + return self + + +class Transition(Record): + episode_id: str + step_id: int = Field(ge=0) + observation: dict[str, JsonValue] + predictions: dict[str, DomainValue] + external_inputs: dict[str, JsonValue] + reward: FiniteFloat + action_latents: dict[str, FiniteFloat] = Field(default_factory=dict) + action_components: dict[str, StrictInt] = Field(default_factory=dict) + successor_observation: dict[str, JsonValue] | None + terminated: bool + truncated: bool + bootstrap_allowed: bool + + +class Trajectory(Record): + format: Literal[ + "sequifier.numeric-forecast-trajectory", + "sequifier.scalar-policy-trajectory", + "sequifier.policy-trajectory", + ] + episode_id: str = Field(min_length=1) + seed: SeedHistory + bindings: list[InputBinding] + prediction_columns: list[str] + transitions: list[Transition] = Field(min_length=1) + collection_cutoff: bool + total_return: FiniteFloat + metadata: dict[str, Any] = Field(default_factory=dict) + + @model_validator(mode="after") + def validate_temporal_contract(self): + # Metadata must round-trip through the same portable JSON representation. + json.dumps(self.metadata, allow_nan=False) + inputs = set(self.seed.observations) + if not self.prediction_columns or len(self.prediction_columns) != len( + set(self.prediction_columns) + ): + raise ValueError("prediction columns must be nonempty and unique") + bound = [binding.input for binding in self.bindings] + if len(bound) != len(set(bound)) or set(bound) != inputs: + raise ValueError( + "trajectory requires exactly one producer per changing input" + ) + external_keys = { + binding.key for binding in self.bindings if binding.source == "environment" + } + observation = { + key: values[-1] for key, values in self.seed.observations.items() + } + for index, transition in enumerate(self.transitions): + if transition.episode_id != self.episode_id or transition.step_id != index: + raise ValueError("transition episode/step IDs are not contiguous") + if transition.observation != observation: + raise ValueError( + f"transition {index} observation breaks trajectory continuity" + ) + if set(transition.predictions) != set(self.prediction_columns): + raise ValueError( + f"transition {index} prediction fields do not match the contract" + ) + if transition.successor_observation is None: + if ( + self.format + not in { + "sequifier.scalar-policy-trajectory", + "sequifier.policy-trajectory", + } + or not transition.terminated + or transition.external_inputs + ): + raise ValueError( + "only a terminal policy transition may omit its successor" + ) + else: + if set(transition.external_inputs) != external_keys: + raise ValueError( + f"transition {index} external input fields do not match bindings" + ) + if set(transition.successor_observation) != inputs: + raise ValueError( + f"transition {index} successor fields do not match inputs" + ) + for binding in self.bindings: + producer = ( + transition.predictions + if binding.source == "prediction" + else transition.external_inputs + ) + if ( + binding.key not in producer + or transition.successor_observation[binding.input] + != producer[binding.key] + ): + raise ValueError( + f"transition {index} successor violates binding for {binding.input!r}" + ) + if transition.bootstrap_allowed != (not transition.terminated): + raise ValueError( + "termination must disable bootstrap; truncation preserves the final observation" + ) + if (transition.terminated or transition.truncated) and index != len( + self.transitions + ) - 1: + raise ValueError("trajectory contains a step after an episode boundary") + observation = transition.successor_observation + last = self.transitions[-1] + if self.collection_cutoff != (not (last.terminated or last.truncated)): + raise ValueError("collection cutoff must leave the episode unfinished") + expected_return = math.fsum( + transition.reward for transition in self.transitions + ) + if not math.isclose( + self.total_return, expected_return, rel_tol=1e-12, abs_tol=1e-12 + ): + raise ValueError("total_return does not equal the sum of rewards") + if self.format == "sequifier.policy-trajectory": + _validate_joint_execution(self) + return self + + +def validate_trajectory(value: Trajectory | dict) -> Trajectory: + """Revalidate even an existing (possibly mutated) trajectory object.""" + return Trajectory.model_validate( + value.model_dump() if isinstance(value, Trajectory) else value + ) + + +def _validate_joint_execution(trajectory): + from sequifier.config.policy_config import ResolvedPolicyBinding + from sequifier.io.actions import encode_action + + execution = trajectory.metadata.get("policy_execution") + if not isinstance(execution, dict): + raise ValueError("policy trajectories require execution metadata") + spec = ResolvedPolicyBinding.model_validate(execution.get("config")) + if trajectory.prediction_columns != spec.action_columns: + raise ValueError("trajectory action fields differ from the binding") + if set(trajectory.seed.observations) != set(spec.observation_columns) or set( + trajectory.seed.static_inputs + ) != set(spec.static_columns): + raise ValueError("trajectory inputs differ from the model binding") + scores = TypeAdapter(list[FiniteFloat]).validate_python( + execution.get("action_log_probabilities"), strict=True + ) + previous = execution.get("previous_action_ids") + if ( + len(scores) != len(trajectory.transitions) + or not isinstance(previous, list) + or len(previous) != len(scores) + ): + raise ValueError("trajectory scores/history must match decision count") + expected = None + for index, step in enumerate(trajectory.transitions): + real_fields = { + name for name, field in spec.actions.items() if field.type == "real" + } + if (set(step.action_latents) | set(step.action_components)) - real_fields: + raise ValueError( + "trajectory transform metadata must name real action fields" + ) + recorded_previous = previous[index] + if isinstance(recorded_previous, dict): + recorded_previous = { + key: value + for key, value in recorded_previous.items() + if key in spec.actions + } + if recorded_previous != expected: + raise ValueError("previous actions must lag executed actions") + expected = encode_action(step.predictions, spec) + if execution.get("action_selection") not in {"deterministic", "stochastic"}: + raise ValueError("trajectory must declare its selection mode") + budgets = TypeAdapter(list[FiniteFloat]).validate_python( + execution.get("remaining_returns"), strict=True + ) + if spec.return_conditioned: + if len(budgets) != len(scores) + 1 or budgets[0] != execution.get( + "initial_desired_return" + ): + raise ValueError( + "return budgets must include every successor and the initial budget" + ) + for index, step in enumerate(trajectory.transitions): + if not math.isclose( + budgets[index + 1], + remaining_return(budgets[index], step.reward, spec.returns.gamma), + rel_tol=1e-12, + abs_tol=1e-12, + ): + raise ValueError( + "return budgets violate the recorded reward arithmetic" + ) + elif budgets: + raise ValueError("an unconditioned policy cannot record return budgets") diff --git a/src/sequifier/trajectory/sources.py b/src/sequifier/trajectory/sources.py new file mode 100644 index 000000000..7a3da2e08 --- /dev/null +++ b/src/sequifier/trajectory/sources.py @@ -0,0 +1,143 @@ +"""Resolve sequence source files, manifests, and declared formats.""" + +import hashlib +import json +from dataclasses import dataclass +from pathlib import Path + +from sequifier.config.policy_train_config import ( + SequenceSourceConfig, + resolve_sequence_source, +) +from sequifier.contract_versions import ContractVersions, require_contract_version +from sequifier.trajectory.store import load_trajectory + + +def file_checksum(path): + digest = hashlib.sha256() + with open(path, "rb") as stream: + for block in iter(lambda: stream.read(1024 * 1024), b""): + digest.update(block) + return digest.hexdigest() + + +def _checksum(path, checksums): + path = Path(path) + if path not in checksums: + checksums[path] = file_checksum(path) + return checksums[path] + + +def _source_files(source, checksums): + manifest = None + if source.manifest_path is None: + paths = [Path(path) for path in source.paths] + else: + manifest_path = Path(source.manifest_path) + content = manifest_path.read_bytes() + checksum = hashlib.sha256(content).hexdigest() + if manifest_path in checksums and checksums[manifest_path] != checksum: + raise ValueError( + f"source manifest changed during resolution: {manifest_path}" + ) + checksums[manifest_path] = checksum + manifest = json.loads(content.decode("utf-8")) + if not isinstance(manifest, dict): + raise ValueError("Source manifest must be a mapping") + manifest_format = manifest.get("format") + if manifest_format == "sequifier.sequence-data": + require_contract_version( + manifest_format, + manifest.get("version"), + ContractVersions.SEQUENCE_DATA_MANIFEST, + ) + entries_field = "shards" + elif manifest_format == "sequifier.trajectory-source": + if "version" in manifest: + raise ValueError( + "Authored trajectory-source manifests must omit version" + ) + entries_field = "episodes" + else: + raise ValueError(f"Unsupported source manifest format {manifest_format!r}") + entries = manifest.get(entries_field) + if not isinstance(entries, list) or not entries: + raise ValueError( + f"source manifest requires a nonempty {entries_field} list" + ) + paths = [] + for entry in entries: + if ( + not isinstance(entry, dict) + or not isinstance(entry.get("path"), str) + or not entry["path"] + ): + raise ValueError("each manifest entry requires a nonempty path") + path = manifest_path.parent / entry["path"] + if manifest_format == "sequifier.sequence-data" and _checksum( + path, checksums + ) != entry.get("sha256"): + raise ValueError(f"sequence shard checksum disagrees with {path}") + if ( + "episode_id" in entry + and load_trajectory(path).episode_id != entry["episode_id"] + ): + raise ValueError(f"manifest episode ID disagrees with {path}") + paths.append(path) + if not paths or len({path.resolve() for path in paths}) != len(paths): + raise ValueError("sources require nonempty, unique paths") + return paths, manifest + + +def source_paths(source: SequenceSourceConfig) -> list[Path]: + return [path.resolve() for path in _source_files(source, {})[0]] + + +@dataclass(frozen=True) +class ResolvedSequenceSource: + source: SequenceSourceConfig + paths: tuple[Path, ...] + kind: str + checksums: dict[Path, str] + + +def resolve_source( + source: SequenceSourceConfig, *, checksums: dict[Path, str] | None = None +) -> ResolvedSequenceSource: + checksums = {} if checksums is None else checksums + if source.split_index is not None: + source = resolve_sequence_source(source, Path.cwd()) + # Retain manifest-relative locations so revalidation also detects retargeted links. + paths, manifest = _source_files(source, checksums) + dependencies = set(paths) + dependencies.update( + Path(path) + for path in ( + source.manifest_path, + source.metadata_config_path, + source.static_data_path, + ) + if path + ) + verified = {path: _checksum(path, checksums) for path in dependencies} + resolved_paths = tuple(path.resolve() for path in paths) + if manifest is not None and manifest.get("format") == "sequifier.sequence-data": + if source.encoding != "raw" or source.episode_end != "unknown": + raise ValueError( + "committed sequence data already declares its encoding and boundaries" + ) + source = source.model_copy(update={"format": "table"}) + kind = source.format + if kind == "auto": + suffixes = {path.suffix.lower() for path in resolved_paths} + if suffixes <= {".json"}: + kind = "trajectory" + elif suffixes <= {".csv", ".parquet"}: + kind = "table" + elif suffixes == {".pt"}: + kind = "prepared" + else: + raise ValueError( + "select a source format explicitly for mixed or unsupported files" + ) + return ResolvedSequenceSource(source, resolved_paths, kind, verified) diff --git a/src/sequifier/trajectory/store.py b/src/sequifier/trajectory/store.py new file mode 100644 index 000000000..1ff0423f5 --- /dev/null +++ b/src/sequifier/trajectory/store.py @@ -0,0 +1,31 @@ +"""Validated JSON trajectory storage with shared atomic publication.""" + +import json +from pathlib import Path + +from sequifier.contract_versions import ContractVersions, require_contract_version +from sequifier.io.atomic import write_json_atomic +from sequifier.trajectory.schema import Trajectory, validate_trajectory + +_TRAJECTORY_VERSIONS = { + "sequifier.numeric-forecast-trajectory": ContractVersions.FORECAST_TRAJECTORY, + "sequifier.scalar-policy-trajectory": ContractVersions.SCALAR_POLICY_TRAJECTORY, + "sequifier.policy-trajectory": ContractVersions.JOINT_POLICY_TRAJECTORY, +} + + +def save_trajectory(trajectory: Trajectory, path: str | Path) -> None: + values = validate_trajectory(trajectory).model_dump(mode="json") + values["version"] = _TRAJECTORY_VERSIONS[values["format"]] + write_json_atomic(path, values) + + +def load_trajectory(path: str | Path) -> Trajectory: + values = json.loads(Path(path).read_text(encoding="utf-8")) + kind = values.get("format") if isinstance(values, dict) else None + if not isinstance(kind, str) or kind not in _TRAJECTORY_VERSIONS: + raise ValueError("Unsupported trajectory format") + require_contract_version( + kind, values.pop("version", None), _TRAJECTORY_VERSIONS[kind] + ) + return validate_trajectory(values) diff --git a/src/sequifier/trajectory/tables.py b/src/sequifier/trajectory/tables.py new file mode 100644 index 000000000..a92ced15b --- /dev/null +++ b/src/sequifier/trajectory/tables.py @@ -0,0 +1,454 @@ +"""Indexed table episode readers and canonical table-to-episode validation.""" + +import tempfile +from bisect import bisect_right +from dataclasses import dataclass +from pathlib import Path + +import polars as pl + +from sequifier.config.metadata import DatasetMetadata, load_dataset_metadata +from sequifier.config.policy_inputs import PolicyEpisodeSpec +from sequifier.config.policy_train_config import SequenceSourceConfig +from sequifier.io.images import ImageStore +from sequifier.io.preprocessing import denormalize_values +from sequifier.io.sequence import ( + SequenceEpisode, + TransitionColumns, + ordered_sequence_rows, +) +from sequifier.io.values import domain_scalar, numeric_scalar +from sequifier.trajectory.prepared import prepared_table + +ACTION_RECORD = "__executed_action" +PREVIOUS_ACTION_RECORD = "__previous_action" +PREVIOUS_ACTION_PREFIX = "__previous_action_" +SUCCESSOR_PREFIX = "__next_" +BOOTSTRAP_COLUMN = "bootstrap_allowed" +GROUP_COLUMN = "__group" +EPISODE_ID_COLUMN = "episodeId" +REWARD_COLUMN = "reward" +TRANSITION_FLAGS = ("terminated", "truncated", BOOTSTRAP_COLUMN) + + +def scan_tables(paths: tuple[Path, ...]): + scans = [] + for path in paths: + if path.suffix.lower() == ".csv": + scans.append(pl.scan_csv(path, infer_schema_length=None)) + elif path.suffix.lower() == ".parquet": + scans.append(pl.scan_parquet(path)) + else: + raise ValueError("table sources require CSV or Parquet files") + return pl.concat(scans, how="vertical_relaxed") + + +class IndexedTable: + """Canonical sorted storage with direct row-group access per episode. + + The temporary directory lives as long as its episode references. CSV parsing + and sorting happen once; cache misses never filter the original source. + """ + + def __init__(self, data, sequence_column): + import pyarrow.parquet as pq + + self.directory = tempfile.TemporaryDirectory(prefix="sequifier-episodes-") + self.path = Path(self.directory.name) / "episodes.parquet" + data.sort(sequence_column, maintain_order=True).sink_parquet( + self.path, row_group_size=8192 + ) + table = pq.ParquetFile(self.path) + self.offsets = [0] + for i in range(table.num_row_groups): + self.offsets.append(self.offsets[-1] + table.metadata.row_group(i).num_rows) + self.ranges = {} + offset = 0 + for batch in table.iter_batches(columns=[sequence_column]): + for identity in batch.column(0).to_pylist(): + if isinstance(identity, bool) or not isinstance(identity, (int, str)): + raise ValueError( + "sequence IDs must be non-null integers or strings" + ) + if identity not in self.ranges: + self.ranges[identity] = [offset, offset + 1] + else: + self.ranges[identity][1] = offset + 1 + offset += 1 + table.close() + + def load(self, identity): + import pyarrow.parquet as pq + + start, end = self.ranges[identity] + first = bisect_right(self.offsets, start) - 1 + last = bisect_right(self.offsets, end - 1) - 1 + with pq.ParquetFile(self.path) as table: + rows = table.read_row_groups(list(range(first, last + 1))) + return pl.from_arrow(rows.slice(start - self.offsets[first], end - start)) + + +def model_decoder(metadata, actions): + """Decode model-encoded scalars using one fixed set of inverse codecs.""" + inverse_maps = ( + { + column: {encoded: label for label, encoded in mapping.items()} + for column, mapping in metadata.id_maps.items() + } + if metadata is not None + else {} + ) + + def decode(value, column): + if isinstance(value, (list, tuple)): + return [decode(item, column) for item in value] + if isinstance(value, bool) or isinstance(value, dict): + return value + value = domain_scalar(value, column) + if metadata is not None: + if column in inverse_maps: + inverse = inverse_maps[column] + if ( + isinstance(value, str) + or value != int(value) + or int(value) not in inverse + ): + from sequifier.special_tokens import SPECIAL_TOKEN_IDS + + if ( + isinstance(value, int) + and value in SPECIAL_TOKEN_IDS.labels_by_id + ): + return SPECIAL_TOKEN_IDS.labels_by_id[value] + raise ValueError(f"{column}: encoded ID has no domain label") + value = inverse[int(value)] + if column in actions and actions[column].type == "categorical": + action_values = actions[column].values + if isinstance(value, str) and not isinstance(action_values[0], str): + try: + value = numeric_scalar(float(value), column) + except ValueError: + pass + matches = [ + action + for action in action_values + if action == value or str(action) == str(value) + ] + if len(matches) == 1: + value = matches[0] + elif ( + metadata.normalize_real_columns + and column in metadata.selected_columns_statistics + ): + value = denormalize_values( + numeric_scalar(value, column), + metadata.selected_columns_statistics[column], + ) + return value + + return decode + + +@dataclass(frozen=True) +class TableEpisodeReference: + source: SequenceSourceConfig + policy: PolicyEpisodeSpec + sequence_id: str | int + table: IndexedTable + metadata: DatasetMetadata | None = None + prepared: bool = False + static_table: IndexedTable | None = None + image_store: ImageStore | None = None + + def load(self) -> SequenceEpisode: + data = ordered_sequence_rows(self.table.load(self.sequence_id)) + decode = model_decoder( + self.metadata if self.prepared or self.source.encoding == "model" else None, + self.policy.actions, + ) + + preceding_row = None + if self.prepared: + decisions = data.filter(pl.col("__decision")) + if decisions.is_empty(): + raise ValueError("prepared episode contains no split-owned decisions") + first = decisions["itemPosition"][0] + previous = data.filter(pl.col("itemPosition") == first - 1) + if len(previous): + preceding_row = previous.row(0, named=True) + data = decisions + static_data = None + if self.static_table is not None: + static_data = self.static_table.load(self.sequence_id) + if self.image_store is not None: + import torch + + images, masks = self.image_store.gather( + torch.tensor([int(self.sequence_id)]), + torch.tensor([data["itemPosition"].to_list()]), + torch.ones((1, len(data)), dtype=torch.bool), + ) + for name, value in {**images, **masks}.items(): + if name == "__preconditioning_image": + if static_data is None: + data = data.with_columns( + pl.Series(name, [value[0].tolist()] * len(data)) + ) + else: + static_data = static_data.with_columns( + pl.Series(name, [value[0].tolist()]) + ) + else: + data = data.with_columns(pl.Series(name, value[0].tolist())) + return table_episode( + data, + self.policy, + sequence_id=self.sequence_id, + preceding_row=preceding_row, + static_data=static_data, + episode_end=self.source.episode_end, + episode_namespace=self.source.episode_namespace, + decode=decode, + ) + + +def table_episode( + data, + policy, + *, + sequence_id, + preceding_row=None, + static_data=None, + episode_end="unknown", + episode_namespace="", + decode=domain_scalar, +) -> SequenceEpisode: + """Read an episode using the policy's canonical feature names.""" + data = ordered_sequence_rows(data) + if decode is domain_scalar: + + def identity_decode(value, column): + return value + + decode = identity_decode + + if data.is_empty(): + raise ValueError("sequence source changed while loading") + + observations = policy.observation_columns + static_inputs = policy.static_columns + action_names = policy.action_columns + structured_actions = ACTION_RECORD in data.columns + structured_previous = PREVIOUS_ACTION_RECORD in data.columns + action_columns = {ACTION_RECORD} if structured_actions else set(action_names) + previous_columns = [PREVIOUS_ACTION_PREFIX + name for name in action_names] + scalar_previous = not structured_previous and any( + column in data.columns for column in previous_columns + ) + if structured_previous: + previous_inputs = {PREVIOUS_ACTION_RECORD} + elif scalar_previous: + previous_inputs = set(previous_columns) + else: + previous_inputs = set() + successor_columns = [SUCCESSOR_PREFIX + name for name in observations] + has_successors = bool(set(successor_columns) & set(data.columns)) + inputs = set(observations) + if static_data is None: + inputs.update(static_inputs) + if action_columns & (inputs | previous_inputs): + raise ValueError( + "current action labels require distinct input/previous-action columns" + ) + if previous_inputs & inputs: + raise ValueError("previous actions require distinct input columns") + if has_successors and set(successor_columns) & ( + inputs | action_columns | previous_inputs + ): + raise ValueError("successors require distinct input and action columns") + required = inputs | {"itemPosition"} | action_columns | previous_inputs + if has_successors: + required.update(successor_columns) + if missing := required - set(data.columns): + raise ValueError(f"policy table is missing columns: {sorted(missing)}") + for record in (ACTION_RECORD, PREVIOUS_ACTION_RECORD): + if ( + record in data.columns + and data.schema[record] != pl.Null + and not isinstance(data.schema[record], pl.Struct) + ): + raise ValueError(f"{record}: action records require a structured column") + + def canonical_action(value): + if value is None: + return None + return { + name: entry + for name, entry in value.items() + if name in policy.actions or entry is not None + } + + def action_at(row): + if structured_actions: + return canonical_action(row[ACTION_RECORD]) + return {name: decode(row[name], name) for name in action_names} + + def values(column, decoder=decode): + return tuple(decoder(value, column) for value in data[column]) + + def optional_values(column, decoder=decode): + if column not in data.columns or data[column].null_count() == len(data): + return None + return values(column, decoder) + + def constant(entries, label): + entries = list(entries) + if not entries or any(value != entries[0] for value in entries[1:]): + raise ValueError(f"{label} must remain constant within an episode") + return entries[0] + + positions = tuple(data["itemPosition"]) + actions = tuple(action_at(row) for row in data.iter_rows(named=True)) + preceding = action_at(preceding_row) if preceding_row is not None else None + if scalar_previous: + previous = tuple( + dict(zip(action_names, entries, strict=True)) + for entries in data.select(previous_columns).iter_rows() + ) + if previous[1:] != actions[:-1]: + raise ValueError( + "previous actions must lag recorded actions by one decision" + ) + preceding = ( + None + if all(value is None for value in previous[0].values()) + else previous[0] + ) + if structured_previous: + previous = tuple(map(canonical_action, data[PREVIOUS_ACTION_RECORD])) + if previous[1:] != actions[:-1]: + raise ValueError( + "previous-action column must lag executed actions by exactly one decision" + ) + preceding = previous[0] + if static_data is not None: + if len(static_data) != 1: + raise ValueError("static data requires exactly one row per sequenceId") + if missing := set(static_inputs) - set(static_data.columns): + raise ValueError(f"static table is missing columns: {sorted(missing)}") + else: + static_data = data + + def feature_value(value, name): + return value if policy.input_encoding == "model" else decode(value, name) + + static = { + name: constant( + (feature_value(value, name) for value in static_data[name]), + f"static input {name}", + ) + for name in static_inputs + } + # Strict boolean and length checks belong to TransitionColumns/SequenceEpisode. + transitions = { + field: optional_values(field, lambda value, _: value) + for field in TRANSITION_FLAGS + } + transitions = TransitionColumns(**transitions).model_dump(exclude_none=True) + declared = episode_end + if declared != "unknown": + current = transitions.get(declared) + if current is not None and not current[-1]: + raise ValueError("episode_end conflicts with the recorded final boundary") + transitions[declared] = (False,) * (len(data) - 1) + (True,) + successors = None + if has_successors: + successors = [] + for entries in data.select(successor_columns).iter_rows(): + present = [value is not None for value in entries] + if any(present) and not all(present): + raise ValueError("a missing successor must mask all observation fields") + successors.append( + { + name: feature_value(value, name) + for name, value in zip(observations, entries, strict=True) + } + if all(present) + else None + ) + groups = optional_values(GROUP_COLUMN, domain_scalar) + identity = ( + constant(values(EPISODE_ID_COLUMN, domain_scalar), "episode ID") + if EPISODE_ID_COLUMN in data.columns + else sequence_id + ) + return SequenceEpisode( + episode_id=episode_namespace + str(identity), + item_positions=positions, + observations={name: values(name, feature_value) for name in observations}, + actions=actions, + static_inputs=static, + starts_episode=positions[0] == 0 and preceding is None, + previous_action=preceding, + group_id=str(constant(groups, "collection group")) + if groups is not None + else None, + transitions=TransitionColumns( + **transitions, + rewards=optional_values( + REWARD_COLUMN, + lambda value, column: numeric_scalar(decode(value, column), "reward"), + ), + successors=tuple(successors) if successors is not None else None, + ), + ) + + +def table_episodes(source, policy, paths, *, prepared=False): + data = prepared_table(paths, source).lazy() if prepared else scan_tables(paths) + if {"sequenceId", "itemPosition"} - set(data.collect_schema().names()): + raise ValueError("policy tables require sequenceId and itemPosition") + if set(source.columns.observations) - set(policy.observation_columns) or set( + source.columns.static_inputs + ) - set(policy.static_columns): + raise ValueError("table columns must name declared policy features") + table = IndexedTable(data, "sequenceId") + static_table = None + if source.static_data_path: + static_table = IndexedTable( + scan_tables((Path(source.static_data_path),)), "sequenceId" + ) + if ( + any(end - start != 1 for start, end in static_table.ranges.values()) + or not table.ranges.keys() <= static_table.ranges.keys() + ): + raise ValueError( + "static data must cover every selected sequence ID without duplicates" + ) + metadata = ( + load_dataset_metadata(source.metadata_config_path) + if source.metadata_config_path + else None + ) + image_store = None + if ( + prepared + and metadata is not None + and (metadata.image_sources or metadata.preconditioning_image is not None) + ): + image_store = ImageStore( + metadata.image_sources, metadata.preconditioning_image, source.project_root + ) + return [ + TableEpisodeReference( + source, + policy, + identity, + table, + metadata=metadata, + prepared=prepared, + static_table=static_table, + image_store=image_store, + ) + for identity in table.ranges + ] diff --git a/src/sequifier/visualize_training.py b/src/sequifier/visualize_training.py index 8d628a575..dfab5bd35 100644 --- a/src/sequifier/visualize_training.py +++ b/src/sequifier/visualize_training.py @@ -14,15 +14,17 @@ from sequifier.helpers import configure_logger from sequifier.logging_paths import dataset_artifact_prefix -from sequifier.training.metrics import TOTAL_TARGET +from sequifier.training.metrics import ( + TOTAL_TARGET, + validate_metrics_csv_header, + validate_metrics_csv_row, +) from sequifier.typechecking import beartype class DataContinuityError(Exception): """Non-monotonic training batch or epoch sequence.""" - pass - @dataclass class TrainingMetrics: @@ -35,6 +37,8 @@ class TrainingMetrics: default_factory=dict ) train_window_batches: dict[float, dict[int, int]] = field(default_factory=dict) + training_label: str = "Training Loss" + validation_label: str = "Validation Loss" class StructuredMetricsParser: @@ -49,14 +53,49 @@ def parse_files(self, training_file: str, validation_file: str) -> TrainingMetri training_rows = self._read_rows(training_file) validation_rows = self._read_rows(validation_file) total_training_rows = [ - row for row in training_rows if row.get("target") == TOTAL_TARGET + row + for row in training_rows + if row.get("target") == TOTAL_TARGET + and ( + row.get("is_primary", "").lower() == "true" + or (not row.get("is_primary") and row.get("metric", "loss") == "loss") + ) ] if not total_training_rows: raise DataContinuityError( f"[{self.model}]: No valid training loss data found." ) run_id = total_training_rows[-1]["run_id"] + total_training_rows = [ + row for row in total_training_rows if row["run_id"] == run_id + ] + last_row = total_training_rows[-1] + policy_rows = last_row.get("algorithm") not in {None, "", "supervised"} + if policy_rows: + phase = last_row.get("phase") + training_metric = last_row.get("metric") + total_training_rows = [ + row + for row in total_training_rows + if row.get("phase") == phase and row.get("metric") == training_metric + ] + validation_rows = [ + row for row in validation_rows if row.get("phase") == phase + ] + primary_validation = [ + row + for row in validation_rows + if row.get("run_id") == run_id + and row.get("is_primary", "").lower() == "true" + ] + validation_metric = ( + primary_validation[-1].get("metric") if primary_validation else None + ) + else: + validation_metric = "loss" metrics = TrainingMetrics() + if total_training_rows[-1].get("algorithm") not in {None, "", "supervised"}: + metrics.training_label = total_training_rows[-1].get("metric", "Training") for row in total_training_rows: if row.get("run_id") != run_id: @@ -85,8 +124,17 @@ def parse_files(self, training_file: str, validation_file: str) -> TrainingMetri metric = row.get("metric") target = row.get("target") value = float(row["value"]) - if metric == "loss" and target == TOTAL_TARGET: + if ( + metric == validation_metric + and (not policy_rows or row.get("is_primary", "").lower() == "true") + ) and target == TOTAL_TARGET: metrics.val_losses[epoch_position] = value + if metric is not None and row.get("algorithm") not in { + None, + "", + "supervised", + }: + metrics.validation_label = metric elif metric == "baseline_loss" and target == TOTAL_TARGET: metrics.baseline_losses[epoch_position] = value elif metric == "loss" and target: @@ -96,12 +144,12 @@ def parse_files(self, training_file: str, validation_file: str) -> TrainingMetri raise DataContinuityError( f"[{self.model}]: No valid validation loss data found." ) - if not metrics.baseline_losses: + if not metrics.baseline_losses and not policy_rows: raise DataContinuityError(f"[{self.model}]: No baseline loss data found.") missing_baselines = sorted( set(metrics.val_losses).difference(metrics.baseline_losses) ) - if missing_baselines: + if missing_baselines and not policy_rows: raise DataContinuityError( f"[{self.model}]: Missing baseline loss data for validation " f"positions {missing_baselines!r}." @@ -112,7 +160,12 @@ def parse_files(self, training_file: str, validation_file: str) -> TrainingMetri @beartype def _read_rows(path: str) -> list[dict[str, str]]: with open(path, "r", encoding="utf-8", newline="") as file: - return list(csv.DictReader(file)) + reader = csv.DictReader(file) + validate_metrics_csv_header(reader.fieldnames, path) + rows = list(reader) + for row in rows: + validate_metrics_csv_row(row, path) + return rows @staticmethod @beartype @@ -121,7 +174,7 @@ def _epoch_position(row: dict[str, str]) -> float: batch = int(row["batch"]) batches_total = int(row["batches_total"]) if row.get("evaluation_kind") == "initial": - return 0.0 + return float(epoch - 1) if row.get("algorithm") != "supervised" else 0.0 if batch > 0 and batches_total > 0: return round(epoch - 1 + batch / batches_total, 8) return float(epoch) @@ -183,7 +236,11 @@ def format_plot_data( """Convert parsed metrics into Plotly-ready arrays.""" val_x = sorted(list(metrics.val_losses.keys())) val_y = [metrics.val_losses[e] for e in val_x] - base_y = [metrics.baseline_losses[e] for e in val_x] + base_y = ( + [metrics.baseline_losses[e] for e in val_x] + if all(e in metrics.baseline_losses for e in val_x) + else [] + ) train_x, train_y = [], [] @@ -261,6 +318,8 @@ def append_chunk() -> None: "train_x": train_x, "train_y": train_y, "var_losses": metrics.var_losses, + "training_label": metrics.training_label, + "validation_label": metrics.validation_label, } @@ -292,11 +351,17 @@ def _generate_single_model_plot( invalid_runs: Optional[dict[str, str]] = None, ) -> None: """Write a single-model training report.""" + policy_metrics = data.get("training_label", "Training Loss") != "Training Loss" + if policy_metrics: + yaxis_type = "linear" has_var_losses = bool(data.get("var_losses")) subplot_titles = ( ("Global Losses", "Normalized Variable Validation Losses") if has_var_losses - else ("Global Losses", "") + else ( + "Training and validation metrics" if policy_metrics else "Global Losses", + "", + ) ) fig = make_subplots(rows=1, cols=2, subplot_titles=subplot_titles) @@ -306,7 +371,7 @@ def _generate_single_model_plot( x=data["val_x"], y=data["val_y"], mode="lines", - name="Validation Loss", + name=data.get("validation_label", "Validation Loss"), hovertemplate=f"{model}
Val Loss: %{{y}}
Epoch: %{{x}}", ), row=1, @@ -317,7 +382,7 @@ def _generate_single_model_plot( x=data["train_x"], y=data["train_y"], mode="lines", - name="Training Loss", + name=data.get("training_label", "Training Loss"), hovertemplate=f"{model}
Train Loss: %{{y}}
Epoch: %{{x}}", ), row=1, @@ -393,6 +458,11 @@ def _generate_multi_model_plot( fig = make_subplots( rows=1, cols=2, subplot_titles=("Validation Losses", "Training Losses") ) + if any( + data.get("training_label", "Training Loss") != "Training Loss" + for data in all_data.values() + ): + yaxis_type = "linear" baseline_val = None colors = pc.qualitative.Plotly @@ -483,7 +553,7 @@ def generate_html_report( output_dir = os.path.join(args.project_root, "outputs", "visualization") os.makedirs(output_dir, exist_ok=True) - yaxis_type = "log" if getattr(args, "log_scale", False) else "linear" + yaxis_type = "log" if args.log_scale else "linear" if len(models) == 1 and report_name is None: model = models[0] @@ -514,7 +584,7 @@ def visualize_training(args: argparse.Namespace) -> None: if not models: raise ValueError("No models provided to visualize.") - bucket_batches = getattr(args, "bucket_training_batches", None) + bucket_batches = args.bucket_training_batches all_data: dict[str, Any] = {} invalid_runs: dict[str, str] = {} diff --git a/tests/test_rl_image_throughput.py b/tests/test_rl_image_throughput.py new file mode 100644 index 000000000..bc29a96da --- /dev/null +++ b/tests/test_rl_image_throughput.py @@ -0,0 +1,215 @@ +"""Verify compact policy histories and padded-image skipping without training.""" + +import json + +import numpy as np +import pytest +import torch +from torch import nn + +from sequifier.config.policy_config import ResolvedPolicyBinding +from sequifier.io.values import portable_fields +from sequifier.model.execution_schema import ExecutionSchema, InputDescriptor +from sequifier.model.images import ImageEncoder +from sequifier.rl.collection import PolicyCursor, PolicyDecision +from sequifier.rl.environment import ( + OnlineResetResult, + OnlineStepResult, + validate_reset, + validate_step, +) +from sequifier.rl.history import ( + PolicyFrame, + PolicyHistory, + _json_payload_bytes, + frame_storage_bytes, + history_inputs, +) + + +@pytest.fixture(scope="module") +def spec(): + inputs = [ + InputDescriptor(f"feature_{index}", column, "feature", "float32", ("batch", 32)) + for index, column in enumerate( + ("observation_state_x", "observation_state_y", "action_x", "action_y") + ) + ] + inputs += [ + InputDescriptor( + "image_0", "__image:camera", "image", "uint8", ("batch", 32, 1, 3, 224, 224) + ), + InputDescriptor( + "image_mask_0", "__image_mask:camera", "metadata", "bool", ("batch", 32, 1) + ), + InputDescriptor( + "attention_valid_mask", + "attention_valid_mask", + "metadata", + "bool", + ("batch", 32), + ), + ] + return ResolvedPolicyBinding( + actions={ + name: {"type": "real", "scale": {"fixed": 10.0}} + for name in ("action_x", "action_y") + }, + previous_action_inputs={"action_x": "action_x", "action_y": "action_y"}, + initial_inputs={"action_x": 256.0, "action_y": 256.0}, + context_length=32, + interface_name="actor", + observation_columns=[ + "observation_state_x", + "observation_state_y", + "__image:camera", + "__image_mask:camera", + ], + input_schema=ExecutionSchema(tuple(inputs), (), {}, {}).to_dict(), + ) + + +def observation(): + rng = np.random.default_rng(17) + return { + "observation_state_x": 100.0, + "observation_state_y": 200.0, + "__image:camera": rng.integers(0, 256, (1, 3, 224, 224), dtype=np.uint8), + "__image_mask:camera": np.array([True]), + } + + +@pytest.mark.parametrize("dtype", [np.uint8, np.bool_]) +@pytest.mark.parametrize("shape", [(), (0,), (2, 0), (1,), (2, 3), (1, 3, 7, 9)]) +def test_compact_json_size_matches_portable_json(dtype, shape): + rng = np.random.default_rng(12) + array = rng.integers(0, 256, shape, dtype=np.uint8).astype(dtype) + value = {"é": array, "nested": [None, 1.25, "漢字", (False, 1)], 17: {None: True}} + expected = len(json.dumps(portable_fields(value), allow_nan=False).encode("utf-8")) + assert _json_payload_bytes(value) == expected + + +def test_validators_can_own_compact_images_without_changing_portable_default(): + raw = observation() + reset = OnlineResetResult(raw) + step = OnlineStepResult(raw, 1.0) + for validator, result in ((validate_reset, reset), (validate_step, step)): + standard = validator(result, {}) + compact = validator(result, {}, compact_images=True) + assert standard.observation is not None + assert compact.observation is not None + assert isinstance(standard.observation["__image:camera"], list) + assert isinstance(compact.observation["__image:camera"], np.ndarray) + assert portable_fields(compact.observation) == standard.observation + assert not np.shares_memory( + raw["__image:camera"], compact.observation["__image:camera"] + ) + + +def test_history_preserves_tensors_checkpoints_and_accounting(spec): + raw = observation() + history = PolicyHistory(spec) + history.reset(raw, {}, "episode") + owned = dict(history.current.observation)["__image:camera"] + assert isinstance(owned, np.ndarray) + assert not np.shares_memory(owned, raw["__image:camera"]) + frame = history.current + portable = frame.state_dict() + legacy = PolicyFrame.from_state_dict(portable) + assert isinstance(portable["observation"]["__image:camera"], list) + baseline = history_inputs(spec, [legacy]) + compact = history.inputs() + for field in ("inputs", "metadata"): + left, right = getattr(baseline, field), getattr(compact, field) + assert all(torch.equal(left[key], right[key]) for key in left) + expected = 1024 + 4 * len(json.dumps(portable, allow_nan=False).encode("utf-8")) + assert frame_storage_bytes(frame) == expected + # The second accounting pass must use the cached cost, not touch pixels. + assert frame.storage_bytes == expected + raw["__image:camera"].fill(0) + assert np.any(owned) + saved = json.loads(json.dumps(history.state_dict())) + restored = PolicyHistory(spec) + restored.load_state_dict(saved) + assert restored.state_dict()["frames"] == saved["frames"] + assert isinstance(dict(restored.current.observation)["__image:camera"], np.ndarray) + + +def test_compact_history_retains_lagged_actions(spec): + history = PolicyHistory(spec) + history.reset(observation(), {}, "episode") + history.advance({"action_x": 123.0, "action_y": 234.0}, observation()) + inputs = history.inputs() + assert inputs.inputs["action_x"][0, -1] == 123.0 + assert inputs.inputs["action_y"][0, -1] == 234.0 + assert inputs.attention_valid_mask.sum() == 2 + + +def test_collection_cursor_uses_compact_images(spec): + class Actor: + config = spec + device = torch.device("cpu") + + def model_inputs(self, inputs): + return inputs.inputs, inputs.metadata + + class Adapter: + def step(self, actions): + return OnlineStepResult(observation(), 0.5, truncated=True) + + cursor = PolicyCursor(Actor(), Adapter(), {}, 17) + cursor.reset(OnlineResetResult(observation()), "episode") + actions = {"action_x": 123.0, "action_y": 234.0} + result = cursor.step(PolicyDecision(actions, actions, 0.0, 0.0, 0.0)) + assert cursor.ended and result.truncated + assert result.observation is not None + assert isinstance(result.observation["__image:camera"], np.ndarray) + assert isinstance( + dict(cursor.history.current.observation)["__image:camera"], np.ndarray + ) + assert cursor.inputs.inputs["action_x"][0, -1] == 123.0 + + +class CountingBackbone(nn.Module): + def __init__(self): + super().__init__() + self.weight = nn.Parameter(torch.ones(3), requires_grad=False) + self.seen = 0 + + def forward(self, values): + self.seen += values.shape[0] + return values.mean(dim=(-1, -2)) * self.weight + + +def test_frozen_encoder_skips_padding_and_preserves_projection_gradients(): + # Exercise the real encoder methods using a tiny stand-in backbone; no + # pretrained weights, network access, model optimizer, or training run. + encoder = ImageEncoder.__new__(ImageEncoder) + nn.Module.__init__(encoder) + encoder.frozen = True + encoder.backbone = CountingBackbone() + encoder.projection = nn.Linear(3, 2) + encoder.output_dim = 2 + encoder.register_buffer("mean", torch.zeros(1, 3, 1, 1)) + encoder.register_buffer("std", torch.ones(1, 3, 1, 1)) + encoder.eval() + images = ( + torch.arange(6 * 3 * 8 * 8) + .remainder(256) + .to(torch.uint8) + .reshape(2, 3, 1, 3, 8, 8) + ) + valid = torch.tensor([[[False], [False], [True]], [[False], [True], [True]]]) + dense = encoder._encode(images.reshape(-1, 3, 8, 8)).reshape(2, 3, 1, 2) + (dense * valid[..., None]).sum().backward() + gradients = { + name: p.grad.clone() for name, p in encoder.projection.named_parameters() + } + encoder.zero_grad() + encoder.backbone.seen = 0 + sparse = encoder(images, valid) + (sparse * valid[..., None]).sum().backward() + assert encoder.backbone.seen == int(valid.sum()) + torch.testing.assert_close(sparse[valid], dense[valid]) + for name, parameter in encoder.projection.named_parameters(): + torch.testing.assert_close(parameter.grad, gradients[name]) diff --git a/tests/unit/test_normalization_parameter_groups.py b/tests/unit/test_normalization_parameter_groups.py deleted file mode 100644 index 36b59f6cd..000000000 --- a/tests/unit/test_normalization_parameter_groups.py +++ /dev/null @@ -1,61 +0,0 @@ -import pytest -from torch import nn - -from sequifier.config.freezing_config import LayerFreezingConfigFields -from sequifier.model.parameter_catalog import ( - ParameterCatalog, - semantic_optimizer_groups, -) -from sequifier.training.runtime import _policy_parameter_ids - - -@pytest.mark.parametrize( - "normalization", - [ - lambda: nn.GroupNorm(2, 4), - lambda: nn.BatchNorm1d(4), - lambda: nn.BatchNorm2d(4), - lambda: nn.BatchNorm3d(4), - ], -) -def test_normalization_classification_freezing_and_optimizer_routing(normalization): - model = nn.Module() - model.ingestion = nn.Sequential(normalization(), nn.Linear(4, 4)) - model.backbone = nn.LayerNorm(4) - model.backbone.weight._sequifier_depth_parameter = True - - catalog = ParameterCatalog(model) - ingestion_normalization = catalog.select( - component="ingestion", semantic_group="normalization" - ) - backbone_normalization = catalog.select( - component="backbone", semantic_group="normalization" - ) - assert {item.parameter_kind for item in ingestion_normalization} == { - "weight", - "bias", - } - assert {item.parameter_kind for item in backbone_normalization} == { - "weight", - "bias", - } - - normalization_ids = { - id(catalog.parameter(item.parameter_id)) for item in ingestion_normalization - } - frozen = _policy_parameter_ids( - model.ingestion, - LayerFreezingConfigFields(freeze=["normalization"]), - "ingestion", - ) - assert frozen == normalization_ids - - optimizer_groups = { - group["group_id"]: {id(parameter) for parameter in group["params"]} - for group in semantic_optimizer_groups(catalog) - } - assert optimizer_groups["ingestion.normalization"] == normalization_ids - assert optimizer_groups["backbone.normalization"] == { - id(catalog.parameter(item.parameter_id)) for item in backbone_normalization - } - assert "ingestion.free_parameter" not in optimizer_groups diff --git a/tests/unit/test_static_preprocessing.py b/tests/unit/test_static_preprocessing.py deleted file mode 100644 index ac2a5b12c..000000000 --- a/tests/unit/test_static_preprocessing.py +++ /dev/null @@ -1,46 +0,0 @@ -"""Regression coverage for reused static preprocessing metadata.""" - -import json - -import polars as pl - -from sequifier.preprocess import Preprocessor - - -def test_reused_integer_static_categories_keep_their_ids(tmp_path): - metadata_path = tmp_path / "configs" / "metadata_configs" / "events.json" - metadata_path.parent.mkdir(parents=True) - (tmp_path / "data").mkdir() - existing = { - "static_columns": ["category"], - "n_classes": {"category": 5}, - "column_data_types": {"category": "Int64"}, - "id_maps": {"category": {10: 3, 20: 4}}, - "selected_columns_statistics": {}, - } - metadata_path.write_text(json.dumps(existing)) - - preprocessor = Preprocessor.__new__(Preprocessor) - preprocessor._static_source = pl.DataFrame( - {"sequenceId": [1, 2], "category": [10, 20]} - ) - preprocessor._preexisting_static_metadata = json.loads(metadata_path.read_text()) - preprocessor.static_categorical_columns = ["category"] - preprocessor.static_real_columns = [] - preprocessor.normalize_real_columns = True - preprocessor.project_root = str(tmp_path) - preprocessor.data_name_root = "events" - preprocessor.merge_output = True - preprocessor.output_n_classes = {} - preprocessor.output_column_data_types = {} - preprocessor.output_id_maps = {} - preprocessor.output_selected_columns_statistics = {} - - preprocessor._finalize_static_source() - - processed = pl.read_parquet(tmp_path / "data" / "events-static.parquet") - assert processed["category"].to_list() == [3, 4] - assert json.loads(metadata_path.read_text())["id_maps"]["category"] == { - "10": 3, - "20": 4, - } diff --git a/tools/check_contract_versions.py b/tools/check_contract_versions.py new file mode 100644 index 000000000..5ca3e64a0 --- /dev/null +++ b/tools/check_contract_versions.py @@ -0,0 +1,214 @@ +"""Check declared contract versions and shipped examples.""" + +import ast +import json +import re +from pathlib import Path + +REGISTRY = "src/sequifier/contract_versions.py" + + +def registry(source): + definition = next( + node + for node in ast.parse(source).body + if isinstance(node, ast.ClassDef) and node.name == "ContractVersions" + ) + values = {} + for node in definition.body: + if isinstance(node, ast.Expr) and isinstance(node.value, ast.Constant): + continue + if ( + not isinstance(node, ast.Assign) + or len(node.targets) != 1 + or not isinstance(node.targets[0], ast.Name) + ): + raise ValueError("ContractVersions requires plain class assignments") + name = node.targets[0].id + if name in values: + raise ValueError(f"Duplicate ContractVersions attribute {name}") + values[name] = ast.literal_eval(node.value) + return values + + +def validate_versions(values): + for name, value in values.items(): + if not isinstance(value, int) or isinstance(value, bool) or value < 1: + raise ValueError(f"{name} must be a positive integer") + + +VERSION_FIELDS = { + "version", + "format_version", + "schema_version", + "manifest_version", + "stored_window_layout_version", + "tensor_payload_version", + "protocol_version", + "initialization_policy_version", + "initialization_seed_derivation_version", + "opset_version", +} + + +POSITIONAL_VERSION_FIELDS = { + "StoredWindowLayout": (2, "version"), + "ModelArtifact": (0, "format_version"), +} + + +def assignment_name(target): + if isinstance(target, ast.Name): + return target.id + if isinstance(target, ast.Attribute): + return target.attr + if isinstance(target, ast.Subscript) and isinstance(target.slice, ast.Constant): + return target.slice.value + return None + + +def has_integer_literal(value): + if isinstance(value, ast.IfExp): + return has_integer_literal(value.body) or has_integer_literal(value.orelse) + return isinstance(value, ast.Constant) and ( + isinstance(value.value, int) + and not isinstance(value.value, bool) + or isinstance(value.value, str) + and value.value.isdecimal() + ) + + +def hardcoded_versions(tree): + """Find literal versions at Python serialization and declaration sites.""" + for node in ast.walk(tree): + assignments = () + if isinstance(node, ast.Dict): + assignments = ( + (key.value if isinstance(key, ast.Constant) else None, value) + for key, value in zip(node.keys, node.values) + ) + elif isinstance(node, ast.Assign): + assignments = ( + (assignment_name(target), node.value) for target in node.targets + ) + elif isinstance(node, ast.AnnAssign): + name = assignment_name(node.target) + assignments = ((name, node.value),) + if name in VERSION_FIELDS and isinstance(node.value, ast.Call): + assignments += tuple( + (name, argument.value) + for argument in node.value.keywords + if argument.arg == "default" + ) + if ( + isinstance(name, str) + and name in VERSION_FIELDS + and isinstance(node.annotation, ast.Subscript) + and isinstance(node.annotation.value, ast.Name) + and node.annotation.value.id == "Literal" + ): + annotation = node.annotation.slice + candidates = ( + annotation.elts + if isinstance(annotation, ast.Tuple) + else [annotation] + ) + assignments += tuple((name, candidate) for candidate in candidates) + elif ( + isinstance(node, ast.Call) + and isinstance(node.func, ast.Name) + and node.func.id in POSITIONAL_VERSION_FIELDS + ): + index, name = POSITIONAL_VERSION_FIELDS[node.func.id] + if index < len(node.args): + assignments = ((name, node.args[index]),) + elif isinstance(node, ast.keyword): + assignments = ((node.arg, node.value),) + for name, value in assignments: + if ( + isinstance(node, (ast.stmt, ast.expr, ast.keyword)) + and ( + name in VERSION_FIELDS + or isinstance(name, str) + and name.endswith("_VERSION") + ) + and has_integer_literal(value) + ): + yield node.lineno + if ( + isinstance(node, ast.Constant) + and isinstance(node.value, str) + and re.search(r"^sequifier[.-][\w.-]+[.-]v[0-9]+(?:$|:)", node.value) + ): + yield node.lineno + if isinstance(node, ast.FunctionDef) and node.name == "protocol_version": + for statement in ast.walk(node): + if ( + isinstance(statement, ast.Return) + and isinstance(statement.value, ast.Constant) + and isinstance(statement.value.value, int) + and not isinstance(statement.value.value, bool) + ): + yield statement.lineno + + +def validate_boundaries(root, values): + errors = [] + for directory in ("src/sequifier", "documentation/example_inputs", "tools"): + for path in (root / directory).rglob("*.py"): + if path.relative_to(root).as_posix() in { + REGISTRY, + "tools/check_contract_versions.py", + }: + continue + tree = ast.parse(path.read_text(), filename=str(path)) + errors.extend( + f"{path.relative_to(root)}:{line}: reference ContractVersions" + for line in hardcoded_versions(tree) + ) + + versioned_identity = re.compile(r"sequifier[.-][\w.-]+\.v[0-9]+\b") + for directory in ( + "src/sequifier/templates", + "documentation/example_inputs/configs", + ): + for path in (root / directory).rglob("*"): + if path.suffix in {".yaml", ".yml"} and versioned_identity.search( + path.read_text() + ): + errors.append( + f"{path.relative_to(root)}: use a version-free adapter config" + ) + + metadata_path = ( + "documentation/example_inputs/configs/ddconfigs/preprocessing_input.json" + ) + metadata = json.loads((root / metadata_path).read_text()) + if metadata.get("tensor_payload_kind") != "tabular": + errors.append(f"{metadata_path}: tensor_payload_kind must be tabular") + for field, name in ( + ("tensor_payload_version", "DATASET_TENSOR_METADATA"), + ("stored_window_layout_version", "STORED_WINDOW_LAYOUT"), + ): + actual = metadata.get(field) + if ( + not isinstance(actual, int) + or isinstance(actual, bool) + or actual != values[name] + ): + errors.append( + f"{metadata_path}: {field} must match ContractVersions.{name}" + ) + if errors: + raise ValueError("\n".join(errors)) + + +def main(): + root = Path(__file__).resolve().parents[1] + current = registry((root / REGISTRY).read_text()) + validate_versions(current) + validate_boundaries(root, current) + + +if __name__ == "__main__": + main() diff --git a/tools/convert_folder.sh b/tools/convert_folder.sh deleted file mode 100644 index 32a9e759e..000000000 --- a/tools/convert_folder.sh +++ /dev/null @@ -1,27 +0,0 @@ -#!/bin/bash - -# Check if a directory is provided -if [ -z "$1" ]; then - echo "Usage: $0 " - exit 1 -fi - -# Find all yaml files recursively and loop through them -find "$1" -type f -name "*.yaml" | while read -r filepath; do - # Get just the filename for checking patterns - filename=$(basename "$filepath") - - # logic: check strict exclusion first - if [[ "$filename" == *"train"* && "$filename" == *"infer"* ]]; then - echo "Skipping ambiguous file (contains both train/infer): $filepath" - continue - fi - - if [[ "$filename" == *"infer"* ]]; then - echo "Converting INFER config: $filepath" - python tools/convert_v0_config_to_v1.py "infer" "$filepath" - elif [[ "$filename" == *"train"* ]]; then - echo "Converting TRAIN config: $filepath" - python tools/convert_v0_config_to_v1.py "train" "$filepath" - fi -done diff --git a/tools/convert_to_pt.py b/tools/convert_to_pt.py index bb42f2ee1..9b27a98be 100644 --- a/tools/convert_to_pt.py +++ b/tools/convert_to_pt.py @@ -1,102 +1,69 @@ -import os -import sys +"""Convert current tabular windows to Sequifier tensor batches.""" + +import argparse +from pathlib import Path -import numpy as np import polars as pl import torch +from sequifier.io.pt_payload import save_pt_payload, tabular_tensor_batch +from sequifier.io.sample_order import ( + curriculum_storage_columns, + source_curriculum_column, +) -def main(): - assert ( - len(sys.argv) >= 2 - ), "Usage: python convert_to_pt.py [output_path] [chunk_size]" - in_path = sys.argv[1] - out_path = ( - sys.argv[2] if len(sys.argv) > 2 else os.path.splitext(in_path)[0] + ".pt" - ) - chunk_size = int(sys.argv[3]) if len(sys.argv) > 3 else None - - assert os.path.exists(in_path), f"Input file not found: {in_path}" - if in_path.endswith(".csv"): - df = pl.read_csv(in_path) - elif in_path.endswith(".parquet"): - df = pl.read_parquet(in_path) +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("input_file", type=Path) + parser.add_argument("output_path", type=Path, nargs="?") + parser.add_argument("chunk_size", type=int, nargs="?") + args = parser.parse_args() + source = args.input_file + output = args.output_path or source.with_suffix(".pt") + if args.chunk_size is not None and args.chunk_size <= 0: + parser.error("chunk_size must be positive") + if source.suffix == ".csv": + data = pl.read_csv(source) + elif source.suffix == ".parquet": + data = pl.read_parquet(source) else: - raise ValueError("Input must be .csv or .parquet") - - required_meta = {"sequenceId", "subsequenceId", "startItemPosition", "inputCol"} - assert required_meta.issubset( - df.columns - ), f"Missing columns: {required_meta - set(df.columns)}" - - seq_cols = sorted([c for c in df.columns if c.isdigit()], key=int) - assert len(seq_cols) > 1, "No numbered sequence columns found" - - feature_names = df["inputCol"].unique().to_list() - aggs = [ - pl.concat_list(seq_cols).filter(pl.col("inputCol") == f).flatten().alias(f) - for f in feature_names - ] - aggs.append(pl.col("startItemPosition").first()) - - grouped = df.group_by(["sequenceId", "subsequenceId"], maintain_order=True).agg( - aggs + parser.error("input must be CSV or Parquet") + + sequence_columns = [name for name in data.columns if name.isdigit()] + if not sequence_columns: + parser.error("input must contain numbered sequence columns") + features = data.get_column("inputCol").unique().sort().to_list() + dtype = ( + torch.float32 + if any(data.schema[name].is_float() for name in sequence_columns) + else torch.int64 ) - grouped = grouped.sort(["sequenceId", "subsequenceId"]) - - seq_ids = torch.from_numpy(grouped["sequenceId"].to_numpy().astype(np.int64)) - sub_ids = torch.from_numpy(grouped["subsequenceId"].to_numpy().astype(np.int64)) - start_pos = torch.from_numpy( - grouped["startItemPosition"].to_numpy().astype(np.int64) + batch = tabular_tensor_batch( + data, dict.fromkeys(features, dtype), features, len(sequence_columns) ) + curriculum = curriculum_storage_columns(data.columns) + if curriculum: + positions = ( + data.group_by("sequenceId", "subsequenceId") + .agg(pl.col(name).first() for name in curriculum) + .sort("sequenceId", "subsequenceId") + ) + batch.sample_positions = torch.tensor(positions.to_numpy(), dtype=torch.int64) + batch.curriculum_columns = tuple( + source_curriculum_column(name) for name in curriculum + ) - sequences_dict = {} - targets_dict = {} - - for f in feature_names: - is_float = df.schema[seq_cols[0]] in (pl.Float32, pl.Float64) - dtype = torch.float32 if is_float else torch.int64 - - data_np = np.stack(grouped[f].to_list()) - tensor = torch.tensor(data_np, dtype=dtype) - - sequences_dict[f] = tensor[:, :-1] - targets_dict[f] = tensor[:, 1:] - - total_rows = len(seq_ids) - - if chunk_size is not None and total_rows > chunk_size: - assert not os.path.isfile( - out_path - ), f"Output path '{out_path}' exists as a file, but chunking requires a directory." - os.makedirs(out_path, exist_ok=True) - - print(f"Converting {in_path} -> {out_path}/ (chunked by {chunk_size})") - dirname = os.path.split(out_path)[-1] - - for i in range(0, total_rows, chunk_size): - end = i + chunk_size - - s_ids_chunk = seq_ids[i:end] - sub_ids_chunk = sub_ids[i:end] - start_pos_chunk = start_pos[i:end] - seq_chunk = {k: v[i:end] for k, v in sequences_dict.items()} - tgt_chunk = {k: v[i:end] for k, v in targets_dict.items()} - - file_name = f"{dirname}-{i // chunk_size}.pt" - full_name = os.path.join(out_path, file_name) - - torch.save( - (seq_chunk, tgt_chunk, s_ids_chunk, sub_ids_chunk, start_pos_chunk), - full_name, - ) + total = len(batch.sequence_ids) + if args.chunk_size is None or total <= args.chunk_size: + save_pt_payload(batch, output) else: - print(f"Converting {in_path} -> {out_path}") - print(f"Samples: {total_rows} | Features: {feature_names}") - torch.save( - (sequences_dict, targets_dict, seq_ids, sub_ids, start_pos), out_path - ) + output.mkdir(parents=True, exist_ok=True) + for index, start in enumerate(range(0, total, args.chunk_size)): + save_pt_payload( + batch.slice_rows(start, start + args.chunk_size), + output / f"{output.name}-{index}.pt", + ) if __name__ == "__main__": diff --git a/tools/convert_v0_config_to_v1.py b/tools/convert_v0_config_to_v1.py deleted file mode 100644 index b64885d6e..000000000 --- a/tools/convert_v0_config_to_v1.py +++ /dev/null @@ -1,224 +0,0 @@ -import argparse - -import yaml - - -# Helper to rename keys while preserving insertion order -def rename_key(d, old, new, transform=None): - if old not in d: - return - # Create a new dict to preserve order - new_d = {} - for k, v in d.items(): - if k == old: - new_d[new] = transform(v) if transform is not None else v - else: - new_d[k] = v - d.clear() - d.update(new_d) - - -def convert_preprocess(config): - rename_key(config, "project_path", "project_root") - rename_key(config, "data_path", "preprocessing_data_path") - rename_key(config, "column_types", "column_data_types") - - assert "group_proportions" in config, "group_proportions missing" - rename_key(config, "group_proportions", "split_ratios") - - if "seq_step_sizes" in config: - steps = config.pop("seq_step_sizes") - if not isinstance(steps, list) or not steps or len(set(steps)) != 1: - raise ValueError( - "seq_step_sizes must contain one repeated stride; choose a scalar window_stride manually" - ) - config["window_stride"] = steps[0] - - # Rename and validate merge settings - if "combine_into_single_file" in config: - config.pop("combine_into_single_file") - - # Enforce new v1.0 logic for merge_output based on format - write_format = config.get("write_format", "parquet") - if write_format == "pt": - config["merge_output"] = False - elif write_format == "parquet": - # Keep the user's choice, defaulting to True if not present to mimic old behavior - config["merge_output"] = config.get("merge_output", True) - else: - config["merge_output"] = True - - if "seed" not in config: - config["seed"] = 1010 - - return config - - -def convert_model(ms, is_hp_search=False): - # Mapping old keys to new keys - if "d_model" in ms: - val = ms["d_model"] - rename_key(ms, "d_model", "dim_model") - ms["initial_embedding_dim"] = val - - if is_hp_search: - ms["joint_embedding_dim"] = [None] * len(val) - - if "d_model_by_column" in ms: - rename_key(ms, "d_model_by_column", "feature_embedding_dims") - - if "nhead" in ms: - rename_key(ms, "nhead", "n_heads") - - if "d_hid" in ms: - rename_key(ms, "d_hid", "dim_feedforward") - - if "nlayers" in ms: - rename_key(ms, "nlayers", "num_layers") - - if "inference_size" in ms: - rename_key(ms, "inference_size", "prediction_length") - elif is_hp_search: - ms["prediction_length"] = 1 - - # Add v1 defaults - if is_hp_search: - ms["activation"] = ["swiglu"] - ms["normalization"] = ["rmsnorm"] - ms["positional_encoding"] = ["learned"] - ms["attention_type"] = ["mha"] - ms["norm_first"] = [True] - ms["n_kv_heads"] = [None] - ms["rope_theta"] = [10000.0] - else: - ms["activation"] = "swiglu" - ms["normalization"] = "rmsnorm" - ms["positional_encoding"] = "learned" - ms["attention_type"] = "mha" - ms["norm_first"] = True - ms["rope_theta"] = 10000.0 - - return ms - - -def convert_training_spec(ts, is_hp_search=False): - if "lr" in ts: - rename_key(ts, "lr", "learning_rate") - - if "iter_save" in ts: - rename_key(ts, "iter_save", "save_interval_epochs") - - if "scheduler_step_iter" in ts: - rename_key(ts, "scheduler_step_iter", "scheduler_step_on") - else: - ts["scheduler_step_on"] = "epoch" - - if not is_hp_search: - ts["enforce_determinism"] = False - ts["max_ram_gb"] = 16 - - return ts - - -def convert_train(config): - rename_key(config, "project_path", "project_root") - rename_key(config, "ddconfig_path", "metadata_config_path") - rename_key(config, "selected_columns", "input_columns") - - if "seed" not in config: - config["seed"] = 1010 - - assert "model_spec" in config, "model_spec missing" - rename_key(config, "model_spec", "model", convert_model) - - assert "training_spec" in config, "training_spec missing" - config["training_spec"] = convert_training_spec(config["training_spec"]) - for field_name in ("training_objective", "device"): - if field_name in config["training_spec"]: - config[field_name] = config["training_spec"].pop(field_name) - - return config - - -def convert_infer(config): - rename_key(config, "project_path", "project_root") - rename_key(config, "ddconfig_path", "metadata_config_path") - rename_key(config, "selected_columns", "input_columns") - rename_key( - config, - "autoregression_extra_steps", - "generation_steps", - lambda val: val + 1 if val is not None else val, - ) - - if "inference_size" in config: - rename_key(config, "inference_size", "prediction_length") - - if "seed" not in config: - config["seed"] = 1010 - - config["deterministic"] = False - - return config - - -def convert_hp_search(config): - rename_key(config, "project_path", "project_root") - rename_key(config, "ddconfig_path", "metadata_config_path") - rename_key(config, "selected_columns", "input_columns") - - # Add new export flags - config["export_generative_model"] = True - config["export_embedding_model"] = False - config["override_input"] = False - - assert ( - "model_hyperparameter_sampling" in config - ), "model_hyperparameter_sampling missing" - config["model_hyperparameter_sampling"] = convert_model( - config["model_hyperparameter_sampling"], is_hp_search=True - ) - - assert ( - "training_hyperparameter_sampling" in config - ), "training_hyperparameter_sampling missing" - config["training_hyperparameter_sampling"] = convert_training_spec( - config["training_hyperparameter_sampling"], is_hp_search=True - ) - - return config - - -def main(): - parser = argparse.ArgumentParser() - parser.add_argument("config_type", type=str) - parser.add_argument("file_path", type=str) - args = parser.parse_args() - - assert args.config_type in [ - "preprocess", - "train", - "infer", - "hyperparameter-search", - ], "Invalid config-type" - - with open(args.file_path, "r") as f: - config = yaml.safe_load(f) - - if args.config_type == "preprocess": - new_config = convert_preprocess(config) - elif args.config_type == "train": - new_config = convert_train(config) - elif args.config_type == "infer": - new_config = convert_infer(config) - elif args.config_type == "hyperparameter-search": - new_config = convert_hp_search(config) - else: - new_config = {} - - with open(args.file_path, "w") as f: - f.write(yaml.dump(new_config, sort_keys=False, default_flow_style=False)) - - -if __name__ == "__main__": - main() diff --git a/tools/extract_validation_losses_to_table.py b/tools/extract_validation_losses_to_table.py index b42736d1e..1cfb38370 100644 --- a/tools/extract_validation_losses_to_table.py +++ b/tools/extract_validation_losses_to_table.py @@ -14,7 +14,6 @@ def process_metrics(file_path: str) -> Path: losses["measurement"] = losses["metric"] + "." + losses["target"] context_columns = [ - "schema_version", "run_id", "session_id", "timestamp_utc", diff --git a/tools/resize_pt_files.py b/tools/resize_pt_files.py index ae7ff89ea..b16f9df6a 100644 --- a/tools/resize_pt_files.py +++ b/tools/resize_pt_files.py @@ -1,225 +1,103 @@ +"""Rechunk current Sequifier tensor batches, retaining split metadata.""" + import argparse import json import os import sys -from typing import Any, Dict, List, Tuple +from collections import Counter +from pathlib import Path import torch - -def unpack_dataset_tuple(data_tuple: Tuple) -> Dict[str, Any]: - """Unpack the standard Sequifier PT tuple.""" - return { - "sequences": data_tuple[0], - "targets": data_tuple[1], - "seq_ids": data_tuple[2], - "sub_ids": data_tuple[3], - "start_pos": data_tuple[4], - } - - -def pack_dataset_tuple(data_dict: Dict[str, Any]) -> Tuple: - """Pack a dataset dict into the standard Sequifier PT tuple.""" - return ( - data_dict["sequences"], - data_dict["targets"], - data_dict["seq_ids"], - data_dict["sub_ids"], - data_dict["start_pos"], - ) - - -def concat_dataset_list(data_list: List[Dict[str, Any]]) -> Dict[str, Any]: - """Concatenate dataset dictionaries along dim 0.""" - if not data_list: - return {} - - if len(data_list) == 1: - return data_list[0] - - combined = {} - first = data_list[0] - - combined["sequences"] = { - k: torch.cat([d["sequences"][k] for d in data_list], dim=0) - for k in first["sequences"] - } - - combined["targets"] = { - k: torch.cat([d["targets"][k] for d in data_list], dim=0) - for k in first["targets"] - } - - combined["seq_ids"] = torch.cat([d["seq_ids"] for d in data_list], dim=0) - combined["sub_ids"] = torch.cat([d["sub_ids"] for d in data_list], dim=0) - combined["start_pos"] = torch.cat([d["start_pos"] for d in data_list], dim=0) - - return combined - - -def slice_dataset(data: Dict[str, Any], start: int, end: int) -> Dict[str, Any]: - """Slice a dataset dictionary without cloning.""" - sliced = {} - sliced["sequences"] = {k: v[start:end] for k, v in data["sequences"].items()} - sliced["targets"] = {k: v[start:end] for k, v in data["targets"].items()} - sliced["seq_ids"] = data["seq_ids"][start:end] - sliced["sub_ids"] = data["sub_ids"][start:end] - sliced["start_pos"] = data["start_pos"][start:end] - return sliced - - -def clone_dataset(data: Dict[str, Any]) -> Dict[str, Any]: - """Clone all tensors in a dataset dictionary.""" - cloned = {} - cloned["sequences"] = {k: v.clone() for k, v in data["sequences"].items()} - cloned["targets"] = {k: v.clone() for k, v in data["targets"].items()} - cloned["seq_ids"] = data["seq_ids"].clone() - cloned["sub_ids"] = data["sub_ids"].clone() - cloned["start_pos"] = data["start_pos"].clone() - return cloned - - -def get_row_size_bytes(data: Dict[str, Any]) -> float: - """Return bytes per row across sequence/target tensors.""" - total_bytes = 0 - for t in data["sequences"].values(): - total_bytes += t.element_size() * t.shape[1] - for t in data["targets"].values(): - total_bytes += t.element_size() * t.shape[1] - - # Add metadata sizes (1 element each) - total_bytes += data["seq_ids"].element_size() - total_bytes += data["sub_ids"].element_size() - total_bytes += data["start_pos"].element_size() - - return total_bytes - - -def process_split( - input_dir: str, - output_dir: str, - dataset_name: str, - target_size_mb: float, - split_suffix: str, -): - os.makedirs(output_dir, exist_ok=True) - - meta_path = os.path.join(input_dir, "metadata.json") - if os.path.exists(meta_path): - with open(meta_path, "r") as f: - old_metadata = json.load(f) - input_files = [entry["path"] for entry in old_metadata.get("batch_files", [])] - expected_total_samples = old_metadata.get("total_samples", 0) - else: - input_files = sorted([f for f in os.listdir(input_dir) if f.endswith(".pt")]) - expected_total_samples = None - print( - f"Warning: No metadata.json found in {input_dir}. Using alphabetical sort." - ) - - data_buffer: List[Dict[str, Any]] = [] - buffer_row_count = 0 - - output_batch_idx = 0 - new_batch_files_metadata = [] - total_samples_processed = 0 - - target_bytes = target_size_mb * 1024 * 1024 +from sequifier.config.metadata import load_dataset_metadata +from sequifier.io.atomic import write_json_atomic +from sequifier.io.pt_payload import ( + concatenate_pt_batches, + load_pt_payload, + save_pt_payload, +) + + +def process_split(input_dir, output_dir, dataset_name, target_size_mb, split_suffix): + source = Path(input_dir) + destination = Path(output_dir) + metadata = load_dataset_metadata(str(source / "metadata.json")) + manifest = json.loads((source / "metadata.json").read_text()) + options = {"layouts": metadata.depth_layouts, "n_classes": metadata.n_classes} + destination.mkdir(parents=True, exist_ok=True) + entries = [] + batches = [] + buffered_rows = 0 target_rows = None - print(f"Processing {input_dir} -> {output_dir}") - - for file_name in input_files: - file_path = os.path.join(input_dir, file_name) - if not os.path.exists(file_path): - continue - - try: - loaded_tuple = torch.load(file_path, map_location="cpu", weights_only=False) - current_data = unpack_dataset_tuple(loaded_tuple) - - current_rows = current_data["seq_ids"].shape[0] - if current_rows == 0: - continue - - if target_rows is None: - bytes_per_row = get_row_size_bytes(current_data) - target_rows = max(1, int(target_bytes / bytes_per_row)) - - data_buffer.append(current_data) - buffer_row_count += current_rows - - if buffer_row_count >= target_rows: - full_data = concat_dataset_list(data_buffer) - - data_buffer = [] - buffer_row_count = 0 - - num_rows = full_data["seq_ids"].shape[0] - - start_idx = 0 - while start_idx + target_rows <= num_rows: - end_idx = start_idx + target_rows - - chunk_data = slice_dataset(full_data, start_idx, end_idx) - - fname = f"{dataset_name}-{split_suffix}-{output_batch_idx}.pt" - out_path = os.path.join(output_dir, fname) - torch.save(pack_dataset_tuple(chunk_data), out_path) - - chunk_len = end_idx - start_idx - new_batch_files_metadata.append( - {"path": fname, "samples": chunk_len} - ) - total_samples_processed += chunk_len - output_batch_idx += 1 - - start_idx = end_idx - - if start_idx < num_rows: - remainder_data = clone_dataset( - slice_dataset(full_data, start_idx, num_rows) - ) - data_buffer = [remainder_data] - buffer_row_count = remainder_data["seq_ids"].shape[0] - - del full_data - - except Exception as e: - print(f"Error processing {file_path}: {e}") - sys.exit(1) - - if buffer_row_count > 0: - full_data = concat_dataset_list(data_buffer) - fname = f"{dataset_name}-{split_suffix}-{output_batch_idx}.pt" - out_path = os.path.join(output_dir, fname) - torch.save(pack_dataset_tuple(full_data), out_path) - - chunk_len = full_data["seq_ids"].shape[0] - new_batch_files_metadata.append({"path": fname, "samples": chunk_len}) - total_samples_processed += chunk_len - - new_metadata = { - "total_samples": total_samples_processed, - "batch_files": new_batch_files_metadata, - } - with open(os.path.join(output_dir, "metadata.json"), "w") as f: - json.dump(new_metadata, f, indent=4) - - if ( - expected_total_samples is not None - and total_samples_processed != expected_total_samples - ): - print( - f"WARNING: Sample count mismatch! Input: {expected_total_samples}, Output: {total_samples_processed}" + def write_shard(batch): + filename = f"{dataset_name}-{split_suffix}-{len(entries)}.pt" + save_pt_payload(batch, destination / filename, **options) + first_target = torch.maximum( + batch.left_pad_lengths, + batch.split_start_item_positions - batch.start_item_positions, ) - else: - print( - f"Success: {split_suffix} processed. Total samples: {total_samples_processed}" + entries.append( + { + "path": filename, + "samples": len(batch.sequence_ids), + "left_pad_length_histogram": dict( + Counter(batch.left_pad_lengths.tolist()) + ), + "target_valid_from_histogram": dict(Counter(first_target.tolist())), + } ) + for entry in manifest["batch_files"]: + batch = load_pt_payload(source / entry["path"], **options) + count = len(batch.sequence_ids) + if count != entry["samples"]: + raise ValueError( + f"{entry['path']}: tensor batch size differs from metadata" + ) + if count == 0: + continue + if target_rows is None: + size = sum( + tensor.numel() * tensor.element_size() + for tensor in ( + *batch.sequences.values(), + *batch.depth_valid_masks.values(), + batch.sequence_ids, + batch.subsequence_ids, + batch.start_item_positions, + batch.left_pad_lengths, + batch.split_start_item_positions, + batch.split_end_item_positions, + batch.sample_positions, + ) + if tensor is not None + ) + target_rows = max(1, int(target_size_mb * 1024 * 1024 * count / size)) + batches.append(batch) + buffered_rows += count + if buffered_rows >= target_rows: + combined = concatenate_pt_batches(batches) + complete_rows = buffered_rows // target_rows * target_rows + for start in range(0, complete_rows, target_rows): + write_shard(combined.slice_rows(start, start + target_rows)) + batches = ( + [combined.slice_rows(complete_rows, buffered_rows)] + if complete_rows < buffered_rows + else [] + ) + buffered_rows -= complete_rows + if batches: + write_shard(concatenate_pt_batches(batches)) + count = sum(entry["samples"] for entry in entries) + if count != manifest["total_samples"]: + raise ValueError(f"{source}: total sample count differs from metadata") + write_json_atomic( + destination / "metadata.json", + {**manifest, "total_samples": count, "batch_files": entries}, + ) + -# ... [main function remains unchanged] ... def main(): parser = argparse.ArgumentParser( description="Fast Rechunker for Sequifier Datasets" @@ -229,6 +107,8 @@ def main(): parser.add_argument("target_size_mb", type=float, help="Target file size in MB") args = parser.parse_args() + if args.target_size_mb <= 0: + parser.error("target_size_mb must be positive") if not os.path.exists(args.data_folder): print("Data folder not found.") diff --git a/tools/stream_folder_via_scp.py b/tools/stream_folder_via_scp.py index f5bcf543b..5071394fc 100644 --- a/tools/stream_folder_via_scp.py +++ b/tools/stream_folder_via_scp.py @@ -69,6 +69,7 @@ def main(): ssh = paramiko.SSHClient() ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy()) + sftp = None try: ssh.connect(args.remote_host, username=args.remote_user, password=password) @@ -125,10 +126,9 @@ def main(): except Exception as e: print(f"Error: {e}") finally: - if "sftp" in locals(): - sftp.close() # type: ignore - if "ssh" in locals(): - ssh.close() + if sftp is not None: + sftp.close() + ssh.close() if __name__ == "__main__":