CorpusCustody

Problem · Install · Commands · Obligations · Rules · Worked run · Manifest · Exit codes · Limitations · Design

This is a mechanical obligation check, not legal advice. It matches SPDX identifiers against a short hand-maintained table and applies fixed rules. A PASS means no conflict was found in that table, not that a lawyer cleared the set. Treat every result as input to a human decision, never as the decision.

corpuscustody is a training-data license and provenance gate. It walks a dataset manifest, resolves the license of every record against an offline SPDX table, and decides whether the combined set is usable for a declared purpose: internal research, a commercial product, or redistribution. It flags incompatible combinations, such as a share-alike corpus mixed into a commercial release, and it refuses to emit a cleared manifest when any record's provenance is unknown rather than assuming permissive.

Python 3.11, standard library only. No third-party dependencies and no network access anywhere in the code.

The mixing problem

A training set is rarely one license. It is a pile of records pulled from many places: a permissive tokenizer here, a Wikipedia dump there, a scraped folder nobody labelled, one corpus grabbed under a non-commercial term and never flagged. Each record on its own is fine. The set is the problem.

The moment records are combined and put to a purpose, the obligations stack. A single share-alike record can force the whole release to carry share-alike terms. A single non-commercial record makes the set unsafe to sell. A single unlabelled record means you cannot honestly say what you are allowed to do with any of it. These conflicts are invisible in a per-record spreadsheet, because the danger lives in the combination and the declared use, not in any one row.

corpuscustody makes that combination explicit. You declare what you intend to do with the set. It resolves every record, applies the rules a careful reviewer would apply, and either clears the set for that purpose or refuses and names the records that blocked it, and why.

Install

Install as an editable package:

pip install -e .

Or run without installing, from the project root:

set PYTHONPATH=src
python -m corpuscustody version

corpuscustody 0.1.0

The set PYTHONPATH=src form is the Windows shell. On a POSIX shell use PYTHONPATH=src python -m corpuscustody version.

Commands

Four subcommands. All read a manifest; gate and report also require a declared --purpose.

Command	Purpose	Requires --purpose
resolve	parse a manifest, print each record's license	no
gate	run the pass or refuse decision for a purpose	yes
report	combined resolve view then gate view	yes
version	print the version	no

Run with no subcommand and it prints help and exits 2:

python -m corpuscustody

usage: corpuscustody [-h] {resolve,gate,report,version} ...

Training-data license and provenance gate.

positional arguments:
  {resolve,gate,report,version}
    resolve             parse a manifest and resolve each record's license
    gate                run the pass or refuse decision for a declared purpose
    report              print the combined resolve and gate report
    version             print the version

options:
  -h, --help            show this help message and exit

--purpose accepts exactly internal, commercial, or redistribute. It is case-insensitive: Commercial normalises to commercial. Pass --out PATH to gate to write a cleared manifest on PASS.

The obligation model

The model keeps attribution, share-alike, notice and source disclosure as separate obligations. Collapsing them into one flag is how a compliant-looking gate starts passing corpora it should refuse.

Every license in the offline table maps to a set of obligations. An obligation is a mechanical yes or no fact about the license, not a judgement. These five obligations are the entire vocabulary the tool reasons over.

Obligation	Key	What it means
Attribution	attribution	attribution or notices must be preserved
Share alike	share_alike	derivatives or the combined work must carry the same license
Non commercial	non_commercial	commercial use is not permitted
No derivatives	no_derivatives	modified or derived works are not permitted
Unknown	unknown	provenance or license is not established

The unknown obligation is special. It is never inferred as permissive. A record whose license cannot be resolved is treated as carrying every restrictive obligation at once, so it can never quietly pass a gate. That is defined once, in src/corpuscustody/spdx.py, as the UNKNOWN sentinel with all five flags set.

Purposes and why they differ

The same set can be safe for one use and unsafe for another. The purpose you declare changes which obligations block and which are only noted.

Purpose	Intent	Distribution
internal	internal research use, not distributed outside the org	none
commercial	a commercial product, distributed, possibly proprietary	yes, and for sale
redistribute	redistributed as a dataset or corpus to third parties	yes

Internal use is the most permissive purpose here, because nothing leaves the organisation. Share-alike and non-commercial obligations do not block internal research: you are not distributing, and you are not selling. Commercial use is the strictest, because it both distributes and sells. Redistribution sits between the two: it distributes, so no-derivatives blocks it, but it is assumed non-commercial, so a share-alike record is a note rather than a hard block.

The compatibility rules

Compatibility is decided between obligation sets, not between license names, so two spellings of the same terms behave identically. The conflicting pair is quoted in the finding.

These are the actual rules the code applies, transcribed from the _RULES table in src/corpuscustody/compat.py. A blank cell means the obligation raises no issue for that purpose. finding blocks the gate. note is informational and does not block.

Obligation	internal	commercial	redistribute
unknown	finding	finding	finding
share_alike	(none)	finding	note
non_commercial	(none)	finding	(none)
no_derivatives	note	finding	finding
attribution	note	note	note

Reading the table as prose, so the intent is unambiguous:

• unknown is a finding for every purpose. Unknown provenance never passes.
• share_alike is a finding for commercial, because a share-alike corpus mixed into a proprietary release forces that release to share alike. For redistribute it is a note: the combined set must then carry the terms.
• non_commercial is a finding for commercial only. Internal and redistribute are assumed non-commercial here, so it is not reported at all for internal and raises nothing for redistribute.
• no_derivatives is a finding for commercial and redistribute, because both typically transform or repackage the data, and a note for internal.
• attribution is never a blocking finding. It is always a note: attribution must be preserved.

Issues are emitted in record order, then in the obligation order declared in spdx.py (attribution, share_alike, non_commercial, no_derivatives, unknown), so identical input yields byte-identical output.

Unknown provenance is a refusal, not a warning

Unknown provenance is a refusal because no obligation set can be derived from it. A warning here would let the one case the gate exists for slip through as a note.

The single most consequential design choice is that an unresolvable license does not degrade to permissive and does not become a soft warning. It becomes a blocking finding for every purpose, and it makes the gate refuse.

Two things resolve to UNKNOWN: an empty license field, and the explicit token UNKNOWN. The resolver in spdx.py returns the UNKNOWN sentinel for both, and for any identifier not in the table. That sentinel carries all five obligations, so it trips the unknown finding no matter the purpose.

This default matters because the opposite choice fails silently and expensively. If unknown resolved to permissive, a scraped folder nobody labelled would sail through and land in a shipped product, and the first sign of trouble would be a takedown or a lawsuit. A refusal is loud and cheap: it stops the pipeline now, names the records, and asks a person to establish provenance before proceeding. That conservatism is deliberate and not adjustable by a flag.

A worked gate run

Below are the real decisions for each of the three sample manifests, captured by running the CLI in this repository. The manifests are described in samples/README.md; they are hand-authored test vectors, not scraped data.

permissive.manifest, commercial: PASS, 0 findings

Five permissive records, attribution at most. It clears for commercial with four attribution notes and no findings.

python -m corpuscustody gate samples/permissive.manifest --purpose commercial

purpose: commercial
records: 5
decision: PASS
findings: 0
notes: 4
  rec-0001 | MIT | attribution: attribution must be preserved
  rec-0002 | Apache-2.0 | attribution: attribution must be preserved
  rec-0004 | BSD-3-Clause | attribution: attribution must be preserved
  rec-0005 | CC-BY-4.0 | attribution: attribution must be preserved

Note that rec-0003 (CC0-1.0) produces no note at all: CC0 carries no obligations, so it is silent.

sharealike.manifest, commercial: REFUSE, 3 findings

The same set that is fine internally mixes share-alike and non-commercial records into a commercial release. Three findings, and the process exits 1.

python -m corpuscustody gate samples/sharealike.manifest --purpose commercial

purpose: commercial
records: 5
decision: REFUSE
findings: 3
  rec-1002 | CC-BY-SA-4.0 | share_alike: share-alike terms attach to the combined set
  rec-1004 | CC-BY-NC-4.0 | non_commercial: non-commercial license, commercial use blocked
  rec-1005 | GPL-3.0-only | share_alike: share-alike terms attach to the combined set
notes: 5
  rec-1001 | MIT | attribution: attribution must be preserved
  rec-1002 | CC-BY-SA-4.0 | attribution: attribution must be preserved
  rec-1003 | Apache-2.0 | attribution: attribution must be preserved
  rec-1004 | CC-BY-NC-4.0 | attribution: attribution must be preserved
  rec-1005 | GPL-3.0-only | attribution: attribution must be preserved

sharealike.manifest, internal and redistribute: PASS

The exact same records pass for internal (share-alike and non-commercial do not block internal use, leaving the five attribution notes) and for redistribute (a PASS with seven notes, where the two share-alike obligations become notes rather than the commercial findings above). Only the declared purpose changed; the records did not.

unknown.manifest, internal: REFUSE, 2 findings

Two records resolve to UNKNOWN, one from an empty license field (rec-2003) and one from the explicit token (rec-2005). Even for the most permissive purpose, the gate refuses.

python -m corpuscustody gate samples/unknown.manifest --purpose internal

purpose: internal
records: 5
decision: REFUSE
findings: 2
  rec-2003 | UNKNOWN | unknown: provenance unknown, cannot clear
  rec-2005 | UNKNOWN | unknown: provenance unknown, cannot clear
notes: 7
  rec-2001 | MIT | attribution: attribution must be preserved
  rec-2002 | CC-BY-4.0 | attribution: attribution must be preserved
  rec-2003 | UNKNOWN | attribution: attribution must be preserved
  rec-2003 | UNKNOWN | no_derivatives: no-derivatives license, transformation blocked
  rec-2004 | Apache-2.0 | attribution: attribution must be preserved
  rec-2005 | UNKNOWN | attribution: attribution must be preserved
  rec-2005 | UNKNOWN | no_derivatives: no-derivatives license, transformation blocked

The UNKNOWN records also emit no_derivatives notes for internal, because the sentinel carries every obligation and no_derivatives is a note under internal.

Manifest format field by field

A manifest is a UTF-8 text file, one record per line. Blank lines and lines whose first non-space character is # are ignored, so manifests carry comments and stay diffable. Each record line has exactly three pipe-separated fields.

record_id | spdx_id | source

Field	Position	Required	Meaning
record_id	1	yes	free text identifier for the record; an empty value is a parse error
spdx_id	2	no	SPDX identifier; empty or unrecognised resolves to UNKNOWN
source	3	no	free text provenance note; not validated

Surrounding whitespace on each field is stripped. Parsing is deliberately dumb: spdx_id is read but not validated at parse time, so parsing and license policy stay in separate modules. Validation happens later, at resolve time. A line with anything other than three fields raises a ManifestError and the process exits 2. A real sample manifest, comment lines and all:

# permissive.manifest
rec-0001 | MIT | github.com/example/tokenizer
rec-0002 | Apache-2.0 | github.com/example/corpus-tools
rec-0003 | CC0-1.0 | zenodo.org/record/000001
rec-0004 | BSD-3-Clause | github.com/example/textnorm
rec-0005 | CC-BY-4.0 | data.example.org/news-2020

Output format and the cleared manifest

The resolve view lists every record with its resolved license and its active obligations (or none), then a license count summary sorted by SPDX identifier.

python -m corpuscustody resolve samples/sharealike.manifest

records: 5
  rec-1001 | MIT | attribution
  rec-1002 | CC-BY-SA-4.0 | attribution,share_alike
  rec-1003 | Apache-2.0 | attribution
  rec-1004 | CC-BY-NC-4.0 | attribution,non_commercial
  rec-1005 | GPL-3.0-only | attribution,share_alike
license counts:
  Apache-2.0: 1
  CC-BY-NC-4.0: 1
  CC-BY-SA-4.0: 1
  GPL-3.0-only: 1
  MIT: 1

The gate view is fixed and diffable: a purpose line, a records count, a decision of PASS or REFUSE, a findings count with one line per finding, then a notes count with one line per note. Each issue line is record_id | spdx_id | obligation: message.

On PASS, passing --out PATH writes a cleared manifest. It is written with LF line endings regardless of platform, so identical input produces byte-identical output. The file lists every checked record, its resolved identifier, and the purpose it was cleared for, under a three-line comment header.

python -m corpuscustody gate samples/permissive.manifest --purpose commercial --out cleared.manifest

# corpuscustody cleared manifest
# purpose: commercial
# record_id | spdx_id | cleared_for
rec-0001 | MIT | commercial
rec-0002 | Apache-2.0 | commercial
rec-0003 | CC0-1.0 | commercial
rec-0004 | BSD-3-Clause | commercial
rec-0005 | CC-BY-4.0 | commercial

No cleared manifest is written on REFUSE, even if --out is given.

Exit codes

The exit code is the machine-readable result. Use it in CI to fail a pipeline when the gate refuses.

Code	Meaning
0	clean: gate passed, or an informational command succeeded
1	gate refused: at least one blocking finding
2	usage error or bad input: missing manifest, bad line, bad purpose

These were confirmed in this session: help with no subcommand exits 2, a passing gate exits 0, a refusing gate exits 1, and resolve on a missing file exits 2 with error: manifest not found: on stderr. In a CI step, a nonzero exit from gate should fail the job. To diff two runs in git, commit the resolve or cleared-manifest output and let the deterministic ordering make the diff meaningful.

Reading the obligation matrix asset

The matrix below is the license-by-obligation grid. A filled cell means the license carries that obligation. The n column is the record count per license summed across all three sample manifests.

The n counts come from running resolve over the three sample manifests and summing: MIT 3, Apache-2.0 3, CC-BY-4.0 2, UNKNOWN 2, and BSD-3-Clause, CC0-1.0, CC-BY-SA-4.0, GPL-3.0-only, CC-BY-NC-4.0 at 1 each, for 15 records. The UNKNOWN row is amber because it is the row a reader must not miss: every obligation cell is filled, so it blocks every purpose.

Limitations, expanded

This is a mechanical obligation check, not legal advice. It matches SPDX identifiers against a short hand-maintained table and applies fixed rules. It does not:

• read or interpret full license texts, only identifiers,
• cover every SPDX identifier, only the common ones in src/corpuscustody/spdx.py,
• model license version differences beyond the identifiers listed,
• handle dual licensing, exceptions, or per-file license expressions,
• decide whether attribution or share-alike terms are actually satisfied downstream, only that they attach,
• distinguish jurisdictions, or account for fair use, database rights, or contractual terms that sit on top of the license,
• replace review by a person qualified to make licensing decisions.

Treat a PASS as "no mechanical conflict was found in the table," not as a legal clearance. The obligation rules are deliberately conservative, so unknown provenance always blocks rather than defaulting to permissive. If the table does not contain an identifier, that identifier resolves to UNKNOWN and blocks; the correct fix is to add the identifier to spdx.py after a person has verified its obligations, not to loosen the default. Every result is input to a human decision, never a substitute for one.

Design decisions

Two choices shape everything else. Each is stated with the alternative rejected and why.

An offline table, not a license API. Obligations live in a small hand-maintained dict in spdx.py. The rejected alternative was to resolve identifiers over the network against a live service. That was rejected because a gate that reaches the network is non-deterministic (the same manifest could pass today and refuse next week), because a network dependency is a supply-chain and availability risk in the pipeline where you least want surprises, and because obligations for the common identifiers are stable and few, so a short audited table is more trustworthy than an opaque remote answer. The cost is real and acknowledged: the table is incomplete, and any identifier it lacks resolves to UNKNOWN and blocks. That cost is paid on purpose, and it fails safe.

Refuse on unknown, not warn. An unresolvable license makes the gate refuse for every purpose. The rejected alternative was to warn and let the set pass. That was rejected because a warning is a decision deferred to nobody: it scrolls past in a log, the pipeline goes green, and the unlabelled record ships. A refusal forces the decision to a person now, while it is cheap to fix. The conservatism is not configurable, because a flag to downgrade refusals to warnings would be the first thing switched on under deadline pressure, which is precisely when it is most dangerous.

Repository layout

Generated from the real tree, __pycache__ and build artifacts omitted.

corpuscustody/
  README.md                     this file
  CHANGELOG.md                  version history
  LICENSE                       MIT license text
  pyproject.toml                package metadata, entry point, build config
  .gitignore                    ignored paths (pycache, build, venv)
  docs/
    assets/
      logo.svg                  wordmark: three record blocks through a gate
      obligation-matrix.svg     license-by-obligation grid, counts from samples
  samples/
    README.md                   description of the three test-vector manifests
    permissive.manifest         five permissive records, passes every purpose
    sharealike.manifest         share-alike and non-commercial mix
    unknown.manifest            two records of unknown provenance
  src/
    corpuscustody/
      __init__.py               package marker, holds __version__
      __main__.py               entry point for python -m corpuscustody
      cli.py                    argument parsing, subcommand dispatch, exit codes
      manifest.py               line-oriented manifest parser, Record dataclass
      spdx.py                   offline SPDX table, obligations, UNKNOWN sentinel
      compat.py                 obligation-to-severity rules per purpose
      gate.py                   PASS/REFUSE decision, cleared manifest writer
      report.py                 deterministic line rendering of resolve and gate
  tests/
    test_corpuscustody.py       unittest suite, stdlib only

Glossary

Term	Meaning
Manifest	a text file listing dataset records, one per line, three pipe-separated fields
Record	one line of a manifest: a record_id, an spdx_id, and a source
SPDX identifier	a standard short name for a license, such as MIT or CC-BY-SA-4.0
Obligation	a mechanical yes/no fact a license imposes (attribution, share alike, etc.)
Purpose	the declared use of the combined set: internal, commercial, or redistribute
Finding	a blocking issue; any finding makes the gate refuse
Note	an informational issue that does not block
Resolve	mapping an spdx_id to a license and its obligations via the offline table
Gate	the PASS or REFUSE decision over a set for a purpose
Cleared manifest	the file written on PASS listing each record and the purpose cleared for
UNKNOWN	the sentinel for unresolvable provenance; carries every obligation, blocks all

Verification

Run the test suite from the project root:

set PYTHONPATH=src
python -m unittest discover -s tests -v

The suite has 29 tests, all passing in this session (Ran 29 tests in 0.009s, timing on an unspecified machine and indicative only). They cover: SPDX resolution including empty, explicit UNKNOWN, and unrecognised identifiers all mapping to the sentinel; manifest parsing including field-count errors, empty record ids, comment and blank handling, and preserved empty license fields; compatibility evaluation including permissive clearing for commercial, share-alike blocking commercial but clearing internal, and unknown blocking every purpose; the gate PASS/REFUSE decision and deterministic cleared-manifest lines; report rendering and license counts; and the CLI end to end, including exit codes 0, 1, and 2 and writing the cleared manifest on pass.

Roadmap

No dates. In rough priority order:

• widen the SPDX table with more common identifiers, each added only after its obligations are verified by a person,
• support license expressions such as MIT OR Apache-2.0 and GPL-3.0 WITH exceptions, resolving to the least restrictive satisfiable combination,
• optionally read SPDX-License-Identifier tags from a directory tree so a manifest can be generated rather than hand-written,
• a machine-readable output mode (JSON) alongside the current line format for tools that would rather parse than diff.

License

MIT. See LICENSE.