Skip to content

Validate the output with independent readers and the input with fuzzing and corpora - #32

Open
CastleSoft wants to merge 2 commits into
MPCoreDeveloper:mainfrom
CastleSoft:validation-checks
Open

CastleSoft wants to merge 2 commits into
MPCoreDeveloper:mainfrom
CastleSoft:validation-checks

Conversation

@CastleSoft

Copy link
Copy Markdown

What this changes

Adds twelve independent validation checks, so readers that share no code with this library judge what it
writes, and fuzzers and public corpora test what it reads. validation.yml (every push and pull request)
puts the example output and a new validation corpus through qpdf, pypdf, pdfminer.six, MuPDF, pdf.js,
fontTools, OTS, PDFium and pyHanko, checks the incremental-update invariants, and replays the fuzz seeds and
four hostile documents. robustness.yml (weekly) runs libFuzzer over the parser and five decoders, reads
the veraPDF and pdf.js corpora one process per file, and judges the PDF/A writer with veraPDF.
compliance.yml now cross-validates PDF/A-4. No shipped package changes. docs/validation.md is the record.

Why, and what is wrong today

A writer that only reads its own output back cannot see the mistakes it shares with itself. The new checks
found defects that this pull request records but does not fix. Fixes follow in separate pull requests:

  • signature validator accepts a /ByteRange gap that holds more than /Contents (signature wrapping)
  • no decoded-size limit on Flate (1.1 MB → >1 GiB), unbounded /Length recursion (uncatchable stack
    overflow), allocations sized by object-stream /N, unchecked JPEG Huffman selectors
  • incremental updates rewrite /ID[0] and append a renumbered copy of the whole document, which also
    rewrites the signature dictionary after signing
  • merges copy a page without /MediaBox; an empty-name dictionary key, an upside-down glyph box and an
    out-of-range date offset each fault real pdf.js-corpus files

The test that fails without this change

Each defect is pinned so CI stays green while it stands and goes red once it is fixed, until its pin is removed:

  • PdfSignatureAttackTests.Does_not_report_a_gap_that_holds_more_than_the_signature_value_as_covering_the_document
    and PdfIncrementalInvariantTests.Keeps_the_first_identifier_element_across_an_incremental_update.
    Both are skipped with the defect named; with the skip removed, each fails (checked locally).
  • Strict --known-defect / --known flags in validation.yml and robustness.yml; a listed defect that
    stops reproducing fails the run.
  • New passing tests: the forged-byte-range theory (6 cases), the emptied-value case, the
    revision-readability and /Size invariants, and the writer-side gap check.

Checklist

  • dotnet build .\SharpCorePDF.slnx -c Release is clean on both legs, with zero warnings
  • A test fails when this change is reverted, and it is in this pull request (the two skipped tests above, verified by un-skipping)
  • PublicAPI.Shipped.txt / PublicAPI.Unshipped.txt: the surface is unchanged
  • CHANGELOG.md has an entry under [Unreleased]
  • Documentation updated: README (CI table, validation section), CONTRIBUTING, docs/README.md, new docs/validation.md
  • No internal story, epic or requirement identifier anywhere
  • Not drive-by: only the files the checks need

Anything else a reviewer should know

  • Constitution: the sentence "Neither Python tool runs in CI" became false, so this proposes PATCH
    amendment 2.0.1 (fact only, no rule changes). It needs your ratification; drop that file if you
    would rather amend separately.
  • Security disclosure: this publishes details and reproducers of the unfixed issues above, by the
    contributor's choice, on the basis that the fixes follow immediately.
  • Report-only until a first run: the Arlington job (continue-on-error) and the judged PDF/A
    pre-flight. They could not run locally (no Docker daemon, no Java); each workflow says what to switch
    on once the baseline exists.
  • Verified locally (Windows, .NET 11): 1,277 tests pass and 2 are skipped on net10.0 and on net11.0;
    every check in validation.yml passes with its known-defect flags; the veraPDF corpus (1,471 files)
    reads cleanly; the pdf.js corpus (987 files) shows exactly the six listed faults.
  • Review tag: review/validation-checks-2026-11-16 on the fork marks this commit for sign-off. It
    is not a release tag and does not match v*.

…ng and corpora

A writer that only reads its own output back cannot see the mistakes it shares with itself.
validation.yml now puts the example output and a validation corpus (written by
tools/SharpCorePDF.Validation, with a manifest of the text, passwords, permissions, revisions and
signature coverage each document must have) through qpdf, pypdf, pdfminer.six, MuPDF, pdf.js,
fontTools, OTS, PDFium and pyHanko, checks the incremental-update invariants, and replays the fuzz
seeds and four hostile documents. robustness.yml runs libFuzzer over the parser and the DCT, CCITT,
LZW, Flate and RunLength decoders, reads the veraPDF and pdf.js corpora one process per file, and
judges the PDF/A writer with veraPDF over the veraPDF corpus. compliance.yml cross-validates PDF/A-4.

The checks found defects that are not fixed here. Each is pinned by a skipped test
(PdfSignatureAttackTests, PdfIncrementalInvariantTests) or a strict --known-defect flag that fails
the run once the defect is fixed, and docs/validation.md lists them with their evidence.

No shipped package changes. The constitution's statement that no Python validator runs in CI is
corrected as a proposed PATCH amendment (2.0.1) for the maintainer to ratify. Local AI assistant
state (.claude/, CLAUDE.local.md) is ignored.
pdf.js 6 has no PDFDocumentProxy.destroy(), so the encryption check's pdf.js leg reported every document as unopenable: the call threw inside the try that was meant to catch a wrong password. The loading task owns the document and destroying it releases both. The fix was made before the previous commit but an editor saved a stale copy of the file over it.
@CastleSoft

Copy link
Copy Markdown
Author

Maintainer notes for the six pull requests (#32 to #37), which are stacked and should merge in this order: #32, #33, #34, #35, #36, #37. Each contains the commits of the ones before it until they merge.

  1. Approve the workflow runs. These come from a first-time contributor, so GitHub holds the CI runs until a maintainer chooses "Approve and run" on each pull request.
  2. Constitution amendment 2.0.1. One of the pull requests amends .specify/memory/constitution.md. Please ratify it, or drop that file from the pull request if you would rather not.
  3. Version. Harden hybrid-reference reading, bound form painting, and make the PDF/A conversion read what the profiles name #37 adds public API (MaxDecodedStreamBytes, MaxFormPaintsPerPage, and two PdfWarningKind members), so the next release is 1.9.0, not a 1.8.x patch. docs/release-checklist.md has the details.
  4. First Robustness run. After Harden hybrid-reference reading, bound form painting, and make the PDF/A conversion read what the profiles name #37 merges, the first robustness.yml run sets the PDF/A false-success baseline; its --max-false-success figure should be set from that run.

Six items from the validation review are in progress across these; #37 carries the last of them, and docs/validation.md lists what is still open.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant