Skip to content

Append only what an incremental update changes, and keep the permanent identifier - #35

Open
CastleSoft wants to merge 5 commits into
MPCoreDeveloper:mainfrom
CastleSoft:incremental-fixes
Open

CastleSoft wants to merge 5 commits into
MPCoreDeveloper:mainfrom
CastleSoft:incremental-fixes

Conversation

@CastleSoft

Copy link
Copy Markdown

What this changes

PdfEditOutputOptions.UseIncrementalUpdate now appends what the edit changed, under the numbers those objects
already had, instead of a renumbered copy of the whole document. A signature over the source stays a signature
over the revision it was made over in the eyes of validators that look at what changed after signing, and the
trailer keeps the permanent half of /ID.

Stacked on #34 (on #33, on #32). Until those merge, the diff here includes them; review 74a4d73 alone.

Why, and what is wrong today

  • The whole document was appended again. The incremental path ran the full rewrite into an appended
    writer, so the catalog, the page tree, every page and every copied object — the signature dictionary of a
    signed source included — got new numbers in the revision. Every earlier byte was kept, so the library's own
    validator still called the signature valid, but the file grew by its own size per edit, and pyHanko, whose
    difference analysis is what Acrobat-style validators do, saw the signature object as rewritten after
    signing: a signature followed by an edit came back CONTIGUOUS_BLOCK_FROM_START, not ENTIRE_REVISION.
  • The permanent identifier changed. Both elements of /ID were recomputed. ISO 32000-1 §14.4 says the first
    "shall not change when the file is incrementally updated".

How it works now, in an appended revision only (a complete rewrite is untouched):

  1. the copier of the edited document keeps each object's number (PdfObjectCopier.KeepObjectNumbers); copiers
    of documents the edit imports from still allocate new numbers, so they cannot collide;
  2. the catalog, the page-tree root, each page (its first occurrence; a duplicate is new) and the information
    dictionary are written under the numbers they already have;
  3. PdfWriter.SkipUnchanged leaves out an object whose serialised form equals what the source states under
    that number — which is what keeps an untouched signature dictionary out of the revision;
  4. the trailer writes [source /ID[0], new identifier].

Measured on the validation corpus: rotating one page of a signed document appended 20,969 bytes and twelve
objects
(a second /Type /Sig among them) before, and 716 bytes and three objects (the page, the page
tree, the information dictionary) after.

The test that fails without this change

Run with the source changes reverted: these three fail.

Plus Appends_the_objects_an_edit_creates (an inserted page is written under a new number) and
Writes_each_object_once_when_the_source_refers_to_its_own_catalog, which pin the edges of the new path. The
existing PdfIncrementalWriterTests — prefix, chaining, offsets, a signature that covers the earlier revision,
refusals — pass unchanged.

Checklist

  • dotnet build .\SharpCorePDF.slnx -c Release is clean on both legs, with zero warnings
  • A test fails when this change is reverted, and it is in this pull request
  • Public API surface unchanged (the new members are internal)
  • CHANGELOG.md has entries under [Unreleased] → Fixed
  • Documentation updated: manual chapter 14 (it described every edit as a complete rewrite), docs/validation.md; the --known-defect flags id-first, appends-only and signed-then-edited.pdf:FirstSignature are removed from validation.yml
  • No internal story, epic or requirement identifier anywhere
  • Not drive-by

Anything else a reviewer should know

  • Verified locally (Windows, .NET 11): 1,321 tests pass on net10.0 and on net11.0, none skipped — the
    last known defect pinned in the suite is closed here. On a freshly written validation corpus, with no flags:
    validate-incremental.py holds every invariant (each revision opens in qpdf, prefix, /ID[0],
    appends-only), and pyHanko reports ENTIRE_REVISION for the signature followed by an edit (modification
    level OTHER, which is right: a page was rotated). All 22 examples run, qpdf passes every output (bar the two
    pages without /MediaBox, a separate defect still flagged), and PDFium and MuPDF render the incremental
    documents identically.
  • Mutation testing (Stryker 5.1.0: PdfWriter, PdfObjectCopier, PdfEditEngine, PdfEditEngine.Incremental):
    81.8 % on the changed lines (36 of 44). The survivors were read: four checked casts on numbers already below
    int.MaxValue; > 0 versus >= 0 on a root number that is never 0 in this mode; the boundary at the
    highest source number (at worst one object written that need not be); and the two Redirect calls for the
    catalog and the information dictionary, which are defence in depth — the copier's copy of either is by
    construction identical to the source, so the unchanged check already leaves it out.

…ng and corpora

A writer that only reads its own output back cannot see the mistakes it shares with itself.
validation.yml now puts the example output and a validation corpus (written by
tools/SharpCorePDF.Validation, with a manifest of the text, passwords, permissions, revisions and
signature coverage each document must have) through qpdf, pypdf, pdfminer.six, MuPDF, pdf.js,
fontTools, OTS, PDFium and pyHanko, checks the incremental-update invariants, and replays the fuzz
seeds and four hostile documents. robustness.yml runs libFuzzer over the parser and the DCT, CCITT,
LZW, Flate and RunLength decoders, reads the veraPDF and pdf.js corpora one process per file, and
judges the PDF/A writer with veraPDF over the veraPDF corpus. compliance.yml cross-validates PDF/A-4.

The checks found defects that are not fixed here. Each is pinned by a skipped test
(PdfSignatureAttackTests, PdfIncrementalInvariantTests) or a strict --known-defect flag that fails
the run once the defect is fixed, and docs/validation.md lists them with their evidence.

No shipped package changes. The constitution's statement that no Python validator runs in CI is
corrected as a proposed PATCH amendment (2.0.1) for the maintainer to ratify. Local AI assistant
state (.claude/, CLAUDE.local.md) is ignored.
pdf.js 6 has no PDFDocumentProxy.destroy(), so the encryption check's pdf.js leg reported every document as unopenable: the call threw inside the try that was meant to catch a wrong password. The loading task owns the document and destroying it releases both. The fix was made before the previous commit but an editor saved a stale copy of the file over it.
Signature wrapping: the validator required only that the byte range start at 0 and end at the end of
the file, so bytes written into the unsigned gap could close the signature value early and add
entries while the digest still matched, and the signature was reported valid and covering the whole
document. The gap must now be exactly one hexadecimal string equal to the dictionary's /Contents;
anything else is MalformedByteRange, integrity Unknown and CoversWholeDocument false.

Decompression bombs: Flate, LZW and RunLength decoded without a limit, so a 1.1 MB document took the
process past 1 GiB. PdfEngineOptions.MaxDecodedStreamBytes (128 MiB) now stops them; decode buffers
above 16 MiB no longer stay in the shared array pool, and buffer growth no longer overflows.

Stack overflow: a chain of streams whose /Length is the next stream was resolved by recursion, and
60,000 links ended the process. The chain is followed no deeper than MaxNestingDepth, then the
stream is found by its endstream keyword, as for a /Length cycle.

Allocations sized by the file: an object stream's /N is bounded by what the stream can hold, and a
JPEG whose output is above the caller's budget is measured from its frame header instead of being
decoded first; the working sizes are computed in 64 bits.

Tests that fail without the fixes: PdfSignatureAttackTests (gap wrapping, gap shape),
PdfFilterTests (limits per filter, content stream past the limit), PdfReaderRecoveryTests (length
chain, indirect lengths, object stream /N), PdfJpegFilterTests (oversized frames, working
buffers). The three hostile documents of validation.yml are read under a 1 GiB heap, and their
--known flags are removed.
…fused

Four defects turned documents into exceptions that were not PdfExceptions, so a caller that handles
the contract's refusal still saw the read fail:

- A JPEG scan selected Huffman tables with 4-bit selectors and a progressive scan named its spectral
  selection, and both were used as indexes unchecked (IndexOutOfRangeException). ParseScan now refuses
  a selector above 3, and a progressive scan unless Ss <= Se <= 63 and Ah, Al <= 13.
- The empty name, a solidus on its own, is a name PDF allows, and four documents of the pdf.js test
  corpus use it as a key; PdfDictionary.Set refused it, so they did not open at all.
- A glyph drawn upside down (a negative font size) gave PdfGlyph.BoundingBox a bottom above its top,
  and the rectangle refused it, failing text extraction for the page.
- PdfDate.TryParse threw for a time-zone offset beyond fourteen hours, so one malformed /CreationDate
  failed the inspection of the whole document; such an offset is now read as no offset.

PdfContainerTests.Refuses_an_entry_without_a_key asserted the second defect and is rewritten: a null
key is still refused, the empty name is kept. Tests that fail without the fixes:
PdfJpegFilterTests (table selectors, progressive scan bounds), PdfObjectModelTests (empty-name key,
in a dictionary and in a document), PdfTextScannerTests (upside-down glyph), PdfSerializationTests
(offsets past fourteen hours). The pdf.js corpus and the hostile documents now read with no fault,
crash or timeout, and their --known flags are removed.
…t identifier

UseIncrementalUpdate ran the full rewrite into an appended writer, so every object of the document,
the signature dictionary of a signed source included, was written again under a new number. Every
earlier byte was kept, but the file grew by its own size per edit, and validators that look at what
changed after a signature (pyHanko, Acrobat) saw the signed field as rewritten: pyHanko placed a
signature followed by an edit as CONTIGUOUS_BLOCK_FROM_START instead of ENTIRE_REVISION.

In an appended revision the edited document's copier now keeps the numbers of its objects, the
catalog, the page tree, the pages and the information dictionary are written under the numbers they
already have, and the writer leaves out an object that would be written exactly as the source states
it. Rotating a page of a signed document now appends 716 bytes and three objects instead of 20,969
bytes and twelve; the signature dictionary is not in the revision.

The trailer also kept neither half of the source's /ID. ISO 32000-1 14.4 makes the first element
permanent across incremental updates; it is now the source's, and only the second element is new.

Tests that fail without the change: PdfIncrementalInvariantTests (the identifier, an update that
appends only what it changed, a signature dictionary left out of the revision); the corpus checks
validate-incremental.py and validate-signatures.py now pass with no --known-defect flag. The manual's
editing chapter described every edit as a complete rewrite and now says what the option does.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant