Repository navigation
Append only what an incremental update changes, and keep the permanent identifier - #35
Open
CastleSoft wants to merge 5 commits into
Open
CastleSoft wants to merge 5 commits into
CastleSoft wants to merge 5 commits into
Conversation
…ng and corpora A writer that only reads its own output back cannot see the mistakes it shares with itself. validation.yml now puts the example output and a validation corpus (written by tools/SharpCorePDF.Validation, with a manifest of the text, passwords, permissions, revisions and signature coverage each document must have) through qpdf, pypdf, pdfminer.six, MuPDF, pdf.js, fontTools, OTS, PDFium and pyHanko, checks the incremental-update invariants, and replays the fuzz seeds and four hostile documents. robustness.yml runs libFuzzer over the parser and the DCT, CCITT, LZW, Flate and RunLength decoders, reads the veraPDF and pdf.js corpora one process per file, and judges the PDF/A writer with veraPDF over the veraPDF corpus. compliance.yml cross-validates PDF/A-4. The checks found defects that are not fixed here. Each is pinned by a skipped test (PdfSignatureAttackTests, PdfIncrementalInvariantTests) or a strict --known-defect flag that fails the run once the defect is fixed, and docs/validation.md lists them with their evidence. No shipped package changes. The constitution's statement that no Python validator runs in CI is corrected as a proposed PATCH amendment (2.0.1) for the maintainer to ratify. Local AI assistant state (.claude/, CLAUDE.local.md) is ignored.
pdf.js 6 has no PDFDocumentProxy.destroy(), so the encryption check's pdf.js leg reported every document as unopenable: the call threw inside the try that was meant to catch a wrong password. The loading task owns the document and destroying it releases both. The fix was made before the previous commit but an editor saved a stale copy of the file over it.
Signature wrapping: the validator required only that the byte range start at 0 and end at the end of the file, so bytes written into the unsigned gap could close the signature value early and add entries while the digest still matched, and the signature was reported valid and covering the whole document. The gap must now be exactly one hexadecimal string equal to the dictionary's /Contents; anything else is MalformedByteRange, integrity Unknown and CoversWholeDocument false. Decompression bombs: Flate, LZW and RunLength decoded without a limit, so a 1.1 MB document took the process past 1 GiB. PdfEngineOptions.MaxDecodedStreamBytes (128 MiB) now stops them; decode buffers above 16 MiB no longer stay in the shared array pool, and buffer growth no longer overflows. Stack overflow: a chain of streams whose /Length is the next stream was resolved by recursion, and 60,000 links ended the process. The chain is followed no deeper than MaxNestingDepth, then the stream is found by its endstream keyword, as for a /Length cycle. Allocations sized by the file: an object stream's /N is bounded by what the stream can hold, and a JPEG whose output is above the caller's budget is measured from its frame header instead of being decoded first; the working sizes are computed in 64 bits. Tests that fail without the fixes: PdfSignatureAttackTests (gap wrapping, gap shape), PdfFilterTests (limits per filter, content stream past the limit), PdfReaderRecoveryTests (length chain, indirect lengths, object stream /N), PdfJpegFilterTests (oversized frames, working buffers). The three hostile documents of validation.yml are read under a 1 GiB heap, and their --known flags are removed.
…fused Four defects turned documents into exceptions that were not PdfExceptions, so a caller that handles the contract's refusal still saw the read fail: - A JPEG scan selected Huffman tables with 4-bit selectors and a progressive scan named its spectral selection, and both were used as indexes unchecked (IndexOutOfRangeException). ParseScan now refuses a selector above 3, and a progressive scan unless Ss <= Se <= 63 and Ah, Al <= 13. - The empty name, a solidus on its own, is a name PDF allows, and four documents of the pdf.js test corpus use it as a key; PdfDictionary.Set refused it, so they did not open at all. - A glyph drawn upside down (a negative font size) gave PdfGlyph.BoundingBox a bottom above its top, and the rectangle refused it, failing text extraction for the page. - PdfDate.TryParse threw for a time-zone offset beyond fourteen hours, so one malformed /CreationDate failed the inspection of the whole document; such an offset is now read as no offset. PdfContainerTests.Refuses_an_entry_without_a_key asserted the second defect and is rewritten: a null key is still refused, the empty name is kept. Tests that fail without the fixes: PdfJpegFilterTests (table selectors, progressive scan bounds), PdfObjectModelTests (empty-name key, in a dictionary and in a document), PdfTextScannerTests (upside-down glyph), PdfSerializationTests (offsets past fourteen hours). The pdf.js corpus and the hostile documents now read with no fault, crash or timeout, and their --known flags are removed.
…t identifier UseIncrementalUpdate ran the full rewrite into an appended writer, so every object of the document, the signature dictionary of a signed source included, was written again under a new number. Every earlier byte was kept, but the file grew by its own size per edit, and validators that look at what changed after a signature (pyHanko, Acrobat) saw the signed field as rewritten: pyHanko placed a signature followed by an edit as CONTIGUOUS_BLOCK_FROM_START instead of ENTIRE_REVISION. In an appended revision the edited document's copier now keeps the numbers of its objects, the catalog, the page tree, the pages and the information dictionary are written under the numbers they already have, and the writer leaves out an object that would be written exactly as the source states it. Rotating a page of a signed document now appends 716 bytes and three objects instead of 20,969 bytes and twelve; the signature dictionary is not in the revision. The trailer also kept neither half of the source's /ID. ISO 32000-1 14.4 makes the first element permanent across incremental updates; it is now the source's, and only the second element is new. Tests that fail without the change: PdfIncrementalInvariantTests (the identifier, an update that appends only what it changed, a signature dictionary left out of the revision); the corpus checks validate-incremental.py and validate-signatures.py now pass with no --known-defect flag. The manual's editing chapter described every edit as a complete rewrite and now says what the option does.
This was referenced Nov 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
PdfEditOutputOptions.UseIncrementalUpdatenow appends what the edit changed, under the numbers those objectsalready had, instead of a renumbered copy of the whole document. A signature over the source stays a signature
over the revision it was made over in the eyes of validators that look at what changed after signing, and the
trailer keeps the permanent half of
/ID.Stacked on #34 (on #33, on #32). Until those merge, the diff here includes them; review
74a4d73alone.Why, and what is wrong today
writer, so the catalog, the page tree, every page and every copied object — the signature dictionary of a
signed source included — got new numbers in the revision. Every earlier byte was kept, so the library's own
validator still called the signature valid, but the file grew by its own size per edit, and pyHanko, whose
difference analysis is what Acrobat-style validators do, saw the signature object as rewritten after
signing: a signature followed by an edit came back
CONTIGUOUS_BLOCK_FROM_START, notENTIRE_REVISION./IDwere recomputed. ISO 32000-1 §14.4 says the first"shall not change when the file is incrementally updated".
How it works now, in an appended revision only (a complete rewrite is untouched):
PdfObjectCopier.KeepObjectNumbers); copiersof documents the edit imports from still allocate new numbers, so they cannot collide;
dictionary are written under the numbers they already have;
PdfWriter.SkipUnchangedleaves out an object whose serialised form equals what the source states underthat number — which is what keeps an untouched signature dictionary out of the revision;
[source /ID[0], new identifier].Measured on the validation corpus: rotating one page of a signed document appended 20,969 bytes and twelve
objects (a second
/Type /Sigamong them) before, and 716 bytes and three objects (the page, the pagetree, the information dictionary) after.
The test that fails without this change
Run with the source changes reverted: these three fail.
PdfIncrementalInvariantTests.Keeps_the_first_identifier_element_across_an_incremental_update(was skipped as a known defect in Validate the output with independent readers and the input with fuzzing and corpora #32)PdfIncrementalInvariantTests.Appends_only_the_objects_an_edit_changesPdfIncrementalInvariantTests.Leaves_the_signature_dictionary_out_of_an_appended_revisionPlus
Appends_the_objects_an_edit_creates(an inserted page is written under a new number) andWrites_each_object_once_when_the_source_refers_to_its_own_catalog, which pin the edges of the new path. Theexisting
PdfIncrementalWriterTests— prefix, chaining, offsets, a signature that covers the earlier revision,refusals — pass unchanged.
Checklist
dotnet build .\SharpCorePDF.slnx -c Releaseis clean on both legs, with zero warningsCHANGELOG.mdhas entries under[Unreleased]→ Fixeddocs/validation.md; the--known-defectflagsid-first,appends-onlyandsigned-then-edited.pdf:FirstSignatureare removed fromvalidation.ymlAnything else a reviewer should know
last known defect pinned in the suite is closed here. On a freshly written validation corpus, with no flags:
validate-incremental.pyholds every invariant (each revision opens in qpdf, prefix,/ID[0],appends-only), and pyHanko reports
ENTIRE_REVISIONfor the signature followed by an edit (modificationlevel
OTHER, which is right: a page was rotated). All 22 examples run, qpdf passes every output (bar the twopages without
/MediaBox, a separate defect still flagged), and PDFium and MuPDF render the incrementaldocuments identically.
PdfWriter,PdfObjectCopier,PdfEditEngine,PdfEditEngine.Incremental):81.8 % on the changed lines (36 of 44). The survivors were read: four
checkedcasts on numbers already belowint.MaxValue;> 0versus>= 0on a root number that is never 0 in this mode; the boundary at thehighest source number (at worst one object written that need not be); and the two
Redirectcalls for thecatalog and the information dictionary, which are defence in depth — the copier's copy of either is by
construction identical to the source, so the unchanged check already leaves it out.