Repository navigation
Write missing media boxes, keep copied reals, harden recovery and the remaining edges - #36
Open
CastleSoft wants to merge 11 commits into
Open
CastleSoft wants to merge 11 commits into
CastleSoft wants to merge 11 commits into
Conversation
…ng and corpora A writer that only reads its own output back cannot see the mistakes it shares with itself. validation.yml now puts the example output and a validation corpus (written by tools/SharpCorePDF.Validation, with a manifest of the text, passwords, permissions, revisions and signature coverage each document must have) through qpdf, pypdf, pdfminer.six, MuPDF, pdf.js, fontTools, OTS, PDFium and pyHanko, checks the incremental-update invariants, and replays the fuzz seeds and four hostile documents. robustness.yml runs libFuzzer over the parser and the DCT, CCITT, LZW, Flate and RunLength decoders, reads the veraPDF and pdf.js corpora one process per file, and judges the PDF/A writer with veraPDF over the veraPDF corpus. compliance.yml cross-validates PDF/A-4. The checks found defects that are not fixed here. Each is pinned by a skipped test (PdfSignatureAttackTests, PdfIncrementalInvariantTests) or a strict --known-defect flag that fails the run once the defect is fixed, and docs/validation.md lists them with their evidence. No shipped package changes. The constitution's statement that no Python validator runs in CI is corrected as a proposed PATCH amendment (2.0.1) for the maintainer to ratify. Local AI assistant state (.claude/, CLAUDE.local.md) is ignored.
pdf.js 6 has no PDFDocumentProxy.destroy(), so the encryption check's pdf.js leg reported every document as unopenable: the call threw inside the try that was meant to catch a wrong password. The loading task owns the document and destroying it releases both. The fix was made before the previous commit but an editor saved a stale copy of the file over it.
Signature wrapping: the validator required only that the byte range start at 0 and end at the end of the file, so bytes written into the unsigned gap could close the signature value early and add entries while the digest still matched, and the signature was reported valid and covering the whole document. The gap must now be exactly one hexadecimal string equal to the dictionary's /Contents; anything else is MalformedByteRange, integrity Unknown and CoversWholeDocument false. Decompression bombs: Flate, LZW and RunLength decoded without a limit, so a 1.1 MB document took the process past 1 GiB. PdfEngineOptions.MaxDecodedStreamBytes (128 MiB) now stops them; decode buffers above 16 MiB no longer stay in the shared array pool, and buffer growth no longer overflows. Stack overflow: a chain of streams whose /Length is the next stream was resolved by recursion, and 60,000 links ended the process. The chain is followed no deeper than MaxNestingDepth, then the stream is found by its endstream keyword, as for a /Length cycle. Allocations sized by the file: an object stream's /N is bounded by what the stream can hold, and a JPEG whose output is above the caller's budget is measured from its frame header instead of being decoded first; the working sizes are computed in 64 bits. Tests that fail without the fixes: PdfSignatureAttackTests (gap wrapping, gap shape), PdfFilterTests (limits per filter, content stream past the limit), PdfReaderRecoveryTests (length chain, indirect lengths, object stream /N), PdfJpegFilterTests (oversized frames, working buffers). The three hostile documents of validation.yml are read under a 1 GiB heap, and their --known flags are removed.
…fused Four defects turned documents into exceptions that were not PdfExceptions, so a caller that handles the contract's refusal still saw the read fail: - A JPEG scan selected Huffman tables with 4-bit selectors and a progressive scan named its spectral selection, and both were used as indexes unchecked (IndexOutOfRangeException). ParseScan now refuses a selector above 3, and a progressive scan unless Ss <= Se <= 63 and Ah, Al <= 13. - The empty name, a solidus on its own, is a name PDF allows, and four documents of the pdf.js test corpus use it as a key; PdfDictionary.Set refused it, so they did not open at all. - A glyph drawn upside down (a negative font size) gave PdfGlyph.BoundingBox a bottom above its top, and the rectangle refused it, failing text extraction for the page. - PdfDate.TryParse threw for a time-zone offset beyond fourteen hours, so one malformed /CreationDate failed the inspection of the whole document; such an offset is now read as no offset. PdfContainerTests.Refuses_an_entry_without_a_key asserted the second defect and is rewritten: a null key is still refused, the empty name is kept. Tests that fail without the fixes: PdfJpegFilterTests (table selectors, progressive scan bounds), PdfObjectModelTests (empty-name key, in a dictionary and in a document), PdfTextScannerTests (upside-down glyph), PdfSerializationTests (offsets past fourteen hours). The pdf.js corpus and the hostile documents now read with no fault, crash or timeout, and their --known flags are removed.
…t identifier UseIncrementalUpdate ran the full rewrite into an appended writer, so every object of the document, the signature dictionary of a signed source included, was written again under a new number. Every earlier byte was kept, but the file grew by its own size per edit, and validators that look at what changed after a signature (pyHanko, Acrobat) saw the signed field as rewritten: pyHanko placed a signature followed by an edit as CONTIGUOUS_BLOCK_FROM_START instead of ENTIRE_REVISION. In an appended revision the edited document's copier now keeps the numbers of its objects, the catalog, the page tree, the pages and the information dictionary are written under the numbers they already have, and the writer leaves out an object that would be written exactly as the source states it. Rotating a page of a signed document now appends 716 bytes and three objects instead of 20,969 bytes and twelve; the signature dictionary is not in the revision. The trailer also kept neither half of the source's /ID. ISO 32000-1 14.4 makes the first element permanent across incremental updates; it is now the source's, and only the second element is new. Tests that fail without the change: PdfIncrementalInvariantTests (the identifier, an update that appends only what it changed, a signature dictionary left out of the revision); the corpus checks validate-incremental.py and validate-signatures.py now pass with no --known-defect flag. The manual's editing chapter described every edit as a complete rewrite and now says what the option does.
…CI runs found - A merge copied a page that states no /MediaBox, and inherits none, without one, so the result broke the rule that every page has a media box and each reader assumed its own size. The page is written with the A4 box the inspection already describes it with, and PageSizeAssumed says so. - A merge or an edit rewrote every real it copied to five decimals: a Type 3 font matrix of 0.000012 came out as 0.00001. A real read from a document now keeps its text when that is a well-formed PDF real and is written back from it. A real too large for the formatter's 32-byte buffer failed the write of the whole document with InvalidOperationException; the first run of the document fuzz target found it, and such a value is now formatted on the allocating path. - The image copy route let InvalidDataException escape for damaged Flate data, with or without a predictor (the flate fuzz target's first find); it now refuses it with PdfSourceFormatException. - The Arlington service names its verdict element arlingtonReport, which validate-arlington.py now reads. Its first run found that a free text annotation lacked the /DA ISO 32000-1 requires and annotation appearances lacked /Resources; both are now written. Tests that fail without the changes: PdfMergeFidelityTests (media box, copied reals, any magnitude), PdfContentReaderTests.Refuses_to_copy_an_image_whose_flate_data_is_damaged, and PdfDocumentEditorTests.States_the_default_appearance_and_resources_of_a_free_text_annotation and States_resources_on_an_appearance_that_needs_none. qpdf now passes every example output with no --known-defect flag, so the qpdf step lists none.
…e edges Recovery - A cross-reference table that parses can still be wrong: bytes inserted or removed after it was written move every object. Reconstruction now also runs when the catalog's offset, or most of a sample of offsets, does not land on the n g obj header of its own object. - Reconstruction indexes the objects of every object stream the scan finds, and takes /Root from the dictionary of the last cross-reference stream, so a PDF 1.5 file whose cross-reference stream is lost reads instead of being refused with "no document catalog". The test that pinned the refusal now pins the rebuild. Hardening - AESV2 always uses a sixteen-byte key; a crypt filter that declares fewer made the file unopenable. - The whole-buffer predictor path sizes a row in 64 bits like the streaming decoder: a declared Columns x Colors that wrapped to zero looped forever under TIFF prediction, and to a negative number threw ArgumentOutOfRangeException. - A stream with a wrong /Length is cut where its zlib data is complete, by its Adler-32, instead of at the first endstream whose truncated bytes inflate; at most sixteen candidates are inflated. - A reference to a generation the table does not list resolves to null (ISO 32000-1, 7.3.10). - A revision appended to a file whose newest section is a cross-reference stream, by an edit or a signature, ends in a cross-reference stream instead of a classic table. qpdf and pyHanko accept an edit and two signatures appended this way. Corpus fault - The public-corpora job faulted on isartor-6-1-11-t01-fail-a.pdf: a file specification that only names an external file is listed as an attachment with ByteLength 0, and copying it threw ArgumentException. It now writes no bytes; an id the document does not have is still refused. Each change has a test that fails without it. Full suite passes on net11.0 and net10.0; the pdf.js corpus run reports no fault, crash or timeout.
The scoped run over the files of the last two commits scored 70.8% on their changed lines. The survivors that were missing assertions now have them: - PdfCrossReferenceStreamWriterTests compares every row of an appended cross-reference stream byte for byte, its /Index subsections, the width of each /W field at the byte boundaries, its /Size and the trailer entries it carries or drops. - The free text /DA test uses three different colour channels; the merge warning names the page as a person counts it; the damaged-Flate refusal says why. - The predictor limit is pinned at the largest row accepted and one column more, and a zero-bit row is refused. - The endstream scan leaves the parser behind the keyword it chose, and the sixteen-candidate cap is pinned at its boundary. - A signature on a classic-table document keeps a classic table, and only an edit's stream lists the head of the free list. PdfWriter's stream path no longer sets /Size before the stream writer, which derives it itself; the statement was the survivor nothing could kill. The remaining survivors are equivalent mutants (ConfigureAwait flips, bounds checks on offsets the scan already validated) or fallbacks for damaged files noted in the pull request.
…ection does The second robustness run, on 54c08d5, found two more faults now that the first ones are fixed: - flate: the image copy route caught the decompressor's InvalidDataException but not the ZLibException it throws for other damaged input (crash-0b3d19d6). Both are now refused with PdfSourceFormatException. ZLibException is not public, so it is recognised by name rather than by catching every IOException, which would also swallow the destination's own failures. - document: a page declaring /Rotate 91 made the edit engine throw ArgumentOutOfRangeException (crash-7201f51f). The edit engine read /Rotate as an integer and required a multiple of 90, while the inspection reads any number and rounds to the nearest quarter turn, so a real /Rotate was also read as none. The edit engine now uses the inspection's reading, and the quarter turns are reduced before they become an integer, so a declared value beyond int's range no longer wraps. Tests that fail without the changes: PdfContentReaderTests.Refuses_to_copy_an_image_whose_flate_data_is_damaged (the fuzz input's payload) and PdfMergeFidelityTests.Edits_a_page_whose_rotation_is_not_a_quarter_turn. Both crash inputs and the two from the first run replay with no fault. Scoped Stryker over the two changed files kills every mutant on the changed lines.
The third robustness run, on 7c97a38, found a stack overflow (crash-d34275a3): a merge walks the outline recursively, one frame per level, and a source nested about fifty thousand levels deep ended the process. The walk now stops at 256 levels and reports BookmarksTruncated, and a bookmark reached a second time - a /First or /Next pointing back up the tree, or a child two parents share - is copied once and reported with BookmarkSkipped, which also ends cycles and the repeated walks of shared subtrees. Tests: PdfMergeOutlineLimitsTests (depth limit, shared child, self cycle). The crash input replays with no fault; the full suite passes.
The robustness run on f529b02 found a crash in the document fuzz target (crash-e154541f). Reconstruction keeps the damaged table's entries beside the ones its scan finds, and the search for a cross-reference stream's trailer sliced the document at each of their offsets unchecked. An offset past the end of the file, or one beyond the range of an index that wraps negative when cast, threw ArgumentOutOfRangeException instead of letting the rebuild go on to find the catalog. Such an entry is now passed over. The catalog search needs no such check: PdfLexer clamps the position it starts at, which a scoped Stryker run confirmed by leaving the same guard there unkillable, so it was not added. Test that fails without the change: PdfReaderRecoveryTests.Rebuilds_a_table_that_states_an_offset_past_the_end_of_the_file, for an offset past the end and one that wraps negative. The crash input replays with no fault. On the changed lines the two surviving mutants are equivalent: offset 0 is the header and an offset equal to the length gives an empty window, so neither boundary changes what is read. The outline-depth fix of f529b02 is now in CHANGELOG.md and the fixed-defects table of docs/validation.md, which it had not been.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #35 (
incremental-fixes); the first commits of this branch are those of #32–#35. Review the last three commits, or merge #32–#35 first and this diff shrinks to them.Fixes the remaining open items of the review — B7 to B10 — and the faults the first runs of the new CI checks found on the fork.
5fbfc92 — Write a missing media box, keep copied reals, and fix what the first CI runs found
/MediaBox(and none inherited) without one; qpdf assumed Letter--known-defect, now removed)PageSizeAssumedsays soPdfMergeFidelityTests.Writes_a_media_box_for_a_page_that_has_none0.000012→0.00001); a real too large for the 32-byte buffer failed the whole writedocumentfuzz targetPdfMergeFidelityTests.Writes_the_reals_it_copies_as_the_source_states_them,…_of_any_magnitude_…InvalidDataExceptionescape for damaged Flate dataflatefuzz targetPdfSourceFormatExceptionPdfContentReaderTests.Refuses_to_copy_an_image_whose_flate_data_is_damaged/DA; appearance streams without/ResourcesPdfDocumentEditorTests.States_the_default_appearance_and_resources_of_a_free_text_annotation,States_resources_on_an_appearance_that_needs_nonevalidate-arlington.pylooked forvalidationReport; the service writesarlingtonReport3a8b578 — Rebuild moved tables, read object streams in recovery, and harden five edges
PdfReaderRecoveryTests.Rebuilds_a_table_whose_offsets_no_longer_point_at_their_objects/Roottaken from the xref stream's dictionaryRebuilds_a_table_from_object_streams_…,Rebuilds_a_document_whose_catalog_lives_in_an_object_stream(previously pinned the refusal)PdfMergeEngineTests.Opens_an_aesv2_document_whose_crypt_filter_declares_a_short_keyColumns × Colorswrapped to 0 (infinite loop) or negative (ArgumentOutOfRangeException)PdfPredictorTests.Refuses_a_predicted_row_larger_than_the_engine_decodesendstreamscan accepted the first candidate that inflated — including truncated zlib data — and inflated every candidate from the startPdfStreamParsingTests.Scans_past_a_candidate_whose_compressed_data_is_cut_short3 1 Rresolved to3 0 obj)PdfReaderRecoveryTests.Resolves_a_reference_to_a_generation_the_table_does_not_list_as_nullPdfIncrementalWriterTests.Appends_a_cross_reference_stream_…,PdfDocumentSignerTests.Signs_a_document_that_uses_cross_reference_streams_with_oneArgumentException(isartor-6-1-11-t01-fail-a.pdf)ByteLength0PdfContentReaderTests.Copies_nothing_for_a_listed_file_that_embeds_no_stream54c08d5 — Pin what the scoped Stryker run found unasserted
The scoped Stryker run (
.clinerules/tooling.md) over every production file of the two commits above scored70.8% on their changed lines. The survivors that were missing assertions now have them: a byte-exact test of
the cross-reference stream writer (rows,
/Index,/Wwidths at each byte boundary,/Size, trailer entries),the predictor limit at its boundary, the sixteen-candidate cap, the parser position after a scanned stream,
classic-vs-stream sections for signatures, and the wording of two messages. One redundant statement in
PdfWriterwas removed.Survivors left on purpose:
ConfigureAwaitflips, bounds checks on offsets the scan already validated,/ObjStmat index 0 of an object header;
/Root, and the"first candidate that inflates" fallback when no candidate's Adler-32 matches;
Verification
--known-defectflag.54c08d5(CI, CodeQL, compliance, independent validation, robustness): https://github.com/CastleSoft/SharpCorePDF/actions?query=branch%3Amain — running at the time of writing; results will be added as a comment.For the maintainer
The workflows on these PRs wait for approval because this is a first-time contributor (Actions → the run → Approve and run workflows). The same workflows already ran on the fork; links above.