Skip to content

Write missing media boxes, keep copied reals, harden recovery and the remaining edges - #36

Open
CastleSoft wants to merge 11 commits into
MPCoreDeveloper:mainfrom
CastleSoft:document-fixes
Open

CastleSoft wants to merge 11 commits into
MPCoreDeveloper:mainfrom
CastleSoft:document-fixes

Conversation

@CastleSoft

Copy link
Copy Markdown

Stacked on #35 (incremental-fixes); the first commits of this branch are those of #32–#35. Review the last three commits, or merge #32–#35 first and this diff shrinks to them.

Fixes the remaining open items of the review — B7 to B10 — and the faults the first runs of the new CI checks found on the fork.

5fbfc92 — Write a missing media box, keep copied reals, and fix what the first CI runs found

Defect Found by Fix Test that fails without it
A merge copied a page with no /MediaBox (and none inherited) without one; qpdf assumed Letter qpdf step (--known-defect, now removed) the page is written with the A4 box the inspection describes it with; PageSizeAssumed says so PdfMergeFidelityTests.Writes_a_media_box_for_a_page_that_has_none
Copied reals rewritten to five decimals (0.000012 → 0.00001); a real too large for the 32-byte buffer failed the whole write review (B8); document fuzz target a parsed real keeps its own text; large values take the allocating path PdfMergeFidelityTests.Writes_the_reals_it_copies_as_the_source_states_them, …_of_any_magnitude_…
Image copy route let InvalidDataException escape for damaged Flate data flate fuzz target refused with PdfSourceFormatException PdfContentReaderTests.Refuses_to_copy_an_image_whose_flate_data_is_damaged
FreeText annotation without /DA; appearance streams without /Resources Arlington check both written PdfDocumentEditorTests.States_the_default_appearance_and_resources_of_a_free_text_annotation, States_resources_on_an_appearance_that_needs_none
validate-arlington.py looked for validationReport; the service writes arlingtonReport Arlington job both element names accepted checked against the job's own report artifact

3a8b578 — Rebuild moved tables, read object streams in recovery, and harden five edges

Defect Fix Test that fails without it
A table that parses but whose offsets moved (inserted BOM, converted line endings) was trusted rebuilt when the catalog's offset, or most of a sample, misses its object PdfReaderRecoveryTests.Rebuilds_a_table_whose_offsets_no_longer_point_at_their_objects
Reconstruction could not see into object streams, so a PDF 1.5 file with a lost xref stream was refused object streams indexed; /Root taken from the xref stream's dictionary Rebuilds_a_table_from_object_streams_…, Rebuilds_a_document_whose_catalog_lives_in_an_object_stream (previously pinned the refusal)
AESV2 with a short declared key could not be opened AESV2 always uses 16 bytes PdfMergeEngineTests.Opens_an_aesv2_document_whose_crypt_filter_declares_a_short_key
Whole-buffer predictor: Columns × Colors wrapped to 0 (infinite loop) or negative (ArgumentOutOfRangeException) 64-bit row size shared with the streaming decoder PdfPredictorTests.Refuses_a_predicted_row_larger_than_the_engine_decodes
endstream scan accepted the first candidate that inflated — including truncated zlib data — and inflated every candidate from the start Adler-32-checked candidates, at most 16 inflated PdfStreamParsingTests.Scans_past_a_candidate_whose_compressed_data_is_cut_short
Generation numbers ignored (3 1 R resolved to 3 0 obj) mismatched generation resolves to null (ISO 32000-1 §7.3.10) PdfReaderRecoveryTests.Resolves_a_reference_to_a_generation_the_table_does_not_list_as_null
Edits and signatures appended a classic table to files that use xref streams the appended section is a cross-reference stream; qpdf and pyHanko accept it PdfIncrementalWriterTests.Appends_a_cross_reference_stream_…, PdfDocumentSignerTests.Signs_a_document_that_uses_cross_reference_streams_with_one
Public-corpora job fault: copying a listed attachment whose filespec embeds no stream threw ArgumentException (isartor-6-1-11-t01-fail-a.pdf) writes no bytes, matching ByteLength 0 PdfContentReaderTests.Copies_nothing_for_a_listed_file_that_embeds_no_stream

54c08d5 — Pin what the scoped Stryker run found unasserted

The scoped Stryker run (.clinerules/tooling.md) over every production file of the two commits above scored
70.8% on their changed lines. The survivors that were missing assertions now have them: a byte-exact test of
the cross-reference stream writer (rows, /Index, /W widths at each byte boundary, /Size, trailer entries),
the predictor limit at its boundary, the sixteen-candidate cap, the parser position after a scanned stream,
classic-vs-stream sections for signatures, and the wording of two messages. One redundant statement in
PdfWriter was removed.

Survivors left on purpose:

  • equivalent mutants — ConfigureAwait flips, bounds checks on offsets the scan already validated, /ObjStm
    at index 0 of an object header;
  • fallbacks for damaged files: which of several xref-stream dictionaries supplies /Root, and the
    "first candidate that inflates" fallback when no candidate's Adler-32 matches;
  • the AESV2 key-length switch for an RC4 revision-3 file that declares 40 bits — no fixture has that shape.

Verification

  • Build: 0 warnings, 0 errors. Tests: 1,357 pass on net11.0 and net10.0.
  • qpdf: every example output passes with no --known-defect flag.
  • pdf.js corpus (991 files) and hostile inputs: no fault, crash or timeout.
  • Both fuzz crash inputs replay with no fault.
  • Fork CI on 54c08d5 (CI, CodeQL, compliance, independent validation, robustness): https://github.com/CastleSoft/SharpCorePDF/actions?query=branch%3Amain — running at the time of writing; results will be added as a comment.

For the maintainer

The workflows on these PRs wait for approval because this is a first-time contributor (Actions → the run → Approve and run workflows). The same workflows already ran on the fork; links above.

…ng and corpora

A writer that only reads its own output back cannot see the mistakes it shares with itself.
validation.yml now puts the example output and a validation corpus (written by
tools/SharpCorePDF.Validation, with a manifest of the text, passwords, permissions, revisions and
signature coverage each document must have) through qpdf, pypdf, pdfminer.six, MuPDF, pdf.js,
fontTools, OTS, PDFium and pyHanko, checks the incremental-update invariants, and replays the fuzz
seeds and four hostile documents. robustness.yml runs libFuzzer over the parser and the DCT, CCITT,
LZW, Flate and RunLength decoders, reads the veraPDF and pdf.js corpora one process per file, and
judges the PDF/A writer with veraPDF over the veraPDF corpus. compliance.yml cross-validates PDF/A-4.

The checks found defects that are not fixed here. Each is pinned by a skipped test
(PdfSignatureAttackTests, PdfIncrementalInvariantTests) or a strict --known-defect flag that fails
the run once the defect is fixed, and docs/validation.md lists them with their evidence.

No shipped package changes. The constitution's statement that no Python validator runs in CI is
corrected as a proposed PATCH amendment (2.0.1) for the maintainer to ratify. Local AI assistant
state (.claude/, CLAUDE.local.md) is ignored.
pdf.js 6 has no PDFDocumentProxy.destroy(), so the encryption check's pdf.js leg reported every document as unopenable: the call threw inside the try that was meant to catch a wrong password. The loading task owns the document and destroying it releases both. The fix was made before the previous commit but an editor saved a stale copy of the file over it.
Signature wrapping: the validator required only that the byte range start at 0 and end at the end of
the file, so bytes written into the unsigned gap could close the signature value early and add
entries while the digest still matched, and the signature was reported valid and covering the whole
document. The gap must now be exactly one hexadecimal string equal to the dictionary's /Contents;
anything else is MalformedByteRange, integrity Unknown and CoversWholeDocument false.

Decompression bombs: Flate, LZW and RunLength decoded without a limit, so a 1.1 MB document took the
process past 1 GiB. PdfEngineOptions.MaxDecodedStreamBytes (128 MiB) now stops them; decode buffers
above 16 MiB no longer stay in the shared array pool, and buffer growth no longer overflows.

Stack overflow: a chain of streams whose /Length is the next stream was resolved by recursion, and
60,000 links ended the process. The chain is followed no deeper than MaxNestingDepth, then the
stream is found by its endstream keyword, as for a /Length cycle.

Allocations sized by the file: an object stream's /N is bounded by what the stream can hold, and a
JPEG whose output is above the caller's budget is measured from its frame header instead of being
decoded first; the working sizes are computed in 64 bits.

Tests that fail without the fixes: PdfSignatureAttackTests (gap wrapping, gap shape),
PdfFilterTests (limits per filter, content stream past the limit), PdfReaderRecoveryTests (length
chain, indirect lengths, object stream /N), PdfJpegFilterTests (oversized frames, working
buffers). The three hostile documents of validation.yml are read under a 1 GiB heap, and their
--known flags are removed.
…fused

Four defects turned documents into exceptions that were not PdfExceptions, so a caller that handles
the contract's refusal still saw the read fail:

- A JPEG scan selected Huffman tables with 4-bit selectors and a progressive scan named its spectral
  selection, and both were used as indexes unchecked (IndexOutOfRangeException). ParseScan now refuses
  a selector above 3, and a progressive scan unless Ss <= Se <= 63 and Ah, Al <= 13.
- The empty name, a solidus on its own, is a name PDF allows, and four documents of the pdf.js test
  corpus use it as a key; PdfDictionary.Set refused it, so they did not open at all.
- A glyph drawn upside down (a negative font size) gave PdfGlyph.BoundingBox a bottom above its top,
  and the rectangle refused it, failing text extraction for the page.
- PdfDate.TryParse threw for a time-zone offset beyond fourteen hours, so one malformed /CreationDate
  failed the inspection of the whole document; such an offset is now read as no offset.

PdfContainerTests.Refuses_an_entry_without_a_key asserted the second defect and is rewritten: a null
key is still refused, the empty name is kept. Tests that fail without the fixes:
PdfJpegFilterTests (table selectors, progressive scan bounds), PdfObjectModelTests (empty-name key,
in a dictionary and in a document), PdfTextScannerTests (upside-down glyph), PdfSerializationTests
(offsets past fourteen hours). The pdf.js corpus and the hostile documents now read with no fault,
crash or timeout, and their --known flags are removed.
…t identifier

UseIncrementalUpdate ran the full rewrite into an appended writer, so every object of the document,
the signature dictionary of a signed source included, was written again under a new number. Every
earlier byte was kept, but the file grew by its own size per edit, and validators that look at what
changed after a signature (pyHanko, Acrobat) saw the signed field as rewritten: pyHanko placed a
signature followed by an edit as CONTIGUOUS_BLOCK_FROM_START instead of ENTIRE_REVISION.

In an appended revision the edited document's copier now keeps the numbers of its objects, the
catalog, the page tree, the pages and the information dictionary are written under the numbers they
already have, and the writer leaves out an object that would be written exactly as the source states
it. Rotating a page of a signed document now appends 716 bytes and three objects instead of 20,969
bytes and twelve; the signature dictionary is not in the revision.

The trailer also kept neither half of the source's /ID. ISO 32000-1 14.4 makes the first element
permanent across incremental updates; it is now the source's, and only the second element is new.

Tests that fail without the change: PdfIncrementalInvariantTests (the identifier, an update that
appends only what it changed, a signature dictionary left out of the revision); the corpus checks
validate-incremental.py and validate-signatures.py now pass with no --known-defect flag. The manual's
editing chapter described every edit as a complete rewrite and now says what the option does.
…CI runs found

- A merge copied a page that states no /MediaBox, and inherits none, without one, so the result broke the
  rule that every page has a media box and each reader assumed its own size. The page is written with the
  A4 box the inspection already describes it with, and PageSizeAssumed says so.
- A merge or an edit rewrote every real it copied to five decimals: a Type 3 font matrix of 0.000012 came
  out as 0.00001. A real read from a document now keeps its text when that is a well-formed PDF real and
  is written back from it. A real too large for the formatter's 32-byte buffer failed the write of the
  whole document with InvalidOperationException; the first run of the document fuzz target found it, and
  such a value is now formatted on the allocating path.
- The image copy route let InvalidDataException escape for damaged Flate data, with or without a
  predictor (the flate fuzz target's first find); it now refuses it with PdfSourceFormatException.
- The Arlington service names its verdict element arlingtonReport, which validate-arlington.py now reads.
  Its first run found that a free text annotation lacked the /DA ISO 32000-1 requires and annotation
  appearances lacked /Resources; both are now written.

Tests that fail without the changes: PdfMergeFidelityTests (media box, copied reals, any magnitude),
PdfContentReaderTests.Refuses_to_copy_an_image_whose_flate_data_is_damaged, and
PdfDocumentEditorTests.States_the_default_appearance_and_resources_of_a_free_text_annotation and
States_resources_on_an_appearance_that_needs_none. qpdf now passes every example output with no
--known-defect flag, so the qpdf step lists none.
…e edges

Recovery
- A cross-reference table that parses can still be wrong: bytes inserted or removed after it was written
  move every object. Reconstruction now also runs when the catalog's offset, or most of a sample of
  offsets, does not land on the n g obj header of its own object.
- Reconstruction indexes the objects of every object stream the scan finds, and takes /Root from the
  dictionary of the last cross-reference stream, so a PDF 1.5 file whose cross-reference stream is lost
  reads instead of being refused with "no document catalog". The test that pinned the refusal now pins
  the rebuild.

Hardening
- AESV2 always uses a sixteen-byte key; a crypt filter that declares fewer made the file unopenable.
- The whole-buffer predictor path sizes a row in 64 bits like the streaming decoder: a declared
  Columns x Colors that wrapped to zero looped forever under TIFF prediction, and to a negative number
  threw ArgumentOutOfRangeException.
- A stream with a wrong /Length is cut where its zlib data is complete, by its Adler-32, instead of at the
  first endstream whose truncated bytes inflate; at most sixteen candidates are inflated.
- A reference to a generation the table does not list resolves to null (ISO 32000-1, 7.3.10).
- A revision appended to a file whose newest section is a cross-reference stream, by an edit or a
  signature, ends in a cross-reference stream instead of a classic table. qpdf and pyHanko accept an edit
  and two signatures appended this way.

Corpus fault
- The public-corpora job faulted on isartor-6-1-11-t01-fail-a.pdf: a file specification that only names an
  external file is listed as an attachment with ByteLength 0, and copying it threw ArgumentException.
  It now writes no bytes; an id the document does not have is still refused.

Each change has a test that fails without it. Full suite passes on net11.0 and net10.0; the pdf.js corpus
run reports no fault, crash or timeout.
The scoped run over the files of the last two commits scored 70.8% on their changed lines. The survivors
that were missing assertions now have them:

- PdfCrossReferenceStreamWriterTests compares every row of an appended cross-reference stream byte for
  byte, its /Index subsections, the width of each /W field at the byte boundaries, its /Size and the
  trailer entries it carries or drops.
- The free text /DA test uses three different colour channels; the merge warning names the page as a
  person counts it; the damaged-Flate refusal says why.
- The predictor limit is pinned at the largest row accepted and one column more, and a zero-bit row is
  refused.
- The endstream scan leaves the parser behind the keyword it chose, and the sixteen-candidate cap is pinned
  at its boundary.
- A signature on a classic-table document keeps a classic table, and only an edit's stream lists the head
  of the free list.

PdfWriter's stream path no longer sets /Size before the stream writer, which derives it itself; the
statement was the survivor nothing could kill. The remaining survivors are equivalent mutants
(ConfigureAwait flips, bounds checks on offsets the scan already validated) or fallbacks for damaged files
noted in the pull request.
…ection does

The second robustness run, on 54c08d5, found two more faults now that the first ones are fixed:

- flate: the image copy route caught the decompressor's InvalidDataException but not the ZLibException
  it throws for other damaged input (crash-0b3d19d6). Both are now refused with
  PdfSourceFormatException. ZLibException is not public, so it is recognised by name rather than by
  catching every IOException, which would also swallow the destination's own failures.
- document: a page declaring /Rotate 91 made the edit engine throw ArgumentOutOfRangeException
  (crash-7201f51f). The edit engine read /Rotate as an integer and required a multiple of 90, while the
  inspection reads any number and rounds to the nearest quarter turn, so a real /Rotate was also read as
  none. The edit engine now uses the inspection's reading, and the quarter turns are reduced before they
  become an integer, so a declared value beyond int's range no longer wraps.

Tests that fail without the changes: PdfContentReaderTests.Refuses_to_copy_an_image_whose_flate_data_is_damaged
(the fuzz input's payload) and PdfMergeFidelityTests.Edits_a_page_whose_rotation_is_not_a_quarter_turn.
Both crash inputs and the two from the first run replay with no fault. Scoped Stryker over the two changed
files kills every mutant on the changed lines.
The third robustness run, on 7c97a38, found a stack overflow (crash-d34275a3): a merge walks the outline
recursively, one frame per level, and a source nested about fifty thousand levels deep ended the
process. The walk now stops at 256 levels and reports BookmarksTruncated, and a bookmark reached a
second time - a /First or /Next pointing back up the tree, or a child two parents share - is copied once
and reported with BookmarkSkipped, which also ends cycles and the repeated walks of shared subtrees.

Tests: PdfMergeOutlineLimitsTests (depth limit, shared child, self cycle). The crash input replays with
no fault; the full suite passes.
The robustness run on f529b02 found a crash in the document fuzz target (crash-e154541f).
Reconstruction keeps the damaged table's entries beside the ones its scan finds, and the search for a
cross-reference stream's trailer sliced the document at each of their offsets unchecked. An offset past
the end of the file, or one beyond the range of an index that wraps negative when cast, threw
ArgumentOutOfRangeException instead of letting the rebuild go on to find the catalog. Such an entry is now
passed over. The catalog search needs no such check: PdfLexer clamps the position it starts at, which a
scoped Stryker run confirmed by leaving the same guard there unkillable, so it was not added.

Test that fails without the change:
PdfReaderRecoveryTests.Rebuilds_a_table_that_states_an_offset_past_the_end_of_the_file, for an offset
past the end and one that wraps negative. The crash input replays with no fault. On the changed lines the
two surviving mutants are equivalent: offset 0 is the header and an offset equal to the length gives an
empty window, so neither boundary changes what is read.

The outline-depth fix of f529b02 is now in CHANGELOG.md and the fixed-defects table of
docs/validation.md, which it had not been.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant