Skip to content

Read the documents the JPEG, empty-name, glyph-box and date faults refused - #34

Open
CastleSoft wants to merge 4 commits into
MPCoreDeveloper:mainfrom
CastleSoft:reader-fixes
Open

CastleSoft wants to merge 4 commits into
MPCoreDeveloper:mainfrom
CastleSoft:reader-fixes

Conversation

@CastleSoft

Copy link
Copy Markdown

What this changes

Four kinds of document that ended a read in an exception the contract does not name — not a
PdfException — are now read: a JPEG whose scan header names tables or coefficients the decoder does not
have, a document that uses the empty name as a dictionary key, a page with a glyph drawn upside down, and a
document whose /CreationDate states a time-zone offset no time zone has. With these fixed, the pdf.js test
corpus (987 files) and the four hostile documents read with no fault, crash or timeout.

Stacked on #33 (itself on #32). Until those merge, the diff here includes them; review 9e2ab83 alone.

Why, and what is wrong today

  • JPEG scan header. The table selectors are four bits and the decoder has four tables of each class, and
    a progressive scan's spectral selection indexes the 64-entry zig-zag table; both were used unchecked
    (IndexOutOfRangeException from JpegDecoder.DecodeBlock). ParseScan now refuses a selector above 3, and
    a progressive scan unless Ss ≤ Se ≤ 63 and Ah, Al ≤ 13; the image is reported undecodable as any other.
  • The empty name. / on its own is a name ISO 32000-1 §7.3.5 allows. PdfDictionary.Set refused an
    empty key with ArgumentException, which surfaced while parsing, so four pdf.js corpus documents
    (issue3188, issue5896, pdfjs_wikipedia, poppler-742-0-fuzzed) did not open at all.
  • A glyph drawn upside down. A negative font size put the computed descender above the ascender, and
    PdfGlyph.BoundingBox handed PdfRectangle a bottom above its top (issue14415). The box is now taken
    between the two whichever way round they are.
  • A date offset beyond fourteen hours. PdfDate.TryParse clamped hours to 0–23 and handed the result to
    DateTimeOffset, which throws past ±14:00 (TAMReview). Such an offset is now read as no offset, so the
    method keeps its contract of answering rather than throwing.

The test that fails without this change

Run with the four source changes reverted: 14 fail, each with the original exception.

  • PdfJpegFilterTests.Refuses_a_scan_that_selects_a_table_beyond_the_four_there_are (4 cases),
    Refuses_a_progressive_scan_outside_the_block (the spectral cases)
  • PdfObjectModelTests.Keeps_an_entry_whose_key_is_the_empty_name, Opens_a_document_that_uses_the_empty_name_as_a_key
  • PdfTextScannerTests.Gives_a_glyph_drawn_upside_down_a_box
  • PdfSerializationTests.Reads_an_offset_no_time_zone_has_as_no_offset (the out-of-range cases; the
    in-range cases pass either way on purpose — they pin that ±14:00 and +05:30 are kept)

Checklist

  • dotnet build .\SharpCorePDF.slnx -c Release is clean on both legs, with zero warnings
  • A test fails when this change is reverted, and it is in this pull request
  • Public API surface unchanged
  • CHANGELOG.md has entries under [Unreleased] → Fixed
  • docs/validation.md moves the four defects to the fixed table; the --known flags they had in validation.yml and robustness.yml are removed
  • No internal story, epic or requirement identifier anywhere
  • Not drive-by

Anything else a reviewer should know

  • An existing test changes its assertion. PdfContainerTests.Refuses_an_entry_without_a_key asserted that
    an empty key is refused — the defect itself. It becomes
    Refuses_an_entry_without_a_key_and_keeps_one_whose_key_is_the_empty_name: a null key is still a
    programmer error (ArgumentNullException), the empty name is kept.
  • Verified locally (Windows, .NET 11): 1,316 tests pass and 1 is skipped (the incremental /ID[0] defect,
    next) on net10.0 and on net11.0. The pdf.js corpus plus the hostile documents: 991 files, 987 read,
    4 refused (PdfException, as before), 0 faults — with no --known flag left in either workflow.
  • Mutation testing (Stryker 5.1.0, the four changed files): 85.7 % on the changed lines (36 of 42). The six
    survivors were read: ≥ 3 on a selector is only observable with a fixture defining table 3 (ours define 0
    and 1, so a selector of 3 fails later anyway), >>> equals >> on a byte, and Ah/Al at exactly 13 is a
    legal value no test meaningfully exercises.

…ng and corpora

A writer that only reads its own output back cannot see the mistakes it shares with itself.
validation.yml now puts the example output and a validation corpus (written by
tools/SharpCorePDF.Validation, with a manifest of the text, passwords, permissions, revisions and
signature coverage each document must have) through qpdf, pypdf, pdfminer.six, MuPDF, pdf.js,
fontTools, OTS, PDFium and pyHanko, checks the incremental-update invariants, and replays the fuzz
seeds and four hostile documents. robustness.yml runs libFuzzer over the parser and the DCT, CCITT,
LZW, Flate and RunLength decoders, reads the veraPDF and pdf.js corpora one process per file, and
judges the PDF/A writer with veraPDF over the veraPDF corpus. compliance.yml cross-validates PDF/A-4.

The checks found defects that are not fixed here. Each is pinned by a skipped test
(PdfSignatureAttackTests, PdfIncrementalInvariantTests) or a strict --known-defect flag that fails
the run once the defect is fixed, and docs/validation.md lists them with their evidence.

No shipped package changes. The constitution's statement that no Python validator runs in CI is
corrected as a proposed PATCH amendment (2.0.1) for the maintainer to ratify. Local AI assistant
state (.claude/, CLAUDE.local.md) is ignored.
pdf.js 6 has no PDFDocumentProxy.destroy(), so the encryption check's pdf.js leg reported every document as unopenable: the call threw inside the try that was meant to catch a wrong password. The loading task owns the document and destroying it releases both. The fix was made before the previous commit but an editor saved a stale copy of the file over it.
Signature wrapping: the validator required only that the byte range start at 0 and end at the end of
the file, so bytes written into the unsigned gap could close the signature value early and add
entries while the digest still matched, and the signature was reported valid and covering the whole
document. The gap must now be exactly one hexadecimal string equal to the dictionary's /Contents;
anything else is MalformedByteRange, integrity Unknown and CoversWholeDocument false.

Decompression bombs: Flate, LZW and RunLength decoded without a limit, so a 1.1 MB document took the
process past 1 GiB. PdfEngineOptions.MaxDecodedStreamBytes (128 MiB) now stops them; decode buffers
above 16 MiB no longer stay in the shared array pool, and buffer growth no longer overflows.

Stack overflow: a chain of streams whose /Length is the next stream was resolved by recursion, and
60,000 links ended the process. The chain is followed no deeper than MaxNestingDepth, then the
stream is found by its endstream keyword, as for a /Length cycle.

Allocations sized by the file: an object stream's /N is bounded by what the stream can hold, and a
JPEG whose output is above the caller's budget is measured from its frame header instead of being
decoded first; the working sizes are computed in 64 bits.

Tests that fail without the fixes: PdfSignatureAttackTests (gap wrapping, gap shape),
PdfFilterTests (limits per filter, content stream past the limit), PdfReaderRecoveryTests (length
chain, indirect lengths, object stream /N), PdfJpegFilterTests (oversized frames, working
buffers). The three hostile documents of validation.yml are read under a 1 GiB heap, and their
--known flags are removed.
…fused

Four defects turned documents into exceptions that were not PdfExceptions, so a caller that handles
the contract's refusal still saw the read fail:

- A JPEG scan selected Huffman tables with 4-bit selectors and a progressive scan named its spectral
  selection, and both were used as indexes unchecked (IndexOutOfRangeException). ParseScan now refuses
  a selector above 3, and a progressive scan unless Ss <= Se <= 63 and Ah, Al <= 13.
- The empty name, a solidus on its own, is a name PDF allows, and four documents of the pdf.js test
  corpus use it as a key; PdfDictionary.Set refused it, so they did not open at all.
- A glyph drawn upside down (a negative font size) gave PdfGlyph.BoundingBox a bottom above its top,
  and the rectangle refused it, failing text extraction for the page.
- PdfDate.TryParse threw for a time-zone offset beyond fourteen hours, so one malformed /CreationDate
  failed the inspection of the whole document; such an offset is now read as no offset.

PdfContainerTests.Refuses_an_entry_without_a_key asserted the second defect and is rewritten: a null
key is still refused, the empty name is kept. Tests that fail without the fixes:
PdfJpegFilterTests (table selectors, progressive scan bounds), PdfObjectModelTests (empty-name key,
in a dictionary and in a document), PdfTextScannerTests (upside-down glyph), PdfSerializationTests
(offsets past fourteen hours). The pdf.js corpus and the hostile documents now read with no fault,
crash or timeout, and their --known flags are removed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant