Speed up PE checksum calculation

[francoisgirinon-netizen]

Pull Request Type

• Bug fix
• New feature
• Code style / formatting / renaming
• Refactoring (no functional or API changes)
• Build / CI related changes
• Documentation
• Other (please describe): Performance

Related Issue

Issue number: N/A

Current Behavior

The PE checksum is a one's complement sum of the 16-bit words of the file. It is
computed with a carry fold after every word, and pe_calc_realchecksum() copies
each 64 KB block through a memory BIO before summing it. On a 3.7 GB PE file this
makes verify and remove-signature slower than necessary.

New Behavior

The 16-bit words are added in a 64-bit accumulator and folded once at the end.
The one's complement sum is mathematically the same, so the checksum is unchanged.
pe_calc_realchecksum() sums the mapped input directly, without the BIO copy.
On a 3.7 GB PE file verify takes 15% less time and remove-signature 15%.

Scope of Changes

• pe.c: add pe_checksum_sum() and pe_checksum_fold(), used by
  pe_calc_checksum() and pe_calc_realchecksum().
  The 64-bit accumulator cannot overflow: a file is limited to 4 GB, so at most
  2^31 words of at most 0xFFFF are added.
• tests/check_pe_checksum.py, cmake/CMakeTest.cmake: tests of the CheckSum field.

Testing

• Existing tests

• New tests added

• Manual testing

• ctest: 387/387 passed on Linux x86-64 (OpenSSL 3.5), without LARGE_FILE_TESTS.

• New tests checksum_*: sign tests/files/unsigned.exe with an overlay of 1, 3, 65537
  or 300001 bytes (filled with 0x00, 0x7F or 0xFF) and compare the CheckSum field of
  the signed file with tests/check_pe_checksum.py, an independent implementation of
  the documented algorithm. They also pass on the previous algorithm. With a wrong fold
  or a wrong exclusion of the CheckSum field in pe_checksum_sum()/pe_checksum_fold(),
  every case fails.

• The result was compared with the previous algorithm on 200000 random and edge case
  inputs (all zero, all 0xFF, odd lengths, any header size, any chunking).

• The signed and unsigned output files are byte identical.

• Not tested: Windows, macOS, 32-bit builds.

Additional Notes

• The odd trailing byte is ignored and the checksum field is excluded exactly as before.

License Declaration

• I hereby agree to license my contribution under the project's license.

[mtrojnar]

Is the performance improvement significant compared to the time already used for comparing cryptographic hash? 15% does not seem like a significant justification.

My point: "if it ain't broke, don't fix it".

[francoisgirinon-netizen]

Measured on the largest PE file that can be verified (4,294,960,000 bytes, SHA-256, file in cache, old and new builds run alternately):

	before	after
verify (median of 4 runs)	14.7 s	12.2 s
sign (last 2 stable runs)	about 16.4 s	about 14.1 s

For reference, openssl dgst -sha256 on the same file takes 11.9 s. The gain is about 2.4 s on this file. Since a signed PE file is limited to 4 GB, this is close to the maximum saving. For typical binaries it is a few milliseconds.

The change is covered by tests that compare the CheckSum field of signed files with an independent implementation. Happy to go either way.