Parquet Java ALP Implementation - #3397
vinooganesh wants to merge 75 commits into
Conversation
Implements ALP encoding for FLOAT and DOUBLE types, which converts floating-point values to integers using decimal scaling, then applies Frame of Reference (FOR) encoding and bit-packing for compression. New files: - AlpConstants.java: Constants for ALP encoding - AlpEncoderDecoder.java: Core encoding/decoding logic - AlpValuesWriter.java: Writer implementation - AlpValuesReaderForFloat/Double.java: Reader implementations Includes comprehensive unit tests and interop test infrastructure.
Restore original comment indentation that was accidentally changed.
Escape <= characters as <= in javadoc comments to avoid malformed HTML errors during documentation generation.
ALP encoding is not yet part of the parquet-format Thrift specification, so it cannot be converted to org.apache.parquet.format.Encoding. Skip it in the testEnumEquivalence test and add a clear error message in the converter for when ALP conversion is attempted.
size and add independent reader/writer verification tests
93c365a to
6d65eaa
Compare
Switch encode/decode from division-based formula to multiply-by-reciprocal using separate POW10_NEGATIVE arrays, matching C++ Arrow's approach: - Encode: fastRound(value * POW10[e] * POW10_NEGATIVE[f]) - Decode: encoded * POW10[f] * POW10_NEGATIVE[e] Add fastRound helpers with sign branching for correct negative value rounding. Remove version byte from page header (8 -> 7 bytes). Empty pages now emit a 7-byte header with numElements=0. Update all hand-crafted binary tests to match the new header format and add comprehensive end-to-end tests for overflow boundaries, large-scale data, preset caching, and NaN bit-pattern preservation.
- Rewrite TestInterOpReadAlp to use LocalInputFile instead of Hadoop FileSystem, fixing failures on Java 24+ where Subject.getSubject is removed. Tests now read C++ ALP parquet files directly without going through Hadoop security/UGI. - Add AlpExceptionCountTest with per-column exception rate reporting against the real Spotify and Arade floating-point datasets from the parquet-testing repository. Useful for comparing Java vs C++ ALP compression ratios.
- Switch findBestFloatParams/findBestDoubleParams from minimizing exception count to minimizing estimated compressed size (length * bitWidth + exceptions * (typeSize + 2 bytes)), matching the C++ ALP cost model. This closes the ~4-5% compression gap vs C++. - Rewrite sampler to collect evenly-spaced sample vectors and run findBestParams on each, then rank by win count. Matches C++ AlpSampler behavior more closely than the previous HashMap-based approach. - Minor fixes: IOExceptionUtils null check, MemoryManager volatile scale, Files utility cleanup, parquet-cli dependency update.
15bc06d to
24c23e5
Compare
- Move shared LE helper methods (getShortLE/getIntLE/getLongLE) to AlpValuesReader base class; remove duplicates from subclasses - Make EncodingParams fields package-private (remove public modifier) - Replace fully-qualified java.util.Arrays.fill calls with imported Arrays.fill in both float and double readers; add missing import to double reader - Add explanatory comments to getBufferedSize() magic numbers (3 for float, 5 for double) explaining the overhead breakdown - Add ALP enabled state to ParquetProperties.toString() - Add ALP support to DefaultV1ValuesWriterFactory for float and double columns
- Revert Files.java, IOExceptionUtils.java, MemoryManager.java, and parquet-cli/pom.xml to master state; these changes are unrelated to ALP and should be submitted in separate PRs - Clarify ParquetMetadataConverter error message: ALP encoding is defined in the ALP paper (enum value 26) but is not yet in the parquet-format Thrift spec, so ALP cannot be written through the Hadoop write path; the error message now explains what needs to happen to remove the block
prtkgaur
left a comment
prtkgaur
left a comment
There was a problem hiding this comment.
The code organization looks good to me and the code follows the spec. I looked for areas of any extra buffer allocations which might impact performance and I think it is optimally written.
I think we should add a few benchmarks and publish numbers from them.
Thanks for working on this Vinoo!
prtkgaur
left a comment
prtkgaur
left a comment
There was a problem hiding this comment.
Wanted to make sure we have the following testing.
For the cross compatibility testing are we making sure that we write both V1 and V2 pages and the implementation in other language is able to read it.
- Add build-time Perl script to patch generated Encoding.java with ALP(10) after Thrift codegen (process-sources phase), since parquet-format 2.12.0 does not yet include ALP in its Thrift spec - Remove guard in ParquetMetadataConverter.getEncoding() that blocked ALP writes; Encoding.ALP now exists in the patched Thrift enum - Add withAlpEncoding() builder methods to ParquetWriter - Add TestInterOpReadAlp: Java V1/V2 write+read round-trip tests and C++ Arrow interop tests (reads alp_spotify1.parquet, alp_arade.parquet, etc.) - Add AlpEncodingBenchmarks JMH benchmark
…d pyarrow interop test - AlpValuesWriter: stop clearing cachedPresets in reset() so preset (e,f) pairs survive page flushes; eliminates redundant full parameter search on every page after the first, cutting write time ~60% - AlpEncodingBenchmarks: clarify Javadoc that comparison is PLAIN+UNCOMPRESSED (no codec), not plain+ZSTD - parquet-benchmarks pom: add explicit annotationProcessorPaths and proc=full for jmh-generator-annprocess so BenchmarkList is generated under Java 23+ - TestInterOpReadAlp: add pyarrow cross-language compatibility test (skips if pyarrow unavailable or does not yet support ALP encoding)
- ParquetProperties: add withAlpVectorSize(int) and withAlpVectorSize(String, int) builder methods plus getAlpVectorSize(ColumnDescriptor) accessor, defaulting to AlpConstants.DEFAULT_VECTOR_SIZE (1024). - AlpConstants: promote validateVectorSize to public so the builder can validate eagerly across packages. - DefaultV1/V2 ValuesWriterFactory: pass the configured vector size to the 4-arg AlpValuesWriter constructors. - ParquetWriter.Builder: expose withAlpVectorSize facades mirroring withAlpEncoding. - TestInterOpReadAlp: add testJavaWriteAlpCustomVectorSize covering 4500 rows at vectorSize=4096 so we cross a full vector boundary and verify round-trip equality. A wrong log_vector_size byte would surface as decode garbage, so round-trip equality is sufficient proof the configured size took effect on the wire. Enables generating ALP test fixtures at different vector sizes (e.g. 4096) for cross-language compatibility testing against the C++/Rust/Go implementations.
Logging and debug output was missing the new alpVectorSize field alongside the existing 'ALP enabled' line. Cosmetic only — no behavior change.
Adds generateAlpFixturesAtMultipleVectorSizes to TestInterOpReadAlp. For each of the four source files in parquet-testing PR apache#100 (alp_spotify1, alp_arade, alp_float_spotify1, alp_float_arade), reads every row, then re-encodes as Java ALP at both vectorSize=1024 and vectorSize=4096. Output goes to ALP_OUTPUT_DIR (default ${user.dir}/alp-java-generated/), producing 8 files total named alp_java_<stem>_vs{1024,4096}.parquet. Each output is verified by reading back through the standard reader path and bit-comparing every value via doubleToRawLongBits / floatToRawIntBits — catches NaN payload and signed-zero divergence, not just numerical equality. Skips when ALP_TEST_DATA_DIR isn't set, so it stays inert in CI on machines without the source datasets. To run: git clone --branch alpFloatingPointDataset \\ https://github.com/prtkgaur/parquet-testing.git ALP_TEST_DATA_DIR=path/to/parquet-testing/data \\ mvn -pl parquet-hadoop \\ -Dtest=TestInterOpReadAlp#generateAlpFixturesAtMultipleVectorSizes \\ test
Extends generateAlpFixturesAtMultipleVectorSizes to vary writer page
version (PARQUET_1_0, PARQUET_2_0) as a third axis alongside dataset
and ALP vector size. Output grows from 8 → 16 files per run:
alp_java_<stem>_v{1,2}_vs{1024,4096}.parquet
Page version is orthogonal to ALP encoding — the page version
difference lives in the parquet protocol layer, not in the ALP
payload — but covering both axes makes the fixture set fully
symmetric for cross-language compatibility verification. C++/Rust/Go
readers can use the V1 and V2 variants to prove their decoders
handle Java-written ALP regardless of how the surrounding pages are
framed. Avoids an asymmetry where the existing PR apache#100 set has C++
at V1 and Java at V2 with no overlap.
All 16 outputs independently verified against the canonical
_expect.csv truth files from parquet-testing PR apache#100 (1.56M values,
0 mismatches).
The reader was asserting that the ALP header's num_elements equals the data page's valuesCount, but those values differ whenever a column has nulls: num_elements is the count of non-null values that went through ALP encoding, while valuesCount is the total row count of the page (which includes null positions tracked by definition levels). The strict equality check made the reader reject every optional float/double column with at least one null value. Relaxes the check to numElements > valuesCount — the header can never legitimately claim more encoded values than the page has rows, but it can claim fewer when nulls are present. The downstream code already uses numElements (not valuesCount) to drive vector allocation and decoding, so the rest of the read path is unchanged. This was surfaced by the corner-case fixture per parquet-testing issue apache#105, which exercises optional columns with null values.
Two new tests in TestInterOpReadAlp: readAllFixtureFilesIndependently Opens every alp_java_*.parquet in ALP_OUTPUT_DIR and asserts each column chunk declares Encoding.ALP and decodes through the standard reader path without error. Separate from the generator's own round-trip verification so reader correctness surfaces as a distinct signal in CI when the fixtures are present. Skips cleanly when ALP_OUTPUT_DIR is empty so it stays inert in default CI environments. generateAndVerifyCornerCaseFixture Writes a single small fixture file (alp_java_cornercases.parquet, ~60 KB) targeting the corner cases enumerated in parquet-testing issue apache#105: vectors with no exceptions, one exception per vector, all exceptions, NaN/Inf/-0.0, constant values (bit_width=0), multi-vector with differing exponents, and optional columns with nulls. Both f32 and f64 variants — 14 columns × 2048 rows total. Reads each column back and bit-exactly verifies every value against the expected pattern via doubleToRawLongBits / floatToRawIntBits. The corner-case fixture is intended as a candidate file for parquet-testing PR apache#100 once naming/design is confirmed. Generating it also surfaced (and verified the fix for) a pre-existing reader bug where optional columns with nulls couldn't be decoded — see the preceding commit.
The corner-case fixture (alp_java_cornercases.parquet) is synthetic — it isn't derived from any raw dataset in parquet-testing PR apache#100, so the existing alp_*_expect.csv files don't cover it. That left cross-language verifiers with no independent ground truth to check the parquet file against; they had to either trust the Java reader or duplicate the construction recipe in their own code. writeCornerCaseCsvTruth now dumps the expected values straight from the construction recipe into alp_java_cornercases_expect.csv next to the parquet, every time the generator runs. The CSV uses the same format conventions as the existing _expect.csv files (comma- separated, header row, no quoting) plus two extensions: • Empty field = null cell (for optional columns) • Special values printed via Java's standard toString: "NaN", "Infinity", "-Infinity", "-0.0". These all parse via C++ std::stod / std::stof per the standard (case-insensitive, "inf" and "infinity" both accepted). The Arrow C++ ALP decoder reads the parquet and compares against this CSV bit-exactly: 27306 non-null cells + 1366 null cells across 14 columns × 2048 rows, 0 mismatches. This makes the corner-case fixture self-documenting and verifiable by any future cross-language tooling without rerunning the Java generator to discover what the expected values are.
The builder tracked the ALP enabled flag and vector size as two independent ColumnProperty scaffolds, merged them into a per-column AlpConfig in buildAlp(), then split them apart again in the copy constructor. Nothing needed that round trip. Hold one ColumnProperty<AlpConfig> instead and expose withAlp(AlpConfig) / withAlp(columnPath, AlpConfig) directly. withAlpEncoding and withAlpVectorSize stay as read-modify-write conveniences over it, so every existing caller -- including ParquetWriter.Builder -- is unchanged. buildAlp and the copy-constructor split are both gone. Vector size validation moves into the AlpConfig constructor, which is where an invalid size should be rejected regardless of how the config was built. That also removes ParquetProperties' last reference to AlpConstants. ColumnProperty.Builder gains getDefaultValue/getValue so a caller can modify one field of an existing value rather than replace the whole thing. The class is package-private, so this is not public API.
Addresses several review comments in one pass, no behaviour change: - AlpConstants and every member drop to package-private. Only ParquetProperties referenced it from outside the package, and the preceding commit removed that, so none of this needs to be public API. - Constants move to the class that uses them. The sampler constants (SAMPLER_*, MAX_PRESET_COMBINATIONS) are writer-only and move to AlpValuesWriter; the rounding magic numbers and power-of-ten tables are codec-only and move to AlpCodec. AlpConstants is left holding just the wire format: header/metadata sizes, mode and encoding markers, vector size bounds, and per-type exponent limits. - Rename AlpEncoderDecoder to AlpCodec, and its test to match. - Replace the wildcard static imports of AlpConstants with explicit ones in all five files. - Fix the interleaved-page-layout box in the AlpValuesWriter and AlpValuesReader javadoc. The "4B × numVectors" cell was seven source characters wide for a glyph that renders as one, so the box could not line up in both the editor and the rendered javadoc. Using the literal glyph makes every row 73 characters in both.
Adding ALP to the Encoding enum previously meant post-processing the thrift-generated Encoding.java with a perl script wired into the build, because parquet.thrift was unpacked from the released parquet-format jar and could not be edited. Since parquet.thrift is now inlined in the repo (apache#3709), ALP = 10 can be declared in the enum directly. Removes src/main/perl/patch-alp-encoding.pl and the exec-maven-plugin execution that ran it. Verified the generated Encoding.java still carries ALP(10) after BYTE_STREAM_SPLIT(9) with no patch step involved. The entry is marked as a local addition: dev/update-parquet-thrift.sh overwrites this file from upstream and would silently drop ALP, so the notice tells whoever re-runs it to re-apply the entry until the encoding is accepted into parquet-format, at which point both the entry's notice and this divergence go away.
isFloatException has to encode the value to check the round trip, so calling it and then encodeFloat did the work twice. The writer's loop was worse: exception check, encode again, plus a placeholder scan that encoded a third time. tryEncodeFloat and tryEncodeDouble now return the exception flag and the encoded value together through a caller-owned EncodeResult. A holder rather than an Optional keeps the per-value path allocation free. Encoded bytes are unchanged, and a new test asserts the new path agrees with the old exception-check-then-encode pair bit for bit. The benchmark is a wash (197.9 vs 194.2 ms/op, error bars overlap), since encoding is small next to compression and page assembly.
3bc0835 to
8e743c6
Compare
|
@wgtmac thanks again for the review! I've gone through all of your comments and replied to each thread individually. A summary of where things landed: The three reader issues are fixed in 338fb99. You were right that numElements could be forged up to Integer.MAX_VALUE, and while fixing it I found the allocation was reachable rather than theoretical, since MultiBufferInputStream.slice allocates before its EOF check. The vector count is bounded against the bytes actually present now, the offset array is validated up front, each vector's reads are confined by the following offset, and skip no longer overflows. The sampling problem you spotted was the most valuable comment in the review. Your arithmetic was exactly right: reset() ran at every page boundary and cleared the sample state, so the threshold was never reached and the preset cache was never built. Fixed in eb0e594, with a regression test that fails without it. The two test comments were both fair. The allocation test was inflating valuesCount rather than forging num_elements so it never reached the path it claimed to, and catchAny was swallowing Errors so it could not really fail. Both fixed in 338fb99. Beyond that: ALP is configured through a single AlpConfig now with withAlp (99ae547), AlpConstants is package private with the constants moved to whichever class actually uses them, AlpEncoderDecoder is renamed to AlpCodec, the wildcard imports are gone and the javadoc box is fixed (78c1d18), ALP is declared directly in parquet.thrift with the perl patch deleted (229e0d9), and the double encoding is gone (8e743c6). There are three places where I would like your opinion rather than assuming I picked right:
The one thing I cannot fix on my own is the interop test. The fixtures are proposed in apache/parquet-testing#100, which is still open, so the test skips in CI today. If you have any pull there it would help, since cross language read back matters for the vote more than for that test alone. parquet-column is at 856 tests with no failures. Whenever you have time for another look, I would appreciate it. |
| public static final boolean DEFAULT_IS_DICTIONARY_ENABLED = true; | ||
| public static final boolean DEFAULT_IS_BYTE_STREAM_SPLIT_ENABLED = false; | ||
| public static final boolean DEFAULT_IS_ALP_ENABLED = false; | ||
| public static final int DEFAULT_ALP_VECTOR_SIZE = AlpConfig.DEFAULT_VECTOR_SIZE; |
There was a problem hiding this comment.
Why rename this constant? Feels like we should just be calling AlpConfig.DEFAULT_VECTOR_SIZE at the call site
There was a problem hiding this comment.
Agreed, and thanks for flagging it — there was no reason to alias the constant. Removed in 7377588, so the call site now reads AlpConfig.DEFAULT_VECTOR_SIZE directly.
| * @param vectorSize values per encoded vector; must be a power of 2 in the supported range | ||
| * @throws IllegalArgumentException if {@code vectorSize} is not a supported vector size | ||
| */ | ||
| public AlpConfig(boolean enabled, int vectorSize) { |
There was a problem hiding this comment.
There probably shouldn't be an "enabled" as part of this config. Whether ALP is on or off is separate from how it is configured.
There was a problem hiding this comment.
That separation makes sense, thank you. Fixed in e3b846a, where AlpConfig holds only the vector size and a column is ALP encoded when it has a config at all.
| DEFAULT_IS_BYTE_STREAM_SPLIT_ENABLED | ||
| ? ByteStreamSplitMode.FLOATING_POINT | ||
| : ByteStreamSplitMode.NONE); | ||
| alp = ColumnProperty.<AlpConfig>builder() |
There was a problem hiding this comment.
Silently overriding byteStreamSplitEnabled
We probably want a slightly more complicated logic here
If (bytestream && ALP) fail (can't use both)
... elif(bytestream) {
bystreamEnabled ...
} elif( alp) {
alpEnabled
}
There was a problem hiding this comment.
Good catch, it was silently dropping BYTE_STREAM_SPLIT. Both that conflict and ALP on a non float or double column are now rejected in newColumnWriteStore in e660c6c, rather than resolved by precedence.
| * @param config the ALP configuration | ||
| * @return this builder for method chaining. | ||
| */ | ||
| public Builder withAlp(AlpConfig config) { |
There was a problem hiding this comment.
These APIs seem a bit over-specified for what we allow?
I feel like we have
withAlp(config) // sets ALP on everywhere
withAlp(columnPath, config) // set config on one column
withAlp() // Sets ALP on with default config
withAlp(columnPath) //sets alp as on for a single column
I'm not sure we need the ability to flip the encoding back off
Then I would make sure that none of this includes implementation details (like config parameters) in the method names.
I would also consider whether we should fail if "ALP" is enabled on a non float/double column
There was a problem hiding this comment.
if we did want the "off functions"
withoutAlp()
There was a problem hiding this comment.
Your sketch is what the API looks like now, thanks for laying it out: e3b846a collapses the six methods into withAlp(), withAlp(config), withAlp(columnPath), withAlp(columnPath, config) and withoutAlp(), with the vector size only reachable through AlpConfig. The non float or double check is in e660c6c, and the one tradeoff worth flagging is that per-column disabling goes away with the boolean setters, so let me know if you would like that kept.
There was a problem hiding this comment.
What do you think? Do we need per column disabling?
There was a problem hiding this comment.
Yes, I think so, and it is in a8c584f as withoutAlp(columnPath). The reason it was missing is that ColumnProperty treated a null per-column value as unset and fell back to the default, so it now checks containsKey instead and an explicit null overrides; every other per-column property either cannot hold a null or already defaults to one, so nothing else changes behaviour.
Two things had to come with it. The schema check was treating "column named in the ALP property" as "ALP enabled here", so disabling ALP on an int column failed with a message claiming ALP was enabled on it, and toString rendered a disabled column as null again, which undoes your other nit. Both are fixed and covered.
DEFAULT_ALP_VECTOR_SIZE was just a second public name for AlpConfig.DEFAULT_VECTOR_SIZE, with one use. Drop it and read the constant from AlpConfig directly.
AlpConfig carried an enabled flag, which mixed up whether a column uses ALP with how it encodes. A column is now ALP encoded when it has a config and is not when it has none, so AlpConfig holds only the vector size and the builder property defaults to null. That collapses the six builder methods into withAlp(), withAlp(config), withAlp(column), withAlp(column, config) and withoutAlp(). The vector size is no longer its own setter, so an implementation detail stops appearing in a method name, and the read-modify-write setters that needed getDefaultValue/getValue on ColumnProperty.Builder are gone along with those two accessors. Per-column disabling goes away with them: withoutAlp() clears the default but cannot turn off a column enabled by name, since ColumnProperty treats a null per-column value as unset.
The writer factories checked ALP before BYTE_STREAM_SPLIT, so enabling both on a column silently dropped BYTE_STREAM_SPLIT. Enabling ALP on a column that is not FLOAT or DOUBLE was ignored just as quietly. Both are now checked against the schema in newColumnWriteStore, before any data is written, and rejected rather than resolved by precedence. The type check only applies to columns named explicitly, so turning ALP on for everything still means the float and double columns. The check returns immediately when no ALP is configured, which is the common case and keeps it off the per-rowgroup path for everyone else.
| + "Statistics enabled: " + statisticsEnabled + '\n' | ||
| + "Size statistics enabled: " + sizeStatisticsEnabled; | ||
| + "Size statistics enabled: " + sizeStatisticsEnabled + '\n' | ||
| + "ALP: " + alp; |
There was a problem hiding this comment.
nit: A little odd here since it will say "ALP: null" when unset probably worth a bit more clarity
There was a problem hiding this comment.
Fixed in bdef58a. It reads ALP: off when nothing is configured now, and ALP: off {col_c=AlpConfig{vectorSize=1024}} when only some columns have it, so the word null no longer appears either way.
| Support for INT32, INT64 and FIXED_LEN_BYTE_ARRAY added in 2.11. | ||
| */ | ||
| BYTE_STREAM_SPLIT = 9; | ||
|
|
There was a problem hiding this comment.
We should do the copy over of parquet-thrift in a precursor pr
There was a problem hiding this comment.
Done as #3791, which merged yesterday, and I've merged master into this branch. parquet.thrift here is now byte-identical to upstream 2.14.0, so the local ALP = 10 block and the "LOCAL ADDITION" notice are gone and the entry comes straight from parquet-format.
| } | ||
|
|
||
| /** | ||
| * Java writes ALP-encoded floats/doubles using V2 (PARQUET_2_0) data pages and reads them back. |
There was a problem hiding this comment.
Java doc here is not in sync. One of the reasons I'm always pushing on not including details in javadocs because they changed very fast :)
RussellSpitzer
left a comment
RussellSpitzer
left a comment
There was a problem hiding this comment.
This looks good to me now. I have a few remaining nits
We need to update the thrift definition so it doesn't say "local addition"
Java docs need cleanup where they refer to modified thrift or really anything not method contract specific
Skips based on ALP not existing should get pulled from tests too
Maybe we should add in that "withoutALP(column)" method?
An unset ALP property is null now, so toString printed "ALP: null" for most writers, and still did for the default when only some columns had ALP set.
The perl patch it described was removed in 229e0d9, so the comment pointed at something that no longer exists.
ALP is in the Encoding enum now, so it round trips like every other encoding and no longer needs excluding.
Turning ALP off for one column while it is on by default was not expressible, because ColumnProperty treated a null per-column value as unset and fell back to the default. It now checks containsKey, so an explicit null overrides. Every other per-column property either cannot hold a null or already defaults to one, so nothing else changes. Two things had to follow. The schema check was reading "named" as "enabled", so disabling ALP on an int column complained that ALP was enabled on it, and toString rendered a disabled column as null again.
|
@RussellSpitzer thanks for the review and the approval. Three of the four nits are done, and I am still working through the thrift one. Java docs — f7f7273. The one you spotted described the build-time patch that was deleted in 229e0d9, so I dropped the detail rather than updating it. I grepped for anything else mentioning the patch or a local addition and that was the only one left. Skips — the
Thrift — still looking into it. There is a wrinkle in doing it as a precursor PR that I want to understand properly before I propose something, so I will follow up on that thread. Your approval is against e660c6c, so it misses the four commits since. Another look whenever you get a chance would be great. |
|
Opened #3791 as the thrift precursor for this PR. It bumps the inlined One thing worth knowing, since it changes the shape of what was suggested. The bump cannot be thrift only: on a clean master it fails Once #3791 merges I will rebase this one and drop the local |
…a-implementation # Conflicts: # parquet-format-structures/src/main/thrift/parquet.thrift
The thrift bump in apache#3791 brought ALP in from upstream, but the merge also brought back the ALP skip in testEnumEquivalence. Remove it again, along with the stale parquet-testing#100 link in TestInterOpReadAlp.
|
@RussellSpitzer with #3791 merged, I've merged master in and worked through what was left of your nits:
One question from last time is still open: Your approval is against e660c6c, so another look whenever you get a chance would be great. |
Use InterOpTester to fetch alp_extended.zstd.parquet at a pinned changeset, like the other interop tests, instead of reading files from a local directory. Every ALP column is compared bit for bit against its PLAIN counterpart, with spot checks on the special values. Drop the fixture generators too. They only produced files for parquet-testing#100, which apache#119 superseded.
6 participants
cc @julienledem @alamb @emkornfield @prtkgaur
Rationale for this change
Reworks the ALP encoding implementation to address emkornfield's architectural feedback on PR #3390. The original buffered all values in memory and decoded eagerly. This makes the writer incremental (encode per-vector as values arrive) and the reader lazy (decode on demand), matching how other Parquet encodings work.
Builds on Julien Le Dem's original implementation (#3390). File structure, integration points, core math, and interop test infrastructure all come from his work. The rework focused on the internal writer/reader plumbing.
What changes are included in this PR?
Architecture (addressing review feedback):
AlpValuesReaderabstract base class for shared decode logic (float/double readers only implementdecodeBody).Spec compliance:
ceil(n * bitWidth / 8)(reusingBytesUtils.paddedByteCountFromBits).max - minoverstated the span on mixed-sign vectors).ByteBuffer's typed getters on aLITTLE_ENDIAN-ordered buffer.BytePackerinstead of custom bit-packing.num_exceptions.> 32float /> 64double throw).Configuration:
withAlp(),withAlp(AlpConfig),withAlp(columnPath)andwithAlp(columnPath, AlpConfig)onParquetProperties.Builderand the HadoopParquetWriter.Builder, pluswithoutAlp()/withoutAlp(columnPath)to turn it back off globally or for one column.AlpConfigcarries only the vector size (default 1024, validated in its constructor). A column is ALP encoded when it has a config at all.DefaultV1ValuesWriterFactoryandDefaultV2ValuesWriterFactoryso per-column overrides work.BYTE_STREAM_SPLITon a column, or on a non FLOAT/DOUBLE column, is rejected rather than resolved by precedence.Reader null tolerance (bug fix):
AlpValuesReaderwas assertingnum_elements == page.valuesCount, which fails on optional columns with nulls (num_elements is the encoded non-null count, valuesCount is the page row count including nulls). Relaxed tonum_elements <= valuesCount.Integration:
DefaultV1ValuesWriterFactoryandDefaultV2ValuesWriterFactoryas a fallback data-page encoding for FLOAT/DOUBLE.Are these changes tested?
Yes, extensively. The full
parquet-columnmodule (840 tests) and all downstream non-ALP modules (arrow/avro/protobuf/thrift/variant/cli, 908 tests) pass; the parquet-hadoop ALP interop tests pass. Coverage spans:Correctness & spec:
Integration (production paths):
null_countcorrect.Robustness & scale:
ByteBufferAllocatorverifies all off-heap buffers are released across 200 write/reset page cycles.Cross-language verification
The Arrow C++ ALP decoder (apache/arrow#48345) reads every Java-written fixture bit-exact against the canonical
_expect.csvtruth tables. Local verification covers the full {V1, V2} × {vs1024, vs4096} matrix plus the corner-case and extreme-value columns: >1.5M values, 0 mismatches. The shared ALP conformance fixture has landed in parquet-testing (apache/parquet-testing#119, which superseded #100).Interop:
TestInterOpReadAlpreads the shared conformance fixture from parquet-testing (#119) and checks every ALP column bit for bit against its PLAIN counterpart.Are there any user-facing changes?
withAlp(...)onParquetProperties.Builderor the HadoopParquetWriter.Builder, globally or per column, and disable it withwithoutAlp(...).AlpConfig(default 1024).Spec status
ALP is part of parquet-format 2.14.0 (
ALP = 10). The inlinedparquet.thriftwas bumped to 2.14.0 in #3791, so the entry comes from upstream; the build-time perl patch, the local thrift addition and thetestEnumEquivalenceALP skip are all gone from this PR.