feat(parquet): add ALP encoding for FLOAT and DOUBLE columns by prtkgaur · Pull Request #1289 · apache/arrow-go

prtkgaur · 2026-09-04T17:56:05Z

Co-Author: @arnav.balyan

Supersedes #704 (this PR started from it and addresses the review there).

Rationale for this change

ALP (Adaptive Lossless floating-Point) is the ALP = 10 encoding for FLOAT and DOUBLE columns in parquet-format (in Preview since 2026-08-01). It scales values by powers of ten into integers, then stores them frame-of-reference + bit-packed, keeping values that don't round-trip as exceptions. Arrow C++ already implements it.

What changes are included in this PR?

Encoding (parquet/internal/encoding)

alp_encoding.go: generic alpEncoder[T]/alpDecoder[T] for float32 and float64, following the spec's page layout (7-byte header, offset array, self-describing vectors).
- The encoder buffers one vector at a time and writes a page per FlushValues.
- The decoder decodes lazily, one vector at a time, honours the page's log_vector_size (8–32768), and bounds-checks every size and offset read from the page.
- Exception values keep their exact IEEE 754 bits, NaN payloads included.
alp_scaling.go: the normative decode encoded * 10^f * 10^-e with the spec's literal power-of-ten tables. alpScaling[T] looks the constants up once per vector instead of once per value.
alp_params.go: picks (exponent, factor) by estimated encoded size, using the paper's sampling scheme: 256 samples per vector, a full search over the first 8 vectors, then the 5 best combinations.
Reuses utils.BitWriter/BitReader for bit packing.

Writer/reader integration

parquet.Encodings.ALP, and Encoding_ALP in the generated thrift code.
The column reader only builds an ALP decoder for FLOAT/DOUBLE columns.
Writing is opt-in, as the spec recommends for Preview encodings:
- WithALPEncoding / WithALPEncodingFor / WithALPEncodingPath allow ALP, and WithEncoding*(Encodings.ALP) selects it.
- Selecting ALP without allowing it panics in NewWriterProperties.
- A dictionary takes precedence, as for every other encoding.
WithALPVectorSize / WithALPVectorSizeFor / WithALPVectorSizePath set the vector size: a power of two from 8 to 32768, default 1024, matching the alp_vector_size property of the Arrow C++ implementation.

Are these changes tested?

Yes.

Unit tests for scaling, parameter search, bit packing, every vector size, partial vectors, exceptions (NaN payloads, ±Inf, −0, subnormals, out-of-range values), partial reads and discards, multi-page encoders, and corrupt or truncated pages.
File-level round trips: V1 and V2 data pages, nulls, all-null pages, many pages, and per-column vector sizes. Also a pqarrow round trip.
TestReadALPConformanceFile reads alp_extended.zstd.parquet from apache/parquet-testing, written by Arrow C++ with 32-, 1024- and 4096-value vectors. It bit-compares every ALP column against its PLAIN copy. The parquet-testing submodule is bumped to include that file.
Interop in the other direction, checked manually: Arrow C++ (pyarrow 27.0.0.dev59) read a Go-written file with float/double/nullable columns at vector sizes 8, 1024 and 32768. Every value was bit-identical to the PLAIN columns, NaN payloads included.
Benchmarks: BenchmarkALP{En,De}codingFloat{32,64}, plus batched decoding.

Are there any user-facing changes?

Yes, all opt-in:

parquet.Encodings.ALP
WithALPEncoding* and WithALPVectorSize* writer properties, with the matching getters
DefaultALPEnabled and DefaultALPVectorSize

Reading ALP needs no option.

ALP turns a float into an integer by multiplying it by 10^exponent and then by 10^-factor. This adds that arithmetic and the search that picks the pair for each vector. The search estimates the size of every pair on a sample of 256 values, counting the packed width of the values that convert and the full cost of those that do not, and breaks ties toward the larger exponent as the ALP paper does. After eight vectors it only tries the five pairs those vectors chose most often. The arithmetic follows the reference implementation's rounding and order of multiplication exactly. Rounding decides which values become exceptions, so a writer that differs by one bit writes a different page.

This adds the ALP page format for FLOAT and DOUBLE columns. A page has a seven-byte header, one offset per vector, and vectors of 1024 values. Each vector stores its scaling, a frame of reference, the bit-packed differences from that frame, and the values that did not survive the scaling as exceptions patched in by position. Packing uses the existing BitWriter and BitReader. The differences are computed in unsigned arithmetic, so a DOUBLE vector spanning most of the int64 range packs at 64 bits instead of overflowing. The decoder accepts any vector size a page declares, from 8 to 32768 values, so it reads pages from writers that do not use 1024. It checks every offset, bit width and exception position against the page before reading, and returns an error when the page holds fewer values than the page header claims instead of a short read the column reader would retry forever. ALP takes encoding id 10. The generated Thrift bindings have no checked-in source, so the new value is added to them by hand. Co-authored-by: Prateek Gaur <prateek.gaur@snowflake.com>

parquet-format marks ALP as a preview encoding, and a reader without it cannot open a file that uses it. Writing ALP therefore takes an explicit opt-in, WithALPEncoding for every column or WithALPEncodingFor for one, and NewWriterProperties panics when a column asks for ALP without it, as it does for any unusable encoding. The opt-in permits ALP but does not select it. A column still chooses it with WithEncoding or WithEncodingFor, and needs dictionary encoding off, because a writer that abandons its dictionary falls back to PLAIN. The column reader returns an error for an ALP page on a column that is not FLOAT or DOUBLE, where the decoder traits would otherwise panic. The tests write and read columns of several pages with and without nulls, under both data page versions.

A round trip through one implementation cannot show that it reads what other writers produce. This moves parquet-testing to the commit that adds alp_extended.zstd.parquet, whose ALP columns use vectors of 32, 1024 and 4096 values next to PLAIN copies of the same data, and checks that every ALP value decodes to the bits of its PLAIN copy. A second test writes an Arrow table with ALP columns and reads it back across compression codecs, several pages and row groups, nulls, and values that can only be stored as exceptions.

The exception rate drives what ALP costs, so the benchmarks encode and decode FLOAT and DOUBLE columns at several sizes with no exceptions, one exception in ten values, and nothing but exceptions. A batched DOUBLE decode reads 137 values at a time, so batches end partway through a vector the way column reader calls do.

# Conflicts: # parquet/file/file_reader_test.go

- Look the power-of-ten constants up once per vector (alpScaling) instead of once per value; encode is ~35-40% and decode ~30-44% faster. - Add the frame of reference back in the encoded integer's width, as the spec requires for FLOAT. - Reuse scratch for PutSpaced and the page header. - Add WithALPVectorSize/WithALPVectorSizeFor/WithALPVectorSizePath (power of two, 8-32768, default 1024) and apply it in the float column writers. - Document ALP in the package docs.

- Treat a FLOAT value whose rounded integer leaves int32 as an exception. Rounding a scaled value just under 2^31 could reach 2^31, which readers rebuild in int32 with the sign flipped (or a 33-bit packing width). - Avoid int overflow on 32-bit platforms when counting vectors and bounds-checking vector offsets in the decoder. - List the encodings the data pages actually used in the column chunk metadata when there is no dictionary, so a column that asked for ALP but fell back to PLAIN is not labelled ALP. - Point the docs at per-column ALP selection, make ALPEncodingEnabled report the default like the other getters, and add ALPVectorSize. - Count vector offsets in EstimatedDataEncodedSize; fix stale comments.

- Cover V2 pages with nulls and compression, and the spaced write/read paths, which the existing tests named but did not reach. - Prove pages are ALP via data-page encoding stats, and check the many-pages test really writes several pages. - Merge the float32/float64 copy-paste round trips into one generic table test; drop tests that only restated types or duplicated the shared encoding suite; move reader-error tests to column_reader_test.go on testutils.MockPageReader; gather file-level ALP tests in alp_test.go. - Use checked allocators, and report allocations in the benchmarks.

EstimatedDataEncodedSize counted buffered values at their raw width, so a vector of NaNs or infinities (each stored with a 2-byte position, plus the vector header and offset) was underestimated and pages could overshoot the page size.

A vector can pack every slot at full width and also store exceptions, so sizing the pending vector as all exceptions still undercounted mixed vectors of NaNs and wide integers. Count both.

Copilot

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

3 open issues

SetData trusts the ALP header’s numElements even when it disagrees with the page’s nvals. You… · New
range over an integer expression requires Go 1.22+. If this module still targets Go <= 1.21, this… · New
This panic message interpolates path without quoting/escaping, and also renders… · New

2 files not reviewed

parquet/file/column_writer_types.gen.go: Generated file
parquet/internal/gen-go/parquet/parquet.go: Generated file

What changed in this PR

Adds Adaptive Lossless floating-Point (ALP) encoding support for Parquet FLOAT/DOUBLE, including writer opt-in controls, reader integration, metadata correctness, and extensive tests/benchmarks.

Changes:

Implement ALP encoder/decoder and scaling/parameter selection logic for float32/float64.
Integrate ALP into writer/reader plumbing (properties, typed encoder/decoder selection, column writer vector sizing, metadata encodings list).
Add comprehensive unit, integration, conformance, and benchmark coverage for ALP.

File	Description
parquet/writer_properties.go	Adds ALP opt-in + vector size writer properties, validation, and getters.
parquet/types.go	Exposes `parquet.Encodings.ALP`.
parquet/reader_writer_properties_test.go	Tests ALP opt-in and vector size validation behavior.
parquet/pqarrow/alp_test.go	End-to-end Arrow write/read roundtrip tests using ALP.
parquet/metadata/column_chunk.go	Makes footer encodings reflect actual data page encodings (important for ALP/dict fallback).
parquet/internal/encoding/typed_encoder.go	Wires ALP into float typed encoder/decoder traits.
parquet/internal/encoding/encoding_test.go	Adds ALP to generic encoding roundtrip test suite.
parquet/internal/encoding/alp_scaling.go	Implements normative ALP scaling/round-trip logic.
parquet/internal/encoding/alp_scaling_test.go	Validates scaling arithmetic, exceptions, and power tables.
parquet/internal/encoding/alp_params.go	Implements exponent/factor selection logic (sampling + presets).
parquet/internal/encoding/alp_params_test.go	Tests parameter search optimality and edge cases.
parquet/internal/encoding/alp_encoding.go	Implements ALP page/vector layout, packing, encoder/decoder.
parquet/internal/encoding/alp_encoding_test.go	Extensive correctness, corruption-resistance, and edge-case tests.
parquet/internal/encoding/alp_encoding_benchmark_test.go	Adds ALP encode/decode benchmarks, including batched decoding.
parquet/file/file_reader_test.go	Adds conformance test against parquet-testing `alp_extended` fixture.
parquet/file/column_writer_types.gen.go.tmpl	Propagates per-column ALP vector size into ALP encoders.
parquet/file/column_writer_test.go	Enables ALP opt-in in writer tests and adds ALP required-column coverage.
parquet/file/column_reader_test.go	Adds reader-side rejection/mismatch tests for ALP on non-float and bad headers.
parquet/file/column_reader.go	Rejects ALP early for non-float physical types to avoid trait panics.
parquet/file/alp_test.go	File-level ALP roundtrip, null handling, page versions, vector sizing, and dict-drop behavior tests.
parquet/doc.go	Documents ALP encoding support and opt-in requirements.

_{Review effort: Lite}

Also quote the column path in the ALP opt-in panic message.

Copilot

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

3 open issues

This uses for i := range <int> (integer range loops), which requires Go 1.22+. If the repo’s… · New
The decoder currently trusts the vector offset table without validating that offsets actually point… · New
The panic message for per-column vector sizes concatenates the raw path without quoting/escaping… · New

3 resolved since last review

2 files not reviewed

parquet/file/column_writer_types.gen.go: Generated file
parquet/internal/gen-go/parquet/parquet.go: Generated file

_{Review effort: Lite}

Also quote the column path in the ALP vector size panic message.

Copilot

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

2 open issues

testing.B does not have a Loop() method in Go versions I’m familiar with, so these benchmarks… · New
slices.Sorted is not a standard-library API (at least up to Go versions I’m aware of), so this… · New

3 resolved since last review

2 files not reviewed

parquet/file/column_writer_types.gen.go: Generated file
parquet/internal/gen-go/parquet/parquet.go: Generated file

_{Review effort: Lite}

zeroshade · 2026-11-10T21:20:30Z

+					for b.Loop() {
+						encoder.Put(values)
+						buf, _ := encoder.FlushValues()
+						buf.Release()
+					}


No change needed: testing.B.Loop was added in Go 1.24, and go.mod declares go 1.26.0. The benchmarks compile and run.

zeroshade · 2026-11-10T21:20:31Z

+		// List what the data pages used rather than what was asked for: a
+		// writer whose dictionary was dropped before its first page writes
+		// PLAIN pages whatever encoding the properties name.
+		for _, e := range slices.Sorted(maps.Keys(encStats.DataEncodingStats)) {


No change needed: slices.Sorted (with maps.Keys) has been in the standard library since Go 1.23, and go.mod declares go 1.26.0.

…ded values Offsets must start right after the offset array and each vector must end where the next begins, so a corrupt offset can't decode another vector's values. Discard now reports a page that encodes fewer values than requested, as Decode does.

…e changes Sample a different position in each stride-wide block, so data that repeats with the stride, such as a NaN every 128 values, can't fill the whole sample. When the shortlist does no better than PLAIN, search again in case the data changed scale, backing off while nothing compresses. Make the benchmark fixtures match their exception rates.

zeroshade · 2026-11-10T21:45:27Z

I did an independent review of this and pushed some fixes directly to the branch, so please pull before making further changes:

85c4a970, be4b786d: SetData rejects a header that claims more values than the page holds, and offsets that point into the offset array. Panic messages quote the column path.
ac12ed5c: the decoder now checks the offset chain the spec requires: the first offset equals the offset array size, and each vector ends where the next begins. Before this, a corrupt offset could decode another vector's values without an error. Discard now reports a page that encodes fewer values than requested, as Decode already did.
eee34375: two parameter-choice fixes and a benchmark fix:
- Sampling shifts its position within each block, so data that repeats with the sampling stride can't fill the sample. With 32768 doubles and a NaN every 128 values in one 32768-value vector, the page goes from 327,564 B to 59,928 B.
- When the preset shortlist does no better than PLAIN, the encoder searches again in case the data changed scale, backing off while nothing compresses. Integers followed by four-decimal prices drop from 10,254 B per 1024 values to under 3 KiB. On random doubles the cost is about 1.5% at 1M values.
- The benchmark fixtures now produce the exception rates their names say.

Other things I checked that look good:

Decoding matches the spec's normative rules, including float32 arithmetic for FLOAT, which matches Arrow C++.
The power-of-ten tables equal the decimal literals.
Fuzzing each decoder for 2 minutes found no panics, and the tests pass on linux/386.
A pyarrow nightly reads Go-written ALP files bit-identically, including NaN payloads, -0, infinities, subnormals and nulls.
The footer encodings-list change only removes duplicates and lists the encodings the pages actually used.

prtkgaur force-pushed the alp-go-encoding branch from db9b5c8 to 6ce73f9 Compare September 5, 2026 01:42

prtkgaur force-pushed the alp-go-encoding branch 2 times, most recently from 22e6bde to 44ffbd8 Compare October 4, 2026 02:39

sfc-gh-pgaur and others added 5 commits October 7, 2026 05:01

prtkgaur force-pushed the alp-go-encoding branch from 44ffbd8 to a017cac Compare October 8, 2026 03:33

zeroshade added 2 commits October 29, 2026 12:15

Merge remote-tracking branch 'upstream/main' into alp-1289

d90a3da

# Conflicts: # parquet/file/file_reader_test.go

zeroshade changed the title ~~[WIP] Add support for ALP encoding~~ feat(parquet): add ALP encoding for FLOAT and DOUBLE columns Oct 29, 2026

zeroshade mentioned this pull request Oct 29, 2026

[WIP] Parquet Go ALP implementation support #704

Closed

zeroshade added 5 commits October 29, 2026 13:13

fix(parquet/metadata): list RLE once when a data page used it

1218254

fix(parquet): bound a buffered ALP vector's estimate by its largest size

b7c6e12

A vector can pack every slot at full width and also store exceptions, so sizing the pending vector as all exceptions still undercounted mixed vectors of NaNs and wide integers. Count both.

zeroshade marked this pull request as ready for review October 29, 2026 17:54

zeroshade self-requested a review as a code owner October 29, 2026 17:54

zeroshade requested review from kou, lidavidm and wgtmac October 29, 2026 17:54

Merge branch 'main' into alp-go-encoding

65a0041

zeroshade requested a balanced review from Copilot November 10, 2026 16:41

Copilot AI reviewed Nov 10, 2026

View reviewed changes

Comment thread parquet/internal/encoding/alp_encoding.go Resolved

Comment thread parquet/internal/encoding/alp_params.go Resolved

Comment thread parquet/writer_properties.go Resolved

Copilot stopped reviewing on behalf of zeroshade due to an error November 10, 2026 17:29
The job was not acquired by a runner even after multiple attempts.

fix(parquet): reject ALP pages claiming more values than the page holds

85c4a97

Also quote the column path in the ALP opt-in panic message.

zeroshade requested a balanced review from Copilot November 10, 2026 17:54

Copilot AI reviewed Nov 10, 2026

View reviewed changes

Comment thread parquet/internal/encoding/alp_params.go Resolved

Comment thread parquet/internal/encoding/alp_encoding.go Resolved

Comment thread parquet/writer_properties.go Resolved

fix(parquet): reject ALP vector offsets into the offset array

be4b786

Also quote the column path in the ALP vector size panic message.

zeroshade requested a balanced review from Copilot November 10, 2026 19:00

Copilot stopped reviewing on behalf of zeroshade due to an error November 10, 2026 19:03
The job was not acquired by a runner even after multiple attempts.

Copilot started reviewing on behalf of zeroshade November 10, 2026 19:22

View session

Copilot AI reviewed Nov 10, 2026

View reviewed changes

zeroshade added 2 commits November 10, 2026 16:37

Navigation Menu

Repository navigation

feat(parquet): add ALP encoding for FLOAT and DOUBLE columns - #1289

feat(parquet): add ALP encoding for FLOAT and DOUBLE columns#1289
prtkgaur wants to merge 17 commits into
apache:mainfrom
prtkgaur:alp-go-encoding

prtkgaur commented Sep 4, 2026 •

edited by zeroshade

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

zeroshade Nov 10, 2026

Uh oh!

zeroshade Nov 10, 2026

Uh oh!

zeroshade commented Nov 10, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

prtkgaur commented Sep 4, 2026 • edited by zeroshade Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

Uh oh!

zeroshade Nov 10, 2026

Choose a reason for hiding this comment

Uh oh!

zeroshade Nov 10, 2026

Choose a reason for hiding this comment

Uh oh!

zeroshade commented Nov 10, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

prtkgaur commented Sep 4, 2026 •

edited by zeroshade

Loading