GH-3751: Preserve field repetition when truncating recursive proto fields - #3752
Open
puskarpeter wants to merge 1 commit into
Open
puskarpeter wants to merge 1 commit into
puskarpeter wants to merge 1 commit into
Conversation
puskarpeter
force-pushed
the
recursion-truncation-repetition
branch
from
September 4, 2026 06:41
162bacc to
f9cf76f
Compare
…oto fields The maxRecursion truncation (PARQUET-1711) replaced recursive fields with a hardcoded optional binary, while ProtoWriteSupport still wraps repeated/map fields' writers in ArrayWriter/RepeatedWriter/MapWriter. Writing data that nests deeper than maxRecursion through a repeated recursive field crashed with a ClassCastException in parquet-specs mode and corrupted the file in the old style (inconsistent repetition levels: reads fail with ParquetDecodingException or return a wrong tree). A map field exhausting the recursion budget collapsed entirely - keys included - into one binary, and writing data through it crashed the same way. Truncate to proto bytes preserving the field's shape instead, reusing the terminate-as-bytes path introduced for empty message types (apacheGH-2142): LIST-wrapped binary for repeated fields in specs mode, repeated binary in the old style, and the MAP structure kept with the recursive value truncated inside key_value (the map branch now runs before the recursion check). Truncated optional fields are unchanged; proto2 required fields now keep their required repetition. Each truncated cell round-trips as the serialized subtree, and with the binary-to-message read path from apacheGH-2142 the truncated messages read back losslessly through ProtoParquetReader. Signed-off-by: Puškár, Peter <peter.puskar@firma.seznam.cz>
puskarpeter
force-pushed
the
recursion-truncation-repetition
branch
from
September 23, 2026 15:38
f9cf76f to
8eaa12b
Compare
puskarpeter
marked this pull request as ready for review
September 23, 2026 15:38
Contributor
Author
|
@wgtmac Now that #3750 is merged I rebased this one on master and marked it ready. It is the follow-up mentioned in #3751: the same terminate-as-proto-bytes path, applied to recursion truncation so repeated and |
wgtmac
approved these changes
Sep 23, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale for this change
PARQUET-1711 truncates recursive proto fields at
parquet.proto.maxRecursiondepth by replacingthem with the serialized proto bytes, but hardcodes the replacement column as
optional binary,ignoring the field's actual repetition.
ProtoWriteSupportstill wraps repeated fields' writers inArrayWriter/RepeatedWriterand map fields inMapWriter, which emit record structure theoptional binarycolumn cannot hold. As a result (see #3751):maxRecursionthrough a repeatedrecursive field crashes with
ClassCastException: PrimitiveColumnIO cannot be cast to GroupColumnIO— a data-dependent failure that passes schema creation and shallow rows;repetition levels and corrupts the file — depending on the data, reading it back either
fails with
ParquetDecodingExceptionor silently returns a wrong tree (elements lost orattached to phantom duplicate nodes);
google.protobuf.Structmapsreached through
list_valuebranches), the whole MAP — including its keys — collapses into onebinary in the schema, and writing data through it crashes with the same
ClassCastException.The existing mock-based tests (
ProtoWriteSupportTest.testRepeatedRecursion/testMapRecursion)never validate against a real
MessageColumnIO, which is why the mismatch went unnoticed.What changes are included in this PR?
ProtoSchemaConverter.addMessageFieldkeeps the field's shape when truncating, reusing theterminate-as-proto-bytes path introduced for empty message types in #3750:
addRepeatedPrimitive;builder.primitive(BINARY, getRepetition(descriptor))—repeated binaryin the oldstyle; truncated optional fields stay
optional binary, byte-for-byte identical to before(proto2
requiredfields in a recursion cycle now keep theirrequiredrepetition instead ofbeing forced
optional);(typed key) is always preserved and a recursive value type is truncated to
optional binaryinside
key_valuewhenaddMapFieldrecurses into the value field — same recursion budget,applied at the level where the recursion actually is.
The writer side needs no further changes: the
getContentTypecheck from #3750 alreadyselects
BinaryWriterbehind LIST/MAP wrappers, so the existingArrayWriter/RepeatedWriter/MapWriterwrapping then lines up with the schema. The read sideis likewise covered by the binary-to-message converter from #3750.
This also makes the "Message fields stored as proto bytes" section of the parquet-protobuf README
(added in #3750) accurate for the recursion case: it describes the truncated column as keeping the
field's repetition, which until now only held for empty message types.
Are these changes tested?
Yes. New
ProtoRecursionTruncationTest(6 tests) writes recursive data deeper thanmaxRecursionthrough the real write path (
ProtoParquetWriter→MessageColumnIO) and reads it back:Trees.WideTree), specs-compliant and old style — previously theClassCastExceptionand the file-corrupting write, respectively; now every element at thetruncation depth round-trips as the serialized subtree;
google.protobuf.Structbehindlist_value) —previously the whole-MAP collapse plus the same
ClassCastExceptionwhen data reached it; nowkeys stay typed and queryable, values round-trip as serialized protos;
Structpath and optional recursion (Trees.BinaryTree) asregression guards for the shapes that already worked;
ProtoParquetReaderround trip: aWideTreedeeper thanmaxRecursion(both modes) and amap-recursive
Structread back equal to the original messages — with the binary-to-messageread path from GH-2142: Write fields of empty message types as proto bytes #3750, truncation is now lossless end to end.
Expected-schema fixtures were regenerated for the new truncation shape:
WideTree.par,Value.par,Struct.par, the inline schemas inProtoSchemaConverterTest, and thetestDeepRecursionStruct fan-out series (now2n+5— a truncated map keeps its key column). Thefull parquet-protobuf suite passes.
Are there any user-facing changes?
Schemas containing repeated or map recursive fields change shape at the truncation depth
(LIST-of-binary / MAP-with-binary-value instead of a single
optional binary) — but writing morethan one element at that depth previously crashed (specs mode) or corrupted the file (old style),
so no valid existing files carry meaningful multi-element data in the old shape. Truncated
optional fields are unchanged; proto2
requiredfields in a recursion cycle now map torequired binary(previously forcedoptional). Data past the truncation depth now round-trips losslessly:each binary cell is the serialized subtree, recoverable with
X.parseFrom(bytes).Closes #3751