Skip to content

GH-3672: Isolate Avro usage in parquet-cli - #3812

Open
makarandhinge wants to merge 2 commits into
apache:masterfrom
makarandhinge:parquet-3672-avro-isolation
Open

makarandhinge wants to merge 2 commits into
apache:masterfrom
makarandhinge:parquet-3672-avro-isolation

Conversation

@makarandhinge

Copy link
Copy Markdown

Note: This PR currently addresses the data-reading side of the Avro isolation work. The remaining part of the isolation will be added to this PR in a follow-up change.

#3672

Rationale for this change

The Parquet-native cat, head, and scan paths were unnecessarily going through Avro schema conversion. This could cause valid Parquet files containing INT96 fields or nested LIST structures to fail during schema conversion.

What changes are included in this PR?

This PR isolates Avro usage from the Parquet-native data-reading paths.

  • Parquet files use GroupReadSupport and read the MessageType directly from the Parquet footer.
  • Native Parquet projections are built without converting the schema to Avro.
  • cat, head, and scan use the native Parquet reader for Parquet input.
  • Non-Parquet and Avro-specific paths continue using the existing Avro reader and schema logic.
  • Column selection behavior is preserved, including single-column output.
  • Multi-file scan validation is preserved.
  • Regression tests cover INT96, nested LIST structures, column projections, and multiple input files.

Are these changes tested?

Yes. Regression tests were added for the affected Parquet CLI paths, including INT96, nested LIST structures, column projection, and multi-file scanning.

Are there any user-facing changes?

This is a bug fix for the Parquet data-reading path. Valid Parquet files that previously failed because of unnecessary Avro schema conversion can now be processed through the native Parquet reader.

The remaining Avro isolation work will be added to this PR later.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant