Repository navigation
Upload: parquet / hive / Delta sources, and suggested column mapping in the wizard - #262
Merged
Merged
Conversation
…a Delta table A cloud-location upload (gs:// or public s3://) can now be parquet as well as CSV: one .parquet file, a folder read recursively (key=value directories become columns; _/. entries are skipped), or a folder with a _delta_log, read as a Delta table so files an optimize or overwrite replaced are never read twice. process_csv plans parts from the parquet footers (UPLOAD_PARQUET_PART_ROWS, default 1.5M rows): whole row groups, or slices of a row group larger than that, so each process:process_parquet_part stays about a minute. The footers give each part's row count, so generated ids get exact offsets and a part that reads a different count fails. Parts go through the same SchemaProcessor, CSV parts and GCS compose as CSV uploads. Footers are read with one suffix-range request each: pyarrow's GCS filesystem serialized concurrent opens (315 footers of the minnie v1941 synapse export: 151 s, now ~8 s). The inspect endpoint previews parquet sources (columns, first row, files, rows) for the column mapping. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
When a schema is chosen (or the position format toggled), fields with no
column yet get the column whose name matches best; a "Suggest matches"
button runs it again. Suggestions are marked until changed, and never
replace a column the user chose.
columnMatch.js splits names into components (_, -, spaces, camelCase). A
column component matches a field component when equal or a prefix of it
("pos" ~ "position"), worth the shorter one's characters; the score is
2 * matched / (characters in both names), so identical names score 1. A pair
is suggested only when every component of one name matched and it scores at
least 0.4, so pre_x -> pre_pt_position_x (0.44) but not pre_pt_root_id, and
not id_partition -> id (0.31). Plain in-order character matching would not
do: "post_x" spells out a subsequence of "pre_pt_position_x".
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Parquet sources
A cloud-location upload (
gs://or publics3://) can now be parquet as well as CSV:.parquetfile.key=valuedirectories become columns. Entries starting with_or.are skipped._delta_log. Only the table's current files are read, so files replaced by an optimize or overwrite aren't counted twice.How the work is split:
UPLOAD_PARQUET_PART_ROWS). A part is whole row groups, or a slice of a row group larger than the target.SchemaProcessor, CSV parts and GCS compose as CSV uploads.Suggested column mapping (wizard step 2)
_,-, spaces and camelCase. A component matches when it is equal or a prefix (pos~position). The score is2 × matched characters / (total characters).pre_xmaps topre_pt_position_x, butpre_pt_root_idandid_partition→idare not suggested.Tests
🤖 Generated with Claude Code