Conversation
The reflection based row coder used the varint encoding for int16 and uint16 fields. Java and Python encode the INT16 schema type as 2 big endian bytes, so rows with such fields could not be exchanged with the other SDKs. Fixes apache#40151. Part of apache#39684.
525272f to
15745eb
Compare
|
Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment |
| case reflect.String: | ||
| return typeDecoderFieldReflect{decode: reflectDecodeString}, nil | ||
| case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64: | ||
| case reflect.Int, reflect.Int8, reflect.Int32, reflect.Int64: |
There was a problem hiding this comment.
Int8 (BYTE) likely has similar inconsistency. Consider a same fix here, or could be a follow up
edit: It's fine: BYTE is backed by uint8. See https://github.com/Eliaaazzz/beam/blob/15745eb513417942b6cfe563961edb508804bd8a/sdks/go/pkg/beam/core/runtime/graphx/schema/schema.go#L658
| ## Breaking Changes | ||
|
|
||
| * X behavior was changed ([#X](https://github.com/apache/beam/issues/X)). | ||
| * (Go) The row coder now encodes `int16` and `uint16` struct fields as 2 byte big endian INT16 values, matching the Java and Python SDKs. Rows with such fields written by earlier Go SDK versions decode differently with this version ([#40151](https://github.com/apache/beam/issues/40151)). |
There was a problem hiding this comment.
Consider changing
Rows with such fields written by earlier Go SDK versions decode differently with this version
To something
This is a upgrade incompatible change for streaming pipelines used rows of int16/uint16 types
moer informative to users (decoding is an implementation detail)
|
Assigning reviewers: R: @shunping for label go. This pull request likely touches a core component ("core" label). Please review with scrutiny. Note: If you would like to opt out of this review, comment Available commands:
The PR bot will only process comments in the main thread (not review comments). |
2 participants
Fixes #40151. Part of #39684.
The reflection based row coder encoded
int16anduint16struct fields with the varint encoding. Java (BigEndianShortCoder) and Python (BigEndianShortCoder) encode the INT16 schema type as 2 big endian bytes, so rows with such fields could not be exchanged with the other SDKs. Thebeam:logical_type:timestamp:v1cases instandard_coders.yamldepend on this encoding for precisions below 5.Changes:
coder.EncodeInt16,DecodeInt16,EncodeUint16andDecodeUint16.int16anduint16fields.uint16is stored as an INT16 schema field.int16oruint16fields written by earlier Go SDK versions decode differently.This is the first PR of a series for #39684 (Go SDK portable logical types).
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, commentfixes #<ISSUE NUMBER>instead.CHANGES.mdwith noteworthy changes.See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.