Repository navigation
feat: send Glue VersionId on Hive table updates (GLUE_SEND_VERSION_ID) - #149
Open
jamespfaulkner wants to merge 4 commits into
Open
jamespfaulkner wants to merge 4 commits into
jamespfaulkner wants to merge 4 commits into
Conversation
Add an opt-in GLUE_SEND_VERSION_ID flag (default off) that makes the GlueSync listener read the current Glue VersionId and pass it on UpdateTable requests for non-Iceberg (Hive) tables, so AWS records requestParameters.versionId on the emitted CloudTrail event. Consumers of that event stream can then identify and fetch the pre-update table version for ALTER_TABLE handling. Glue enforces VersionId as a compare-and-swap, so a concurrent writer fails with ConcurrentModificationException. The retry/fallback control flow is extracted into OptimisticUpdateRetry (pure, independently tested): refresh the version and retry up to a bound, then fall back to an unconditional update so the sync never regresses below today's last-write-wins behaviour (surfaced via the version_conflict_fallback outcome tag). When the flag is on, Hive tables are forced to skipArchive=false so the prior version is retained and the carried VersionId resolves via GetTableVersion; an explicit per-table apiary.gluesync.skipArchive property still wins. Iceberg tables are unaffected. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Previously a non-EntityNotFound failure of the getTable read (throttle, timeout, access denied, etc.) added by the versionId path propagated as a sync failure for that event — a new failure surface vs the single-call behaviour before the flag. The read is best-effort, so readVersionId now returns null on any AmazonClientException (other than EntityNotFound, which still propagates so the caller creates the missing table), and the update proceeds unconditionally. Surfaced via a new version_unavailable_fallback outcome tag. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…come retryOnConflict now returns the result of whichever branch ran (Supplier<T>), so each branch returns its own UpdateOutcome directly instead of signalling "version unavailable" through a mutable boolean[] captured in the lambda. Also add an end-to-end test proving that, with GLUE_SEND_VERSION_ID on, an ALTER_TABLE for a table missing in Glue (EntityNotFoundException from the getTable version read) still drives a create via updateOrCreateTable. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds an opt-in
GLUE_SEND_VERSION_IDflag (default off) to the GlueSync listener. When enabled, the listener reads the current GlueVersionId(viaGetTable) and passes it onUpdateTablerequests for non-Iceberg (Hive) tables, so AWS recordsrequestParameters.versionIdon the emitted CloudTrailUpdateTableevent.Why
The emitted
versionIdis the version the table was at immediately before the update. Consumers of the Glue/CloudTrail event stream can use it to deterministically identify and fetch the pre-update table version (viaGetTableVersion) when handlingALTER_TABLEevents, instead of guessing. Today noversionIdis emitted, so that pre-update state can't be resolved reliably.How
GlueTableService.update()— for Hive tables, reads the currentVersionIdand sets it on theUpdateTable. Iceberg tables are skipped (identified bymetadata_location), avoiding the extraGetTableand optimistic-locking overhead.VersionIdas a compare-and-swap, so a concurrent writer fails the write withConcurrentModificationException. The retry/fallback control flow is extracted into a small, independently-testedOptimisticUpdateRetry: refresh the version and retry up to a bound, then fall back to an unconditional update so the sync never regresses below today's last-write-wins behaviour. The fallback is observable via a newversion_conflict_fallbackvalue on theglue_listener_eventoutcometag.versionIdto resolve to fetchable content, the prior version must be archived, so when the flag is on, Hive tables are forced toskipArchive=false. An explicit per-tableapiary.gluesync.skipArchiveproperty still wins; Iceberg tables keepskipArchive=true(avoids table-version explosion).Behaviour is unchanged when the flag is unset (default), and the flag is honoured identically by the listener and the resync CLI.
Testing
OptimisticUpdateRetryTest— unit-tests the retry logic in isolation (success first try, retry-then-succeed, all-conflict → single fallback, non-conflict exception propagation, single-attempt, invalidmaxAttempts).GlueTableServiceTest/GluePartitionServiceTest— versionId set/skipped (Hive vs Iceberg vs disabled), conflict retry + fallback,skipArchiveprecedence, env-var parsing.mvn clean installgreen (146 tests in this module, 0 failures).Docs
README (
GLUE_SEND_VERSION_IDenv var + "Table update VersionId" section + updated SkipArchive precedence) and CHANGELOG updated.🤖 Generated with Claude Code