Skip to content

fix(ctrltest): judge garbage collection from a live read and precondition every write - #7

Merged
graveland merged 2 commits into
mainfrom
graveland/gcsim-uid-precondition
Nov 5, 2026
Merged

graveland merged 2 commits into
mainfrom
graveland/gcsim-uid-precondition

Conversation

@graveland

@graveland graveland commented Nov 5, 2026 •

Copy link
Copy Markdown
Collaborator

Diff: 62% test, 36% code, 2% generated (776 lines changed)

Problem

ctrltest's simulated garbage collector deletes dependents from a List it took at the start of the sweep. It deletes by name, with no preconditions. If an owner and its children are recreated under the same names before the delete lands, the sim deletes the new children. If a listed child is adopted by a live owner in the meantime (same UID), the sim deletes that child too. In multigres-operator's TestThrash/child_CR_thrash, under load, this deleted a recreated Shard's pool-0 Pod and PVCs, leaving two PRIMARY Pods and Terminating PVCs. The real GC cannot do this, so consumers end up chasing failures that exist only in the harness.

Fix

Candidates go through collect, modelled on upstream attemptToDeleteItem:

  • read the dependent live
  • classify each owner from a fresh lookup
  • strip the references to dead or foreground-deleting owners if a live owner remains
  • otherwise delete under the live read's UID and resourceVersion

A conflict triggers a re-read. The delete is retried under the UID alone only for the same incarnation with unchanged owners (upstream deleteObject), so objects under constant writes are still collected. An unresolvable owner stops the attempt with nothing changed. Orphan strips and finalizer removal read fresh and use JSON merge patches under a resourceVersion lock. Finalizer removal also checks the owner's UID and retries on conflict. The orphan finalizer is removed only after every strip has landed.

Tests

gc_stale_test.go steps a sim against envtest from a stale List, with hooks between the sim's read and its write. The cases are a recreated child, adoption before and after the read, a recreated owner, a concurrent orphan strip, write churn, and a genuine orphan, which is still collected. Of 15 guard mutations, 14 fail the tests. The survivor drops the UID from the first delete and is equivalent, since a recreated object always has a new resourceVersion.

Risks

  • Under foreground deletion, a dependent with another live owner now loses the reference instead of being deleted.
  • An unresolvable owner kind now holds a foreground owner, as it would in a real cluster.
  • Each candidate costs extra metadata GETs.
  • blockOwnerDeletion and propagation policy are still not modelled (documented).
  • A second commit tidies go.sum. main's Lint job fails its tidy check without it.

…tion every write

The simulated garbage collector acted on a List taken at the start of
the sweep: it deleted dependents by name with no preconditions, so a
dependent recreated under the same name after the List (new UID), or
the same dependent adopted by a live owner since (same UID), was
deleted anyway. Orphan strips and finalizer removal were strategic
merge patches of the listed copy with no resourceVersion.

Each candidate now goes through collect, which follows upstream's
attemptToDeleteItem: read the dependent's metadata live, classify each
owner from a fresh lookup (solid, dangling, or waiting on foreground
deletion), strip the dangling and waiting references if a solid owner
remains, and otherwise delete under the live read's UID and
resourceVersion. A conflict is followed by one re-read, and the delete
is retried under the UID alone only if the object is the same
incarnation with unchanged owner references, as upstream's deleteObject
does, so an object under constant unrelated writes is still collected.
An owner the sim cannot resolve ends the attempt with nothing done
rather than counting as solid.

Orphan strips and finalizer removal are JSON merge patches under a
resourceVersion lock from a fresh read (strategic merge is refused for
custom resources); finalizer removal checks the owner's UID and retries
on conflict, like upstream's removeFinalizer. The orphan finalizer now
comes off only after every strip has landed, so the comment saying
orphaning always finishes in one sweep is removed, as is the one
claiming strips update the listed objects in place, which they no
longer do.

The new tests drive a hand-stepped sim against envtest from a stale
List, with hooks between the sim's read and its write.

Signed-off-by: Brent Graveland <graveland@supabase.io>
go mod tidy adds checksums for the otel, stdr and gonum modules in the
build graph, which the Lint job's tidy check requires.

Signed-off-by: Brent Graveland <graveland@supabase.io>
@graveland
graveland merged commit b6d7343 into main Nov 5, 2026
11 checks passed
@graveland
graveland deleted the graveland/gcsim-uid-precondition branch November 5, 2026 17:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant