Skip to content

MST/2 T06: durable retention store, GC operation log and mark audit #16

Description

@Ivanbeethoven

Context

The retention graph and collector exist and are proven against an in-memory
store:

  • src/ceres/snapshot/retention.rsRetentionCoordinator + RetentionStore
    trait: de-duplicated (parent, child) edges, root coverage for
    Lease/Pin/Prepare, mark-sweep collection that CASes LIVE → DELETING
    before any reaper runs, NoopReaper as the fail-closed default.
  • src/jupiter/migration/m20260917_000100_add_mst2_retention.rs — tables
    mst2_retention_node, mst2_retention_edge (unique (parent_id, child_id)),
    mst2_retention_root (unique (node_id, root_key)).
  • src/ceres/snapshot/runtime.rs — a resolved view pins its metadata-root page
    and chunk projection as retention roots; release_lease drops the root and
    re-resolving lifts a DELETING node back to LIVE.

Missing: the durable side. The tables exist but nothing reads or writes
them; retention state lives in one process's memory and is lost on restart,
which is exactly when a partially-collected graph is most dangerous.

Scope

  1. PostgresRetentionStore implementing the existing trait
    (node, root_covers, live_incoming, retain, mark_deleting, remove,
    release_root, all_live, children) against the migrated tables, with the
    guarantees the in-memory version gets from its single mutex:
    • retain inserts the node, the unique edges and the root rows in one
      transaction
      ; re-adding an existing edge must not increment anything;
    • mark_deleting is a conditional update (WHERE state = 'LIVE') that
      reports whether it changed a row — this is the CAS the coordinator relies
      on;
    • concurrent retain against a node being collected resolves to exactly one
      of "retained (stays LIVE)" or "already DELETING (caller falls back)"; it
      must never leave a DELETING node with a live incoming edge.
  2. Durable GC operation log + crash replay. Each collection step that
    changes counts (mark, edge removal, node removal, reference decrement) is
    recorded so a crash mid-pass can be replayed without decrementing twice.
    Replaying must be idempotent by construction, not by hope.
  3. Mark audit. A periodic pass comparing the root-reachable set against the
    recorded reference counts; on any mismatch it must stop physical
    reclamation
    and report, rather than deleting on a wrong count (spec 10 §6).
  4. DELETING is terminal. A node marked DELETING must not be re-referenced;
    completing cleanup and then explicitly rebuilding is the only path back.

Acceptance

  • The 12 coordinator rules currently covered in retention.rs unit tests run
    against the Postgres implementation, not only the in-memory one.
  • Crash-replay test: kill the process between mark and remove, restart, replay,
    assert counts are consistent and no node is removed twice.
  • Concurrent test: a collection pass racing a retain for the same node ends in
    one of the two legal outcomes, never in a DELETING node with a live edge.
  • Mark-audit test: corrupt a count deliberately, assert reclamation stops and
    the audit reports (physical deletion is still off by default — see the
    GitRetentionPort issue).
  • GC-01..10 subset that does not depend on the Git-side port.

References

  • Spec 10 §§5–6, spec 16 §6 GC-01..10
  • src/ceres/snapshot/retention.rs (rules to port), the retention migration

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions