Skip to content

cache: replace read/dirty with a concurrent hash-trie - #6

Merged
twmb merged 2 commits into
mainfrom
trie-cache
Sep 27, 2026
Merged

twmb merged 2 commits into
mainfrom
trie-cache

Conversation

@twmb

@twmb twmb commented Apr 21, 2026 •

Copy link
Copy Markdown
Owner

This replaces the sync.Map-style read map and locked dirty map with a
concurrent hash-trie modeled on Go's internal HashTrieMap, which also backs
sync.Map since Go 1.24. Loads are lock-free, inserts and deletes lock only the
trie node they change, and Delete unlinks the entry immediately.

Against main (Apple M2, Go 1.27, interleaved runs):

main this PR
Get hit 12.7ns 3.3ns
Get hit, MaxAge 11.5ns 6.9ns
Get hit, MaxIdleAge, one hot key 50.6ns 7.7ns
Get miss, unique keys 560ns, 3 allocs 108ns, 2 allocs
Set existing key, MaxAge 146ns 21ns
Live heap per key, 1M int keys ~119B ~138B

The heap per key is higher: an uncompressed 16-way trie has about 0.36
interior nodes per key. Leaf buckets would bring it to about 108B, but they
need a version-checked lock-free read path, which is not worth ~30B per key.

Behavior changes:

  • Get runs the miss function in the calling goroutine unless it has a stale
    value to return. A panic in an inline miss reaches the caller, and Gets
    waiting on that load run their own load.
  • Gets that collapsed onto a load all return its result, including an error
    that is not cached (MaxErrorAge(0)). On main, each waiter ran its own load.
  • Expiry uses the monotonic clock.
  • TryGet returns an expired value as Stale while its stale window is open,
    whether Get or Set wrote it; main only did this for values written by Set
    or Swap. The window starts when the value expires, and Get, TryGet, and
    Clean agree on it.
  • Expire does nothing to a loading or already expired value, and can no
    longer be lost to a concurrent Set or Swap (it replaces the loading
    instead of writing the old one's expiry; main had the same race).
  • CompareAndSwap and CompareAndDelete only match unexpired, unerrored values.
  • MaxIdleAge skips an extension that moves the expiry by less than 1s (or
    age/64), so an entry can expire up to 1s early.
  • The autoclean goroutine holds its cache weakly: a cache that is never
    stopped is still garbage collected, and the goroutine exits.
    StopAutoClean now waits for the goroutine to exit.
  • NewItem and NewSet honor AutoCleanInterval; main ignored it.
  • Requires Go 1.25: the package uses a generic type alias (1.24) and the
    tests use sync.WaitGroup.Go (1.25). Go 1.24 and 1.25 are both past end of
    support.

The public API is otherwise unchanged. The tests cover 100% of the
statements in cache.go and trie.go, stable across repeated runs. A
linearizability checker runs every operation, Get with its singleflight
rules, and expiry under a frozen test clock, including with forced hash
collisions; it found the Expire race above.

Every remaining allocation, for a Cache[int, int]:

  • New key: a 32B trie entry. Lock-free readers need it on the heap for the
    key's life.
  • New value (miss, refresh, Set, Swap, successful CAS): a 48B loading. Readers
    read values without locks, so a value is never updated in place. sync.Pool
    reuse is unsafe because a reader may still hold the old pointer.
  • Errored load with no stale: a 48B side struct, so successful values do not
    carry 16B for the error.
  • Stale refresh: the loading and its stale in one 96B allocation, plus the
    goroutine's closure.
  • Expire: a 48B copy of the loading, with the expiry set, swapped in so a
    concurrent Set or Swap cannot make the Expire lost.
  • Trie growth: a 160B indirect node, about 0.36 per key.
  • A miss closure that captures variables is heap allocated on every Get,
    including hits: the stale refresh passes it to a goroutine, and Go's escape
    analysis is per function. Avoiding it needs a new API.

Changelog

  • cache.Cache is now backed by a concurrent hash-trie. Loads are lock-free,
    and Get, Set, and Delete are faster and allocate less.
  • Get runs the miss function inline when it has no stale value to return;
    a panic in the miss function now reaches the caller.
  • Expiry uses the monotonic clock; TryGet returns expired values loaded by
    Get as Stale while their stale window is open, as it already did for values
    written by Set.
  • MaxIdleAge extensions under 1s are skipped; the autoclean goroutine no
    longer keeps an unstopped cache alive; NewItem and NewSet honor
    AutoCleanInterval.
  • Requires Go 1.25.

@twmb
twmb force-pushed the trie-cache branch 4 times, most recently from b4b66ce to ef85207 Compare September 27, 2026 05:41
Comment thread cache/cache.go
The sync.Map-style read map and locked dirty map are replaced with a
concurrent hash-trie modeled on Go's internal HashTrieMap. Loads are
lock-free, inserts and deletes lock only the trie node they change, and
Delete unlinks the entry immediately rather than waiting for a promotion.

Get runs the miss function in the calling goroutine unless it has a stale
value to return while the load runs. If an inline miss panics or calls
runtime.Goexit, the panic reaches the caller and Gets waiting on that load
run their own load. Gets that collapsed onto a load all return its result,
including an error that is not cached.

Expiry uses the monotonic clock, so a wall clock step no longer moves
TTLs. Expire does nothing to a loading or already expired value, and never
moves an expiry later. Expire replaces the loading with an expired copy
rather than writing its expiry, so a concurrent Set or Swap can no longer
make it lost. A stale window starts when the value expires, and
TryGet, Get, and Clean agree on it. CompareAndSwap and CompareAndDelete
only match unexpired, unerrored values. MaxIdleAge skips an extension
that would move the expiry by less than 1s (or age/64), so hot keys do
not write their expiry on every hit.

The autoclean goroutine holds its cache through a weak pointer, so a cache
that is never stopped is still garbage collected and its goroutine exits.
StopAutoClean waits for the goroutine to exit. NewItem and NewSet now
honor AutoCleanInterval, which they previously ignored.

A loading holds its error and stale in a side struct that only errored or
stale-carrying loadings allocate, so a Cache[int, int] value takes 48
bytes. Lookups do not make the key escape, Clean does not allocate, Get
with no expiry does not read the clock, and Set.Get no longer allocates
a wrapper closure per call.

The tests cover every statement of the trie and cache. They include the
stdlib sync.Map tests, a property test against a mutex guarded map,
structural checks of the trie, hook-driven tests for specific races, and
a linearizability checker. The checker runs every operation, Get with its
singleflight rules, and expiry under a frozen test clock, including with
forced hash collisions. CI runs them with the race detector on amd64
and arm64, and without it on 386.

This requires Go 1.25: the package uses a generic type alias (1.24) and
the tests use sync.WaitGroup.Go (1.25).
@twmb
twmb marked this pull request as ready for review September 27, 2026 17:20
@twmb
twmb merged commit 40b9fda into main Sep 27, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant