MatrixCache

matrixcache is a standalone, Rust-native multi-tier cache library extracted from TemporalStore and reusable by any Rust service. It manages a hot in-memory (DRAM) tier, a persistent-memory-like resident tier, and an SSD tier, with admission control, cross-tier eviction, read-through refill, pinned handles, invalidation, and asynchronous writeback with backpressure accounting.

Features

• Multi-tier placement across DRAM, a PMEM-like resident tier, and an SSD tier.
• Admission policy, cross-tier eviction, and read-through refill.
• Pinned (no-promotion) handles and pinned-acquire APIs.
• Shared-buffer reads for retrieval and scan paths that can consume cached bytes without copying them.
• Asynchronous writeback with backpressure counters and cache-pressure helpers.
• Persistent-tier auto-recovery and restart refill.
• Latency and tier metrics for observability.

SSD backend

RocksDB is the default SSD key-value backend, enabled by the rocksdb-ssd feature (on by default), so cargo build and cargo test exercise the RocksDB path. SSD writes use RocksDB as the block authority and do not double-write raw block shadow files. A small file-backed compatibility store is available with --no-default-features for lightweight local diagnostics; it is not intended as a production backend.

The default feature set compiles RocksDB from source, which requires a native build toolchain plus clang/libclang and cmake:

# Ubuntu / Debian
sudo apt-get install -y build-essential pkg-config libssl-dev clang libclang-dev cmake

Usage

cargo build
cargo test
cargo test --no-default-features       # file-backed compatibility store
cargo run --release --example rocksdb_backend_bench -- --iterations 5000 --json-output /tmp/matrixcache-rocksdb-backend.json --emit-operator-log --require-passed
cargo run --release --no-default-features --example metrics_server

The rocksdb_backend_bench example drives the multi-tier cache against the RocksDB SSD backend and prints a JSON report (backend, tier evictions, resident hot key count, cold SSD refills, PMEM soak activity, pressure and replacement-soak status, p99 latency, average latency, and QPS) that is useful as local performance/behavior evidence. Each archive also includes an operator_log object with the compact fields most useful in build logs and scale dashboards: pass/fail, QPS, average, p95 and p99 latency, tier evictions, refills, and write-back backpressure. Validate archived reports before publishing or comparing them. The validator checks report shape, cache evidence counters, QPS math, average latency math, and that operator_log agrees with the detailed report:

tools/validate_backend_report.py /tmp/matrixcache-rocksdb-backend.json --expect-backend rocksdb --min-iterations 5000 --min-replacement-soak-iterations 5000 --min-cold-ssd-refills 1 --min-memory-evictions 1 --min-pmem-evictions 1 --min-disk-fills 1 --min-async-writeback-backpressure 1 --max-refill-failures 0
tools/emit_backend_operator_log.py /tmp/matrixcache-rocksdb-backend.json

Compare a new backend archive with a known-good run before accepting a cache optimization as a scale improvement. The comparator checks p95/p99 latency, average latency, QPS, replacement-loop max latency, and required cache evidence counters:

tools/compare_backend_reports.py /tmp/matrixcache-rocksdb-backend-baseline.json /tmp/matrixcache-rocksdb-backend.json --max-hot-get-p95-regression 1.30 --max-cold-refill-p95-regression 1.45 --max-hot-get-p99-regression 1.35 --max-cold-refill-p99-regression 1.50 --max-hot-get-avg-regression 1.35 --max-cold-refill-avg-regression 1.50 --min-hot-get-qps-ratio 0.80 --min-cold-refill-qps-ratio 0.75 --min-counter-ratio 0.90

Durability

Each durable tier decides separately whether a write also survives the machine losing power, and the two default differently. Both are set on CacheOptions.

option	default	what the default means
ssd_block_durability	true	every SSD block write is flushed, and so is the directory entry after the rename
pmem_block_durability	false	the persistent tier writes and renames, and flushes neither

A block is always whole either way, because it arrives by rename -- no reader ever sees a torn block. What the setting decides is whether a block the cache believed it had written is still there after the machine stops abruptly.

The persistent tier's name invites the opposite assumption, so it is worth stating plainly: real persistent memory is durable without being flushed, and this tier is files standing in for it. Files are not.

Flushing is most of what a block write costs. Measured on one machine, 500 puts of 64-byte values:

	us/put	fsync calls
SSD tier flushed	~7,600	1,000
SSD tier not flushed	~420	0
persistent tier flushed	~5,500	872, for 436 blocks
persistent tier not flushed	~330	0

Those ratios are one machine's virtual disk and will be smaller on real flash; the call counts will not be. examples/manifest_append_cost.rs takes both settings as arguments, so the numbers can be re-measured rather than believed.

Choosing. On a tier that is purely a cache, a block lost to a crash is a miss, and not flushing is the cheaper trade. It stops being a miss if something recovers the tier and expects it to be complete: with auto_recover_on_start set, recovery restores a tier with holes in it and no way to know which entries used to be there. CacheOptions::validate() reports that combination.

auto_recover_on_start is false by default, so a default cache does not read its SSD tier back at all -- and is paying for durability it never uses.

Checking a configuration

CacheOptions::validate() reads a configuration back and says what the cache will actually do where that differs from what was asked for: a replacement policy name nobody offers, a tier given a size and no path, recovery from a tier that was never flushed, or a shard count that makes a tier refuse values it has room for. MultiLayerCache::try_with_options refuses the findings that mean the cache cannot do its job and starts anyway on the rest.

Choosing a shard count

ShardedMultiLayerCache::with_options(options, shards) takes a count and nothing here used to say what to pass. Sixteen, unless you have measured otherwise on your own workload -- and the reason is not the one most people reach for.

Ratios rather than absolute times below, because the times move by about 30% between a busy machine and an idle one while the ratios do not. All of it comes from examples/shard_count_bench.rs, which prints the table and refuses to pass if the findings stop holding.

Sharding does not buy you concurrent reads. A memory hit is served under a shared lock, so readers already proceed together at one shard: four readers cost less per operation at one shard than a single reader does. Adding shards then makes reads slower, not faster -- 1,897ns at one shard against 3,189ns at a thousand -- because each shard's own tables are smaller but the work per lookup is not.

Sharding buys you concurrent writes. A write takes the exclusive lock, so at one shard four writers queue: they cost about 3x what a single writer costs, where four perfectly serialised would be 4x. Sixteen shards is between 4x and 10x cheaper per write than one for four writers -- 4.4x on a shared CI runner, 9.7x on an idle development box. If your workload only reads, a shard count buys you nothing; if it writes concurrently, this is the whole reason the type exists.

It costs you nothing in hit rate. A sharded cache divides its byte budget by its shard count, so the shard holding the hot keys might be expected to run out while the others sit idle. It does not happen, because the keys divide in the same proportion as the capacity: 75.00% at one shard and at 256, and 74.98% at 1,024, where sixteen entries a shard is small enough for the unevenness of the draw to show.

Past about 64 shards, writes can start evicting each other. Under the default replacement policy an entry's hotness decides this: a write starts at a small hotness set by its block kind, every read adds to it, and eviction gives up the lowest. Reading a resident therefore makes it expensive to evict, and a freshly written key sits near the bottom of the order. In a large shard that is harmless -- there is always something colder. In a small one the low-hotness population is mostly the write set itself, so admitting one member evicts another and nearly every write evicts: measured at 0.04 evictions per write or fewer up to 64 shards, and 0.93 at 256, with the write cost rising by more than ten times. A write-heavy cache that is also read-hot should stay well below that.

So: sixteen for a mixed workload, fewer if you only read, and measure before going past sixty-four.

Sixteen is for this size of cache; the rule is entries per shard

One number sets two things: how many locks writers can spread across, and how finely the byte budget is divided. They pull in opposite directions.

• More shards, less queueing. Four writers at one shard cost 3.1x to 3.9x what a single writer costs -- four threads taking turns on one lock. At sixteen shards four writers cost less per write than one writer does.
• More shards, a smaller budget each. Reading a resident raises its hotness and eviction gives up the lowest, so a read-heavy phase leaves a freshly written key near the bottom of the order. If a shard is small enough, what is near the bottom is mostly the write set, and the writes evict each other.

Neither tier policy helps, but the structure does. Driven on the sweep's own write arms at 256 shards, varying only the DRAM policy, then driving ReplacementSlru directly on the same shape -- same capacity, same 256 segments, same 64 entries each, same key space and read volume:

	hit rate	evictions per write
weightedhotnesslru (tier)	82.77%	0.996
slru (tier)	82.77%	0.996
ReplacementSlru (store)	82.59%	0.144

Both tier policies score entries over one access order, so neither reserves anything for an arrival: when a shard is oversubscribed, every admission evicts. ReplacementSlru holds its hot list to a share of each segment's budget, so a recently written set sits in room that reading cannot take, and rewriting it mostly updates in place instead of displacing something. Seven times fewer evictions, with the hit rate unchanged.

Which half of that structure does the work is worth knowing, because the two cost very different amounts to reproduce. Turning the store's maintainer off leaves the placement rule -- a rewrite re-attaching at the hot end -- and nothing else:

	hit rate	evictions per write
hot and warm held to their shares	82.59%	0.144
placement only, no shares	82.59%	1.000

Placement alone buys nothing: 1.000, which is what both tier policies already do. The whole of it is the shares. That is also why set_insertion_point_spec, which is the tier's own placement rule, leaves the rate at 1.000 at every setting -- it is the half that does not matter.

That store is not what a tier selects, so this is not a knob available to a cache today -- but it does mean the structure of the policy is a lever here and the choice between the two tier policies is not. Scale the shard count for the cache you have, and read the entries-per-shard table below rather than reaching for slru.

None of that is the probationary-and-protected scheme the slru policy applies. That policy does hold a read-twice entry above every entry below it, but it is not the default and is not what any number here was measured under; ReplacementSlru is a third thing again, a standalone store with its own hot, warm and cold lists.

The second effect is not about the shard count. It is about how many entries a shard still holds, which is the count divided into the capacity:

entries per shard	evictions per write
16,384	0.000
1,024	0.000
256	0.045
64	0.927
16	0.837

So sixteen is not a constant to carry elsewhere -- it is 16,384 entries divided by a thousand. The sweep's cache holds 16,384 entries, so 256 shards leaves 64 each and the writes thrash; a cache ten times the size at the same count would leave 640 each and behave like the 256-entry row. Scale the count with the capacity, and leave a shard enough entries that the write set and the entries a read-heavy phase has protected can both sit in it: about a thousand was comfortable here, sixty-four was not.

Spend shards on that headroom rather than on locks you do not need. The locking half saturates early and is the cheaper half to satisfy -- sixteen to 256 shards buys about 1.2x for four writers, because sixteen locks already make a collision rare at that thread count. (Four writers is as far as the measurement goes here; the lock half would matter more with far more writers, and this machine cannot resolve that.) Measure on your own size with examples/shard_count_bench.rs.

Placing keys on a cluster

One cache picks the shard for a key with hash(key) % shard_count, which holds up because a shard count is fixed for the life of the cache. Across machines neither half survives. The node count changes while data is live, and % n reassigns nearly every key when it does; and the hash behind it, DefaultHasher, is documented as changing between releases, so two nodes on different builds would not agree about where anything lives.

CacheClusterTopology answers the same question with a hash ring:

use matrixcache::{CacheClusterTopology, CacheKey, CacheNodeState};

let mut cluster = CacheClusterTopology::new();
cluster.add_nodes([("cache-a", 1), ("cache-b", 1), ("cache-big", 4)])?;

let key = CacheKey::string(0, "greeting");
let owner = cluster.owner(&key);              // where it lives
let copies = cluster.owners(&key, 3);         // and its two backups, elsewhere

cluster.set_node_state("cache-b", CacheNodeState::Down);   // its keys move on
cluster.set_node_state("cache-b", CacheNodeState::Live);   // and come back

What it costs when membership changes, over 100,000 keys and a ninth node joining eight:

	keys reassigned
hash % node_count	88.9%
this ring	12.1%

An ideal reassignment is 1/9, or 11.1%. Every key that moves, moves to the new node: none is shuffled between two nodes that were both already there.

Weight is relative capacity -- cache-big above takes four ordinary nodes' worth of the key space. A node marked Down keeps its membership and owns nothing, so its keys pass to the next node on the ring and every other key stays put; marking it Live again restores exactly what it had.

At 1,000 nodes and 200,000 keys the busiest node holds 1.40x its share and the quietest 0.70x, most of which is the sampling noise of 200 keys per node rather than the ring.

What a change costs. The ring says where a key belongs. handoffs_to says what has to move for that to be true:

let plan = before.handoffs_to(&after);
for handoff in &plan {
    // handoff.from streams every key it holds whose route hash falls in one
    // of handoff.ranges to handoff.to.
}

Ownership only changes at a ring point, so the two memberships' points laid together cut the hash space into stretches each membership agrees with itself on; the stretches whose owner differs are the work. Without this a node that gains keys gains misses instead, and serves nothing until reads have refilled it from underneath.

Nothing here moves data. It says what would have to.

Failure domains. Three copies on three nodes in one rack are one power supply away from being no copies. Give a node the thing it fails together with -- a rack, a power domain, an availability zone -- and owners places a key's copies in different ones:

let mut cluster = CacheClusterTopology::new();
cluster.add_nodes_in_zones([
    ("cache-a", 1, "rack-1"),
    ("cache-b", 1, "rack-2"),
    ("cache-c", 1, "rack-3"),
])?;

let copies = cluster.owners(&key, 3);        // one per rack
cluster.failure_domain_count();              // 3: the most copies that can be kept apart

A node added without a zone is a domain of its own, so saying nothing spreads copies as widely as the cluster allows and a wrong zone is the only way to claim a separation that is not there. When more copies are asked for than there are domains, the ones that cannot be separated are still placed, on distinct nodes -- a third copy in a rack already used is worth less than a third copy elsewhere and more than no third copy, and failure_domain_count is how a caller knows which it is getting.

Measuring

The examples are measurements rather than demonstrations. Each prints a table and says what the number means:

• read_path_cost -- where the time goes on a memory-tier hit, and what the promotion bookkeeping costs
• hit_concurrency_bench -- how reads and zero-copy handles scale with threads
• eviction_bench, steady_state_put_bench -- eviction cost at capacity, and whether it grows with the cache
• manifest_append_cost -- the write path's syscalls, with the durability settings above
• rocksdb_backend_bench -- multi-tier DRAM/PMEM/RocksDB-SSD pressure, cold read refill, restart refill, replacement soak, and writeback backpressure. Add --json-output <path> to archive the report and --require-passed to fail the process when the cache contract is not satisfied. Validate archived reports with tools/validate_backend_report.py; CI uses the same validator against the lightweight --no-default-features compatibility backend, while production evidence should use the default RocksDB backend. Use tools/emit_backend_operator_log.py <report.json> to print the compact log-friendly summary.
• scan_resistance_bench, admission_filter_bench -- what the admission policy is worth against a scan
• soak -- long-running memory-pressure and latency stability. Add --json to append a machine-readable report to stdout, or --json-output <path> to archive the same report as a standalone file for Grafana/comparison scripts. Add --require-passed when CI or a scale script should fail the process on a missed memory, hit-rate, or p99 latency gate. Use --duration-seconds and --sample-seconds for short validation runs. The JSON latency section includes average, p50, p95, p99, and max estimates from the same histogram buckets exported to Prometheus. Validate an archived report with tools/validate_soak_report.py; compare it to a baseline with tools/compare_soak_reports.py.
• metrics_server -- serves Prometheus text metrics for Grafana. Import docs/grafana/matrixcache-dashboard.json and see docs/grafana.md for a local scrape setup. The exporter includes direct p50/p95/p99 latency gauges alongside Prometheus histograms.
• batch_write_cost -- bounded sharded write/control-path benchmark for put_batch, pinned insert/release, and acquire/release. It compares colocated batches against fanout batches and prints the sharded batch counters used by the Grafana/Prometheus path.
• batch_read_cost -- bounded read/retrieval benchmark for copied reads, shared-buffer reads, no-promotion scans, pinned acquire reads, and sharded colocated reads. Add --json-output <path> to archive a machine-readable report, then validate it with tools/validate_batch_read_report.py.
• batch_concurrency_bench -- concurrent batch-read throughput benchmark. Add --json-output <path> to archive scale evidence and validate it with tools/validate_batch_concurrency_report.py.
• hit_concurrency_bench -- concurrent memory-hit throughput benchmark. Add --json-output <path> to archive scale evidence and validate it with tools/validate_hit_concurrency_report.py.
• tools/run_scale_reports.py -- runs the read/retrieval scale benchmarks above, validates each JSON report, and writes a manifest that points to the archived reports. Add --markdown-output <path> when the same run should also leave an operator/Grafana-friendly summary. Validate the resulting manifest with tools/validate_scale_report_manifest.py.
• tools/compare_scale_reports.py -- compares two scale report manifests and fails on configured latency or throughput regressions. Add --json-output <path> or --markdown-output <path> to archive the comparison. Add --emit-operator-log to print a compact logfmt line for build logs and Grafana annotations. Validate archived comparison JSON with tools/validate_scale_comparison_report.py.

Benchmarks that report a ratio measure both sides inside one pass and print the spread across passes. A run whose spread is wide measured the machine rather than the cache, and the number should be discarded rather than quoted.

Minimum Supported Rust Version

MSRV is 1.88, set by the rocksdb dependency behind the default rocksdb-ssd feature.

Contributing, security, and license

See CONTRIBUTING.md, CODE_OF_CONDUCT.md, SECURITY.md, and CHANGELOG.md. Licensed under the Apache License, Version 2.0 (LICENSE).

Product and crate names are trademarks; see TRADEMARKS.md.

Third-party dependency licenses and attributions are listed in THIRD-PARTY-NOTICES.md.