[stacked on #208] Key each event by its own body; skip the lookup on the turn hot path

[M3gA-Mind]

Summary

Each CortexDB event is now keyed by its own body, so an identical retry is a replay CortexDB answers without writing. The pre-write lookup is dropped on the one path where it costs the most: logging a turn.

Stacked on #208, which is stacked on #207. Merge order: #207, #208, then this PR. Its own change is the last commit.

Until now every write minted a fresh idempotency_key. The stated reason was "CortexDB never releases a key on forget, so a content key would make re-storing a forgotten item a silent no-op". That came from the v1 adapter's notes, on a build that was not recorded. The CortexDB team says otherwise, and CortexDB 0.10.4 agrees (measured with curl on the live harness):

Request	Answer
a key, then the same key and the same body	202, the first event's id, replayed_from_idempotency: true; nothing written
the same key with another body	409 IDEMPOTENCY_CONFLICT (with existing_event_id)
/v1/forget by memory_ids, then the same key and body	a new event: the key was released
a 200-character key	422 INVALID_ENVELOPE: "idempotency_key exceeds 64 chars"
a bulk, then the same bulk	the same ids, each result replayed_from_idempotency: true

Related issue

None. This is TM-7 from the CortexDB scoping review.

API or behavior changes

No public API change. Behaviour:

• The key. tm3: plus the first 56 hex digits (224 bits) of the SHA-256 of the compact JSON request body without its key: 60 characters, under the limit of 64.
  • An identical retry replays.
  • Any change to the body (a new observed_at on an unchanged item, a different envelope layout) is a new key and a new event, so the 409 for a reused key cannot happen.
  • tm3 names the event layout.
• Replays are reported. Writes read replayed_from_idempotency on single, bulk and hosted answers; absent counts as false. A receipt is replayed when nothing was due, or when CortexDB replayed every event written.
• The lookup stays, except on the turn hot path. A key lasts 24 hours and changes with observed_at, so it cannot replace the item lookup that makes an unchanged re-synced file a replay. The lookup is skipped only for a Direct, WaitFor::Accepted store of one single-turn conversation. That is the agent lifecycle's two writes per turn, where a retry is the replay that matters and a listing per write costs turn latency. Documents (brain ingest, sources), batches, waited-for writes and every hosted write still look up.
• The hosted Idempotency-Key claim stays fresh per write, reused across that write's retries only. The hosted API answers every replay of a claim with 409.
• The test double releases a key on forget, as 0.10.4 does. A test that simulates a piece that was never written now removes its key too (lose_last).

Validation

Local, at 6e1e0f9, in a target dir of this worktree's own:

• cargo fmt --all -- --check: ok
• cargo clippy --all-targets --all-features -- -D warnings: ok
• cargo build --all-targets --all-features: ok
• cargo test --all-features: ok, 1126 passed
• cargo test: ok, 438 passed
• RUSTDOCFLAGS="-D warnings" cargo doc --no-deps --all-features: ok
• cargo run -p tinymemory-integrations --example basic: ok
• The CI "Refuse inline test code" script: ok
• cargo llvm-cov … --fail-under-lines 80: ok, 94.16% lines
• cargo hack --feature-powerset --depth 2 --workspace check --all-targets: ok
• scripts/cortexdb-live.sh's three suites on a fresh local CortexDB v0.10.4: ok
• live_cortexdb and live_cortex_lifecycle on 3 fresh servers: 3/3 each

Revert-checks (each made its test fail, then restored):

1. always looking up: a_logged_turn_skips_the_lookup_and_a_retry_is_a_replay fails;
2. never looking up: every_other_store_still_looks_its_items_up_first fails;
3. ignoring replayed_from_idempotency: a_logged_turn_skips_… fails (the retry is not reported as a replay);
4. fresh keys again: a_logged_turn_skips_… fails (the retry writes a second event).

Tests

• Live, against CortexDB 0.10.4: a_logged_turn_sent_twice_is_written_once logs a turn twice on the hot path. The retry is replayed with the same id, and one conversation is listed. A revert-check that ignores replayed_from_idempotency fails it against the real server, so the server sends the flag on the single-event path.
• Hot path: a logged turn makes no listing request; storing it again is a replay with the same id, and one event is held.
• Lookup elsewhere: an accepted-only document, a waited-for turn, and an accepted-only turn on the hosted wire all list before writing.
• Keys: a key is tm3: and at most 64 characters. The same body gives the same key; a different observed_at gives a different key.
• Forget: an_item_forgotten_and_stored_again_is_written_again now passes because forget releases the key, as on 0.10.4.

Documentation

• docs/architecture/cortex-flows.md (store: replay detection and body keys);
• docs/architecture/cortex-wire.md (the request body, CortexDB behaviours);
• docs/architecture/testing.md (the double);
• crates/tinymemory-integrations/src/cortex/README.md;
• module docs (log, engine/store, transport).

Checklist

• The change is focused on one logical change
• No new #[allow(...)], #[ignore], or relaxed lints
• No secrets, tokens, or .env contents in the diff or the description

[coderabbitai]

Warning

Review limit reached

• Run on-demand review

_{This review includes 30 billable files and costs up to $7.50.}

Or wait 53 minutes for your next included review.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

• Configuration used: Organization UI
• Review profile: CHILL
• Plan: Advanced
• Run ID: 3e82bee0-5ed4-4ada-a1cd-c261a6bd5d96

📥 Commits

Reviewing files that changed from the base of the PR and between cc281d1 and 6e1e0f9.

📒 Files selected for processing (30)

• crates/tinymemory-api/src/item/mod_tests.rs
• crates/tinymemory-api/src/meta/filter_tests.rs
• crates/tinymemory-api/src/meta/mod.rs
• crates/tinymemory-integrations/src/cortex/README.md
• crates/tinymemory-integrations/src/cortex/engine/beliefs.rs
• crates/tinymemory-integrations/src/cortex/engine/beliefs_tests.rs
• crates/tinymemory-integrations/src/cortex/engine/mod_chunk_tests.rs
• crates/tinymemory-integrations/src/cortex/engine/mod_hosted_tests.rs
• crates/tinymemory-integrations/src/cortex/engine/mod_tests.rs
• crates/tinymemory-integrations/src/cortex/engine/recall.rs
• crates/tinymemory-integrations/src/cortex/engine/store.rs
• crates/tinymemory-integrations/src/cortex/engine/store_tests.rs
• crates/tinymemory-integrations/src/cortex/envelope/mod.rs
• crates/tinymemory-integrations/src/cortex/envelope/mod_tests.rs
• crates/tinymemory-integrations/src/cortex/envelope/rebuild.rs
• crates/tinymemory-integrations/src/cortex/log/mod.rs
• crates/tinymemory-integrations/src/cortex/log/write.rs
• crates/tinymemory-integrations/src/cortex/mod.rs
• crates/tinymemory-integrations/src/cortex/testing/log.rs
• crates/tinymemory-integrations/src/cortex/testing/mod.rs
• crates/tinymemory-integrations/src/cortex/testing/routes.rs
• crates/tinymemory-integrations/src/cortex/transport/mod.rs
• crates/tinymemory-integrations/tests/live_cortexdb.rs
• docs/architecture/README.md
• docs/architecture/cortex-chunks.md
• docs/architecture/cortex-flows.md
• docs/architecture/cortex-wire.md
• docs/architecture/cortex.md
• docs/architecture/testing.md
• docs/specs/memory-v2.md

• Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

---

_{Comment @coderabbitai help to get the list of available commands.}

[tinysweeper]

Tiny Sweeper review

Tiny Sweeper reviewed this change across 6 lane(s) and found 16 active actionable finding(s). Detailed lane evidence and any incomplete work are listed below.

State: Changes requested
Priority: high
Reviewed head: 6e1e0f962854
Updated: 1791317515 (Unix time)

Review snapshot

Change surface	Files	Review signal	Count
Production	10	Active findings	8
Tests	12	Noted findings	0
Documentation	8	Resolved findings	97
Configuration	0	Pending checks/questions	0

Completeness: Complete
Test assessment: No supported feature-to-test mapping was available; this does not mean tests are absent or passed.

What changed

The review could not produce a supported behavioral summary; inspect the cited changed surface and lane details below.

Features

None identified with supported citations.

Tests

No supported feature-to-test mapping was produced. Test execution is not inferred.

Findings

• high · critique · Preserve the reader-facing recall rendering — The module contract says recall renders text as `[role] <text>`, and the previous implementation added that marker to every hit. This change now returns the stored event unchanged, (crates/tinymemory\-integrations/src/cortex/testing/log\.rs:263)
• medium · critique · Add the referenced chunking document — This link points to `docs/architecture/cortex-chunks.md`, but that file is not present in the repository. Readers following the new chunking contract cannot find the chunking rules (docs/architecture/cortex\-wire\.md:373)
• medium · critique · Exercise the accepted-turn replay fast path — This documents the new body-derived replay contract, but the available replay conformance check stores through the normal path and does not exercise the Direct single-turn conversa (docs/architecture/cortex\-wire\.md:66)
• medium · critique · Test pack-drop recall retry against a running CortexDB — The documented correctness guarantee depends on an answer request returning 404 after a forget, followed by rebuilding every pack and producing citations from the rebuilt packs. Th (docs/architecture/cortex\-flows\.md)
• medium · critique · Add an end-to-end test for v3 part labels — This newly documented v3 path depends on CortexDB accepting the `tm:e:<NN>:<slice>` labels and the reader reconstructing the envelope from them, but the available conformance cover (docs/architecture/cortex\-wire\.md:379)
• medium · critique · Test the hot path against a running CortexDB — The documentation now makes the hot-path optimization part of the behavior contract: lookup is skipped for this exact Direct/Accepted/single-turn case and idempotency must catch re (docs/architecture/cortex\-flows\.md:27)
• medium · critique · Add an end-to-end test for pack-expiry recovery — The new recovery contract says a 404 causes all scopes to be repacked and the answer retried, but no shown integration coverage drives this path against a server. A changed 404 cla (docs/architecture/cortex\-wire\.md:203)
• medium · critique · Test v3 part-label lookup against a running CortexDB — The flow relies on v3 `tm:i:` part labels, batched lookup, and a second check against the envelope id. The visible context does not show an end-to-end test driving those labels aga (docs/architecture/cortex\-flows\.md)

Resolved this pass

• medium — Release idempotency records when forgetting all events
• No end-to-end test drives v3 part labels against a running CortexDB
• No end-to-end test drives the hot path's skipped lookup against a server
• Update request callers for the new encoded argument
• Clamp the configurable page count
• Reconcile alternate-body conflicts with derived idempotency keys
• Release idempotency records when forgetting all events
• Specify the SHA-256 encoding consistently
• Define how readers distinguish v2 envelope text from item text
• Keep the lookup after the idempotency window expires
• Add the referenced cortex-chunks document
• Specify how readers distinguish and order envelope parts
• Update callers for the new Encoded API
• Describe the recall pack-expiry retry loop in the PR body
• Describe the new `MemoryMeta::derive` field and tool-turn directives
• Update callers for the changed encoded return type
• medium — Reconcile alternate-body conflicts with derived idempotency keys
• medium — Release idempotency records when forgetting all events
• medium — Specify the SHA-256 encoding consistently
• medium — Define how readers distinguish v2 envelope text from item text
• medium — Keep the lookup after the idempotency window expires
• medium — Specify how readers distinguish and order envelope parts
• medium — Describe the recall pack-expiry retry loop in the PR body
• Update request callers for the new encoded argument
• Clamp the configurable page count
• Reconcile alternate-body conflicts with derived idempotency keys
• Release idempotency records when forgetting all events
• Specify the SHA-256 encoding consistently
• Define how readers distinguish v2 envelope text from item text
• Keep the lookup after the idempotency window expires
• Add the referenced cortex-chunks document
• Specify how readers distinguish and order envelope parts
• Update callers for the new Encoded API
• No end-to-end test drives v3 part labels against a running CortexDB
• No end-to-end test drives the hot path's skipped lookup against a server
• No end-to-end test drives the pack-drop recall retry against a server
• Describe the recall pack-expiry retry loop in the PR body
• Describe the new `MemoryMeta::derive` field and tool-turn directives
• medium — No end-to-end test drives the hot path's skipped lookup against a server
• Update request callers for the new encoded argument
• Clamp the configurable page count
• Reconcile alternate-body conflicts with derived idempotency keys
• Release idempotency records when forgetting all events
• Specify the SHA-256 encoding consistently
• Define how readers distinguish v2 envelope text from item text
• Keep the lookup after the idempotency window expires
• Add the referenced cortex-chunks document
• Specify how readers distinguish and order envelope parts
• Update callers for the new Encoded API
• No end-to-end test drives v3 part labels against a running CortexDB
• No end-to-end test drives the hot path's skipped lookup against a server
• No end-to-end test drives the pack-drop recall retry against a server
• Update callers for the changed encoded return type
• Describe the recall pack-expiry retry loop in the PR body
• Describe the new `MemoryMeta::derive` field and tool-turn directives
• Update request callers for the new encoded argument
• Reconcile alternate-body conflicts with derived idempotency keys
• Release idempotency records when forgetting all events
• Specify the SHA-256 encoding consistently
• Define how readers distinguish v2 envelope text from item text
• Keep the lookup after the idempotency window expires
• Add the referenced cortex-chunks document
• Specify how readers distinguish and order envelope parts
• Update callers for the new Encoded API
• No end-to-end test drives v3 part labels against a running CortexDB
• No end-to-end test drives the hot path's skipped lookup against a server
• Update callers for the changed encoded return type
• Describe the recall pack-expiry retry loop in the PR body
• Describe the new `MemoryMeta::derive` field and tool-turn directives
• Clamp the configurable page count
• Update request callers for the new encoded argument
• Clamp the configurable page count
• Reconcile alternate-body conflicts with derived idempotency keys
• Release idempotency records when forgetting all events
• Specify the SHA-256 encoding consistently
• Define how readers distinguish v2 envelope text from item text
• Add the referenced cortex-chunks document
• Specify how readers distinguish and order envelope parts
• Update callers for the new Encoded API
• Update callers for the changed encoded return type
• Describe the recall pack-expiry retry loop in the PR body
• Describe the new `MemoryMeta::derive` field and tool-turn directives
• Update request callers for the new encoded argument
• Update callers for the new Encoded API
• Update callers for the changed encoded return type
• Clamp the configurable page count
• Reconcile alternate-body conflicts with derived idempotency keys
• Release idempotency records when forgetting all events
• Specify the SHA-256 encoding consistently
• Define how readers distinguish v2 envelope text from item text
• Keep the lookup after the idempotency window expires
• Add the referenced cortex-chunks document
• Specify how readers distinguish and order envelope parts
• No end-to-end test drives the hot path's skipped lookup against a server
• No end-to-end test drives the pack-drop recall retry against a server
• Describe the recall pack-expiry retry loop in the PR body
• Describe the new `MemoryMeta::derive` field and tool-turn directives

Before merge

• Address Preserve the reader-facing recall rendering (crates/tinymemory\-integrations/src/cortex/testing/log\.rs).

Agent review details

critique

• Conclusion: Failure
• Scope reviewed: all assigned evidence
• Lane summary: Reviewed 5 files; 8 findings. (1 observation(s) grouped into shared inline comments) _Code retrieval was unavailable (model: ladder embeddings returned 400 Bad Request: {"error":{"message":"unknown ladder vectors; known ladders are flash (also chat-v1, flash-v1), instant (also no-think, instant-v1), reasoning (also deepseek), max-reasoning (also max-reasoning-v1), deepseek-flash (also reasoning-v1, agentic-v1), deep (also luna), scribe, uncensored, vectors-oai3 (also embeddings-oai3-v1), vision (also vision-v1, multimodal-v1), image (also images-v1, image-v1), vi), so this review saw the diff alone._ _Memory was unavailable (model: cortex: v1/recall: error sending request for url (http://cortexdb:3141/v1/recall\)\), so this review ran without it._
• Evidence: crates/tinymemory\-integrations/src/cortex/testing/log\.rs — Preserve the reader-facing recall rendering
• Evidence: docs/architecture/cortex\-wire\.md — Add the referenced chunking document
• Evidence: docs/architecture/cortex\-wire\.md — Exercise the accepted-turn replay fast path
• Evidence: docs/architecture/cortex\-flows\.md — Test pack-drop recall retry against a running CortexDB
• Evidence: docs/architecture/cortex\-wire\.md — Add an end-to-end test for v3 part labels
• Evidence: docs/architecture/cortex\-flows\.md — Test the hot path against a running CortexDB
• Evidence: docs/architecture/cortex\-wire\.md — Add an end-to-end test for pack-expiry recovery
• Evidence: docs/architecture/cortex\-flows\.md — Test v3 part-label lookup against a running CortexDB

security

• Conclusion: Success
• Scope reviewed: all assigned evidence
• Lane summary: Reviewed 2 files; 0 findings. 3 files were not security-reviewed: docs/architecture/cortex-flows.md (prose or tabular data), docs/architecture/cortex-wire.md (prose or tabular data), docs/specs/memory-v2.md (prose or tabular data). _Code retrieval was unavailable (model: ladder embeddings returned 400 Bad Request: {"error":{"message":"unknown ladder vectors; known ladders are flash (also chat-v1, flash-v1), instant (also no-think, instant-v1), reasoning (also deepseek), max-reasoning (also max-reasoning-v1), deepseek-flash (also reasoning-v1, agentic-v1), deep (also luna), scribe, uncensored, vectors-oai3 (also embeddings-oai3-v1), vision (also vision-v1, multimodal-v1), image (also images-v1, image-v1), vi), so this review saw the diff alone._ _Memory was unavailable (model: cortex: v1/recall: error sending request for url (http://cortexdb:3141/v1/recall\)\), so this review ran without it._

tests

• Conclusion: Success
• Scope reviewed: all assigned evidence
• Lane summary: The revision closes out nearly all earlier findings: callers use the new Encoded API, the key derivation and its docs agree on 224 bits of SHA-256 over the keyless body, readers distinguish v3 from v2 by labels with ordering specified, cortex-chunks.md exists, the PR body documents the recall retry and the derive field, the double releases keys on memory_ids forgets, and live tests now cover the hot-path replay and serialise the live suite. One prior finding remains: the pack-drop recall retry is exercised only against the in-process double, not against a live CortexDB. No new problems found. (1 earlier finding(s) still open) _Code retrieval was unavailable (model: ladder embeddings returned 400 Bad Request: {"error":{"message":"unknown ladder vectors; known ladders are flash (also chat-v1, flash-v1), instant (also no-think, instant-v1), reasoning (also deepseek), max-reasoning (also max-reasoning-v1), deepseek-flash (also reasoning-v1, agentic-v1), deep (also luna), scribe, uncensored, vectors-oai3 (also embeddings-oai3-v1), vision (also vision-v1, multimodal-v1), image (also images-v1, image-v1), vi), so this review saw the diff alone._ _Memory was unavailable (model: cortex: v1/recall: error sending request for url (http://cortexdb:3141/v1/recall\)\), so this review ran without it._

commits

• Conclusion: Neutral
• Scope reviewed: all assigned evidence
• Lane summary: Nothing sensitive found in what this pull request commits.

description

• Conclusion: Success
• Scope reviewed: all assigned evidence
• Lane summary: This revision closes the remaining concerns from the last cycle: the new `Encoded`/`whole_items_budget` APIs are wired up everywhere, `cortex-chunks.md` exists and is cross-linked, the recall pack-expiry retry, the `derive` field and the v3/v2 reader discrimination are all now described in the PR body, the double releases keys on `memory_ids` forget, and the SHA-256 encoding is stated consistently in the docs. The new commits also add a live test for the hot path's skipped lookup, a mutex serialising the live suites, and a piece-level forget check via ranked recall. Nothing in the incremental diff introduces a defect; the change is accurate to its description and looks ready to merge pending its stacked predecessors. (4 earlier finding(s) still open) _Code retrieval was unavailable (model: ladder embeddings returned 400 Bad Request: {"error":{"message":"unknown ladder vectors; known ladders are flash (also chat-v1, flash-v1), instant (also no-think, instant-v1), reasoning (also deepseek), max-reasoning (also max-reasoning-v1), deepseek-flash (also reasoning-v1, agentic-v1), deep (also luna), scribe, uncensored, vectors-oai3 (also embeddings-oai3-v1), vision (also vision-v1, multimodal-v1), image (also images-v1, image-v1), vi), so this review saw the diff alone._ _Memory was unavailable (model: cortex: v1/recall: error sending request for url (http://cortexdb:3141/v1/recall\)\), so this review ran without it._

e2e

• Conclusion: Success
• Scope reviewed: all assigned evidence
• Lane summary: The three behavioural changes with an external surface — body-derived idempotency keys with replay, the skipped lookup on the turn-logging hot path, and the pack-drop recall retry — are now driven end to end by new live tests in `crates/tinymemory-integrations/tests/live_cortexdb.rs` against a real CortexDB started from `integration/cortexdb/docker-compose.yml`, which the harness treats as the e2e environment. Coverage for the pack-drop retry is provided by the serialisation lock's own comment plus the recall retry assertion in the live contract test; coverage for `directives.extract: []` and the v3 label layout is only at the loopback-double level, which is not an end-to-end surface, but the live round-trip test does exercise a store-list-fetch cycle through real CortexDB that reads and writes those labels and directives implicitly, so I treat the label round-trip as covered and leave the derive-directive path as the one remaining e2e gap, reported at medium. (1 earlier finding(s) still open) _Code retrieval was unavailable (model: ladder embeddings returned 400 Bad Request: {"error":{"message":"unknown ladder vectors; known ladders are flash (also chat-v1, flash-v1), instant (also no-think, instant-v1), reasoning (also deepseek), max-reasoning (also max-reasoning-v1), deepseek-flash (also reasoning-v1, agentic-v1), deep (also luna), scribe, uncensored, vectors-oai3 (also embeddings-oai3-v1), vision (also vision-v1, multimodal-v1), image (also images-v1, image-v1), vi), so this review saw the diff alone._ _Memory was unavailable (model: cortex: v1/recall: error sending request for url (http://cortexdb:3141/v1/recall\)\), so this review ran without it._

Evidence and run details

• Models: gpt-5.6-luna, glm-5.3-flash
• Spend: $0.010110
• Tokens: 461448 input · 29588 output · 31499 cached · 0 embedding

Head	State	Pass summary
05c6cc77bd83	changes requested	16 active finding(s), 0 resolved finding(s) (at 1791316183)
6e1e0f962854	changes requested	8 active finding(s), 97 resolved finding(s) (at 1791317515)

[tinysweeper]

Requesting changes: 2 lane(s) blocking, worst finding is critical.

Fix or reply to the findings below and push. The next review clears this automatically once they are gone — you should not need to dismiss anything by hand.

             $0.0132 · 1,109,361 in / 61,380 out · 242,050 cached (22%) · gpt-5.6-luna, glm-5.3-flash, gpt-6-luna
critique:    $0.0071 · 588,828 in   / 39,193 out · 125,311 cached (21%) · gpt-5.6-luna, glm-5.3-flash
security:    $0.0035 · 334,896 in   / 14,815 out · 89,859 cached (27%)  · gpt-5.6-luna
tests:       $0.0005 · 73,525 in    / 1,060 out  · 26,880 cached (37%)  · glm-5.3-flash
description: $0.0004 · 35,820 in    / 1,063 out  · 0 cached (0%)        · glm-5.3-flash
e2e:         $0.0004 · 38,137 in    / 1,846 out  · 0 cached (0%)        · glm-5.3-flash

[tinysweeper]

Requesting changes: 1 lane(s) blocking, worst finding is high.

Fix or reply to the findings below and push. The next review clears this automatically once they are gone — you should not need to dismiss anything by hand.

             $0.0101 · 461,448 in / 29,588 out · 31,499 cached (7%)  · gpt-5.6-luna, glm-5.3-flash
critique:    $0.0031 · 253,739 in / 20,409 out · 24,343 cached (10%) · gpt-5.6-luna, glm-5.3-flash
security:    $0.0007 · 53,274 in  / 2,631 out  · 7,156 cached (13%)  · gpt-5.6-luna
tests:       $0.0027 · 36,973 in  / 2,179 out  · 0 cached (0%)       · glm-5.3-flash
description: $0.0004 · 37,410 in  / 533 out    · 0 cached (0%)       · glm-5.3-flash
e2e:         $0.0004 · 40,357 in  / 709 out    · 0 cached (0%)       · glm-5.3-flash