Repository navigation
fix(kg): project-namespaced pk primary key; scope observation/embedding paths per project - #17
Conversation
…ng paths per project The Entity/Observation primary key was global `id` alone, but every clear/delete and most reads are scoped by `project_id`. When two projects shared a DB file, a prior project's rows survived the project-scoped clear and collided with the current run's identically-id'd entities on CREATE, which were then silently dropped — cross-project data loss, surfaced as the "duplicated primary key" / "COPY FROM JSON unavailable" warning flood. Relations bled too, because the CreateRelation endpoint MATCH was project-blind. Implements ADR docs/kg-entity-key-fix-design.md (Option A1): - Add `pk STRING PRIMARY KEY = project_id <0x1F> id` to Entity and Observation via a single central helper (src/kglib/pk.go); `id` stays a plain, human-facing, journal-stable column. Separator is 0x1F (Unit Separator), not NUL: cgo truncates C strings at NUL, which would collapse every pk to its project_id. - Qualify endpoint/lookup MATCHes by project: CreateRelation matches on pk (closes the cross-project bleed); CreateObservation binds its parent by pk; GetEntity matches on pk with an id+project_id fallback for un-migrated DBs. - Scope observation READ paths (GetTopObservations, batchGetObservations) through the parent entity's project, matching GetObservations. - Scope the embeddings write/invalidation path by project (SetEmbedding now takes projectID and matches on pk; removed the project-blind entityProjectID lookup). - Switch entity writes CREATE -> MERGE on pk (idempotent; ON CREATE populates the full row, ON MATCH refreshes only name/type/updated_at). - Migration: bump FormatStamp.SchemaVersion to 2 so CheckFormat trips the existing archiveIfMismatch rebuild + journal replay (Kuzu v0.11.3 has no ADD PRIMARY KEY). The journal is the only copy of hand-writes — the old DB is archived, never deleted. - Federated cross-graph joins continue to key on local id/name, never pk. Tests: cross-project shared-id retention, zero-warning re-index, observation non-bleed across projects, and schema-version rebuild + journal replay. Co-Authored-By: Claude <noreply@anthropic.com>
There was a problem hiding this comment.
Review Summary
- Files reviewed:
docs/kg-entity-key-fix-design.md,src/kglib/pk.go,src/kglib/schema.go,src/kglib/entity.go,src/kglib/observation.go,src/kglib/relation.go,src/kglib/embeddings.go,src/kglib/search.go,src/kglib/store.go,src/kglib/format.go,src/kg/migrate.go,src/kg/internal/knowledge/indexer.go,src/kg/internal/knowledge/log_indexer.go,src/kg/internal/knowledge/indexer_pdf.go,src/kg/internal/knowledge/store_wrappers.go,src/kg/internal/knowledge/indexer_crossproject_test.go,src/kg/personal.go,src/kg/helpers.go, plus test files touched by the diff. - Overall verdict: REQUEST CHANGES level content, but per the mapping rule (Major, no Critical) the required verdict is comment.
The core fix is well-executed: the pk = project_id + 0x1F + id namespacing is applied consistently through every entity/observation write path (CreateEntity, createEntityWithID, CreateObservation, SetEmbedding, batchCreateEntities, bulkLoadExecEntities, bulkLoadExecObservations, bulkLoadRelations), the write-open gate correctly refuses pre-pk databases, GetEntity degrades gracefully for legacy read-only opens, and the schema-version stamp correctly forces archive+rebuild. I traced every call site of the changed signatures (SetEmbedding, batchGetObservations) and found no stale callers. One real gap remains in the same file the PR fixes.
Major Issues (should fix)
src/kglib/relation.go:69-73(GetRelations),:104-149(DeleteRelation),:151-168(TraverseRelations) — these three sibling functions still match theto/edge endpoint by bareidwith noproject_idfilter (to.id = $to_idwith onlyfrom.project_idchecked, or no filter at all onto), the exact "project-blind endpoint match" pattern the ADR calls out and thatCreateRelation(same file, lines 24-42) was fixed to avoid. The new comment onCreateRelation("Endpoints are matched on pk, never on bare id") is therefore only true for that one function in the file. Today this is masked by an invariant the rest of the PR establishes (every edge-creation path —CreateRelation,bulkLoadRelations,CreateObservation,bulkLoadExecObservations— now binds both endpoints to the sameprojectID, so no current-schema database can contain a cross-project edge), but it is untested (the newindexer_crossproject_test.goonly exercises entity/observation bleed, notDeleteRelation/GetRelations/TraverseRelations) and silently reintroduces the original bug class the moment any future write path creates a cross-project edge. Fix: qualifyto(and, forGetRelations/TraverseRelations, the returned neighbour) byproject_idorpk, matching the standard already applied toCreateRelation,GetEntity, and the observation reads.
Minor Issues (optional)
src/kg/internal/knowledge/indexer.go:676-679andsrc/kg/internal/knowledge/log_indexer.go:339-341— theMERGE ... ON MATCH SETfallback refreshesname/type/updated_atbut notvisibility(and the log-indexer variant doesn't carry visibility at all). In the common path this is unreachable becauseclearCodeDerivedDataempties this project's code-derived rows before the batch runs, soON MATCHrarely fires today — but if that invariant ever changes, a symbol whose visibility flips on re-index would keep the stale value under the COPY-unavailable fallback. Consider addinge.visibility = $visibilitytoON MATCH SETfor completeness.src/kglib/pk.go— the "Unit Separator survives the cgo round trip, NUL does not" claim is the linchpin of the whole fix and is stated as "verified" but isn't exercised by an explicit round-trip unit test (e.g., write apkcontaining0x1f, reopen, assert the raw stored value and aMATCHby that exact string both work). Given how much of the fix depends on this exact byte surviving go-kuzu's C-string marshalling, a small dedicated test would pin the invariant against a future go-kuzu bump silently breaking it again.
Security Findings
- None.
Positive Observations
- The
pknamespacing is centralized in one helper (kglib.PK) and re-exported rather than duplicated, and every write call site I traced (indexer, log indexer, PDF indexer, embeddings, entity/observation/relation CRUD) derives it from that single function. - The write-open hard-refusal for pre-
pkdatabases, paired with the schema-version stamp bump andarchiveIfMismatch, gives a clean, well-tested migration story (TestSchemaVersionBumpRebuildsAndRestoresHandWrites) that correctly distinguishes a Kuzu-version mismatch from a schema-version mismatch in user-facing error text. GetEntity's fallback to(id, project_id)for legacy read-only opens is a thoughtful touch that keeps old graphs readable without forcing a rebuild just to view them.- New regression tests (
indexer_crossproject_test.go) directly reproduce the reported bug (shared code-derived IDs across two projects in one DB) and assert both the entity-count and observation-bleed symptoms described in the design doc.
…elations Addresses PR #17 reviewer Major: GetRelations, DeleteRelation, and TraverseRelations pinned only the `from` endpoint by project_id, leaving the `to`/neighbour match project-blind — the same pattern CreateRelation was fixed to avoid. Currently masked (no write path can create a cross-project edge), but untested and a latent reintroduction of the cross-project bleed the moment any future path does. Add `AND to.project_id = $project_id` to all three (project_id column exists on every schema version, so no !hasPK branch; return contract stays to.id, never pk). New regression test TestRelationReadsAndDeletesAreProjectScopedOnBothEnds constructs a cross-project edge via raw Cypher and asserts Get/Traverse never cross and Delete scoped to one project leaves the other's edge intact (verified fail-before/pass-after). Co-Authored-By: Claude <noreply@anthropic.com>
There was a problem hiding this comment.
Review Summary
- Files reviewed:
docs/kg-entity-key-fix-design.md,src/kglib/pk.go,src/kglib/schema.go,src/kglib/entity.go,src/kglib/observation.go,src/kglib/relation.go,src/kglib/embeddings.go,src/kglib/search.go,src/kglib/store.go,src/kglib/format.go,src/kg/migrate.go,src/kg/index.go,src/kg/internal/knowledge/indexer.go,log_indexer.go,log_app_indexer.go,indexer_pdf.go,store_wrappers.go,indexer_crossproject_test.go, plus the test files touched by the diff. - The Major issue from the prior review round (
GetRelations/DeleteRelation/TraverseRelationsbeing project-blind on thetoendpoint) is now fixed — both endpoints are qualified byproject_id, andindexer_crossproject_test.go's newTestRelationReadsAndDeletesAreProjectScopedOnBothEndsexercises exactly that regression with a raw cross-project edge. I traced every other write/read path (CreateEntity,createEntityWithID,CreateObservation,CreateRelation,SetEmbedding,batchGetObservations,GetTopObservations,batchCreateEntities,bulkLoadExecEntities/Observations,bulkLoadRelations,batchCreateRelations's fallback) and found them all consistently namespaced viaPK()/project_id. Federated cross-graph linking (graph_federated.go) correctly keys on localid/name, notpk, as it must. Two gaps remain.
Major Issues (should fix)
src/kglib/embeddings.go:186-192(SetObservationEmbedding) — this sibling ofSetEmbeddingwas not updated: it still doesMATCH (o:Observation {id: $id}) SET o.embedding = $embeddingwith noproject_id/pkqualification at all, the exact project-blind-match pattern this PR fixes everywhere else (see the comment onSetEmbedding, lines 162-165, describing precisely this hazard). Called fromBatchEmbed(line 77) after the caller already hasprojectIDin scope but drops it before the write. Practical collision risk is low (observation ids are UUIDs), but it's an inconsistent, untested gap in the same invariant the rest of the PR establishes — fix by threadingprojectIDthrough and matching via the parent Entity'spk/project_idthe wayGetTopObservations/batchGetObservationsnow do.src/kglib/store.go:113-141interacting withsrc/kglib/format.go:266-270andsrc/kg/migrate.go:148-151— a database that is unstamped (no readable.format.jsonsidecar — e.g. a pre-stamping-era DB, or any DB whose sidecar was lost/corrupted independently of the.dbdirectory) and also predates thepkcolumn now has no self-healing path.CheckFormatreturnsFormatUnstamped, notFormatMismatch, for any nil/unreadable stamp regardless ofpkpresence (format.go:266-270), soarchiveIfMismatch(migrate.go) treats it as fine and does nothing.kg index(index.go:124-134) then callsarchiveIfMismatch(no-op) followed directly byknowledge.OpenStoreread-write, which hits the new!store.hasPKgate (store.go:127-133) and hard-errors with "...must be rebuilt: run 'kg index' (or 'kg migrate'...)" — butkg indexis the command that just failed, andkg migrate's own scope loop (migrate.go:148-151:if status != kglib.FormatMismatch { continue }) explicitly skips unstamped databases too, so neither documented remedy actually repairs it. This contradicts the stated design promise in migrate.go's own doc comment ("never an error and never a manual recovery") and isn't covered by any test (no test opens an unstamped+pre-pk DB read-write). Fix: haveCheckFormat(or a parallel checkarchiveIfMismatchperforms) also treat "unstamped AND missing pk column" as a mismatch requiring archive+rebuild, so the existing rebuild machinery actually reaches this case.
Minor Issues (optional)
src/kglib/pk.go— the claim that the Unit Separator (0x1F) survives the cgo string round-trip where NUL does not is still asserted only in a comment, not pinned by a dedicated round-trip test, even though the whole fix depends on this exact byte surviving go-kuzu's marshalling. (Carried over from the prior review round; still true but unchanged in risk.)src/kg/internal/knowledge/indexer.go:677-679andlog_indexer.go:339-344— theMERGE ... ON MATCH SETfallback paths still don't refreshvisibilityon re-index, masked today byclearCodeDerivedDataemptying the project's rows first. (Carried over, unchanged.)
Security Findings
- None identified.
Positive Observations
- The previously-flagged cross-project relation-endpoint bug is fixed and now has an explicit regression test with a hand-crafted cross-project edge, which is the right way to pin this invariant.
pk.go's switch from NUL to Unit-Separator as the join byte is a real, self-caught fix (NUL truncates across the cgo boundary) — good catch by the author before this even reached review.GetEntity's legacy fallback for pre-pk read-only opens, and the newhasPK/schema-version-aware error messages instore.go/migrate.go, are thoughtful compatibility handling for the common cases.TestSchemaVersionBumpRebuildsAndRestoresHandWritesandindexer_crossproject_test.goare well-targeted, scenario-driven tests rather than incidental coverage.
…pk DBs Addresses the two Major findings from PR #17's second review round. 1. SetObservationEmbedding was project-blind (`MATCH (o:Observation {id:$id})`), the one write path still on bare-id matching. Thread projectID through from BatchEmbed (which already scopes the batch via GetUnembeddedObservations) and match on the project-namespaced pk, consistent with SetEmbedding. Collision risk was low (observation ids are UUIDs) but the inconsistency was real and untested. 2. An unstamped database that also predates the pk column had no recovery path: CheckFormat returns FormatUnstamped (not FormatMismatch), so archiveIfMismatch did nothing, then the read-write open hit the !hasPK gate and hard-errored pointing at `kg index`/`kg migrate` — but index had just failed and migrate skipped unstamped DBs too. Both documented remedies dead-ended, contradicting migrate.go's "never an error, never manual recovery" promise. Fix: detect the unstamped+missing-pk case via a read-only open (databaseMissingPK / Store.HasPK) and route it through the existing archive+rebuild+journal-replay machinery, in both archiveIfMismatch and the migrate scope loop. An unstamped-but-current DB (has pk) is still left alone and simply stamped on next write. Adds TestUnstampedPrePKDatabaseRebuildsAndRestoresHandWrites. Co-Authored-By: Claude <noreply@anthropic.com>
There was a problem hiding this comment.
Review Summary
- Files reviewed (all 25 changed):
docs/kg-entity-key-fix-design.md,src/kglib/pk.go,src/kglib/schema.go,src/kglib/entity.go,src/kglib/observation.go,src/kglib/relation.go,src/kglib/embeddings.go,src/kglib/search.go,src/kglib/store.go,src/kglib/format.go,src/kg/migrate.go,src/kg/internal/knowledge/{indexer.go,log_indexer.go,log_app_indexer.go,indexer_pdf.go,store_wrappers.go}, plus every touched test file (indexer_crossproject_test.go,migrate_test.go,embeddings_test.go,federated_test.go,journal_test.go,null_columns_test.go,search_test.go,visibility_test.go,graph_federated_test.go). - Both Major gaps raised in prior review rounds on this PR are now closed, with dedicated regression tests:
relation.go'sGetRelations/DeleteRelation/TraverseRelationsnow qualify theto/neighbour endpoint byproject_idas well asfrom(lines 69-73, 104-149, 151-168), exercised end-to-end by the newTestRelationReadsAndDeletesAreProjectScopedOnBothEndsinindexer_crossproject_test.go, which builds a real cross-project edge via raw Cypher and asserts it survives a project-A delete and is invisible to project-A reads.embeddings.go'sSetObservationEmbeddingnow takesprojectIDand matches onpk(same pattern asSetEmbedding), called correctly fromBatchEmbedwith the in-scopeprojectID.- The unstamped-but-pre-
pk"dead end" database (can't be write-opened, previously unrepaired by eitherkg indexorkg migrate) is now detected via the newStore.HasPK()/databaseMissingPKand routed into the existing archive+rebuild machinery in botharchiveIfMismatch(used bykg index) andmigrate's own scan loop, covered by the newTestUnstampedPrePKDatabaseRebuildsAndRestoresHandWrites, which constructs a real pre-pk DB with a hand-written journal entry and asserts it survives the rebuild.
- I traced every write/read path touching
pk/id/project_id(CreateEntity,createEntityWithID,batchCreateEntities,bulkLoadExecEntities,bulkLoadExecObservations,bulkLoadRelations,CreateObservation,CreateRelation,GetEntity,GetTopObservations,batchGetObservations,KeywordSearchFiltered,VectorSearch) and found them all consistently namespaced, with theGetEntity/openStoreWithConfiglegacy fallback correctly gated ons.hasPK. No stale callers of the changedSetEmbedding/SetObservationEmbedding/batchGetObservationssignatures remain. Thekgmodule'sgo.moduses a localreplaceforkglib, so the cross-module blast radius this repo structure usually risks doesn't apply here —kgalways builds against this PR'skglibchanges.
Critical Issues (must fix before merge)
None.
Major Issues (should fix)
None.
Minor Issues (optional)
src/kglib/entity.go:122-128(detectColumns) — the refactor fromif len(row) > 1 && stringOrEmpty(row[1]) == "visibility"toswitch name := stringOrEmpty(row[1]); namedrops thelen(row) > 1bounds guard entirely;row[1]is now indexed unconditionally.CALL table_info('Entity')reliably returns multi-column rows in practice, so this is unlikely to trigger, but it's a defensive check that existed and was silently removed rather than deliberately judged unnecessary. Fix: restore alen(row) > 1(or equivalent) guard before indexing.src/kg/migrate.go:62+:196—databaseMissingPK(dbPath)is called twice per unstamped database on akg migraterun (once in the main loop's status check, once again insidearchiveIfMismatchwhenindexScopeDBis reached), each doing a full read-onlyOpenStore/Close. Harmless but redundant; consider threading the already-computedmissingbool through instead of re-deriving it.
Security Findings
None.
Positive Observations
- The two Major findings from earlier review rounds are fixed correctly and each is now pinned by a purpose-built regression test rather than just a comment —
TestRelationReadsAndDeletesAreProjectScopedOnBothEndsandTestUnstampedPrePKDatabaseRebuildsAndRestoresHandWritesboth construct the exact failure condition they guard against (a raw cross-project edge; a real pre-pk+unstamped DB with a journal) rather than asserting on an artifact. CurrentSchemaVersion/FormatStamp.SchemaVersionis a clean, minimal mechanism for forcing archive+rebuild on a PK-shape change that Kuzu can'tALTERin place, and it composes correctly with the existing Kuzu-version mismatch path (openDiagnostic,archiveIfMismatchnow both report which of the two actually differs).PK()is the single, centrally-exported source of the namespacing format (kglib.PK, re-exported asknowledge.PK), so every bulk-load/COPY and every parameterized write across both modules derives the same byte format — the one place this could have silently drifted (indexer bulk paths vs.kglibwrites) is instead unified.
… scope test - kglib/entity.go: restore len(row)>1 guard in detectColumns before indexing row[1] (defensive bounds check dropped in an earlier refactor). - kg/migrate.go,index.go: thread a precomputed missing-pk hint into archiveIfMismatch so `kg migrate` does one read-only open per unstamped-pre-pk database instead of two; standalone `kg index` passes nil and computes on demand. - knowledge/indexer.go: refresh e.visibility on the MERGE ON MATCH fallback so a re-indexed symbol whose visibility flips isn't left stale (log_indexer has no visibility attribute, left as-is). - kglib/pk_test.go: pin the 0x1F pk-separator cgo round-trip invariant with a real write + reopen + query test (asserts the stored pk is byte-identical, not truncated as a NUL join would be, and resolves by exact pk). - kglib/embeddings_test.go: TestSetObservationEmbedding_ProjectScoped — the project-scoping regression test for the SetObservationEmbedding fix in caa19b2, orphaned when that commit was staged file-by-file. Co-Authored-By: Claude <noreply@anthropic.com>
There was a problem hiding this comment.
Review Summary
- Files reviewed:
docs/kg-entity-key-fix-design.md,src/kglib/pk.go,pk_test.go,schema.go,entity.go,observation.go,relation.go,embeddings.go,embeddings_test.go,search.go,search_test.go,store.go,format.go,federated_test.go,journal_test.go,null_columns_test.go,visibility_test.go,src/kg/index.go,migrate.go,migrate_test.go,src/kg/internal/knowledge/{indexer.go, log_indexer.go, log_app_indexer.go, indexer_pdf.go, store_wrappers.go, indexer_crossproject_test.go, graph_federated_test.go}. - This PR fixes the global-
id-as-PK bug (cross-project collisions under a shared DB file) by introducing a project-namespacedpk = project_id + 0x1F + idcolumn, rewriting every write/match path to key on it, and adding an archive+rebuild path for legacy unstamped/pre-pkdatabases. - I independently traced every write path (
CreateEntity,createEntityWithID,CreateObservation,CreateRelation,SetEmbedding,SetObservationEmbedding,batchCreateEntities,bulkLoadExecEntities/Observations,bulkLoadRelations) and every read path that needed project-scoping (GetEntity,GetRelations,DeleteRelation,TraverseRelations,GetTopObservations,batchGetObservations) and all are consistently namespaced, with no stale callers of the changedSetEmbedding/SetObservationEmbedding/batchGetObservationssignatures. - The two Major gaps raised in earlier review rounds on this same PR (project-blind
toendpoint onGetRelations/DeleteRelation/TraverseRelations, and the unqualifiedSetObservationEmbedding) are fixed and each is pinned by a dedicated regression test (TestRelationReadsAndDeletesAreProjectScopedOnBothEnds,TestSetObservationEmbedding_ProjectScoped). The unstamped-pre-pkdead-end database is now routed intoarchiveIfMismatch/kg migrate's rebuild machinery, covered byTestUnstampedPrePKDatabaseRebuildsAndRestoresHandWrites. - The two Minor items from earlier rounds are also now addressed:
entity.go:122-131(detectColumns) restores thelen(row) <= 1bounds guard before indexing, and the cgo round-trip claim inpk.go(Unit Separator survives, NUL doesn't) is now pinned bypk_test.go'sTestPKSeparatorSurvivesCGoRoundTrip, which proves it against a realStorewrite + fresh read-only reopen rather than asserting it only in a comment. src/kg/go.moduses a localreplace github.com/cortexa-llc/mcp/kglib => ../kglib, so thekgmodule always builds against this PR'skglibchanges — the usual cross-module blast-radius risk in this monorepo doesn't apply here.
Critical Issues (must fix before merge)
None.
Major Issues (should fix)
None.
Minor Issues (optional)
src/kglib/pk.go:30-33— the collision-freedom ofpkrests entirely on the unenforced assumption that neitherproject_idnoridever contains0x1F; nothing validates or sanitizesproject_id(derived from a repo directory base name) at the point it's accepted. Pathologically low risk on real filesystems, but consider a cheap assertion/sanitization inPK()(or at theproject_idderivation site) rather than relying purely on convention — not blocking given how unlikely a directory name containing a Unit Separator byte is in practice.src/kg/internal/knowledge/log_indexer.go:329-341(bulkLoadExecEntitiesMERGE fallback) still doesn't carryvisibilityin eitherON CREATE/ON MATCHbranch, unlike the code-indexer's equivalent inindexer.go. Likely intentional (exec-log entities don't use visibility today), but worth a one-line comment confirming that so a future reader doesn't mistake it for a second instance of the gap already fixed elsewhere in this PR.
Security Findings
None.
Positive Observations
- Every Major/Minor finding raised across this PR's prior review rounds is now fixed with a purpose-built regression test rather than just a comment, including a real cgo round-trip test for the separator-byte invariant the whole fix depends on.
- The cross-project test fixtures (
indexer_crossproject_test.go) construct real collisions via raw Cypher rather than only unit-testingPK()in isolation, which is the right level to catch this class of bug. - The migration path correctly distinguishes "unstamped but current" (leave alone) from "unstamped and pre-
pk" (archive + rebuild), and threads the already-computedmissingbool fromkg migrate's loop intoarchiveIfMismatchto avoid a redundant read-only open — resolving the earlier redundancy note.
…arify exec-log visibility omission Addresses the two remaining optional Minor findings on PR #17. - helpers.go: projectIDFromCwd now guards the invariant kglib.PK rests on — that project_id never contains the 0x1F Unit Separator. A repo directory basename holding 0x1F is effectively impossible, so this defends the invariant non-fatally at its single derivation site (replace the byte with '_' + a stderr warning) rather than panicking in hot-path PK() or adding an error return across projectIDFromCwd's many string-consuming callers. Adds a focused test. - log_indexer.go: one comment confirming exec-log entities intentionally carry no visibility attribute, so the MERGE's omission is deliberate and not a second instance of the ON CREATE/ON MATCH gap fixed in indexer.go. No behavior change. Co-Authored-By: Claude <noreply@anthropic.com>
There was a problem hiding this comment.
Review Summary
- This PR has been through several review rounds; prior rounds' Major/Critical findings (project-blind relation endpoints, unqualified
SetObservationEmbedding, the unstamped-pre-pk dead end) are all fixed and test-pinned, confirmed by re-readingrelation.go,embeddings.go, andmigrate.go/migrate_test.go. - This pass focuses on what changed since the last approved round:
src/kg/helpers.go/helpers_test.go(new0x1F-sanitization guard onprojectIDFromCwd, closing a previously-noted Minor), and themissingHint-threading added tosrc/kg/migrate.go's scan loop (closing another previously-noted Minor about a redundant read-only open). The threading change introduces a new correctness gap; the sanitization fix is incomplete at one other call site.
Major Issues (should fix)
src/kg/migrate.go:199-216— in themigratecommand's scan loop, when a database'sCheckFormatstatus isFormatUnstamped,databaseMissingPK(dbPath)is called to decide whether to rebuild it. If that call errors (e.g. the.dbdirectory is corrupted or otherwise fails to open read-only —CheckFormatnever opens the database, only its sidecar, so this is reachable independently of the stamp check), the conditionmErr == nil && missingis false, soneedsRebuildsilently staysfalseand the loop justcontinues — butchecked++already ran at line 199, before this block. The database is counted as successfully checked and the run can end with"✅ %d database(s) checked; all readable by this build", which is false for this database. This directly violates the invariant the surrounding code documents a few lines above (unchecked"gates the summary, because the one thing this command must never do is print an all-clear for a database it never looked at") — every other failure mode in this function correctly callsfailf, this one doesn't. Fix: onmErr != nil, callfailf("scope %s: cannot determine whether %s predates the primary key: %v", label, dbPath, mErr)andcontinue, the same as theCheckFormaterror path just above it.
Minor Issues (optional)
src/kg/helpers.go:79vssrc/kg/graph.go:305— the new guard comment onprojectIDFromCwdclaims it defends the0x1F-separator invariant "at its single derivation site," butrunFederatedGraphderivesproject_idfor the federated-graph filter viafilepath.Base(root)directly, bypassingprojectIDFromCwdand its sanitization.graph_federated.go'slayerIdentities/loadLayerGraphmatch this value literally against the storedproject_idcolumn (not viaPK()), so in the (admittedly pathological) case where a project root's basename contains the0x1Fbyte, indexed entities would carry the sanitizedproject_idwhile--federatedwould filter on the raw, un-sanitized one and silently return zero matches. Fix: useprojectIDFromCwd(root)atgraph.go:305instead offilepath.Base(root)for consistency, or note explicitly in the helpers.go comment that it is not in fact the only derivation site.
Security Findings
None.
Positive Observations
- The two Minor items flagged in the previous (approved) review round are both addressed: the
0x1F/project_id sanitization guard is real and tested (TestProjectIDFromCwdReplacesSeparator), and the redundant doubledatabaseMissingPKread-only open is now avoided viamissingHintthreading — the fix for the latter is what introduced the Major finding above, so the next round should re-verify the threading change rather than re-litigate the original redundancy. - The core
pknamespacing fix itself (schema, writes, relation endpoints, embeddings, migration/rebuild path) remains correctly and consistently applied everywhere it was re-examined in this pass.
… project_id through sanitizer Addresses the Major + Minor from PR #17's latest review. Major (a regression from the earlier migrate double-open dedup): in the `kg migrate` scan loop, when databaseMissingPK errored (e.g. a corrupt .db that won't open read-only) the error was swallowed — needsRebuild stayed false and the loop continued, but checked++ had already run, so the run could print a false "all readable by this build" all-clear for a database it never resolved. Now failf + continue (mirroring the CheckFormat-error path), so failf's unchecked count gates the summary and no all-clear is emitted. Adds a regression test (TestMigrateDoesNotCountScopeDatabasesWhosePKCannotBeDetermined) that fails with the false all-clear before the fix. Minor + tree audit: graph.go (federated filter) derived project_id via filepath.Base(root), bypassing projectIDFromCwd's 0x1F sanitization — the federated filter matches project_id literally, so a sanitized index vs raw filter would mismatch. Auditing the tree surfaced two more raw sites: handle_server.go (the MCP server's project_id, feeding entity/observation/ relation writes — a second live bug) and helpers.go openStore's returned projectID. All three now route through projectIDFromCwd, making its "sole funnel" doc comment actually true. Co-Authored-By: Claude <noreply@anthropic.com>
There was a problem hiding this comment.
Review Summary
- Files reviewed: full diff (
gh pr diff 17) acrossdocs/kg-entity-key-fix-design.md,src/kglib/{pk.go,pk_test.go,schema.go,entity.go,observation.go,relation.go,embeddings.go,embeddings_test.go,search.go,search_test.go,store.go,format.go,federated_test.go,journal_test.go,null_columns_test.go,visibility_test.go},src/kg/{helpers.go,helpers_test.go,graph.go,handle_server.go,index.go,migrate.go,migrate_test.go,migrate_scopes_test.go},src/kg/internal/knowledge/{indexer.go,indexer_pdf.go,log_indexer.go,log_app_indexer.go,store_wrappers.go,indexer_crossproject_test.go,graph_federated_test.go}. - This PR has been through several prior review rounds (visible in thread history). All previously-raised Critical/Major findings — project-blind
GetRelations/DeleteRelation/TraverseRelationsendpoints, unqualifiedSetObservationEmbedding, the unstamped-pre-pkdead-end database, thechecked++-before-failfordering bug inmigrate.go's scan loop, and thegraph.go:305/helpers.goseparator-sanitization inconsistency — are confirmed fixed in this diff, each with a dedicated regression test (TestRelationReadsAndDeletesAreProjectScopedOnBothEnds,TestSetObservationEmbedding_ProjectScoped,TestUnstampedPrePKDatabaseRebuildsAndRestoresHandWrites,TestMigrateDoesNotCountScopeDatabasesWhosePKCannotBeDetermined,TestProjectIDFromCwdReplacesSeparator). I independently re-traced every write/read path touchingpk/id/project_idand found them all consistently namespaced, with no stale callers of the changed signatures. - One new, purely cosmetic issue surfaced this pass (see Minor below); no Critical or Major issues found.
Critical Issues (must fix before merge)
None.
Major Issues (should fix)
None.
Minor Issues (optional)
src/kg/internal/knowledge/store_wrappers.go:33— the comment on the re-exportedPKsays "the NUL-join format must match byte-for-byte," butsrc/kglib/pk.go:20-26explicitly rejected NUL (it gets truncated by go-kuzu's cgo string marshalling) and uses the Unit Separator (0x1F) instead. The code is correct (it just callskglib.PK), but the comment itself misstates the format it's describing. Fix: change "NUL-join" to "0x1F-separator" (or just "pk-derivation") to matchpk.go's actual documented format.
Security Findings
None.
Positive Observations
- Every Major/Critical gap raised across this PR's five prior review rounds is now fixed and pinned by a purpose-built regression test rather than just a comment, including a real cgo round-trip test for the separator-byte invariant (
pk_test.go) and a test proving themigratescan loop correctly fails closed when pk-introspection errors (rather than silently printing an all-clear). - The
kgmodule'sgo.moduses a localreplaceforkglib, so the usual cross-module blast-radius risk in this monorepo (three separatego.mods, nogo.work) doesn't apply here —kgalways builds against this PR'skglibchanges. - Design doc (
docs/kg-entity-key-fix-design.md) is unusually thorough and the implementation matches its accepted design (Option A1) faithfully, including the documented rationale for rejecting NUL in favor of0x1F.
…tor) The re-exported PK comment in store_wrappers.go still described a NUL join, but pk.go rejected NUL (go-kuzu's cgo string marshalling truncates at NUL) and uses the 0x1F Unit Separator. Comment-only; the code just calls kglib.PK. Co-Authored-By: Claude <noreply@anthropic.com>
There was a problem hiding this comment.
Review Summary
- Files reviewed (full diff,
gh pr diff 17):docs/kg-entity-key-fix-design.md,src/kglib/{pk.go,pk_test.go,schema.go,entity.go,observation.go,relation.go,embeddings.go,embeddings_test.go,search.go,search_test.go,store.go,format.go,federated_test.go,journal_test.go,null_columns_test.go,visibility_test.go},src/kg/{helpers.go,helpers_test.go,graph.go,handle_server.go,index.go,migrate.go,migrate_test.go,migrate_scopes_test.go},src/kg/internal/knowledge/{indexer.go,indexer_pdf.go,log_indexer.go,log_app_indexer.go,store_wrappers.go,indexer_crossproject_test.go,graph_federated_test.go}. - This PR fixes the global-
id-as-primary-key bug (cross-project collisions in a shared Kuzu DB file) via a project-namespacedpk = project_id + 0x1F + idcolumn, rewritten write/match paths, and an archive+rebuild path for legacy unstamped/pre-pkdatabases. - Independently re-traced every write path (
CreateEntity,createEntityWithID,CreateObservation,CreateRelation,SetEmbedding,SetObservationEmbedding,batchCreateEntities,bulkLoadExecEntities/Observations,bulkLoadRelations) and read path (GetEntity,GetRelations,DeleteRelation,TraverseRelations,GetTopObservations,batchGetObservations) — all consistently namespaced onpk/project_id, with no stale callers of the changed signatures. - All prior Major/Critical findings from earlier rounds of this PR (project-blind
toendpoint onGetRelations/DeleteRelation/TraverseRelations; unqualifiedSetObservationEmbedding; the unstamped-pre-pkdead-end database; thechecked++-before-resolution semantics inmigrate.go's scan loop; thegraph.go:305vshelpers.goseparator-sanitization bypass; thedetectColumnsbounds-guard removal; thestore_wrappers.go"NUL-join" comment misstatement) remain fixed, each pinned by a dedicated regression test (TestRelationReadsAndDeletesAreProjectScopedOnBothEnds,TestSetObservationEmbedding_ProjectScoped,TestUnstampedPrePKDatabaseRebuildsAndRestoresHandWrites,TestMigrateDoesNotCountScopeDatabasesWhosePKCannotBeDetermined,TestProjectIDFromCwdReplacesSeparator,TestPKSeparatorSurvivesCGoRoundTrip). src/kg/go.moduses a localreplace github.com/cortexa-llc/mcp/kglib => ../kglib, so the usual cross-module blast-radius risk in this three-go.mod/no-go.workmonorepo doesn't apply —kgalways builds against this PR'skglibchanges.
Critical Issues (must fix before merge)
None.
Major Issues (should fix)
None.
Minor Issues (optional)
None new. (Previously-noted minors — entity.go bounds guard, pk.go round-trip test, store_wrappers.go comment wording, graph.go:305 sanitization bypass — are all addressed in this revision.)
Security Findings
None.
Positive Observations
- Every Major/Critical gap raised across this PR's many prior review rounds is fixed and backed by a purpose-built regression test rather than just a comment fix, including a real cgo round-trip test for the
0x1F-separator invariant and a test proving themigratescan loop fails closed (rather than printing a false all-clear) when pk-introspection errors. - The design doc (
docs/kg-entity-key-fix-design.md) is thorough and the implementation faithfully matches its accepted design, including the documented rationale for choosing0x1Fover NUL given go-kuzu's cgo string marshalling. - Comments throughout the diff (
CreateRelation,SetEmbedding,GetRelations,migrate.go) explicitly name the hazard each fix closes, making the invariants easy for future readers to verify rather than just trust.
Problem
kg indexfloodsduplicated primary key/COPY FROM JSON unavailablewarnings. These are not cosmetic — they flag silent cross-project data loss.The
Entity/Observationprimary key is globalidalone (schema.go), but every clear/delete and nearly every read is scoped byproject_id. When two projects share a DB file, a prior project's rows survive the project-scoped clear and collide with the current run's identically-id'd entities onCREATE, which are then silently dropped (old row wins). Relations bled too:CreateRelation's endpointMATCHwas project-blind, so edges could bind to the wrong project's node.Fix — ADR Option A1
Full design:
docs/kg-entity-key-fix-design.md(included in this PR).pk STRING PRIMARY KEY = project_id <0x1F> idto Entity and Observation via one central helper (src/kglib/pk.go).idstays a plain, human-facing, journal-stable column.\x00: cgo truncates C strings at NUL, collapsing every pk back to itsproject_id(verified failure) — so NUL would have silently re-introduced the collisions.CreateRelationmatches endpoints onpk(closes the bleed);CreateObservationbinds its parent bypk;GetEntitymatches onpkwith anid+project_idfallback for un-migrated DBs.GetTopObservations,batchGetObservations) now scope through the parent entity's project.SetEmbeddingtakesprojectID, matches onpk; removed the project-blindentityProjectID).CREATE→MERGEonpk(ON CREATEpopulates the full row;ON MATCHrefreshes only name/type/updated_at).FormatStamp.SchemaVersion→ 2 soCheckFormattrips the existingarchiveIfMismatchrebuild + journal replay (Kuzu v0.11.3 has noADD PRIMARY KEY). The journal is the only copy of hand-writes — the old DB is archived, never deleted.id/name, neverpk.Tests
duplicated primary key/COPY FROM JSON unavailable).go build,go vet,gofmt -l, andgo test ./...clean in bothsrc/kgandsrc/kglib.Review history
Implemented and reviewed via two engineer→reviewer rounds before this PR. The first review caught a real cross-project observation read bleed the initial tests missed (now fixed + regression-tested); a focused re-review confirmed all findings closed with no new defects.
Migration note for reviewers
Existing stamped DBs rebuild from source exactly once on first index after this lands (schema bump). Hand-written entities/relations replay from the journal. A truly unstamped legacy DB still reads fine; a write against one returns an actionable "run kg index" message rather than self-healing.
Generated with Claude Code