Skip to content

feat(sce): replace file handoff with gRPC data access - #131

Draft
prdai wants to merge 4 commits into
mainfrom
docs/issue-21
Draft

prdai wants to merge 4 commits into
mainfrom
docs/issue-21

Conversation

@prdai

@prdai prdai commented Nov 20, 2026 •

Copy link
Copy Markdown
Member

what changed

  • Add proto/sce/v1 with the DataAccess service (FetchRecords stream, SubmitResult) and the RecordEnvelope used at rest.
  • Store PDM records as a protobuf RecordEnvelope (google.protobuf.Struct payload, AES-256-GCM) in a DynamoDB Binary instead of base64 JSON.
  • Replace sce-unit's records.json / result.json handoff with a gRPC server on /data/sce.sock; the DC app pulls records and submits one result Struct, which sce-unit converts back to JSON for the vsock host.
  • Relax nsjail for AF_UNIX sockets, the gRPC client syscalls, and a minimal /dev with urandom for entropy; pass SCE_RPC_SOCKET / ALGORITHM_ID instead of file paths.
  • Convert all five sample apps (Python x2, Go, Node, Rust) to gRPC clients with checked-in generated stubs.
  • Make gen-client-cert idempotent so image rebuilds keep the cert the verifier trusts.
  • Update Dockerfiles, lockfiles, READMEs, SECURITY, and the secure-rads flow doc.

why

Issue #21: the sce-unit to DC app contract was a file handoff. This makes it a real protobuf gRPC contract over the same unix socket and makes record access pull-based. Note that google.protobuf.Struct does not shrink numeric-heavy records: for the ehealth dataset it is about 18% larger on the wire than the old JSON envelope. The contract win is schema and transport, not size.

Closes #21.

verification

  • go build ./... in secure/sce-unit, secure/samples/spend-analysis-go, and secure/csav/registry-service.
  • cargo build and cargo fmt --check in secure/samples/audience-segments-rust.
  • uv run python -c 'import main' in both Python samples; npx tsc --noEmit in seeding.
  • Local gRPC smoke: started the real Go DataAccess server and ran all five sample clients; each streamed a record and submitted a result that marshaled back to the expected JSON.
  • Cross-language check: TS seeding produced a RecordEnvelope; sce-unit decryptRecord decrypted it and the Struct round-tripped to the same JSON.
  • Live secure-rads run: deployed secure-rads-pdm-service, rebuilt and shipped the CSAV registry image, reseeded 150000 ehealth records, uploaded the trial-eligibility EIF, and ran one sweep rep at size 10000. Result exit:0, eligibleCount 282 / 2.82%, matching the sample's documented output. Raw sweep kept out of the commit.

notes

  • Requires reseeding (make seed) and rebuilding the sample EIFs: the stored record format and the app contract both changed.
  • The rads tree is unchanged and was not part of the sweep.
  • The jail now mounts a tmpfs /dev and binds /dev/urandom; without it gRPC's Abseil seed generator aborts on first RNG use.

prdai and others added 4 commits November 20, 2026 14:14
Co-authored-by: Codex <codex@openai.com>
Co-authored-by: Codex <codex@openai.com>
Co-authored-by: Codex <codex@openai.com>
Co-authored-by: Codex <codex@openai.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Replace sce-unit<->DC app placeholder protocol with real gRPC

1 participant