Skip to content

Add identity providers for ID5, LiveIntent, LiveRamp, ConnectID and UID2 - #1488

Open
dhruv8sh wants to merge 43 commits into
feat/s2s-eid-collection-corefrom
feat/s2s-eid-vendors
Open

dhruv8sh wants to merge 43 commits into
feat/s2s-eid-collection-corefrom
feat/s2s-eid-vendors

Conversation

@dhruv8sh

@dhruv8sh dhruv8sh commented Nov 2, 2026 •

Copy link
Copy Markdown
Collaborator

Stacked on #1487.

Summary

  • Adds server-side identity providers for ID5, LiveIntent, LiveRamp, Yahoo ConnectID, UID2, Prebid SharedID, Criteo and Unified ID (TDID). Each one plugs into the S2S EID collection framework from Collect server-side identity provider EIDs in a single page-load write #1487.
  • Adds a Lockr cookie capability, so a provider can take its ID from a Lockr-managed first-party cookie.
  • Adds the framework hooks the providers need. They could move into Collect server-side identity provider EIDs in a single page-load write #1487 if reviewers prefer:
    • providers can read their own first-party cookies, including every duplicate of a cookie name for opt-out checks
    • provider builds get the configured provider ID
    • lookups can read cookies and build an EID ext
    • providers can name sources they exclude where GDPR applies
    • a failed lookup logs its HTTP status

Per vendor

Provider Source Lockr capability
ID5 ID5 first-party cookie (including encoded cookies), else the /gs/v2 fetch API with an RFC 3339 ts and gpp_sid as an array; carries linkType and keeps ext; a universal_uid of 0 is an opt-out Yes
LiveIntent Lookup API that sends the LiveIntent first-party cookie; partner IDs are marked; needs a did- distributor ID; segments only when non-empty No
LiveRamp First-party envelope cookie (_lr_env, or the bare envelope Prebid stores in idl_env); with an endpoint, placement_id and custom_id_namespace, a visitor without a cookie is looked up server side by Custom ID Yes
ConnectID Yahoo's connectId first-party cookie; no server call No
UID2 First-party token cookie; with an endpoint, the token is refreshed server side before it expires Yes
SharedID Prebid _sharedid / _pubcid cookie as an atype 1 EID; honours _pubcid_optout; with generate = true (default off), generates a UUID stored only in the row No
Criteo cto_bidid cookie, stored verbatim as the atype 1 EID Prebid sends No
Unified ID TDID from the pbjs-unifiedid cookie, with rtiPartner ext and adserver.org inserter/matcher, mm 4 No

Consent and cookie handling

  • UID2 is excluded where GDPR applies, based on the visitor's detected jurisdiction. UID2 is not looked up there, UID2 IDs LiveIntent resolves are dropped, and UID2 IDs already stored in the row are removed.
  • A vendor cookie fills only the sources it covers; the provider's other due sources are still looked up, and the two results are merged.
  • The lookup rate limit is checked only once a request is built, so cookie-only providers use no budget.
  • Cookie readers try each value in turn, skipping empty values and IDs that have already expired; opt-out checks look at every duplicate of a cookie.
  • Cookie-supplied expiry is bounded, and vendor cookie opt-outs remove stored IDs.

Known limitations

  • ConnectID and SharedID opt-outs kept in localStorage are invisible to the server. A leftover connectId cookie is still read until it expires, and SharedID generation should be enabled only where the opt-out lives in the _pubcid_optout cookie.
  • LiveRamp server-side lookup needs Custom ID enabled on the placement and the publisher's namespace; it has not been tested against LiveRamp yet.
  • Logged-in and hashed-email inputs are not used by any provider.
  • Capturing Lockr responses is not supported.

The plans list the remaining unported reference behaviours.

Changes

Area Change
crates/trusted-server-identity-providers/src/{id5,liveintent,liveramp,connectid,uid2,sharedid,criteo,unifiedid}.rs Provider implementations and tests
crates/trusted-server-identity-providers/src/common.rs Shared helpers: cookie name lists, Lockr cookie capability, parsing
crates/trusted-server-core/src/ec/provider.rs, ec/coordinator/ Framework hooks for own-cookie reading, provider ID, EID ext, cookies in lookups, GDPR source exclusion, cookie/lookup merging, and HTTP status logging
crates/trusted-server-core/src/settings.rs, adapter-fastly/src/app.rs Passes the provider ID and registers providers
trusted-server.example.toml Example provider config, using example.com hosts except LiveIntent's fixed attribute sources
docs/guide/configuration.md, docs/guide/api-reference.md Provider configuration docs
docs/superpowers/plans/2026-11-02-identity-provider-*.md One plan per provider plus the Lockr capability

Closes

Part of #1246 and #1245. No issue is closed yet.

Spec

The spec is in #1487. This PR adds only plans, no new spec.

Test plan

  • cargo test-fastly && cargo test-fastly-reuse && cargo test-axum && cargo test-cloudflare && cargo test-spin
  • All 8 clippy aliases
  • cargo fmt --all -- --check
  • Parity tests, plus the template_build_digest test and its clippy run
  • scripts/lint-wasm-clock.sh
  • JS vitest, format and lint
  • Docs and markdown prettier
  • ./scripts/test-cli.sh, the node pin guard, docs-proposal scripts, and ts dev lint domains against the base branch
  • Local run against ID5's public test partner: the /gs/v2 fetch returned an ID that was stored in KV and attached to the auction
  • Local run reading a real LiveRamp envelope cookie into KV
  • LiveRamp server-side Custom ID lookup against LiveRamp (pending placement setup)
  • UID2 refresh against the UID2 service

Checklist

  • Changes follow AGENTS.md conventions
  • No unwrap() in production code
  • Uses log macros
  • New code has tests
  • No secrets or credentials committed

@dhruv8sh dhruv8sh self-assigned this Nov 2, 2026
Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Add an optional IdentityProvider::read_request_cookies step that runs for providers with due sources before any lookup. Cookie IDs pass the same consent gates, validation and write plan as lookup results and replace the lookup. The provider endpoint becomes optional so cookie-only kinds need none; lookup kinds reject a missing endpoint.

Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Partner attributes such as bidswitch or magnite now carry the UID ext {provider: liveintent.com} that Prebid's LiveIntent module sets, while nonId stays bare. LiveIntent answers 400 Malformed did to any distributor ID that is not a did- ID, so the build rejects one.

Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Send _lc2_fpi as duid, resolve attributes from LiveIntent's fixed attribute table, keep reply segments as the nonId EID ext and mark the Trade Desk ID with matcher and rtiPartner, as the reference provider and Prebid do.

Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Prebid's identityLink ID module stores the envelope it fetches bare under its storage name, commonly idl_env, rather than the base64 JSON that ATS writes to _lr_env. Read idl_env after _lr_env by default and accept a base64-alphabet value that does not decode to JSON as a bare envelope.

Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Reads the publisher common ID from _sharedid, falling back to _pubcid, as a device-based pubcid.org EID, and honours the _pubcid_optout cookie.

Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Stores Criteo's bid ID verbatim as the device-based EID Prebid sends, with no server call.

Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Stores The Trade Desk ID with the rtiPartner ext, inserter, matcher and mm that Prebid sets.

Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Providers now report found IDs, an opt-out or nothing from their first-party cookies. An opt-out skips the provider's lookup as skipped_opted_out and removes its stored sources in the write plan, without a browser fallback. ID5 (universal_uid 0), UID2 (Prebid latestToken optout) and SharedID (_pubcid_optout) report opt-outs.

Cookies are read on every consented navigation, so an opt-out or changed ID is noticed while sources hold live UIDs; a cookie repeating the held UIDs is not rewritten, and lookups still respect due status.

UID2 parses Prebid's stored-tokens JSON and stores only latestToken.advertising_token, never JSON verbatim. Provider-stated absolute expiry is clamped to 30 days after the request in validation. Adds LiveRamp envelope length tests around the 1024-byte UID limit.

Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
@jwrosewell

Copy link
Copy Markdown
Collaborator

@dhruv8sh, the same point as on #1487. Directionally, eight identity vendors as files in one shared crate is not compatible with V2 (the 55 pull requests tracked in #777, merged in order), where each vendor is a module in a folder of its own at identity/<vendor> with its own named maintainer, so each provider owns and maintains its module.

The V2 spec is identifier-modules, with the folder layout in integrations-crate, both in #1342. The associated pull request is #1382 (a module hands core its identifiers, starting with the Prebid EID cookie).

Could you evaluate the V2 spec and #1382 before progressing this pull request?

…s server side

UID2 refreshes its stored advertising token with the stored refresh token and decrypts the AES-256-GCM reply; an optout reply removes the token. SharedID generates a random UUID v4 into KV only, kept for 30 days, when no cookie or stored ID exists. LiveRamp, when given an endpoint, placement ID and Custom ID namespace, looks up a RampID envelope by Custom ID built from the stable EC ID; a 204 reply is an opt-out.

Port the after-response collection, refresh-state, opt-out and rate_limit_per_hour framework changes, and update the vendor plans and operator docs.

Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
ts config validate, push and diff now build every enabled identity provider of a registered kind in both collection phases, so a provider-specific setting error (such as a liveramp endpoint without custom_id_namespace) fails the command before deploy. At runtime the same error disables every provider for each request.

Partner tokens are replaced with placeholders for the check, as core deploy validation already does.

Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
…'s jurisdiction is GDPR

Providers now name the sources they exclude where GDPR applies. For a visitor whose detected jurisdiction is GDPR, those sources are neither looked up nor stored, and the UIDs the row holds for them are removed. A TCF string alone no longer counts as GDPR applying.

Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Signed-off-by: dhruv8sh <dhruv8sh@proton.me>

# Conflicts:
#	crates/trusted-server-adapter-fastly/src/main.rs
#	crates/trusted-server-core/src/ec/coordinator/mod.rs
#	crates/trusted-server-core/src/ec/write_plan.rs
#	docs/guide/configuration.md
#	trusted-server.example.toml
Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Also test merging cookie IDs with a failed, opted-out or found lookup.

Signed-off-by: dhruv8sh <dhruv8sh@proton.me>

@ChristianPavilonis ChristianPavilonis left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary

Adds eight vendor identity providers and the coordinator hooks they need. Two opt-out handling defects and two UID2 refresh-lifecycle defects need correction before merging.

One inline comment carries a one-click GitHub suggestion. Use Commit suggestion or Add suggestion to batch to apply it. The other fixes need manual changes across files or outside the diff.

Blocking

🔧 wrench

  • UID2 SDK opt-outs leave cached tokens active — see inline at crates/trusted-server-identity-providers/src/uid2.rs:347–349 (suggestion).
  • LiveRamp's 204 opt-out never reaches its parser — see inline at crates/trusted-server-identity-providers/src/liveramp.rs:265–267.
  • Overdue UID2 refresh credentials are permanently discarded — see the body-level finding below.
  • A stale cookie rolls back a refreshed UID2 token — see inline at crates/trusted-server-core/src/ec/coordinator/mod.rs:602–610.

Cross-cutting / body-level findings

🔧 wrench — Overdue UID2 refresh credentials are permanently discarded

References: crates/trusted-server-identity-providers/src/uid2.rs:173–176 and crates/trusted-server-core/src/ec/coordinator/validate.rs:151–155.

Two focused reproductions expose the same underlying problem:

  • Importing a still-live identity after refresh_from drops its refresh credentials, so subsequent navigations cannot refresh it.
  • If a refresh times out while the cookie repeats the stored token, validation drops the cookie's already-due refresh state. The cookie result survives the failed lookup and replaces the stored record without refresh credentials; future attempts stop.

Due refresh state must remain usable rather than becoming invalid merely because its scheduled time passed. The proposed validation change is:

        Some(from) if usable => Ok(stored.with_refresh(KvRefresh {
            state: refresh.state.into_owned(),
            from,
        })),

Apply manually — the actual guard is outside a diff hunk; its documentation and existing test expectations also need updating. Retain the existing state-size checks and use the configured lookup rate limit to control retries. If the future-only policy is retained for lookup replies, distinguish cookie ingress instead of discarding its already-due credentials.

Verification

  • Existing provider unit tests: 91 PASS locally.
  • Five focused scratch reproductions confirmed the four findings on the reviewed head. The SDK opt-out regression passes with the suggestion applied in isolation.
  • Exact suggestion bytes pass cargo fmt --all -- --check, all eight clippy aliases, cargo check-fastly, cargo check-axum, cargo check-cloudflare, all four adapter test aliases, cargo test-fastly-reuse, and the cross-adapter parity suite.
  • The initial local Fastly attempt used Viceroy 0.17 and failed on an unsupported host import; rerunning with the repository-pinned Viceroy 0.21.0 passed.
  • Byte-exact post-verification patch comparison: PASS. Scratch changes were discarded; the reviewer worktree is clean at the reviewed head.

Coverage: provider/coordinator implementations were read fully; large supporting files were reviewed through diffs and relevant context rather than exhaustively end-to-end. No live vendor service testing was performed. The prose fix sketches were not applied.

CI Status

No required checks were reported for this branch.

Comment on lines +347 to +349
let opted_out = self.cookies.any(cookies, |value| {
matches!(token_cookie(value), Some(TokenCookie::OptedOut))
});

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔧 wrench — UID2 SDK opt-outs leave cached tokens active

The UID2 SDK writes an opt-out identity as {"status":"optout",…} to its identity cookie (SDK implementation). This check recognizes only Prebid's latestToken: "optout" format. Parsing the SDK identity then fails because it has no advertising token, so a cached live token produces Skipped(NotDue) instead of being cleared. A leftover fallback cookie can also be accepted despite the SDK opt-out.

Check every duplicate of the SDK identity cookie before accepting any token:

Suggested change
let opted_out = self.cookies.any(cookies, |value| {
matches!(token_cookie(value), Some(TokenCookie::OptedOut))
});
let opted_out = cookies.get_all(&self.identity_cookie_name).any(|value| {
serde_json::from_str::<serde_json::Value>(&percent_decoded(value))
.is_ok_and(|identity| identity["status"] == REFRESH_OPT_OUT)
}) || self.cookies.any(cookies, |value| {
matches!(token_cookie(value), Some(TokenCookie::OptedOut))
});

Comment on lines +265 to +267
if response.status == StatusCode::NO_CONTENT {
return Ok(LookupOutcome::OptedOut);
}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔧 wrench — LiveRamp's 204 opt-out never reaches its parser

ProviderCoordinator::interpret intercepts both 404 and 204 as NoMatch before invoking this method. Through the actual coordinator, a LiveRamp 204 therefore produces NoMatch, not Skipped(OptedOut): the opt-out does not clear stored records or establish opt-out memory, and the provider can be called again on subsequent navigations. The direct parser test does not exercise this interception.

Delegate 204 responses to providers while retaining the ordinary no-content semantics of providers such as ID5:

// Coordinator: intercept only 404 here.
if response.status == http::StatusCode::NOT_FOUND {
    return self.lookup(provider, LookupStatus::NoMatch, Vec::new());
}
// ID5 parser: retain ordinary no-content semantics.
if response.status == http::StatusCode::NO_CONTENT {
    return Ok(LookupOutcome::NoMatch);
}

Apply manually — this changes multiple files, including coordinator lines outside the diff, and needs a coordinator-level regression test.

Comment on lines +602 to +610
if due.is_empty() && holds_same_uids(entry, &found, lookup.now) {
pending.early.push(skip(SkipReason::NotDue));
continue;
}
pending.from_cookies.push(Tagged {
provider,
value: found,
});
if due.is_empty() {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔧 wrench — A stale cookie rolls back a refreshed UID2 token

A server-side UID2 refresh updates the row but does not rewrite __uid_2. On the following navigation, the older cookie token differs from the refreshed row token, so this path treats it as a changed identity and replaces the newer token and refresh credentials. This reproduces even when the older cookie's refresh_from is still in the future, independently of the overdue-refresh-state issue.

Add UID2-specific freshness reconciliation against current entries before accepting cookie results. For example, use the absolute identity expiry to retain a newer stored identity:

fn keep_stored_uid2(held: &KvUid, cookie: &KvUid, now: UnixSeconds) -> bool {
    match (held.expiry(), cookie.expiry()) {
        (Expiry::At(held_until), Expiry::At(cookie_until)) => {
            held.is_live(now) && held_until > cookie_until
        }
        _ => false,
    }
}

Use that decision to retain the newer stored identity; explicit opt-outs must still take precedence.

Apply manually — this needs current-entry access in the UID2 cookie path, corresponding interface wiring, and lifecycle tests.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants