Repository navigation
✨ Every administrative action logged, including a tenant-level log (EMS-AUDIT) - #3902
Merged
Merged
Conversation
…MS-AUDIT) Each tenant gets a signed log of its own for what happens outside election events: Admin Portal sign-ins and failures, staff accounts and roles, tenant settings, templates and the lifecycle of election events. Changes carry their old and new values, in the tenant log and in each event's log. - electoral-log: Audit statement kinds, a database per tenant, retiring a deleted event's database, event type and log type filters - hasura: audit_log_outbox and the audit_change() trigger on every table an admin role can write, with its coverage test - windmill: outbox worker, Keycloak events of the tenant realm and admin events, exports with scope, range and SQL, monitoring lines - harvest: tenant log list and export routes, audit entries in the routes that change state - keycloak: custom-event-listener for the tenant realm and admin events - admin-portal: Settings > Logs, entry popup, export scopes Parent issue: sequentech/meta#13781 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- An auditor who holds only the tenant log's permissions can open Settings > Logs: the portal reads the schema and the tenant row as tenant-logs-read - The steps of a sign-in are recorded as LOGIN_STEP, so one sign-in is one LOGIN - Saving the Google service account key writes its entry, without the key - One-time codes are redacted in Keycloak admin events - The tenant log's permissions get their own migration, with the log - The whole-log CSV and PDF export records LogsExported with the file's hash - A deleted event's entry names the event; the writer's start is retried until the database is ready Parent issue: sequentech/meta#13781 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. 🗂️ Base branches to auto review: 2
Please check the settings in the CodeRabbit UI or the ⚙️ Run configuration
You can disable this status message by setting the Use the checkbox below for a quick retry:
Comment |
Parent issue: sequentech/meta#13781 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s, an undeclared action enum, the log crate's coverage - scripts/coverage/run.py: a profile without scope_exceptions is measured instead of failing with a KeyError - hasura: declare RestorePrivateKeyOutcome, which AuthorizeTallyCustodyOutput uses, so the metadata is consistent - electoral-log: a server that is down names its databases and fails every other operation Parent issue: sequentech/meta#13781 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Parent issue: sequentech/meta#13781 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#3900 merged a tenant log of its own into ovcs. EMS-AUDIT's replaces it: - One tenant log: the per-tenant database, outbox and trigger, listed by listElectoralLog with no election event. #3900's boards in the base database, listTenantLog, POST /tenant-log and its Settings > Logs go. - StatementBody::TenantActivity keeps Borsh tag 43 and its statement types 62-63; Audit moves to tag 44 and its types to 64-68. - Kept from #3900, written into the same log: Harvest's request entries (TenantAdminAction, with accepted, denied or failed), MESSAGE_SENT without the message's text, and entries stamped with the time their producer saw them. An event of an unknown tenant is dead-lettered instead of creating a log. - Keycloak listener: one realm parser (LogRealm); a message sent to a tenant account leaves Keycloak without its text. - #3900's repairs of ovcs's checks are taken as they are. - sequent-core WASM package regenerated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…eycloak account is its own The backend journeys, which had stopped before any journey on ovcs, found two defects: - The audit entry of an imported event read `election_event.name`, a column dropped when names moved to the presentation, so every import failed. It now reads the name in the event's default language. - Step signs in to Keycloak's admin API as the master realm's user named by KEYCLOAK_ADMIN_CLIENT_ID (a password grant through admin-cli), not as a service account. Its realm updates during an import were logged in the new event's log as an operator's changes. That user, in the master realm, is now Step's own account. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ereslibre
added a commit
that referenced
this pull request
Oct 17, 2026
…dmin keys after commit, new tenants' realms (part 2) (#3929) Parent issue: sequentech/meta#13781 Design package [EMS-AUDIT](https://project-1.sequent.sh/#package/EMS-AUDIT), second part. #3902 gave each tenant a signed tenant log; this pull request closes the gaps it listed or left open, apart from the route-by-route entries (see "Split with #3928" below). ## What changes | Area | Change | | --- | --- | | Imports | Importing an election event or a tenant configuration records the file's SHA-256, computed by the task from the file it read, and compares it with the checksum the admin declared (`undeclared`, `matches` or `mismatch`). The entry also counts what the file held: elections, contests, candidates, areas and areas per contest, or configuration sections. A failed commit of a tenant configuration import no longer records the import. | | Entry popup | A new **The imported file** section shows the checksum verdict, the SHA-256, the counts and the document, instead of raw extra details. The deleted election event section names **The log's system key**. Translated into the portal's 8 languages. | | Deleted events in exports | One account's export with "every election event" now also reads the logs of deleted events from their retired databases, merged by time; their rows name `retired event <ID>`. Each retired database is checked against the catalog, opened read-only under the event database lock, and closed to connections again, including when the read fails. The deletion entry keeps the retired log's system public key, since the event's keys are deleted with it. | | Admin public keys | `AdminPublicKey` is posted to the tenant log only once the transaction that saved the key commits. A rollback leaves nothing behind (before, it could leave a public key in the log whose key was then lost). The delivery id is per account and key, so the key is posted once. The tenant log migration backfills keys created before keys went to the tenant log, and running it again does nothing. The outbox can't carry this statement (it only posts `Audit` bodies), so a small post-commit task does it instead. | | New tenants | Creating a tenant prepares its Keycloak realm for the tenant log whatever the realm template: `custom-event-listener`, admin events with details, and the `tenant-logs-read` / `tenant-logs-export` roles granted to the `admin` group. | | Monitoring | Every audit outbox worker pass writes `{"target":"audit_log_outbox","pending_entries":N,"oldest_pending_seconds":S}`, zeros included, so security monitoring can alert on entries waiting more than 5 minutes. This replaces the `waiting`/`oldest_seconds` names. It is documented in the developer page. | | Permissions | `/export-tenant-logs` needs `tenant-logs-read` as well as `tenant-logs-export` (403 otherwise). The portal shows the tenant log's Export only to people who have both. | | Fix | Tenant log exports could not be downloaded: `fetch-document` built a private document's S3 key from the event the caller named. The tenant log asks with an empty event, so the link pointed at `tenant-…/event-/document-…` (NoSuchKey). The key now comes from the document's own event. | | Docs | Settings > Logs reference: the popup's new sections, deleted events in account exports, the export permissions. Developer page: the backlog line and its alert threshold. Auditor tutorial: retired logs, which have no system key in the manifest. | ## Split with #3928 #3928 (VOTE-AUDIT, meta#14259) closes the "routes write no AdminChange" gap from #3902 for every Harvest route, including messaging accounts, certificate authorities, countries and deleting a tenant. It does this by letting the `audit_change()` trigger see Step's own writes. To avoid logging each change twice, this pull request adds no per-route entries. The two pull requests touch different files. ## Acceptance (meta#13781) Ticked where #3902 (merged) and this pull request together meet the criterion. Items 2, 3 and 5 also need #3928. - [x] Signing in to the Admin Portal, failing (wrong password, wrong authenticator code, unknown account), being locked out, setting up or removing an authenticator, resetting or changing a password and signing out each write one KeycloakUserEvent entry to the tenant log, with the client, IP address and authentication used. An unknown username is recorded as unknown. Token refreshes and code exchanges are not logged. *(#3902; this PR adds the listener to new tenants' realms)* - [ ] Every tenant-level change writes one AdminChange entry to the tenant log with the person, the object, the action and each changed field's old and new value, and no secret values. This covers the Settings tabs, the tenant record, election types, trustees, templates, staff accounts, roles and permissions, countries, backup and restore, and creating, importing, archiving, unarchiving and deleting election events. *(Imports now record the file's checksum and counts here. Messaging accounts, countries with old values and deleting a tenant are in #3928.)* - [ ] Event-level create, update, delete, import, configure, approve and schedule operations write AdminChange entries with old and new values to their event's log (baseline). *(Changes made through Hasura are covered since #3902; the Harvest routes are in #3928.)* - [x] A change made through Hasura to any table an admin role can write is logged in the same transaction. The coverage test fails when such a table has no trigger. *(#3902)* - [ ] A change whose transaction rolls back leaves no entry. A committed change always gets exactly one entry, even when immudb or the worker is down for a while: its entries wait and are posted in order afterwards. *(The outbox does this, and this PR moves AdminPublicKey to after commit. Certificate-authority routes still post their own entry after commit, on a best-effort basis; with #3928 they also get an AdminChange through the outbox.)* - [x] A change to the tenant realm made in Keycloak's console is logged as KeycloakAdminEvent, with the operator and the new value. Changes Step makes through its service account appear as SYSTEM entries next to the admin's AdminChange. *(#3902; new tenants' realms have admin events with details since this PR)* - [x] Windmill puts an event without an election event in the tenant log. An event it can't place becomes an ERROR entry and the rest of its batch is still posted. An unreadable message goes to the dead-letter queue, and later messages are processed. *(#3902)* - [x] Deleting an election event writes its tenant log entry, with the event log's last entry and state hash, before anything is removed. The event's log database is unloaded, not deleted. *(#3902; the entry also keeps the retired log's system key since this PR. Checked on the local stack.)* - [x] Settings shows the Logs tab only with tenant-logs-read, and its other tabs only with tenant-write: an Auditor sees only Logs, a Security Officer every tab, a Configuration Manager every tab but Logs, and an SBEI no Settings. Without tenant-logs-read, listElectoralLog and the exports refuse (403), including direct API calls. *(#3902; this PR adds the tenant-logs-read check to the tenant export)* - [x] Settings > Logs has the same header, date range, filters, columns, details popup and export menu as Election Event > Logs, and differs only in its subtitle, timezone, statement kinds and export scopes. *(#3902)* - [x] Times show in the tenant log's timezone (PhST for COMELEC) and in my time. Filters by username, statement kind, event type, log type and created range (inclusive, to the minute, with PhST boundary cases verified) return exactly the matching entries in the tenant log and in event logs. *(#3902)* - [x] Clicking an entry in either list opens a popup (the shared Dialog, not a side drawer) with who, when (the log's zone, my time and UTC), what, the old and new values, the sign-in details or the Keycloak change, the log and the signing keys; Close and Escape dismiss it. *(#3902; this PR adds the imported file and the deleted log's system key)* - [x] Exports reconcile exactly with the list for the same scope and range. They cover the tenant log (whole, or one account with or without its election event entries) and event logs (All Posts, one Post, one country, one account), over a date range, as CSV, SQL or PDF. Each export writes LogsExported with the file's SHA-256. The event export checks logs-export and the tenant export checks tenant-logs-export. *(#3902. This PR adds deleted events' logs to one account's export and fixes the tenant export's download. CSV was checked on the local stack; PDF was not, because it does not render in the dev image.)* - [x] The SQL export loads into PostgreSQL, and every entry's signatures verify with the exported public keys. *(#3902; test `postgres_audit_log_outbox`)* - [x] Every posted entry appears as one JSON line in Windmill's output, without old or new values or secrets. The outbox's waiting time is available to VOTE-SECOPS's alerts. *(#3902; this PR writes the waiting time on every worker pass)* - [x] The one-off task gives every existing tenant its log and key, and AdminPublicKey entries are written to the tenant log. *(#3902; this PR posts keys after commit and backfills keys made earlier)* - [x] The second organization's tenant log works by configuration only (its timezone, accounts and labels); no code refers to COMELEC. *(#3902; nothing in this PR names a client)* - [ ] Apply every linked source requirement, preserving its exact targets, alternatives and lifecycle stage. This package estimate is shared and must not be summed per row. *(A rule for the whole package, not something one pull request can tick)* ## Not in this pull request - EMS-SBEI-QUORUM's signing entries still use their own `signing_log_outbox`. - Tenant log checkpoints and an audit task for the tenant log (VOTE-AUDIT). - Erasing or reopening a retired event database (OPS-RETENTION). - Beyond's client-setup chart still configures only `jboss-logging` for the tenant realm. Existing realms are set up by the one-off task, and new tenants by this pull request. ## Upgrade notes - Monitoring that parsed the outbox line's `waiting`/`oldest_seconds` must read `pending_entries`/`oldest_pending_seconds`. - Roles that exported the tenant log with only `tenant-logs-export` need `tenant-logs-read` too. ## Tests - electoral-log: 124 tests with the database tests (`--include-ignored`), including reading a retired database and closing it again. - windmill: 1705 lib tests pass. `decrypted_archives_use_private_owned_files_and_leave_no_plaintext` needs an `openssl` binary on the host, and passes with one. `postgres_audit_log_outbox` (8, including the post-commit key announcement, the backfill and retired events in exports) and `postgres_audit_change` (18) pass. - harvest: 561 pass, including the new 403 for an export without read permission. - admin-portal: 2233 jest tests, `LogEntryDialog` stories (17, with a new import story), story inventory, `typecheck:stories`, and the journeys `settings/logs`, `settings/access` and `events/operations` (29). - End to end on a local stack built from this branch (Keycloak, Hasura, Harvest, Windmill, PostgreSQL, MinIO, production portals), using the backend E2E bootstrap. All 11 UI journeys pass. The stack also checked, by hand: the import entry's checksum and counts; retired databases read and closed again; the account export's download and its retired rows; a new tenant's realm settings and roles; the backlog line. That run found the export download bug fixed here. Not measured locally: the per-package coverage gates. ## Demo Video with voiceover, to upload to meta#13781: import, then an event edit, deletion, the account export with the retired log, a new tenant's realm, and the backlog line. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
ereslibre
added a commit
that referenced
this pull request
Oct 17, 2026
…DIT, SW-F-0364) (#3928) Parent issue: sequentech/meta#14259 Requirement [SW-F-0364](https://project-1.sequent.sh/#requirement/SW-F-0364): log all system activity from start to finish. #3900 and #3902 log what a person changes through Hasura, accounts and roles, Keycloak events and the tenant-level actions. #3902's "Not in this pull request" left the rest open: most event-level routes (areas, elections, dates, voting status, ceremonies, tallies, publications, scheduled events, reports, applications, the approval matrix, monitoring, signing settings) write with Harvest's or Windmill's own SQL, which `audit_change()` skipped, so they wrote no `AdminChange`. This pull request closes that gap. > **Scope agreed with EMS-AUDIT (meta#13781):** this pull request carries the route coverage. The tenant deletion and country-limit entries come from that session's work; its own pull request keeps the rest of EMS-AUDIT. ## What changes | Area | Change | | --- | --- | | Database | Migration `1791800000000_audit_changes_step_makes`: `audit_change()` also logs the writes of a transaction whose admin Step names in the setting `sequent.audit_actor` (`{user_id, username, role, origin, tenant_id}`), with old and new values; a Hasura session still takes precedence. The trigger on the tables only Step writes for an admin: `keys_ceremony`, `messaging_account`, `messaging_event_config`, `monitoring_config`, `signing_checks`, `staff_certificate`, `tally_results_publication`, `tally_sheet_import`, `tally_sheet_import_item`. `webhook_key` is a secret (SQL and Rust rules) | | Windmill | `audit_log::act_for(&tx, &actor, origin, tenant_id)`. Results publication (publish, revoke, website policy), monitoring settings (`save_as`, `reset_to_preset_as`, `set_mode_as`) and a signing approval (as the signer whose signature runs the action) name their admin. `insert_tenant` takes the creating admin | | Harvest | `audit::act_for(&tx, &claims)` right after the transaction opens in every state-changing admin route: areas, elections, election dates and schedule recompute, keys and tally ceremonies, trustee custody, ballot publication, voting status, tally sheets and their imports, signing certificates, issuers, checks and rules, messaging accounts and settings, applications, scheduled events, lifecycle policies, uploads and previews, certificate authority import and delete | | Harvest, own entries | Where the trigger cannot see the change: area and candidate imports (one entry with the document and declared checksum), export requests (event, tally results, ballot publication, applications, tasks, trustees, certificate authorities, reports), the approval matrix (old and new version), a report's password set (named, never recorded), a document's password revealed, an electoral-log console query, a voter information letter (it resets the voter's password), the country limits before and after | | Windmill, own entries | Deleting a tenant: an `AdminChange` staged in the deleting transaction, in the log of the tenant whose admin deleted it (the route passes the admin as an optional task input) | | Guard | `harvest/tests/support/audited_routes.rs` fails when a route file commits a transaction without naming its admin or staging an entry. Exemptions, with their reasons, in `tests/fixtures/unaudited-committing-routes.txt` (voting portal routes, reads that commit, callbacks) | | Docs | Developer page "Tenant log and audit entries": sections 5, 6.1, 6.2 and 13 | Also fixed on the way: - `/encrypt-report` logged the report's password (`info!("body …")` and the tracing span). - `/miru/upload-signature` recorded its input, password included, in its tracing span. - `/import-candidates` wrote its task row before checking the permission; it now checks first (its `route_permissions` case is updated). - `/delete-election-event` passes the admin's role like every other route. ## Not in this pull request - What a Windmill task writes on its own after the route answered (ballot styles of a publication, a tally's execution, documents of reports and letters, reconciliation patches) has no admin; the route's entry records the decision. - Keycloak changes when an application is confirmed or rejected are recorded only by Keycloak's admin events. - `/fetch-document` downloads write no entry. ## Tests - windmill: the whole suite, 3088 pass, including `postgres_audit_change` (24 tests; 6 new: an admin named by Harvest or Windmill, the setting ending with the transaction, Hasura's precedence, the new tables, a keys ceremony, `webhook_key`). - harvest: 570 pass, including the route guard, `act_for` on the migrated test database, import and export entries, and the route permission inventory. - hasura: migrations apply, roll back and reapply to an identical schema; 115 metadata tests pass. - `reuse lint` passes. ## Demo A narrated video of the feature on a local stack is attached to meta#14259. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Parent issue: https://github.com/sequentech/meta/issues/13781
Design package EMS-AUDIT. Every administrative action is logged, including the ones that happen outside an election event. Each tenant gets a signed log of its own, shown in Settings > Logs, and every change carries its old and new values, in the tenant log and in each event's log.
What changes
Auditstatement body with five kinds (AdminChange,KeycloakAdminEvent,LogsExported,LogWriterStarted,LogWriterStopped). A log database per tenant (<base>_t<tenant>). A deleted event's database is retired (renamed, closed to connections), not dropped.event_typeandlog_typecolumns so lists filter by themaudit_log_outbox, and theaudit_change()trigger on the 26 tables an admin role can write through Hasura. It reads the Hasura session, diffs JSON key by key, redacts secrets and writes in the same transaction. A coverage test fails when such a table has no triggerLogsExportedwith the file's SHA-256. One JSON line per posted entry for monitoring. One-off task for existing tenants (log, key, realm listener, the two roles)listElectoralLogwithout an election event reads the tenant log (needstenant-logs-read, 403 otherwise).export_tenant_logs(needstenant-logs-export). The event export checkslogs-exportinstead ofreport-write. Audit entries in the routes for staff accounts, roles, permissions, phone blacklist, realm attributes, password policy, voter authentication and the Google service account keycustom-event-listenerhandles the tenant realm (sign-ins, failures, lockouts, credential changes; no token refreshes) and admin events, with secrets redacted. An unknown account is recorded without the name typedtenant-logs-read. Entry popup with who, when, what, old and new values, sign-in or Keycloak details and signatures, in both logs. Export scopes and SQL. "Timezone for tenant logs" in Languages. An auditor with only the log's permissions sees only LogsNot in this pull request
signing_log_outbox.AdminChangeyet (areas, elections, dates, ceremonies, tally, publications, scheduled events, reports). Changes made to those tables through Hasura are covered by the trigger.Upgrade notes
electoral-log-admin upgrade-event-databasesbefore the new Windmill and Harvest start: appends write the newevent_typeandlog_typecolumns.report-writeneedlogs-export.Tests
postgres_audit_log_outbox(6 tests: signing, exactly-once posting, ordering after an outage, rerouting, deletion, and an SQL export loaded into PostgreSQL with every signature verified);postgres_audit_change(18 tests of the trigger).Failing on
ovcsitself, not caused by this change:migrate_realm_permissions::the_migration_adds_the_signing_and_messaging_permissions_with_their_labels(portal labels and migration labels differ since #3885), journeysevents.spec.ts:215and the/trusteeroute smoke, and Hasura metadata uses an undeclared enumRestorePrivateKeyOutcome. PDF export does not render in the dev image (its Chromium crashes with the flagssequent-corepasses); CSV and SQL do.🤖 Generated with Claude Code