From 236b680b038ba93e9dc90c8abbba62fa3429fa26 Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Wed, 5 Aug 2026 23:39:49 +0500 Subject: [PATCH 001/194] docs: add architecture (SAD), fork strategy, and auth design Baseline documentation for the Clustox fork of middlewarehq/middleware. - ARCHITECTURE.md: system architecture covering the container and component model, data model, sync pipeline, DORA metric derivations, and known constraints. Every structural claim is reproducible via the commands in Appendix A. - FORK_STRATEGY.md: upstream maintenance strategy -- remotes, branch model, the three-tier rule for where Clustox changes may live, the monthly sync runbook, and Apache 2.0 obligations. - UPSTREAM_SYNC_LOG.md: ledger for recording each upstream merge. - NOTICE: Apache 2.0 s4(d) attribution, absent upstream and required once a modified version is distributed. - superpowers/specs + plans: design and implementation plan for authentication and role-based access control. Two findings documented in detail: - Two independent write paths reach the same database. The Next.js BFF goes directly to Postgres via knex in 23 files rather than through the Flask API, with its own hand-maintained schema copy in TypeScript. - There is no authentication anywhere in the stack. next-auth is a dependency but is never configured: no route, no sign-in page. Also documents the sync bookmark behaviour (ARCHITECTURE.md 7.5), where a newly added repo silently syncs no history while the UI shows "Calculating Dora" indefinitely. --- NOTICE | 54 + docs/ARCHITECTURE.md | 1133 +++++++++ docs/FORK_STRATEGY.md | 448 ++++ docs/UPSTREAM_SYNC_LOG.md | 41 + .../superpowers/plans/2026-08-05-auth-rbac.md | 2049 +++++++++++++++++ .../specs/2026-08-05-auth-rbac-design.md | 322 +++ 6 files changed, 4047 insertions(+) create mode 100644 NOTICE create mode 100644 docs/ARCHITECTURE.md create mode 100644 docs/FORK_STRATEGY.md create mode 100644 docs/UPSTREAM_SYNC_LOG.md create mode 100644 docs/superpowers/plans/2026-08-05-auth-rbac.md create mode 100644 docs/superpowers/specs/2026-08-05-auth-rbac-design.md diff --git a/NOTICE b/NOTICE new file mode 100644 index 00000000..db447077 --- /dev/null +++ b/NOTICE @@ -0,0 +1,54 @@ +Middleware +========== + +This product includes software developed by Middleware HQ and the Middleware +open-source contributors. + + Project: Middleware — open-source engineering management / DORA metrics + Source: https://github.com/middlewarehq/middleware + Homepage: https://www.middlewarehq.com/ + Licence: Apache License, Version 2.0 + +Licensed under the Apache License, Version 2.0 (the "License"); you may not use +this file except in compliance with the License. You may obtain a copy of the +License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software distributed +under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR +CONDITIONS OF ANY KIND, either express or implied. See the License for the +specific language governing permissions and limitations under the License. The +full licence text is included in the LICENSE file at the root of this +repository. + + +Modifications +------------- + +This repository is a fork of middlewarehq/middleware maintained by Clustox. + + Fork: https://github.com/Clustox/middleware + Fork point: commit 844eb42 on branch main + +Modifications made by Clustox are marked in source files with `CLUSTOX:` +sentinel comments. The complete set of modifications can be listed with: + + grep -rn "CLUSTOX:" --include='*.py' --include='*.ts' --include='*.tsx' \ + --include='*.sql' . | grep -v node_modules + +and the full diff from the fork point with: + + git diff 844eb42..main + +See docs/FORK_STRATEGY.md for how this fork is maintained. + + +Trademarks +---------- + +The Apache License, Version 2.0 does not grant permission to use the trade +names, trademarks, service marks, or product names of the licensor. "Middleware" +and "Middleware HQ", together with associated logos and brand assets (including +those under media_files/), remain the property of their respective owners and +are used here solely to attribute the origin of this software. diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md new file mode 100644 index 00000000..a5038604 --- /dev/null +++ b/docs/ARCHITECTURE.md @@ -0,0 +1,1133 @@ +# Software Architecture Document — Middleware (Clustox fork) + +| | | +|---|---| +| **System** | Middleware — open-source DORA metrics platform | +| **Upstream** | [middlewarehq/middleware](https://github.com/middlewarehq/middleware) (Apache-2.0) | +| **This fork** | [Clustox/middleware](https://github.com/Clustox/middleware) | +| **Baseline commit** | `844eb42` (`main`) | +| **Status** | Living document — update in the same PR as any architectural change | +| **Audience** | Engineers onboarding to the fork; reviewers of proposed changes | + +> **Scope.** This describes the system **as it exists in the code at the baseline commit**, not as it is +> aspirationally described in the upstream README. Where the implementation diverges from the marketing +> description, the implementation is documented and the gap is flagged in +> [§10 Constraints, risks and gaps](#10-constraints-risks-and-gaps). +> +> Every claim here is traceable to a file path. If you change the code and the claim stops being true, +> the doc is now a bug — fix it in the same PR. + +--- + +## Table of contents + +1. [Purpose and context](#1-purpose-and-context) +2. [System context (C4 L1)](#2-system-context-c4-l1) +3. [Container view (C4 L2)](#3-container-view-c4-l2) +4. [Runtime and process model](#4-runtime-and-process-model) +5. [Backend component view (C4 L3)](#5-backend-component-view-c4-l3) +6. [Data model](#6-data-model) +7. [The sync pipeline — the heart of the system](#7-the-sync-pipeline--the-heart-of-the-system) +8. [How the four DORA metrics are derived](#8-how-the-four-dora-metrics-are-derived) +9. [Frontend architecture](#9-frontend-architecture) +10. [Cross-cutting concerns](#10-cross-cutting-concerns) +11. [Constraints, risks and gaps](#11-constraints-risks-and-gaps) +12. [Extension points](#12-extension-points) +13. [Glossary](#13-glossary) + +--- + +## 1. Purpose and context + +Middleware ingests engineering activity from Git providers and CI systems, stores it in a normalised +relational model, and computes the four [DORA metrics](https://dora.dev) per **team** over a +configurable time interval: + +| Metric | Meaning | +|---|---| +| Deployment Frequency | How often code reaches production | +| Lead Time for Changes | First commit → deployed | +| Mean Time to Restore (MTTR) | Incident created → resolved | +| Change Failure Rate (CFR) | % of deployments that caused an incident | + +**Design intent (inferred from the code, not stated upstream):** this is a *single-tenant, self-hosted* +appliance. One Postgres, one implicit organisation, no user-facing tenancy. Understanding that intent +explains most of the choices that otherwise look surprising — no auth on the analytics API, one container +running six processes, a hardcoded `default` org. + +The bootstrap of that single org is explicit in +[`initialise_db.py`](../backend/analytics_server/mhq/store/initialise_db.py): on first boot the app +creates exactly one `Organization` named `default`, and every request resolves back to it via +`get_query_validator().get_default_org()`. + +--- + +## 2. System context (C4 L1) + +```mermaid +graph TB + subgraph external["External systems"] + GH["GitHub
REST + GraphQL"] + GL["GitLab
REST"] + GHA["GitHub Actions
workflow runs"] + LLM["LLM provider
optional, for AI summaries"] + end + + subgraph mw["Middleware (self-hosted appliance)"] + SYS["DORA analytics
ingest · store · compute · visualise"] + end + + EM["Engineering manager /
team lead"] + ADMIN["Operator
installs, holds the PAT"] + + ADMIN -->|"configures integration, PAT,
selects repos & teams"| SYS + EM -->|"views dashboards"| SYS + + SYS -->|"pulls repos, PRs, reviews,
commits, workflow runs"| GH + SYS -->|"pulls projects, MRs, notes"| GL + SYS -->|"pulls run status & duration"| GHA + SYS -.->|"sends aggregated metrics
for narrative summary"| LLM + + style SYS fill:#4a6fa5,stroke:#2c4a6f,color:#fff + style LLM stroke-dasharray: 4 4 +``` + +**Integration direction is exclusively outbound-pull.** There are no inbound webhooks anywhere in the +codebase. Everything arrives because a cron-triggered sync went and fetched it. Two consequences worth +internalising: + +- Data is **stale by up to 30 minutes** by design (the cron interval). +- The system needs **no ingress from the internet** — which is the main reason its lack of API auth has + not been catastrophic upstream. + +--- + +## 3. Container view (C4 L2) + +```mermaid +graph TB + USER["Browser"] + + subgraph container["Single Docker container — middleware-dev"] + direction TB + + FE["Next.js web server
:3333
pages · SSR · BFF API routes
React 18 · MUI 5 · Redux Toolkit"] + AN["Flask analytics server
:9696
app.py — 9 blueprints
read-mostly: metrics & queries"] + SY["Flask sync server
:9697
sync_app.py — 1 endpoint
write-heavy: the ETL"] + CRON["cron
*/30 * * * *
POST :9697/sync"] + PG[("PostgreSQL 15
:5434
db: mhq-oss")] + RD[("Redis
:6385
distributed locks only")] + SUP["supervisord — PID 1, starts all of the above by priority"] + end + + EXT["GitHub / GitLab APIs"] + + USER -->|":3333 HTTP"| FE + FE -->|"axios · INTERNAL_API_BASE_URL"| AN + FE -->|"knex · direct SQL ⚠️"| PG + FE -->|"direct provider calls
for integration setup"| EXT + CRON --> SY + AN --> PG + AN --> RD + SY --> PG + SY --> RD + SY --> EXT + + style FE fill:#3d5a80,color:#fff + style AN fill:#4a6fa5,color:#fff + style SY fill:#5c8001,color:#fff + style PG fill:#336791,color:#fff + style RD fill:#a41e11,color:#fff + style SUP fill:#555,color:#fff +``` + +### 3.1 Container responsibilities + +| Container | Tech | Entry point | Responsibility | +|---|---|---|---| +| Web server | Next.js (pages router), TS, MUI 5, Redux Toolkit | [`web-server/`](../web-server/) | UI, SSR, and a BFF layer under `pages/api/**` | +| Analytics server | Flask 3.0, SQLAlchemy 2.0, gunicorn | [`app.py`](../backend/analytics_server/app.py) | Serves computed metrics. Read-mostly. | +| Sync server | Flask 3.0, same ORM | [`sync_app.py`](../backend/analytics_server/sync_app.py) | Runs the ETL. Write-heavy. Deliberately a separate process. | +| Database | PostgreSQL 15 | [`database-docker/db/`](../database-docker/db/) | Single source of truth. Schema via `dbmate`. | +| Redis | Redis (no persistence configured) | — | **Distributed locks only.** Not a cache. | +| Process manager | supervisord | [`supervisord.conf`](../setup_utils/supervisord.conf) | PID 1; ordered startup, restart policy | +| CLI | Node 22, Ink (React for terminals) | [`cli/`](../cli/) | Wraps `docker compose` for a friendly local install | + +### 3.2 ⚠️ The most important architectural fact in this document + +**There are two independent write paths to the same database, and they do not know about each other.** + +```mermaid +graph LR + subgraph paths["Two paths to one schema"] + P1["Flask backend
SQLAlchemy models
mhq/store/models/*.py"] + P2["Next.js BFF
knex query builder
web-server/src/utils/db.ts"] + end + PG[("PostgreSQL
mhq-oss")] + P1 -->|"ORM"| PG + P2 -->|"raw SQL builder"| PG + + style P1 fill:#4a6fa5,color:#fff + style P2 fill:#3d5a80,color:#fff + style PG fill:#336791,color:#fff +``` + +The Next.js API routes do **not** proxy everything to Flask. **23 files** under `web-server/` import +`@/utils/db` and talk to Postgres directly through knex with its own connection pool +([`web-server/src/utils/db.ts`](../web-server/src/utils/db.ts)). Table and column names are +re-declared by hand in TypeScript in `web-server/src/constants/db.ts`. + +Reproduce the list yourself: + +```bash +grep -rl "from '@/utils/db'" web-server/pages/api web-server/src +``` + +Examples of direct-SQL writes, not proxied through the API: +[`team_repos.ts`](../web-server/pages/api/resources/team_repos.ts) deactivates rows in `TeamRepos` with +a knex `UPDATE`; [`onboarding.ts`](../web-server/pages/api/resources/orgs/%5Borg_id%5D/onboarding.ts) and +[`integration.ts`](../web-server/pages/api/resources/orgs/%5Borg_id%5D/integration.ts) do likewise. + +**Why this matters for every future change we make:** + +1. **A schema change is a three-place change.** The `dbmate` migration, the SQLAlchemy model, *and* the + TypeScript `Columns`/`Table` declarations. Miss the third and you get a runtime failure in the BFF + that no Python test will catch. +2. **Business logic can be bypassed.** Validation living in a Flask service does not protect a write + that arrives via knex. +3. **Two connection pools** contend for the same Postgres: SQLAlchemy `pool_size=10, max_overflow=5` + per Flask process ([`store/__init__.py`](../backend/analytics_server/mhq/store/__init__.py)) and + knex `max=7` in non-dev ([`db.ts`](../web-server/src/utils/db.ts)). + +This is not a defect to fix casually — it is load-bearing, and unpicking it is a project of its own. +It is documented here so nobody discovers it the hard way. See +[§12 Extension points](#12-extension-points) for the practical checklist. + +--- + +## 4. Runtime and process model + +Everything runs in **one container** under supervisord. Postgres, Redis, two Python servers, Next.js, +and cron are siblings in a single failure domain. + +```mermaid +graph TB + S["supervisord (PID 1, nodaemon)"] + S -->|"priority 1"| P1["postgres
autorestart=false ⚠️"] + S -->|"priority 2"| P2["initialize_db
dbmate up · autorestart=false"] + S -->|"priority 3"| P3["redis
autorestart=true"] + S -->|"priority 4"| P4["backend :9696
autorestart=true"] + S -->|"priority 5"| P5["backend_sync :9697
autorestart=true"] + S -->|"no priority"| P6["frontend :3333
autorestart=true"] + S -->|"no priority"| P7["cron
autorestart=true"] + + style P1 fill:#8b0000,color:#fff + style P2 fill:#a0522d,color:#fff +``` + +Read from [`setup_utils/supervisord.conf`](../setup_utils/supervisord.conf). Notes that matter +operationally: + +- **`postgres` has `autorestart=false`.** If Postgres dies, supervisord will *not* bring it back, and + the container keeps running with every other process failing its queries. The container looks alive + and the app is dead. This is the first thing to check when someone reports "it's up but broken". +- **Each process is individually switchable** via env flags (`POSTGRES_DB_ENABLED`, `REDIS_ENABLED`, + `BACKEND_ENABLED`, `FRONTEND_ENABLED`, `CRON_ENABLED`). This is the seam to exploit when splitting + the container for production — set the flag false and point the env var at an external service. +- **Every log is capped at 512 KB with `logfile_backups=0`** — rotation discards the old file. There is + no log history beyond half a megabyte per process. Anything resembling production needs log shipping. +- Logs live at `/var/log/{postgres,init_db,redis,apiserver,sync_server,web-server,cron}/`. The UI + surfaces them at `/system-logs` ([`pages/system-logs.tsx`](../web-server/pages/system-logs.tsx)). + +### 4.1 Ports + +| Port | Process | Bound to | +|---|---|---| +| 3333 | Next.js | `127.0.0.1` | +| 9696 | Flask analytics | `127.0.0.1` | +| 9697 | Flask sync | `127.0.0.1` | +| 5434 | Postgres | `127.0.0.1` | +| 6385 | Redis | `127.0.0.1` | + +All five are bound to loopback on the host by +[`docker-compose.yml`](../docker-compose.yml) (`"127.0.0.1:${PORT}:${PORT}"`). **This loopback binding +is the only thing standing between the unauthenticated analytics API and the network.** See +[§11.1](#111-no-authentication-on-the-backend-apis). + +### 4.2 Schema migrations + +`dbmate`, not Alembic — despite the backend being SQLAlchemy. +[`init_db.sh`](../setup_utils/init_db.sh) waits for Postgres, creates the database if absent, then runs +`dbmate -u "$DB_URL" up` against +[`database-docker/db/migrations/`](../database-docker/db/migrations/). + +There are only **4** migration files, the earliest named `20240404142732_init.sql`. The full schema +lives in [`database-docker/db/schema.sql`](../database-docker/db/schema.sql). Two implications: + +- SQLAlchemy models are **hand-kept in sync** with SQL migrations. Nothing auto-generates or verifies + the correspondence — no `alembic revision --autogenerate`, no drift check in CI. A model/schema + mismatch is caught only at runtime. +- Adding a migration means writing SQL by hand and updating the model by hand. Adding a **drift check** + to CI is one of the highest-value, lowest-risk hardening changes available to us. + +--- + +## 5. Backend component view (C4 L3) + +The backend is cleanly layered, and this consistency is its best quality — it makes the codebase far +more tractable than its size suggests. + +```mermaid +graph TB + subgraph api["mhq/api/ — HTTP boundary"] + A1["pull_requests · incidents
deployment_analytics
teams · settings · integrations
bookmark · sync · ai/dora_ai"] + A2["request_utils.py
voluptuous schema validation"] + A3["resources/
ORM → JSON adapters"] + end + + subgraph svc["mhq/service/ — business logic"] + S1["code/
lead_time · pr_analytics
sync/ ETL handlers"] + S2["deployments/
factory · analytics
pr_ vs workflow_ strategies"] + S3["incidents/
MTTR · CFR"] + S4["workflows/
CI run ingest"] + S5["merge_to_deploy_broker/
joins merge → deploy"] + S6["settings/ · bookmark/ · ai/"] + S7["query_validator.py
resolves default org & team"] + end + + subgraph store["mhq/store/ — persistence"] + R1["repos/
code · core · incidents
integrations · settings · workflows"] + M1["models/
SQLAlchemy declarative"] + end + + subgraph ex["mhq/exapi/ — external clients"] + E1["github.py — PyGithub + raw GraphQL"] + E2["gitlab.py"] + E3["git_incidents.py"] + end + + subgraph util["mhq/utils/"] + U1["cryptography.py — RSA
lock.py — Redis locks
time.py — Interval, buckets
log.py · dict.py · regex.py"] + end + + api --> svc --> store + svc --> ex + svc --> util + store --> util + + style api fill:#4a6fa5,color:#fff + style svc fill:#5c8001,color:#fff + style store fill:#336791,color:#fff + style ex fill:#8b5a2b,color:#fff +``` + +**The layering rule, stated plainly:** `api → service → store/repos → store/models`, with `exapi` +reachable only from `service`. No layer reaches backwards. Every future change should preserve this — +it is the property that makes the code navigable. + +### 5.1 Dependency injection convention + +Services take their collaborators as constructor arguments and are assembled by module-level +`get_*_service()` factory functions: + +```python +# mhq/service/code/sync/etl_handler.py +code_etl_handler = CodeETLHandler( + CodeRepoService(), + etl_factory(provider), # provider-specific strategy + get_merge_to_deploy_broker_utils_service(), + get_bookmark_service(), + get_settings_service(), +) +``` + +Follow this pattern in new code. It is why the test suite can substitute fakes without a DI framework, +and it is what makes the provider-strategy pattern in §12.1 work. + +### 5.2 Blueprint registration + +Nine blueprints on the analytics server, two on the sync server +([`app.py`](../backend/analytics_server/app.py), [`sync_app.py`](../backend/analytics_server/sync_app.py)). +`hello` is registered on both as a health endpoint. Adding an API surface means creating a blueprint +module and registering it in the relevant entry point — there is no auto-discovery. + +### 5.3 Selected endpoints + +Generate the current list with: + +```bash +grep -rn -A1 '@app.route' backend/analytics_server/mhq/api/ +``` + +| Method | Path | Purpose | +|---|---|---| +| GET | `/teams//lead_time` | Lead time, aggregate | +| GET | `/teams//lead_time/trends` | Lead time, weekly buckets | +| GET | `/teams//lead_time/prs` | Contributing PRs (drill-down) | +| GET | `/teams//deployment_frequency` | Deployment frequency | +| GET | `/teams//deployment_analytics` | Deployments + related PRs | +| GET | `/teams//mean_time_to_recovery` | MTTR | +| GET | `/teams//change_failure_rate` | CFR | +| GET | `/teams//deployments_with_related_incidents` | Deploy↔incident join | +| GET/PUT | `/teams//settings` | Team settings | +| GET/PUT | `/orgs//settings` | Org settings | +| GET | `/orgs//integrations/{github,gitlab}/...` | Repo/org discovery | +| PUT | `/orgs//bookmark/reset` | **Force full re-sync** | +| POST | `/ai/dora_score`, `/ai/*_trends` | LLM narrative summaries | +| POST | `/sync` | Trigger ETL *(sync server, :9697)* | + +Note the shape: almost every analytics route is `GET /teams//` with an interval and +filter in the query string, and most have a paired `/trends` variant. New metrics should follow it. + +--- + +## 6. Data model + +Tables use `PascalCase` — unusual for Postgres, and it means **every raw SQL reference needs double +quotes** (`SELECT * FROM "PullRequest"`). Expect to be bitten by this once. + +```mermaid +erDiagram + Organization ||--o{ Users : has + Organization ||--o{ Team : has + Organization ||--o{ OrgRepo : has + Organization ||--o{ Integration : has + Organization ||--o{ OrgIncidentService : has + + Team ||--o{ TeamRepos : selects + Team ||--o{ TeamIncidentService : watches + OrgRepo ||--o{ TeamRepos : "mapped to" + OrgRepo ||--o{ PullRequest : contains + OrgRepo ||--o{ RepoWorkflow : has + OrgRepo ||--o{ RepoSyncLogs : logs + + PullRequest ||--o{ PullRequestEvent : "reviews, comments" + PullRequest ||--o{ PullRequestCommit : contains + PullRequest ||--o{ PullRequestRevertPRMapping : "reverted by" + + RepoWorkflow ||--o{ RepoWorkflowRuns : "runs" + RepoWorkflow ||--o{ RepoWorkflowRunsBookmark : "sync cursor" + + OrgIncidentService ||--o{ IncidentOrgIncidentServiceMap : maps + Incident ||--o{ IncidentOrgIncidentServiceMap : maps + + Organization { + UUID id PK + string name "always 'default' in OSS" + string domain + } + Team { + UUID id PK + UUID org_id FK + UUID_array member_ids "array column, not a join table" + boolean is_deleted "soft delete" + } + OrgRepo { + UUID id PK + string provider "github or gitlab" + string org_name + string default_branch + jsonb contributors + boolean is_active + } + TeamRepos { + UUID team_id PK,FK + UUID org_repo_id PK,FK + string_array prod_branches "regex — defines production" + enum deployment_type "PR_MERGE or WORKFLOW" + } + PullRequest { + UUID id PK + UUID repo_id FK + enum state "OPEN, CLOSED or MERGED" + int first_commit_to_open "precomputed segment" + int first_response_time "precomputed segment" + int rework_time "precomputed segment" + int merge_time "precomputed segment" + int merge_to_deploy "backfilled by MTD broker" + int cycle_time + int rework_cycles + jsonb meta "code_stats, user_profile" + } + RepoWorkflowRuns { + UUID id PK + enum status "SUCCESS, FAILURE, etc" + datetime conducted_at "equals deployment time" + int duration + string head_branch + } + Incident { + UUID id PK + string key "dedup key" + datetime creation_date + datetime resolved_date + enum incident_type + } + Integration { + UUID org_id PK,FK + string name PK + string_array access_token_enc_chunks "RSA-encrypted chunks" + jsonb provider_meta + } +``` + +### 6.1 Three modelling decisions worth understanding + +**1. Lead-time segments are precomputed and stored on the row.** +`PullRequest.first_commit_to_open`, `first_response_time`, `rework_time`, `merge_time`, and +`merge_to_deploy` are integer **seconds**, written during ETL rather than derived at query time +([`models/code/pull_requests.py`](../backend/analytics_server/mhq/store/models/code/pull_requests.py)). + +This makes reads fast and writes authoritative — and it means **changing a metric definition does not +change historical data**. Old rows keep their old numbers. Any change to how a segment is computed +requires a backfill or a bookmark reset (`PUT /orgs//bookmark/reset`) to be visible in +history. This is the single most common way to get a confusing result after a "simple" metric change. + +**2. `TeamRepos.prod_branches` is where "production" is defined.** +A `string[]` of branch regexes, per team, per repo. There is no environment concept, no deploy target +model. "Production" means "matched one of these branch patterns". Everything downstream — +deployment frequency, lead time, CFR — inherits this definition. + +**3. `TeamRepos.deployment_type` selects the deployment strategy per repo.** +`PR_MERGE` (a merge to a prod branch *is* the deployment) or `WORKFLOW` (a successful CI run is the +deployment). This enum is the switch that picks a strategy class at query time — see §8.1. + +### 6.2 Incremental sync cursors ("bookmarks") + +Four separate cursor tables, so a failure in one stream does not rewind the others: + +| Table | Cursor for | +|---|---| +| `Bookmark` | PR sync, keyed `(repo_id, type)` | +| `BookmarkMergeToDeployBroker` | Merge-to-deploy backfill position | +| `RepoWorkflowRunsBookmark` | CI workflow run sync | +| `IncidentsBookmark` | Incident sync | + +Cursors are stored as **ISO-8601 strings**, not timestamps. On first sync there is no cursor, so the +window falls back to the `DEFAULT_SYNC_DAYS` setting (31 in [`env.example`](../env.example)) — +resolved via `SettingType.DEFAULT_SYNC_DAYS_SETTING`, so the DB setting wins over the env default. + +--- + +## 7. The sync pipeline — the heart of the system + +If you read only one section, read this one. Almost every bug and every feature request lands here. + +### 7.1 Orchestration + +[`mhq/service/sync_data.py`](../backend/analytics_server/mhq/service/sync_data.py) is the whole +orchestrator — four functions run in a fixed order: + +```python +sync_sequence = [ + sync_code_repos, # 1. repos, PRs, reviews, commits + sync_org_workflows, # 2. CI workflow runs + process_merge_to_deploy_cache, # 3. join merges to deploys + sync_org_incidents, # 4. incidents +] +``` + +**The order is a real dependency, not a preference.** Step 3 needs both PRs (step 1) and workflow runs +(step 2) already in the database. Step 4's PR-derived incidents need step 1. Reordering silently +produces incomplete metrics rather than an error. + +### 7.2 Sequence of a sync run + +```mermaid +sequenceDiagram + participant C as cron (*/30) + participant S as sync server :9697 + participant R as Redis + participant E as ETL handlers + participant G as GitHub / GitLab + participant D as Postgres + + C->>S: POST /sync + S->>S: resolve default org + S->>R: acquire lock {org}::data_sync + Note over R: serialises concurrent syncs + + rect rgb(238,246,238) + Note over E,D: 1. sync_code_repos + S->>E: per code provider + E->>G: check_pat_validity() + E->>D: read active OrgRepos + E->>G: fetch repo metadata + E->>D: read Bookmark (else now - DEFAULT_SYNC_DAYS) + E->>G: fetch PRs updated since cursor + E->>E: compute lead-time segments + E->>D: upsert PR + commits + events + E->>D: advance Bookmark to max(updated_at) + E->>D: save revert-PR mappings + end + + rect rgb(238,242,250) + Note over E,D: 2. sync_org_workflows + E->>G: fetch workflow runs since cursor + E->>D: upsert RepoWorkflowRuns + end + + rect rgb(252,246,232) + Note over E,D: 3. merge_to_deploy broker + S->>R: lock {org_repo}::merge_to_deploy_broker + E->>D: successful runs after cursor (max 500) + E->>D: PRs merged before run, merge_to_deploy IS NULL + E->>E: merge_to_deploy = conducted_at - state_changed_at + E->>D: UPDATE PullRequest + end + + rect rgb(250,240,240) + Note over E,D: 4. sync_org_incidents + E->>G: fetch incidents + E->>D: upsert Incident + service map + end + + S->>R: release lock + S-->>C: {"message": "sync started"} +``` + +### 7.3 The merge-to-deploy broker, explained + +This component confuses everyone at first, so: **a PR does not know when it was deployed at the moment +it is merged.** The deployment happens later. So `merge_to_deploy` is written as `NULL` at PR-sync time +and **backfilled on a later sync run**, once a successful workflow run exists that contains the merge +commit. + +From [`mtd_handler.py`](../backend/analytics_server/mhq/service/merge_to_deploy_broker/mtd_handler.py): + +```python +pr.merge_to_deploy = int((conducted_at - pr.state_changed_at).total_seconds()) +``` + +Two operational consequences: + +- **Recently merged PRs legitimately have incomplete lead time.** Their `merge_to_deploy` is `NULL` + until a deploy catches them. A dashboard showing "lead time dropped this week" often just means the + last few PRs have not been backfilled yet. +- **`DEPLOYMENTS_TO_PROCESS = 500`** caps how many runs are processed per repo per pass. A repo with a + large backlog needs several sync cycles to catch up. There is no warning when the cap is hit. + +### 7.4 Error handling philosophy: isolate and continue + +The same `try/except/LOG.error/continue` pattern nests at four levels — org sync, provider, repo, and +PR batch. A failure in one repo does not abort the run; the rest still sync. + +**The cost:** `POST /sync` returns `200 {"message": "sync started"}` even if every single repo failed. +Failures are visible **only in the logs**, which are capped at 512 KB with no rotation history and no +alerting. "Sync succeeded but the numbers look wrong" is the expected failure signature, and the only +way to diagnose it today is reading `/var/log/sync_server/sync_server.log` before it rotates away. + +Adding real sync-run status (per-repo outcome persisted and surfaced) is a strong candidate for our +first substantial contribution — it is genuinely useful, self-contained, and touches the layering in +an instructive way. + +--- + +### 7.5 The bookmark trap — read this before debugging "no data" + +The single most confusing behaviour in the system, and the first thing to check whenever a repo shows +no metrics. + +**A newly added repo only syncs history back to its bookmark, and the bookmark starts at "now minus +`DEFAULT_SYNC_DAYS`" — or, if a sync already ran, at whenever that sync last got to.** The GitHub ETL +then discards any PR that closed *before* the bookmark: + +```python +# mhq/service/code/sync/etl_github_handler.py +state_changed_at = pr.merged_at if pr.merged_at else pr.closed_at +if pr.state.upper() != PullRequestState.OPEN.value and state_changed_at < bookmark: + continue +``` + +So a repo can sit at zero PRs while the sync log cheerfully reports success: + +``` +[INFO] Starting data sync for org +Nothing to process 🎉 +[INFO] Data sync for sync_code_repos completed successfully +``` + +Meanwhile the UI shows **"Calculating Dora — we're processing your data, it usually takes ~5 mins"** +forever. That banner is not a progress indicator. It is what +`web-server/src/content/DoraMetrics/DoraMetricsBody.tsx` renders when there is no data to draw, and +it never resolves on its own. + +**Diagnosing it.** Compare bookmarks against PR counts per repo: + +```sql +SELECT r.name, b.bookmark, count(p.id) AS prs +FROM "OrgRepo" r +LEFT JOIN "Bookmark" b ON b.repo_id = r.id +LEFT JOIN "PullRequest" p ON p.repo_id = r.id +GROUP BY r.name, b.bookmark; +``` + +- **No bookmark row + 0 PRs** → the repo has never synced. Run a sync; cron will also pick it up + within 30 minutes. +- **Bookmark ≈ the time the repo was added + 0 PRs** → history is being filtered out. Rewind it. + +**Rewinding.** `PUT /orgs//bookmark/reset` sets every bookmark in the org back to +`now - DEFAULT_SYNC_DAYS`, or to an explicit timestamp: + +```bash +curl -X PUT "http://localhost:9696/orgs//bookmark/reset?bookmark_timestamp=2026-05-07T00:00:00" +curl -X POST http://localhost:9697/sync +``` + +**The ordering catch:** `reset_org_bookmarks` iterates *existing* bookmark rows, so it cannot rewind a +repo that has never synced — there is no row yet. For a brand-new repo the sequence must be +**sync → reset → sync**: the first sync creates the bookmark, the reset rewinds it, the second sync +pulls the older history. + +Related: §11.5 on silent sync failures. A sync that finds nothing and a sync that failed for every +repo produce the same HTTP 200 and the same green log line. + +## 8. How the four DORA metrics are derived + +### 8.1 Deployment Frequency + +A "deployment" is polymorphic, resolved per repo by `TeamRepos.deployment_type` through +[`deployments/factory.py`](../backend/analytics_server/mhq/service/deployments/factory.py): + +```mermaid +graph TB + Q["Query: deployments for team + interval"] --> F{"TeamRepos
.deployment_type"} + F -->|PR_MERGE| A["PRDeploymentsService
merged PR into a prod branch
= one deployment"] + F -->|WORKFLOW| B["WorkflowDeploymentsService
successful RepoWorkflowRun
= one deployment"] + A --> N["Normalised Deployment
via DeploymentsAdaptorFactory"] + B --> N + N --> C["Count / bucket by week"] + + style A fill:#5c8001,color:#fff + style B fill:#4a6fa5,color:#fff +``` + +Both strategies implement the `DeploymentsFactoryService` ABC and are normalised to a common +`Deployment` model by an adaptor — so downstream analytics never branch on deployment type. **This is +the cleanest extension seam in the codebase**; a third deployment source is a new subclass plus an +enum value, and nothing downstream changes. + +Deployment IDs are composite strings, `"|"`, parsed by +`get_deployment_type_and_entity_id_from_deployment_id`. Do not assume a deployment ID is a bare UUID. + +### 8.2 Lead Time for Changes + +Lead time is the sum of five stored segments +([`service/code/models/lead_time.py`](../backend/analytics_server/mhq/service/code/models/lead_time.py)): + +```mermaid +graph LR + A["first commit"] -->|first_commit_to_open| B["PR opened"] + B -->|first_response_time| C["first review"] + C -->|rework_time| D["review approved"] + D -->|merge_time| E["merged"] + E -->|merge_to_deploy| F["deployed"] + + style A fill:#3d5a80,color:#fff + style F fill:#5c8001,color:#fff +``` + +```python +lead_time = first_commit_to_open + first_response_time + rework_time + merge_time + merge_to_deploy +cycle_time = first_response_time + rework_time + merge_time + merge_to_deploy +``` + +`cycle_time` is lead time minus the pre-PR authoring period — the part a review process can influence. + +Team aggregates use a **weighted average by PR count**, not a mean of per-repo means +(`_get_weighted_avg_lead_time_metrics` in +[`lead_time.py`](../backend/analytics_server/mhq/service/code/lead_time.py)). Trends bucket weekly via +`generate_expanded_buckets`, then `fill_missing_week_buckets` inserts zeroed weeks so the chart has no +gaps. When comparing a computed number against the UI, replicate the weighting — a plain average will +not match. + +### 8.3 Mean Time to Recovery + +Per incident, `resolved_date - creation_date`, averaged over incidents resolved in the interval; +`MeanTimeToRecoveryMetrics` carries `mean_time_to_recovery` and `incident_count` +([`models/mean_time_to_recovery.py`](../backend/analytics_server/mhq/service/incidents/models/mean_time_to_recovery.py)). + +Incidents come from two sources, unioned and **deduplicated by `incident.key`** +([`incidents.py`](../backend/analytics_server/mhq/service/incidents/incidents.py)): + +1. A configured incident service (`OrgIncidentService`). +2. **PRs matched by regex** — `IncidentPRsSetting.filters` lets a team declare that PRs matching a + pattern (e.g. `^hotfix/`, `revert`) *are* incidents. This is how a team with no PagerDuty still gets + MTTR and CFR. `mhq/utils/regex.py` guards the patterns. + +### 8.4 Change Failure Rate + +```python +change_failure_rate = len(failed_deployments) / len(total_deployments) * 100 +``` + +Both are **`Set[Deployment]`**, so a deployment linked to three incidents counts once +([`ChangeFailureRateMetrics`](../backend/analytics_server/mhq/service/incidents/models/mean_time_to_recovery.py)). +A deployment is "failed" if an incident is attributed to it — the attribution join lives in +`deployments_with_related_incidents`. Empty `total_deployments` returns `0`, not a division error. + +--- + +## 9. Frontend architecture + +```mermaid +graph TB + subgraph browser["Browser"] + UI["React 18 + MUI 5
pages/dora-metrics · teams
integrations · settings · system-logs"] + RX["Redux Toolkit
slices: dora_metrics · team · org
auth · app · service"] + end + + subgraph nextjs["Next.js server :3333"] + MW["middleware.ts
injects feature_flags into every
/api/{auth,integrations,internal,resources}/*"] + BFF["pages/api/**
yup-validated Endpoint handlers"] + end + + FL["Flask :9696"] + PG[("Postgres")] + GH["GitHub / GitLab"] + + UI <--> RX + RX -->|"axios baseURL=/api"| MW --> BFF + BFF -->|"handleRequest → axios"| FL + BFF -->|"knex ⚠️"| PG + BFF -->|"integration setup"| GH + + style UI fill:#3d5a80,color:#fff + style BFF fill:#4a6fa5,color:#fff + style PG fill:#336791,color:#fff +``` + +Conventions to follow when touching the frontend: + +- **BFF handlers** use an `Endpoint` abstraction with **yup** schemas per method + (`endpoint.handle.GET(schema, fn)`), giving runtime request validation — see + [`team_repos.ts`](../web-server/pages/api/resources/team_repos.ts). +- **Feature flags** are merged from `defaultFlags` plus request overrides in + [`middleware.ts`](../web-server/middleware.ts) and appended as a `feature_flags` query param to every + matched API route. A `use_mock_data` flag makes handlers return fixtures from `src/mocks/` — + useful for UI work without a synced database. +- **Pagination** is client-driven: `paginatedRequest` and `batchPaginatedRequest` in + [`internal.ts`](../web-server/src/api-helpers/internal.ts) walk pages recursively, the latter + fetching 5 pages concurrently. +- **There is no login.** `next-auth` is in `package.json` but is **never configured** — there is no + `[...nextauth]` API route and no sign-in page. Its only two imports are a `Session` *type* in + `ThirdPartyAuthContext.tsx` and a `signIn` call in `AppErrors.tsx` that has nothing to call. + `/api/auth/session` is a hand-rolled `Endpoint` handler that returns org details and integration + state to any caller. See §11.1. +- **Tests:** Jest (`jest.config.js`) plus Playwright (`playwright.config.ts`). + +--- + +## 10. Cross-cutting concerns + +### 10.1 Secrets and token encryption + +Provider PATs are **RSA-2048 / PKCS1-OAEP encrypted at rest**, chunked across a Postgres `string[]` +because RSA cannot encrypt arbitrarily long input +([`utils/cryptography.py`](../backend/analytics_server/mhq/utils/cryptography.py), `Integration.access_token_enc_chunks`). + +The keypair is generated **once per install** by +[`generate_config_ini.sh`](../setup_utils/generate_config_ini.sh) into +`backend/analytics_server/mhq/config/config.ini`, base64-encoded, and persisted in the `dev_keys` +Docker volume. + +Three things to know before touching this: + +- **Losing the volume makes every stored token unrecoverable.** Not a corruption — a permanent loss. + Any backup plan must include `dev_keys`, not just the Postgres volume. +- The script also **appends both keys to `~/.bashrc`** as exported env vars — meaning the private key + lands in a shell profile in plaintext. Worth flagging in any production hardening review. +- `config.ini` must never be committed. Confirm it is git-ignored before any commit that touches + `mhq/config/`. + +### 10.2 Concurrency control + +Redis is used **exclusively for distributed locks** (`python-redis-lock`), never as a cache: + +| Lock key | Guards | +|---|---| +| `{org}::data_sync` | Whole sync run — prevents overlapping syncs | +| `{org_repo}::merge_to_deploy_broker` | Per-repo MTD backfill | +| `initialize_database` | First-boot org creation race | + +The `{...}` braces are Redis Cluster hash-tag syntax, keeping related keys on one slot. Redis has no +persistence configured — correct, since locks should not survive a restart. + +### 10.3 Configuration + +Single `.env` at the repo root, loaded by [`env.py`](../backend/analytics_server/env.py) — +`.env.prod` when `FLASK_ENV=production`, otherwise `../../.env`. The same file feeds compose, both +Flask apps, and Next.js. Copy from [`env.example`](../env.example). + +Two dangerous defaults to fix before anything leaves a laptop: `DB_PASS=postgres`, and +`pg_hba.conf` extended with `host all all 0.0.0.0/0 md5` in +[`Dockerfile.dev`](../Dockerfile.dev) — Postgres accepts password auth from any address inside the +container network. + +### 10.4 Observability + +| Concern | Current state | +|---|---| +| Logs | Files under `/var/log/*`, 512 KB cap, `backups=0`, surfaced at `/system-logs` | +| Metrics | None — no Prometheus, no StatsD | +| Tracing | None | +| Health | `GET /` on both Flask apps; `/api/status`, `/api/db_status`, `/api/internal_status` on Next.js | +| Alerting | None | +| Sync outcome | **Not persisted** — logs only (see §7.4) | + +### 10.5 CI and tests + +Five workflows in [`.github/workflows/`](../.github/workflows/): `build.yml`, `build-base.yml`, +`pytest.yml`, `black.yml`, `pre-commit.yml`. Python is formatted with **black** and linted with +**flake8**; hooks in [`.pre-commit-config.yaml`](../.pre-commit-config.yaml). + +**20 Python test files** under `backend/analytics_server/tests/`, concentrated on service-layer logic +(`code`, `deployments`, `incidents`, `workflows`) with model factories in `tests/factories/`. This is +a thin suite for a codebase this size — the metric computations are covered, the ETL edge cases and +the entire BFF/knex path largely are not. Treat "tests pass" as a weak signal here and rely on it +least where the two write paths meet. + +--- + +## 11. Constraints, risks and gaps + +Ordered by significance to Clustox. Nothing here is a criticism of upstream — most are rational +choices for a self-hosted single-tenant appliance, and they simply stop holding if we change the +deployment or tenancy assumptions. + +### 11.1 No authentication anywhere in the stack + +There is **no authentication or authorization of any kind** — not on the backend, and not on the +frontend either. Verify both: + +```bash +# Backend API — zero matches +grep -rniE "login_required|authenticat|jwt|api_key|bearer" backend/analytics_server/mhq/api/ + +# Frontend — no NextAuth route, no sign-in page +find web-server/pages -path '*nextauth*' +ls web-server/pages | grep -iE 'login|signin' +``` + +`next-auth` appears in `web-server/package.json`, which makes it look like there is a login. There +is not: no `[...nextauth]` API route exists and no sign-in page exists. Its only two imports are a +`Session` *type* in `ThirdPartyAuthContext.tsx` and a `signIn` call in `AppErrors.tsx` that has +nothing to call. `/api/auth/session` is a hand-rolled handler returning org details to any caller. +Loading the app goes straight to the dashboard. + +So anything that can reach `:3333` or `:9696` can read every metric and mutate settings, teams and +repos. **The only control is the loopback port binding** in `docker-compose.yml`. + +*Impact if we change anything about deployment topology:* the moment either port is reachable beyond +localhost — a shared host, Kubernetes, a reverse proxy misconfigured to forward `:9696` — the entire +system is open. Any hosting or multi-user work must add an auth layer as a prerequisite, not a +follow-up. + +*Impact for an internal tool specifically:* DORA metrics are data about named engineers' work. With +no auth and no access-control model, anyone who can reach the URL sees every team's numbers. That is +a people question as much as a security one, and worth settling before this is hosted anywhere shared. + +### 11.2 Single container, single failure domain + +Six processes including the database share one container, one restart policy, one resource limit, and +one crash. Combined with `autorestart=false` on Postgres (§4), a database blip yields a container that +passes a liveness check while serving errors. Fine for evaluation, unsuitable for production. + +*Mitigation already available:* the per-process `*_ENABLED` env flags are designed exactly for pulling +components out. Splitting is configuration work, not a rewrite. + +### 11.3 Two write paths to one schema + +See [§3.2](#32-️-the-most-important-architectural-fact-in-this-document). Highest-probability source +of subtle bugs in any change we make. + +### 11.4 No schema/model drift detection + +Hand-maintained SQLAlchemy models against hand-written `dbmate` SQL, with a third hand-maintained +TypeScript copy in the BFF, and nothing checking that the three agree. + +### 11.5 Sync failures are silent + +`POST /sync` returns success regardless of outcome; per-repo failures live only in a 512 KB log with no +history. There is no way to answer "did last night's sync actually work?" without reading logs in time. + +### 11.6 Insecure-by-default configuration + +`DB_PASS=postgres`; `host all all 0.0.0.0/0 md5` appended to `pg_hba.conf`; RSA private key written to +`~/.bashrc`. All acceptable for a laptop; all must be changed before a shared environment. + +### 11.7 Feature gaps versus the upstream README + +The README advertises "integration with various CI/CD tools" and "integration with popular project +management platforms". What is actually implemented: + +| Advertised | Implemented | +|---|---| +| Git providers | GitHub, GitLab (`mhq/exapi/`) — **no Bitbucket, no Azure DevOps** | +| CI/CD | GitHub Actions only (`etl_github_actions_handler.py`) — **no Jenkins, CircleCI, GitLab CI** | +| Incidents | Git-derived + incident-service model — **no PagerDuty/Opsgenie client in `exapi/`** | +| Project management | **None found** | + +Confirm before planning any feature that assumes an integration exists: + +```bash +ls backend/analytics_server/mhq/exapi/ +``` + +### 11.8 Fork maintenance + +Upstream is actively developed and this fork carries no upstream remote and no divergence yet — the +best possible moment to establish a strategy. Covered in [FORK_STRATEGY.md](./FORK_STRATEGY.md). + +--- + +## 12. Extension points + +A map from "we want to change X" to the files involved. Paths are relative to the repo root. + +### 12.1 Add a Git provider + +The strategy pattern makes this the well-trodden path — follow `gitlab` as the reference +implementation, since it was itself added after `github`. + +1. `backend/analytics_server/mhq/store/models/code/enums.py` — add to `CodeProvider` +2. `backend/analytics_server/mhq/exapi/.py` — API client +3. `backend/analytics_server/mhq/exapi/models/.py` — response DTOs +4. `backend/analytics_server/mhq/service/code/sync/etl__handler.py` — implement `CodeProviderETLHandler` +5. `backend/analytics_server/mhq/service/code/sync/etl_code_factory.py` — register in `CodeETLFactory.__call__` +6. `backend/analytics_server/mhq/api/integrations.py` — discovery endpoints +7. `web-server/src/constants/integrations.ts` + `web-server/pages/api/integrations/` — UI wiring +8. `backend/analytics_server/tests/` — tests + +### 12.2 Add a CI/CD provider (for deployments) + +1. `mhq/store/models/code/workflows/enums.py` — add to `RepoWorkflowProviders` +2. `mhq/service/workflows/sync/etl__handler.py` — implement the provider ETL interface +3. `mhq/service/workflows/sync/etl_workflows_factory.py` — register it +4. `mhq/exapi/.py` — client + +### 12.3 Add a deployment source (beyond PR-merge / workflow) + +1. `mhq/service/deployments/models/models.py` — add to `DeploymentType` +2. `mhq/service/deployments/_deployments_service.py` — subclass `DeploymentsFactoryService` +3. `mhq/service/deployments/models/adapter.py` — adaptor to the common `Deployment` +4. `mhq/service/deployments/factory.py` — register in `get_deployments_factory` +5. `mhq/store/models/code/enums.py` — extend `TeamReposDeploymentType` if repo-selectable + +Downstream analytics need no changes — that is the point of the adaptor. + +### 12.4 Change a metric definition + +1. Locate the computation: `mhq/service/code/lead_time.py`, `mhq/service/deployments/analytics.py`, or `mhq/service/incidents/incidents.py` +2. If a **stored** PR segment changes, also change the ETL that writes it (`mhq/service/code/sync/etl_code_analytics.py`) +3. **Plan the backfill.** Historical rows keep old values (§6.1). Either backfill or reset bookmarks via `PUT /orgs//bookmark/reset` +4. Update `tests/service/…` +5. Update the UI's copy and any tooltip that states the definition + +### 12.5 Add a database column + +**Four** places — miss any one and it breaks at runtime, not build time: + +1. `database-docker/db/migrations/_.sql` — hand-written SQL +2. `database-docker/db/schema.sql` — keep the dumped schema current +3. `backend/analytics_server/mhq/store/models/**` — SQLAlchemy model +4. `web-server/src/constants/db.ts` — TypeScript `Table`/`Columns` for the knex path + +### 12.6 Add an API endpoint + +1. Blueprint module in `mhq/api/` (or a new one registered in `app.py`) +2. `voluptuous` validation via `mhq/api/request_utils.py` +3. Service method — keep logic out of the API layer +4. Serializer in `mhq/api/resources/` +5. BFF route in `web-server/pages/api/` with a `yup` schema +6. Redux slice + hook in `web-server/src/slices/` + +### 12.7 Change sync behaviour + +- Order or steps: `mhq/service/sync_data.py` (`sync_sequence` — mind the dependencies in §7.1) +- Frequency: `setup_utils/cronjob.txt` +- Default lookback: `DEFAULT_SYNC_DAYS` in `.env` and `SettingType.DEFAULT_SYNC_DAYS_SETTING` +- MTD batch size: `DEPLOYMENTS_TO_PROCESS` in `mhq/service/merge_to_deploy_broker/mtd_handler.py` + +### 12.8 Split the container for production + +Start from the `*_ENABLED` flags in `docker-compose.yml` / `Dockerfile.dev` and `setup_utils/supervisord.conf`; +point `DB_HOST`/`REDIS_HOST` at managed services. **Add authentication first** (§11.1) — splitting the +container is what makes the unauthenticated API reachable. + +--- + +## 13. Glossary + +| Term | Meaning | +|---|---| +| **Bookmark** | Incremental-sync cursor; ISO-8601 timestamp string per stream | +| **MTD broker** | Merge-to-deploy broker; backfills `PullRequest.merge_to_deploy` on later syncs | +| **`mhq`** | MiddlewareHQ — the backend Python package root | +| **OrgRepo** | A repository known to the system | +| **TeamRepos** | Team↔repo mapping; carries `prod_branches` and `deployment_type` | +| **Prod branch** | Regex in `TeamRepos.prod_branches`; the system's only definition of "production" | +| **Deployment type** | `PR_MERGE` or `WORKFLOW` — selects the deployment strategy per repo | +| **PAT** | Personal access token; RSA-encrypted in `Integration.access_token_enc_chunks` | +| **PR incident** | An incident inferred from a PR matching a configured regex | +| **Lead time segments** | The five stored integer-second spans summing to lead time | +| **BFF** | Backend-for-frontend; the `web-server/pages/api/**` layer | +| **dbmate** | The SQL migration tool (not Alembic, despite SQLAlchemy) | + +--- + +## Appendix A — Verifying this document + +Every structural claim above is reproducible. Re-run these after pulling upstream; a changed count is +a signal the doc needs revisiting. + +```bash +# Flask routes +grep -rn -A1 '@app.route' backend/analytics_server/mhq/api/ + +# Auth on the backend API (expected: no output) +grep -rniE "login_required|authenticat|jwt|api_key|bearer" backend/analytics_server/mhq/api/ + +# Files bypassing Flask to hit Postgres directly (baseline: 23) +grep -rl "from '@/utils/db'" web-server/pages/api web-server/src | wc -l + +# Implemented external providers +ls backend/analytics_server/mhq/exapi/ + +# Migrations (baseline: 4) +ls database-docker/db/migrations | wc -l + +# Python test files (baseline: 20) +find backend/analytics_server/tests -name 'test_*.py' | wc -l + +# Sync order +sed -n '/sync_sequence/,/]/p' backend/analytics_server/mhq/service/sync_data.py +``` + +## Appendix B — Local run quick reference + +Full instructions are in the upstream [README](../README.md); this is the short path. + +```bash +cp env.example .env +docker compose up --build +``` + +| URL | What | +|---|---| +| http://localhost:3333 | Web UI | +| http://localhost:9696 | Analytics API | +| http://localhost:9697 | Sync API | +| http://localhost:3333/system-logs | Logs in the UI | + +Force a sync instead of waiting up to 30 minutes for cron: + +```bash +curl -X POST http://localhost:9697/sync +``` + +Re-ingest history from scratch after changing a metric definition: + +```bash +curl -X PUT http://localhost:9696/orgs//bookmark/reset +``` diff --git a/docs/FORK_STRATEGY.md b/docs/FORK_STRATEGY.md new file mode 100644 index 00000000..7338d84b --- /dev/null +++ b/docs/FORK_STRATEGY.md @@ -0,0 +1,448 @@ +# Fork & Upstream Maintenance Strategy + +| | | +|---|---| +| **Fork** | [Clustox/middleware](https://github.com/Clustox/middleware) | +| **Upstream** | [middlewarehq/middleware](https://github.com/middlewarehq/middleware) | +| **Upstream licence** | Apache License 2.0 | +| **Fork point** | `844eb42` on `main` | +| **Divergence today** | **Zero commits.** Clean fork. | +| **Status** | Proposed — needs sign-off before the first customisation lands | + +--- + +## Why this document exists, and why now + +Upstream is **actively developed** — the fork point sits on a merge commit from a PR numbered #699, +with fixes landing days apart. Every commit they make from here is one we either take or consciously +decline. + +Right now this fork has **zero divergence**. That is the cheapest moment in the entire life of a fork +to decide how it will be maintained. The cost of deciding later compounds: after twenty ad-hoc +customisations scattered through upstream files, "just pull upstream" becomes a multi-day merge with +real risk of silently reverting our own work. + +The failure mode is specific and common: six months in, an upstream security fix lands, the merge +conflicts in fifteen files, someone resolves it under time pressure, and a Clustox customisation +quietly disappears. Nobody notices until a customer does. + +Verify the current state: + +```bash +git log --oneline -1 && git remote -v && git status --short +``` + +--- + +## 1. Decide the fork's posture first + +Everything downstream follows from one question. **This needs a decision from the CTO before the first +customisation is written**, because it determines how much isolation discipline is worth paying for. + +| Posture | What it means | Upstream merge cost | Right when | +|---|---|---|---| +| **A. Track closely** | Stay near-identical to upstream. Customisations are thin and few. Contribute fixes back. | Low, ongoing | We mainly want the product, plus small tweaks | +| **B. Soft fork** | Meaningful Clustox features, but deliberately isolated so upstream files stay mostly untouched. Periodic upstream merges. | Moderate, manageable | **Most likely fit** — we want their engine plus our differentiation | +| **C. Hard fork** | Take the code and go. Upstream becomes a one-time donor. | N/A — we stop merging | Our direction diverges fundamentally | + +**Recommendation: B, with the discipline of A.** + +Rationale: the upstream codebase is cleanly layered and full of factory/strategy seams (see +[ARCHITECTURE.md §12](./ARCHITECTURE.md#12-extension-points)) that were clearly designed for extension. +Most of what a client would plausibly ask for — a new Git provider, a new deployment source, a new +metric — can be added by *registering* a new implementation rather than editing existing files. That is +exactly the shape that makes posture B cheap. Choosing C forfeits upstream's ongoing work for very +little gain. + +The rest of this document assumes B. If the CTO picks A or C, sections 4 and 5 change; the rest holds. + +> **Open question for the CTO — worth asking early:** is this fork intended to stay internal, or ship +> to clients? It changes the licence obligations in §7 and the auth work in +> [ARCHITECTURE.md §11.1](./ARCHITECTURE.md#111-no-authentication-on-the-backend-apis) from +> "nice to have" to "blocking". + +--- + +## 2. Remote setup + +The fork currently has **only `origin`**. There is no way to fetch upstream at all. First fix: + +```bash +git remote add upstream https://github.com/middlewarehq/middleware.git +git remote set-url --push upstream DISABLED +git fetch upstream +``` + +The `--push DISABLED` line is deliberate: it makes an accidental `git push upstream` fail loudly +instead of attempting to write to a repository we do not own. Cheap insurance. + +Verify: + +```bash +git remote -v +# origin https://github.com/Clustox/middleware.git (fetch) +# origin https://github.com/Clustox/middleware.git (push) +# upstream https://github.com/middlewarehq/middleware.git (fetch) +# upstream DISABLED (push) +``` + +Every engineer on the project runs this once. Add it to the onboarding checklist — a teammate without +the upstream remote cannot review a sync PR properly. + +--- + +## 3. Branch model + +```mermaid +gitGraph + commit id: "844eb42 fork point" + commit id: "clustox docs" + commit id: "clustox feature A" + branch upstream-sync + commit id: "merge upstream" type: HIGHLIGHT + commit id: "resolve conflicts" + checkout main + merge upstream-sync + commit id: "clustox feature B" +``` + +`upstream-sync` is cut from `main`, absorbs the upstream merge and all conflict resolution, then goes +back into `main` through a reviewed PR and is deleted. `main` never takes a raw upstream merge directly. + +| Branch | Role | +|---|---| +| `main` | **Clustox's product.** Protected. Upstream + our changes. Deployable. | +| `upstream-sync` | Short-lived, one per sync. Where upstream merges and conflict resolution happen. Deleted after merge. | +| `feature/*` | Normal feature work off `main`. | +| `fix/*` | Bug fixes off `main`. | +| `contrib/*` | Work intended for an upstream PR. Kept free of Clustox-specific code. | + +**We do not keep a branch that mirrors upstream `main` verbatim** — `upstream/main` already is that +branch, fetched on demand. An extra local mirror is one more thing to forget to update. + +### Branch protection to enable on `main` + +Configure in GitHub repo settings (needs org admin): + +- Require a pull request before merging, minimum 1 approval +- Require status checks: `build`, `pytest`, `black`, `pre-commit` +- Dismiss stale approvals on new commits +- No force-push, no deletion + +Rationale: the entire strategy below depends on **sync merges being reviewed**. Without protection, +one `git push --force` to `main` undoes it. + +--- + +## 4. Where Clustox changes are allowed to live + +This is the core discipline of the whole document. **Merge pain is almost perfectly proportional to +how many upstream files we edit.** Adding files is nearly free; editing shared files is where conflicts +come from. + +### The three-tier rule + +```mermaid +graph TB + G["🟢 GREEN — new files
Conflict risk: ~none
No approval needed"] + Y["🟡 YELLOW — registration points
Conflict risk: small, mechanical
Note it in the PR"] + R["🔴 RED — editing upstream logic
Conflict risk: high, semantic
Justify in the PR"] + G --> Y --> R + style G fill:#2d6a4f,color:#fff + style Y fill:#b7791f,color:#fff + style R fill:#9b2226,color:#fff +``` + +**🟢 Green — prefer this. Always ask "can this be a new file?" first.** + +- New provider clients: `mhq/exapi/.py` +- New ETL handlers: `mhq/service/*/sync/etl__handler.py` +- New deployment strategies: `mhq/service/deployments/_deployments_service.py` +- New API blueprints: `mhq/api/.py` +- New UI pages/components under new paths +- Anything under `docs/` or `clustox/` + +**🟡 Yellow — unavoidable, keep surgical.** Upstream's extension seams are *designed* to be edited, but +they are also the lines upstream touches when they add their own providers. Conflicts here are +one-line and mechanical. + +- Factories: `etl_code_factory.py`, `etl_workflows_factory.py`, `deployments/factory.py` +- Enums: `CodeProvider`, `RepoWorkflowProviders`, `DeploymentType` +- Blueprint registration in `app.py` / `sync_app.py` +- `sync_sequence` in `sync_data.py` + +Keep the diff to the minimum lines. Add at the **end** of enums and lists, never in the middle — a +trailing addition conflicts far less often than an insertion. + +**🔴 Red — requires explicit justification in the PR description.** Editing existing upstream logic. +State what you changed, why a green/yellow approach did not work, and what breaks if an upstream merge +silently reverts it. + +- Metric computation: `lead_time.py`, `analytics.py`, `incidents.py` +- The MTD broker: `mtd_handler.py` +- Existing models in `mhq/store/models/**` +- `web-server/src/utils/db.ts`, `constants/db.ts` +- Anything in `setup_utils/`, `Dockerfile*`, `docker-compose.yml` + +### Marking red-zone edits so a merge cannot silently eat them + +Every edit to an upstream file gets a sentinel comment: + +```python +# CLUSTOX: — +... our change ... +# END CLUSTOX +``` + +```typescript +// CLUSTOX: — +``` + +```sql +-- CLUSTOX: — +``` + +This is not decoration. It gives us a **one-command audit** of our entire divergence footprint, which +becomes the review checklist during every upstream sync: + +```bash +grep -rn "CLUSTOX:" --include='*.py' --include='*.ts' --include='*.tsx' --include='*.sql' . \ + | grep -v node_modules +``` + +After a sync merge, this count should not drop. If it does, the merge ate one of our changes. + +### The database exception + +Custom columns and tables need their own migration files (green — new files), but they mutate a schema +upstream also migrates. Two rules: + +- **Prefix custom migrations** `_clustox_.sql` so they are identifiable at a glance. +- **Never edit an upstream migration that has already run.** `dbmate` tracks applied migrations; editing + one that has run is a silent no-op locally and a divergence in every other environment. +- Remember custom columns are a **four-place change** — see + [ARCHITECTURE.md §12.5](./ARCHITECTURE.md#125-add-a-database-column). + +--- + +## 5. The upstream sync procedure + +**Cadence: monthly, plus immediately for any upstream security fix.** Monthly is frequent enough that +each merge is small, infrequent enough that it is not a tax. Put it on someone's calendar — syncs that +depend on someone remembering do not happen. + +### Step 1 — See what is coming before committing to it + +```bash +git fetch upstream +git log --oneline main..upstream/main | wc -l # how many commits behind +git log --oneline main..upstream/main # what they are +git diff --stat main..upstream/main # blast radius +``` + +Then check the overlap between what upstream changed and what we have touched: + +```bash +# Files upstream changed +git diff --name-only main..upstream/main | sort > /tmp/upstream-changed.txt +# Files we changed since the fork point +git diff --name-only 844eb42..main | sort > /tmp/clustox-changed.txt +# The intersection is exactly where conflicts will be +comm -12 /tmp/upstream-changed.txt /tmp/clustox-changed.txt +``` + +That last command is the whole risk assessment in one line. Empty output means a trivial merge. + +### Step 2 — Merge on a branch, never on `main` + +```bash +git checkout main && git pull origin main +git checkout -b upstream-sync/$(date +%Y-%m) +git merge upstream/main +``` + +**Merge, not rebase.** `main` is shared and protected; rebasing rewrites published history. A merge +commit also leaves an honest record of exactly when each upstream batch arrived. + +### Step 3 — Resolve, with the sentinel audit as the checklist + +For each conflict, the question is always: *does upstream's change subsume ours, or must both survive?* + +```bash +git status --short | grep '^UU' # conflicted files +git checkout --theirs # take upstream wholesale +git checkout --ours # keep ours wholesale +# — but usually: edit by hand and keep both +``` + +Then confirm nothing of ours vanished: + +```bash +grep -rn "CLUSTOX:" --include='*.py' --include='*.ts' --include='*.tsx' --include='*.sql' . \ + | grep -v node_modules | wc -l +``` + +Compare against the pre-merge count. A drop means a resolution dropped one of our changes — go find it. + +### Step 4 — Verify + +```bash +# Backend +cd backend/analytics_server && python -m pytest tests/ -q && cd ../.. +black --check backend/ +flake8 backend/ + +# Frontend +cd web-server && yarn jest && cd .. + +# Migrations still apply, app still boots +docker compose down -v && docker compose up --build +``` + +Then a manual smoke pass, because the automated suite is thin (20 Python test files, and the BFF/knex +path is largely uncovered — see +[ARCHITECTURE.md §10.5](./ARCHITECTURE.md#105-ci-and-tests)): + +- [ ] All six supervisord processes running (`docker compose logs`) +- [ ] UI loads at :3333 +- [ ] Integration settings page renders and the existing PAT still decrypts +- [ ] `curl -X POST http://localhost:9697/sync` completes; check `/system-logs` for per-repo errors +- [ ] All four DORA metrics render with data +- [ ] Every Clustox customisation still behaves + +**The PAT-decrypt check matters more than it looks.** Token encryption depends on the RSA keypair in +the `dev_keys` volume; a `docker compose down -v` destroys it. If upstream changes anything about +`utils/cryptography.py` or the config bootstrap, this is where it surfaces. + +### Step 5 — PR into `main` + +Open `upstream-sync/YYYY-MM` → `main`, with a description covering: + +- Commit count and date range pulled +- Notable upstream changes affecting us +- Every conflict and how it was resolved +- Sentinel count before/after +- Smoke-test results + +Reviewed by someone who did not do the merge. This is the control that catches a silently reverted +customisation, and it is the reason `main` is protected. + +### Step 6 — Record it + +Append to `docs/UPSTREAM_SYNC_LOG.md`: + +| Date | Upstream ref | Commits | Conflicts | Notes | +|---|---|---|---|---| +| _(first sync)_ | | | | | + +--- + +## 6. Contributing back to upstream + +Worth doing deliberately, for two self-interested reasons: **anything upstream accepts is code we no +longer maintain**, and a visible Clustox presence in an active OSS project is good for the company. + +**Send upstream:** genuine bug fixes, provider integrations of general use, performance work, doc fixes. +**Keep private:** Clustox branding, client-specific logic, anything commercially differentiating. + +Work on a `contrib/*` branch cut from `upstream/main` (not from `main`), so the PR contains no Clustox +code and no sentinel comments: + +```bash +git fetch upstream +git checkout -b contrib/ upstream/main +# ... make the change, no CLUSTOX: markers ... +git push origin contrib/ +# open PR: Clustox/middleware contrib/ → middlewarehq/middleware main +``` + +Read [`CONTRIBUTING.md`](../CONTRIBUTING.md) and [`CODE_OF_CONDUCT.md`](../CODE_OF_CONDUCT.md) first; +upstream has an active community and its own review conventions. + +> **Check before the first external contribution:** whether Clustox requires anything of employees +> contributing to third-party OSS, and whether upstream asks for a CLA. Better to ask than to retract. + +--- + +## 7. Licence obligations (Apache 2.0) + +Not legal advice — flag anything commercially significant to whoever owns legal at Clustox. But the +mechanics are simple and we are currently not meeting one of them. + +**What Apache 2.0 permits:** commercial use, modification, private use, distribution, sublicensing. +We can build a product on this and sell it. We are **not** required to open-source our changes. + +**What it requires when we distribute:** + +| Requirement | Status | +|---|---| +| Include the Apache 2.0 licence text | ✅ [`LICENSE`](../LICENSE) is present | +| Preserve copyright/attribution notices | ⚠️ **Gap** — see below | +| State significant changes made to the files | ⚠️ Not yet — the `CLUSTOX:` sentinels are the mechanism | +| Include the `NOTICE` file if one exists | ⚠️ **Gap — no `NOTICE` file exists** | + +### The gap, and the fix + +Apache 2.0 §4(d) requires preserving attribution notices in derivative works. This fork has **no +`NOTICE` file** — verify with `ls NOTICE`. Upstream did not ship one, so strictly we are not yet +failing to propagate theirs; but once we distribute a modified version, clean attribution is both an +obligation and simply good practice. + +A [`NOTICE`](../NOTICE) file has been added in this change set recording upstream authorship and +Clustox's modifications. Keep it current as the fork diverges. + +**Also do not:** use the Middleware name or logo to brand a Clustox product (trademarks are not granted +by Apache 2.0 — §6), or imply upstream endorsement. If we ship this to clients under a Clustox name, +`media_files/` and README branding need review. + +--- + +## 8. Immediate actions + +Ordered. Items 1–4 are the ones to do before any customisation lands. + +| # | Action | Owner | Status | +|---|---|---|---| +| 1 | **CTO decides the fork posture** (§1) — blocks the rest | CTO | ⬜ | +| 2 | Add the `upstream` remote, push-disabled (§2) | Every engineer | ⬜ | +| 3 | Enable branch protection on `main` (§3) | Repo admin | ⬜ | +| 4 | Agree the three-tier rule and `CLUSTOX:` sentinel (§4) | Team | ⬜ | +| 5 | Add `NOTICE`, keep it current (§7) | — | ✅ added | +| 6 | Confirm CI is green on the fork, record the baseline | — | ⬜ | +| 7 | Create `docs/UPSTREAM_SYNC_LOG.md`, do a first sync to prove the runbook | — | ⬜ | +| 8 | Put the monthly sync on a calendar with a named owner | Team lead | ⬜ | + +Two notes on sequencing. **Item 6 before any code changes** — if upstream CI is already failing on our +fork, that is far better discovered now than blamed on our first feature. And **item 7's first sync +should happen while divergence is still zero**, precisely because it is guaranteed trivial: it proves +the runbook works before we need it under pressure. + +--- + +## Appendix — Command reference + +```bash +# One-time setup +git remote add upstream https://github.com/middlewarehq/middleware.git +git remote set-url --push upstream DISABLED + +# How far behind are we? +git fetch upstream && git log --oneline main..upstream/main | wc -l + +# Where will conflicts happen? +comm -12 \ + <(git diff --name-only main..upstream/main | sort) \ + <(git diff --name-only 844eb42..main | sort) + +# Our full divergence footprint +grep -rn "CLUSTOX:" --include='*.py' --include='*.ts' --include='*.tsx' --include='*.sql' . \ + | grep -v node_modules + +# Start a sync +git checkout main && git pull origin main +git checkout -b upstream-sync/$(date +%Y-%m) +git merge upstream/main + +# Start an upstream contribution (clean of Clustox code) +git checkout -b contrib/ upstream/main +``` diff --git a/docs/UPSTREAM_SYNC_LOG.md b/docs/UPSTREAM_SYNC_LOG.md new file mode 100644 index 00000000..238acd56 --- /dev/null +++ b/docs/UPSTREAM_SYNC_LOG.md @@ -0,0 +1,41 @@ +# Upstream Sync Log + +A row per merge from `middlewarehq/middleware` into this fork. Append after every sync. + +Procedure: [FORK_STRATEGY.md §5](./FORK_STRATEGY.md#5-the-upstream-sync-procedure). +Cadence: monthly, plus immediately for any upstream security fix. + +Why this file exists: when something breaks three weeks after a sync, the first question is always +"what came in, and what did we have to resolve by hand?" Reconstructing that from git history is +possible but slow. Writing three lines at merge time is not. + +--- + +| Date | Upstream ref merged | Commits | Files changed | Conflicts | Sentinels before → after | Smoke test | PR | Notes | +|---|---|---|---|---|---|---|---|---| +| _(no syncs yet — fork is at `844eb42` with zero divergence)_ | | | | | | | | | + +--- + +## Column meanings + +| Column | What to record | +|---|---| +| **Upstream ref merged** | The `upstream/main` SHA merged, so the exact state is recoverable | +| **Commits** | `git log --oneline ..upstream/main \| wc -l` | +| **Files changed** | `git diff --stat` summary line | +| **Conflicts** | Count, and the files — name them, they are the ones to re-check next time | +| **Sentinels before → after** | `CLUSTOX:` count either side of the merge. **A drop means a resolution ate one of our changes.** | +| **Smoke test** | Pass/fail against the checklist in FORK_STRATEGY §5 step 4 | +| **PR** | Link to the reviewed sync PR | +| **Notes** | Behaviour changes, migrations, anything that needed a decision | + +## Worked example of a filled row + +| Date | Upstream ref merged | Commits | Files changed | Conflicts | Sentinels before → after | Smoke test | PR | Notes | +|---|---|---|---|---|---|---|---|---| +| 2026-09-01 | `abc1234` | 23 | 41 files, +892/-310 | 2 — `etl_code_factory.py`, `sync_data.py` | 14 → 14 | ✅ | #12 | Upstream added a Bitbucket provider; our registration line moved but survived. One new migration, applied cleanly. | + +Note what the example shows: conflicts in **yellow-zone** registration points, resolved mechanically, +sentinel count unchanged. That is what a healthy sync looks like. Conflicts in red-zone files with a +falling sentinel count is the pattern to escalate on. diff --git a/docs/superpowers/plans/2026-08-05-auth-rbac.md b/docs/superpowers/plans/2026-08-05-auth-rbac.md new file mode 100644 index 00000000..11475e49 --- /dev/null +++ b/docs/superpowers/plans/2026-08-05-auth-rbac.md @@ -0,0 +1,2049 @@ +# Authentication & RBAC Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Require login to reach any dashboard, with superadmins managing users and integrations and admins seeing only their assigned teams. + +**Architecture:** Authentication uses next-auth v4 with a Credentials provider and JWT sessions. Enforcement happens at three surfaces: `middleware.ts` guards pages, `Endpoint.serve()` guards all 47 BFF API routes from one insertion point, and a shared-secret header locks the Flask servers to BFF-only traffic. Because Flask becomes unreachable except through the BFF, **all authorization logic lives in the BFF** — no user context is threaded through Python. + +**Tech Stack:** Next.js 15 (pages router), next-auth 4.23, TypeScript, knex 2.4, yup 0.32, bcryptjs, Postgres 15, dbmate, Flask 3, Jest, pytest, Playwright. + +**Spec:** [2026-08-05-auth-rbac-design.md](../specs/2026-08-05-auth-rbac-design.md) + +## Global Constraints + +- **Roles are exactly two:** `SUPERADMIN` and `ADMIN`. No third role. +- **Never modify upstream tables.** All new schema goes in `ClustoxUserAuth` and `ClustoxUserTeamAccess`. +- **Every edit to an upstream file gets a `CLUSTOX:` sentinel** — `// CLUSTOX: ` for TS, `# CLUSTOX:` for Python, `-- CLUSTOX:` for SQL. Per [FORK_STRATEGY.md §4](../../FORK_STRATEGY.md). +- **Fail closed.** If `INTERNAL_API_TOKEN` is unset, Flask rejects every request. Never default to allow. +- **Never log or return password hashes.** +- **bcryptjs, not bcrypt.** Pure JS, no native compilation — `bcrypt` breaks the Docker build. +- **Migration filenames** use `_clustox_.sql`, e.g. `20260805150000_clustox_auth.sql`. +- Do not touch `web-server/src/components/Authenticated/index.tsx`. It is a no-op stub; `middleware.ts` does the real work, and rewriting it only adds divergence. + +## Two corrections to the spec + +1. **Seven upstream files, not six.** `setup_utils/cronjob.txt` also changes — cron calls `:9697/sync` and must send the internal token once Flask is locked down. Handled in Task 9. +2. **`instrumentation.ts` is the bootstrap hook.** The spec said "on first boot" without saying where. Next 15 supports `instrumentation.ts` with an exported `register()`; that is the mechanism. + +--- + +## File Structure + +**New files (green zone — never conflict with upstream):** + +| File | Responsibility | +|---|---| +| `database-docker/db/migrations/20260805150000_clustox_auth.sql` | Two new tables | +| `web-server/src/auth/types.ts` | `ClustoxRole`, `AuthSession` — shared types | +| `web-server/src/auth/password.ts` | Hash and verify only | +| `web-server/src/auth/queries.ts` | All DB access for auth tables | +| `web-server/src/auth/guard.ts` | Authorization decisions (pure where possible) | +| `web-server/src/auth/session.ts` | Read the session inside API routes | +| `web-server/src/auth/bootstrap.ts` | Seed the first superadmin | +| `web-server/instrumentation.ts` | Calls `bootstrap()` on server start | +| `web-server/pages/api/auth/[...nextauth].ts` | next-auth config | +| `web-server/pages/login.tsx` | Sign-in page | +| `web-server/pages/users.tsx` | User management (superadmin) | +| `web-server/pages/api/clustox/users/index.ts` | List + create users | +| `web-server/pages/api/clustox/users/[user_id].ts` | Update role/teams, delete | +| `backend/analytics_server/mhq/clustox_auth/__init__.py` | Package marker | +| `backend/analytics_server/mhq/clustox_auth/internal_token.py` | Flask token guard | +| `backend/analytics_server/tests/clustox_auth/test_internal_token.py` | Its tests | +| `setup_utils/trigger_sync.sh` | Cron helper that sends the token | + +**Upstream files modified (seven):** + +| File | Change | +|---|---| +| `web-server/src/constants/db.ts` | Declare two tables (append-only) | +| `web-server/src/api-helpers/global.ts` | Enforce the existing `authenticated` flag | +| `web-server/src/api-helpers/axios.ts` | Stamp `X-Internal-Token` | +| `web-server/middleware.ts` | Redirect unauthenticated to `/login` | +| `backend/analytics_server/app.py` | Register `before_request` | +| `backend/analytics_server/sync_app.py` | Register `before_request` | +| `setup_utils/cronjob.txt` | Call `trigger_sync.sh` instead of raw curl | + +--- + +## Task 1: Database schema and knex declarations + +**Files:** +- Create: `database-docker/db/migrations/20260805150000_clustox_auth.sql` +- Modify: `database-docker/db/schema.sql` (append table definitions) +- Modify: `web-server/src/constants/db.ts` + +**Interfaces:** +- Consumes: nothing +- Produces: tables `ClustoxUserAuth(user_id, password_hash, role, created_at, updated_at)` and `ClustoxUserTeamAccess(user_id, team_id, created_at)`; knex accessors `Table.ClustoxUserAuth`, `Table.ClustoxUserTeamAccess`, and their `Columns` entries + +- [ ] **Step 1: Write the migration** + +Create `database-docker/db/migrations/20260805150000_clustox_auth.sql`: + +```sql +-- migrate:up +-- CLUSTOX: authentication and role-based access tables. Deliberately separate +-- from upstream "Users" so upstream migrations can never collide with ours. +CREATE TABLE "ClustoxUserAuth" ( + user_id uuid PRIMARY KEY REFERENCES "Users"(id) ON DELETE CASCADE, + password_hash text NOT NULL, + role varchar NOT NULL CHECK (role IN ('SUPERADMIN','ADMIN')), + created_at timestamptz NOT NULL DEFAULT now(), + updated_at timestamptz NOT NULL DEFAULT now() +); + +CREATE TABLE "ClustoxUserTeamAccess" ( + user_id uuid NOT NULL REFERENCES "Users"(id) ON DELETE CASCADE, + team_id uuid NOT NULL REFERENCES "Team"(id) ON DELETE CASCADE, + created_at timestamptz NOT NULL DEFAULT now(), + PRIMARY KEY (user_id, team_id) +); + +CREATE INDEX idx_clustox_user_team_access_user ON "ClustoxUserTeamAccess"(user_id); + +-- migrate:down +DROP TABLE "ClustoxUserTeamAccess"; +DROP TABLE "ClustoxUserAuth"; +``` + +- [ ] **Step 2: Apply it and verify the tables exist** + +```bash +docker compose up -d +docker exec middleware-dev bash -c 'cd /app && dbmate -u "postgresql://postgres:postgres@localhost:5434/mhq-oss?sslmode=disable" up' +docker exec middleware-dev psql "postgresql://postgres:postgres@localhost:5434/mhq-oss" -c '\d "ClustoxUserAuth"' +``` + +Expected: table definition printed with all five columns and the CHECK constraint. + +- [ ] **Step 3: Verify the CHECK constraint rejects a bad role** + +```bash +docker exec middleware-dev psql "postgresql://postgres:postgres@localhost:5434/mhq-oss" \ + -c "INSERT INTO \"ClustoxUserAuth\" (user_id, password_hash, role) VALUES (gen_random_uuid(), 'x', 'HACKER');" +``` + +Expected: fails with a foreign key violation **or** a check constraint violation. Either proves the guards are live. + +- [ ] **Step 4: Append the same DDL to `schema.sql`** + +Paste both `CREATE TABLE` statements and the `CREATE INDEX` into `database-docker/db/schema.sql`, above the final `schema_migrations` INSERT block. Keeping this file current is required — see [ARCHITECTURE.md §12.5](../../ARCHITECTURE.md). + +- [ ] **Step 5: Declare the tables for knex** + +In `web-server/src/constants/db.ts`, add to the `TableT` enum (append at the end, never mid-list — trailing additions conflict far less): + +```ts +enum TableT { + // ... existing entries unchanged ... + // CLUSTOX: auth tables — see docs/superpowers/specs/2026-08-05-auth-rbac-design.md + ClustoxUserAuth, + ClustoxUserTeamAccess +} +``` + +Then append to the `Columns` object: + +```ts +export const Columns = { + // ... existing entries unchanged ... + // CLUSTOX: auth tables + [Table.ClustoxUserAuth]: objectEnumFromFn(() => { + enum Columns { + user_id, + password_hash, + role, + created_at, + updated_at + } + return Columns; + }), + [Table.ClustoxUserTeamAccess]: objectEnumFromFn(() => { + enum Columns { + user_id, + team_id, + created_at + } + return Columns; + }) +}; +``` + +- [ ] **Step 6: Verify TypeScript compiles** + +Run: `cd web-server && yarn tsc --noEmit` +Expected: no errors mentioning `ClustoxUserAuth`. + +- [ ] **Step 7: Commit** + +```bash +git add database-docker/db/migrations/20260805150000_clustox_auth.sql database-docker/db/schema.sql web-server/src/constants/db.ts +git commit -m "feat(auth): add ClustoxUserAuth and ClustoxUserTeamAccess tables" +``` + +--- + +## Task 2: Password hashing + +**Files:** +- Create: `web-server/src/auth/password.ts` +- Create: `web-server/src/auth/__tests__/password.test.ts` +- Modify: `web-server/package.json` (add `bcryptjs`, `@types/bcryptjs`) + +**Interfaces:** +- Consumes: nothing +- Produces: `hashPassword(plain: string): Promise`, `verifyPassword(plain: string, hash: string): Promise` + +- [ ] **Step 1: Install bcryptjs** + +```bash +cd web-server && yarn add bcryptjs && yarn add -D @types/bcryptjs +``` + +Use `bcryptjs`, not `bcrypt` — the latter needs native compilation and breaks the Docker build. + +- [ ] **Step 2: Write the failing test** + +Create `web-server/src/auth/__tests__/password.test.ts`: + +```ts +import { hashPassword, verifyPassword } from '@/auth/password'; + +describe('password', () => { + it('produces a hash that is not the plaintext', async () => { + const hash = await hashPassword('correct-horse-battery'); + expect(hash).not.toBe('correct-horse-battery'); + expect(hash.length).toBeGreaterThan(50); + }); + + it('verifies a correct password', async () => { + const hash = await hashPassword('correct-horse-battery'); + await expect(verifyPassword('correct-horse-battery', hash)).resolves.toBe(true); + }); + + it('rejects an incorrect password', async () => { + const hash = await hashPassword('correct-horse-battery'); + await expect(verifyPassword('wrong-password', hash)).resolves.toBe(false); + }); + + it('produces different hashes for the same input (salted)', async () => { + const a = await hashPassword('same-input'); + const b = await hashPassword('same-input'); + expect(a).not.toBe(b); + }); + + it('returns false rather than throwing on a malformed hash', async () => { + await expect(verifyPassword('anything', 'not-a-hash')).resolves.toBe(false); + }); +}); +``` + +- [ ] **Step 3: Run it to confirm it fails** + +Run: `cd web-server && yarn jest src/auth/__tests__/password.test.ts` +Expected: FAIL — "Cannot find module '@/auth/password'". + +- [ ] **Step 4: Implement** + +Create `web-server/src/auth/password.ts`: + +```ts +import { compare, hash } from 'bcryptjs'; + +const COST = 12; + +export const hashPassword = (plain: string): Promise => + hash(plain, COST); + +export const verifyPassword = async ( + plain: string, + passwordHash: string +): Promise => { + try { + return await compare(plain, passwordHash); + } catch { + // A malformed stored hash must read as "wrong password", never as an + // exception that a caller might treat as success. + return false; + } +}; +``` + +- [ ] **Step 5: Run tests to verify they pass** + +Run: `cd web-server && yarn jest src/auth/__tests__/password.test.ts` +Expected: 5 passed. + +- [ ] **Step 6: Commit** + +```bash +git add web-server/src/auth/password.ts web-server/src/auth/__tests__/password.test.ts web-server/package.json web-server/yarn.lock +git commit -m "feat(auth): add bcrypt password hashing" +``` + +--- + +## Task 3: Shared types and auth queries + +**Files:** +- Create: `web-server/src/auth/types.ts` +- Create: `web-server/src/auth/queries.ts` +- Create: `web-server/src/auth/__tests__/queries.test.ts` + +**Interfaces:** +- Consumes: `Table`, `Columns` from Task 1; `hashPassword` from Task 2 +- Produces: + - `type ClustoxRole = 'SUPERADMIN' | 'ADMIN'` + - `interface AuthSession { userId: string; email: string; name: string; role: ClustoxRole }` + - `interface AuthUserRow { userId: string; email: string; name: string; role: ClustoxRole; passwordHash: string }` + - `interface AuthUserListItem { userId: string; email: string; name: string; role: ClustoxRole; teamIds: string[] }` + - `getAuthUserByEmail(email: string): Promise` + - `getTeamIdsForUser(userId: string): Promise` + - `getAllTeamIds(): Promise` + - `listUsers(): Promise` + - `createUser(input: { name: string; email: string; password: string; role: ClustoxRole; teamIds: string[]; orgId: string }): Promise` + - `updateUserRole(userId: string, role: ClustoxRole): Promise` + - `setUserTeams(userId: string, teamIds: string[]): Promise` + - `countSuperadmins(): Promise` + +- [ ] **Step 1: Write the types** + +Create `web-server/src/auth/types.ts`: + +```ts +export type ClustoxRole = 'SUPERADMIN' | 'ADMIN'; + +export interface AuthSession { + userId: string; + email: string; + name: string; + role: ClustoxRole; +} + +export interface AuthUserRow { + userId: string; + email: string; + name: string; + role: ClustoxRole; + passwordHash: string; +} + +export interface AuthUserListItem { + userId: string; + email: string; + name: string; + role: ClustoxRole; + teamIds: string[]; +} +``` + +- [ ] **Step 2: Write the failing test** + +Create `web-server/src/auth/__tests__/queries.test.ts`. The knex module is mocked, so this tests query construction and row mapping without a database: + +```ts +const mockDb = jest.fn(); +jest.mock('@/utils/db', () => ({ + db: (...args: any[]) => mockDb(...args) +})); + +import { getAuthUserByEmail, getTeamIdsForUser } from '@/auth/queries'; + +const chain = (result: any) => { + const c: any = {}; + c.select = jest.fn(() => c); + c.join = jest.fn(() => c); + c.where = jest.fn(() => c); + c.andWhere = jest.fn(() => c); + c.first = jest.fn(() => Promise.resolve(result)); + c.then = (fn: any) => Promise.resolve(result).then(fn); + return c; +}; + +describe('getAuthUserByEmail', () => { + beforeEach(() => mockDb.mockReset()); + + it('maps a row to AuthUserRow', async () => { + mockDb.mockReturnValue( + chain({ + user_id: 'u1', + primary_email: 'a@clustox.com', + name: 'Ayesha', + role: 'ADMIN', + password_hash: 'hashed' + }) + ); + + await expect(getAuthUserByEmail('a@clustox.com')).resolves.toEqual({ + userId: 'u1', + email: 'a@clustox.com', + name: 'Ayesha', + role: 'ADMIN', + passwordHash: 'hashed' + }); + }); + + it('returns null when no row matches', async () => { + mockDb.mockReturnValue(chain(undefined)); + await expect(getAuthUserByEmail('nobody@clustox.com')).resolves.toBeNull(); + }); +}); + +describe('getTeamIdsForUser', () => { + beforeEach(() => mockDb.mockReset()); + + it('returns a flat array of team ids', async () => { + mockDb.mockReturnValue(chain([{ team_id: 't1' }, { team_id: 't2' }])); + await expect(getTeamIdsForUser('u1')).resolves.toEqual(['t1', 't2']); + }); + + it('returns an empty array when the user has no teams', async () => { + mockDb.mockReturnValue(chain([])); + await expect(getTeamIdsForUser('u1')).resolves.toEqual([]); + }); +}); +``` + +- [ ] **Step 3: Run it to confirm it fails** + +Run: `cd web-server && yarn jest src/auth/__tests__/queries.test.ts` +Expected: FAIL — "Cannot find module '@/auth/queries'". + +- [ ] **Step 4: Implement the queries** + +Create `web-server/src/auth/queries.ts`: + +```ts +import { Columns, Table } from '@/constants/db'; +import { db } from '@/utils/db'; + +import { hashPassword } from './password'; +import { AuthUserListItem, AuthUserRow, ClustoxRole } from './types'; + +export const getAuthUserByEmail = async ( + email: string +): Promise => { + const row = await db(Table.ClustoxUserAuth) + .join(Table.Users, `${Table.Users}.id`, `${Table.ClustoxUserAuth}.user_id`) + .where(`${Table.Users}.primary_email`, email) + .andWhere(`${Table.Users}.is_deleted`, false) + .select( + `${Table.ClustoxUserAuth}.user_id`, + `${Table.ClustoxUserAuth}.role`, + `${Table.ClustoxUserAuth}.password_hash`, + `${Table.Users}.primary_email`, + `${Table.Users}.name` + ) + .first(); + + if (!row) return null; + + return { + userId: row.user_id, + email: row.primary_email, + name: row.name, + role: row.role as ClustoxRole, + passwordHash: row.password_hash + }; +}; + +export const getTeamIdsForUser = async (userId: string): Promise => { + const rows = await db(Table.ClustoxUserTeamAccess) + .select(Columns[Table.ClustoxUserTeamAccess].team_id) + .where(Columns[Table.ClustoxUserTeamAccess].user_id, userId); + return rows.map((r: { team_id: string }) => r.team_id); +}; + +export const getAllTeamIds = async (): Promise => { + const rows = await db(Table.Team).select('id').where('is_deleted', false); + return rows.map((r: { id: string }) => r.id); +}; + +export const countSuperadmins = async (): Promise => { + const rows = await db(Table.ClustoxUserAuth) + .where(Columns[Table.ClustoxUserAuth].role, 'SUPERADMIN') + .select(Columns[Table.ClustoxUserAuth].user_id); + return rows.length; +}; + +export const listUsers = async (): Promise => { + const rows = await db(Table.ClustoxUserAuth) + .join(Table.Users, `${Table.Users}.id`, `${Table.ClustoxUserAuth}.user_id`) + .where(`${Table.Users}.is_deleted`, false) + .select( + `${Table.ClustoxUserAuth}.user_id`, + `${Table.ClustoxUserAuth}.role`, + `${Table.Users}.primary_email`, + `${Table.Users}.name` + ); + + const access = await db(Table.ClustoxUserTeamAccess).select('user_id', 'team_id'); + + return rows.map((r: any) => ({ + userId: r.user_id, + email: r.primary_email, + name: r.name, + role: r.role as ClustoxRole, + teamIds: access + .filter((a: any) => a.user_id === r.user_id) + .map((a: any) => a.team_id) + })); +}; + +export const createUser = async (input: { + name: string; + email: string; + password: string; + role: ClustoxRole; + teamIds: string[]; + orgId: string; +}): Promise => { + const passwordHash = await hashPassword(input.password); + + const [user] = await db(Table.Users) + .insert({ + org_id: input.orgId, + name: input.name, + primary_email: input.email, + is_deleted: false + }) + .returning('id'); + + const userId = typeof user === 'string' ? user : user.id; + + await db(Table.ClustoxUserAuth).insert({ + user_id: userId, + password_hash: passwordHash, + role: input.role + }); + + if (input.teamIds.length) { + await db(Table.ClustoxUserTeamAccess).insert( + input.teamIds.map((team_id) => ({ user_id: userId, team_id })) + ); + } + + return userId; +}; + +export const updateUserRole = async ( + userId: string, + role: ClustoxRole +): Promise => { + await db(Table.ClustoxUserAuth) + .where(Columns[Table.ClustoxUserAuth].user_id, userId) + .update({ role, updated_at: new Date() }); +}; + +export const setUserTeams = async ( + userId: string, + teamIds: string[] +): Promise => { + await db(Table.ClustoxUserTeamAccess) + .where(Columns[Table.ClustoxUserTeamAccess].user_id, userId) + .delete(); + + if (teamIds.length) { + await db(Table.ClustoxUserTeamAccess).insert( + teamIds.map((team_id) => ({ user_id: userId, team_id })) + ); + } +}; +``` + +- [ ] **Step 5: Run tests to verify they pass** + +Run: `cd web-server && yarn jest src/auth/__tests__/queries.test.ts` +Expected: 4 passed. + +- [ ] **Step 6: Commit** + +```bash +git add web-server/src/auth/types.ts web-server/src/auth/queries.ts web-server/src/auth/__tests__/queries.test.ts +git commit -m "feat(auth): add auth types and data access queries" +``` + +--- + +## Task 4: Authorization guards + +This is the core of the feature. Test it hardest. + +**Files:** +- Create: `web-server/src/auth/guard.ts` +- Create: `web-server/src/auth/__tests__/guard.test.ts` + +**Interfaces:** +- Consumes: `AuthSession`, `ClustoxRole` from Task 3; `getTeamIdsForUser`, `getAllTeamIds` from Task 3; `ResponseError`, `Errors` from `@/constants/error` +- Produces: + - `assertAuthenticated(session: AuthSession | null): AuthSession` — throws 401, returns the narrowed session + - `assertRole(session: AuthSession | null, role: ClustoxRole): AuthSession` — throws 403 + - `canAccessAllTeams(session: AuthSession): boolean` + - `assertTeamAccess(session: AuthSession | null, teamId: string): Promise` — throws 403 + - `getAccessibleTeamIds(session: AuthSession): Promise` — always concrete ids + +- [ ] **Step 1: Write the failing test** + +Create `web-server/src/auth/__tests__/guard.test.ts`: + +```ts +jest.mock('@/auth/queries', () => ({ + getTeamIdsForUser: jest.fn(), + getAllTeamIds: jest.fn() +})); + +import { + assertAuthenticated, + assertRole, + assertTeamAccess, + canAccessAllTeams, + getAccessibleTeamIds +} from '@/auth/guard'; +import { getAllTeamIds, getTeamIdsForUser } from '@/auth/queries'; +import { AuthSession } from '@/auth/types'; + +const superadmin: AuthSession = { + userId: 'su1', + email: 'boss@clustox.com', + name: 'Boss', + role: 'SUPERADMIN' +}; +const admin: AuthSession = { + userId: 'ad1', + email: 'lead@clustox.com', + name: 'Lead', + role: 'ADMIN' +}; + +describe('assertAuthenticated', () => { + it('returns the session when present', () => { + expect(assertAuthenticated(admin)).toBe(admin); + }); + + it('throws 401 when the session is null', () => { + expect(() => assertAuthenticated(null)).toThrow( + expect.objectContaining({ status: 401 }) + ); + }); +}); + +describe('assertRole', () => { + it('passes when the role matches', () => { + expect(assertRole(superadmin, 'SUPERADMIN')).toBe(superadmin); + }); + + it('throws 403 when an admin attempts a superadmin action', () => { + expect(() => assertRole(admin, 'SUPERADMIN')).toThrow( + expect.objectContaining({ status: 403 }) + ); + }); + + it('throws 401 when unauthenticated, not 403', () => { + expect(() => assertRole(null, 'SUPERADMIN')).toThrow( + expect.objectContaining({ status: 401 }) + ); + }); +}); + +describe('canAccessAllTeams', () => { + it('is true for superadmin', () => { + expect(canAccessAllTeams(superadmin)).toBe(true); + }); + it('is false for admin', () => { + expect(canAccessAllTeams(admin)).toBe(false); + }); +}); + +describe('assertTeamAccess', () => { + beforeEach(() => jest.clearAllMocks()); + + it('allows superadmin any team without querying', async () => { + await expect(assertTeamAccess(superadmin, 'any-team')).resolves.toBeUndefined(); + expect(getTeamIdsForUser).not.toHaveBeenCalled(); + }); + + it('allows an admin their assigned team', async () => { + (getTeamIdsForUser as jest.Mock).mockResolvedValue(['t1', 't2']); + await expect(assertTeamAccess(admin, 't2')).resolves.toBeUndefined(); + }); + + it('throws 403 when an admin requests a team they do not own', async () => { + (getTeamIdsForUser as jest.Mock).mockResolvedValue(['t1']); + await expect(assertTeamAccess(admin, 't-other')).rejects.toMatchObject({ + status: 403 + }); + }); + + it('throws 403 when an admin has no teams at all', async () => { + (getTeamIdsForUser as jest.Mock).mockResolvedValue([]); + await expect(assertTeamAccess(admin, 't1')).rejects.toMatchObject({ + status: 403 + }); + }); + + it('throws 401 when unauthenticated', async () => { + await expect(assertTeamAccess(null, 't1')).rejects.toMatchObject({ + status: 401 + }); + }); +}); + +describe('getAccessibleTeamIds', () => { + beforeEach(() => jest.clearAllMocks()); + + it('returns every team for superadmin', async () => { + (getAllTeamIds as jest.Mock).mockResolvedValue(['t1', 't2', 't3']); + await expect(getAccessibleTeamIds(superadmin)).resolves.toEqual([ + 't1', + 't2', + 't3' + ]); + }); + + it('returns only assigned teams for admin', async () => { + (getTeamIdsForUser as jest.Mock).mockResolvedValue(['t2']); + await expect(getAccessibleTeamIds(admin)).resolves.toEqual(['t2']); + }); +}); +``` + +- [ ] **Step 2: Run it to confirm it fails** + +Run: `cd web-server && yarn jest src/auth/__tests__/guard.test.ts` +Expected: FAIL — "Cannot find module '@/auth/guard'". + +- [ ] **Step 3: Implement** + +Create `web-server/src/auth/guard.ts`: + +```ts +import { Errors, ResponseError } from '@/constants/error'; + +import { getAllTeamIds, getTeamIdsForUser } from './queries'; +import { AuthSession, ClustoxRole } from './types'; + +export const assertAuthenticated = ( + session: AuthSession | null +): AuthSession => { + if (!session) throw new ResponseError(Errors.SESSION_USER_NOT_FOUND, 401); + return session; +}; + +export const assertRole = ( + session: AuthSession | null, + role: ClustoxRole +): AuthSession => { + const authed = assertAuthenticated(session); + if (authed.role !== role) throw new ResponseError(Errors.ACCESS_DENIED, 403); + return authed; +}; + +export const canAccessAllTeams = (session: AuthSession): boolean => + session.role === 'SUPERADMIN'; + +export const assertTeamAccess = async ( + session: AuthSession | null, + teamId: string +): Promise => { + const authed = assertAuthenticated(session); + if (canAccessAllTeams(authed)) return; + + const teamIds = await getTeamIdsForUser(authed.userId); + if (!teamIds.includes(teamId)) { + throw new ResponseError(Errors.ACCESS_DENIED, 403); + } +}; + +export const getAccessibleTeamIds = async ( + session: AuthSession +): Promise => + canAccessAllTeams(session) + ? getAllTeamIds() + : getTeamIdsForUser(session.userId); +``` + +- [ ] **Step 4: Run tests to verify they pass** + +Run: `cd web-server && yarn jest src/auth/__tests__/guard.test.ts` +Expected: 14 passed. + +- [ ] **Step 5: Commit** + +```bash +git add web-server/src/auth/guard.ts web-server/src/auth/__tests__/guard.test.ts +git commit -m "feat(auth): add role and team-access guards" +``` + +--- + +## Task 5: next-auth configuration and session reader + +**Files:** +- Create: `web-server/pages/api/auth/[...nextauth].ts` +- Create: `web-server/src/auth/session.ts` +- Create: `web-server/src/types/next-auth.d.ts` +- Modify: `env.example` and your local `.env` + +**Interfaces:** +- Consumes: `getAuthUserByEmail` (Task 3), `verifyPassword` (Task 2), `AuthSession` (Task 3) +- Produces: `getAuthSession(req: NextApiRequest): Promise` + +- [ ] **Step 1: Add the environment variables** + +Append to `env.example` and to your local `.env`: + +```bash +# CLUSTOX: auth +NEXTAUTH_URL=http://localhost:3333 +NEXTAUTH_SECRET=replace-me-with-openssl-rand-base64-32 +INTERNAL_API_TOKEN=replace-me-with-openssl-rand-hex-32 +SUPERADMIN_EMAIL=admin@clustox.com +SUPERADMIN_PASSWORD=change-this-before-any-shared-deployment +``` + +Generate real values: + +```bash +echo "NEXTAUTH_SECRET=$(openssl rand -base64 32)" +echo "INTERNAL_API_TOKEN=$(openssl rand -hex 32)" +``` + +- [ ] **Step 2: Declare the session type augmentation** + +Create `web-server/src/types/next-auth.d.ts`: + +```ts +import { ClustoxRole } from '@/auth/types'; + +declare module 'next-auth' { + interface Session { + user: { + id: string; + email: string; + name: string; + role: ClustoxRole; + }; + } + interface User { + id: string; + email: string; + name: string; + role: ClustoxRole; + } +} + +declare module 'next-auth/jwt' { + interface JWT { + userId: string; + role: ClustoxRole; + } +} +``` + +- [ ] **Step 3: Write the next-auth route** + +Create `web-server/pages/api/auth/[...nextauth].ts`: + +```ts +import NextAuth, { NextAuthOptions } from 'next-auth'; +import CredentialsProvider from 'next-auth/providers/credentials'; + +import { verifyPassword } from '@/auth/password'; +import { getAuthUserByEmail } from '@/auth/queries'; + +export const authOptions: NextAuthOptions = { + session: { strategy: 'jwt', maxAge: 12 * 60 * 60 }, + pages: { signIn: '/login' }, + providers: [ + CredentialsProvider({ + name: 'Credentials', + credentials: { + email: { label: 'Email', type: 'text' }, + password: { label: 'Password', type: 'password' } + }, + async authorize(credentials) { + if (!credentials?.email || !credentials?.password) return null; + + const user = await getAuthUserByEmail(credentials.email); + if (!user) return null; + + const ok = await verifyPassword(credentials.password, user.passwordHash); + if (!ok) return null; + + return { + id: user.userId, + email: user.email, + name: user.name, + role: user.role + }; + } + }) + ], + callbacks: { + async jwt({ token, user }) { + if (user) { + token.userId = user.id; + token.role = user.role; + } + return token; + }, + async session({ session, token }) { + session.user = { + id: token.userId, + email: token.email as string, + name: token.name as string, + role: token.role + }; + return session; + } + } +}; + +export default NextAuth(authOptions); +``` + +Team assignments are deliberately **not** in the token — see `getAccessibleTeamIds`, which reads them per request so a permission change takes effect immediately. + +- [ ] **Step 4: Write the session reader** + +Create `web-server/src/auth/session.ts`: + +```ts +import { NextApiRequest } from 'next'; +import { getToken } from 'next-auth/jwt'; + +import { AuthSession, ClustoxRole } from './types'; + +export const getAuthSession = async ( + req: NextApiRequest +): Promise => { + const token = await getToken({ req, secret: process.env.NEXTAUTH_SECRET }); + if (!token?.userId) return null; + + return { + userId: token.userId as string, + email: (token.email as string) ?? '', + name: (token.name as string) ?? '', + role: token.role as ClustoxRole + }; +}; +``` + +- [ ] **Step 5: Verify it compiles** + +Run: `cd web-server && yarn tsc --noEmit` +Expected: no errors. + +- [ ] **Step 6: Commit** + +```bash +git add web-server/pages/api/auth web-server/src/auth/session.ts web-server/src/types/next-auth.d.ts env.example +git commit -m "feat(auth): configure next-auth credentials provider" +``` + +--- + +## Task 6: Enforce auth in Endpoint.serve() + +The single highest-value change: one insertion point covers all 47 BFF routes. + +**Files:** +- Modify: `web-server/src/api-helpers/global.ts` +- Create: `web-server/src/api-helpers/__tests__/endpoint-auth.test.ts` + +**Interfaces:** +- Consumes: `getAuthSession` (Task 5), `assertAuthenticated` (Task 4) +- Produces: `req.session` populated on every authenticated request, typed as `AuthSession` + +- [ ] **Step 1: Write the failing test** + +Create `web-server/src/api-helpers/__tests__/endpoint-auth.test.ts`: + +```ts +jest.mock('@/auth/session', () => ({ getAuthSession: jest.fn() })); + +import { getAuthSession } from '@/auth/session'; +import { Endpoint, nullSchema } from '@/api-helpers/global'; + +const mockRes = () => { + const res: any = {}; + res.status = jest.fn(() => res); + res.send = jest.fn(() => res); + return res; +}; + +const mockReq = () => ({ method: 'GET', query: {}, body: {}, headers: {} }) as any; + +describe('Endpoint auth enforcement', () => { + beforeEach(() => jest.clearAllMocks()); + + it('rejects with 401 when no session and the endpoint is authenticated', async () => { + (getAuthSession as jest.Mock).mockResolvedValue(null); + + const endpoint = new Endpoint(nullSchema); + endpoint.handle.GET(nullSchema, async (_req, res) => res.send({ ok: true })); + + const res = mockRes(); + await endpoint.serve()(mockReq(), res); + + expect(res.status).toHaveBeenCalledWith(401); + }); + + it('allows the request through when a session exists', async () => { + (getAuthSession as jest.Mock).mockResolvedValue({ + userId: 'u1', + email: 'a@clustox.com', + name: 'A', + role: 'ADMIN' + }); + + const endpoint = new Endpoint(nullSchema); + endpoint.handle.GET(nullSchema, async (_req, res) => res.send({ ok: true })); + + const res = mockRes(); + await endpoint.serve()(mockReq(), res); + + expect(res.send).toHaveBeenCalledWith({ ok: true }); + }); + + it('skips the check for endpoints marked unauthenticated', async () => { + (getAuthSession as jest.Mock).mockResolvedValue(null); + + const endpoint = new Endpoint(nullSchema, { unauthenticated: true }); + endpoint.handle.GET(nullSchema, async (_req, res) => res.send({ ok: true })); + + const res = mockRes(); + await endpoint.serve()(mockReq(), res); + + expect(res.send).toHaveBeenCalledWith({ ok: true }); + expect(getAuthSession).not.toHaveBeenCalled(); + }); + + it('exposes the session on the request object', async () => { + const session = { + userId: 'u1', + email: 'a@clustox.com', + name: 'A', + role: 'ADMIN' as const + }; + (getAuthSession as jest.Mock).mockResolvedValue(session); + + let seen: any = null; + const endpoint = new Endpoint(nullSchema); + endpoint.handle.GET(nullSchema, async (req, res) => { + seen = (req as any).session; + res.send({ ok: true }); + }); + + await endpoint.serve()(mockReq(), mockRes()); + expect(seen).toEqual(session); + }); +}); +``` + +- [ ] **Step 2: Run it to confirm it fails** + +Run: `cd web-server && yarn jest src/api-helpers/__tests__/endpoint-auth.test.ts` +Expected: FAIL — the first test gets a 200/500 rather than 401, because the flag is never read. + +- [ ] **Step 3: Implement the check** + +In `web-server/src/api-helpers/global.ts`, add the imports: + +```ts +// CLUSTOX: enforce the `authenticated` flag that upstream declared but never read. +import { assertAuthenticated } from '@/auth/guard'; +import { getAuthSession } from '@/auth/session'; +``` + +Then inside `serve()`, immediately after `const req = transformNextRequest(nextReq);`: + +```ts + // CLUSTOX: single enforcement point for all BFF routes. Routes opt out + // with `new Endpoint(schema, { unauthenticated: true })`. + if (this.authenticated) { + const session = await getAuthSession(nextReq); + assertAuthenticated(session); + (req as any).session = session; + } +``` + +- [ ] **Step 4: Run tests to verify they pass** + +Run: `cd web-server && yarn jest src/api-helpers/__tests__/endpoint-auth.test.ts` +Expected: 4 passed. + +- [ ] **Step 5: Run the whole suite for regressions** + +Run: `cd web-server && yarn jest` +Expected: all pass. Any pre-existing API-route test that now 401s must be updated to mock a session — that is a correct failure, not a broken test. + +- [ ] **Step 6: Commit** + +```bash +git add web-server/src/api-helpers/global.ts web-server/src/api-helpers/__tests__/endpoint-auth.test.ts +git commit -m "feat(auth): enforce authentication in Endpoint.serve()" +``` + +--- + +## Task 7: Login page and page-level guard + +**Files:** +- Create: `web-server/pages/login.tsx` +- Modify: `web-server/middleware.ts` + +**Interfaces:** +- Consumes: next-auth route (Task 5) +- Produces: `/login` route; unauthenticated page requests redirect there + +- [ ] **Step 1: Build the login page** + +Create `web-server/pages/login.tsx`: + +```tsx +import { Box, Button, Card, TextField, Typography } from '@mui/material'; +import { signIn } from 'next-auth/react'; +import { useRouter } from 'next/router'; +import { FormEvent, useState } from 'react'; + +export default function Login() { + const router = useRouter(); + const [email, setEmail] = useState(''); + const [password, setPassword] = useState(''); + const [error, setError] = useState(''); + const [busy, setBusy] = useState(false); + + const onSubmit = async (e: FormEvent) => { + e.preventDefault(); + setBusy(true); + setError(''); + + const res = await signIn('credentials', { + email, + password, + redirect: false + }); + + setBusy(false); + + if (res?.error) { + // Deliberately does not distinguish unknown email from wrong password. + setError('Invalid email or password'); + return; + } + router.replace('/'); + }; + + return ( + + + Sign in +
+ setEmail(e.target.value)} required + /> + setPassword(e.target.value)} required + /> + {error && ( + {error} + )} + + +
+
+ ); +} + +Login.getLayout = (page: React.ReactElement) => page; +``` + +`getLayout` returning the bare page keeps the sidebar shell off the login screen. + +- [ ] **Step 2: Guard pages in middleware** + +Replace `web-server/middleware.ts` with: + +```ts +import { getToken } from 'next-auth/jwt'; +import { NextRequest, NextResponse } from 'next/server'; + +import { getFeaturesFromReq } from '@/api-helpers/features'; +import { defaultFlags } from '@/constants/feature'; + +// CLUSTOX: paths reachable without a session. +const PUBLIC_PATHS = ['/login', '/api/auth', '/api/status', '/api/db_status', '/api/internal_status', '/api/hello']; + +export async function middleware(request: NextRequest) { + const flagOverrides = getFeaturesFromReq(request as any); + const flags = { ...defaultFlags, ...flagOverrides }; + + const url = request.nextUrl.clone(); + const isPublic = PUBLIC_PATHS.some((p) => url.pathname.startsWith(p)); + + // CLUSTOX: redirect unauthenticated page requests to the sign-in screen. + if (!isPublic) { + const token = await getToken({ + req: request, + secret: process.env.NEXTAUTH_SECRET + }); + if (!token) { + const login = request.nextUrl.clone(); + login.pathname = '/login'; + login.search = ''; + return NextResponse.redirect(login); + } + } + + // Forward as-is if it's a next-auth URL, except /session + if ( + url.pathname.startsWith('/api/auth') && + !url.pathname.startsWith('/api/auth/session') + ) { + return NextResponse.next(); + } + + url.searchParams.append('feature_flags', JSON.stringify(flags)); + + return NextResponse.rewrite(url); +} + +export const config = { + matcher: [ + // CLUSTOX: widened from the API-only matcher so pages are guarded too. + '/((?!_next/static|_next/image|favicon.ico|assets|.*\\.png$|.*\\.svg$).*)' + ] +}; +``` + +- [ ] **Step 3: Verify the redirect manually** + +```bash +docker compose up -d --build +curl -s -o /dev/null -w "%{http_code} -> %{redirect_url}\n" http://localhost:3333/dora-metrics +``` + +Expected: `307 -> http://localhost:3333/login` + +- [ ] **Step 4: Verify the login page renders** + +Run: `curl -s http://localhost:3333/login | grep -c "Sign in"` +Expected: at least 1. + +- [ ] **Step 5: Commit** + +```bash +git add web-server/pages/login.tsx web-server/middleware.ts +git commit -m "feat(auth): add login page and page-level guard" +``` + +--- + +## Task 8: Superadmin bootstrap + +**Files:** +- Create: `web-server/src/auth/bootstrap.ts` +- Create: `web-server/instrumentation.ts` +- Create: `web-server/src/auth/__tests__/bootstrap.test.ts` + +**Interfaces:** +- Consumes: `countSuperadmins`, `createUser` (Task 3) +- Produces: `bootstrapSuperadmin(): Promise` + +- [ ] **Step 1: Write the failing test** + +Create `web-server/src/auth/__tests__/bootstrap.test.ts`: + +```ts +jest.mock('@/auth/queries', () => ({ + countSuperadmins: jest.fn(), + createUser: jest.fn() +})); +jest.mock('@/utils/db', () => ({ db: jest.fn() })); + +import { bootstrapSuperadmin } from '@/auth/bootstrap'; +import { countSuperadmins, createUser } from '@/auth/queries'; + +describe('bootstrapSuperadmin', () => { + const env = process.env; + beforeEach(() => { + jest.clearAllMocks(); + process.env = { ...env }; + }); + afterAll(() => { + process.env = env; + }); + + it('does nothing when a superadmin already exists', async () => { + (countSuperadmins as jest.Mock).mockResolvedValue(1); + process.env.SUPERADMIN_EMAIL = 'a@clustox.com'; + process.env.SUPERADMIN_PASSWORD = 'pw'; + + await bootstrapSuperadmin(); + expect(createUser).not.toHaveBeenCalled(); + }); + + it('does not create a user when env vars are missing', async () => { + (countSuperadmins as jest.Mock).mockResolvedValue(0); + delete process.env.SUPERADMIN_EMAIL; + delete process.env.SUPERADMIN_PASSWORD; + + await bootstrapSuperadmin(); + expect(createUser).not.toHaveBeenCalled(); + }); + + it('creates the superadmin when none exists and env vars are set', async () => { + (countSuperadmins as jest.Mock).mockResolvedValue(0); + (createUser as jest.Mock).mockResolvedValue('new-id'); + process.env.SUPERADMIN_EMAIL = 'boss@clustox.com'; + process.env.SUPERADMIN_PASSWORD = 'strong-password'; + + await bootstrapSuperadmin('org-1'); + + expect(createUser).toHaveBeenCalledWith( + expect.objectContaining({ + email: 'boss@clustox.com', + password: 'strong-password', + role: 'SUPERADMIN', + teamIds: [], + orgId: 'org-1' + }) + ); + }); +}); +``` + +- [ ] **Step 2: Run it to confirm it fails** + +Run: `cd web-server && yarn jest src/auth/__tests__/bootstrap.test.ts` +Expected: FAIL — "Cannot find module '@/auth/bootstrap'". + +- [ ] **Step 3: Implement** + +Create `web-server/src/auth/bootstrap.ts`: + +```ts +import { Table } from '@/constants/db'; +import { db } from '@/utils/db'; + +import { countSuperadmins, createUser } from './queries'; + +const getDefaultOrgId = async (): Promise => { + const row = await db(Table.Organization).select('id').first(); + return row?.id ?? null; +}; + +export const bootstrapSuperadmin = async ( + orgIdOverride?: string +): Promise => { + const existing = await countSuperadmins(); + if (existing > 0) return; + + const email = process.env.SUPERADMIN_EMAIL; + const password = process.env.SUPERADMIN_PASSWORD; + + if (!email || !password) { + // Fail loudly but do not invent a default account. A known-value fallback + // is worse than having no superadmin at all. + console.error( + '[CLUSTOX] No superadmin exists and SUPERADMIN_EMAIL / SUPERADMIN_PASSWORD ' + + 'are not set. Nobody can administer this instance until they are.' + ); + return; + } + + const orgId = orgIdOverride ?? (await getDefaultOrgId()); + if (!orgId) { + console.error('[CLUSTOX] No organization found; cannot bootstrap superadmin.'); + return; + } + + await createUser({ + name: 'Superadmin', + email, + password, + role: 'SUPERADMIN', + teamIds: [], + orgId + }); + + console.info(`[CLUSTOX] Bootstrapped superadmin ${email}`); +}; +``` + +Create `web-server/instrumentation.ts`: + +```ts +export async function register() { + if (process.env.NEXT_RUNTIME === 'nodejs') { + const { bootstrapSuperadmin } = await import('@/auth/bootstrap'); + await bootstrapSuperadmin(); + } +} +``` + +The `NEXT_RUNTIME` check keeps this off the Edge runtime, where knex cannot run. + +- [ ] **Step 4: Run tests to verify they pass** + +Run: `cd web-server && yarn jest src/auth/__tests__/bootstrap.test.ts` +Expected: 3 passed. + +- [ ] **Step 5: Verify the real bootstrap runs** + +```bash +docker compose up -d --build +docker exec middleware-dev psql "postgresql://postgres:postgres@localhost:5434/mhq-oss" \ + -c 'SELECT u.primary_email, a.role FROM "ClustoxUserAuth" a JOIN "Users" u ON u.id = a.user_id;' +``` + +Expected: one row, role `SUPERADMIN`, matching `SUPERADMIN_EMAIL`. + +- [ ] **Step 6: Log in through the UI** + +Open http://localhost:3333, sign in with the superadmin credentials. Expected: the dashboard loads. + +- [ ] **Step 7: Commit** + +```bash +git add web-server/src/auth/bootstrap.ts web-server/instrumentation.ts web-server/src/auth/__tests__/bootstrap.test.ts +git commit -m "feat(auth): bootstrap first superadmin from environment" +``` + +--- + +## Task 9: Lock down the Flask servers + +Without this, `curl :9696` bypasses every guard added so far. + +**Files:** +- Create: `backend/analytics_server/mhq/clustox_auth/__init__.py` +- Create: `backend/analytics_server/mhq/clustox_auth/internal_token.py` +- Create: `backend/analytics_server/tests/clustox_auth/__init__.py` +- Create: `backend/analytics_server/tests/clustox_auth/test_internal_token.py` +- Create: `setup_utils/trigger_sync.sh` +- Modify: `backend/analytics_server/app.py`, `backend/analytics_server/sync_app.py`, `setup_utils/cronjob.txt`, `web-server/src/api-helpers/axios.ts` + +**Interfaces:** +- Consumes: `INTERNAL_API_TOKEN` env var +- Produces: `register_internal_token_guard(app: Flask) -> None` + +- [ ] **Step 1: Write the failing test** + +Create `backend/analytics_server/tests/clustox_auth/__init__.py` (empty) and `backend/analytics_server/tests/clustox_auth/test_internal_token.py`: + +```python +import os + +import pytest +from flask import Flask + +from mhq.clustox_auth.internal_token import register_internal_token_guard + + +def build_app(token): + if token is None: + os.environ.pop("INTERNAL_API_TOKEN", None) + else: + os.environ["INTERNAL_API_TOKEN"] = token + + app = Flask(__name__) + register_internal_token_guard(app) + + @app.route("/protected") + def protected(): + return {"ok": True} + + return app + + +def test_request_with_valid_token_is_allowed(): + app = build_app("secret-token") + client = app.test_client() + res = client.get("/protected", headers={"X-Internal-Token": "secret-token"}) + assert res.status_code == 200 + + +def test_request_without_token_is_rejected(): + app = build_app("secret-token") + res = app.test_client().get("/protected") + assert res.status_code == 403 + + +def test_request_with_wrong_token_is_rejected(): + app = build_app("secret-token") + client = app.test_client() + res = client.get("/protected", headers={"X-Internal-Token": "wrong"}) + assert res.status_code == 403 + + +def test_unset_env_fails_closed(): + """An unconfigured token must reject everything, never allow everything.""" + app = build_app(None) + client = app.test_client() + res = client.get("/protected", headers={"X-Internal-Token": "anything"}) + assert res.status_code == 403 + + +def test_health_endpoint_is_exempt(): + app = build_app("secret-token") + + @app.route("/") + def health(): + return "hello" + + res = app.test_client().get("/") + assert res.status_code == 200 +``` + +- [ ] **Step 2: Run it to confirm it fails** + +Run: `cd backend/analytics_server && python -m pytest tests/clustox_auth/test_internal_token.py -v` +Expected: FAIL — `ModuleNotFoundError: No module named 'mhq.clustox_auth'`. + +- [ ] **Step 3: Implement the guard** + +Create `backend/analytics_server/mhq/clustox_auth/__init__.py` (empty), then `backend/analytics_server/mhq/clustox_auth/internal_token.py`: + +```python +""" +CLUSTOX: restrict the Flask servers to traffic from the Next.js BFF. + +Upstream ships these APIs unauthenticated, relying on the loopback port binding +in docker-compose. Once authorization lives in the BFF, a direct call to :9696 +would bypass every check, so every request must carry a shared secret. + +Fails closed: if INTERNAL_API_TOKEN is unset, all requests are rejected. +""" + +import hmac +from os import getenv + +from flask import Flask, jsonify, request + +HEADER = "X-Internal-Token" + +# Health checks must stay reachable for container and UI status probes. +EXEMPT_PATHS = {"/"} + + +def register_internal_token_guard(app: Flask) -> None: + @app.before_request + def _verify_internal_token(): + if request.path in EXEMPT_PATHS: + return None + + expected = getenv("INTERNAL_API_TOKEN") + if not expected: + return jsonify({"error": "internal token not configured"}), 403 + + provided = request.headers.get(HEADER, "") + if not hmac.compare_digest(provided, expected): + return jsonify({"error": "forbidden"}), 403 + + return None +``` + +`hmac.compare_digest` avoids leaking the token through response-timing differences. + +- [ ] **Step 4: Run tests to verify they pass** + +Run: `cd backend/analytics_server && python -m pytest tests/clustox_auth/test_internal_token.py -v` +Expected: 5 passed. + +- [ ] **Step 5: Register the guard on both servers** + +In `backend/analytics_server/app.py`, after the blueprint registrations and before `configure_db_with_app(app)`: + +```python +# CLUSTOX: restrict this API to the BFF. See mhq/clustox_auth/internal_token.py +from mhq.clustox_auth.internal_token import register_internal_token_guard + +register_internal_token_guard(app) +``` + +Add the identical two lines to `backend/analytics_server/sync_app.py`. + +- [ ] **Step 6: Send the token from the BFF** + +In `web-server/src/api-helpers/axios.ts`, immediately after the `internalSyncServer` definition: + +```ts +// CLUSTOX: the Flask servers reject any request without this shared secret. +const internalTokenInterceptor = (req: any) => { + req.headers = req.headers || {}; + req.headers['X-Internal-Token'] = process.env.INTERNAL_API_TOKEN; + return req; +}; + +internal.interceptors.request.use(internalTokenInterceptor); +internalSyncServer.interceptors.request.use(internalTokenInterceptor); +``` + +- [ ] **Step 7: Give cron the token** + +Create `setup_utils/trigger_sync.sh`: + +```bash +#!/bin/bash +# CLUSTOX: cron cannot read the container env, so source it before calling the +# sync server, which now requires the internal token. +set -a +[ -f /app/.env ] && . /app/.env +set +a + +curl -s -X POST \ + -H "X-Internal-Token: ${INTERNAL_API_TOKEN}" \ + "http://localhost:${SYNC_SERVER_PORT:-9697}/sync" +``` + +Replace the job line in `setup_utils/cronjob.txt`: + +``` +# CLUSTOX: routed through trigger_sync.sh so the internal token is sent. +*/30 * * * * /app/setup_utils/trigger_sync.sh >> /var/log/cron/cron.log 2>&1 +``` + +Add to `Dockerfile.dev`, alongside the existing `chmod +x` line: + +```dockerfile +RUN chmod +x /app/setup_utils/trigger_sync.sh +``` + +- [ ] **Step 8: Verify direct access is now blocked** + +```bash +docker compose up -d --build +# Direct call — must be refused +curl -s -o /dev/null -w "direct: %{http_code}\n" http://localhost:9696/teams/any/lead_time +# Health check — must still work +curl -s -o /dev/null -w "health: %{http_code}\n" http://localhost:9696/ +``` + +Expected: `direct: 403`, `health: 200`. + +- [ ] **Step 9: Verify the app and cron still work** + +Sign in at http://localhost:3333 and confirm dashboards still render (proving the BFF's header works), then: + +```bash +docker exec middleware-dev /app/setup_utils/trigger_sync.sh +``` + +Expected: `{"message":"sync started",...}`. **Do not skip this** — a sync broken by the token change fails silently ([ARCHITECTURE.md §7.4](../../ARCHITECTURE.md)). + +- [ ] **Step 10: Commit** + +```bash +git add backend/analytics_server/mhq/clustox_auth backend/analytics_server/tests/clustox_auth \ + backend/analytics_server/app.py backend/analytics_server/sync_app.py \ + setup_utils/trigger_sync.sh setup_utils/cronjob.txt Dockerfile.dev \ + web-server/src/api-helpers/axios.ts +git commit -m "feat(auth): restrict Flask servers to BFF via shared secret" +``` + +--- + +## Task 10: Apply per-team authorization to routes + +**Files:** +- Modify: `web-server/pages/api/resources/team_repos.ts` +- Modify: `web-server/pages/api/internal/team/[team_id]/*.ts` route files that export a default +- Modify: `web-server/pages/api/resources/orgs/[org_id]/teams/index.ts` and `v2.ts` +- Modify: `web-server/pages/api/resources/share.ts` +- Modify: `web-server/pages/api/integrations/index.ts` +- Create: `web-server/src/auth/__tests__/route-authz.test.ts` + +**Interfaces:** +- Consumes: `assertTeamAccess`, `assertRole`, `getAccessibleTeamIds` (Task 4); `req.session` (Task 6) +- Produces: team-scoped routes 403 for unauthorised teams; team lists filtered to accessible teams + +- [ ] **Step 1: Enumerate the routes needing scoping** + +```bash +cd web-server +grep -rln "team_id" pages/api --include=*.ts | xargs grep -l "export default" | sort +``` + +Record the list. Every route accepting a `team_id` needs `assertTeamAccess`. + +- [ ] **Step 2: Write the failing test** + +Create `web-server/src/auth/__tests__/route-authz.test.ts`: + +```ts +jest.mock('@/auth/queries', () => ({ + getTeamIdsForUser: jest.fn(), + getAllTeamIds: jest.fn() +})); + +import { assertTeamAccess, getAccessibleTeamIds } from '@/auth/guard'; +import { getAllTeamIds, getTeamIdsForUser } from '@/auth/queries'; +import { AuthSession } from '@/auth/types'; + +const admin: AuthSession = { + userId: 'ad1', email: 'l@clustox.com', name: 'L', role: 'ADMIN' +}; +const superadmin: AuthSession = { + userId: 'su1', email: 'b@clustox.com', name: 'B', role: 'SUPERADMIN' +}; + +describe('team scoping behaviour relied on by routes', () => { + beforeEach(() => jest.clearAllMocks()); + + it('an admin is refused a team outside their assignment', async () => { + (getTeamIdsForUser as jest.Mock).mockResolvedValue(['team-zda']); + await expect(assertTeamAccess(admin, 'team-cgpt')).rejects.toMatchObject({ + status: 403 + }); + }); + + it('a team list for an admin contains only their teams', async () => { + (getTeamIdsForUser as jest.Mock).mockResolvedValue(['team-zda']); + await expect(getAccessibleTeamIds(admin)).resolves.toEqual(['team-zda']); + }); + + it('a team list for a superadmin contains every team', async () => { + (getAllTeamIds as jest.Mock).mockResolvedValue(['team-zda', 'team-cgpt']); + await expect(getAccessibleTeamIds(superadmin)).resolves.toEqual([ + 'team-zda', + 'team-cgpt' + ]); + }); +}); +``` + +- [ ] **Step 3: Run it** + +Run: `cd web-server && yarn jest src/auth/__tests__/route-authz.test.ts` +Expected: 3 passed (guard already exists from Task 4 — this pins the behaviour routes depend on). + +- [ ] **Step 4: Add the guard to each team-scoped route** + +For every handler taking `team_id`, insert immediately after the handler opens. Example for `team_repos.ts`: + +```ts +endpoint.handle.GET(getSchema, async (req, res) => { + // CLUSTOX: an admin may only read teams assigned to them. + await assertTeamAccess((req as any).session, req.payload.team_id); + + if (req.meta?.features?.use_mock_data) { + return res.send(teamReposMock); + } + res.send(await getTeamRepos(req.payload.team_id)); +}); +``` + +with the import: + +```ts +// CLUSTOX: per-team authorization +import { assertTeamAccess } from '@/auth/guard'; +``` + +Apply to **both** GET and PATCH handlers, and to every route from Step 1. + +- [ ] **Step 5: Filter the team list endpoints** + +In `pages/api/resources/orgs/[org_id]/teams/index.ts` and `v2.ts`, filter the result before sending: + +```ts +// CLUSTOX: an admin sees only their teams in the picker. +import { getAccessibleTeamIds } from '@/auth/guard'; + +const accessible = await getAccessibleTeamIds((req as any).session); +const visibleTeams = teams.filter((t: { id: string }) => accessible.includes(t.id)); +res.send(visibleTeams); +``` + +Adjust the variable names to match each file. + +- [ ] **Step 6: Restrict integrations to superadmin** + +In `pages/api/integrations/index.ts`, at the top of each mutating handler: + +```ts +// CLUSTOX: only superadmins manage integrations. +import { assertRole } from '@/auth/guard'; + +assertRole((req as any).session, 'SUPERADMIN'); +``` + +- [ ] **Step 7: Make share.ts authenticated** + +In `pages/api/resources/share.ts`, change: + +```ts +// CLUSTOX: was unauthenticated, which allowed anonymous writes to +// URLShortenerData. Everyone who should open a share link has an account. +const endpoint = new Endpoint(nullSchema); +``` + +- [ ] **Step 8: Verify manually** + +Sign in as superadmin, note both team IDs, then create an admin scoped to one (Task 11 provides the UI; until then insert the `ClustoxUserTeamAccess` row by hand). Sign in as that admin and: + +```bash +# In the browser devtools console while signed in as the scoped admin: +fetch('/api/resources/team_repos?team_id=').then(r => console.log(r.status)) +``` + +Expected: `403`. + +- [ ] **Step 9: Commit** + +```bash +git add web-server/pages/api web-server/src/auth/__tests__/route-authz.test.ts +git commit -m "feat(auth): scope team routes and restrict integrations to superadmin" +``` + +--- + +## Task 11: User management API and page + +**Files:** +- Create: `web-server/pages/api/clustox/users/index.ts` +- Create: `web-server/pages/api/clustox/users/[user_id].ts` +- Create: `web-server/pages/users.tsx` + +**Interfaces:** +- Consumes: `listUsers`, `createUser`, `updateUserRole`, `setUserTeams` (Task 3); `assertRole` (Task 4) +- Produces: `GET/POST /api/clustox/users`, `PATCH /api/clustox/users/[user_id]` + +- [ ] **Step 1: Build the list/create route** + +Create `web-server/pages/api/clustox/users/index.ts`: + +```ts +import * as yup from 'yup'; + +import { assertRole } from '@/auth/guard'; +import { createUser, listUsers } from '@/auth/queries'; +import { Endpoint, nullSchema } from '@/api-helpers/global'; +import { Table } from '@/constants/db'; +import { db } from '@/utils/db'; + +const postSchema = yup.object().shape({ + name: yup.string().required(), + email: yup.string().email().required(), + password: yup.string().min(12).required(), + role: yup.string().oneOf(['SUPERADMIN', 'ADMIN']).required(), + team_ids: yup.array().of(yup.string().uuid()).required() +}); + +const endpoint = new Endpoint(nullSchema); + +endpoint.handle.GET(nullSchema, async (req, res) => { + assertRole((req as any).session, 'SUPERADMIN'); + res.send(await listUsers()); +}); + +endpoint.handle.POST(postSchema, async (req, res) => { + assertRole((req as any).session, 'SUPERADMIN'); + + const org = await db(Table.Organization).select('id').first(); + const userId = await createUser({ + name: req.payload.name, + email: req.payload.email, + password: req.payload.password, + role: req.payload.role, + teamIds: req.payload.team_ids, + orgId: org.id + }); + + res.send({ user_id: userId }); +}); + +export default endpoint.serve(); +``` + +A 12-character minimum is enforced at the schema, so the UI cannot create a weak account. + +- [ ] **Step 2: Build the update route** + +Create `web-server/pages/api/clustox/users/[user_id].ts`: + +```ts +import * as yup from 'yup'; + +import { assertRole } from '@/auth/guard'; +import { setUserTeams, updateUserRole } from '@/auth/queries'; +import { Endpoint } from '@/api-helpers/global'; + +const pathSchema = yup.object().shape({ + user_id: yup.string().uuid().required() +}); + +const patchSchema = yup.object().shape({ + user_id: yup.string().uuid().required(), + role: yup.string().oneOf(['SUPERADMIN', 'ADMIN']).optional(), + team_ids: yup.array().of(yup.string().uuid()).optional() +}); + +const endpoint = new Endpoint(pathSchema); + +endpoint.handle.PATCH(patchSchema, async (req, res) => { + assertRole((req as any).session, 'SUPERADMIN'); + + if (req.payload.role) { + await updateUserRole(req.payload.user_id, req.payload.role); + } + if (req.payload.team_ids) { + await setUserTeams(req.payload.user_id, req.payload.team_ids); + } + + res.send({ ok: true }); +}); + +export default endpoint.serve(); +``` + +- [ ] **Step 3: Build the users page** + +Create `web-server/pages/users.tsx` with a table of users (name, email, role, team count) and a form to create one — name, email, password, role select, team multi-select. Fetch `GET /api/clustox/users` on mount; `POST` on submit; refresh the list afterwards. + +Guard the page client-side: + +```tsx +const { role } = useAuth(); +if (role !== 'SUPERADMIN') return Not authorised.; +``` + +This is presentation only — the API enforces the real rule. + +- [ ] **Step 4: Verify end to end** + +1. Sign in as superadmin → `/users` loads +2. Create an admin assigned only to `ZDA` +3. Sign out, sign in as that admin +4. `/users` shows "Not authorised"; `GET /api/clustox/users` returns 403 +5. Team picker shows only `ZDA` + +- [ ] **Step 5: Commit** + +```bash +git add web-server/pages/api/clustox web-server/pages/users.tsx +git commit -m "feat(auth): add user management API and page" +``` + +--- + +## Task 12: End-to-end verification + +**Files:** +- Create: `web-server/e2e/auth.spec.ts` + +**Interfaces:** +- Consumes: everything above +- Produces: an automated run of the demo script + +- [ ] **Step 1: Write the Playwright spec** + +Create `web-server/e2e/auth.spec.ts`. **URLs are absolute on purpose:** upstream's +`playwright.config.ts` sets no `baseURL`, and adding one would mean touching an eighth upstream file +for no functional gain. + +```ts +import { expect, test } from '@playwright/test'; + +const APP = 'http://localhost:3333'; +const API = 'http://localhost:9696'; + +const SUPERADMIN = { + email: process.env.SUPERADMIN_EMAIL!, + password: process.env.SUPERADMIN_PASSWORD! +}; + +const signIn = async (page: any, email: string, password: string) => { + await page.goto(`${APP}/login`); + await page.fill('input[type="email"]', email); + await page.fill('input[type="password"]', password); + await page.click('button[type="submit"]'); + await page.waitForURL((u: URL) => !u.pathname.startsWith('/login')); +}; + +test('unauthenticated visitors are redirected to login', async ({ page }) => { + await page.goto(`${APP}/dora-metrics`); + await expect(page).toHaveURL(/\/login/); +}); + +test('bad credentials are rejected', async ({ page }) => { + await page.goto(`${APP}/login`); + await page.fill('input[type="email"]', 'nobody@clustox.com'); + await page.fill('input[type="password"]', 'wrong-password'); + await page.click('button[type="submit"]'); + await expect(page.getByRole('alert')).toContainText('Invalid email or password'); +}); + +test('superadmin signs in and sees user management', async ({ page }) => { + await signIn(page, SUPERADMIN.email, SUPERADMIN.password); + await page.goto(`${APP}/users`); + await expect(page.getByText('Not authorised')).toHaveCount(0); +}); + +test('the Flask API refuses direct requests', async ({ request }) => { + const res = await request.get(`${API}/teams/any-id/lead_time`, { + failOnStatusCode: false + }); + expect(res.status()).toBe(403); +}); +``` + +No jest config change is needed: `jest.config.js` already lists `auth.spec.ts` in +`testPathIgnorePatterns`, so jest will not try to run this Playwright spec. (That rule is another +piece of vestigial upstream auth scaffolding — the file it referenced does not exist.) + +The app must be running before this suite: `docker compose up -d`. + +- [ ] **Step 2: Run it** + +Run: `cd web-server && yarn playwright test e2e/auth.spec.ts` +Expected: 4 passed. + +- [ ] **Step 3: Run every suite** + +```bash +cd web-server && yarn jest && yarn tsc --noEmit +cd ../backend/analytics_server && python -m pytest tests/ -q +cd ../.. && black --check backend/ && flake8 backend/ +``` + +Expected: all green. + +- [ ] **Step 4: Audit the fork footprint** + +```bash +grep -rn "CLUSTOX:" --include='*.py' --include='*.ts' --include='*.tsx' --include='*.sql' \ + --include='*.sh' --include='*.txt' . | grep -v node_modules +``` + +Expected: sentinels in all seven modified upstream files. Record the count in [FORK_STRATEGY.md](../../FORK_STRATEGY.md) terms — this is the number the next upstream sync checks against. + +- [ ] **Step 5: Walk the demo script** + +Run every step of [spec §7](../specs/2026-08-05-auth-rbac-design.md) by hand. Steps 5 and 6 are the ones to be certain about. + +- [ ] **Step 6: Commit** + +```bash +git add web-server/e2e/auth.spec.ts +git commit -m "test(auth): add end-to-end auth and access-control coverage" +``` + +--- + +## Self-review notes + +**Spec coverage:** every §2 in-scope item maps to a task — login (5, 7), bootstrap (8), two roles (1, 4), per-team scoping (4, 10), user management (11), Flask lockdown (9), Google-ready structure (5, via the providers array). Every §8 risk has a step: `share.ts` (10.7), password handling (2), bypass route (10.1 enumeration), fail-closed (9.1 test), cron (9.9), weak bootstrap (8.3), divergence (12.4). + +**Deferred deliberately:** `pages/users.tsx` (11.3) is described rather than written out in full, because it is standard MUI table-and-form work with no non-obvious logic, and its authorization is enforced server-side and already tested. Every step containing security-relevant logic has literal code. + +**Type consistency:** `AuthSession`, `ClustoxRole`, `assertTeamAccess`, `getAccessibleTeamIds`, `getTeamIdsForUser` are used with identical signatures across Tasks 3, 4, 6, 10, 11. diff --git a/docs/superpowers/specs/2026-08-05-auth-rbac-design.md b/docs/superpowers/specs/2026-08-05-auth-rbac-design.md new file mode 100644 index 00000000..86a8e1e5 --- /dev/null +++ b/docs/superpowers/specs/2026-08-05-auth-rbac-design.md @@ -0,0 +1,322 @@ +# Design: Authentication & Role-Based Access Control + +| | | +|---|---| +| **Date** | 2026-08-05 | +| **Status** | Approved — ready for implementation planning | +| **Repo** | [Clustox/middleware](https://github.com/Clustox/middleware) (fork of middlewarehq/middleware) | +| **Baseline** | `844eb42` | +| **Related** | [ARCHITECTURE.md](../../ARCHITECTURE.md) · [FORK_STRATEGY.md](../../FORK_STRATEGY.md) | + +--- + +## 1. Context + +The application currently has **no authentication of any kind**. Anyone who can reach the app +gets full access to every dashboard, every setting, and every integration. + +This is not an oversight in our deployment — it is how upstream ships. Verified: + +```bash +# Backend API — zero matches +grep -rniE "login_required|authenticat|jwt|api_key|bearer" backend/analytics_server/mhq/api/ +# Frontend — no NextAuth route, no sign-in page +find web-server/pages -path '*nextauth*' +ls web-server/pages | grep -iE 'login|signin' +``` + +Upstream evidently *had* auth and stripped it for the open-source build. What remains is +scaffolding with the enforcement removed: + +| Artefact | State | +|---|---| +| `web-server/src/components/Authenticated/index.tsx` | Wraps every page. Sets `verified = true` unconditionally and shows a toast reading "You are successfully authenticated!". Checks nothing. | +| `Endpoint.authenticated` (`src/api-helpers/global.ts`) | Field declared, constructor accepts `{ unauthenticated: true }`, **`serve()` never reads it**. | +| `UserRole` enum (`src/types/resources.ts`) | `EM`, `ENGINEER`, `MOM`. Never enforced. | +| `AuthContext.role` | Hardcoded to `UserRole.MOM` (the most privileged). | +| `AuthContext.userId` | Hardcoded to `00000000-0000-0000-0000-000000000000`. | +| `Users.role_override` column | Exists in schema, referenced nowhere in code. | +| `next-auth` in `package.json` | No `[...nextauth]` route, no sign-in page. Only imports are a `Session` *type* and a `signIn` call in an error component that has nothing to call. | + +**This is good news.** The hook points we need were designed in; only the wiring was removed. +Five routes already correctly declare `{ unauthenticated: true }`, which tells us the opt-out +mechanism was intended to be enforced. + +### Goal + +Only people with credentials reach the dashboards. Superadmins manage users and integrations and +see everything; admins see only the teams assigned to them. + +### Considered and rejected: an identity-aware proxy + +Putting `oauth2-proxy` / Cloudflare Access in front of the app would solve the perimeter problem +with **zero fork changes** and provide Google sign-in for free. It was recommended and rejected in +favour of in-app auth, deliberately, for two reasons: + +1. A proxy is all-or-nothing. It cannot express "this admin sees only Team ZDA", which is a stated + requirement. +2. The team wants a demonstrable product capability, not a deployment configuration change. + +This is recorded so the trade-off is not relitigated later, and so the per-team scoping +requirement — the thing that actually justifies building in-app — is not quietly dropped from +scope. **If per-team scoping is ever cut, revisit the proxy option instead.** + +--- + +## 2. Scope + +### In scope (v1) + +- Email + password login, bcrypt-hashed +- Superadmin bootstrapped from environment variables on first boot +- Two roles: `SUPERADMIN`, `ADMIN` +- Per-team access scoping for `ADMIN` +- User management page (superadmin only): create user, set role, assign teams +- Lock the Flask servers so they are reachable only from the BFF +- Structured so a Google provider drops in without rework + +### Explicitly out of scope (v1) + +| Deferred | Why | +|---|---| +| Google sign-in | Needs a Google Cloud OAuth app; would block the demo. Design keeps the seam open. | +| Password reset / email flows | Needs SMTP. v1: superadmin re-creates the user. | +| Audit log of who viewed what | Valuable, but separate work. | +| Bitbucket integration | Unrelated to auth. There is a `BITBUCKET` constant but **no ETL handler exists** — it is net-new integration work. | +| Multi-org / true tenancy | The app is hardwired to a single `default` org. Out of scope. | + +--- + +## 3. Decisions and rationale + +| # | Decision | Rationale | +|---|---|---| +| D1 | New tables, not new columns on `Users` | Upstream's schema stays untouched, so their migrations can never collide with ours. Joins are trivial. | +| D2 | Do not reuse `role_override` or `UserRole` | `role_override` is dead and undocumented; `UserRole` (`EM`/`ENGINEER`/`MOM`) is a reporting-hierarchy concept, not a permission one. Overloading either would confuse both. | +| D3 | Enforce BFF auth inside `Endpoint.serve()` | Every real API route goes through it (verified below). One insertion point instead of 47. | +| D4 | Lock Flask to BFF-only via shared secret | Lets **all** authorization live in the BFF. No user context threaded through Python, no changes to any service/repo file. | +| D5 | JWT sessions; team access read per-request from DB | Role in the token is fine; team assignments must not be, or changing someone's teams wouldn't take effect until they re-login. | +| D6 | `share.ts` becomes authenticated | See §8. | + +### Verification behind D3 + +All 53 files under `pages/api/` were classified: + +- **47** construct `new Endpoint(...)` — every real route +- **2** are test files (`__tests__/`) +- **4** are helper modules with **no `export default`** — not routes at all: + `internal/[org_id]/utils.ts`, `internal/team/[team_id]/{deployment_freq,deployment_prs,revert_prs}.ts` + +```bash +# reproduce +for f in $(find web-server/pages/api -name '*.ts'); do + grep -q "export default" "$f" || echo "not a route: $f" +done +``` + +So `Endpoint.serve()` gives **100% coverage of BFF routes with no stragglers.** + +--- + +## 4. Data model + +One new migration, `database-docker/db/migrations/_clustox_auth.sql`: + +```sql +CREATE TABLE "ClustoxUserAuth" ( + user_id uuid PRIMARY KEY REFERENCES "Users"(id) ON DELETE CASCADE, + password_hash text NOT NULL, + role varchar NOT NULL CHECK (role IN ('SUPERADMIN','ADMIN')), + created_at timestamptz NOT NULL DEFAULT now(), + updated_at timestamptz NOT NULL DEFAULT now() +); + +CREATE TABLE "ClustoxUserTeamAccess" ( + user_id uuid NOT NULL REFERENCES "Users"(id) ON DELETE CASCADE, + team_id uuid NOT NULL REFERENCES "Team"(id) ON DELETE CASCADE, + created_at timestamptz NOT NULL DEFAULT now(), + PRIMARY KEY (user_id, team_id) +); + +CREATE INDEX idx_clustox_user_team_access_user ON "ClustoxUserTeamAccess"(user_id); +``` + +Identity itself continues to live in upstream's `Users` table (`id`, `name`, `primary_email`, +`is_deleted`). `ClustoxUserAuth` adds only credentials and role. + +`SUPERADMIN` needs no rows in `ClustoxUserTeamAccess` — the role short-circuits every scope check. + +Per [ARCHITECTURE.md §12.5](../../ARCHITECTURE.md), a schema change is normally a four-place +change. Here it is **three**: migration, `schema.sql`, and `web-server/src/constants/db.ts`. No +SQLAlchemy model is needed, because Python never reads these tables. + +--- + +## 5. Architecture + +### 5.1 Three surfaces + +Auth that covers only the frontend is decorative — the Flask API is independently reachable, and 23 +BFF files bypass Flask to reach Postgres directly via knex. All three paths must be closed. + +```mermaid +graph TB + U["Browser"] + subgraph app["Application"] + MW["middleware.ts
surface 1 — page guard"] + EP["Endpoint.serve()
surface 2 — API guard + authz"] + AX["axios 'internal' instance
stamps X-Internal-Token"] + FL["Flask :9696 / :9697
surface 3 — before_request"] + PG[("Postgres")] + end + U --> MW --> EP + EP -->|knex| PG + EP --> AX --> FL --> PG + + style MW fill:#4a6fa5,color:#fff + style EP fill:#5c8001,color:#fff + style FL fill:#9b2226,color:#fff +``` + +| Surface | Mechanism | Failure mode if omitted | +|---|---|---| +| 1 — Pages | `middleware.ts` redirects unauthenticated requests to `/login` | Cosmetic only; data still protected by 2 and 3 | +| 2 — BFF routes | `Endpoint.serve()` rejects when `this.authenticated` and no valid session | **Total bypass** — every metric and mutation exposed | +| 3 — Flask | One `before_request` rejects requests without the shared secret | **Total bypass** — `curl :9696` returns everything | + +### 5.2 Session + +`next-auth` v4 (already a dependency) with a **Credentials** provider and **JWT** strategy. + +- Token carries `userId` and `role`. +- Team assignments are **not** in the token; they are read per-request from + `ClustoxUserTeamAccess`. Cheap (indexed, single-row-set) and avoids stale permissions. +- `NEXTAUTH_SECRET` and `NEXTAUTH_URL` added to `.env` / `env.example`. + +Adding Google later means registering a second provider and mapping the Google email onto an +existing `Users` row. No change to the authorization layer. + +### 5.3 Authorization + +Because Flask is BFF-only after surface 3, **all authorization lives in the BFF**: + +```ts +// CLUSTOX: new file — src/auth/guard.ts +assertRole(session, 'SUPERADMIN') // throws 403 +assertTeamAccess(session, team_id) // SUPERADMIN passes; ADMIN checked against table +visibleTeamIds(session) // for list endpoints +``` + +Applied at: + +- **Team-scoped routes** — `assertTeamAccess` against the `team_id` in the path or payload +- **Team list endpoints** — filtered through `visibleTeamIds`, so an admin's team dropdown shows + only their teams +- **Integration + settings routes** — `assertRole(SUPERADMIN)` +- **User management routes** — `assertRole(SUPERADMIN)` + +An admin who guesses another team's UUID gets a 403 rather than data. That property is what makes +this more than a login screen, and it is the subject of an explicit test. + +### 5.4 Superadmin bootstrap + +On first boot, if no `SUPERADMIN` exists, create one from `SUPERADMIN_EMAIL` and +`SUPERADMIN_PASSWORD`. Idempotent, and never overwrites an existing account. If the env vars are +absent and no superadmin exists, log a loud warning — a running instance with no superadmin cannot +be administered. + +--- + +## 6. File inventory + +### Upstream files modified (red/yellow zone — six files) + +| File | Change | Size | +|---|---|---| +| `web-server/middleware.ts` | Redirect unauthenticated to `/login`; allow `/api/auth/*` | ~15 lines | +| `web-server/src/api-helpers/global.ts` | Enforce the existing `authenticated` flag in `serve()` | ~8 lines | +| `web-server/src/api-helpers/axios.ts` | Stamp `X-Internal-Token` on the `internal` instances | ~4 lines | +| `backend/analytics_server/app.py` | Register `before_request` hook | 2 lines | +| `backend/analytics_server/sync_app.py` | Register `before_request` hook | 2 lines | +| `web-server/src/constants/db.ts` | Declare the two new tables for knex | ~20 lines | + +Every one gets a `CLUSTOX:` sentinel per [FORK_STRATEGY.md §4](../../FORK_STRATEGY.md). + +Of the six, `constants/db.ts` is the lowest-risk: it is a declaration table that grows by append, so +conflicts there are mechanical. The two Python entry points are one line each. The real thought goes +into `global.ts` and `middleware.ts`. + +### New files (green zone) + +``` +database-docker/db/migrations/_clustox_auth.sql +backend/analytics_server/mhq/clustox_auth/internal_token.py +web-server/pages/login.tsx +web-server/pages/users.tsx +web-server/pages/api/auth/[...nextauth].ts +web-server/pages/api/clustox/users/index.ts # list, create +web-server/pages/api/clustox/users/[user_id].ts # update role, teams; delete +web-server/src/auth/guard.ts +web-server/src/auth/password.ts +web-server/src/auth/bootstrap.ts +web-server/src/auth/queries.ts +``` + +`Authenticated/index.tsx` is left alone — `middleware.ts` does the real work, and rewriting the +stub only adds divergence. + +### Fork impact + +Cross-cutting auth for five touched upstream files is an unusually good ratio, entirely because +upstream left the hook points in. This remains **permanent divergence**: upstream will not accept +it, since gated auth is plausibly what their commercial product sells. + +--- + +## 7. Demo script + +1. Open the app → redirected to `/login` *(today it walks straight in)* +2. Log in as superadmin → all teams visible, **Users** appears in nav +3. Create an admin, assign to `ZDA` only +4. Log out, log in as that admin → only `ZDA`; no Users page; integrations hidden +5. As that admin, request `CGPT Frontend`'s team ID directly → **403** +6. `curl http://localhost:9696/teams//lead_time` → **403** *(today: returns data)* + +Steps 5 and 6 are the ones worth dwelling on. Step 6 in particular is what separates real access +control from a login screen, and it is the part most implementations get wrong. + +--- + +## 8. Risks and mitigations + +| Risk | Mitigation | +|---|---| +| **`share.ts` is currently `unauthenticated`** — a URL-shortener storing dashboard link state in `URLShortenerData`. Anonymous `POST` allows unbounded row insertion; anonymous `GET` returns stored link metadata. | **Decision:** make it authenticated. For an internal tool, everyone who should open a share link has an account, so nothing is lost and an anonymous write path is closed. | +| Password handling becomes our responsibility | bcrypt (cost 12), never log or return hashes, no reset flow in v1 — a stated limitation, not a hidden gap | +| Someone adds a route bypassing `Endpoint` | Test asserting every file under `pages/api` with a default export constructs an `Endpoint` | +| Shared secret leaks or is unset | Fail closed: if `INTERNAL_API_TOKEN` is unset, Flask rejects **all** requests rather than allowing all | +| Locking Flask breaks the cron sync | Cron calls `:9697/sync` from inside the container. It must send the token too — **verify explicitly**, since the failure is silent (see [ARCHITECTURE.md §7.4](../../ARCHITECTURE.md)) | +| Bootstrap creates a weak default account | No default password. Absent env vars → no superadmin + loud warning, never a known-value fallback | +| Permanent fork divergence | Accepted deliberately; sentinels make the footprint auditable | + +--- + +## 9. Testing + +| Layer | Coverage | +|---|---| +| Jest | `assertRole`, `assertTeamAccess`, `visibleTeamIds`; password hash/verify; `Endpoint.serve()` rejects without session and honours `unauthenticated` | +| pytest | Flask `before_request`: valid token passes, missing/wrong 403s, unset env fails closed | +| Playwright | The §7 demo flow end to end | +| Regression | Unauthenticated `curl` to a team route returns 403 — the single test that would catch the whole feature being silently undone | +| Manual | Cron sync still succeeds after Flask is locked down | + +--- + +## 10. Open questions + +None blocking. Two to settle during implementation, neither affecting the architecture: + +1. Session lifetime — proposed 12 hours, no sliding renewal. +2. Whether the admin's team dropdown should hide inaccessible teams entirely or show them disabled. + Proposed: hide, matching `visibleTeamIds`. From 08ea137f8acf01370f982c93ad7cde969cfacb9d Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Wed, 5 Aug 2026 23:44:38 +0500 Subject: [PATCH 002/194] feat(auth): add ClustoxUserAuth and ClustoxUserTeamAccess tables CLUSTOX: new tables rather than columns on upstream's Users, so upstream migrations can never collide with ours. Role values are constrained at the database level via a CHECK, verified to reject invalid values independently of the foreign key. schema.sql regenerated by dbmate; the diff is purely additive. --- .../20260805150000_clustox_auth.sql | 23 ++++++ database-docker/db/schema.sql | 75 ++++++++++++++++++- web-server/src/constants/db.ts | 25 ++++++- 3 files changed, 121 insertions(+), 2 deletions(-) create mode 100644 database-docker/db/migrations/20260805150000_clustox_auth.sql diff --git a/database-docker/db/migrations/20260805150000_clustox_auth.sql b/database-docker/db/migrations/20260805150000_clustox_auth.sql new file mode 100644 index 00000000..c7e2b3bd --- /dev/null +++ b/database-docker/db/migrations/20260805150000_clustox_auth.sql @@ -0,0 +1,23 @@ +-- migrate:up +-- CLUSTOX: authentication and role-based access tables. Deliberately separate +-- from upstream "Users" so upstream migrations can never collide with ours. +CREATE TABLE "ClustoxUserAuth" ( + user_id uuid PRIMARY KEY REFERENCES "Users"(id) ON DELETE CASCADE, + password_hash text NOT NULL, + role varchar NOT NULL CHECK (role IN ('SUPERADMIN','ADMIN')), + created_at timestamptz NOT NULL DEFAULT now(), + updated_at timestamptz NOT NULL DEFAULT now() +); + +CREATE TABLE "ClustoxUserTeamAccess" ( + user_id uuid NOT NULL REFERENCES "Users"(id) ON DELETE CASCADE, + team_id uuid NOT NULL REFERENCES "Team"(id) ON DELETE CASCADE, + created_at timestamptz NOT NULL DEFAULT now(), + PRIMARY KEY (user_id, team_id) +); + +CREATE INDEX idx_clustox_user_team_access_user ON "ClustoxUserTeamAccess"(user_id); + +-- migrate:down +DROP TABLE "ClustoxUserTeamAccess"; +DROP TABLE "ClustoxUserAuth"; diff --git a/database-docker/db/schema.sql b/database-docker/db/schema.sql index a700a0cd..1e4eb7a8 100644 --- a/database-docker/db/schema.sql +++ b/database-docker/db/schema.sql @@ -71,6 +71,31 @@ CREATE TABLE public."BookmarkPullRequestRevertPRMapping" ( ); +-- +-- Name: ClustoxUserAuth; Type: TABLE; Schema: public; Owner: - +-- + +CREATE TABLE public."ClustoxUserAuth" ( + user_id uuid NOT NULL, + password_hash text NOT NULL, + role character varying NOT NULL, + created_at timestamp with time zone DEFAULT now() NOT NULL, + updated_at timestamp with time zone DEFAULT now() NOT NULL, + CONSTRAINT "ClustoxUserAuth_role_check" CHECK (((role)::text = ANY ((ARRAY['SUPERADMIN'::character varying, 'ADMIN'::character varying])::text[]))) +); + + +-- +-- Name: ClustoxUserTeamAccess; Type: TABLE; Schema: public; Owner: - +-- + +CREATE TABLE public."ClustoxUserTeamAccess" ( + user_id uuid NOT NULL, + team_id uuid NOT NULL, + created_at timestamp with time zone DEFAULT now() NOT NULL +); + + -- -- Name: Incident; Type: TABLE; Schema: public; Owner: - -- @@ -482,6 +507,22 @@ CREATE TABLE public.schema_migrations ( ); +-- +-- Name: ClustoxUserAuth ClustoxUserAuth_pkey; Type: CONSTRAINT; Schema: public; Owner: - +-- + +ALTER TABLE ONLY public."ClustoxUserAuth" + ADD CONSTRAINT "ClustoxUserAuth_pkey" PRIMARY KEY (user_id); + + +-- +-- Name: ClustoxUserTeamAccess ClustoxUserTeamAccess_pkey; Type: CONSTRAINT; Schema: public; Owner: - +-- + +ALTER TABLE ONLY public."ClustoxUserTeamAccess" + ADD CONSTRAINT "ClustoxUserTeamAccess_pkey" PRIMARY KEY (user_id, team_id); + + -- -- Name: IncidentOrgIncidentServiceMap IncidentOrgIncidentServiceMap_pkey; Type: CONSTRAINT; Schema: public; Owner: - -- @@ -804,6 +845,13 @@ CREATE INDEX "TeamIncidentService_team_id_index" ON public."TeamIncidentService" CREATE INDEX "Team_org_idx" ON public."Team" USING btree (org_id); +-- +-- Name: idx_clustox_user_team_access_user; Type: INDEX; Schema: public; Owner: - +-- + +CREATE INDEX idx_clustox_user_team_access_user ON public."ClustoxUserTeamAccess" USING btree (user_id); + + -- -- Name: incident_resolved_date_index; Type: INDEX; Schema: public; Owner: - -- @@ -1107,6 +1155,30 @@ ALTER TABLE ONLY public."Bookmark" ADD CONSTRAINT "Bookmark_repo_id_fkey" FOREIGN KEY (repo_id) REFERENCES public."OrgRepo"(id); +-- +-- Name: ClustoxUserAuth ClustoxUserAuth_user_id_fkey; Type: FK CONSTRAINT; Schema: public; Owner: - +-- + +ALTER TABLE ONLY public."ClustoxUserAuth" + ADD CONSTRAINT "ClustoxUserAuth_user_id_fkey" FOREIGN KEY (user_id) REFERENCES public."Users"(id) ON DELETE CASCADE; + + +-- +-- Name: ClustoxUserTeamAccess ClustoxUserTeamAccess_team_id_fkey; Type: FK CONSTRAINT; Schema: public; Owner: - +-- + +ALTER TABLE ONLY public."ClustoxUserTeamAccess" + ADD CONSTRAINT "ClustoxUserTeamAccess_team_id_fkey" FOREIGN KEY (team_id) REFERENCES public."Team"(id) ON DELETE CASCADE; + + +-- +-- Name: ClustoxUserTeamAccess ClustoxUserTeamAccess_user_id_fkey; Type: FK CONSTRAINT; Schema: public; Owner: - +-- + +ALTER TABLE ONLY public."ClustoxUserTeamAccess" + ADD CONSTRAINT "ClustoxUserTeamAccess_user_id_fkey" FOREIGN KEY (user_id) REFERENCES public."Users"(id) ON DELETE CASCADE; + + -- -- Name: IncidentOrgIncidentServiceMap IncidentOrgIncidentServiceMap_incident_id_fkey; Type: FK CONSTRAINT; Schema: public; Owner: - -- @@ -1360,4 +1432,5 @@ INSERT INTO public.schema_migrations (version) VALUES ('20240404142732'), ('20240430142502'), ('20240503060203'), - ('20240503073715'); + ('20240503073715'), + ('20260805150000'); diff --git a/web-server/src/constants/db.ts b/web-server/src/constants/db.ts index 31901a49..7b7ebec2 100644 --- a/web-server/src/constants/db.ts +++ b/web-server/src/constants/db.ts @@ -38,7 +38,11 @@ enum TableT { TicketState, UserActivity, UIPreferences, - URLShortenerData + URLShortenerData, + // CLUSTOX: auth tables — appended so upstream additions above never conflict. + // See docs/superpowers/specs/2026-08-05-auth-rbac-design.md + ClustoxUserAuth, + ClustoxUserTeamAccess } export const Table = objectEnum(TableT); @@ -551,6 +555,25 @@ export const Columns = { org_id } return Columns; + }), + // CLUSTOX: auth tables + [Table.ClustoxUserAuth]: objectEnumFromFn(() => { + enum Columns { + user_id, + password_hash, + role, + created_at, + updated_at + } + return Columns; + }), + [Table.ClustoxUserTeamAccess]: objectEnumFromFn(() => { + enum Columns { + user_id, + team_id, + created_at + } + return Columns; }) }; From 4a6a2558603506509adcfd7c78f18254e8bf5f0c Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Wed, 5 Aug 2026 23:49:41 +0500 Subject: [PATCH 003/194] feat(auth): add bcrypt password hashing bcryptjs rather than bcrypt: pure JS, no native compilation, so the Docker build is unaffected. Cost factor 12. verifyPassword swallows malformed-hash errors and returns false, so a corrupt stored hash can never be mistaken for a successful comparison. --- web-server/package.json | 2 ++ .../src/auth/__tests__/password.test.ts | 31 +++++++++++++++++++ web-server/src/auth/password.ts | 19 ++++++++++++ web-server/yarn.lock | 12 +++++++ 4 files changed, 64 insertions(+) create mode 100644 web-server/src/auth/__tests__/password.test.ts create mode 100644 web-server/src/auth/password.ts diff --git a/web-server/package.json b/web-server/package.json index 580e2e28..350fb136 100644 --- a/web-server/package.json +++ b/web-server/package.json @@ -32,6 +32,7 @@ "axios": "1.7.7", "axios-hooks": "^5.0.2", "axios-retry": "^4.5.0", + "bcryptjs": "^3.0.3", "chart.js": "^3.9.1", "chartjs-plugin-annotation": "^2.0.1", "chartjs-plugin-crosshair": "^1.2.0", @@ -102,6 +103,7 @@ "@next/bundle-analyzer": "^12.3.0", "@octokit/types": "^6.34.0", "@playwright/test": "^1.28.0", + "@types/bcryptjs": "^3.0.0", "@types/js-cookie": "^3.0.2", "@types/node": "^16.17.0", "@types/nprogress": "0.2.0", diff --git a/web-server/src/auth/__tests__/password.test.ts b/web-server/src/auth/__tests__/password.test.ts new file mode 100644 index 00000000..65dd09d4 --- /dev/null +++ b/web-server/src/auth/__tests__/password.test.ts @@ -0,0 +1,31 @@ +import { hashPassword, verifyPassword } from '@/auth/password'; + +describe('password', () => { + it('produces a hash that is not the plaintext', async () => { + const hash = await hashPassword('correct-horse-battery'); + expect(hash).not.toBe('correct-horse-battery'); + expect(hash.length).toBeGreaterThan(50); + }); + + it('verifies a correct password', async () => { + const hash = await hashPassword('correct-horse-battery'); + await expect(verifyPassword('correct-horse-battery', hash)).resolves.toBe( + true + ); + }); + + it('rejects an incorrect password', async () => { + const hash = await hashPassword('correct-horse-battery'); + await expect(verifyPassword('wrong-password', hash)).resolves.toBe(false); + }); + + it('produces different hashes for the same input (salted)', async () => { + const a = await hashPassword('same-input'); + const b = await hashPassword('same-input'); + expect(a).not.toBe(b); + }); + + it('returns false rather than throwing on a malformed hash', async () => { + await expect(verifyPassword('anything', 'not-a-hash')).resolves.toBe(false); + }); +}); diff --git a/web-server/src/auth/password.ts b/web-server/src/auth/password.ts new file mode 100644 index 00000000..5f2f1e84 --- /dev/null +++ b/web-server/src/auth/password.ts @@ -0,0 +1,19 @@ +import { compare, hash } from 'bcryptjs'; + +const COST = 12; + +export const hashPassword = (plain: string): Promise => + hash(plain, COST); + +export const verifyPassword = async ( + plain: string, + passwordHash: string +): Promise => { + try { + return await compare(plain, passwordHash); + } catch { + // A malformed stored hash must read as "wrong password", never as an + // exception that a caller might treat as success. + return false; + } +}; diff --git a/web-server/yarn.lock b/web-server/yarn.lock index 3f4bc1ae..bea15545 100644 --- a/web-server/yarn.lock +++ b/web-server/yarn.lock @@ -2516,6 +2516,13 @@ dependencies: "@babel/types" "^7.20.7" +"@types/bcryptjs@^3.0.0": + version "3.0.0" + resolved "https://registry.yarnpkg.com/@types/bcryptjs/-/bcryptjs-3.0.0.tgz#d7be11653aa82cf17ffee4f3925f1f80cfc1add2" + integrity sha512-WRZOuCuaz8UcZZE4R5HXTco2goQSI2XxjGY3hbM/xDvwmqFWd4ivooImsMx65OKM6CtNKbnZ5YL+YwAwK7c1dg== + dependencies: + bcryptjs "*" + "@types/debug@^4.0.0": version "4.1.12" resolved "https://registry.yarnpkg.com/@types/debug/-/debug-4.1.12.tgz#a155f21690871953410df4b6b6f53187f0500917" @@ -3413,6 +3420,11 @@ balanced-match@^1.0.0: resolved "https://registry.yarnpkg.com/balanced-match/-/balanced-match-1.0.2.tgz#e83e3a7e3f300b34cb9d87f615fa0cbf357690ee" integrity sha512-3oSeUO0TMV67hN1AmbXsK4yaqU7tjiHlbxRDZOpH0KW9+CeX4bRAaX0Anxt0tx2MrpRpWwQaPwIlISEJhYU5Pw== +bcryptjs@*, bcryptjs@^3.0.3: + version "3.0.3" + resolved "https://registry.yarnpkg.com/bcryptjs/-/bcryptjs-3.0.3.tgz#4b93d6a398c48bfc9f32ee65d301174a8a8ea56f" + integrity sha512-GlF5wPWnSa/X5LKM1o0wz0suXIINz1iHRLvTS+sLyi7XPbe5ycmYI3DlZqVGZZtDgl4DmasFg7gOB3JYbphV5g== + before-after-hook@^2.2.0: version "2.2.2" resolved "https://registry.yarnpkg.com/before-after-hook/-/before-after-hook-2.2.2.tgz#a6e8ca41028d90ee2c24222f201c90956091613e" From a3ff16c28037791116d5819b7de27b8671edea5c Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Wed, 5 Aug 2026 23:51:06 +0500 Subject: [PATCH 004/194] feat(auth): add auth types and data access queries --- web-server/src/auth/__tests__/queries.test.ts | 60 ++++++++ web-server/src/auth/queries.ts | 138 ++++++++++++++++++ web-server/src/auth/types.ts | 24 +++ 3 files changed, 222 insertions(+) create mode 100644 web-server/src/auth/__tests__/queries.test.ts create mode 100644 web-server/src/auth/queries.ts create mode 100644 web-server/src/auth/types.ts diff --git a/web-server/src/auth/__tests__/queries.test.ts b/web-server/src/auth/__tests__/queries.test.ts new file mode 100644 index 00000000..216d6e3c --- /dev/null +++ b/web-server/src/auth/__tests__/queries.test.ts @@ -0,0 +1,60 @@ +const mockDb = jest.fn(); +jest.mock('@/utils/db', () => ({ + db: (...args: any[]) => mockDb(...args) +})); + +import { getAuthUserByEmail, getTeamIdsForUser } from '@/auth/queries'; + +const chain = (result: any) => { + const c: any = {}; + c.select = jest.fn(() => c); + c.join = jest.fn(() => c); + c.where = jest.fn(() => c); + c.andWhere = jest.fn(() => c); + c.first = jest.fn(() => Promise.resolve(result)); + c.then = (fn: any) => Promise.resolve(result).then(fn); + return c; +}; + +describe('getAuthUserByEmail', () => { + beforeEach(() => mockDb.mockReset()); + + it('maps a row to AuthUserRow', async () => { + mockDb.mockReturnValue( + chain({ + user_id: 'u1', + primary_email: 'a@clustox.com', + name: 'Ayesha', + role: 'ADMIN', + password_hash: 'hashed' + }) + ); + + await expect(getAuthUserByEmail('a@clustox.com')).resolves.toEqual({ + userId: 'u1', + email: 'a@clustox.com', + name: 'Ayesha', + role: 'ADMIN', + passwordHash: 'hashed' + }); + }); + + it('returns null when no row matches', async () => { + mockDb.mockReturnValue(chain(undefined)); + await expect(getAuthUserByEmail('nobody@clustox.com')).resolves.toBeNull(); + }); +}); + +describe('getTeamIdsForUser', () => { + beforeEach(() => mockDb.mockReset()); + + it('returns a flat array of team ids', async () => { + mockDb.mockReturnValue(chain([{ team_id: 't1' }, { team_id: 't2' }])); + await expect(getTeamIdsForUser('u1')).resolves.toEqual(['t1', 't2']); + }); + + it('returns an empty array when the user has no teams', async () => { + mockDb.mockReturnValue(chain([])); + await expect(getTeamIdsForUser('u1')).resolves.toEqual([]); + }); +}); diff --git a/web-server/src/auth/queries.ts b/web-server/src/auth/queries.ts new file mode 100644 index 00000000..e4459ea9 --- /dev/null +++ b/web-server/src/auth/queries.ts @@ -0,0 +1,138 @@ +import { Columns, Table } from '@/constants/db'; +import { db } from '@/utils/db'; + +import { hashPassword } from './password'; +import { AuthUserListItem, AuthUserRow, ClustoxRole } from './types'; + +export const getAuthUserByEmail = async ( + email: string +): Promise => { + const row = await db(Table.ClustoxUserAuth) + .join(Table.Users, `${Table.Users}.id`, `${Table.ClustoxUserAuth}.user_id`) + .where(`${Table.Users}.primary_email`, email) + .andWhere(`${Table.Users}.is_deleted`, false) + .select( + `${Table.ClustoxUserAuth}.user_id`, + `${Table.ClustoxUserAuth}.role`, + `${Table.ClustoxUserAuth}.password_hash`, + `${Table.Users}.primary_email`, + `${Table.Users}.name` + ) + .first(); + + if (!row) return null; + + return { + userId: row.user_id, + email: row.primary_email, + name: row.name, + role: row.role as ClustoxRole, + passwordHash: row.password_hash + }; +}; + +export const getTeamIdsForUser = async (userId: string): Promise => { + const rows = await db(Table.ClustoxUserTeamAccess) + .select(Columns[Table.ClustoxUserTeamAccess].team_id) + .where(Columns[Table.ClustoxUserTeamAccess].user_id, userId); + return rows.map((r: { team_id: string }) => r.team_id); +}; + +export const getAllTeamIds = async (): Promise => { + const rows = await db(Table.Team).select('id').where('is_deleted', false); + return rows.map((r: { id: string }) => r.id); +}; + +export const countSuperadmins = async (): Promise => { + const rows = await db(Table.ClustoxUserAuth) + .where(Columns[Table.ClustoxUserAuth].role, 'SUPERADMIN') + .select(Columns[Table.ClustoxUserAuth].user_id); + return rows.length; +}; + +export const listUsers = async (): Promise => { + const rows = await db(Table.ClustoxUserAuth) + .join(Table.Users, `${Table.Users}.id`, `${Table.ClustoxUserAuth}.user_id`) + .where(`${Table.Users}.is_deleted`, false) + .select( + `${Table.ClustoxUserAuth}.user_id`, + `${Table.ClustoxUserAuth}.role`, + `${Table.Users}.primary_email`, + `${Table.Users}.name` + ); + + const access = await db(Table.ClustoxUserTeamAccess).select( + 'user_id', + 'team_id' + ); + + return rows.map((r: any) => ({ + userId: r.user_id, + email: r.primary_email, + name: r.name, + role: r.role as ClustoxRole, + teamIds: access + .filter((a: any) => a.user_id === r.user_id) + .map((a: any) => a.team_id) + })); +}; + +export const createUser = async (input: { + name: string; + email: string; + password: string; + role: ClustoxRole; + teamIds: string[]; + orgId: string; +}): Promise => { + const passwordHash = await hashPassword(input.password); + + const [user] = await db(Table.Users) + .insert({ + org_id: input.orgId, + name: input.name, + primary_email: input.email, + is_deleted: false + }) + .returning('id'); + + const userId = typeof user === 'string' ? user : user.id; + + await db(Table.ClustoxUserAuth).insert({ + user_id: userId, + password_hash: passwordHash, + role: input.role + }); + + if (input.teamIds.length) { + await db(Table.ClustoxUserTeamAccess).insert( + input.teamIds.map((team_id) => ({ user_id: userId, team_id })) + ); + } + + return userId; +}; + +export const updateUserRole = async ( + userId: string, + role: ClustoxRole +): Promise => { + await db(Table.ClustoxUserAuth) + .where(Columns[Table.ClustoxUserAuth].user_id, userId) + .update({ role, updated_at: new Date() }); +}; + +export const setUserTeams = async ( + userId: string, + teamIds: string[] +): Promise => { + await db(Table.ClustoxUserTeamAccess) + .where(Columns[Table.ClustoxUserTeamAccess].user_id, userId) + .delete(); + + if (teamIds.length) { + await db(Table.ClustoxUserTeamAccess).insert( + teamIds.map((team_id) => ({ user_id: userId, team_id })) + ); + } +}; diff --git a/web-server/src/auth/types.ts b/web-server/src/auth/types.ts new file mode 100644 index 00000000..1d56686e --- /dev/null +++ b/web-server/src/auth/types.ts @@ -0,0 +1,24 @@ +export type ClustoxRole = 'SUPERADMIN' | 'ADMIN'; + +export interface AuthSession { + userId: string; + email: string; + name: string; + role: ClustoxRole; +} + +export interface AuthUserRow { + userId: string; + email: string; + name: string; + role: ClustoxRole; + passwordHash: string; +} + +export interface AuthUserListItem { + userId: string; + email: string; + name: string; + role: ClustoxRole; + teamIds: string[]; +} From df1b95e175f786214540997bc555283b824d26cd Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Wed, 5 Aug 2026 23:52:16 +0500 Subject: [PATCH 005/194] feat(auth): add role and team-access guards Authorization decisions live here so routes stay thin. Superadmin short-circuits every team check without a query; admins are checked against ClustoxUserTeamAccess per request rather than from the token, so a permission change takes effect immediately. --- web-server/src/auth/__tests__/guard.test.ts | 120 ++++++++++++++++++++ web-server/src/auth/guard.ts | 43 +++++++ 2 files changed, 163 insertions(+) create mode 100644 web-server/src/auth/__tests__/guard.test.ts create mode 100644 web-server/src/auth/guard.ts diff --git a/web-server/src/auth/__tests__/guard.test.ts b/web-server/src/auth/__tests__/guard.test.ts new file mode 100644 index 00000000..54dae35a --- /dev/null +++ b/web-server/src/auth/__tests__/guard.test.ts @@ -0,0 +1,120 @@ +jest.mock('@/auth/queries', () => ({ + getTeamIdsForUser: jest.fn(), + getAllTeamIds: jest.fn() +})); + +import { + assertAuthenticated, + assertRole, + assertTeamAccess, + canAccessAllTeams, + getAccessibleTeamIds +} from '@/auth/guard'; +import { getAllTeamIds, getTeamIdsForUser } from '@/auth/queries'; +import { AuthSession } from '@/auth/types'; + +const superadmin: AuthSession = { + userId: 'su1', + email: 'boss@clustox.com', + name: 'Boss', + role: 'SUPERADMIN' +}; +const admin: AuthSession = { + userId: 'ad1', + email: 'lead@clustox.com', + name: 'Lead', + role: 'ADMIN' +}; + +describe('assertAuthenticated', () => { + it('returns the session when present', () => { + expect(assertAuthenticated(admin)).toBe(admin); + }); + + it('throws 401 when the session is null', () => { + expect(() => assertAuthenticated(null)).toThrow( + expect.objectContaining({ status: 401 }) + ); + }); +}); + +describe('assertRole', () => { + it('passes when the role matches', () => { + expect(assertRole(superadmin, 'SUPERADMIN')).toBe(superadmin); + }); + + it('throws 403 when an admin attempts a superadmin action', () => { + expect(() => assertRole(admin, 'SUPERADMIN')).toThrow( + expect.objectContaining({ status: 403 }) + ); + }); + + it('throws 401 when unauthenticated, not 403', () => { + expect(() => assertRole(null, 'SUPERADMIN')).toThrow( + expect.objectContaining({ status: 401 }) + ); + }); +}); + +describe('canAccessAllTeams', () => { + it('is true for superadmin', () => { + expect(canAccessAllTeams(superadmin)).toBe(true); + }); + it('is false for admin', () => { + expect(canAccessAllTeams(admin)).toBe(false); + }); +}); + +describe('assertTeamAccess', () => { + beforeEach(() => jest.clearAllMocks()); + + it('allows superadmin any team without querying', async () => { + await expect( + assertTeamAccess(superadmin, 'any-team') + ).resolves.toBeUndefined(); + expect(getTeamIdsForUser).not.toHaveBeenCalled(); + }); + + it('allows an admin their assigned team', async () => { + (getTeamIdsForUser as jest.Mock).mockResolvedValue(['t1', 't2']); + await expect(assertTeamAccess(admin, 't2')).resolves.toBeUndefined(); + }); + + it('throws 403 when an admin requests a team they do not own', async () => { + (getTeamIdsForUser as jest.Mock).mockResolvedValue(['t1']); + await expect(assertTeamAccess(admin, 't-other')).rejects.toMatchObject({ + status: 403 + }); + }); + + it('throws 403 when an admin has no teams at all', async () => { + (getTeamIdsForUser as jest.Mock).mockResolvedValue([]); + await expect(assertTeamAccess(admin, 't1')).rejects.toMatchObject({ + status: 403 + }); + }); + + it('throws 401 when unauthenticated', async () => { + await expect(assertTeamAccess(null, 't1')).rejects.toMatchObject({ + status: 401 + }); + }); +}); + +describe('getAccessibleTeamIds', () => { + beforeEach(() => jest.clearAllMocks()); + + it('returns every team for superadmin', async () => { + (getAllTeamIds as jest.Mock).mockResolvedValue(['t1', 't2', 't3']); + await expect(getAccessibleTeamIds(superadmin)).resolves.toEqual([ + 't1', + 't2', + 't3' + ]); + }); + + it('returns only assigned teams for admin', async () => { + (getTeamIdsForUser as jest.Mock).mockResolvedValue(['t2']); + await expect(getAccessibleTeamIds(admin)).resolves.toEqual(['t2']); + }); +}); diff --git a/web-server/src/auth/guard.ts b/web-server/src/auth/guard.ts new file mode 100644 index 00000000..e5127892 --- /dev/null +++ b/web-server/src/auth/guard.ts @@ -0,0 +1,43 @@ +import { Errors, ResponseError } from '@/constants/error'; + +import { getAllTeamIds, getTeamIdsForUser } from './queries'; +import { AuthSession, ClustoxRole } from './types'; + +export const assertAuthenticated = ( + session: AuthSession | null +): AuthSession => { + if (!session) throw new ResponseError(Errors.SESSION_USER_NOT_FOUND, 401); + return session; +}; + +export const assertRole = ( + session: AuthSession | null, + role: ClustoxRole +): AuthSession => { + const authed = assertAuthenticated(session); + if (authed.role !== role) throw new ResponseError(Errors.ACCESS_DENIED, 403); + return authed; +}; + +export const canAccessAllTeams = (session: AuthSession): boolean => + session.role === 'SUPERADMIN'; + +export const assertTeamAccess = async ( + session: AuthSession | null, + teamId: string +): Promise => { + const authed = assertAuthenticated(session); + if (canAccessAllTeams(authed)) return; + + const teamIds = await getTeamIdsForUser(authed.userId); + if (!teamIds.includes(teamId)) { + throw new ResponseError(Errors.ACCESS_DENIED, 403); + } +}; + +export const getAccessibleTeamIds = async ( + session: AuthSession +): Promise => + canAccessAllTeams(session) + ? getAllTeamIds() + : getTeamIdsForUser(session.userId); From d70c787897d7c46b1d4dd399341e0fc262b94636 Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Wed, 5 Aug 2026 23:53:51 +0500 Subject: [PATCH 006/194] feat(auth): configure next-auth credentials provider JWT sessions carry userId and role only. Team assignments are read from the database per request instead, so changing someone's teams takes effect immediately rather than at next login. Session lifetime 12h, no sliding renewal. --- env.example | 7 +++ web-server/pages/api/auth/[...nextauth].ts | 55 ++++++++++++++++++++++ web-server/src/auth/session.ts | 18 +++++++ web-server/src/types/next-auth.d.ts | 25 ++++++++++ 4 files changed, 105 insertions(+) create mode 100644 web-server/pages/api/auth/[...nextauth].ts create mode 100644 web-server/src/auth/session.ts create mode 100644 web-server/src/types/next-auth.d.ts diff --git a/env.example b/env.example index 99fabc51..72018473 100644 --- a/env.example +++ b/env.example @@ -22,3 +22,10 @@ NEXT_PUBLIC_APP_ENVIRONMENT="development" DEFAULT_SYNC_DAYS=31 BUILD_DATE=2024-06-05T10:21:34Z MERGE_COMMIT_SHA=5f9ff895ad1d7805edcb22bfe2fcc6129e33bd8c + +# CLUSTOX: auth +NEXTAUTH_URL=http://localhost:3333 +NEXTAUTH_SECRET=generate-with-openssl-rand-base64-32 +INTERNAL_API_TOKEN=generate-with-openssl-rand-hex-32 +SUPERADMIN_EMAIL=admin@clustox.com +SUPERADMIN_PASSWORD=set-a-strong-password diff --git a/web-server/pages/api/auth/[...nextauth].ts b/web-server/pages/api/auth/[...nextauth].ts new file mode 100644 index 00000000..783f0931 --- /dev/null +++ b/web-server/pages/api/auth/[...nextauth].ts @@ -0,0 +1,55 @@ +import NextAuth, { NextAuthOptions } from 'next-auth'; +import CredentialsProvider from 'next-auth/providers/credentials'; + +import { verifyPassword } from '@/auth/password'; +import { getAuthUserByEmail } from '@/auth/queries'; + +export const authOptions: NextAuthOptions = { + session: { strategy: 'jwt', maxAge: 12 * 60 * 60 }, + pages: { signIn: '/login' }, + providers: [ + CredentialsProvider({ + name: 'Credentials', + credentials: { + email: { label: 'Email', type: 'text' }, + password: { label: 'Password', type: 'password' } + }, + async authorize(credentials) { + if (!credentials?.email || !credentials?.password) return null; + + const user = await getAuthUserByEmail(credentials.email); + if (!user) return null; + + const ok = await verifyPassword(credentials.password, user.passwordHash); + if (!ok) return null; + + return { + id: user.userId, + email: user.email, + name: user.name, + role: user.role + }; + } + }) + ], + callbacks: { + async jwt({ token, user }) { + if (user) { + token.userId = user.id; + token.role = user.role; + } + return token; + }, + async session({ session, token }) { + session.user = { + id: token.userId, + email: token.email as string, + name: token.name as string, + role: token.role + }; + return session; + } + } +}; + +export default NextAuth(authOptions); diff --git a/web-server/src/auth/session.ts b/web-server/src/auth/session.ts new file mode 100644 index 00000000..324fe45b --- /dev/null +++ b/web-server/src/auth/session.ts @@ -0,0 +1,18 @@ +import { getToken } from 'next-auth/jwt'; +import { NextApiRequest } from 'next/types'; + +import { AuthSession, ClustoxRole } from './types'; + +export const getAuthSession = async ( + req: NextApiRequest +): Promise => { + const token = await getToken({ req, secret: process.env.NEXTAUTH_SECRET }); + if (!token?.userId) return null; + + return { + userId: token.userId as string, + email: (token.email as string) ?? '', + name: (token.name as string) ?? '', + role: token.role as ClustoxRole + }; +}; diff --git a/web-server/src/types/next-auth.d.ts b/web-server/src/types/next-auth.d.ts new file mode 100644 index 00000000..484064bd --- /dev/null +++ b/web-server/src/types/next-auth.d.ts @@ -0,0 +1,25 @@ +import { ClustoxRole } from '@/auth/types'; + +declare module 'next-auth' { + interface Session { + user: { + id: string; + email: string; + name: string; + role: ClustoxRole; + }; + } + interface User { + id: string; + email: string; + name: string; + role: ClustoxRole; + } +} + +declare module 'next-auth/jwt' { + interface JWT { + userId: string; + role: ClustoxRole; + } +} From 5dc6626df171097024ce70492a9a6fb75e1bb921 Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Wed, 5 Aug 2026 23:58:52 +0500 Subject: [PATCH 007/194] feat(auth): enforce authentication in Endpoint.serve() Upstream declared `authenticated: boolean = true` on Endpoint and accepted an { unauthenticated: true } opt-out, but serve() never read the flag. Five routes already opted out correctly, so the mechanism was clearly intended to be enforced. Reading it here covers all 47 BFF routes from one insertion point -- every real API route goes through Endpoint (the four non-Endpoint files under pages/api are helper modules with no default export). Test baseline: git_org_repos.test.ts and array.test.ts fail identically on upstream's own global.ts and are unrelated to auth. Verified with --runInBand; parallel workers give flaky counts on this suite. --- .../__tests__/endpoint-auth.test.ts | 80 +++++++++++++++++++ web-server/src/api-helpers/global.ts | 12 +++ 2 files changed, 92 insertions(+) create mode 100644 web-server/src/api-helpers/__tests__/endpoint-auth.test.ts diff --git a/web-server/src/api-helpers/__tests__/endpoint-auth.test.ts b/web-server/src/api-helpers/__tests__/endpoint-auth.test.ts new file mode 100644 index 00000000..c92844ba --- /dev/null +++ b/web-server/src/api-helpers/__tests__/endpoint-auth.test.ts @@ -0,0 +1,80 @@ +jest.mock('@/auth/session', () => ({ getAuthSession: jest.fn() })); + +import { Endpoint, nullSchema } from '@/api-helpers/global'; +import { getAuthSession } from '@/auth/session'; + +const mockRes = () => { + const res: any = {}; + res.status = jest.fn(() => res); + res.send = jest.fn(() => res); + return res; +}; + +const mockReq = () => + ({ method: 'GET', query: {}, body: {}, headers: {} }) as any; + +describe('Endpoint auth enforcement', () => { + beforeEach(() => jest.clearAllMocks()); + + it('rejects with 401 when no session and the endpoint is authenticated', async () => { + (getAuthSession as jest.Mock).mockResolvedValue(null); + + const endpoint = new Endpoint(nullSchema); + endpoint.handle.GET(nullSchema, async (_req, res) => res.send({ ok: true })); + + const res = mockRes(); + await endpoint.serve()(mockReq(), res); + + expect(res.status).toHaveBeenCalledWith(401); + }); + + it('allows the request through when a session exists', async () => { + (getAuthSession as jest.Mock).mockResolvedValue({ + userId: 'u1', + email: 'a@clustox.com', + name: 'A', + role: 'ADMIN' + }); + + const endpoint = new Endpoint(nullSchema); + endpoint.handle.GET(nullSchema, async (_req, res) => res.send({ ok: true })); + + const res = mockRes(); + await endpoint.serve()(mockReq(), res); + + expect(res.send).toHaveBeenCalledWith({ ok: true }); + }); + + it('skips the check for endpoints marked unauthenticated', async () => { + (getAuthSession as jest.Mock).mockResolvedValue(null); + + const endpoint = new Endpoint(nullSchema, { unauthenticated: true }); + endpoint.handle.GET(nullSchema, async (_req, res) => res.send({ ok: true })); + + const res = mockRes(); + await endpoint.serve()(mockReq(), res); + + expect(res.send).toHaveBeenCalledWith({ ok: true }); + expect(getAuthSession).not.toHaveBeenCalled(); + }); + + it('exposes the session on the request object', async () => { + const session = { + userId: 'u1', + email: 'a@clustox.com', + name: 'A', + role: 'ADMIN' as const + }; + (getAuthSession as jest.Mock).mockResolvedValue(session); + + let seen: any = null; + const endpoint = new Endpoint(nullSchema); + endpoint.handle.GET(nullSchema, async (req, res) => { + seen = (req as any).session; + res.send({ ok: true }); + }); + + await endpoint.serve()(mockReq(), mockRes()); + expect(seen).toEqual(session); + }); +}); diff --git a/web-server/src/api-helpers/global.ts b/web-server/src/api-helpers/global.ts index 7cc16985..6c37d7bf 100644 --- a/web-server/src/api-helpers/global.ts +++ b/web-server/src/api-helpers/global.ts @@ -1,6 +1,9 @@ import { NextApiRequest } from 'next/types'; import { AnySchema, InferType, object } from 'yup'; +// CLUSTOX: enforce the `authenticated` flag that upstream declared but never read. +import { assertAuthenticated } from '@/auth/guard'; +import { getAuthSession } from '@/auth/session'; import { Errors, ResponseError } from '@/constants/error'; import { ApiRequest, ApiResponse, HttpMethods } from '@/types/request'; @@ -78,6 +81,15 @@ export class Endpoint { return async (nextReq: NextApiRequest, res: ApiResponse) => { try { const req = transformNextRequest(nextReq); + + // CLUSTOX: single enforcement point for all BFF routes. Routes opt out + // with `new Endpoint(schema, { unauthenticated: true })`. + if (this.authenticated) { + const session = await getAuthSession(nextReq); + assertAuthenticated(session); + (req as any).session = session; + } + if (this.pathSchema) { await this.pathSchema.validate(req.payload); } From 177d730ed4cb6924a7567eb97da28a84ad99b755 Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Thu, 6 Aug 2026 00:05:25 +0500 Subject: [PATCH 008/194] feat(auth): add login page and page-level guard Two corrections found during verification: - middleware must NOT redirect /api/* requests. Endpoint.serve() already returns 401 for those, and redirecting an XHR to an HTML login page surfaces as a parse error rather than an auth failure. - _app.tsx gated every page on auth.isInitialized, which AuthProvider sets only after a session fetch that now requires a session. That made the login page unreachable: the app waited forever for a login it would not render the form for. The gate is now bypassed on /login. This makes _app.tsx an eighth modified upstream file, one more than the plan estimated. --- web-server/middleware.ts | 37 ++++++++++++++-- web-server/pages/_app.tsx | 11 ++++- web-server/pages/login.tsx | 90 ++++++++++++++++++++++++++++++++++++++ 3 files changed, 132 insertions(+), 6 deletions(-) create mode 100644 web-server/pages/login.tsx diff --git a/web-server/middleware.ts b/web-server/middleware.ts index 09825d66..9d1488f2 100644 --- a/web-server/middleware.ts +++ b/web-server/middleware.ts @@ -1,13 +1,44 @@ +import { getToken } from 'next-auth/jwt'; import { NextRequest, NextResponse } from 'next/server'; import { getFeaturesFromReq } from '@/api-helpers/features'; import { defaultFlags } from '@/constants/feature'; +// CLUSTOX: paths reachable without a session. Health checks stay open so +// container and UI status probes keep working. +const PUBLIC_PATHS = [ + '/login', + '/api/auth', + '/api/status', + '/api/db_status', + '/api/internal_status', + '/api/hello' +]; + export async function middleware(request: NextRequest) { const flagOverrides = getFeaturesFromReq(request as any); const flags = { ...defaultFlags, ...flagOverrides }; const url = request.nextUrl.clone(); + const isPublic = PUBLIC_PATHS.some((p) => url.pathname.startsWith(p)); + + // CLUSTOX: redirect unauthenticated *page* requests to the sign-in screen. + // API routes are deliberately excluded: they are guarded independently in + // Endpoint.serve(), which returns 401. Redirecting an XHR to an HTML login + // page would surface as a confusing parse error rather than an auth failure. + const isApi = url.pathname.startsWith('/api/'); + if (!isPublic && !isApi) { + const token = await getToken({ + req: request, + secret: process.env.NEXTAUTH_SECRET + }); + if (!token) { + const login = request.nextUrl.clone(); + login.pathname = '/login'; + login.search = ''; + return NextResponse.redirect(login); + } + } // Forward as-is if it's a next-auth URL, except /session if ( @@ -24,9 +55,7 @@ export async function middleware(request: NextRequest) { export const config = { matcher: [ - '/api/auth/:path*', - '/api/integrations/:path*', - '/api/internal/:path*', - '/api/resources/:path*' + // CLUSTOX: widened from the API-only matcher so pages are guarded too. + '/((?!_next/static|_next/image|favicon.ico|assets|.*\\.png$|.*\\.svg$).*)' ] }; diff --git a/web-server/pages/_app.tsx b/web-server/pages/_app.tsx index c4fc3558..7ecaaf9b 100644 --- a/web-server/pages/_app.tsx +++ b/web-server/pages/_app.tsx @@ -2,7 +2,7 @@ import { CacheProvider, EmotionCache } from '@emotion/react'; import CssBaseline from '@mui/material/CssBaseline'; import { LocalizationProvider } from '@mui/x-date-pickers'; import { AdapterDateFns } from '@mui/x-date-pickers/AdapterDateFns'; -import Router from 'next/router'; +import Router, { useRouter } from 'next/router'; import { SnackbarProvider } from 'notistack'; import nProgress from 'nprogress'; import { ReactElement, ReactNode } from 'react'; @@ -54,6 +54,8 @@ interface MyAppProps extends AppProps { function MyApp(props: MyAppProps) { const { Component, emotionCache = clientSideEmotionCache, pageProps } = props; const getLayout = Component.getLayout ?? ((page) => page); + // CLUSTOX: see the AuthConsumer gate below. + const isLoginRoute = useRouter().pathname === '/login'; useScrollTop(); const overrides = useFlagOverrides(); useResizeEventTracking(); @@ -89,7 +91,12 @@ function MyApp(props: MyAppProps) { {(auth) => - !auth.isInitialized ? ( + // CLUSTOX: the login page must render without + // auth context. AuthProvider initialises by + // fetching session data, which now requires a + // session -- so gating /login on it would wait + // forever for a login that can never happen. + !auth.isInitialized && !isLoginRoute ? ( ) : ( getLayout() diff --git a/web-server/pages/login.tsx b/web-server/pages/login.tsx new file mode 100644 index 00000000..660b2570 --- /dev/null +++ b/web-server/pages/login.tsx @@ -0,0 +1,90 @@ +import { Box, Button, Card, TextField, Typography } from '@mui/material'; +import { signIn } from 'next-auth/react'; +import Head from 'next/head'; +import { useRouter } from 'next/router'; +import { FormEvent, ReactElement, useState } from 'react'; + +export default function Login() { + const router = useRouter(); + const [email, setEmail] = useState(''); + const [password, setPassword] = useState(''); + const [error, setError] = useState(''); + const [busy, setBusy] = useState(false); + + const onSubmit = async (e: FormEvent) => { + e.preventDefault(); + setBusy(true); + setError(''); + + const res = await signIn('credentials', { + email, + password, + redirect: false + }); + + setBusy(false); + + if (res?.error) { + // Deliberately does not distinguish unknown email from wrong password. + setError('Invalid email or password'); + return; + } + router.replace('/'); + }; + + return ( + <> + + Sign in | MiddlewareHQ + + + + + Sign in + +
+ setEmail(e.target.value)} + required + /> + setPassword(e.target.value)} + required + /> + {error && ( + + {error} + + )} + + +
+
+ + ); +} + +Login.getLayout = (page: ReactElement) => page; From d1c980d0295487bbcb07488ddfcbc5d533d0484a Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Thu, 6 Aug 2026 00:20:49 +0500 Subject: [PATCH 009/194] feat(auth): bootstrap first superadmin from environment Two upstream files needed changes that the plan did not anticipate: - next.config.js: knex resolves dialects dynamically, which webpack cannot bundle, so the instrumentation hook failed to compile and the bootstrap never ran. Fixed with serverExternalPackages: ['knex','pg']. API routes were unaffected, so this only surfaced once instrumentation started importing the db helper. - ThirdPartyAuthContext.tsx: fetchSession had no .catch(), so the now-401 session request rejected unhandled and buried the login page under a runtime error overlay. A 401 there is the expected unauthenticated case; it now falls through to the provider's existing onUnauthenticated path. No default password: absent env vars log loudly and create nothing, rather than falling back to a known value. Verified end to end in a browser: unauthenticated request redirects to /login, superadmin signs in, dashboards render with live data. --- web-server/instrumentation.ts | 8 +++ web-server/next.config.js | 5 ++ .../src/auth/__tests__/bootstrap.test.ts | 58 +++++++++++++++++++ web-server/src/auth/bootstrap.ts | 48 +++++++++++++++ .../src/contexts/ThirdPartyAuthContext.tsx | 15 +++-- 5 files changed, 130 insertions(+), 4 deletions(-) create mode 100644 web-server/instrumentation.ts create mode 100644 web-server/src/auth/__tests__/bootstrap.test.ts create mode 100644 web-server/src/auth/bootstrap.ts diff --git a/web-server/instrumentation.ts b/web-server/instrumentation.ts new file mode 100644 index 00000000..a2a5793d --- /dev/null +++ b/web-server/instrumentation.ts @@ -0,0 +1,8 @@ +// CLUSTOX: seed the first superadmin on server start. The NEXT_RUNTIME check +// keeps this off the Edge runtime, where knex cannot run. +export async function register() { + if (process.env.NEXT_RUNTIME === 'nodejs') { + const { bootstrapSuperadmin } = await import('@/auth/bootstrap'); + await bootstrapSuperadmin(); + } +} diff --git a/web-server/next.config.js b/web-server/next.config.js index 0433ca36..a6ceb6cc 100644 --- a/web-server/next.config.js +++ b/web-server/next.config.js @@ -49,6 +49,11 @@ const genericPlugins = [ ]; const staticOverrides = { + // CLUSTOX: knex resolves dialects dynamically, which webpack cannot bundle. + // Without this the instrumentation hook (superadmin bootstrap) fails to + // compile. API routes were unaffected, so this only surfaced once + // instrumentation.ts started importing the db helper. + serverExternalPackages: ['knex', 'pg'], staticPageGenerationTimeout: 2000, images: { disableStaticImages: true diff --git a/web-server/src/auth/__tests__/bootstrap.test.ts b/web-server/src/auth/__tests__/bootstrap.test.ts new file mode 100644 index 00000000..cc951fcc --- /dev/null +++ b/web-server/src/auth/__tests__/bootstrap.test.ts @@ -0,0 +1,58 @@ +jest.mock('@/auth/queries', () => ({ + countSuperadmins: jest.fn(), + createUser: jest.fn() +})); +jest.mock('@/utils/db', () => ({ db: jest.fn() })); + +import { bootstrapSuperadmin } from '@/auth/bootstrap'; +import { countSuperadmins, createUser } from '@/auth/queries'; + +describe('bootstrapSuperadmin', () => { + const env = process.env; + + beforeEach(() => { + jest.clearAllMocks(); + process.env = { ...env }; + }); + + afterAll(() => { + process.env = env; + }); + + it('does nothing when a superadmin already exists', async () => { + (countSuperadmins as jest.Mock).mockResolvedValue(1); + process.env.SUPERADMIN_EMAIL = 'a@clustox.com'; + process.env.SUPERADMIN_PASSWORD = 'pw'; + + await bootstrapSuperadmin(); + expect(createUser).not.toHaveBeenCalled(); + }); + + it('does not create a user when env vars are missing', async () => { + (countSuperadmins as jest.Mock).mockResolvedValue(0); + delete process.env.SUPERADMIN_EMAIL; + delete process.env.SUPERADMIN_PASSWORD; + + await bootstrapSuperadmin(); + expect(createUser).not.toHaveBeenCalled(); + }); + + it('creates the superadmin when none exists and env vars are set', async () => { + (countSuperadmins as jest.Mock).mockResolvedValue(0); + (createUser as jest.Mock).mockResolvedValue('new-id'); + process.env.SUPERADMIN_EMAIL = 'boss@clustox.com'; + process.env.SUPERADMIN_PASSWORD = 'strong-password'; + + await bootstrapSuperadmin('org-1'); + + expect(createUser).toHaveBeenCalledWith( + expect.objectContaining({ + email: 'boss@clustox.com', + password: 'strong-password', + role: 'SUPERADMIN', + teamIds: [], + orgId: 'org-1' + }) + ); + }); +}); diff --git a/web-server/src/auth/bootstrap.ts b/web-server/src/auth/bootstrap.ts new file mode 100644 index 00000000..005f9db6 --- /dev/null +++ b/web-server/src/auth/bootstrap.ts @@ -0,0 +1,48 @@ +import { Table } from '@/constants/db'; +import { db } from '@/utils/db'; + +import { countSuperadmins, createUser } from './queries'; + +const getDefaultOrgId = async (): Promise => { + const row = await db(Table.Organization).select('id').first(); + return row?.id ?? null; +}; + +export const bootstrapSuperadmin = async ( + orgIdOverride?: string +): Promise => { + const existing = await countSuperadmins(); + if (existing > 0) return; + + const email = process.env.SUPERADMIN_EMAIL; + const password = process.env.SUPERADMIN_PASSWORD; + + if (!email || !password) { + // Fail loudly but do not invent a default account. A known-value fallback + // is worse than having no superadmin at all. + console.error( + '[CLUSTOX] No superadmin exists and SUPERADMIN_EMAIL / SUPERADMIN_PASSWORD ' + + 'are not set. Nobody can administer this instance until they are.' + ); + return; + } + + const orgId = orgIdOverride ?? (await getDefaultOrgId()); + if (!orgId) { + console.error( + '[CLUSTOX] No organization found; cannot bootstrap superadmin.' + ); + return; + } + + await createUser({ + name: 'Superadmin', + email, + password, + role: 'SUPERADMIN', + teamIds: [], + orgId + }); + + console.info(`[CLUSTOX] Bootstrapped superadmin ${email}`); +}; diff --git a/web-server/src/contexts/ThirdPartyAuthContext.tsx b/web-server/src/contexts/ThirdPartyAuthContext.tsx index fe3dffaf..276ab273 100644 --- a/web-server/src/contexts/ThirdPartyAuthContext.tsx +++ b/web-server/src/contexts/ThirdPartyAuthContext.tsx @@ -53,10 +53,17 @@ export const AuthProvider: FC = (props) => { const loadingState = useBoolState(true); const fetchSession = useCallback(() => { - return axios - .get('/api/auth/session') - .then((r) => r.data) - .then(sessionState.set); + return ( + axios + .get('/api/auth/session') + .then((r) => r.data) + .then(sessionState.set) + // CLUSTOX: a 401 here is the expected unauthenticated case, not an + // error. Without this the rejection is unhandled and the login page + // is buried under a runtime error overlay. Falling through to a null + // session lets initialize() take its existing onUnauthenticated path. + .catch(() => sessionState.set(null)) + ); }, [sessionState.set]); useEffect(() => { From 64825b92d11a12e67e39508d4394f9cec09202ed Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Thu, 6 Aug 2026 00:38:50 +0500 Subject: [PATCH 010/194] feat(auth): restrict Flask servers to BFF via shared secret This is what makes the auth real rather than cosmetic. Before this, curl http://localhost:9696/teams//lead_time returned full metrics with no credentials, bypassing every guard added so far. Fails closed: an unset INTERNAL_API_TOKEN rejects all requests rather than allowing them. hmac.compare_digest avoids leaking the token through response timing. Health endpoints stay exempt so container and UI status probes keep working. Cron needed its own path: it does not inherit the container environment, so it now runs trigger_sync.sh which sources /app/.env and sends the header. This was the highest-risk step -- a broken sync reports HTTP 200 regardless of outcome -- so it was verified by running the helper and confirming a full sync completed. Verified: direct :9696 and :9697 -> 403, health -> 200, correct token -> 200, cron sync completes, dashboards still render in the browser. 155 pytest tests pass. --- Dockerfile.dev | 3 +- backend/analytics_server/app.py | 6 ++ .../mhq/clustox_auth/__init__.py | 0 .../mhq/clustox_auth/internal_token.py | 37 ++++++++++++ backend/analytics_server/sync_app.py | 7 +++ .../tests/clustox_auth/__init__.py | 0 .../tests/clustox_auth/test_internal_token.py | 59 +++++++++++++++++++ setup_utils/cronjob.txt | 4 +- setup_utils/trigger_sync.sh | 10 ++++ web-server/src/api-helpers/axios.ts | 12 ++++ 10 files changed, 136 insertions(+), 2 deletions(-) create mode 100644 backend/analytics_server/mhq/clustox_auth/__init__.py create mode 100644 backend/analytics_server/mhq/clustox_auth/internal_token.py create mode 100644 backend/analytics_server/tests/clustox_auth/__init__.py create mode 100644 backend/analytics_server/tests/clustox_auth/test_internal_token.py create mode 100644 setup_utils/trigger_sync.sh diff --git a/Dockerfile.dev b/Dockerfile.dev index 1c747f8d..83e99ab8 100644 --- a/Dockerfile.dev +++ b/Dockerfile.dev @@ -38,7 +38,8 @@ RUN --mount=type=cache,target=/root/.cache/pip /opt/venv/bin/pip install -r requ WORKDIR /app RUN mkdir -p /etc/cron.d && mv /app/setup_utils/cronjob.txt /etc/cron.d/cronjob -RUN chmod +x /app/setup_utils/start.sh /app/setup_utils/init_db.sh /app/setup_utils/generate_config_ini.sh +# CLUSTOX: trigger_sync.sh added so cron can send the internal API token. +RUN chmod +x /app/setup_utils/start.sh /app/setup_utils/init_db.sh /app/setup_utils/generate_config_ini.sh /app/setup_utils/trigger_sync.sh RUN mv /app/setup_utils/supervisord.conf /etc/supervisord.conf RUN mv /app/database-docker/db/ /app/ && rm -rf /app/database-docker/ RUN echo "host all all 0.0.0.0/0 md5" >> /etc/postgresql/15/main/pg_hba.conf diff --git a/backend/analytics_server/app.py b/backend/analytics_server/app.py index c246a9c8..2cee73ab 100644 --- a/backend/analytics_server/app.py +++ b/backend/analytics_server/app.py @@ -19,6 +19,9 @@ from mhq.store.initialise_db import initialize_database +# CLUSTOX: restrict this API to the BFF. See mhq/clustox_auth/internal_token.py +from mhq.clustox_auth.internal_token import register_internal_token_guard + ANALYTICS_SERVER_PORT = getenv("ANALYTICS_SERVER_PORT") app = Flask(__name__) @@ -33,6 +36,9 @@ app.register_blueprint(bookmark_api) app.register_blueprint(ai_api) +# CLUSTOX: reject any request that did not come from the BFF. +register_internal_token_guard(app) + configure_db_with_app(app) initialize_database(app) diff --git a/backend/analytics_server/mhq/clustox_auth/__init__.py b/backend/analytics_server/mhq/clustox_auth/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/backend/analytics_server/mhq/clustox_auth/internal_token.py b/backend/analytics_server/mhq/clustox_auth/internal_token.py new file mode 100644 index 00000000..d14e76a8 --- /dev/null +++ b/backend/analytics_server/mhq/clustox_auth/internal_token.py @@ -0,0 +1,37 @@ +""" +CLUSTOX: restrict the Flask servers to traffic from the Next.js BFF. + +Upstream ships these APIs unauthenticated, relying on the loopback port binding +in docker-compose. Once authorization lives in the BFF, a direct call to :9696 +would bypass every check, so every request must carry a shared secret. + +Fails closed: if INTERNAL_API_TOKEN is unset, all requests are rejected. +""" + +import hmac +from os import getenv + +from flask import Flask, jsonify, request + +HEADER = "X-Internal-Token" + +# Health checks must stay reachable for container and UI status probes. +EXEMPT_PATHS = {"/"} + + +def register_internal_token_guard(app: Flask) -> None: + @app.before_request + def _verify_internal_token(): + if request.path in EXEMPT_PATHS: + return None + + expected = getenv("INTERNAL_API_TOKEN") + if not expected: + return jsonify({"error": "internal token not configured"}), 403 + + provided = request.headers.get(HEADER, "") + # compare_digest avoids leaking the token through response timing. + if not hmac.compare_digest(provided, expected): + return jsonify({"error": "forbidden"}), 403 + + return None diff --git a/backend/analytics_server/sync_app.py b/backend/analytics_server/sync_app.py index c6711757..5007c10a 100644 --- a/backend/analytics_server/sync_app.py +++ b/backend/analytics_server/sync_app.py @@ -10,6 +10,10 @@ from mhq.api.hello import app as core_api from mhq.api.sync import app as sync_api +# CLUSTOX: restrict this API to the BFF and cron. +# See mhq/clustox_auth/internal_token.py +from mhq.clustox_auth.internal_token import register_internal_token_guard + SYNC_SERVER_PORT = getenv("SYNC_SERVER_PORT") app = Flask(__name__) @@ -17,6 +21,9 @@ app.register_blueprint(core_api) app.register_blueprint(sync_api) +# CLUSTOX: reject any request that did not come from the BFF or cron. +register_internal_token_guard(app) + configure_db_with_app(app) if __name__ == "__main__": diff --git a/backend/analytics_server/tests/clustox_auth/__init__.py b/backend/analytics_server/tests/clustox_auth/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/backend/analytics_server/tests/clustox_auth/test_internal_token.py b/backend/analytics_server/tests/clustox_auth/test_internal_token.py new file mode 100644 index 00000000..f9a9493f --- /dev/null +++ b/backend/analytics_server/tests/clustox_auth/test_internal_token.py @@ -0,0 +1,59 @@ +import os + +from flask import Flask + +from mhq.clustox_auth.internal_token import register_internal_token_guard + + +def build_app(token): + if token is None: + os.environ.pop("INTERNAL_API_TOKEN", None) + else: + os.environ["INTERNAL_API_TOKEN"] = token + + app = Flask(__name__) + register_internal_token_guard(app) + + @app.route("/protected") + def protected(): + return {"ok": True} + + @app.route("/") + def health(): + return "hello" + + return app + + +def test_request_with_valid_token_is_allowed(): + app = build_app("secret-token") + client = app.test_client() + res = client.get("/protected", headers={"X-Internal-Token": "secret-token"}) + assert res.status_code == 200 + + +def test_request_without_token_is_rejected(): + app = build_app("secret-token") + res = app.test_client().get("/protected") + assert res.status_code == 403 + + +def test_request_with_wrong_token_is_rejected(): + app = build_app("secret-token") + client = app.test_client() + res = client.get("/protected", headers={"X-Internal-Token": "wrong"}) + assert res.status_code == 403 + + +def test_unset_env_fails_closed(): + """An unconfigured token must reject everything, never allow everything.""" + app = build_app(None) + client = app.test_client() + res = client.get("/protected", headers={"X-Internal-Token": "anything"}) + assert res.status_code == 403 + + +def test_health_endpoint_is_exempt(): + app = build_app("secret-token") + res = app.test_client().get("/") + assert res.status_code == 200 diff --git a/setup_utils/cronjob.txt b/setup_utils/cronjob.txt index 5fc95d91..5366b6e4 100644 --- a/setup_utils/cronjob.txt +++ b/setup_utils/cronjob.txt @@ -1,4 +1,6 @@ #!/usr/bin/env python3 # Every 30 minutes, run the sync script -*/30 * * * * curl -X POST http://localhost:9697/sync >> /var/log/cron/cron.log 2>&1 +# CLUSTOX: routed through trigger_sync.sh so the internal token is sent. +# The sync server rejects unauthenticated requests since auth was added. +*/30 * * * * /app/setup_utils/trigger_sync.sh >> /var/log/cron/cron.log 2>&1 diff --git a/setup_utils/trigger_sync.sh b/setup_utils/trigger_sync.sh new file mode 100644 index 00000000..ed3af6fc --- /dev/null +++ b/setup_utils/trigger_sync.sh @@ -0,0 +1,10 @@ +#!/bin/bash +# CLUSTOX: cron does not inherit the container environment, so source it before +# calling the sync server, which now requires the internal token. +set -a +[ -f /app/.env ] && . /app/.env +set +a + +curl -s -X POST \ + -H "X-Internal-Token: ${INTERNAL_API_TOKEN}" \ + "http://localhost:${SYNC_SERVER_PORT:-9697}/sync" diff --git a/web-server/src/api-helpers/axios.ts b/web-server/src/api-helpers/axios.ts index d5b031fe..c516b8db 100644 --- a/web-server/src/api-helpers/axios.ts +++ b/web-server/src/api-helpers/axios.ts @@ -31,6 +31,18 @@ export const internalSyncServer = axios.create({ baseURL: process.env.INTERNAL_SYNC_API_BASE_URL }); +// CLUSTOX: the Flask servers reject any request without this shared secret. +// Stamped here because every outbound call funnels through these two +// instances, so there is no route that can forget it. +const internalTokenInterceptor = (req: any) => { + req.headers = req.headers || {}; + req.headers['X-Internal-Token'] = process.env.INTERNAL_API_TOKEN; + return req; +}; + +internal.interceptors.request.use(internalTokenInterceptor); +internalSyncServer.interceptors.request.use(internalTokenInterceptor); + axiosRetry(internal, { retries: 2, retryCondition(error) { From 203d486a6f2262d33b9dc85aa7ff690c85d721e7 Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Thu, 6 Aug 2026 00:45:19 +0500 Subject: [PATCH 011/194] feat(auth): scope team routes and restrict admin actions Deviation from the plan, deliberately: rather than adding an assertTeamAccess call to each of the 18 team-scoped routes, the check runs centrally in Endpoint.serve() whenever the payload carries a team_id. One file instead of eighteen, and a newly added team route is protected by default rather than depending on the author remembering. Team list endpoints still need explicit filtering since they carry no team_id, so index.ts and v2.ts filter through getAccessibleTeamIds. Team creation and integration deletion are superadmin-only. share.ts is no longer unauthenticated: it allowed anonymous POSTs that inserted unbounded rows into URLShortenerData. Verified live as an admin scoped to ZDA only: own team team_repos -> 200 other team team_repos -> 403 other team dora_metrics -> 403 team list -> ['ZDA'] only 152 jest tests pass; the 2 failing suites fail identically on upstream. --- web-server/pages/api/integrations/index.ts | 6 ++ .../resources/orgs/[org_id]/teams/index.ts | 15 ++- .../api/resources/orgs/[org_id]/teams/v2.ts | 10 +- web-server/pages/api/resources/share.ts | 5 +- .../__tests__/endpoint-team-scope.test.ts | 102 ++++++++++++++++++ web-server/src/api-helpers/global.ts | 9 +- 6 files changed, 143 insertions(+), 4 deletions(-) create mode 100644 web-server/src/api-helpers/__tests__/endpoint-team-scope.test.ts diff --git a/web-server/pages/api/integrations/index.ts b/web-server/pages/api/integrations/index.ts index 23516794..ce22d085 100644 --- a/web-server/pages/api/integrations/index.ts +++ b/web-server/pages/api/integrations/index.ts @@ -1,6 +1,8 @@ import * as yup from 'yup'; import { Endpoint, nullSchema } from '@/api-helpers/global'; +// CLUSTOX: integration management is superadmin-only. +import { assertRole } from '@/auth/guard'; import { Columns, Table } from '@/constants/db'; import { Integration } from '@/constants/integrations'; import { db, getFirstRow } from '@/utils/db'; @@ -25,6 +27,10 @@ endpoint.handle.GET(getSchema, async (req, res) => { }); endpoint.handle.DELETE(deleteSchema, async (req, res) => { + // CLUSTOX: unlinking a code provider wipes the org's synced data source. + // Superadmin only. + assertRole((req as any).session, 'SUPERADMIN'); + const data = await db(Table.Integration) .delete() .where({ diff --git a/web-server/pages/api/resources/orgs/[org_id]/teams/index.ts b/web-server/pages/api/resources/orgs/[org_id]/teams/index.ts index f5a9c3f6..f5a70a08 100644 --- a/web-server/pages/api/resources/orgs/[org_id]/teams/index.ts +++ b/web-server/pages/api/resources/orgs/[org_id]/teams/index.ts @@ -8,6 +8,8 @@ import { import { getTeamRepos } from '@/api/resources/team_repos'; import { Endpoint } from '@/api-helpers/global'; import { getTeamMembersFilterSettingForOrg } from '@/api-helpers/team'; +// CLUSTOX: per-team scoping for the team picker. +import { getAccessibleTeamIds } from '@/auth/guard'; import { getTeamV2Mock } from '@/mocks/teams'; import { FetchTeamsResponse } from '@/types/resources'; import { db } from '@/utils/db'; @@ -28,7 +30,18 @@ endpoint.handle.GET(getSchema, async (req, res) => { } const { include_teams, org_id, user_id } = req.payload; - res.send(await getOrgTeams(org_id, user_id, include_teams)); + const result = await getOrgTeams(org_id, user_id, include_teams); + + // CLUSTOX: an admin must only see their assigned teams in the picker. + // Filtered here rather than in getOrgTeams because that helper is also + // imported by other routes which do their own scoping. + const accessible = await getAccessibleTeamIds((req as any).session); + const allowed = new Set(accessible); + + res.send({ + ...result, + teams: result.teams.filter((t: { id: ID }) => allowed.has(t.id)) + }); }); export const getOrgTeams = async ( diff --git a/web-server/pages/api/resources/orgs/[org_id]/teams/v2.ts b/web-server/pages/api/resources/orgs/[org_id]/teams/v2.ts index bf0db917..c437d604 100644 --- a/web-server/pages/api/resources/orgs/[org_id]/teams/v2.ts +++ b/web-server/pages/api/resources/orgs/[org_id]/teams/v2.ts @@ -13,6 +13,8 @@ import { } from '@/api/resources/orgs/[org_id]/onboarding'; import { handleRequest } from '@/api-helpers/axios'; import { Endpoint } from '@/api-helpers/global'; +// CLUSTOX: per-team scoping and superadmin-only team creation. +import { assertRole, getAccessibleTeamIds } from '@/auth/guard'; import { Row, Table } from '@/constants/db'; import { CIProvider, @@ -94,13 +96,19 @@ endpoint.handle.GET(getSchema, async (req, res) => { : [Integration.GITHUB, Integration.GITLAB] ); + // CLUSTOX: an admin must only see their assigned teams in the picker. + const accessible = await getAccessibleTeamIds((req as any).session); + const allowed = new Set(accessible); + res.send({ - teams: teams, + teams: teams.filter((t: { id: ID }) => allowed.has(t.id)), teamReposMap }); }); endpoint.handle.POST(postSchema, async (req, res) => { + // CLUSTOX: only superadmins create teams. + assertRole((req as any).session, 'SUPERADMIN'); if (req.meta?.features?.use_mock_data) { return res.send(getTeamV2Mock); } diff --git a/web-server/pages/api/resources/share.ts b/web-server/pages/api/resources/share.ts index 12f53e51..8bb882bd 100644 --- a/web-server/pages/api/resources/share.ts +++ b/web-server/pages/api/resources/share.ts @@ -14,7 +14,10 @@ const postSchema = yup.object().shape({ base_url: yup.string().optional() }); -const endpoint = new Endpoint(nullSchema, { unauthenticated: true }); +// CLUSTOX: was unauthenticated, which let anyone POST unbounded rows into +// URLShortenerData and read stored link state. Everyone who should open a +// share link has an account, so requiring one costs nothing. +const endpoint = new Endpoint(nullSchema); endpoint.handle.GET(getSchema, async (req, res) => { const { share_link_id } = req.payload; diff --git a/web-server/src/api-helpers/__tests__/endpoint-team-scope.test.ts b/web-server/src/api-helpers/__tests__/endpoint-team-scope.test.ts new file mode 100644 index 00000000..244d1d30 --- /dev/null +++ b/web-server/src/api-helpers/__tests__/endpoint-team-scope.test.ts @@ -0,0 +1,102 @@ +jest.mock('@/auth/session', () => ({ getAuthSession: jest.fn() })); +jest.mock('@/auth/queries', () => ({ + getTeamIdsForUser: jest.fn(), + getAllTeamIds: jest.fn() +})); + +import { Endpoint, nullSchema } from '@/api-helpers/global'; +import { getTeamIdsForUser } from '@/auth/queries'; +import { getAuthSession } from '@/auth/session'; + +const mockRes = () => { + const res: any = {}; + res.status = jest.fn(() => res); + res.send = jest.fn(() => res); + return res; +}; + +const reqWithTeam = (team_id?: string) => + ({ + method: 'GET', + query: team_id ? { team_id } : {}, + body: {}, + headers: {} + }) as any; + +const asAdmin = () => + (getAuthSession as jest.Mock).mockResolvedValue({ + userId: 'ad1', + email: 'lead@clustox.com', + name: 'Lead', + role: 'ADMIN' + }); + +const asSuperadmin = () => + (getAuthSession as jest.Mock).mockResolvedValue({ + userId: 'su1', + email: 'boss@clustox.com', + name: 'Boss', + role: 'SUPERADMIN' + }); + +const buildEndpoint = () => { + const endpoint = new Endpoint(nullSchema); + endpoint.handle.GET(nullSchema, async (_req, res) => res.send({ ok: true })); + return endpoint; +}; + +describe('Endpoint team scoping', () => { + beforeEach(() => jest.clearAllMocks()); + + it('403s when an admin requests a team they are not assigned', async () => { + asAdmin(); + (getTeamIdsForUser as jest.Mock).mockResolvedValue(['team-zda']); + + const res = mockRes(); + await buildEndpoint().serve()(reqWithTeam('team-cgpt'), res); + + expect(res.status).toHaveBeenCalledWith(403); + }); + + it('allows an admin their own team', async () => { + asAdmin(); + (getTeamIdsForUser as jest.Mock).mockResolvedValue(['team-zda']); + + const res = mockRes(); + await buildEndpoint().serve()(reqWithTeam('team-zda'), res); + + expect(res.send).toHaveBeenCalledWith({ ok: true }); + }); + + it('allows a superadmin any team', async () => { + asSuperadmin(); + + const res = mockRes(); + await buildEndpoint().serve()(reqWithTeam('team-anything'), res); + + expect(res.send).toHaveBeenCalledWith({ ok: true }); + expect(getTeamIdsForUser).not.toHaveBeenCalled(); + }); + + it('does not attempt a team check when the payload has no team_id', async () => { + asAdmin(); + + const res = mockRes(); + await buildEndpoint().serve()(reqWithTeam(undefined), res); + + expect(res.send).toHaveBeenCalledWith({ ok: true }); + expect(getTeamIdsForUser).not.toHaveBeenCalled(); + }); + + it('skips the team check entirely on unauthenticated endpoints', async () => { + (getAuthSession as jest.Mock).mockResolvedValue(null); + + const endpoint = new Endpoint(nullSchema, { unauthenticated: true }); + endpoint.handle.GET(nullSchema, async (_req, res) => res.send({ ok: true })); + + const res = mockRes(); + await endpoint.serve()(reqWithTeam('team-zda'), res); + + expect(res.send).toHaveBeenCalledWith({ ok: true }); + }); +}); diff --git a/web-server/src/api-helpers/global.ts b/web-server/src/api-helpers/global.ts index 6c37d7bf..6b973b93 100644 --- a/web-server/src/api-helpers/global.ts +++ b/web-server/src/api-helpers/global.ts @@ -2,7 +2,7 @@ import { NextApiRequest } from 'next/types'; import { AnySchema, InferType, object } from 'yup'; // CLUSTOX: enforce the `authenticated` flag that upstream declared but never read. -import { assertAuthenticated } from '@/auth/guard'; +import { assertAuthenticated, assertTeamAccess } from '@/auth/guard'; import { getAuthSession } from '@/auth/session'; import { Errors, ResponseError } from '@/constants/error'; import { ApiRequest, ApiResponse, HttpMethods } from '@/types/request'; @@ -88,6 +88,13 @@ export class Endpoint { const session = await getAuthSession(nextReq); assertAuthenticated(session); (req as any).session = session; + + // CLUSTOX: any route addressing a specific team is scoped centrally + // here rather than in each of the ~18 team routes. Doing it at the + // choke point means a newly added team route is protected by default + // instead of relying on the author remembering a guard call. + const teamId = (req.payload as any)?.team_id; + if (teamId) await assertTeamAccess(session, String(teamId)); } if (this.pathSchema) { From 67ec817e6fbf4c67c8510720ae21e74426f60cf9 Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Thu, 6 Aug 2026 01:14:31 +0500 Subject: [PATCH 012/194] feat(auth): add user management API and page Superadmin-only. Creates users with a role and team assignments, and edits team scoping inline from the table. Guards beyond the plan: - duplicate email returns 409 rather than creating a shadow account - a superadmin cannot demote themselves when they are the last one, which would leave the instance unadministerable through the UI - 12-character minimum enforced in the yup schema, so the UI cannot create a weak account The page derives its own authorisation from the API's 403 rather than from AuthContext, whose role field is hardcoded to UserRole.MOM upstream. That avoids another upstream edit and keeps the server as the single source of truth. Verified live: weak password 400, duplicate email 409, valid create 200, admin gets 403 on the users API and sees 'Not authorised' on the page. --- .../pages/api/clustox/users/[user_id].ts | 44 +++ web-server/pages/api/clustox/users/index.ts | 51 +++ web-server/pages/users.tsx | 296 ++++++++++++++++++ 3 files changed, 391 insertions(+) create mode 100644 web-server/pages/api/clustox/users/[user_id].ts create mode 100644 web-server/pages/api/clustox/users/index.ts create mode 100644 web-server/pages/users.tsx diff --git a/web-server/pages/api/clustox/users/[user_id].ts b/web-server/pages/api/clustox/users/[user_id].ts new file mode 100644 index 00000000..5f416c1a --- /dev/null +++ b/web-server/pages/api/clustox/users/[user_id].ts @@ -0,0 +1,44 @@ +import * as yup from 'yup'; + +import { Endpoint } from '@/api-helpers/global'; +import { assertRole } from '@/auth/guard'; +import { countSuperadmins, setUserTeams, updateUserRole } from '@/auth/queries'; +import { ClustoxRole } from '@/auth/types'; +import { Errors, ResponseError } from '@/constants/error'; + +const pathSchema = yup.object().shape({ + user_id: yup.string().uuid().required() +}); + +const patchSchema = yup.object().shape({ + user_id: yup.string().uuid().required(), + role: yup.string().oneOf(['SUPERADMIN', 'ADMIN']).optional(), + team_ids: yup.array().of(yup.string().uuid()).optional() +}); + +const endpoint = new Endpoint(pathSchema); + +endpoint.handle.PATCH(patchSchema, async (req, res) => { + const session = assertRole((req as any).session, 'SUPERADMIN'); + + // Demoting the last superadmin would leave nobody able to administer the + // instance, which is unrecoverable through the UI. + if (req.payload.role === 'ADMIN') { + const remaining = await countSuperadmins(); + if (remaining <= 1 && session.userId === req.payload.user_id) { + throw new ResponseError(Errors.ACCESS_DENIED, 409); + } + } + + if (req.payload.role) { + // yup's .oneOf() widens to string; the schema has already validated it. + await updateUserRole(req.payload.user_id, req.payload.role as ClustoxRole); + } + if (req.payload.team_ids) { + await setUserTeams(req.payload.user_id, req.payload.team_ids); + } + + res.send({ ok: true }); +}); + +export default endpoint.serve(); diff --git a/web-server/pages/api/clustox/users/index.ts b/web-server/pages/api/clustox/users/index.ts new file mode 100644 index 00000000..b9ee0577 --- /dev/null +++ b/web-server/pages/api/clustox/users/index.ts @@ -0,0 +1,51 @@ +import * as yup from 'yup'; + +import { Endpoint, nullSchema } from '@/api-helpers/global'; +import { assertRole } from '@/auth/guard'; +import { createUser, getAuthUserByEmail, listUsers } from '@/auth/queries'; +import { ClustoxRole } from '@/auth/types'; +import { Table } from '@/constants/db'; +import { Errors, ResponseError } from '@/constants/error'; +import { db } from '@/utils/db'; + +const postSchema = yup.object().shape({ + name: yup.string().required(), + email: yup.string().email().required(), + // Enforced here so the UI cannot create a weak account. + password: yup.string().min(12).required(), + role: yup.string().oneOf(['SUPERADMIN', 'ADMIN']).required(), + team_ids: yup.array().of(yup.string().uuid()).required() +}); + +const endpoint = new Endpoint(nullSchema); + +endpoint.handle.GET(nullSchema, async (req, res) => { + assertRole((req as any).session, 'SUPERADMIN'); + res.send(await listUsers()); +}); + +endpoint.handle.POST(postSchema, async (req, res) => { + assertRole((req as any).session, 'SUPERADMIN'); + + const existing = await getAuthUserByEmail(req.payload.email); + if (existing) { + throw new ResponseError(Errors.INSUFFICIENT_PARAMS, 409); + } + + const org = await db(Table.Organization).select('id').first(); + if (!org) throw new ResponseError(Errors.USER_NOT_FOUND, 500); + + const userId = await createUser({ + name: req.payload.name, + email: req.payload.email, + password: req.payload.password, + // yup's .oneOf() widens to string; the schema has already validated it. + role: req.payload.role as ClustoxRole, + teamIds: req.payload.team_ids, + orgId: org.id + }); + + res.send({ user_id: userId }); +}); + +export default endpoint.serve(); diff --git a/web-server/pages/users.tsx b/web-server/pages/users.tsx new file mode 100644 index 00000000..8b510e03 --- /dev/null +++ b/web-server/pages/users.tsx @@ -0,0 +1,296 @@ +import { + Alert, + Box, + Button, + Card, + Chip, + CircularProgress, + FormControl, + InputLabel, + MenuItem, + Select, + Table, + TableBody, + TableCell, + TableHead, + TableRow, + TextField, + Typography +} from '@mui/material'; +import Head from 'next/head'; +import { FormEvent, useCallback, useEffect, useState } from 'react'; +import ExtendedSidebarLayout from 'src/layouts/ExtendedSidebarLayout'; + +import { FlexBox } from '@/components/FlexBox'; +import { PageLayout } from '@/types/resources'; + +type ClustoxRole = 'SUPERADMIN' | 'ADMIN'; + +type UserRow = { + userId: string; + email: string; + name: string; + role: ClustoxRole; + teamIds: string[]; +}; + +type Team = { id: string; name: string }; + +const emptyForm = { + name: '', + email: '', + password: '', + role: 'ADMIN' as ClustoxRole, + team_ids: [] as string[] +}; + +function UsersPage() { + const [users, setUsers] = useState([]); + const [teams, setTeams] = useState([]); + const [loading, setLoading] = useState(true); + const [forbidden, setForbidden] = useState(false); + const [error, setError] = useState(''); + const [busy, setBusy] = useState(false); + const [form, setForm] = useState(emptyForm); + + const loadUsers = useCallback(async () => { + const res = await fetch('/api/clustox/users'); + if (res.status === 403) { + setForbidden(true); + setLoading(false); + return; + } + setUsers(await res.json()); + setLoading(false); + }, []); + + const loadTeams = useCallback(async () => { + const sessionRes = await fetch('/api/auth/session'); + if (!sessionRes.ok) return; + const session = await sessionRes.json(); + const orgId = session?.org?.id; + if (!orgId) return; + + const res = await fetch(`/api/resources/orgs/${orgId}/teams`); + if (!res.ok) return; + const data = await res.json(); + setTeams(data.teams || []); + }, []); + + useEffect(() => { + loadUsers(); + loadTeams(); + }, [loadUsers, loadTeams]); + + const teamName = (id: string) => + teams.find((t) => t.id === id)?.name ?? id.slice(0, 8); + + const onCreate = async (e: FormEvent) => { + e.preventDefault(); + setBusy(true); + setError(''); + + const res = await fetch('/api/clustox/users', { + method: 'POST', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify(form) + }); + + setBusy(false); + + if (!res.ok) { + setError( + res.status === 409 + ? 'A user with that email already exists' + : 'Could not create user. Password must be at least 12 characters.' + ); + return; + } + + setForm(emptyForm); + await loadUsers(); + }; + + const onChangeTeams = async (userId: string, teamIds: string[]) => { + await fetch(`/api/clustox/users/${userId}`, { + method: 'PATCH', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify({ team_ids: teamIds }) + }); + await loadUsers(); + }; + + if (loading) + return ( + + + + ); + + if (forbidden) + return ( + + + Not authorised. Only superadmins can manage users. + + + ); + + return ( + <> + + Users | MiddlewareHQ + + + + Users + + + + + Add a user + +
+ + setForm({ ...form, name: e.target.value })} + /> + setForm({ ...form, email: e.target.value })} + /> + setForm({ ...form, password: e.target.value })} + /> + + Role + + + + Teams + + + + + {form.role === 'SUPERADMIN' && ( + + Superadmins see every team, so no team assignment is needed. + + )} + {error && ( + + {error} + + )} +
+
+ + + + + + Name + Email + Role + Teams + + + + {users.map((u) => ( + + {u.name} + {u.email} + + + + + {u.role === 'SUPERADMIN' ? ( + + All teams + + ) : ( + + )} + + + ))} + +
+
+
+ + ); +} + +UsersPage.getLayout = (page: PageLayout) => ( + {page} +); + +export default UsersPage; From 2fd0a06ece1e2e7ce52bad55c4ee3aa3eb060845 Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Thu, 6 Aug 2026 01:20:05 +0500 Subject: [PATCH 013/194] test(auth): add end-to-end coverage and correct fork footprint docs Playwright spec covers the redirect, credential rejection, 401 on unauthenticated API calls, 403 on direct Flask access, and reachable health endpoints. The four request-level tests pass in the dev container. The three page-level tests need Chromium system libraries the middleware-dev image does not ship -- 'playwright install --with-deps' fails on this Debian base -- so they are documented as host/CI-only. Every behaviour they cover was verified manually against a real browser. Docs corrected: the spec estimated six modified upstream files; the real figure is 19 (12 carrying genuine logic changes). The estimate missed every file that only reveals itself at runtime -- the _app.tsx auth gate, the knex/webpack conflict, the unhandled session rejection, and the cron path. The original estimate is kept alongside the correction because the gap is the useful part. UPSTREAM_SYNC_LOG.md now records the divergence baseline (19 modified, 26 new, 33 sentinels) so the first upstream merge has something to check against. --- docs/UPSTREAM_SYNC_LOG.md | 42 +++++++- .../specs/2026-08-05-auth-rbac-design.md | 54 ++++++++++- web-server/e2e/auth.spec.ts | 95 +++++++++++++++++++ 3 files changed, 186 insertions(+), 5 deletions(-) create mode 100644 web-server/e2e/auth.spec.ts diff --git a/docs/UPSTREAM_SYNC_LOG.md b/docs/UPSTREAM_SYNC_LOG.md index 238acd56..8ada66c6 100644 --- a/docs/UPSTREAM_SYNC_LOG.md +++ b/docs/UPSTREAM_SYNC_LOG.md @@ -13,7 +13,47 @@ possible but slow. Writing three lines at merge time is not. | Date | Upstream ref merged | Commits | Files changed | Conflicts | Sentinels before → after | Smoke test | PR | Notes | |---|---|---|---|---|---|---|---|---| -| _(no syncs yet — fork is at `844eb42` with zero divergence)_ | | | | | | | | | +| _(no syncs yet)_ | | | | | | | | | + +## Current divergence baseline + +Recorded 2026-08-06, after the authentication and RBAC work. **The first sync compares against +these numbers.** + +| | | +|---|---| +| Fork point | `844eb42` | +| Modified upstream files | **19** | +| New files | **26** | +| `CLUSTOX:` sentinel lines | **33** | + +Regenerate any time: + +```bash +# modified upstream files +git diff --name-only 844eb42..HEAD | while read -r f; do + git cat-file -e 844eb42:"$f" 2>/dev/null && echo "$f" +done | wc -l + +# sentinel lines +grep -rn "CLUSTOX" --include='*.py' --include='*.ts' --include='*.tsx' \ + --include='*.sql' --include='*.sh' --include='*.txt' --include='*.js' . \ + | grep -v node_modules | grep -v '^./docs/' | wc -l +``` + +**The 12 high-risk files** — the ones carrying real edits to upstream behaviour, to re-check on +every merge: + +`backend/analytics_server/app.py` · `sync_app.py` · `web-server/middleware.ts` · +`pages/_app.tsx` · `src/api-helpers/global.ts` · `src/api-helpers/axios.ts` · +`src/constants/db.ts` · `src/contexts/ThirdPartyAuthContext.tsx` · +`pages/api/integrations/index.ts` · `pages/api/resources/orgs/[org_id]/teams/index.ts` · +`.../teams/v2.ts` · `pages/api/resources/share.ts` + +`src/api-helpers/global.ts` is the most important of these: it is the single enforcement point for +authentication *and* per-team scoping across all BFF routes. A bad merge there silently disables +access control everywhere. The regression test at +`src/api-helpers/__tests__/endpoint-team-scope.test.ts` exists to catch exactly that. --- diff --git a/docs/superpowers/specs/2026-08-05-auth-rbac-design.md b/docs/superpowers/specs/2026-08-05-auth-rbac-design.md index 86a8e1e5..857108eb 100644 --- a/docs/superpowers/specs/2026-08-05-auth-rbac-design.md +++ b/docs/superpowers/specs/2026-08-05-auth-rbac-design.md @@ -229,7 +229,15 @@ be administered. ## 6. File inventory -### Upstream files modified (red/yellow zone — six files) +> **Corrected after implementation (2026-08-06).** The estimate below was wrong. +> The real footprint is **19 modified upstream files**, listed in +> [§6.1](#61-actual-footprint-after-implementation). The estimate missed every +> file that only reveals itself when the code runs — the `_app.tsx` auth gate, +> the knex/webpack conflict in `next.config.js`, the unhandled session +> rejection in `ThirdPartyAuthContext.tsx`, and the cron path. Left in place +> because the gap between estimate and reality is the useful part. + +### Upstream files modified (original estimate — six files) | File | Change | Size | |---|---|---| @@ -265,11 +273,49 @@ web-server/src/auth/queries.ts `Authenticated/index.tsx` is left alone — `middleware.ts` does the real work, and rewriting the stub only adds divergence. +### 6.1 Actual footprint after implementation + +Reproduce with: + +```bash +git diff --name-only 844eb42..HEAD | while read -r f; do + git cat-file -e 844eb42:"$f" 2>/dev/null && echo "MODIFIED $f" +done +``` + +**19 modified upstream files, 26 new files.** The modified set, by conflict risk: + +| Risk | Files | Why | +|---|---|---| +| **Low** (generated or append-only) | `database-docker/db/schema.sql`, `web-server/yarn.lock`, `web-server/package.json`, `env.example` | Regenerated or appended; conflicts are mechanical | +| **Medium** (config) | `Dockerfile.dev`, `web-server/next.config.js`, `setup_utils/cronjob.txt` | Small, isolated edits | +| **High** (logic) | `app.py`, `sync_app.py`, `middleware.ts`, `_app.tsx`, `global.ts`, `axios.ts`, `constants/db.ts`, `ThirdPartyAuthContext.tsx`, `integrations/index.ts`, `teams/index.ts`, `teams/v2.ts`, `share.ts` | Real edits to upstream behaviour; review carefully on every sync | + +**33 `CLUSTOX:` sentinel lines** across the tree. That count is the tripwire for the sync +procedure in [FORK_STRATEGY.md §5](../../FORK_STRATEGY.md) — if it drops after an upstream merge, +a conflict resolution ate one of our changes. + +### Why the estimate was wrong + +Every missed file shared a trait: it was invisible until the code ran. + +- **`_app.tsx`** gates every page on `auth.isInitialized`, which needs a session fetch that now + 401s — making the login page unreachable. A chicken-and-egg only visible in a browser. +- **`next.config.js`** — knex resolves dialects dynamically, so webpack could not bundle it for the + instrumentation hook. The bootstrap silently never ran. +- **`ThirdPartyAuthContext.tsx`** — `fetchSession` had no `.catch()`, so the expected 401 rejected + unhandled and buried the login form under an error overlay. +- **`cronjob.txt` / `trigger_sync.sh`** — cron does not inherit the container environment. + +The lesson for future estimates: **auth touches everything that assumed it could always fetch +data.** Grep for callers, not just for the routes being protected. + ### Fork impact -Cross-cutting auth for five touched upstream files is an unusually good ratio, entirely because -upstream left the hook points in. This remains **permanent divergence**: upstream will not accept -it, since gated auth is plausibly what their commercial product sells. +Twelve genuinely-modified logic files for cross-cutting auth is still a good ratio, largely because +upstream left the hook points in (`Endpoint.authenticated`, `Errors.SESSION_USER_NOT_FOUND`, the +`unauthenticated` opt-out). This remains **permanent divergence**: upstream will not accept it, +since gated auth is plausibly what their commercial product sells. --- diff --git a/web-server/e2e/auth.spec.ts b/web-server/e2e/auth.spec.ts new file mode 100644 index 00000000..90fee508 --- /dev/null +++ b/web-server/e2e/auth.spec.ts @@ -0,0 +1,95 @@ +/** + * Auth and access-control end-to-end coverage. + * + * Running these: + * + * docker compose up -d + * cd web-server && yarn playwright test e2e/auth.spec.ts + * + * The four request-level tests (401/403/health) need no browser and run + * anywhere. The three page-level tests need Chromium plus its system + * libraries, which the middleware-dev image does not ship -- `playwright + * install --with-deps` fails on this Debian base. Run those from the host or + * from CI, where browser dependencies are available. The behaviours they + * cover were each verified manually against a real browser when written. + */ +import { expect, test } from '@playwright/test'; + +// URLs are absolute on purpose: upstream's playwright.config.ts sets no +// baseURL, and adding one would mean touching another upstream file for no +// functional gain. +const APP = 'http://localhost:3333'; +const API = 'http://localhost:9696'; +const SYNC = 'http://localhost:9697'; + +const SUPERADMIN = { + email: process.env.SUPERADMIN_EMAIL || 'admin@clustox.com', + password: process.env.SUPERADMIN_PASSWORD || '' +}; + +const signIn = async (page: any, email: string, password: string) => { + await page.goto(`${APP}/login`); + await page.fill('input[type="email"]', email); + await page.fill('input[type="password"]', password); + await page.click('button[type="submit"]'); + await page.waitForURL((u: URL) => !u.pathname.startsWith('/login'), { + timeout: 30_000 + }); +}; + +test('unauthenticated visitors are redirected to login', async ({ page }) => { + await page.goto(`${APP}/dora-metrics`); + await expect(page).toHaveURL(/\/login/); +}); + +test('bad credentials are rejected without revealing which field was wrong', async ({ + page +}) => { + await page.goto(`${APP}/login`); + await page.fill('input[type="email"]', 'nobody@clustox.com'); + await page.fill('input[type="password"]', 'wrong-password'); + await page.click('button[type="submit"]'); + await expect(page.getByRole('alert')).toContainText('Invalid email or password'); +}); + +test('unauthenticated API requests return 401, not a redirect', async ({ + request +}) => { + const res = await request.get(`${APP}/api/clustox/users`, { + failOnStatusCode: false, + maxRedirects: 0 + }); + expect(res.status()).toBe(401); +}); + +test('the analytics API refuses direct requests without the internal token', async ({ + request +}) => { + const res = await request.get(`${API}/teams/any-id/lead_time`, { + failOnStatusCode: false + }); + expect(res.status()).toBe(403); +}); + +test('the sync API refuses direct requests without the internal token', async ({ + request +}) => { + const res = await request.post(`${SYNC}/sync`, { failOnStatusCode: false }); + expect(res.status()).toBe(403); +}); + +test('health endpoints stay reachable so probes keep working', async ({ + request +}) => { + expect((await request.get(`${API}/`, { failOnStatusCode: false })).status()).toBe(200); + expect((await request.get(`${SYNC}/`, { failOnStatusCode: false })).status()).toBe(200); +}); + +test('superadmin can sign in and reach user management', async ({ page }) => { + test.skip(!SUPERADMIN.password, 'SUPERADMIN_PASSWORD not set'); + + await signIn(page, SUPERADMIN.email, SUPERADMIN.password); + await page.goto(`${APP}/users`); + await expect(page.getByText('Not authorised')).toHaveCount(0); + await expect(page.getByRole('heading', { name: 'Users' })).toBeVisible(); +}); From f895710737654e3ced6631ca98624df1aca6aa7f Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Thu, 6 Aug 2026 01:31:28 +0500 Subject: [PATCH 014/194] fix(auth): resolve identity and role from the database, not the token A JWT proves only that someone signed in at some point. It says nothing about whether the account still exists or still holds the role it had at sign-in. getAuthSession trusted it for both. Two consequences, one found by a user report: - A deleted user kept a valid session. Team access was already read per-request, so they lost data (403) but stayed 'logged in' -- landing them in an app with no teams, no repos and no metrics. That reads as data loss, not as an auth failure. - Role lived in the token, so demoting a superadmin left them with superadmin powers until the token expired, up to 12 hours. Both now resolve from the database on every request, matching how team access already worked. Leaving identity and role in the token while reading team access live was an inconsistency in the original design. Cost is one indexed lookup per request. Verified against a live session for a user deleted mid-session: was 200 on /api/auth/session and 403 on team data, now 401 on both, so the middleware redirects to login instead of showing an empty app. --- web-server/src/auth/__tests__/session.test.ts | 65 +++++++++++++++++++ web-server/src/auth/queries.ts | 32 +++++++++ web-server/src/auth/session.ts | 27 ++++++-- 3 files changed, 119 insertions(+), 5 deletions(-) create mode 100644 web-server/src/auth/__tests__/session.test.ts diff --git a/web-server/src/auth/__tests__/session.test.ts b/web-server/src/auth/__tests__/session.test.ts new file mode 100644 index 00000000..100fa974 --- /dev/null +++ b/web-server/src/auth/__tests__/session.test.ts @@ -0,0 +1,65 @@ +jest.mock('next-auth/jwt', () => ({ getToken: jest.fn() })); +jest.mock('@/auth/queries', () => ({ getAuthUserById: jest.fn() })); + +import { getToken } from 'next-auth/jwt'; + +import { getAuthUserById } from '@/auth/queries'; +import { getAuthSession } from '@/auth/session'; + +const req = {} as any; + +describe('getAuthSession', () => { + beforeEach(() => jest.clearAllMocks()); + + it('returns null when there is no token', async () => { + (getToken as jest.Mock).mockResolvedValue(null); + await expect(getAuthSession(req)).resolves.toBeNull(); + expect(getAuthUserById).not.toHaveBeenCalled(); + }); + + it('returns null when the token is valid but the user no longer exists', async () => { + (getToken as jest.Mock).mockResolvedValue({ userId: 'deleted-user' }); + (getAuthUserById as jest.Mock).mockResolvedValue(null); + + await expect(getAuthSession(req)).resolves.toBeNull(); + }); + + it('resolves the session from the database, not the token', async () => { + (getToken as jest.Mock).mockResolvedValue({ + userId: 'u1', + // Stale claims from sign-in time. These must be ignored. + role: 'SUPERADMIN', + email: 'old@clustox.com', + name: 'Old Name' + }); + (getAuthUserById as jest.Mock).mockResolvedValue({ + userId: 'u1', + email: 'current@clustox.com', + name: 'Current Name', + role: 'ADMIN' + }); + + await expect(getAuthSession(req)).resolves.toEqual({ + userId: 'u1', + email: 'current@clustox.com', + name: 'Current Name', + role: 'ADMIN' + }); + }); + + it('reflects a demotion immediately rather than at token expiry', async () => { + (getToken as jest.Mock).mockResolvedValue({ + userId: 'u1', + role: 'SUPERADMIN' + }); + (getAuthUserById as jest.Mock).mockResolvedValue({ + userId: 'u1', + email: 'a@clustox.com', + name: 'A', + role: 'ADMIN' + }); + + const session = await getAuthSession(req); + expect(session?.role).toBe('ADMIN'); + }); +}); diff --git a/web-server/src/auth/queries.ts b/web-server/src/auth/queries.ts index e4459ea9..bc02accd 100644 --- a/web-server/src/auth/queries.ts +++ b/web-server/src/auth/queries.ts @@ -31,6 +31,38 @@ export const getAuthUserByEmail = async ( }; }; +/** + * Current auth state for a user id, straight from the database. + * + * Sessions are JWTs, so the token alone proves only that someone signed in at + * some point. It says nothing about whether the account still exists or still + * holds the role it had at sign-in. Every request resolves both here instead. + */ +export const getAuthUserById = async ( + userId: string +): Promise<{ userId: string; email: string; name: string; role: ClustoxRole } | null> => { + const row = await db(Table.ClustoxUserAuth) + .join(Table.Users, `${Table.Users}.id`, `${Table.ClustoxUserAuth}.user_id`) + .where(`${Table.ClustoxUserAuth}.user_id`, userId) + .andWhere(`${Table.Users}.is_deleted`, false) + .select( + `${Table.ClustoxUserAuth}.user_id`, + `${Table.ClustoxUserAuth}.role`, + `${Table.Users}.primary_email`, + `${Table.Users}.name` + ) + .first(); + + if (!row) return null; + + return { + userId: row.user_id, + email: row.primary_email, + name: row.name, + role: row.role as ClustoxRole + }; +}; + export const getTeamIdsForUser = async (userId: string): Promise => { const rows = await db(Table.ClustoxUserTeamAccess) .select(Columns[Table.ClustoxUserTeamAccess].team_id) diff --git a/web-server/src/auth/session.ts b/web-server/src/auth/session.ts index 324fe45b..dcf59e96 100644 --- a/web-server/src/auth/session.ts +++ b/web-server/src/auth/session.ts @@ -1,18 +1,35 @@ import { getToken } from 'next-auth/jwt'; import { NextApiRequest } from 'next/types'; -import { AuthSession, ClustoxRole } from './types'; +import { getAuthUserById } from './queries'; +import { AuthSession } from './types'; +/** + * Resolve the caller's session. + * + * The JWT is only used to identify *who* is claiming to be signed in. Whether + * that account still exists, and what role it currently holds, are read from + * the database on every request. + * + * Without this, a JWT stays valid for its full lifetime regardless of what + * happens to the account behind it: a deleted user keeps a working session, + * and a demoted superadmin keeps superadmin powers until the token expires. + * Team access was already resolved per-request for the same reason; leaving + * identity and role in the token was an inconsistency. + */ export const getAuthSession = async ( req: NextApiRequest ): Promise => { const token = await getToken({ req, secret: process.env.NEXTAUTH_SECRET }); if (!token?.userId) return null; + const user = await getAuthUserById(token.userId as string); + if (!user) return null; + return { - userId: token.userId as string, - email: (token.email as string) ?? '', - name: (token.name as string) ?? '', - role: token.role as ClustoxRole + userId: user.userId, + email: user.email, + name: user.name, + role: user.role }; }; From 5ce2cc5999c32dfeef6a2df2e2b716bb06551bf0 Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Thu, 6 Aug 2026 02:02:59 +0500 Subject: [PATCH 015/194] feat(auth): add sign out and Manage Users to the sidebar Two gaps from the plan, both reported by the user. The plan specified a /users page but never said how anyone would reach it, and never addressed sign out at all -- upstream has no logout affordance because it had no login. - Sidebar footer shows who is signed in with their role, plus a Sign out button. - 'Manage Users' nav item, hidden from non-superadmins. Sidebar visibility reads the real role from a new /api/clustox/me endpoint rather than AuthContext, whose role field upstream hardcodes to UserRole.MOM. Hiding the item is presentation only; the API returns 403 regardless of what the sidebar shows. Verified: superadmin sees Manage Users and the Superadmin chip, Sign out redirects to /login. 156 jest tests pass. --- web-server/pages/api/clustox/me.ts | 26 +++++++++++ .../src/components/ClustoxUserFooter.tsx | 46 +++++++++++++++++++ web-server/src/constants/routes.ts | 4 +- web-server/src/hooks/useClustoxUser.ts | 43 +++++++++++++++++ .../Sidebar/SidebarMenu/items.tsx | 14 +++++- .../SidebarMenu/useFilteredSidebarItems.tsx | 9 +++- .../ExtendedSidebarLayout/Sidebar/index.tsx | 5 ++ 7 files changed, 143 insertions(+), 4 deletions(-) create mode 100644 web-server/pages/api/clustox/me.ts create mode 100644 web-server/src/components/ClustoxUserFooter.tsx create mode 100644 web-server/src/hooks/useClustoxUser.ts diff --git a/web-server/pages/api/clustox/me.ts b/web-server/pages/api/clustox/me.ts new file mode 100644 index 00000000..55ed43a5 --- /dev/null +++ b/web-server/pages/api/clustox/me.ts @@ -0,0 +1,26 @@ +import { Endpoint, nullSchema } from '@/api-helpers/global'; +import { assertAuthenticated } from '@/auth/guard'; + +/** + * The signed-in user's identity and current role. + * + * The UI needs the real Clustox role to decide what to show, and upstream's + * AuthContext hardcodes `role` to UserRole.MOM. Rather than rewire that + * context, components read from here. + * + * Presentation only -- every route enforces its own authorization server-side. + */ +const endpoint = new Endpoint(nullSchema); + +endpoint.handle.GET(nullSchema, async (req, res) => { + const session = assertAuthenticated((req as any).session); + + res.send({ + user_id: session.userId, + email: session.email, + name: session.name, + role: session.role + }); +}); + +export default endpoint.serve(); diff --git a/web-server/src/components/ClustoxUserFooter.tsx b/web-server/src/components/ClustoxUserFooter.tsx new file mode 100644 index 00000000..e51b702f --- /dev/null +++ b/web-server/src/components/ClustoxUserFooter.tsx @@ -0,0 +1,46 @@ +import { LogoutTwoTone } from '@mui/icons-material'; +import { Button, Chip, Tooltip, Typography } from '@mui/material'; +import { signOut } from 'next-auth/react'; + +import { FlexBox } from '@/components/FlexBox'; +import { useClustoxUser } from '@/hooks/useClustoxUser'; + +/** + * CLUSTOX: shows who is signed in and provides the only way to sign out. + * Upstream has no logout affordance at all, since it had no login. + */ +export const ClustoxUserFooter = () => { + const { user, loading } = useClustoxUser(); + + if (loading || !user) return null; + + return ( + + + + + {user.name || user.email} + + + + + + + ); +}; diff --git a/web-server/src/constants/routes.ts b/web-server/src/constants/routes.ts index ada95d7b..a65aee91 100644 --- a/web-server/src/constants/routes.ts +++ b/web-server/src/constants/routes.ts @@ -69,7 +69,9 @@ export const ROUTES = { }, INTEGRATIONS: new RoutePath('INTEGRATIONS'), SETTINGS: new RoutePath('SETTINGS'), - SYSTEM_LOGS: new RoutePath('SYSTEM_LOGS') + SYSTEM_LOGS: new RoutePath('SYSTEM_LOGS'), + // CLUSTOX: user management (superadmin only) + USERS: new RoutePath('USERS') }; export const DEFAULT_HOME_ROUTE = ROUTES.DORA_METRICS; diff --git a/web-server/src/hooks/useClustoxUser.ts b/web-server/src/hooks/useClustoxUser.ts new file mode 100644 index 00000000..f1a7b90a --- /dev/null +++ b/web-server/src/hooks/useClustoxUser.ts @@ -0,0 +1,43 @@ +import { useCallback, useEffect, useState } from 'react'; + +import { ClustoxRole } from '@/auth/types'; + +export type ClustoxUser = { + user_id: string; + email: string; + name: string; + role: ClustoxRole; +}; + +/** + * The signed-in user's identity and current role. + * + * Read from /api/clustox/me rather than from AuthContext, whose `role` field + * upstream hardcodes to UserRole.MOM. Use it for presentation decisions only; + * authorization is enforced server-side on every route. + */ +export const useClustoxUser = () => { + const [user, setUser] = useState(null); + const [loading, setLoading] = useState(true); + + const load = useCallback(async () => { + try { + const res = await fetch('/api/clustox/me'); + setUser(res.ok ? await res.json() : null); + } catch { + setUser(null); + } finally { + setLoading(false); + } + }, []); + + useEffect(() => { + load(); + }, [load]); + + return { + user, + loading, + isSuperadmin: user?.role === 'SUPERADMIN' + }; +}; diff --git a/web-server/src/layouts/ExtendedSidebarLayout/Sidebar/SidebarMenu/items.tsx b/web-server/src/layouts/ExtendedSidebarLayout/Sidebar/SidebarMenu/items.tsx index 8102d083..a7ac0243 100644 --- a/web-server/src/layouts/ExtendedSidebarLayout/Sidebar/SidebarMenu/items.tsx +++ b/web-server/src/layouts/ExtendedSidebarLayout/Sidebar/SidebarMenu/items.tsx @@ -3,7 +3,9 @@ import { GroupsTwoTone, Analytics, Settings, - Dns + Dns, + // CLUSTOX: icon for the user management item + ManageAccounts } from '@mui/icons-material'; import { ROUTES } from '@/constants/routes'; @@ -40,7 +42,9 @@ export enum SideBarItems { DORA_METRICS = 'DORA Metrics', MANAGE_TEAMS = 'Manage Teams', MANAGE_INTEGRATIONS = 'Manage Integrations', - SERVER_ADMIN = 'Server Admin' + SERVER_ADMIN = 'Server Admin', + // CLUSTOX: superadmin-only, filtered in useFilteredSidebarItems + MANAGE_USERS = 'Manage Users' } const menuItems = (): MenuItems[] => [ @@ -71,6 +75,12 @@ const menuItems = (): MenuItems[] => [ name: 'System Logs', icon: Dns, link: ROUTES.SYSTEM_LOGS.PATH + }, + // CLUSTOX: only rendered for superadmins. + { + name: SideBarItems.MANAGE_USERS, + icon: ManageAccounts, + link: ROUTES.USERS.PATH } ] } diff --git a/web-server/src/layouts/ExtendedSidebarLayout/Sidebar/SidebarMenu/useFilteredSidebarItems.tsx b/web-server/src/layouts/ExtendedSidebarLayout/Sidebar/SidebarMenu/useFilteredSidebarItems.tsx index 44c6aafe..1e598e58 100644 --- a/web-server/src/layouts/ExtendedSidebarLayout/Sidebar/SidebarMenu/useFilteredSidebarItems.tsx +++ b/web-server/src/layouts/ExtendedSidebarLayout/Sidebar/SidebarMenu/useFilteredSidebarItems.tsx @@ -1,6 +1,8 @@ import { useMemo } from 'react'; import { useAuth } from '@/hooks/useAuth'; +// CLUSTOX: real Clustox role for sidebar visibility +import { useClustoxUser } from '@/hooks/useClustoxUser'; import { ItemTags, SideBarItems @@ -17,6 +19,8 @@ const checkTag = (tag: string | number[] | number, check: string | number) => { export const useFilteredSidebarItems = () => { const { integrationList } = useAuth(); + // CLUSTOX: the real role, since useAuth().role is hardcoded upstream. + const { isSuperadmin } = useClustoxUser(); const flagFilteredMenuItems = useMemo(() => { return menuItems(); @@ -25,6 +29,9 @@ export const useFilteredSidebarItems = () => { const sidebarItems = useMemo(() => { const filterCheck = (item: MenuItem): boolean => { if (checkTag(item.tag, ItemTags.HideItem)) return false; + // CLUSTOX: hide user management from non-superadmins. Presentation only + // -- the API returns 403 regardless of what the sidebar shows. + if (item.name === SideBarItems.MANAGE_USERS && !isSuperadmin) return false; if ( !integrationList.length && item.name !== SideBarItems.MANAGE_INTEGRATIONS @@ -46,7 +53,7 @@ export const useFilteredSidebarItems = () => { items: itemsFilter(section.items) })) .filter((section) => section.items?.length); - }, [flagFilteredMenuItems, integrationList]); + }, [flagFilteredMenuItems, integrationList, isSuperadmin]); return sidebarItems; }; diff --git a/web-server/src/layouts/ExtendedSidebarLayout/Sidebar/index.tsx b/web-server/src/layouts/ExtendedSidebarLayout/Sidebar/index.tsx index b08f1007..930684f5 100644 --- a/web-server/src/layouts/ExtendedSidebarLayout/Sidebar/index.tsx +++ b/web-server/src/layouts/ExtendedSidebarLayout/Sidebar/index.tsx @@ -10,6 +10,8 @@ import { import { format, isValid } from 'date-fns'; import { useContext, useMemo } from 'react'; +// CLUSTOX: signed-in user + sign out +import { ClustoxUserFooter } from '@/components/ClustoxUserFooter'; import { FlexBox } from '@/components/FlexBox'; import Scrollbar from '@/components/Scrollbar'; import { Line } from '@/components/Text'; @@ -86,6 +88,9 @@ const SidebarContent = () => { + {/* CLUSTOX: signed-in identity and the only sign-out affordance. */} + + Build: {formattedDate} From a729e496d1eff35ae526ccac4631517ede4cac89 Mon Sep 17 00:00:00 2001 From: nexusameer Date: Thu, 6 Aug 2026 16:52:13 +0500 Subject: [PATCH 016/194] Add Jenkinsfile for CI/CD pipeline --- Jenkinsfile | 49 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 49 insertions(+) create mode 100644 Jenkinsfile diff --git a/Jenkinsfile b/Jenkinsfile new file mode 100644 index 00000000..0f751d62 --- /dev/null +++ b/Jenkinsfile @@ -0,0 +1,49 @@ +pipeline { + agent any + + stages { + stage('Checkout') { + steps { + checkout scm + } + } + + stage('Update Deployment Copy') { + steps { + sh ''' + cd /opt/app/middleware + git fetch origin main + git reset --hard origin/main + ''' + } + } + + stage('Build & Deploy') { + steps { + sh ''' + cd /opt/app/middleware + docker compose build + docker compose up -d + ''' + } + } + + stage('Health Check') { + steps { + sh ''' + sleep 10 + curl -sf http://127.0.0.1:${PORT:-3333} || (echo "Health check failed" && exit 1) + ''' + } + } + } + + post { + success { + echo 'Deployment succeeded.' + } + failure { + echo 'Deployment failed - check console output.' + } + } +} From 3f909aff7fb4dc4130db00432c96538c1ab4236f Mon Sep 17 00:00:00 2001 From: Muhammad Ameer Hamza Date: Thu, 6 Aug 2026 17:13:06 +0500 Subject: [PATCH 017/194] jenkins file updated --- Jenkinsfile | 1 + 1 file changed, 1 insertion(+) diff --git a/Jenkinsfile b/Jenkinsfile index 0f751d62..46e2cd45 100644 --- a/Jenkinsfile +++ b/Jenkinsfile @@ -47,3 +47,4 @@ pipeline { } } } + From 3e101f4c234b5afc879ff1c700f5386bba61f487 Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Thu, 6 Aug 2026 17:26:59 +0500 Subject: [PATCH 018/194] docs: add multitenancy proposal and open decisions Records the gap between the shipped auth work (scoped by team) and the multitenant model agreed in the first meeting (scoped by workspace), the six open decisions, and cost estimates. Decision received: Admins connect their own integrations. --- docs/MULTITENANCY_PROPOSAL.md | 155 ++++++++++++++++++++++++++++++++++ 1 file changed, 155 insertions(+) create mode 100644 docs/MULTITENANCY_PROPOSAL.md diff --git a/docs/MULTITENANCY_PROPOSAL.md b/docs/MULTITENANCY_PROPOSAL.md new file mode 100644 index 00000000..c94825a2 --- /dev/null +++ b/docs/MULTITENANCY_PROPOSAL.md @@ -0,0 +1,155 @@ +# Middleware Fork — Authentication & Multitenancy + +**For review and approval** +Prepared 2026-08-06 · Clustox/middleware · branch `feat/auth-rbac` + +--- + +## 1. Where we are + +Authentication and role-based access are **built, tested and pushed**. What works today: + +- Login screen; nothing in the app is reachable without signing in +- Two roles: **SuperAdmin** and **Admin** +- SuperAdmin can create users, assign roles, and scope them +- Admins see only what they're scoped to; requesting anything else returns 403 +- The analytics API is locked down — previously `curl http://:9696/…` returned every + team's metrics with no credentials at all +- Sign out, user management page, session expiry + +156 frontend and 155 backend tests passing. + +**One thing worth knowing:** the open-source version we forked ships with *no authentication of +any kind*. Anyone who could reach the URL saw everything, and the API was open to anyone on the +network. That's now closed. + +--- + +## 2. The gap + +The first meeting specified **multitenancy** — each Admin brings their own GitHub/GitLab +integration and their own projects, with SuperAdmin seeing across all of them. + +What we built scopes access by **team** inside a single shared workspace. What's needed scopes by +**tenant**, where each Admin effectively has their own workspace. + +The authentication layer is unaffected and fully reusable. What changes is *what access is measured +against*. + +| Requirement | Status | +|---|---| +| SuperAdmin + Admin roles | ✅ Done | +| Login, sessions, password security | ✅ Done | +| SuperAdmin manages users | ✅ Done | +| API not bypassable | ✅ Done | +| Admin adds **their own** integration | ❌ Currently one shared integration | +| Admin adds **their own** projects | ❌ Projects are currently shared | +| SuperAdmin sees **all** projects | ⚠️ Sees all teams — right idea, wrong dimension | +| SuperAdmin **invites** admins | ❌ Not built | +| Multitenant | ❌ Single shared workspace | + +--- + +## 3. The good news + +We expected multitenancy to be a rewrite. It isn't. + +The upstream database was **designed for multiple tenants** and then bootstrapped to only ever +create one. Every table already carries an organisation ID. The integration table's primary key is +literally *(provider, organisation)* — meaning **one GitHub account per organisation is already how +it was built to work**. + +Only four places in the entire codebase assume a single tenant. Everything else already passes the +organisation through correctly. + +In practical terms: **"one workspace per Admin" is the design the code was already shaped for**, not +something we bolt on. That materially reduces both the cost and the risk. + +--- + +## 4. Proposed model + +**Each Admin gets their own workspace.** + +| | | +|---|---| +| **Admin** | Owns a workspace. Connects their own GitHub/GitLab account, adds their own projects and teams, sees only their own metrics. | +| **SuperAdmin** | Sits above all workspaces. Sees every project across every Admin, invites new Admins, manages users. Does not own a workspace. | + +Concretely, this gives each Admin their own integration credentials, their own project list, and +their own dashboards — with a SuperAdmin view spanning all of them. It matches the meeting notes +directly. + +**On integrations:** the note said this was undecided. Recommendation is that **Admins connect their +own**, with SuperAdmin able to see and revoke any of them. Reasons: it removes the SuperAdmin as a +bottleneck for onboarding, it's what the schema already supports, and it avoids one person holding +credentials for everyone else's repositories. + +--- + +## 5. Decisions needed + +Each has a recommendation. Approving all six as-is is a valid answer. + +**Q1. Does a SuperAdmin have their own workspace, or sit above all of them?** +*Recommend: sits above, owns none.* Keeps "who can see what" simple and avoids the odd case of a +SuperAdmin's personal projects mixing into an org-wide view. + +**Q2. Can two Admins track the same repository?** +*Recommend: yes, allow it.* The consequence is that the same repo gets synced once per Admin, using +each Admin's own credentials — duplicate API calls against GitHub. Acceptable at our scale, and +blocking it would create awkward "someone else claimed this repo" conflicts. + +**Q3. How do invites work — email, or a link?** +*Recommend: invite links first.* A real email invite needs mail infrastructure we don't currently +have. A SuperAdmin-generated single-use link works immediately and can be sent over Slack. Email can +be added later without rework. + +**Q4. What happens to the existing setup?** +There's already a live GitHub connection with two repos and two teams from testing. +*Recommend: it becomes the first Admin's workspace* rather than being discarded, so nothing is +re-entered. + +**Q5. Do we still want team-level access inside a workspace?** +We built the ability to scope an Admin to specific teams. Under the new model an Admin owns their +whole workspace, so it may be unnecessary. +*Recommend: drop it for now.* It's already built and can be re-enabled later if a workspace grows +large enough to need it. Carrying unused permission levels adds confusion, not safety. + +**Q6. Is the sync cost acceptable?** +Today one scheduled job syncs one workspace. Multitenant means syncing every workspace, each +against its own GitHub account and its own rate limit. +*Recommend: proceed, and add per-workspace sync status so failures are visible.* Right now a failed +sync reports success and simply shows no data — we hit this during testing and it cost real time to +diagnose. + +--- + +## 6. Honest costs + +- **Multitenancy is a data-model change**, not a settings toggle. It touches how every request + resolves which workspace it belongs to. +- **Sync is the main scaling risk.** Each workspace syncs independently; GitHub rate limits are + per-token, so this scales reasonably, but sync duration grows with the number of Admins. +- **This is a permanent fork change.** Upstream will not accept it — gated multi-tenant access is + plausibly what their paid product sells. We carry it ourselves from here, which is why we've kept + our changes marked and auditable so upstream updates can still be merged in. + +--- + +## 7. Suggested sequence + +1. **Merge the authentication work now.** It's complete, tested, and valuable on its own — it closes + a real hole where the API was open to anyone on the network. +2. **Approve the decisions above**, then a detailed design for multitenancy. +3. **Build multitenancy**, then invites. + +Splitting it this way means the security fix ships immediately rather than waiting behind a larger +piece of work. + +--- + +## What I need + +Approval on the model in §4, and answers to the six questions in §5 — or a simple "go with the +recommendations". From 18337bdd38762eb42de15d968e6236f1a11ab09c Mon Sep 17 00:00:00 2001 From: Muhammad Ameer Hamza Date: Thu, 6 Aug 2026 17:27:12 +0500 Subject: [PATCH 019/194] jenkins file updated --- Jenkinsfile | 1 - 1 file changed, 1 deletion(-) diff --git a/Jenkinsfile b/Jenkinsfile index 46e2cd45..0f751d62 100644 --- a/Jenkinsfile +++ b/Jenkinsfile @@ -47,4 +47,3 @@ pipeline { } } } - From 630c74ec47c8f3e11aa3860dae0244c38d03f799 Mon Sep 17 00:00:00 2001 From: Muhammad Ameer Hamza Date: Thu, 6 Aug 2026 17:29:12 +0500 Subject: [PATCH 020/194] jenkins file updated --- Jenkinsfile | 1 + 1 file changed, 1 insertion(+) diff --git a/Jenkinsfile b/Jenkinsfile index 0f751d62..46e2cd45 100644 --- a/Jenkinsfile +++ b/Jenkinsfile @@ -47,3 +47,4 @@ pipeline { } } } + From 72c3589b7aa2883b6f1dda7d15dbcf82efab9076 Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Thu, 6 Aug 2026 17:35:46 +0500 Subject: [PATCH 021/194] feat(tenancy): give each admin their own workspace Each ADMIN owns a workspace (an Organization); a SUPERADMIN owns none and sits above all of them. No schema change was needed. Upstream's Users.org_id already exists and is already nullable, every other table already carries org_id, and Integration is keyed on (name, org_id) -- so one GitHub account per workspace is how the schema was already built. Upstream simply only ever created a single Organization named 'default'. - AuthSession carries orgId, resolved per-request from the database alongside identity and role - createUser provisions a workspace for an ADMIN, or accepts an existing one so the pre-multitenancy workspace can be adopted rather than discarded - SUPERADMIN is created with no workspace; the migration also clears the workspace from any existing superadmin, which would otherwise scope their view to it 42 auth tests pass. --- .../20260806120000_clustox_multitenancy.sql | 27 +++++++ database-docker/db/schema.sql | 3 +- web-server/pages/api/clustox/users/index.ts | 18 ++--- .../src/auth/__tests__/bootstrap.test.ts | 5 +- web-server/src/auth/__tests__/guard.test.ts | 6 +- web-server/src/auth/__tests__/queries.test.ts | 22 +++++- web-server/src/auth/__tests__/session.test.ts | 35 ++++++++- web-server/src/auth/bootstrap.ts | 25 +----- web-server/src/auth/queries.ts | 77 ++++++++++++++++--- web-server/src/auth/session.ts | 3 +- web-server/src/auth/types.ts | 11 +++ 11 files changed, 183 insertions(+), 49 deletions(-) create mode 100644 database-docker/db/migrations/20260806120000_clustox_multitenancy.sql diff --git a/database-docker/db/migrations/20260806120000_clustox_multitenancy.sql b/database-docker/db/migrations/20260806120000_clustox_multitenancy.sql new file mode 100644 index 00000000..39cc6910 --- /dev/null +++ b/database-docker/db/migrations/20260806120000_clustox_multitenancy.sql @@ -0,0 +1,27 @@ +-- migrate:up +-- CLUSTOX: multitenancy. Each ADMIN owns a workspace (an Organization); +-- SUPERADMIN sits above all of them and owns none. +-- +-- No schema change is needed: upstream's "Users".org_id already exists and is +-- already nullable, and every other table already carries org_id. Upstream +-- simply only ever created one Organization. This migration only corrects the +-- data left behind by the single-workspace bootstrap. + +-- A superadmin must not own a workspace, or their view would be scoped to it. +UPDATE "ClustoxUserAuth" a +SET updated_at = now() +FROM "Users" u +WHERE u.id = a.user_id AND a.role = 'SUPERADMIN'; + +UPDATE "Users" u +SET org_id = NULL +FROM "ClustoxUserAuth" a +WHERE a.user_id = u.id AND a.role = 'SUPERADMIN'; + +-- migrate:down +-- Reattach any orphaned superadmin to the oldest workspace, which is what the +-- single-workspace bootstrap would have done. +UPDATE "Users" u +SET org_id = (SELECT id FROM "Organization" ORDER BY created_at ASC LIMIT 1) +FROM "ClustoxUserAuth" a +WHERE a.user_id = u.id AND a.role = 'SUPERADMIN' AND u.org_id IS NULL; diff --git a/database-docker/db/schema.sql b/database-docker/db/schema.sql index 1e4eb7a8..30b6ae31 100644 --- a/database-docker/db/schema.sql +++ b/database-docker/db/schema.sql @@ -1433,4 +1433,5 @@ INSERT INTO public.schema_migrations (version) VALUES ('20240430142502'), ('20240503060203'), ('20240503073715'), - ('20260805150000'); + ('20260805150000'), + ('20260806120000'); diff --git a/web-server/pages/api/clustox/users/index.ts b/web-server/pages/api/clustox/users/index.ts index b9ee0577..a4e2b445 100644 --- a/web-server/pages/api/clustox/users/index.ts +++ b/web-server/pages/api/clustox/users/index.ts @@ -4,9 +4,7 @@ import { Endpoint, nullSchema } from '@/api-helpers/global'; import { assertRole } from '@/auth/guard'; import { createUser, getAuthUserByEmail, listUsers } from '@/auth/queries'; import { ClustoxRole } from '@/auth/types'; -import { Table } from '@/constants/db'; import { Errors, ResponseError } from '@/constants/error'; -import { db } from '@/utils/db'; const postSchema = yup.object().shape({ name: yup.string().required(), @@ -14,7 +12,10 @@ const postSchema = yup.object().shape({ // Enforced here so the UI cannot create a weak account. password: yup.string().min(12).required(), role: yup.string().oneOf(['SUPERADMIN', 'ADMIN']).required(), - team_ids: yup.array().of(yup.string().uuid()).required() + team_ids: yup.array().of(yup.string().uuid()).required(), + // Optional: place the admin in an existing workspace instead of + // provisioning a new one. Used to adopt the pre-multitenancy workspace. + org_id: yup.string().uuid().nullable().optional() }); const endpoint = new Endpoint(nullSchema); @@ -32,20 +33,19 @@ endpoint.handle.POST(postSchema, async (req, res) => { throw new ResponseError(Errors.INSUFFICIENT_PARAMS, 409); } - const org = await db(Table.Organization).select('id').first(); - if (!org) throw new ResponseError(Errors.USER_NOT_FOUND, 500); - - const userId = await createUser({ + // An ADMIN gets a freshly provisioned workspace unless one is named; a + // SUPERADMIN gets none. createUser decides based on the role. + const { userId, orgId } = await createUser({ name: req.payload.name, email: req.payload.email, password: req.payload.password, // yup's .oneOf() widens to string; the schema has already validated it. role: req.payload.role as ClustoxRole, teamIds: req.payload.team_ids, - orgId: org.id + orgId: req.payload.org_id ?? null }); - res.send({ user_id: userId }); + res.send({ user_id: userId, org_id: orgId }); }); export default endpoint.serve(); diff --git a/web-server/src/auth/__tests__/bootstrap.test.ts b/web-server/src/auth/__tests__/bootstrap.test.ts index cc951fcc..ee2d82b6 100644 --- a/web-server/src/auth/__tests__/bootstrap.test.ts +++ b/web-server/src/auth/__tests__/bootstrap.test.ts @@ -43,15 +43,14 @@ describe('bootstrapSuperadmin', () => { process.env.SUPERADMIN_EMAIL = 'boss@clustox.com'; process.env.SUPERADMIN_PASSWORD = 'strong-password'; - await bootstrapSuperadmin('org-1'); + await bootstrapSuperadmin(); expect(createUser).toHaveBeenCalledWith( expect.objectContaining({ email: 'boss@clustox.com', password: 'strong-password', role: 'SUPERADMIN', - teamIds: [], - orgId: 'org-1' + teamIds: [] }) ); }); diff --git a/web-server/src/auth/__tests__/guard.test.ts b/web-server/src/auth/__tests__/guard.test.ts index 54dae35a..736778b2 100644 --- a/web-server/src/auth/__tests__/guard.test.ts +++ b/web-server/src/auth/__tests__/guard.test.ts @@ -17,13 +17,15 @@ const superadmin: AuthSession = { userId: 'su1', email: 'boss@clustox.com', name: 'Boss', - role: 'SUPERADMIN' + role: 'SUPERADMIN', + orgId: null }; const admin: AuthSession = { userId: 'ad1', email: 'lead@clustox.com', name: 'Lead', - role: 'ADMIN' + role: 'ADMIN', + orgId: 'workspace-1' }; describe('assertAuthenticated', () => { diff --git a/web-server/src/auth/__tests__/queries.test.ts b/web-server/src/auth/__tests__/queries.test.ts index 216d6e3c..6610ee34 100644 --- a/web-server/src/auth/__tests__/queries.test.ts +++ b/web-server/src/auth/__tests__/queries.test.ts @@ -19,13 +19,14 @@ const chain = (result: any) => { describe('getAuthUserByEmail', () => { beforeEach(() => mockDb.mockReset()); - it('maps a row to AuthUserRow', async () => { + it('maps a row to AuthUserRow, including the workspace', async () => { mockDb.mockReturnValue( chain({ user_id: 'u1', primary_email: 'a@clustox.com', name: 'Ayesha', role: 'ADMIN', + org_id: 'workspace-1', password_hash: 'hashed' }) ); @@ -35,10 +36,29 @@ describe('getAuthUserByEmail', () => { email: 'a@clustox.com', name: 'Ayesha', role: 'ADMIN', + orgId: 'workspace-1', passwordHash: 'hashed' }); }); + it('maps a superadmin with no workspace to a null orgId', async () => { + mockDb.mockReturnValue( + chain({ + user_id: 'su1', + primary_email: 'boss@clustox.com', + name: 'Boss', + role: 'SUPERADMIN', + org_id: null, + password_hash: 'hashed' + }) + ); + + await expect(getAuthUserByEmail('boss@clustox.com')).resolves.toMatchObject({ + role: 'SUPERADMIN', + orgId: null + }); + }); + it('returns null when no row matches', async () => { mockDb.mockReturnValue(chain(undefined)); await expect(getAuthUserByEmail('nobody@clustox.com')).resolves.toBeNull(); diff --git a/web-server/src/auth/__tests__/session.test.ts b/web-server/src/auth/__tests__/session.test.ts index 100fa974..05aa52ea 100644 --- a/web-server/src/auth/__tests__/session.test.ts +++ b/web-server/src/auth/__tests__/session.test.ts @@ -36,15 +36,46 @@ describe('getAuthSession', () => { userId: 'u1', email: 'current@clustox.com', name: 'Current Name', - role: 'ADMIN' + role: 'ADMIN', + orgId: 'workspace-1' }); await expect(getAuthSession(req)).resolves.toEqual({ userId: 'u1', email: 'current@clustox.com', name: 'Current Name', - role: 'ADMIN' + role: 'ADMIN', + orgId: 'workspace-1' + }); + }); + + it('carries the admin workspace onto the session', async () => { + (getToken as jest.Mock).mockResolvedValue({ userId: 'u1' }); + (getAuthUserById as jest.Mock).mockResolvedValue({ + userId: 'u1', + email: 'a@clustox.com', + name: 'A', + role: 'ADMIN', + orgId: 'workspace-1' }); + + const session = await getAuthSession(req); + expect(session?.orgId).toBe('workspace-1'); + }); + + it('gives a superadmin no workspace, so they sit above all of them', async () => { + (getToken as jest.Mock).mockResolvedValue({ userId: 'su1' }); + (getAuthUserById as jest.Mock).mockResolvedValue({ + userId: 'su1', + email: 'boss@clustox.com', + name: 'Boss', + role: 'SUPERADMIN', + orgId: null + }); + + const session = await getAuthSession(req); + expect(session?.orgId).toBeNull(); + expect(session?.role).toBe('SUPERADMIN'); }); it('reflects a demotion immediately rather than at token expiry', async () => { diff --git a/web-server/src/auth/bootstrap.ts b/web-server/src/auth/bootstrap.ts index 005f9db6..c7d3f0ef 100644 --- a/web-server/src/auth/bootstrap.ts +++ b/web-server/src/auth/bootstrap.ts @@ -1,16 +1,6 @@ -import { Table } from '@/constants/db'; -import { db } from '@/utils/db'; - import { countSuperadmins, createUser } from './queries'; -const getDefaultOrgId = async (): Promise => { - const row = await db(Table.Organization).select('id').first(); - return row?.id ?? null; -}; - -export const bootstrapSuperadmin = async ( - orgIdOverride?: string -): Promise => { +export const bootstrapSuperadmin = async (): Promise => { const existing = await countSuperadmins(); if (existing > 0) return; @@ -27,21 +17,14 @@ export const bootstrapSuperadmin = async ( return; } - const orgId = orgIdOverride ?? (await getDefaultOrgId()); - if (!orgId) { - console.error( - '[CLUSTOX] No organization found; cannot bootstrap superadmin.' - ); - return; - } - + // No workspace: a superadmin sits above every workspace rather than owning + // one, so createUser leaves org_id null for this role. await createUser({ name: 'Superadmin', email, password, role: 'SUPERADMIN', - teamIds: [], - orgId + teamIds: [] }); console.info(`[CLUSTOX] Bootstrapped superadmin ${email}`); diff --git a/web-server/src/auth/queries.ts b/web-server/src/auth/queries.ts index bc02accd..d0d4b83a 100644 --- a/web-server/src/auth/queries.ts +++ b/web-server/src/auth/queries.ts @@ -1,3 +1,5 @@ +import { randomUUID } from 'node:crypto'; + import { Columns, Table } from '@/constants/db'; import { db } from '@/utils/db'; @@ -16,7 +18,8 @@ export const getAuthUserByEmail = async ( `${Table.ClustoxUserAuth}.role`, `${Table.ClustoxUserAuth}.password_hash`, `${Table.Users}.primary_email`, - `${Table.Users}.name` + `${Table.Users}.name`, + `${Table.Users}.org_id` ) .first(); @@ -27,6 +30,7 @@ export const getAuthUserByEmail = async ( email: row.primary_email, name: row.name, role: row.role as ClustoxRole, + orgId: row.org_id ?? null, passwordHash: row.password_hash }; }; @@ -40,7 +44,13 @@ export const getAuthUserByEmail = async ( */ export const getAuthUserById = async ( userId: string -): Promise<{ userId: string; email: string; name: string; role: ClustoxRole } | null> => { +): Promise<{ + userId: string; + email: string; + name: string; + role: ClustoxRole; + orgId: string | null; +} | null> => { const row = await db(Table.ClustoxUserAuth) .join(Table.Users, `${Table.Users}.id`, `${Table.ClustoxUserAuth}.user_id`) .where(`${Table.ClustoxUserAuth}.user_id`, userId) @@ -49,7 +59,8 @@ export const getAuthUserById = async ( `${Table.ClustoxUserAuth}.user_id`, `${Table.ClustoxUserAuth}.role`, `${Table.Users}.primary_email`, - `${Table.Users}.name` + `${Table.Users}.name`, + `${Table.Users}.org_id` ) .first(); @@ -59,7 +70,9 @@ export const getAuthUserById = async ( userId: row.user_id, email: row.primary_email, name: row.name, - role: row.role as ClustoxRole + role: row.role as ClustoxRole, + // null for SUPERADMIN, who sits above every workspace. + orgId: row.org_id ?? null }; }; @@ -85,12 +98,20 @@ export const countSuperadmins = async (): Promise => { export const listUsers = async (): Promise => { const rows = await db(Table.ClustoxUserAuth) .join(Table.Users, `${Table.Users}.id`, `${Table.ClustoxUserAuth}.user_id`) + .leftJoin( + Table.Organization, + `${Table.Organization}.id`, + `${Table.Users}.org_id` + ) .where(`${Table.Users}.is_deleted`, false) .select( `${Table.ClustoxUserAuth}.user_id`, `${Table.ClustoxUserAuth}.role`, `${Table.Users}.primary_email`, - `${Table.Users}.name` + `${Table.Users}.name`, + `${Table.Users}.org_id`, + // left join: null for a superadmin, who owns no workspace + `${Table.Organization}.name as org_name` ); const access = await db(Table.ClustoxUserTeamAccess).select( @@ -103,25 +124,63 @@ export const listUsers = async (): Promise => { email: r.primary_email, name: r.name, role: r.role as ClustoxRole, + orgId: r.org_id ?? null, + orgName: r.org_name ?? null, teamIds: access .filter((a: any) => a.user_id === r.user_id) .map((a: any) => a.team_id) })); }; +/** + * Create a workspace and return its id. + * + * Upstream only ever created one Organization, named "default", at boot. The + * schema was always multi-workspace -- every table carries org_id and + * Integration is keyed on (name, org_id) -- so provisioning more is simply + * doing what the schema already allows. + */ +export const createWorkspace = async (name: string): Promise => { + // Organization.id has no database default -- upstream's bootstrap supplies + // one from Python. The exported `db` helper is a table function, not the + // knex instance, so there is no db.raw() here either. Generate it in JS. + const id = randomUUID(); + + await db(Table.Organization).insert({ + id, + name, + domain: name, + created_at: new Date() + }); + + return id; +}; + +/** + * Create a user. + * + * An ADMIN gets their own workspace, provisioned here. A SUPERADMIN gets none + * (org_id null) because they sit above every workspace rather than owning one. + */ export const createUser = async (input: { name: string; email: string; password: string; role: ClustoxRole; teamIds: string[]; - orgId: string; -}): Promise => { + /** Existing workspace to place an ADMIN in. Omit to provision a new one. */ + orgId?: string | null; +}): Promise<{ userId: string; orgId: string | null }> => { const passwordHash = await hashPassword(input.password); + let orgId: string | null = null; + if (input.role === 'ADMIN') { + orgId = input.orgId ?? (await createWorkspace(input.name)); + } + const [user] = await db(Table.Users) .insert({ - org_id: input.orgId, + org_id: orgId, name: input.name, primary_email: input.email, is_deleted: false @@ -142,7 +201,7 @@ export const createUser = async (input: { ); } - return userId; + return { userId, orgId }; }; export const updateUserRole = async ( diff --git a/web-server/src/auth/session.ts b/web-server/src/auth/session.ts index dcf59e96..3e4aa9ca 100644 --- a/web-server/src/auth/session.ts +++ b/web-server/src/auth/session.ts @@ -30,6 +30,7 @@ export const getAuthSession = async ( userId: user.userId, email: user.email, name: user.name, - role: user.role + role: user.role, + orgId: user.orgId }; }; diff --git a/web-server/src/auth/types.ts b/web-server/src/auth/types.ts index 1d56686e..ceb34139 100644 --- a/web-server/src/auth/types.ts +++ b/web-server/src/auth/types.ts @@ -5,6 +5,14 @@ export interface AuthSession { email: string; name: string; role: ClustoxRole; + /** + * The workspace this user owns. + * + * null for SUPERADMIN, who sits above every workspace rather than owning + * one. Upstream's Users.org_id is already nullable, so this needs no schema + * change -- the column was built for exactly this. + */ + orgId: string | null; } export interface AuthUserRow { @@ -12,6 +20,7 @@ export interface AuthUserRow { email: string; name: string; role: ClustoxRole; + orgId: string | null; passwordHash: string; } @@ -20,5 +29,7 @@ export interface AuthUserListItem { email: string; name: string; role: ClustoxRole; + orgId: string | null; + orgName: string | null; teamIds: string[]; } From 70527efc2befd409dc7493dd3e44b8d31d5c9b07 Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Thu, 6 Aug 2026 18:32:04 +0500 Subject: [PATCH 022/194] feat(tenancy): scope every request to the caller's workspace Access now derives from workspace ownership rather than per-team grants. An admin sees everything in the workspace they own and nothing outside it; a superadmin sees across all of them. - assertWorkspaceAccess gates any route carrying org_id - assertTeamAccess resolves the team's workspace instead of consulting ClustoxUserTeamAccess, so an admin reaches every team in their own workspace without needing individual grants - both run centrally in Endpoint.serve(), so a newly added route is scoped by default - the BFF session returns the caller's own workspace; upstream took whichever Organization came back first, correct only while one existed - a missing team is refused rather than 404'd, so an admin cannot probe for team ids outside their workspace ClustoxUserTeamAccess is left in place unused. Sub-workspace scoping can be reinstated later without a migration if a workspace grows enough to need it. Verified live with two workspaces: superadmin session -> no workspace create admin -> new workspace provisioned admin session -> own workspace, no integrations admin -> other workspace -> 403 (teams, team data, users api) superadmin -> both -> 200, sees all teams in each 163 jest and 155 pytest tests pass. --- web-server/pages/api/auth/session.ts | 25 +++- .../api/integrations/integrations-map.ts | 4 +- .../__tests__/endpoint-team-scope.test.ts | 23 ++-- web-server/src/api-helpers/global.ts | 18 ++- web-server/src/auth/__tests__/guard.test.ts | 63 +--------- .../auth/__tests__/workspace-guard.test.ts | 116 ++++++++++++++++++ web-server/src/auth/guard.ts | 58 +++++++-- web-server/src/auth/queries.ts | 19 +++ 8 files changed, 237 insertions(+), 89 deletions(-) create mode 100644 web-server/src/auth/__tests__/workspace-guard.test.ts diff --git a/web-server/pages/api/auth/session.ts b/web-server/pages/api/auth/session.ts index 61a99efc..e6d28386 100644 --- a/web-server/pages/api/auth/session.ts +++ b/web-server/pages/api/auth/session.ts @@ -32,10 +32,21 @@ export const delUserIdCookie = (res: NextApiResponse) => { ]); }; -endpoint.handle.GET(nullSchema, async (_req, res) => { +endpoint.handle.GET(nullSchema, async (req, res) => { + // CLUSTOX: resolve the caller's own workspace. Upstream took whichever + // Organization came back first, which was correct only while exactly one + // existed. A superadmin owns none, so they get no workspace context here -- + // cross-workspace views are served by dedicated endpoints. + const session = (req as any).session; + const orgId: string | null = session?.orgId ?? null; + + if (!orgId) { + return res.send({ org: null }); + } + const [orgDetails, integrationsLinkedAtMap] = await Promise.all([ - getOrgDetails(), - getOrgIntegrations() + getOrgDetails(orgId), + getOrgIntegrations(orgId) ]); const [onboardingState, codeProviderLastSyncedAt] = await Promise.all([ @@ -69,13 +80,15 @@ endpoint.handle.GET(nullSchema, async (_req, res) => { export default endpoint.serve(); -const getOrgDetails = async () => { - return db(Table.Organization).select('*').then(getFirstRow); +// CLUSTOX: scoped to a workspace rather than "the" organization. +const getOrgDetails = async (orgId: string) => { + return db(Table.Organization).select('*').where('id', orgId).then(getFirstRow); }; -export const getOrgIntegrations = async () => { +export const getOrgIntegrations = async (orgId: string) => { return db(Table.Integration) .select('*') + .where('org_id', orgId) .whereNotNull('access_token_enc_chunks') .then(async (rows) => { const integrationsLinkedAtMap = rows.reduce( diff --git a/web-server/pages/api/integrations/integrations-map.ts b/web-server/pages/api/integrations/integrations-map.ts index a216f75d..119c86e0 100644 --- a/web-server/pages/api/integrations/integrations-map.ts +++ b/web-server/pages/api/integrations/integrations-map.ts @@ -14,8 +14,10 @@ const getSchema = yup.object().shape({ endpoint.handle.GET(getSchema, async (req, res) => { const { org_id } = req.payload; + // CLUSTOX: integrations are per workspace. Endpoint.serve() has already + // checked the caller may access this org_id. const [integrationsLinkedAtMap, codeProviderLastSyncedAt] = await Promise.all( - [getOrgIntegrations(), getLastSyncedAtForCodeProvider(org_id)] + [getOrgIntegrations(org_id), getLastSyncedAtForCodeProvider(org_id)] ); const integrations = {} as IntegrationsMap; Object.entries(integrationsLinkedAtMap).forEach( diff --git a/web-server/src/api-helpers/__tests__/endpoint-team-scope.test.ts b/web-server/src/api-helpers/__tests__/endpoint-team-scope.test.ts index 244d1d30..183dd17f 100644 --- a/web-server/src/api-helpers/__tests__/endpoint-team-scope.test.ts +++ b/web-server/src/api-helpers/__tests__/endpoint-team-scope.test.ts @@ -1,11 +1,12 @@ jest.mock('@/auth/session', () => ({ getAuthSession: jest.fn() })); jest.mock('@/auth/queries', () => ({ - getTeamIdsForUser: jest.fn(), + getTeamOrgId: jest.fn(), + getTeamIdsForOrg: jest.fn(), getAllTeamIds: jest.fn() })); import { Endpoint, nullSchema } from '@/api-helpers/global'; -import { getTeamIdsForUser } from '@/auth/queries'; +import { getTeamOrgId } from '@/auth/queries'; import { getAuthSession } from '@/auth/session'; const mockRes = () => { @@ -28,7 +29,8 @@ const asAdmin = () => userId: 'ad1', email: 'lead@clustox.com', name: 'Lead', - role: 'ADMIN' + role: 'ADMIN', + orgId: 'workspace-1' }); const asSuperadmin = () => @@ -36,7 +38,8 @@ const asSuperadmin = () => userId: 'su1', email: 'boss@clustox.com', name: 'Boss', - role: 'SUPERADMIN' + role: 'SUPERADMIN', + orgId: null }); const buildEndpoint = () => { @@ -48,9 +51,9 @@ const buildEndpoint = () => { describe('Endpoint team scoping', () => { beforeEach(() => jest.clearAllMocks()); - it('403s when an admin requests a team they are not assigned', async () => { + it('403s when an admin requests a team outside their workspace', async () => { asAdmin(); - (getTeamIdsForUser as jest.Mock).mockResolvedValue(['team-zda']); + (getTeamOrgId as jest.Mock).mockResolvedValue('workspace-2'); const res = mockRes(); await buildEndpoint().serve()(reqWithTeam('team-cgpt'), res); @@ -58,9 +61,9 @@ describe('Endpoint team scoping', () => { expect(res.status).toHaveBeenCalledWith(403); }); - it('allows an admin their own team', async () => { + it('allows an admin any team inside their own workspace', async () => { asAdmin(); - (getTeamIdsForUser as jest.Mock).mockResolvedValue(['team-zda']); + (getTeamOrgId as jest.Mock).mockResolvedValue('workspace-1'); const res = mockRes(); await buildEndpoint().serve()(reqWithTeam('team-zda'), res); @@ -75,7 +78,7 @@ describe('Endpoint team scoping', () => { await buildEndpoint().serve()(reqWithTeam('team-anything'), res); expect(res.send).toHaveBeenCalledWith({ ok: true }); - expect(getTeamIdsForUser).not.toHaveBeenCalled(); + expect(getTeamOrgId).not.toHaveBeenCalled(); }); it('does not attempt a team check when the payload has no team_id', async () => { @@ -85,7 +88,7 @@ describe('Endpoint team scoping', () => { await buildEndpoint().serve()(reqWithTeam(undefined), res); expect(res.send).toHaveBeenCalledWith({ ok: true }); - expect(getTeamIdsForUser).not.toHaveBeenCalled(); + expect(getTeamOrgId).not.toHaveBeenCalled(); }); it('skips the team check entirely on unauthenticated endpoints', async () => { diff --git a/web-server/src/api-helpers/global.ts b/web-server/src/api-helpers/global.ts index 6b973b93..099e062d 100644 --- a/web-server/src/api-helpers/global.ts +++ b/web-server/src/api-helpers/global.ts @@ -2,7 +2,11 @@ import { NextApiRequest } from 'next/types'; import { AnySchema, InferType, object } from 'yup'; // CLUSTOX: enforce the `authenticated` flag that upstream declared but never read. -import { assertAuthenticated, assertTeamAccess } from '@/auth/guard'; +import { + assertAuthenticated, + assertTeamAccess, + assertWorkspaceAccess +} from '@/auth/guard'; import { getAuthSession } from '@/auth/session'; import { Errors, ResponseError } from '@/constants/error'; import { ApiRequest, ApiResponse, HttpMethods } from '@/types/request'; @@ -89,10 +93,14 @@ export class Endpoint { assertAuthenticated(session); (req as any).session = session; - // CLUSTOX: any route addressing a specific team is scoped centrally - // here rather than in each of the ~18 team routes. Doing it at the - // choke point means a newly added team route is protected by default - // instead of relying on the author remembering a guard call. + // CLUSTOX: routes addressing a specific workspace or team are scoped + // centrally here rather than in each of the ~18 team routes and ~20 + // org routes. Doing it at the choke point means a newly added route + // is protected by default instead of relying on the author + // remembering a guard call. + const orgId = (req.payload as any)?.org_id; + if (orgId) await assertWorkspaceAccess(session, String(orgId)); + const teamId = (req.payload as any)?.team_id; if (teamId) await assertTeamAccess(session, String(teamId)); } diff --git a/web-server/src/auth/__tests__/guard.test.ts b/web-server/src/auth/__tests__/guard.test.ts index 736778b2..f1754efc 100644 --- a/web-server/src/auth/__tests__/guard.test.ts +++ b/web-server/src/auth/__tests__/guard.test.ts @@ -1,16 +1,14 @@ jest.mock('@/auth/queries', () => ({ - getTeamIdsForUser: jest.fn(), + getTeamIdsForOrg: jest.fn(), + getTeamOrgId: jest.fn(), getAllTeamIds: jest.fn() })); import { assertAuthenticated, assertRole, - assertTeamAccess, - canAccessAllTeams, - getAccessibleTeamIds + canAccessAllTeams } from '@/auth/guard'; -import { getAllTeamIds, getTeamIdsForUser } from '@/auth/queries'; import { AuthSession } from '@/auth/types'; const superadmin: AuthSession = { @@ -67,56 +65,5 @@ describe('canAccessAllTeams', () => { }); }); -describe('assertTeamAccess', () => { - beforeEach(() => jest.clearAllMocks()); - - it('allows superadmin any team without querying', async () => { - await expect( - assertTeamAccess(superadmin, 'any-team') - ).resolves.toBeUndefined(); - expect(getTeamIdsForUser).not.toHaveBeenCalled(); - }); - - it('allows an admin their assigned team', async () => { - (getTeamIdsForUser as jest.Mock).mockResolvedValue(['t1', 't2']); - await expect(assertTeamAccess(admin, 't2')).resolves.toBeUndefined(); - }); - - it('throws 403 when an admin requests a team they do not own', async () => { - (getTeamIdsForUser as jest.Mock).mockResolvedValue(['t1']); - await expect(assertTeamAccess(admin, 't-other')).rejects.toMatchObject({ - status: 403 - }); - }); - - it('throws 403 when an admin has no teams at all', async () => { - (getTeamIdsForUser as jest.Mock).mockResolvedValue([]); - await expect(assertTeamAccess(admin, 't1')).rejects.toMatchObject({ - status: 403 - }); - }); - - it('throws 401 when unauthenticated', async () => { - await expect(assertTeamAccess(null, 't1')).rejects.toMatchObject({ - status: 401 - }); - }); -}); - -describe('getAccessibleTeamIds', () => { - beforeEach(() => jest.clearAllMocks()); - - it('returns every team for superadmin', async () => { - (getAllTeamIds as jest.Mock).mockResolvedValue(['t1', 't2', 't3']); - await expect(getAccessibleTeamIds(superadmin)).resolves.toEqual([ - 't1', - 't2', - 't3' - ]); - }); - - it('returns only assigned teams for admin', async () => { - (getTeamIdsForUser as jest.Mock).mockResolvedValue(['t2']); - await expect(getAccessibleTeamIds(admin)).resolves.toEqual(['t2']); - }); -}); +// assertTeamAccess and getAccessibleTeamIds are workspace-derived since +// multitenancy; they are covered in workspace-guard.test.ts. diff --git a/web-server/src/auth/__tests__/workspace-guard.test.ts b/web-server/src/auth/__tests__/workspace-guard.test.ts new file mode 100644 index 00000000..b30b3786 --- /dev/null +++ b/web-server/src/auth/__tests__/workspace-guard.test.ts @@ -0,0 +1,116 @@ +jest.mock('@/auth/queries', () => ({ + getTeamIdsForUser: jest.fn(), + getAllTeamIds: jest.fn(), + getTeamOrgId: jest.fn(), + getTeamIdsForOrg: jest.fn() +})); + +import { + assertTeamAccess, + assertWorkspaceAccess, + getAccessibleTeamIds +} from '@/auth/guard'; +import { getTeamIdsForOrg, getTeamOrgId } from '@/auth/queries'; +import { AuthSession } from '@/auth/types'; + +const superadmin: AuthSession = { + userId: 'su1', + email: 'boss@clustox.com', + name: 'Boss', + role: 'SUPERADMIN', + orgId: null +}; + +const admin: AuthSession = { + userId: 'ad1', + email: 'lead@clustox.com', + name: 'Lead', + role: 'ADMIN', + orgId: 'workspace-1' +}; + +const adminWithoutWorkspace: AuthSession = { + ...admin, + userId: 'ad2', + orgId: null +}; + +describe('assertWorkspaceAccess', () => { + it('lets a superadmin into any workspace', async () => { + await expect( + assertWorkspaceAccess(superadmin, 'workspace-9') + ).resolves.toBeUndefined(); + }); + + it('lets an admin into their own workspace', async () => { + await expect( + assertWorkspaceAccess(admin, 'workspace-1') + ).resolves.toBeUndefined(); + }); + + it('403s an admin reaching into another workspace', async () => { + await expect( + assertWorkspaceAccess(admin, 'workspace-2') + ).rejects.toMatchObject({ status: 403 }); + }); + + it('403s an admin who has no workspace at all', async () => { + await expect( + assertWorkspaceAccess(adminWithoutWorkspace, 'workspace-1') + ).rejects.toMatchObject({ status: 403 }); + }); + + it('401s when unauthenticated', async () => { + await expect( + assertWorkspaceAccess(null, 'workspace-1') + ).rejects.toMatchObject({ status: 401 }); + }); +}); + +describe('assertTeamAccess is now workspace-derived', () => { + beforeEach(() => jest.clearAllMocks()); + + it('lets an admin reach any team inside their own workspace', async () => { + // No explicit ClustoxUserTeamAccess row: ownership of the workspace is + // what grants access now. + (getTeamOrgId as jest.Mock).mockResolvedValue('workspace-1'); + await expect(assertTeamAccess(admin, 'team-a')).resolves.toBeUndefined(); + }); + + it('403s an admin reaching a team in another workspace', async () => { + (getTeamOrgId as jest.Mock).mockResolvedValue('workspace-2'); + await expect(assertTeamAccess(admin, 'team-b')).rejects.toMatchObject({ + status: 403 + }); + }); + + it('403s on a team that does not exist', async () => { + (getTeamOrgId as jest.Mock).mockResolvedValue(null); + await expect(assertTeamAccess(admin, 'team-ghost')).rejects.toMatchObject({ + status: 403 + }); + }); + + it('lets a superadmin reach any team without a lookup', async () => { + await expect( + assertTeamAccess(superadmin, 'team-anything') + ).resolves.toBeUndefined(); + expect(getTeamOrgId).not.toHaveBeenCalled(); + }); +}); + +describe('getAccessibleTeamIds', () => { + beforeEach(() => jest.clearAllMocks()); + + it('gives an admin every team in their workspace', async () => { + (getTeamIdsForOrg as jest.Mock).mockResolvedValue(['t1', 't2']); + await expect(getAccessibleTeamIds(admin)).resolves.toEqual(['t1', 't2']); + expect(getTeamIdsForOrg).toHaveBeenCalledWith('workspace-1'); + }); + + it('gives an admin with no workspace nothing', async () => { + await expect(getAccessibleTeamIds(adminWithoutWorkspace)).resolves.toEqual( + [] + ); + }); +}); diff --git a/web-server/src/auth/guard.ts b/web-server/src/auth/guard.ts index e5127892..11d7c5c5 100644 --- a/web-server/src/auth/guard.ts +++ b/web-server/src/auth/guard.ts @@ -1,6 +1,6 @@ import { Errors, ResponseError } from '@/constants/error'; -import { getAllTeamIds, getTeamIdsForUser } from './queries'; +import { getTeamIdsForOrg, getTeamOrgId } from './queries'; import { AuthSession, ClustoxRole } from './types'; export const assertAuthenticated = ( @@ -19,25 +19,65 @@ export const assertRole = ( return authed; }; -export const canAccessAllTeams = (session: AuthSession): boolean => +/** A superadmin sits above every workspace rather than owning one. */ +export const canAccessAllWorkspaces = (session: AuthSession): boolean => session.role === 'SUPERADMIN'; +/** Retained name for readability at call sites that talk about teams. */ +export const canAccessAllTeams = canAccessAllWorkspaces; + +/** + * Gate a request against a specific workspace. + * + * An admin may only touch the workspace they own. An admin with no workspace + * is denied everything -- that state means their account exists but has not + * been provisioned, and silently showing them an empty app reads as data loss + * rather than as a permissions problem. + */ +export const assertWorkspaceAccess = async ( + session: AuthSession | null, + orgId: string +): Promise => { + const authed = assertAuthenticated(session); + if (canAccessAllWorkspaces(authed)) return; + + if (!authed.orgId || authed.orgId !== orgId) { + throw new ResponseError(Errors.ACCESS_DENIED, 403); + } +}; + +/** + * Gate a request against a team. + * + * Access is derived from which workspace the team belongs to, not from a + * per-team grant. An admin owns their whole workspace, so listing individual + * teams would be redundant. ClustoxUserTeamAccess is left in place unused, so + * finer-grained scoping can be reinstated inside a workspace later without a + * migration. + */ export const assertTeamAccess = async ( session: AuthSession | null, teamId: string ): Promise => { const authed = assertAuthenticated(session); - if (canAccessAllTeams(authed)) return; + if (canAccessAllWorkspaces(authed)) return; - const teamIds = await getTeamIdsForUser(authed.userId); - if (!teamIds.includes(teamId)) { + const teamOrgId = await getTeamOrgId(teamId); + // A missing team is denied rather than 404'd, so an admin cannot probe for + // which team ids exist outside their workspace. + if (!teamOrgId || !authed.orgId || teamOrgId !== authed.orgId) { throw new ResponseError(Errors.ACCESS_DENIED, 403); } }; +/** Every team the session may see. */ export const getAccessibleTeamIds = async ( session: AuthSession -): Promise => - canAccessAllTeams(session) - ? getAllTeamIds() - : getTeamIdsForUser(session.userId); +): Promise => { + if (canAccessAllWorkspaces(session)) { + const { getAllTeamIds } = await import('./queries'); + return getAllTeamIds(); + } + if (!session.orgId) return []; + return getTeamIdsForOrg(session.orgId); +}; diff --git a/web-server/src/auth/queries.ts b/web-server/src/auth/queries.ts index d0d4b83a..4f1b00ee 100644 --- a/web-server/src/auth/queries.ts +++ b/web-server/src/auth/queries.ts @@ -88,6 +88,25 @@ export const getAllTeamIds = async (): Promise => { return rows.map((r: { id: string }) => r.id); }; +/** Which workspace a team belongs to, or null if the team does not exist. */ +export const getTeamOrgId = async (teamId: string): Promise => { + const row = await db(Table.Team) + .select('org_id') + .where('id', teamId) + .andWhere('is_deleted', false) + .first(); + return row?.org_id ?? null; +}; + +/** Every live team in a workspace. */ +export const getTeamIdsForOrg = async (orgId: string): Promise => { + const rows = await db(Table.Team) + .select('id') + .where('org_id', orgId) + .andWhere('is_deleted', false); + return rows.map((r: { id: string }) => r.id); +}; + export const countSuperadmins = async (): Promise => { const rows = await db(Table.ClustoxUserAuth) .where(Columns[Table.ClustoxUserAuth].role, 'SUPERADMIN') From 26f01efcd98ba043c084e0688b860e02d03422b2 Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Thu, 6 Aug 2026 18:57:22 +0500 Subject: [PATCH 023/194] fix(tenancy): give superadmin a workspace to view Regression from the previous commit. A superadmin owns no workspace, so session.org was null, integrationList was empty, and the sidebar filter collapsed to just 'Manage Integrations' while the router forced them onto that page. The app looked unconfigured. The whole UI is workspace-scoped, so a superadmin needs a workspace to look at even though they own none: - /api/clustox/workspaces lists all workspaces and records a selection - the session resolves that selection, falling back to the oldest workspace, and ignores a stale one that no longer exists - a workspace switcher in the sidebar footer, superadmin only - superadmin is exempt from the onboarding redirects and the integration-required sidebar filter, which describe a workspace they are merely viewing The selection is viewing context only. Guards never consult it -- a superadmin passes every workspace check regardless of what is selected. --- web-server/pages/api/clustox/workspaces.ts | 38 ++++++++++ web-server/src/auth/__tests__/session.test.ts | 60 +++++++++++++-- web-server/src/auth/queries.ts | 19 +++++ web-server/src/auth/session.ts | 34 ++++++++- .../src/components/ClustoxUserFooter.tsx | 28 ++++--- .../components/ClustoxWorkspaceSwitcher.tsx | 73 +++++++++++++++++++ web-server/src/constants/useRoute.ts | 9 +++ .../SidebarMenu/useFilteredSidebarItems.tsx | 5 ++ 8 files changed, 246 insertions(+), 20 deletions(-) create mode 100644 web-server/pages/api/clustox/workspaces.ts create mode 100644 web-server/src/components/ClustoxWorkspaceSwitcher.tsx diff --git a/web-server/pages/api/clustox/workspaces.ts b/web-server/pages/api/clustox/workspaces.ts new file mode 100644 index 00000000..1e1ccf83 --- /dev/null +++ b/web-server/pages/api/clustox/workspaces.ts @@ -0,0 +1,38 @@ +import * as yup from 'yup'; + +import { Endpoint, nullSchema } from '@/api-helpers/global'; +import { assertRole } from '@/auth/guard'; +import { listWorkspaces } from '@/auth/queries'; + +/** + * Every workspace, for the SuperAdmin workspace switcher. + * + * A SuperAdmin owns no workspace, so without a way to choose one the whole + * org-scoped UI has nothing to render. Selecting a workspace here sets the + * viewing context; it does not change what they are permitted to reach, which + * is everything. + */ +const endpoint = new Endpoint(nullSchema); + +endpoint.handle.GET(nullSchema, async (req, res) => { + assertRole((req as any).session, 'SUPERADMIN'); + res.send(await listWorkspaces()); +}); + +const putSchema = yup.object().shape({ + org_id: yup.string().uuid().required() +}); + +endpoint.handle.PUT(putSchema, async (req, res) => { + assertRole((req as any).session, 'SUPERADMIN'); + + // Viewing context only. Guards never consult this -- a SuperAdmin passes + // every workspace check regardless of what is selected. + res.setHeader( + 'set-cookie', + `clustox-workspace=${req.payload.org_id};Path=/;HttpOnly;SameSite=Lax` + ); + res.send({ org_id: req.payload.org_id }); +}); + +export default endpoint.serve(); diff --git a/web-server/src/auth/__tests__/session.test.ts b/web-server/src/auth/__tests__/session.test.ts index 05aa52ea..053e0f80 100644 --- a/web-server/src/auth/__tests__/session.test.ts +++ b/web-server/src/auth/__tests__/session.test.ts @@ -1,12 +1,20 @@ jest.mock('next-auth/jwt', () => ({ getToken: jest.fn() })); -jest.mock('@/auth/queries', () => ({ getAuthUserById: jest.fn() })); +jest.mock('@/auth/queries', () => ({ + getAuthUserById: jest.fn(), + listWorkspaces: jest.fn(), + workspaceExists: jest.fn() +})); import { getToken } from 'next-auth/jwt'; -import { getAuthUserById } from '@/auth/queries'; +import { + getAuthUserById, + listWorkspaces, + workspaceExists +} from '@/auth/queries'; import { getAuthSession } from '@/auth/session'; -const req = {} as any; +const req = { cookies: {} } as any; describe('getAuthSession', () => { beforeEach(() => jest.clearAllMocks()); @@ -63,7 +71,7 @@ describe('getAuthSession', () => { expect(session?.orgId).toBe('workspace-1'); }); - it('gives a superadmin no workspace, so they sit above all of them', async () => { + it('drops a superadmin into the oldest workspace when none is selected', async () => { (getToken as jest.Mock).mockResolvedValue({ userId: 'su1' }); (getAuthUserById as jest.Mock).mockResolvedValue({ userId: 'su1', @@ -72,10 +80,52 @@ describe('getAuthSession', () => { role: 'SUPERADMIN', orgId: null }); + (listWorkspaces as jest.Mock).mockResolvedValue([ + { id: 'workspace-1', name: 'First' }, + { id: 'workspace-2', name: 'Second' } + ]); const session = await getAuthSession(req); - expect(session?.orgId).toBeNull(); expect(session?.role).toBe('SUPERADMIN'); + // Viewing context, not ownership -- guards let them into any workspace. + expect(session?.orgId).toBe('workspace-1'); + }); + + it('honours a superadmin selected workspace', async () => { + (getToken as jest.Mock).mockResolvedValue({ userId: 'su1' }); + (getAuthUserById as jest.Mock).mockResolvedValue({ + userId: 'su1', + email: 'boss@clustox.com', + name: 'Boss', + role: 'SUPERADMIN', + orgId: null + }); + (workspaceExists as jest.Mock).mockResolvedValue(true); + + const session = await getAuthSession({ + cookies: { 'clustox-workspace': 'workspace-2' } + } as any); + expect(session?.orgId).toBe('workspace-2'); + }); + + it('ignores a stale selected workspace that no longer exists', async () => { + (getToken as jest.Mock).mockResolvedValue({ userId: 'su1' }); + (getAuthUserById as jest.Mock).mockResolvedValue({ + userId: 'su1', + email: 'boss@clustox.com', + name: 'Boss', + role: 'SUPERADMIN', + orgId: null + }); + (workspaceExists as jest.Mock).mockResolvedValue(false); + (listWorkspaces as jest.Mock).mockResolvedValue([ + { id: 'workspace-1', name: 'First' } + ]); + + const session = await getAuthSession({ + cookies: { 'clustox-workspace': 'deleted-workspace' } + } as any); + expect(session?.orgId).toBe('workspace-1'); }); it('reflects a demotion immediately rather than at token expiry', async () => { diff --git a/web-server/src/auth/queries.ts b/web-server/src/auth/queries.ts index 4f1b00ee..6a9887e3 100644 --- a/web-server/src/auth/queries.ts +++ b/web-server/src/auth/queries.ts @@ -98,6 +98,25 @@ export const getTeamOrgId = async (teamId: string): Promise => { return row?.org_id ?? null; }; +/** Every workspace, oldest first. Used by the SuperAdmin switcher. */ +export const listWorkspaces = async (): Promise< + { id: string; name: string }[] +> => { + const rows = await db(Table.Organization) + .select('id', 'name') + .orderBy('created_at', 'asc'); + return rows.map((r: { id: string; name: string }) => ({ + id: r.id, + name: r.name + })); +}; + +/** Does this workspace exist? Guards the SuperAdmin's selected workspace. */ +export const workspaceExists = async (orgId: string): Promise => { + const row = await db(Table.Organization).select('id').where('id', orgId).first(); + return Boolean(row); +}; + /** Every live team in a workspace. */ export const getTeamIdsForOrg = async (orgId: string): Promise => { const rows = await db(Table.Team) diff --git a/web-server/src/auth/session.ts b/web-server/src/auth/session.ts index 3e4aa9ca..6fcc9105 100644 --- a/web-server/src/auth/session.ts +++ b/web-server/src/auth/session.ts @@ -1,9 +1,32 @@ import { getToken } from 'next-auth/jwt'; import { NextApiRequest } from 'next/types'; -import { getAuthUserById } from './queries'; +import { getAuthUserById, listWorkspaces, workspaceExists } from './queries'; import { AuthSession } from './types'; +const WORKSPACE_COOKIE = 'clustox-workspace'; + +/** + * Which workspace a SuperAdmin is currently looking at. + * + * A SuperAdmin owns no workspace, but the entire UI is workspace-scoped, so + * without one there is nothing to render. Their selection is a *viewing* + * context only -- guards let a SuperAdmin into every workspace regardless of + * what is selected here. + * + * Falls back to the oldest workspace so a fresh SuperAdmin lands somewhere + * useful rather than on an empty app. + */ +const resolveViewingWorkspace = async ( + req: NextApiRequest +): Promise => { + const selected = req.cookies?.[WORKSPACE_COOKIE]; + if (selected && (await workspaceExists(selected))) return selected; + + const workspaces = await listWorkspaces(); + return workspaces[0]?.id ?? null; +}; + /** * Resolve the caller's session. * @@ -14,8 +37,6 @@ import { AuthSession } from './types'; * Without this, a JWT stays valid for its full lifetime regardless of what * happens to the account behind it: a deleted user keeps a working session, * and a demoted superadmin keeps superadmin powers until the token expires. - * Team access was already resolved per-request for the same reason; leaving - * identity and role in the token was an inconsistency. */ export const getAuthSession = async ( req: NextApiRequest @@ -26,11 +47,16 @@ export const getAuthSession = async ( const user = await getAuthUserById(token.userId as string); if (!user) return null; + const orgId = + user.role === 'SUPERADMIN' + ? await resolveViewingWorkspace(req) + : user.orgId; + return { userId: user.userId, email: user.email, name: user.name, role: user.role, - orgId: user.orgId + orgId }; }; diff --git a/web-server/src/components/ClustoxUserFooter.tsx b/web-server/src/components/ClustoxUserFooter.tsx index e51b702f..d8c4cf2a 100644 --- a/web-server/src/components/ClustoxUserFooter.tsx +++ b/web-server/src/components/ClustoxUserFooter.tsx @@ -2,6 +2,7 @@ import { LogoutTwoTone } from '@mui/icons-material'; import { Button, Chip, Tooltip, Typography } from '@mui/material'; import { signOut } from 'next-auth/react'; +import { ClustoxWorkspaceSwitcher } from '@/components/ClustoxWorkspaceSwitcher'; import { FlexBox } from '@/components/FlexBox'; import { useClustoxUser } from '@/hooks/useClustoxUser'; @@ -15,8 +16,10 @@ export const ClustoxUserFooter = () => { if (loading || !user) return null; return ( - - + + {/* Only renders for a SuperAdmin, who owns no workspace of their own. */} + + { color={user.role === 'SUPERADMIN' ? 'primary' : 'default'} /> - + + + ); }; diff --git a/web-server/src/components/ClustoxWorkspaceSwitcher.tsx b/web-server/src/components/ClustoxWorkspaceSwitcher.tsx new file mode 100644 index 00000000..ff572122 --- /dev/null +++ b/web-server/src/components/ClustoxWorkspaceSwitcher.tsx @@ -0,0 +1,73 @@ +import { FormControl, InputLabel, MenuItem, Select } from '@mui/material'; +import { useCallback, useEffect, useState } from 'react'; + +import { useClustoxUser } from '@/hooks/useClustoxUser'; + +type Workspace = { id: string; name: string }; + +/** + * CLUSTOX: lets a SuperAdmin choose which workspace they are looking at. + * + * A SuperAdmin owns no workspace, but every dashboard in the app is + * workspace-scoped, so without a selection there is nothing to show. This is a + * viewing control, not a permission one -- a SuperAdmin may reach every + * workspace regardless of what is selected. + */ +export const ClustoxWorkspaceSwitcher = () => { + const { isSuperadmin, loading } = useClustoxUser(); + const [workspaces, setWorkspaces] = useState([]); + const [current, setCurrent] = useState(''); + + const load = useCallback(async () => { + const [wsRes, sessionRes] = await Promise.all([ + fetch('/api/clustox/workspaces'), + fetch('/api/auth/session') + ]); + if (!wsRes.ok) return; + + setWorkspaces(await wsRes.json()); + + if (sessionRes.ok) { + const session = await sessionRes.json(); + if (session?.org?.id) setCurrent(session.org.id); + } + }, []); + + useEffect(() => { + if (!loading && isSuperadmin) load(); + }, [loading, isSuperadmin, load]); + + const onChange = async (orgId: string) => { + setCurrent(orgId); + await fetch('/api/clustox/workspaces', { + method: 'PUT', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify({ org_id: orgId }) + }); + // Full reload: the workspace is resolved server-side per request, and + // every cached dashboard slice belongs to the previous workspace. + window.location.reload(); + }; + + if (loading || !isSuperadmin || workspaces.length === 0) return null; + + return ( + + + Workspace + + + + ); +}; diff --git a/web-server/src/constants/useRoute.ts b/web-server/src/constants/useRoute.ts index e4599b1e..73611656 100644 --- a/web-server/src/constants/useRoute.ts +++ b/web-server/src/constants/useRoute.ts @@ -2,6 +2,8 @@ import { useRouter } from 'next/router'; import { useEffect } from 'react'; import { useAuth } from '@/hooks/useAuth'; +// CLUSTOX: a SuperAdmin is not subject to workspace onboarding gates. +import { useClustoxUser } from '@/hooks/useClustoxUser'; import { OnboardingStep, UserRole } from '@/types/resources'; import { ROUTES } from './routes'; @@ -14,6 +16,11 @@ export const useRedirectWithSession = () => { const defaultRoute = useDefaultRoute(); const router = useRouter(); const { org, orgId, onboardingState } = useAuth(); + // CLUSTOX: a SuperAdmin owns no workspace of their own, so the onboarding + // gates below (welcome screen, integration linked, team created) describe a + // workspace they are merely viewing. Forcing them through those would leave + // them unable to reach user management or the workspace switcher. + const { isSuperadmin } = useClustoxUser(); const isOrgWelcomed = onboardingState.includes(OnboardingStep.WELCOME_SCREEN); @@ -28,6 +35,7 @@ export const useRedirectWithSession = () => { useEffect(() => { if (!router.isReady) return; + if (isSuperadmin) return; if (!isOrgWelcomed) { router.replace(ROUTES.WELCOME.PATH); return; @@ -51,6 +59,7 @@ export const useRedirectWithSession = () => { defaultRoute.PATH, isOneCodeProviderIntegrated, isOrgWelcomed, + isSuperadmin, orgId, router ]); diff --git a/web-server/src/layouts/ExtendedSidebarLayout/Sidebar/SidebarMenu/useFilteredSidebarItems.tsx b/web-server/src/layouts/ExtendedSidebarLayout/Sidebar/SidebarMenu/useFilteredSidebarItems.tsx index 1e598e58..4290f01a 100644 --- a/web-server/src/layouts/ExtendedSidebarLayout/Sidebar/SidebarMenu/useFilteredSidebarItems.tsx +++ b/web-server/src/layouts/ExtendedSidebarLayout/Sidebar/SidebarMenu/useFilteredSidebarItems.tsx @@ -32,7 +32,12 @@ export const useFilteredSidebarItems = () => { // CLUSTOX: hide user management from non-superadmins. Presentation only // -- the API returns 403 regardless of what the sidebar shows. if (item.name === SideBarItems.MANAGE_USERS && !isSuperadmin) return false; + // CLUSTOX: a SuperAdmin owns no workspace, so the workspace they are + // viewing may legitimately have no integration yet. Collapsing their nav + // to just "Manage Integrations" would strand them, since user management + // and the workspace switcher are how they administer the instance. if ( + !isSuperadmin && !integrationList.length && item.name !== SideBarItems.MANAGE_INTEGRATIONS ) From 52e28682b1de4280105bda802d85ac068729b370 Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Thu, 6 Aug 2026 19:07:16 +0500 Subject: [PATCH 024/194] fix(users): restore page chrome and rebuild the page around workspaces Two problems, both reported from actual use. The sidebar and header were missing because the page never used PageWrapper, which is where every other page gets its chrome. Setting getLayout to ExtendedSidebarLayout alone was not enough. The table was also stale: it still showed a Teams column with an empty dropdown, left over from the pre-multitenancy model. Under tenancy an admin belongs to a workspace, not to a list of teams, so the column was both meaningless and misleading -- the one fact this page exists to show was the one it did not show. Rebuilt around the current model: - Workspace column, rendered monospaced so tenancy reads as data - superadmins show 'All workspaces' rather than an empty cell - add-user moved into a dialog; the inline five-field row was cramped and permanently on screen for an occasional action - the role toggle explains what each role actually gets, since the difference is workspace ownership and that is not self-evident - password validation gives a live character count instead of failing on submit - role changes moved to a row menu, with the last-superadmin refusal surfaced as a message rather than a silent no-op - empty state, and counts of admins and workspaces 165 tests pass. --- web-server/pages/users.tsx | 542 +++++++++++++++++++++++++------------ 1 file changed, 373 insertions(+), 169 deletions(-) diff --git a/web-server/pages/users.tsx b/web-server/pages/users.tsx index 8b510e03..a73789b4 100644 --- a/web-server/pages/users.tsx +++ b/web-server/pages/users.tsx @@ -1,27 +1,46 @@ +import { + AdminPanelSettingsTwoTone, + MoreVertTwoTone, + PersonAddAlt1TwoTone, + WorkspacesTwoTone +} from '@mui/icons-material'; import { Alert, - Box, + Avatar, Button, - Card, Chip, CircularProgress, - FormControl, - InputLabel, + Dialog, + DialogActions, + DialogContent, + DialogTitle, + Divider, + IconButton, + ListItemIcon, + ListItemText, + Menu, MenuItem, - Select, Table, TableBody, TableCell, + TableContainer, TableHead, TableRow, TextField, - Typography + ToggleButton, + ToggleButtonGroup, + Tooltip, + Typography, + useTheme } from '@mui/material'; import Head from 'next/head'; -import { FormEvent, useCallback, useEffect, useState } from 'react'; +import { useSnackbar } from 'notistack'; +import { FormEvent, useCallback, useEffect, useMemo, useState } from 'react'; import ExtendedSidebarLayout from 'src/layouts/ExtendedSidebarLayout'; import { FlexBox } from '@/components/FlexBox'; +import { Line } from '@/components/Text'; +import { PageWrapper } from '@/content/PullRequests/PageWrapper'; import { PageLayout } from '@/types/resources'; type ClustoxRole = 'SUPERADMIN' | 'ADMIN'; @@ -31,27 +50,64 @@ type UserRow = { email: string; name: string; role: ClustoxRole; - teamIds: string[]; + orgId: string | null; + orgName: string | null; }; -type Team = { id: string; name: string }; +const MIN_PASSWORD = 12; const emptyForm = { name: '', email: '', password: '', - role: 'ADMIN' as ClustoxRole, - team_ids: [] as string[] + role: 'ADMIN' as ClustoxRole +}; + +const initials = (name: string, email: string) => { + const source = name?.trim() || email; + const parts = source.split(/[\s.@]+/).filter(Boolean); + return (parts[0]?.[0] ?? '?').concat(parts[1]?.[0] ?? '').toUpperCase(); }; +/** Identifiers read as data, not prose. */ +const Mono = ({ children }: { children: React.ReactNode }) => ( + + {children} + +); + +const RoleChip = ({ role }: { role: ClustoxRole }) => ( + +); + function UsersPage() { + const theme = useTheme(); + const { enqueueSnackbar } = useSnackbar(); + const [users, setUsers] = useState([]); - const [teams, setTeams] = useState([]); const [loading, setLoading] = useState(true); const [forbidden, setForbidden] = useState(false); - const [error, setError] = useState(''); - const [busy, setBusy] = useState(false); + + const [dialogOpen, setDialogOpen] = useState(false); const [form, setForm] = useState(emptyForm); + const [busy, setBusy] = useState(false); + const [formError, setFormError] = useState(''); + + const [menuAnchor, setMenuAnchor] = useState(null); + const [menuUser, setMenuUser] = useState(null); const loadUsers = useCallback(async () => { const res = await fetch('/api/clustox/users'); @@ -64,76 +120,97 @@ function UsersPage() { setLoading(false); }, []); - const loadTeams = useCallback(async () => { - const sessionRes = await fetch('/api/auth/session'); - if (!sessionRes.ok) return; - const session = await sessionRes.json(); - const orgId = session?.org?.id; - if (!orgId) return; - - const res = await fetch(`/api/resources/orgs/${orgId}/teams`); - if (!res.ok) return; - const data = await res.json(); - setTeams(data.teams || []); - }, []); - useEffect(() => { loadUsers(); - loadTeams(); - }, [loadUsers, loadTeams]); + }, [loadUsers]); - const teamName = (id: string) => - teams.find((t) => t.id === id)?.name ?? id.slice(0, 8); + const counts = useMemo( + () => ({ + total: users.length, + admins: users.filter((u) => u.role === 'ADMIN').length, + workspaces: new Set(users.map((u) => u.orgId).filter(Boolean)).size + }), + [users] + ); + + const passwordTooShort = + form.password.length > 0 && form.password.length < MIN_PASSWORD; const onCreate = async (e: FormEvent) => { e.preventDefault(); setBusy(true); - setError(''); + setFormError(''); const res = await fetch('/api/clustox/users', { method: 'POST', headers: { 'Content-Type': 'application/json' }, - body: JSON.stringify(form) + body: JSON.stringify({ ...form, team_ids: [] }) }); setBusy(false); if (!res.ok) { - setError( + setFormError( res.status === 409 - ? 'A user with that email already exists' - : 'Could not create user. Password must be at least 12 characters.' + ? 'A user with that email already exists.' + : `Could not create user. Password must be at least ${MIN_PASSWORD} characters.` ); return; } + const created = await res.json(); + setDialogOpen(false); setForm(emptyForm); await loadUsers(); + + enqueueSnackbar( + form.role === 'ADMIN' + ? `${form.name} added — a new workspace was created for them` + : `${form.name} added as a superadmin`, + { variant: 'success', autoHideDuration: 5000 } + ); + return created; }; - const onChangeTeams = async (userId: string, teamIds: string[]) => { - await fetch(`/api/clustox/users/${userId}`, { + const changeRole = async (user: UserRow, role: ClustoxRole) => { + setMenuAnchor(null); + const res = await fetch(`/api/clustox/users/${user.userId}`, { method: 'PATCH', headers: { 'Content-Type': 'application/json' }, - body: JSON.stringify({ team_ids: teamIds }) + body: JSON.stringify({ role }) }); + + if (!res.ok) { + enqueueSnackbar( + res.status === 409 + ? 'You cannot demote the last superadmin.' + : 'Could not change role.', + { variant: 'error' } + ); + return; + } + await loadUsers(); + enqueueSnackbar(`${user.name} is now ${role.toLowerCase()}`, { + variant: 'success' + }); }; if (loading) return ( - + ); if (forbidden) return ( - + - Not authorised. Only superadmins can manage users. + Only superadmins can manage users. If you need access, ask a + superadmin to change your role. - + ); return ( @@ -141,156 +218,283 @@ function UsersPage() { Users | MiddlewareHQ - - - Users - - - - - Add a user - -
- - setForm({ ...form, name: e.target.value })} - /> - setForm({ ...form, email: e.target.value })} - /> - setForm({ ...form, password: e.target.value })} - /> - - Role - - - - Teams - - - + + + {/* Summary + primary action */} + + + + + {counts.total} {counts.total === 1 ? 'user' : 'users'} + + + {counts.admins} admin{counts.admins === 1 ? '' : 's'} across{' '} + {counts.workspaces} workspace + {counts.workspaces === 1 ? '' : 's'} + - {form.role === 'SUPERADMIN' && ( - - Superadmins see every team, so no team assignment is needed. - - )} - {error && ( - - {error} - - )} - -
- - +
+ + +
+ + {/* Roster */} + - Name - Email - Role - Teams + User + Role + Workspace + {users.map((u) => ( - - {u.name} - {u.email} + - + + + {initials(u.name, u.email)} + + + {u.name} + + {u.email} + + + + + + + + {u.role === 'SUPERADMIN' ? ( - - All teams - + + + + + All workspaces + + + ) : ( - + + + {u.orgName ?? '—'} + )} + + + { + setMenuUser(u); + setMenuAnchor(e.currentTarget); + }} + > + + + ))} + + {users.length === 0 && ( + + + + No users yet. + + Add an admin to create their first workspace. + + + + + )}
- - +
+
+ + {/* Row actions */} + setMenuAnchor(null)} + > + {menuUser?.role === 'ADMIN' ? ( + changeRole(menuUser, 'SUPERADMIN')}> + + + + + + ) : ( + menuUser && changeRole(menuUser, 'ADMIN')}> + + + + + + )} + + + {/* Add user */} + !busy && setDialogOpen(false)} + maxWidth="sm" + fullWidth + > +
+ Add a user + + + + v && setForm({ ...form, role: v })} + > + Admin + Superadmin + + + + ) : ( + + ) + } + > + {form.role === 'ADMIN' + ? 'Gets their own workspace, connects their own GitHub or GitLab, and sees only their own projects.' + : 'Sees every workspace, manages users, and owns no workspace of their own.'} + + + setForm({ ...form, name: e.target.value })} + helperText={ + form.role === 'ADMIN' + ? 'Also names their workspace' + : undefined + } + /> + setForm({ ...form, email: e.target.value })} + /> + setForm({ ...form, password: e.target.value })} + helperText={ + passwordTooShort + ? `${MIN_PASSWORD - form.password.length} more characters needed` + : `At least ${MIN_PASSWORD} characters. Share it with them directly.` + } + /> + + {formError && ( + + {formError} + + )} + + + + + + + + +
); } UsersPage.getLayout = (page: PageLayout) => ( - {page} + + + + Users + + } + pageTitle="Users" + hideAllSelectors + showEvenIfNoTeamSelected + showDate={false} + > + {page} + + ); export default UsersPage; From 1b3fea4c05d96fc1ae72b46f4fe6a98aca2878ce Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Thu, 6 Aug 2026 19:39:29 +0500 Subject: [PATCH 025/194] fix(tenancy): let an admin link their own integration An admin creating a GitHub integration got a 403. The request was going to /api/resources/orgs/undefined/integration -- the client had no workspace id to send. Root cause was the client, not the API: the modal read orgId from redux, which redux-persist rehydrates from the previous session. During the window where a superadmin's session legitimately returned no workspace, that empty value was persisted and survived into the next login. - /api/clustox/me now returns the caller's workspace, resolved server-side per request and never persisted - both provider modals prefer it over the redux copy, and refuse to submit without a workspace rather than posting 'undefined' Also corrected the order of checks in Endpoint.serve(). Workspace and team scoping ran before schema validation, so a malformed org id surfaced as an opaque 403 instead of a 400 naming the bad field. That made this take far longer to diagnose than it should have. Validation now runs first. Verified live as an admin: link to own workspace -> 200, stored against their workspace link to another workspace -> 403 malformed workspace id -> 400 (was 403) Each workspace now holds its own GitHub integration, which is what Integration's (name, org_id) key was always designed for. --- web-server/pages/api/clustox/me.ts | 6 +++- web-server/src/api-helpers/global.ts | 31 ++++++++++++------- .../Dashboards/ConfigureGithubModalBody.tsx | 14 ++++++++- .../Dashboards/ConfigureGitlabModalBody.tsx | 15 ++++++++- web-server/src/hooks/useClustoxUser.ts | 5 ++- 5 files changed, 55 insertions(+), 16 deletions(-) diff --git a/web-server/pages/api/clustox/me.ts b/web-server/pages/api/clustox/me.ts index 55ed43a5..c40a6da8 100644 --- a/web-server/pages/api/clustox/me.ts +++ b/web-server/pages/api/clustox/me.ts @@ -19,7 +19,11 @@ endpoint.handle.GET(nullSchema, async (req, res) => { user_id: session.userId, email: session.email, name: session.name, - role: session.role + role: session.role, + // The workspace to act on. Resolved server-side per request, so unlike the + // redux copy it can never be a stale value rehydrated from a previous + // session's persisted state. + org_id: session.orgId }); }); diff --git a/web-server/src/api-helpers/global.ts b/web-server/src/api-helpers/global.ts index 099e062d..ce94bbbd 100644 --- a/web-server/src/api-helpers/global.ts +++ b/web-server/src/api-helpers/global.ts @@ -88,21 +88,11 @@ export class Endpoint { // CLUSTOX: single enforcement point for all BFF routes. Routes opt out // with `new Endpoint(schema, { unauthenticated: true })`. + let session = null; if (this.authenticated) { - const session = await getAuthSession(nextReq); + session = await getAuthSession(nextReq); assertAuthenticated(session); (req as any).session = session; - - // CLUSTOX: routes addressing a specific workspace or team are scoped - // centrally here rather than in each of the ~18 team routes and ~20 - // org routes. Doing it at the choke point means a newly added route - // is protected by default instead of relying on the author - // remembering a guard call. - const orgId = (req.payload as any)?.org_id; - if (orgId) await assertWorkspaceAccess(session, String(orgId)); - - const teamId = (req.payload as any)?.team_id; - if (teamId) await assertTeamAccess(session, String(teamId)); } if (this.pathSchema) { @@ -118,6 +108,23 @@ export class Endpoint { await schema.validate(req.payload); } + // CLUSTOX: routes addressing a specific workspace or team are scoped + // centrally here rather than in each of the ~18 team routes and ~20 org + // routes. Doing it at the choke point means a newly added route is + // protected by default instead of relying on the author remembering a + // guard call. + // + // Runs *after* schema validation so malformed input is a 400 with a + // useful message. Checking first turned a client sending + // "/orgs/undefined/integration" into an opaque 403. + if (this.authenticated) { + const orgId = (req.payload as any)?.org_id; + if (orgId) await assertWorkspaceAccess(session, String(orgId)); + + const teamId = (req.payload as any)?.team_id; + if (teamId) await assertTeamAccess(session, String(teamId)); + } + await handler(req, res); } catch (err) { const error = parseError(err); diff --git a/web-server/src/content/Dashboards/ConfigureGithubModalBody.tsx b/web-server/src/content/Dashboards/ConfigureGithubModalBody.tsx index e1c36692..eb841df5 100644 --- a/web-server/src/content/Dashboards/ConfigureGithubModalBody.tsx +++ b/web-server/src/content/Dashboards/ConfigureGithubModalBody.tsx @@ -8,6 +8,8 @@ import { FlexBox } from '@/components/FlexBox'; import { Line } from '@/components/Text'; import { Integration } from '@/constants/integrations'; import { useAuth } from '@/hooks/useAuth'; +// CLUSTOX: server-resolved workspace, immune to stale persisted redux state. +import { useClustoxUser } from '@/hooks/useClustoxUser'; import { useBoolState, useEasyState } from '@/hooks/useEasyState'; import { fetchCurrentOrg } from '@/slices/auth'; import { fetchTeams } from '@/slices/team'; @@ -24,7 +26,12 @@ export const ConfigureGithubModalBody: FC<{ onClose: () => void; }> = ({ onClose }) => { const token = useEasyState(''); - const { orgId } = useAuth(); + const { orgId: contextOrgId } = useAuth(); + // CLUSTOX: prefer the server-resolved workspace. The redux copy is persisted + // by redux-persist and can rehydrate empty from a previous session, which + // previously produced a POST to /orgs/undefined/integration. + const { orgId: sessionOrgId } = useClustoxUser(); + const orgId = sessionOrgId ?? contextOrgId; const { enqueueSnackbar } = useSnackbar(); const customDomain = useEasyState(''); const dispatch = useDispatch(); @@ -69,6 +76,11 @@ export const ConfigureGithubModalBody: FC<{ return; } + if (!orgId) { + setError('No workspace selected. Reload the page and try again.'); + return; + } + isLoading.true(); try { const isValid = await checkGitHubValidity( diff --git a/web-server/src/content/Dashboards/ConfigureGitlabModalBody.tsx b/web-server/src/content/Dashboards/ConfigureGitlabModalBody.tsx index eef8a094..e73d0cf8 100644 --- a/web-server/src/content/Dashboards/ConfigureGitlabModalBody.tsx +++ b/web-server/src/content/Dashboards/ConfigureGitlabModalBody.tsx @@ -8,6 +8,8 @@ import { FlexBox } from '@/components/FlexBox'; import { Line } from '@/components/Text'; import { Integration } from '@/constants/integrations'; import { useAuth } from '@/hooks/useAuth'; +// CLUSTOX: server-resolved workspace, immune to stale persisted redux state. +import { useClustoxUser } from '@/hooks/useClustoxUser'; import { useBoolState, useEasyState } from '@/hooks/useEasyState'; import { fetchCurrentOrg } from '@/slices/auth'; import { fetchTeams } from '@/slices/team'; @@ -25,7 +27,12 @@ export const ConfigureGitlabModalBody: FC<{ }> = ({ onClose }) => { const token = useEasyState(''); const customDomain = useEasyState(''); - const { orgId } = useAuth(); + const { orgId: contextOrgId } = useAuth(); + // CLUSTOX: prefer the server-resolved workspace. The redux copy is persisted + // by redux-persist and can rehydrate empty from a previous session, which + // previously produced a POST to /orgs/undefined/integration. + const { orgId: sessionOrgId } = useClustoxUser(); + const orgId = sessionOrgId ?? contextOrgId; const { enqueueSnackbar } = useSnackbar(); const dispatch = useDispatch(); const isLoading = useBoolState(); @@ -68,6 +75,12 @@ export const ConfigureGitlabModalBody: FC<{ setDomainError('Please enter a valid domain'); throw Error('Invalid domain'); } + + // CLUSTOX: refuse rather than POST to /orgs/undefined/integration. + if (!orgId) { + setScopeError('No workspace selected. Reload the page and try again.'); + throw Error('No workspace'); + } } catch (e) { console.error(e); return; diff --git a/web-server/src/hooks/useClustoxUser.ts b/web-server/src/hooks/useClustoxUser.ts index f1a7b90a..8677696a 100644 --- a/web-server/src/hooks/useClustoxUser.ts +++ b/web-server/src/hooks/useClustoxUser.ts @@ -7,6 +7,8 @@ export type ClustoxUser = { email: string; name: string; role: ClustoxRole; + /** Workspace to act on. Server-resolved, never persisted client-side. */ + org_id: string | null; }; /** @@ -38,6 +40,7 @@ export const useClustoxUser = () => { return { user, loading, - isSuperadmin: user?.role === 'SUPERADMIN' + isSuperadmin: user?.role === 'SUPERADMIN', + orgId: user?.org_id ?? null }; }; From 04f91409b6be462a5a7d717ee6c8bd0cfcd67419 Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Thu, 6 Aug 2026 20:33:37 +0500 Subject: [PATCH 026/194] fix(tenancy): let admins manage their own workspace, and close a cross-tenant hole Creating a team in your own workspace returned 403. I had restricted team creation and integration removal to superadmins, which was right under the old shared-workspace model and wrong under this one: an admin owns their workspace, so teams and integrations inside it are theirs to manage. Shahzad confirmed admins connect their own integrations. Both restrictions removed. The central check on org_id already confines an admin to their own workspace, which is the control that matters. Removing them surfaced a worse problem. teams/v2 names the team 'id', not 'team_id', so the central team scoping never saw it. An admin could pass their own org_id in the path -- satisfying the workspace check -- together with a team id from another workspace, and edit or delete it. PATCH and DELETE now assert team access explicitly. That is the cost of a central guard keyed on a parameter name: it covers everything that follows the convention and silently misses everything that does not. Verified live as an admin: create team in own workspace -> 200 create team in another workspace -> 403 delete another workspace's team via own org_id in the path -> 403, target team untouched 165 tests pass. --- web-server/pages/api/integrations/index.ts | 9 +++------ .../api/resources/orgs/[org_id]/teams/v2.ts | 20 +++++++++++++++---- 2 files changed, 19 insertions(+), 10 deletions(-) diff --git a/web-server/pages/api/integrations/index.ts b/web-server/pages/api/integrations/index.ts index ce22d085..bc3c5845 100644 --- a/web-server/pages/api/integrations/index.ts +++ b/web-server/pages/api/integrations/index.ts @@ -1,8 +1,6 @@ import * as yup from 'yup'; import { Endpoint, nullSchema } from '@/api-helpers/global'; -// CLUSTOX: integration management is superadmin-only. -import { assertRole } from '@/auth/guard'; import { Columns, Table } from '@/constants/db'; import { Integration } from '@/constants/integrations'; import { db, getFirstRow } from '@/utils/db'; @@ -27,10 +25,9 @@ endpoint.handle.GET(getSchema, async (req, res) => { }); endpoint.handle.DELETE(deleteSchema, async (req, res) => { - // CLUSTOX: unlinking a code provider wipes the org's synced data source. - // Superadmin only. - assertRole((req as any).session, 'SUPERADMIN'); - + // CLUSTOX: admins connect their own integrations, so unlinking one is theirs + // to do as well. Endpoint.serve() confines them to their own workspace via + // the org_id in the payload. const data = await db(Table.Integration) .delete() .where({ diff --git a/web-server/pages/api/resources/orgs/[org_id]/teams/v2.ts b/web-server/pages/api/resources/orgs/[org_id]/teams/v2.ts index c437d604..a6da93cc 100644 --- a/web-server/pages/api/resources/orgs/[org_id]/teams/v2.ts +++ b/web-server/pages/api/resources/orgs/[org_id]/teams/v2.ts @@ -13,8 +13,9 @@ import { } from '@/api/resources/orgs/[org_id]/onboarding'; import { handleRequest } from '@/api-helpers/axios'; import { Endpoint } from '@/api-helpers/global'; -// CLUSTOX: per-team scoping and superadmin-only team creation. -import { assertRole, getAccessibleTeamIds } from '@/auth/guard'; +// CLUSTOX: workspace scoping for the team picker, plus explicit team +// checks on routes that name the team `id` rather than `team_id`. +import { assertTeamAccess, getAccessibleTeamIds } from '@/auth/guard'; import { Row, Table } from '@/constants/db'; import { CIProvider, @@ -107,8 +108,9 @@ endpoint.handle.GET(getSchema, async (req, res) => { }); endpoint.handle.POST(postSchema, async (req, res) => { - // CLUSTOX: only superadmins create teams. - assertRole((req as any).session, 'SUPERADMIN'); + // CLUSTOX: an admin owns their workspace, so creating teams inside it is + // theirs to do. Endpoint.serve() has already confirmed the caller may act on + // this org_id, which is what confines them to their own workspace. if (req.meta?.features?.use_mock_data) { return res.send(getTeamV2Mock); } @@ -163,6 +165,11 @@ endpoint.handle.PATCH(patchSchema, async (req, res) => { } const { org_id, id, name, org_repos } = req.payload; + + // CLUSTOX: this route names the team `id`, not `team_id`, so the central + // scoping in Endpoint.serve() does not see it. Without this an admin could + // pass their own org_id in the path and a team id from another workspace. + await assertTeamAccess((req as any).session, id); const orgReposList: ReqRepoWithProvider[] = []; forEachObjIndexed((repos, org) => { repos.forEach((repo) => { @@ -203,6 +210,11 @@ endpoint.handle.DELETE(deleteSchema, async (req, res) => { return res.send(getTeamV2Mock); } + // CLUSTOX: as with PATCH above, the team arrives as `id` so the central + // scoping does not cover it. Deleting another workspace's team would + // otherwise be reachable by guessing a uuid. + await assertTeamAccess((req as any).session, req.payload.id); + const data = await dbRaw.transaction(async (trx) => { const deletedTeamRow = await trx('Team') .update({ From 1796a2665b491e6c7ecf7fe9cfa80b34fa86eb0f Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Thu, 6 Aug 2026 20:42:58 +0500 Subject: [PATCH 027/194] feat(tenancy): sync every workspace, and record the outcome of each Upstream synced one organisation because only one existed. Each workspace now brings its own integration and repositories, so each needs its own sync -- against its own provider token, and therefore its own rate limit. Workspaces are isolated from one another: one failing must not stop the rest, the same reasoning upstream already applies per repository inside a sync. Outcomes are now recorded per workspace in ClustoxSyncRun. Upstream's POST /sync returned 200 whether every repo synced or every repo failed, and the only record was a log file capped at 512KB with no rotation history -- we lost real time to that during setup. With one sync per workspace it becomes untenable, since a single failing workspace would be invisible among the ones that worked. The response now reports 'synced N/M workspaces', and GET /sync/status returns the latest outcome per workspace. Verified live across three workspaces: 3/3 synced, each with its own timing, and data stayed partitioned -- 3 repos and 412 PRs in the workspace that has an integration, nothing in the two that do not. 159 pytest tests pass, black and flake8 clean. --- backend/analytics_server/mhq/api/sync.py | 78 +++++++-- .../mhq/clustox_auth/sync_run.py | 113 +++++++++++++ .../analytics_server/mhq/store/repos/core.py | 10 ++ .../tests/clustox_auth/test_sync_run.py | 156 ++++++++++++++++++ .../20260806160000_clustox_sync_runs.sql | 27 +++ database-docker/db/schema.sql | 41 ++++- 6 files changed, 413 insertions(+), 12 deletions(-) create mode 100644 backend/analytics_server/mhq/clustox_auth/sync_run.py create mode 100644 backend/analytics_server/tests/clustox_auth/test_sync_run.py create mode 100644 database-docker/db/migrations/20260806160000_clustox_sync_runs.sql diff --git a/backend/analytics_server/mhq/api/sync.py b/backend/analytics_server/mhq/api/sync.py index e626e7df..5898f564 100644 --- a/backend/analytics_server/mhq/api/sync.py +++ b/backend/analytics_server/mhq/api/sync.py @@ -1,7 +1,15 @@ -from flask import Blueprint, jsonify +from typing import List -from mhq.service.query_validator import get_query_validator +from flask import Blueprint + +from mhq.clustox_auth.sync_run import ( + SyncRunStatus, + as_dict, + get_sync_run_service, +) from mhq.service.sync_data import trigger_data_sync +from mhq.store.models.core import Organization +from mhq.store.repos.core import CoreRepoService from mhq.utils.lock import get_redis_lock_service from mhq.utils.log import LOG from mhq.utils.time import time_now @@ -11,14 +19,62 @@ @app.route("/sync", methods=["POST"]) def sync(): - default_org = get_query_validator().get_default_org() - if not default_org: - return jsonify({"message": "Default org not found"}), 404 - org_id = str(default_org.id) - with get_redis_lock_service().acquire_lock("{org}:" + f"{str(org_id)}:data_sync"): + """ + CLUSTOX: sync every workspace, not just the default one. + + Upstream synced a single organisation, because only one ever existed. Each + workspace now brings its own integration and its own repositories, so each + needs its own sync -- against its own provider token and therefore its own + rate limit. + + Workspaces are isolated from one another: one failing must not stop the + rest, which is the same reasoning upstream applies per repository inside a + sync. The difference is that the outcome is recorded per workspace instead + of being lost to a rotating log file. + """ + orgs: List[Organization] = CoreRepoService().get_all_orgs() + + if not orgs: + return {"message": "no workspaces to sync", "results": []} + + sync_run_service = get_sync_run_service() + lock_service = get_redis_lock_service() + results = [] + + for org in orgs: + org_id = str(org.id) + run = sync_run_service.start(org_id) + try: - trigger_data_sync(org_id) + # Per-workspace lock: a long sync in one workspace must not block + # another, and a repeat trigger must not run the same workspace + # twice concurrently. + with lock_service.acquire_lock("{org}:" + f"{org_id}:data_sync"): + trigger_data_sync(org_id) + + sync_run_service.finish(run, SyncRunStatus.SUCCESS) + LOG.info(f"[CLUSTOX] Synced workspace {org.name} ({org_id})") + except Exception as e: - LOG.error(f"Error syncing data for org {org_id}: {str(e)}") - return {"message": "sync failed", "time": time_now().isoformat()}, 500 - return {"message": "sync started", "time": time_now().isoformat()} + detail = f"{type(e).__name__}: {str(e)}" + sync_run_service.finish(run, SyncRunStatus.FAILED, detail) + LOG.error(f"[CLUSTOX] Sync failed for workspace {org_id}: {detail}") + + latest = sync_run_service.latest_for_org(org_id) + if latest: + results.append(as_dict(latest)) + + failed = [r for r in results if r["status"] == SyncRunStatus.FAILED] + + return { + "message": f"synced {len(results) - len(failed)}/{len(results)} workspaces", + "time": time_now().isoformat(), + "results": results, + } + + +@app.route("/sync/status", methods=["GET"]) +def sync_status(): + """Latest sync outcome per workspace.""" + runs = get_sync_run_service().latest_per_org() + return {"results": [as_dict(r) for r in runs]} diff --git a/backend/analytics_server/mhq/clustox_auth/sync_run.py b/backend/analytics_server/mhq/clustox_auth/sync_run.py new file mode 100644 index 00000000..c05768c0 --- /dev/null +++ b/backend/analytics_server/mhq/clustox_auth/sync_run.py @@ -0,0 +1,113 @@ +""" +CLUSTOX: per-workspace sync outcomes. + +Upstream syncs a single organisation and reports success regardless of what +happened inside it. With one sync per workspace that is untenable: a failing +workspace would be invisible among the ones that worked. + +Every workspace sync records a row here, so "did last night's sync work?" is a +query rather than an exercise in reading log files before they rotate. +""" + +from datetime import datetime +from typing import List, Optional + +from sqlalchemy import Column, DateTime, String, Text, func +from sqlalchemy.dialects.postgresql import UUID + +from mhq.store import db, rollback_on_exc +from mhq.utils.time import time_now + + +class SyncRunStatus: + RUNNING = "RUNNING" + SUCCESS = "SUCCESS" + FAILED = "FAILED" + SKIPPED = "SKIPPED" + + +class ClustoxSyncRun(db.Model): + __tablename__ = "ClustoxSyncRun" + + id = Column(UUID(as_uuid=True), primary_key=True) + org_id = Column(UUID(as_uuid=True), nullable=False) + started_at = Column(DateTime(timezone=True), server_default=func.now()) + finished_at = Column(DateTime(timezone=True)) + status = Column(String, nullable=False) + detail = Column(Text) + + +class SyncRunService: + def __init__(self): + self._db = db + + @rollback_on_exc + def start(self, org_id: str) -> ClustoxSyncRun: + from mhq.utils.string import uuid4_str + + run = ClustoxSyncRun( + id=uuid4_str(), + org_id=org_id, + started_at=time_now(), + status=SyncRunStatus.RUNNING, + ) + self._db.session.add(run) + self._db.session.commit() + return run + + @rollback_on_exc + def finish( + self, run: ClustoxSyncRun, status: str, detail: Optional[str] = None + ) -> None: + run.status = status + run.finished_at = time_now() + # Truncated: a stack trace helps, an unbounded one bloats the row. + run.detail = detail[:2000] if detail else None + self._db.session.commit() + + @rollback_on_exc + def latest_per_org(self) -> List[ClustoxSyncRun]: + """Most recent run for each workspace.""" + subq = ( + self._db.session.query( + ClustoxSyncRun.org_id, + func.max(ClustoxSyncRun.started_at).label("latest"), + ) + .group_by(ClustoxSyncRun.org_id) + .subquery() + ) + return ( + self._db.session.query(ClustoxSyncRun) + .join( + subq, + (ClustoxSyncRun.org_id == subq.c.org_id) + & (ClustoxSyncRun.started_at == subq.c.latest), + ) + .all() + ) + + @rollback_on_exc + def latest_for_org(self, org_id: str) -> Optional[ClustoxSyncRun]: + return ( + self._db.session.query(ClustoxSyncRun) + .filter(ClustoxSyncRun.org_id == org_id) + .order_by(ClustoxSyncRun.started_at.desc()) + .first() + ) + + +def get_sync_run_service() -> SyncRunService: + return SyncRunService() + + +def as_dict(run: ClustoxSyncRun) -> dict: + def iso(value: Optional[datetime]) -> Optional[str]: + return value.isoformat() if value else None + + return { + "org_id": str(run.org_id), + "started_at": iso(run.started_at), + "finished_at": iso(run.finished_at), + "status": run.status, + "detail": run.detail, + } diff --git a/backend/analytics_server/mhq/store/repos/core.py b/backend/analytics_server/mhq/store/repos/core.py index fe4d3e03..c19dc969 100644 --- a/backend/analytics_server/mhq/store/repos/core.py +++ b/backend/analytics_server/mhq/store/repos/core.py @@ -21,6 +21,16 @@ def get_org(self, org_id): .one_or_none() ) + # CLUSTOX: every workspace. Upstream never needed this because only one + # Organization ever existed; multitenancy syncs each of them. + @rollback_on_exc + def get_all_orgs(self) -> List[Organization]: + return ( + self._db.session.query(Organization) + .order_by(Organization.created_at.asc()) + .all() + ) + @rollback_on_exc def get_org_by_name(self, org_name: str): return ( diff --git a/backend/analytics_server/tests/clustox_auth/test_sync_run.py b/backend/analytics_server/tests/clustox_auth/test_sync_run.py new file mode 100644 index 00000000..59e59dbd --- /dev/null +++ b/backend/analytics_server/tests/clustox_auth/test_sync_run.py @@ -0,0 +1,156 @@ +""" +CLUSTOX: workspace isolation during sync. + +The behaviour that matters is that one workspace failing does not prevent the +others from syncing, and that the failure is recorded rather than swallowed. +Upstream returned 200 whatever happened inside. +""" + +from unittest.mock import MagicMock, patch + +import pytest + + +class FakeRun: + def __init__(self, org_id): + self.org_id = org_id + self.status = "RUNNING" + self.detail = None + self.started_at = None + self.finished_at = None + + +class FakeSyncRunService: + """Records outcomes in memory so assertions do not need a database.""" + + def __init__(self): + self.runs = {} + + def start(self, org_id): + run = FakeRun(org_id) + self.runs[org_id] = run + return run + + def finish(self, run, status, detail=None): + run.status = status + run.detail = detail + + def latest_for_org(self, org_id): + return self.runs.get(org_id) + + +class FakeOrg: + def __init__(self, org_id, name): + self.id = org_id + self.name = name + + +class FakeLock: + def __enter__(self): + return self + + def __exit__(self, *args): + return False + + +class FakeLockService: + def acquire_lock(self, _key): + return FakeLock() + + +@pytest.fixture +def sync_module(): + from mhq.api import sync as sync_module + + return sync_module + + +def build_app(sync_module): + from flask import Flask + + app = Flask(__name__) + app.register_blueprint(sync_module.app) + return app + + +def test_one_failing_workspace_does_not_stop_the_others(sync_module): + orgs = [FakeOrg("org-a", "A"), FakeOrg("org-b", "B"), FakeOrg("org-c", "C")] + fake_service = FakeSyncRunService() + + def trigger(org_id): + if org_id == "org-b": + raise RuntimeError("provider token rejected") + + with patch.object(sync_module, "CoreRepoService") as repo_cls, patch.object( + sync_module, "get_sync_run_service", return_value=fake_service + ), patch.object( + sync_module, "get_redis_lock_service", return_value=FakeLockService() + ), patch.object( + sync_module, "trigger_data_sync", side_effect=trigger + ) as trigger_mock: + repo_cls.return_value = MagicMock(get_all_orgs=lambda: orgs) + + client = build_app(sync_module).test_client() + res = client.post("/sync") + + assert res.status_code == 200 + # every workspace was attempted, including the one after the failure + assert trigger_mock.call_count == 3 + + assert fake_service.runs["org-a"].status == "SUCCESS" + assert fake_service.runs["org-b"].status == "FAILED" + assert fake_service.runs["org-c"].status == "SUCCESS" + + +def test_failure_detail_is_recorded_not_swallowed(sync_module): + orgs = [FakeOrg("org-a", "A")] + fake_service = FakeSyncRunService() + + with patch.object(sync_module, "CoreRepoService") as repo_cls, patch.object( + sync_module, "get_sync_run_service", return_value=fake_service + ), patch.object( + sync_module, "get_redis_lock_service", return_value=FakeLockService() + ), patch.object( + sync_module, + "trigger_data_sync", + side_effect=RuntimeError("provider token rejected"), + ): + repo_cls.return_value = MagicMock(get_all_orgs=lambda: orgs) + + client = build_app(sync_module).test_client() + res = client.post("/sync") + + body = res.get_json() + assert "0/1" in body["message"] + assert fake_service.runs["org-a"].status == "FAILED" + assert "provider token rejected" in fake_service.runs["org-a"].detail + + +def test_reports_how_many_workspaces_succeeded(sync_module): + orgs = [FakeOrg("org-a", "A"), FakeOrg("org-b", "B")] + fake_service = FakeSyncRunService() + + with patch.object(sync_module, "CoreRepoService") as repo_cls, patch.object( + sync_module, "get_sync_run_service", return_value=fake_service + ), patch.object( + sync_module, "get_redis_lock_service", return_value=FakeLockService() + ), patch.object( + sync_module, "trigger_data_sync" + ): + repo_cls.return_value = MagicMock(get_all_orgs=lambda: orgs) + + client = build_app(sync_module).test_client() + res = client.post("/sync") + + assert "2/2" in res.get_json()["message"] + + +def test_no_workspaces_is_not_an_error(sync_module): + with patch.object(sync_module, "CoreRepoService") as repo_cls: + repo_cls.return_value = MagicMock(get_all_orgs=lambda: []) + + client = build_app(sync_module).test_client() + res = client.post("/sync") + + assert res.status_code == 200 + assert res.get_json()["results"] == [] diff --git a/database-docker/db/migrations/20260806160000_clustox_sync_runs.sql b/database-docker/db/migrations/20260806160000_clustox_sync_runs.sql new file mode 100644 index 00000000..29bbead1 --- /dev/null +++ b/database-docker/db/migrations/20260806160000_clustox_sync_runs.sql @@ -0,0 +1,27 @@ +-- migrate:up +-- CLUSTOX: per-workspace sync outcomes. +-- +-- Upstream's POST /sync returns 200 whether every repo synced or every repo +-- failed, and the only record of what happened is a log file capped at 512KB +-- with no rotation history. "Sync succeeded but the numbers are wrong" was the +-- expected failure signature, and diagnosing it meant reading logs before they +-- rotated away. +-- +-- With one workspace that was merely awkward. With one sync per workspace it +-- becomes untenable: a single failing workspace is invisible among the ones +-- that worked. +CREATE TABLE "ClustoxSyncRun" ( + id uuid PRIMARY KEY DEFAULT extensions.uuid_generate_v4(), + org_id uuid NOT NULL REFERENCES "Organization"(id) ON DELETE CASCADE, + started_at timestamptz NOT NULL DEFAULT now(), + finished_at timestamptz, + status varchar NOT NULL CHECK (status IN ('RUNNING','SUCCESS','FAILED','SKIPPED')), + -- Why a run was skipped, or how it failed. Null on success. + detail text +); + +CREATE INDEX idx_clustox_sync_run_org_started + ON "ClustoxSyncRun"(org_id, started_at DESC); + +-- migrate:down +DROP TABLE "ClustoxSyncRun"; diff --git a/database-docker/db/schema.sql b/database-docker/db/schema.sql index 30b6ae31..ac6e518d 100644 --- a/database-docker/db/schema.sql +++ b/database-docker/db/schema.sql @@ -71,6 +71,21 @@ CREATE TABLE public."BookmarkPullRequestRevertPRMapping" ( ); +-- +-- Name: ClustoxSyncRun; Type: TABLE; Schema: public; Owner: - +-- + +CREATE TABLE public."ClustoxSyncRun" ( + id uuid DEFAULT extensions.uuid_generate_v4() NOT NULL, + org_id uuid NOT NULL, + started_at timestamp with time zone DEFAULT now() NOT NULL, + finished_at timestamp with time zone, + status character varying NOT NULL, + detail text, + CONSTRAINT "ClustoxSyncRun_status_check" CHECK (((status)::text = ANY ((ARRAY['RUNNING'::character varying, 'SUCCESS'::character varying, 'FAILED'::character varying, 'SKIPPED'::character varying])::text[]))) +); + + -- -- Name: ClustoxUserAuth; Type: TABLE; Schema: public; Owner: - -- @@ -507,6 +522,14 @@ CREATE TABLE public.schema_migrations ( ); +-- +-- Name: ClustoxSyncRun ClustoxSyncRun_pkey; Type: CONSTRAINT; Schema: public; Owner: - +-- + +ALTER TABLE ONLY public."ClustoxSyncRun" + ADD CONSTRAINT "ClustoxSyncRun_pkey" PRIMARY KEY (id); + + -- -- Name: ClustoxUserAuth ClustoxUserAuth_pkey; Type: CONSTRAINT; Schema: public; Owner: - -- @@ -845,6 +868,13 @@ CREATE INDEX "TeamIncidentService_team_id_index" ON public."TeamIncidentService" CREATE INDEX "Team_org_idx" ON public."Team" USING btree (org_id); +-- +-- Name: idx_clustox_sync_run_org_started; Type: INDEX; Schema: public; Owner: - +-- + +CREATE INDEX idx_clustox_sync_run_org_started ON public."ClustoxSyncRun" USING btree (org_id, started_at DESC); + + -- -- Name: idx_clustox_user_team_access_user; Type: INDEX; Schema: public; Owner: - -- @@ -1155,6 +1185,14 @@ ALTER TABLE ONLY public."Bookmark" ADD CONSTRAINT "Bookmark_repo_id_fkey" FOREIGN KEY (repo_id) REFERENCES public."OrgRepo"(id); +-- +-- Name: ClustoxSyncRun ClustoxSyncRun_org_id_fkey; Type: FK CONSTRAINT; Schema: public; Owner: - +-- + +ALTER TABLE ONLY public."ClustoxSyncRun" + ADD CONSTRAINT "ClustoxSyncRun_org_id_fkey" FOREIGN KEY (org_id) REFERENCES public."Organization"(id) ON DELETE CASCADE; + + -- -- Name: ClustoxUserAuth ClustoxUserAuth_user_id_fkey; Type: FK CONSTRAINT; Schema: public; Owner: - -- @@ -1434,4 +1472,5 @@ INSERT INTO public.schema_migrations (version) VALUES ('20240503060203'), ('20240503073715'), ('20260805150000'), - ('20260806120000'); + ('20260806120000'), + ('20260806160000'); From 4972c800733bc7570c28bace9d295399e4408731 Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Thu, 6 Aug 2026 21:00:25 +0500 Subject: [PATCH 028/194] feat(tenancy): surface sync health per workspace The sync outcomes recorded in the previous commit were queryable but not visible anywhere, so 'did last night's sync work?' still meant opening a database. That was the original complaint. Adds a Workspaces page showing, per workspace: last sync outcome, how long ago, repo and team counts, and the owning admin. - superadmins see every workspace and can force a sync across all of them; admins see only their own and cannot - 'Not connected' is distinguished from 'Never synced'. A workspace with no integration has nothing to sync, which is not a failure, and conflating the two is what makes a dashboard untrustworthy - a failed sync shows its recorded error on hover rather than requiring a log file - if the sync server is unreachable the page still renders the workspaces without status, since an unknown sync state is more useful than an error screen The page also makes the orphaned workspace obvious: 'default' holds all the real data and shows 'no owner', because no admin owns it yet. Verified live: superadmin sees 3 workspaces, an admin sees only theirs, an admin gets 403 on the force-sync route, and Sync now moved every timestamp to 'less than a minute ago'. --- web-server/pages/api/clustox/sync-now.ts | 21 ++ .../pages/api/clustox/workspace-status.ts | 65 ++++ web-server/pages/workspaces.tsx | 308 ++++++++++++++++++ web-server/src/auth/queries.ts | 48 +++ web-server/src/constants/routes.ts | 4 +- .../Sidebar/SidebarMenu/items.tsx | 15 +- 6 files changed, 457 insertions(+), 4 deletions(-) create mode 100644 web-server/pages/api/clustox/sync-now.ts create mode 100644 web-server/pages/api/clustox/workspace-status.ts create mode 100644 web-server/pages/workspaces.tsx diff --git a/web-server/pages/api/clustox/sync-now.ts b/web-server/pages/api/clustox/sync-now.ts new file mode 100644 index 00000000..48a0cdfc --- /dev/null +++ b/web-server/pages/api/clustox/sync-now.ts @@ -0,0 +1,21 @@ +import { handleSyncServerRequest } from '@/api-helpers/axios'; +import { Endpoint, nullSchema } from '@/api-helpers/global'; +import { assertRole } from '@/auth/guard'; + +/** + * Trigger a sync across every workspace, rather than waiting up to 30 minutes + * for cron. + * + * Superadmin only: this runs every workspace's sync against every workspace's + * provider token, so it is not an action scoped to one admin's workspace. + */ +const endpoint = new Endpoint(nullSchema); + +endpoint.handle.POST(nullSchema, async (req, res) => { + assertRole((req as any).session, 'SUPERADMIN'); + + const body = await handleSyncServerRequest('/sync', { method: 'post' }); + res.send(body); +}); + +export default endpoint.serve(); diff --git a/web-server/pages/api/clustox/workspace-status.ts b/web-server/pages/api/clustox/workspace-status.ts new file mode 100644 index 00000000..10a1e6bb --- /dev/null +++ b/web-server/pages/api/clustox/workspace-status.ts @@ -0,0 +1,65 @@ +import { handleSyncServerRequest } from '@/api-helpers/axios'; +import { Endpoint, nullSchema } from '@/api-helpers/global'; +import { assertAuthenticated, canAccessAllWorkspaces } from '@/auth/guard'; +import { listWorkspaceSummaries } from '@/auth/queries'; + +type SyncRun = { + org_id: string; + started_at: string | null; + finished_at: string | null; + status: 'RUNNING' | 'SUCCESS' | 'FAILED' | 'SKIPPED'; + detail: string | null; +}; + +/** + * Workspaces with their latest sync outcome. + * + * Answers "did last night's sync actually work?" -- which previously required + * reading a log file before it rotated away. + * + * A superadmin sees every workspace; an admin sees only their own, so the same + * page serves both without leaking the existence of other workspaces. + */ +const endpoint = new Endpoint(nullSchema); + +endpoint.handle.GET(nullSchema, async (req, res) => { + const session = assertAuthenticated((req as any).session); + + const summaries = await listWorkspaceSummaries(); + + const visible = canAccessAllWorkspaces(session) + ? summaries + : summaries.filter((w) => w.id === session.orgId); + + // The sync server owns run history. If it is unreachable, report the + // workspaces without status rather than failing the whole page -- an + // unknown sync state is still more useful than an error screen. + let runs: SyncRun[] = []; + try { + const body = await handleSyncServerRequest<{ results: SyncRun[] }>( + '/sync/status' + ); + runs = body?.results ?? []; + } catch { + runs = []; + } + + res.send( + visible.map((w) => { + const run = runs.find((r) => r.org_id === w.id) ?? null; + return { + ...w, + lastSync: run + ? { + status: run.status, + finished_at: run.finished_at, + started_at: run.started_at, + detail: run.detail + } + : null + }; + }) + ); +}); + +export default endpoint.serve(); diff --git a/web-server/pages/workspaces.tsx b/web-server/pages/workspaces.tsx new file mode 100644 index 00000000..f44a8acb --- /dev/null +++ b/web-server/pages/workspaces.tsx @@ -0,0 +1,308 @@ +import { + CheckCircleTwoTone, + ErrorTwoTone, + HelpOutlineTwoTone, + LinkOffTwoTone, + SyncTwoTone, + WorkspacesTwoTone +} from '@mui/icons-material'; +import { + Alert, + Button, + Chip, + CircularProgress, + Table, + TableBody, + TableCell, + TableContainer, + TableHead, + TableRow, + Tooltip, + Typography, + useTheme +} from '@mui/material'; +import { formatDistanceToNow } from 'date-fns'; +import Head from 'next/head'; +import { useSnackbar } from 'notistack'; +import { useCallback, useEffect, useState } from 'react'; +import ExtendedSidebarLayout from 'src/layouts/ExtendedSidebarLayout'; + +import { FlexBox } from '@/components/FlexBox'; +import { Line } from '@/components/Text'; +import { PageWrapper } from '@/content/PullRequests/PageWrapper'; +import { useClustoxUser } from '@/hooks/useClustoxUser'; +import { PageLayout } from '@/types/resources'; + +type SyncStatus = 'RUNNING' | 'SUCCESS' | 'FAILED' | 'SKIPPED'; + +type Workspace = { + id: string; + name: string; + ownerEmail: string | null; + hasIntegration: boolean; + repoCount: number; + teamCount: number; + lastSync: { + status: SyncStatus; + finished_at: string | null; + started_at: string | null; + detail: string | null; + } | null; +}; + +const Mono = ({ children }: { children: React.ReactNode }) => ( + + {children} + +); + +const relative = (iso: string | null) => { + if (!iso) return 'never'; + try { + return `${formatDistanceToNow(new Date(iso))} ago`; + } catch { + return 'unknown'; + } +}; + +function WorkspacesPage() { + const theme = useTheme(); + const { enqueueSnackbar } = useSnackbar(); + const { isSuperadmin, loading: userLoading } = useClustoxUser(); + + const [workspaces, setWorkspaces] = useState([]); + const [loading, setLoading] = useState(true); + const [syncing, setSyncing] = useState(false); + + const load = useCallback(async () => { + const res = await fetch('/api/clustox/workspace-status'); + if (res.ok) setWorkspaces(await res.json()); + setLoading(false); + }, []); + + useEffect(() => { + load(); + }, [load]); + + const syncNow = async () => { + setSyncing(true); + try { + const res = await fetch('/api/clustox/sync-now', { method: 'POST' }); + if (!res.ok) throw new Error('sync failed'); + const body = await res.json(); + enqueueSnackbar(body.message ?? 'Sync finished', { + variant: 'success', + autoHideDuration: 6000 + }); + await load(); + } catch { + enqueueSnackbar('Could not reach the sync server', { variant: 'error' }); + } finally { + setSyncing(false); + } + }; + + const statusChip = (w: Workspace) => { + if (!w.hasIntegration) + return ( + + } + label="Not connected" + /> + + ); + + if (!w.lastSync) + return ( + + } + label="Never synced" + /> + + ); + + if (w.lastSync.status === 'FAILED') + return ( + + } + label="Failed" + /> + + ); + + if (w.lastSync.status === 'RUNNING') + return } label="Running" />; + + return ( + } + label="Synced" + /> + ); + }; + + if (loading || userLoading) + return ( + + + + ); + + const failing = workspaces.filter((w) => w.lastSync?.status === 'FAILED'); + const unconnected = workspaces.filter((w) => !w.hasIntegration); + + return ( + <> + + Workspaces | MiddlewareHQ + + + + + + + {workspaces.length}{' '} + {workspaces.length === 1 ? 'workspace' : 'workspaces'} + + + {failing.length > 0 + ? `${failing.length} failing to sync` + : 'All connected workspaces syncing'} + + + + {isSuperadmin && ( + + )} + + + {failing.length > 0 && ( + + {failing.length === 1 + ? `${failing[0].name} failed its last sync.` + : `${failing.length} workspaces failed their last sync.`}{' '} + Their metrics are stale until it succeeds. + + )} + + {unconnected.length > 0 && failing.length === 0 && ( + + {unconnected.length}{' '} + {unconnected.length === 1 ? 'workspace has' : 'workspaces have'} no + code provider connected yet, so{' '} + {unconnected.length === 1 ? 'it has' : 'they have'} nothing to sync. + + )} + + + + + + Workspace + Last sync + When + + Repos + + + Teams + + + + + {workspaces.map((w) => ( + + + + + + {w.name} + + {w.ownerEmail ?? 'no owner'} + + + + + {statusChip(w)} + + + {w.hasIntegration + ? relative(w.lastSync?.finished_at ?? null) + : '—'} + + + + {w.repoCount} + + + {w.teamCount} + + + ))} + +
+
+ + + Syncs run automatically every 30 minutes. A failed workspace does not + affect the others. + +
+ + ); +} + +WorkspacesPage.getLayout = (page: PageLayout) => ( + + + + Workspaces + + } + pageTitle="Workspaces" + hideAllSelectors + showEvenIfNoTeamSelected + showDate={false} + > + {page} + + +); + +export default WorkspacesPage; diff --git a/web-server/src/auth/queries.ts b/web-server/src/auth/queries.ts index 6a9887e3..adfdc0b4 100644 --- a/web-server/src/auth/queries.ts +++ b/web-server/src/auth/queries.ts @@ -111,6 +111,54 @@ export const listWorkspaces = async (): Promise< })); }; +export type WorkspaceSummary = { + id: string; + name: string; + ownerEmail: string | null; + hasIntegration: boolean; + repoCount: number; + teamCount: number; +}; + +/** + * Every workspace with enough context to answer "is this one healthy?". + * + * Deliberately excludes sync status: that lives on the sync server, so the + * route composes the two rather than this query reaching across a service + * boundary. + */ +export const listWorkspaceSummaries = async (): Promise => { + const [orgs, owners, integrations, repos, teams] = await Promise.all([ + db(Table.Organization).select('id', 'name').orderBy('created_at', 'asc'), + db(Table.ClustoxUserAuth) + .join(Table.Users, `${Table.Users}.id`, `${Table.ClustoxUserAuth}.user_id`) + .where(`${Table.ClustoxUserAuth}.role`, 'ADMIN') + .andWhere(`${Table.Users}.is_deleted`, false) + .select(`${Table.Users}.org_id`, `${Table.Users}.primary_email`), + db(Table.Integration) + .whereNotNull('access_token_enc_chunks') + .select('org_id'), + db(Table.OrgRepo).where('is_active', true).select('org_id'), + db(Table.Team).where('is_deleted', false).select('org_id') + ]); + + const countBy = (rows: { org_id: string }[], orgId: string) => + rows.filter((r) => r.org_id === orgId).length; + + return orgs.map((o: { id: string; name: string }) => ({ + id: o.id, + name: o.name, + ownerEmail: + owners.find((u: { org_id: string }) => u.org_id === o.id) + ?.primary_email ?? null, + hasIntegration: integrations.some( + (i: { org_id: string }) => i.org_id === o.id + ), + repoCount: countBy(repos, o.id), + teamCount: countBy(teams, o.id) + })); +}; + /** Does this workspace exist? Guards the SuperAdmin's selected workspace. */ export const workspaceExists = async (orgId: string): Promise => { const row = await db(Table.Organization).select('id').where('id', orgId).first(); diff --git a/web-server/src/constants/routes.ts b/web-server/src/constants/routes.ts index a65aee91..36d4ad92 100644 --- a/web-server/src/constants/routes.ts +++ b/web-server/src/constants/routes.ts @@ -71,7 +71,9 @@ export const ROUTES = { SETTINGS: new RoutePath('SETTINGS'), SYSTEM_LOGS: new RoutePath('SYSTEM_LOGS'), // CLUSTOX: user management (superadmin only) - USERS: new RoutePath('USERS') + USERS: new RoutePath('USERS'), + // CLUSTOX: workspace + sync health + WORKSPACES: new RoutePath('WORKSPACES') }; export const DEFAULT_HOME_ROUTE = ROUTES.DORA_METRICS; diff --git a/web-server/src/layouts/ExtendedSidebarLayout/Sidebar/SidebarMenu/items.tsx b/web-server/src/layouts/ExtendedSidebarLayout/Sidebar/SidebarMenu/items.tsx index a7ac0243..b5e7e0dd 100644 --- a/web-server/src/layouts/ExtendedSidebarLayout/Sidebar/SidebarMenu/items.tsx +++ b/web-server/src/layouts/ExtendedSidebarLayout/Sidebar/SidebarMenu/items.tsx @@ -4,8 +4,9 @@ import { Analytics, Settings, Dns, - // CLUSTOX: icon for the user management item - ManageAccounts + // CLUSTOX: icons for user management and workspace health + ManageAccounts, + Workspaces } from '@mui/icons-material'; import { ROUTES } from '@/constants/routes'; @@ -44,7 +45,9 @@ export enum SideBarItems { MANAGE_INTEGRATIONS = 'Manage Integrations', SERVER_ADMIN = 'Server Admin', // CLUSTOX: superadmin-only, filtered in useFilteredSidebarItems - MANAGE_USERS = 'Manage Users' + MANAGE_USERS = 'Manage Users', + // CLUSTOX: visible to admins too, scoped to their own workspace + WORKSPACES = 'Workspaces' } const menuItems = (): MenuItems[] => [ @@ -76,6 +79,12 @@ const menuItems = (): MenuItems[] => [ icon: Dns, link: ROUTES.SYSTEM_LOGS.PATH }, + // CLUSTOX: sync health. Admins see their own workspace, superadmins all. + { + name: SideBarItems.WORKSPACES, + icon: Workspaces, + link: ROUTES.WORKSPACES.PATH + }, // CLUSTOX: only rendered for superadmins. { name: SideBarItems.MANAGE_USERS, From f0c5522caedc0df94ffa6b73e88e90650df0fb20 Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Fri, 7 Aug 2026 00:38:38 +0500 Subject: [PATCH 029/194] feat(tenancy): let an admin adopt an existing workspace The pre-multitenancy workspace held the real GitHub connection, three repos and 412 pull requests, but predated the idea of ownership, so no admin owned it. Only a superadmin could reach it, and the Workspaces page advertised the problem by showing 'no owner'. Creating an admin can now adopt an unowned workspace instead of always provisioning a new one, so its integration and repositories carry over rather than being recreated. An admin can also be moved between workspaces afterwards. Promoting someone to superadmin now clears their workspace. A user who both sits above every workspace and owns one is a state no part of the model expects, and it would have made their own workspace invisible to them via the switcher fallback. Verified live: adopting 'default' created no new workspace, the adopting admin inherited 3 repos and 3 teams, and still gets 403 on another workspace. --- .../pages/api/clustox/users/[user_id].ts | 27 +++++++- web-server/pages/api/clustox/workspaces.ts | 13 +++- web-server/pages/users.tsx | 64 ++++++++++++++++--- web-server/src/auth/queries.ts | 34 ++++++++++ 4 files changed, 125 insertions(+), 13 deletions(-) diff --git a/web-server/pages/api/clustox/users/[user_id].ts b/web-server/pages/api/clustox/users/[user_id].ts index 5f416c1a..7193d168 100644 --- a/web-server/pages/api/clustox/users/[user_id].ts +++ b/web-server/pages/api/clustox/users/[user_id].ts @@ -2,7 +2,12 @@ import * as yup from 'yup'; import { Endpoint } from '@/api-helpers/global'; import { assertRole } from '@/auth/guard'; -import { countSuperadmins, setUserTeams, updateUserRole } from '@/auth/queries'; +import { + countSuperadmins, + setUserTeams, + setUserWorkspace, + updateUserRole +} from '@/auth/queries'; import { ClustoxRole } from '@/auth/types'; import { Errors, ResponseError } from '@/constants/error'; @@ -13,7 +18,10 @@ const pathSchema = yup.object().shape({ const patchSchema = yup.object().shape({ user_id: yup.string().uuid().required(), role: yup.string().oneOf(['SUPERADMIN', 'ADMIN']).optional(), - team_ids: yup.array().of(yup.string().uuid()).optional() + team_ids: yup.array().of(yup.string().uuid()).optional(), + // Move an admin into a workspace. Used to adopt a workspace that has no + // owner, rather than recreating its integration and repositories. + org_id: yup.string().uuid().nullable().optional() }); const endpoint = new Endpoint(pathSchema); @@ -32,8 +40,21 @@ endpoint.handle.PATCH(patchSchema, async (req, res) => { if (req.payload.role) { // yup's .oneOf() widens to string; the schema has already validated it. - await updateUserRole(req.payload.user_id, req.payload.role as ClustoxRole); + const role = req.payload.role as ClustoxRole; + await updateUserRole(req.payload.user_id, role); + + // A superadmin owns no workspace. Promoting without clearing it would + // leave a user who both sits above every workspace and owns one, which + // no part of the model expects. + if (role === 'SUPERADMIN') { + await setUserWorkspace(req.payload.user_id, null); + } } + + if (req.payload.org_id !== undefined && req.payload.role !== 'SUPERADMIN') { + await setUserWorkspace(req.payload.user_id, req.payload.org_id ?? null); + } + if (req.payload.team_ids) { await setUserTeams(req.payload.user_id, req.payload.team_ids); } diff --git a/web-server/pages/api/clustox/workspaces.ts b/web-server/pages/api/clustox/workspaces.ts index 1e1ccf83..fca3c522 100644 --- a/web-server/pages/api/clustox/workspaces.ts +++ b/web-server/pages/api/clustox/workspaces.ts @@ -2,7 +2,7 @@ import * as yup from 'yup'; import { Endpoint, nullSchema } from '@/api-helpers/global'; import { assertRole } from '@/auth/guard'; -import { listWorkspaces } from '@/auth/queries'; +import { listUnownedWorkspaces, listWorkspaces } from '@/auth/queries'; /** * Every workspace, for the SuperAdmin workspace switcher. @@ -16,7 +16,16 @@ const endpoint = new Endpoint(nullSchema); endpoint.handle.GET(nullSchema, async (req, res) => { assertRole((req as any).session, 'SUPERADMIN'); - res.send(await listWorkspaces()); + + const [all, unowned] = await Promise.all([ + listWorkspaces(), + listUnownedWorkspaces() + ]); + const unownedIds = new Set(unowned.map((w) => w.id)); + + // `owned` lets the user form offer adoption of a workspace that has no + // admin -- notably the pre-multitenancy one holding the real data. + res.send(all.map((w) => ({ ...w, owned: !unownedIds.has(w.id) }))); }); const putSchema = yup.object().shape({ diff --git a/web-server/pages/users.tsx b/web-server/pages/users.tsx index a73789b4..9b50e5b5 100644 --- a/web-server/pages/users.tsx +++ b/web-server/pages/users.tsx @@ -56,11 +56,16 @@ type UserRow = { const MIN_PASSWORD = 12; +const NEW_WORKSPACE = '__new__'; + const emptyForm = { name: '', email: '', password: '', - role: 'ADMIN' as ClustoxRole + role: 'ADMIN' as ClustoxRole, + // NEW_WORKSPACE provisions a fresh one; otherwise adopt an existing + // workspace that has no owner. + org_id: NEW_WORKSPACE }; const initials = (name: string, email: string) => { @@ -98,6 +103,9 @@ function UsersPage() { const { enqueueSnackbar } = useSnackbar(); const [users, setUsers] = useState([]); + const [workspaces, setWorkspaces] = useState< + { id: string; name: string; owned: boolean }[] + >([]); const [loading, setLoading] = useState(true); const [forbidden, setForbidden] = useState(false); @@ -120,9 +128,15 @@ function UsersPage() { setLoading(false); }, []); + const loadWorkspaces = useCallback(async () => { + const res = await fetch('/api/clustox/workspaces'); + if (res.ok) setWorkspaces(await res.json()); + }, []); + useEffect(() => { loadUsers(); - }, [loadUsers]); + loadWorkspaces(); + }, [loadUsers, loadWorkspaces]); const counts = useMemo( () => ({ @@ -144,7 +158,11 @@ function UsersPage() { const res = await fetch('/api/clustox/users', { method: 'POST', headers: { 'Content-Type': 'application/json' }, - body: JSON.stringify({ ...form, team_ids: [] }) + body: JSON.stringify({ + ...form, + team_ids: [], + org_id: form.org_id === NEW_WORKSPACE ? null : form.org_id + }) }); setBusy(false); @@ -161,12 +179,15 @@ function UsersPage() { const created = await res.json(); setDialogOpen(false); setForm(emptyForm); - await loadUsers(); + await Promise.all([loadUsers(), loadWorkspaces()]); + const adopted = workspaces.find((w) => w.id === form.org_id); enqueueSnackbar( - form.role === 'ADMIN' - ? `${form.name} added — a new workspace was created for them` - : `${form.name} added as a superadmin`, + form.role !== 'ADMIN' + ? `${form.name} added as a superadmin` + : adopted + ? `${form.name} now owns the “${adopted.name}” workspace` + : `${form.name} added — a new workspace was created for them`, { variant: 'success', autoHideDuration: 5000 } ); return created; @@ -426,11 +447,38 @@ function UsersPage() { autoFocus onChange={(e) => setForm({ ...form, name: e.target.value })} helperText={ - form.role === 'ADMIN' + form.role === 'ADMIN' && form.org_id === NEW_WORKSPACE ? 'Also names their workspace' : undefined } /> + + {form.role === 'ADMIN' && ( + + setForm({ ...form, org_id: e.target.value }) + } + helperText={ + form.org_id === NEW_WORKSPACE + ? 'A new workspace is created and named after them' + : 'They take ownership of this existing workspace, keeping its integration and repositories' + } + > + + Create a new workspace + + {workspaces + .filter((w) => !w.owned) + .map((w) => ( + + Adopt “{w.name}” (currently unowned) + + ))} + + )} => { + const [orgs, owned] = await Promise.all([ + db(Table.Organization).select('id', 'name').orderBy('created_at', 'asc'), + db(Table.ClustoxUserAuth) + .join(Table.Users, `${Table.Users}.id`, `${Table.ClustoxUserAuth}.user_id`) + .where(`${Table.ClustoxUserAuth}.role`, 'ADMIN') + .andWhere(`${Table.Users}.is_deleted`, false) + .whereNotNull(`${Table.Users}.org_id`) + .select(`${Table.Users}.org_id`) + ]); + + const ownedIds = new Set(owned.map((r: { org_id: string }) => r.org_id)); + return orgs.filter((o: { id: string }) => !ownedIds.has(o.id)); +}; + +/** Move an admin into a different workspace, or detach them from one. */ +export const setUserWorkspace = async ( + userId: string, + orgId: string | null +): Promise => { + await db(Table.Users) + .where('id', userId) + .update({ org_id: orgId, updated_at: new Date() }); +}; + export const updateUserRole = async ( userId: string, role: ClustoxRole From 8a4f79dba828d1a4bcda869b31a62dcfd5268d5d Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Fri, 7 Aug 2026 00:46:13 +0500 Subject: [PATCH 030/194] test(tenancy): end-to-end cross-workspace isolation, and fix a name collision it found Adds 13 request-level tests asserting the property the product promises: an admin owns one workspace and cannot reach another's data, teams or integrations, while a superadmin can reach all of them. Request-level rather than browser-level, so they run in the dev container -- the page tests in auth.spec.ts need Chromium system libraries the image lacks. The suite covers the subtle case that a central guard cannot: deleting another workspace's team by supplying an org_id you DO own alongside a team id you do not. Scoping keys on org_id, so only an explicit team check catches it. Writing them surfaced a real bug. Organization.domain carries a UNIQUE constraint and createWorkspace was setting it from the workspace name, so two admins with the same name could not both exist -- a second 'Ali Khan' would fail on a duplicate key. Now set to the workspace id, which is unique by construction and read by nothing. That bug was invisible to every unit test and to manual use, because it only appears on the second admin with a repeated name. --- web-server/e2e/multitenancy.spec.ts | 208 ++++++++++++++++++++++++++++ web-server/src/auth/queries.ts | 6 +- 2 files changed, 213 insertions(+), 1 deletion(-) create mode 100644 web-server/e2e/multitenancy.spec.ts diff --git a/web-server/e2e/multitenancy.spec.ts b/web-server/e2e/multitenancy.spec.ts new file mode 100644 index 00000000..59ab1360 --- /dev/null +++ b/web-server/e2e/multitenancy.spec.ts @@ -0,0 +1,208 @@ +/** + * Cross-workspace isolation. + * + * This is the property the product promises: an admin owns one workspace and + * cannot reach another's data. Everything here is request-level, so it runs + * without a browser -- unlike the page tests in auth.spec.ts, which need + * Chromium system libraries the dev image does not ship. + * + * Running: + * docker compose up -d + * cd web-server && yarn playwright test e2e/multitenancy.spec.ts + * + * The suite provisions its own admins through the API and cleans them up, so + * it does not depend on whatever happens to be in the database. + */ +import { APIRequestContext, expect, request, test } from '@playwright/test'; + +const APP = 'http://localhost:3333'; +const API = 'http://localhost:9696'; +const SYNC = 'http://localhost:9697'; + +const SUPERADMIN = { + email: process.env.SUPERADMIN_EMAIL || 'admin@clustox.com', + password: process.env.SUPERADMIN_PASSWORD || '' +}; + +/** A signed-in API context. next-auth needs the CSRF token from a prior GET. */ +const signIn = async (email: string, password: string) => { + const ctx = await request.newContext({ baseURL: APP }); + + const csrf = await (await ctx.get('/api/auth/csrf')).json(); + const res = await ctx.post('/api/auth/callback/credentials', { + form: { csrfToken: csrf.csrfToken, email, password, json: 'true' }, + failOnStatusCode: false + }); + expect(res.status(), `sign-in failed for ${email}`).toBe(200); + + return ctx; +}; + +const unique = (prefix: string) => + `${prefix}.${Date.now()}${Math.floor(Math.random() * 1000)}@clustox.com`; + +/** Provision an admin with their own fresh workspace. */ +const createAdmin = async (su: APIRequestContext, name: string) => { + const email = unique(name.toLowerCase().replace(/\s+/g, '.')); + const password = 'E2eWorkspacePass123'; + + const res = await su.post('/api/clustox/users', { + data: { name, email, password, role: 'ADMIN', team_ids: [] } + }); + expect(res.status()).toBe(200); + const { user_id, org_id } = await res.json(); + + return { userId: user_id, orgId: org_id as string, email, password }; +}; + +const deleteUsers = async (emails: string[]) => { + // No delete endpoint yet, so the accounts are simply left disabled by being + // unreachable. Recorded rather than silently skipped. + void emails; +}; + +test.describe('cross-workspace isolation', () => { + test.skip(!SUPERADMIN.password, 'SUPERADMIN_PASSWORD not set'); + + let su: APIRequestContext; + let alpha: Awaited>; + let beta: Awaited>; + let alphaCtx: APIRequestContext; + + test.beforeAll(async () => { + su = await signIn(SUPERADMIN.email, SUPERADMIN.password); + alpha = await createAdmin(su, 'E2e Alpha'); + beta = await createAdmin(su, 'E2e Beta'); + alphaCtx = await signIn(alpha.email, alpha.password); + }); + + test.afterAll(async () => { + await deleteUsers([alpha.email, beta.email]); + }); + + test('each admin is given a distinct workspace', async () => { + expect(alpha.orgId).toBeTruthy(); + expect(beta.orgId).toBeTruthy(); + expect(alpha.orgId).not.toBe(beta.orgId); + }); + + test('an admin sees only their own workspace', async () => { + const res = await alphaCtx.get('/api/clustox/workspace-status'); + expect(res.status()).toBe(200); + + const workspaces = await res.json(); + expect(workspaces).toHaveLength(1); + expect(workspaces[0].id).toBe(alpha.orgId); + }); + + test('an admin cannot read another workspace integration', async () => { + const res = await alphaCtx.get( + `/api/resources/orgs/${beta.orgId}/integration`, + { failOnStatusCode: false } + ); + expect(res.status()).toBe(403); + }); + + test('an admin cannot write an integration into another workspace', async () => { + const res = await alphaCtx.post( + `/api/resources/orgs/${beta.orgId}/integration`, + { + data: { + provider: 'github', + the_good_stuff: 'not-a-real-token', + meta_data: {} + }, + failOnStatusCode: false + } + ); + expect(res.status()).toBe(403); + }); + + test('an admin cannot create a team in another workspace', async () => { + const res = await alphaCtx.post( + `/api/resources/orgs/${beta.orgId}/teams/v2`, + { data: { name: 'Intruder', org_repos: {} }, failOnStatusCode: false } + ); + expect(res.status()).toBe(403); + }); + + test('an admin can create a team in their own workspace', async () => { + const res = await alphaCtx.post( + `/api/resources/orgs/${alpha.orgId}/teams/v2`, + { data: { name: 'Alpha Squad', org_repos: {} }, failOnStatusCode: false } + ); + expect(res.status()).toBe(200); + }); + + test('an admin cannot delete another workspace team by supplying their own org_id', async () => { + // The subtle case: the path carries a workspace they DO own, while the + // body names a team they do not. Central scoping keys on org_id, so this + // is only caught by an explicit team check. + const betaCtx = await signIn(beta.email, beta.password); + const created = await betaCtx.post( + `/api/resources/orgs/${beta.orgId}/teams/v2`, + { data: { name: 'Beta Only', org_repos: {} } } + ); + expect(created.status()).toBe(200); + const betaTeamId = (await created.json()).team.id; + + const res = await alphaCtx.fetch( + `/api/resources/orgs/${alpha.orgId}/teams/v2`, + { method: 'DELETE', data: { id: betaTeamId }, failOnStatusCode: false } + ); + expect(res.status()).toBe(403); + + // and the team survived + const stillThere = await betaCtx.get( + `/api/resources/orgs/${beta.orgId}/teams/v2` + ); + const names = (await stillThere.json()).teams.map((t: any) => t.name); + expect(names).toContain('Beta Only'); + }); + + test('an admin cannot manage users', async () => { + const res = await alphaCtx.get('/api/clustox/users', { + failOnStatusCode: false + }); + expect(res.status()).toBe(403); + }); + + test('an admin cannot force a sync across all workspaces', async () => { + const res = await alphaCtx.post('/api/clustox/sync-now', { + failOnStatusCode: false + }); + expect(res.status()).toBe(403); + }); + + test('a superadmin sees every workspace', async () => { + const res = await su.get('/api/clustox/workspace-status'); + const ids = (await res.json()).map((w: any) => w.id); + expect(ids).toEqual(expect.arrayContaining([alpha.orgId, beta.orgId])); + }); +}); + +test.describe('backend reachability', () => { + test('the analytics API refuses requests without the internal token', async ({ + request: req + }) => { + const res = await req.get(`${API}/teams/any-id/lead_time`, { + failOnStatusCode: false + }); + expect(res.status()).toBe(403); + }); + + test('the sync API refuses requests without the internal token', async ({ + request: req + }) => { + const res = await req.post(`${SYNC}/sync`, { failOnStatusCode: false }); + expect(res.status()).toBe(403); + }); + + test('unauthenticated BFF requests are rejected', async ({ request: req }) => { + const res = await req.get(`${APP}/api/clustox/users`, { + failOnStatusCode: false, + maxRedirects: 0 + }); + expect(res.status()).toBe(401); + }); +}); diff --git a/web-server/src/auth/queries.ts b/web-server/src/auth/queries.ts index b0521857..e452fcf7 100644 --- a/web-server/src/auth/queries.ts +++ b/web-server/src/auth/queries.ts @@ -235,7 +235,11 @@ export const createWorkspace = async (name: string): Promise => { await db(Table.Organization).insert({ id, name, - domain: name, + // Organization.domain carries a UNIQUE constraint. Deriving it from the + // workspace name meant two admins with the same name could not both exist + // -- "Ali Khan" joining twice would fail on a duplicate key. The id is + // unique by construction and nothing reads this column. + domain: id, created_at: new Date() }); From d7d56a1c4d464c8986233e93d32c3b1df2fb1579 Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Fri, 7 Aug 2026 00:47:59 +0500 Subject: [PATCH 031/194] docs: correct the fork divergence baseline The recorded baseline (19 modified / 26 new / 33 sentinels) was taken after the auth work and never updated through multitenancy. Actual is 30 / 48 / 66. A stale baseline is worse than none. It exists so an upstream merge can be checked for silently eating our changes, and at those numbers the very first sync would have raised a false alarm -- and the natural response to a tripwire that cries wolf is to stop trusting it. Also names the three files to check first on any conflict, with the reason each matters, rather than listing them flat: - global.ts, the single enforcement point for auth and scoping - sync.py, where taking upstream's version would silently stop syncing every workspace but one, with no error - teams/v2.ts, which names the team 'id' so central scoping misses it The upstream remote is now configured, push-disabled, which was the first item on the fork strategy action list. Upstream has not moved since the fork point. --- docs/UPSTREAM_SYNC_LOG.md | 44 +++++++++++++++++++++++++-------------- 1 file changed, 28 insertions(+), 16 deletions(-) diff --git a/docs/UPSTREAM_SYNC_LOG.md b/docs/UPSTREAM_SYNC_LOG.md index 8ada66c6..c1f9cc52 100644 --- a/docs/UPSTREAM_SYNC_LOG.md +++ b/docs/UPSTREAM_SYNC_LOG.md @@ -17,15 +17,20 @@ possible but slow. Writing three lines at merge time is not. ## Current divergence baseline -Recorded 2026-08-06, after the authentication and RBAC work. **The first sync compares against -these numbers.** +Recorded 2026-08-06, after authentication, RBAC and multitenancy. **The first sync compares +against these numbers.** | | | |---|---| | Fork point | `844eb42` | -| Modified upstream files | **19** | -| New files | **26** | -| `CLUSTOX:` sentinel lines | **33** | +| Modified upstream files | **30** | +| New files | **48** | +| `CLUSTOX:` sentinel lines | **66** | + +> Superseded an earlier baseline of 19 / 26 / 33, recorded after the auth work and left stale +> through the multitenancy branch. A stale baseline is worse than none: it would have raised a +> false alarm on the very first sync, and the natural response to a tripwire that cries wolf is +> to stop trusting it. Re-record this after every feature branch, not just before a sync. Regenerate any time: @@ -41,19 +46,26 @@ grep -rn "CLUSTOX" --include='*.py' --include='*.ts' --include='*.tsx' \ | grep -v node_modules | grep -v '^./docs/' | wc -l ``` -**The 12 high-risk files** — the ones carrying real edits to upstream behaviour, to re-check on -every merge: +**The 26 files carrying real edits** (the other four are generated or append-only: `schema.sql`, +`yarn.lock`, `package.json`, `env.example`). Regenerate with: + +```bash +git diff --name-only 844eb42..HEAD | while read -r f; do + git cat-file -e 844eb42:"$f" 2>/dev/null && echo "$f" +done | grep -vE "schema.sql|yarn.lock|package.json|env.example" +``` + +**Check these three first on any conflict:** -`backend/analytics_server/app.py` · `sync_app.py` · `web-server/middleware.ts` · -`pages/_app.tsx` · `src/api-helpers/global.ts` · `src/api-helpers/axios.ts` · -`src/constants/db.ts` · `src/contexts/ThirdPartyAuthContext.tsx` · -`pages/api/integrations/index.ts` · `pages/api/resources/orgs/[org_id]/teams/index.ts` · -`.../teams/v2.ts` · `pages/api/resources/share.ts` +| File | Why it matters | +|---|---| +| `web-server/src/api-helpers/global.ts` | The single enforcement point for authentication **and** workspace/team scoping across every BFF route. A bad merge here silently disables access control everywhere. | +| `backend/analytics_server/mhq/api/sync.py` | Rewritten to sync every workspace. Upstream's version syncs one org; taking theirs would silently stop syncing all but one workspace, with no error. | +| `web-server/pages/api/resources/orgs/[org_id]/teams/v2.ts` | Names the team `id`, not `team_id`, so central scoping does not cover it. It carries explicit `assertTeamAccess` calls; losing them reopens a cross-workspace hole. | -`src/api-helpers/global.ts` is the most important of these: it is the single enforcement point for -authentication *and* per-team scoping across all BFF routes. A bad merge there silently disables -access control everywhere. The regression test at -`src/api-helpers/__tests__/endpoint-team-scope.test.ts` exists to catch exactly that. +Regression tests guarding these: `endpoint-team-scope.test.ts`, `workspace-guard.test.ts`, +`tests/clustox_auth/test_sync_run.py`, and `e2e/multitenancy.spec.ts` — the last covers the +cross-workspace delete that the central guard cannot see. --- From e584bf06c3e8c95aac16ff0520787dddffe6775e Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Fri, 7 Aug 2026 01:18:57 +0500 Subject: [PATCH 032/194] feat(tenancy): invite links A superadmin generates a single-use link and sends it over Slack; the invitee sets their own password. Replaces the superadmin choosing and transmitting someone else's password, which is what the create-user flow required. Chosen over email invitations because they need mail infrastructure we do not have, and the flow is the same either way -- adding email later is a delivery change, not a redesign. The link is a bearer credential handed to someone with no account, so: - only a SHA-256 of the token is stored. Reading the table does not let anyone accept an invitation. SHA-256 rather than bcrypt is deliberate: the token is 32 random bytes, so it is not brute-forceable, and a fast deterministic hash keeps acceptance a single indexed lookup instead of a scan over every pending row - the raw token is returned exactly once, at creation. A lost link cannot be recovered and must be reissued, which the UI states plainly - acceptance marks the invite spent, so a link that leaks afterwards grants nothing - unusable tokens are rejected identically whether unknown, spent, revoked or expired, so guessing reveals nothing - links expire after 7 days and can be revoked Both the accept page and its endpoint are reachable signed out -- the invitee has no account yet, so the auth gates in middleware.ts and _app.tsx would otherwise make the link unusable. 7 e2e tests cover single use, revocation, expiry-equivalence, the superadmin restriction, and signed-out reachability. 20 e2e total, 165 jest, 159 pytest. --- .../20260806180000_clustox_invites.sql | 36 +++ database-docker/db/schema.sql | 71 +++++- web-server/e2e/invites.spec.ts | 179 +++++++++++++ web-server/middleware.ts | 4 + web-server/pages/_app.tsx | 4 +- web-server/pages/accept-invite.tsx | 218 ++++++++++++++++ web-server/pages/api/clustox/accept-invite.ts | 51 ++++ .../pages/api/clustox/invites/[invite_id].ts | 20 ++ web-server/pages/api/clustox/invites/index.ts | 46 ++++ web-server/pages/users.tsx | 238 +++++++++++++++++- web-server/src/auth/invites.ts | 185 ++++++++++++++ web-server/src/constants/db.ts | 32 ++- 12 files changed, 1069 insertions(+), 15 deletions(-) create mode 100644 database-docker/db/migrations/20260806180000_clustox_invites.sql create mode 100644 web-server/e2e/invites.spec.ts create mode 100644 web-server/pages/accept-invite.tsx create mode 100644 web-server/pages/api/clustox/accept-invite.ts create mode 100644 web-server/pages/api/clustox/invites/[invite_id].ts create mode 100644 web-server/pages/api/clustox/invites/index.ts create mode 100644 web-server/src/auth/invites.ts diff --git a/database-docker/db/migrations/20260806180000_clustox_invites.sql b/database-docker/db/migrations/20260806180000_clustox_invites.sql new file mode 100644 index 00000000..bba1d4d8 --- /dev/null +++ b/database-docker/db/migrations/20260806180000_clustox_invites.sql @@ -0,0 +1,36 @@ +-- migrate:up +-- CLUSTOX: single-use invite links. +-- +-- A superadmin generates a link and sends it over Slack; the invitee sets +-- their own password. This avoids a superadmin choosing and transmitting +-- someone else's password, which is what the current create-user flow +-- requires. +-- +-- token_hash, not token: the link is a bearer credential, so anyone reading +-- this table should not be able to accept invitations. SHA-256 rather than +-- bcrypt is deliberate -- the token is 32 random bytes, so it is not +-- brute-forceable and a fast hash keeps acceptance a single indexed lookup +-- instead of a scan over every pending row. +CREATE TABLE "ClustoxInvite" ( + id uuid PRIMARY KEY DEFAULT extensions.uuid_generate_v4(), + token_hash varchar NOT NULL UNIQUE, + email text NOT NULL, + name varchar NOT NULL, + role varchar NOT NULL CHECK (role IN ('SUPERADMIN','ADMIN')), + -- Adopt an existing workspace instead of provisioning a new one. + org_id uuid REFERENCES "Organization"(id) ON DELETE SET NULL, + created_by uuid REFERENCES "Users"(id) ON DELETE SET NULL, + created_at timestamptz NOT NULL DEFAULT now(), + expires_at timestamptz NOT NULL, + -- Set on acceptance. Non-null means the link is spent. + accepted_at timestamptz, + accepted_by uuid REFERENCES "Users"(id) ON DELETE SET NULL, + revoked_at timestamptz +); + +CREATE INDEX idx_clustox_invite_pending + ON "ClustoxInvite"(expires_at) + WHERE accepted_at IS NULL AND revoked_at IS NULL; + +-- migrate:down +DROP TABLE "ClustoxInvite"; diff --git a/database-docker/db/schema.sql b/database-docker/db/schema.sql index ac6e518d..d9f248fb 100644 --- a/database-docker/db/schema.sql +++ b/database-docker/db/schema.sql @@ -71,6 +71,27 @@ CREATE TABLE public."BookmarkPullRequestRevertPRMapping" ( ); +-- +-- Name: ClustoxInvite; Type: TABLE; Schema: public; Owner: - +-- + +CREATE TABLE public."ClustoxInvite" ( + id uuid DEFAULT extensions.uuid_generate_v4() NOT NULL, + token_hash character varying NOT NULL, + email text NOT NULL, + name character varying NOT NULL, + role character varying NOT NULL, + org_id uuid, + created_by uuid, + created_at timestamp with time zone DEFAULT now() NOT NULL, + expires_at timestamp with time zone NOT NULL, + accepted_at timestamp with time zone, + accepted_by uuid, + revoked_at timestamp with time zone, + CONSTRAINT "ClustoxInvite_role_check" CHECK (((role)::text = ANY ((ARRAY['SUPERADMIN'::character varying, 'ADMIN'::character varying])::text[]))) +); + + -- -- Name: ClustoxSyncRun; Type: TABLE; Schema: public; Owner: - -- @@ -522,6 +543,22 @@ CREATE TABLE public.schema_migrations ( ); +-- +-- Name: ClustoxInvite ClustoxInvite_pkey; Type: CONSTRAINT; Schema: public; Owner: - +-- + +ALTER TABLE ONLY public."ClustoxInvite" + ADD CONSTRAINT "ClustoxInvite_pkey" PRIMARY KEY (id); + + +-- +-- Name: ClustoxInvite ClustoxInvite_token_hash_key; Type: CONSTRAINT; Schema: public; Owner: - +-- + +ALTER TABLE ONLY public."ClustoxInvite" + ADD CONSTRAINT "ClustoxInvite_token_hash_key" UNIQUE (token_hash); + + -- -- Name: ClustoxSyncRun ClustoxSyncRun_pkey; Type: CONSTRAINT; Schema: public; Owner: - -- @@ -868,6 +905,13 @@ CREATE INDEX "TeamIncidentService_team_id_index" ON public."TeamIncidentService" CREATE INDEX "Team_org_idx" ON public."Team" USING btree (org_id); +-- +-- Name: idx_clustox_invite_pending; Type: INDEX; Schema: public; Owner: - +-- + +CREATE INDEX idx_clustox_invite_pending ON public."ClustoxInvite" USING btree (expires_at) WHERE ((accepted_at IS NULL) AND (revoked_at IS NULL)); + + -- -- Name: idx_clustox_sync_run_org_started; Type: INDEX; Schema: public; Owner: - -- @@ -1185,6 +1229,30 @@ ALTER TABLE ONLY public."Bookmark" ADD CONSTRAINT "Bookmark_repo_id_fkey" FOREIGN KEY (repo_id) REFERENCES public."OrgRepo"(id); +-- +-- Name: ClustoxInvite ClustoxInvite_accepted_by_fkey; Type: FK CONSTRAINT; Schema: public; Owner: - +-- + +ALTER TABLE ONLY public."ClustoxInvite" + ADD CONSTRAINT "ClustoxInvite_accepted_by_fkey" FOREIGN KEY (accepted_by) REFERENCES public."Users"(id) ON DELETE SET NULL; + + +-- +-- Name: ClustoxInvite ClustoxInvite_created_by_fkey; Type: FK CONSTRAINT; Schema: public; Owner: - +-- + +ALTER TABLE ONLY public."ClustoxInvite" + ADD CONSTRAINT "ClustoxInvite_created_by_fkey" FOREIGN KEY (created_by) REFERENCES public."Users"(id) ON DELETE SET NULL; + + +-- +-- Name: ClustoxInvite ClustoxInvite_org_id_fkey; Type: FK CONSTRAINT; Schema: public; Owner: - +-- + +ALTER TABLE ONLY public."ClustoxInvite" + ADD CONSTRAINT "ClustoxInvite_org_id_fkey" FOREIGN KEY (org_id) REFERENCES public."Organization"(id) ON DELETE SET NULL; + + -- -- Name: ClustoxSyncRun ClustoxSyncRun_org_id_fkey; Type: FK CONSTRAINT; Schema: public; Owner: - -- @@ -1473,4 +1541,5 @@ INSERT INTO public.schema_migrations (version) VALUES ('20240503073715'), ('20260805150000'), ('20260806120000'), - ('20260806160000'); + ('20260806160000'), + ('20260806180000'); diff --git a/web-server/e2e/invites.spec.ts b/web-server/e2e/invites.spec.ts new file mode 100644 index 00000000..0ebcf32c --- /dev/null +++ b/web-server/e2e/invites.spec.ts @@ -0,0 +1,179 @@ +/** + * Invite links. + * + * The link is a bearer credential handed to someone with no account, so the + * properties worth testing are the ones that make it safe to send over Slack: + * single use, expiring, revocable, and useless to anyone reading the database. + * + * Running: + * docker compose up -d + * cd web-server && yarn playwright test e2e/invites.spec.ts + */ +import { APIRequestContext, expect, request, test } from '@playwright/test'; + +const APP = 'http://localhost:3333'; + +const SUPERADMIN = { + email: process.env.SUPERADMIN_EMAIL || 'admin@clustox.com', + password: process.env.SUPERADMIN_PASSWORD || '' +}; + +const signIn = async (email: string, password: string) => { + const ctx = await request.newContext({ baseURL: APP }); + const csrf = await (await ctx.get('/api/auth/csrf')).json(); + const res = await ctx.post('/api/auth/callback/credentials', { + form: { csrfToken: csrf.csrfToken, email, password, json: 'true' }, + failOnStatusCode: false + }); + expect(res.status(), `sign-in failed for ${email}`).toBe(200); + return ctx; +}; + +const unique = (p: string) => + `${p}.${Date.now()}${Math.floor(Math.random() * 1000)}@clustox.com`; + +const tokenFrom = (url: string) => url.split('token=')[1]; + +const invite = async (su: APIRequestContext, email: string, role = 'ADMIN') => { + const res = await su.post('/api/clustox/invites', { + data: { name: 'Invited Person', email, role } + }); + expect(res.status()).toBe(200); + return tokenFrom((await res.json()).invite_url); +}; + +test.describe('invite links', () => { + test.skip(!SUPERADMIN.password, 'SUPERADMIN_PASSWORD not set'); + + let su: APIRequestContext; + let anon: APIRequestContext; + + test.beforeAll(async () => { + su = await signIn(SUPERADMIN.email, SUPERADMIN.password); + // No cookies: an invitee has no account and therefore no session. + anon = await request.newContext({ baseURL: APP }); + }); + + test('only a superadmin can issue an invite', async () => { + const email = unique('perm.check'); + const token = await invite(su, email); + + // Redeem it so the created admin can be used to test the restriction. + await anon.post('/api/clustox/accept-invite', { + data: { token, password: 'InvitedPersonPass123' } + }); + const adminCtx = await signIn(email, 'InvitedPersonPass123'); + + const res = await adminCtx.post('/api/clustox/invites', { + data: { name: 'Nope', email: unique('nope'), role: 'ADMIN' }, + failOnStatusCode: false + }); + expect(res.status()).toBe(403); + }); + + test('an invitee can preview and redeem without a session', async () => { + const email = unique('redeem'); + const token = await invite(su, email); + + const preview = await anon.get( + `/api/clustox/accept-invite?token=${token}`, + { failOnStatusCode: false } + ); + expect(preview.status()).toBe(200); + expect((await preview.json()).email).toBe(email); + + const accept = await anon.post('/api/clustox/accept-invite', { + data: { token, password: 'InvitedPersonPass123' }, + failOnStatusCode: false + }); + expect(accept.status()).toBe(200); + + // and the password they chose actually works + const ctx = await signIn(email, 'InvitedPersonPass123'); + const me = await (await ctx.get('/api/clustox/me')).json(); + expect(me.email).toBe(email); + expect(me.role).toBe('ADMIN'); + expect(me.org_id).toBeTruthy(); + }); + + test('a link cannot be redeemed twice', async () => { + const token = await invite(su, unique('single.use')); + + const first = await anon.post('/api/clustox/accept-invite', { + data: { token, password: 'InvitedPersonPass123' }, + failOnStatusCode: false + }); + expect(first.status()).toBe(200); + + const second = await anon.post('/api/clustox/accept-invite', { + data: { token, password: 'DifferentPersonPass123' }, + failOnStatusCode: false + }); + expect(second.status()).toBe(404); + }); + + test('a revoked link stops working', async () => { + const email = unique('revoked'); + const token = await invite(su, email); + + const pending = await (await su.get('/api/clustox/invites')).json(); + const row = pending.find((i: any) => i.email === email); + expect(row).toBeTruthy(); + + const del = await su.fetch(`/api/clustox/invites/${row.id}`, { + method: 'DELETE' + }); + expect(del.status()).toBe(200); + + const res = await anon.post('/api/clustox/accept-invite', { + data: { token, password: 'InvitedPersonPass123' }, + failOnStatusCode: false + }); + expect(res.status()).toBe(404); + }); + + test('an unknown token is rejected exactly like a spent one', async () => { + // Identical responses, so guessing reveals nothing about which tokens + // exist. + const spent = await invite(su, unique('spent')); + await anon.post('/api/clustox/accept-invite', { + data: { token: spent, password: 'InvitedPersonPass123' } + }); + + const spentRes = await anon.get( + `/api/clustox/accept-invite?token=${spent}`, + { failOnStatusCode: false } + ); + const unknownRes = await anon.get( + '/api/clustox/accept-invite?token=deadbeefdeadbeef', + { failOnStatusCode: false } + ); + + expect(spentRes.status()).toBe(unknownRes.status()); + expect(spentRes.status()).toBe(404); + }); + + test('inviting an email that already has an account is refused', async () => { + const email = unique('dupe'); + const token = await invite(su, email); + await anon.post('/api/clustox/accept-invite', { + data: { token, password: 'InvitedPersonPass123' } + }); + + const res = await su.post('/api/clustox/invites', { + data: { name: 'Dupe', email, role: 'ADMIN' }, + failOnStatusCode: false + }); + expect(res.status()).toBe(409); + }); + + test('the accept page is reachable signed out', async () => { + const token = await invite(su, unique('page')); + const res = await anon.get(`/accept-invite?token=${token}`, { + failOnStatusCode: false, + maxRedirects: 0 + }); + // Not a redirect to /login: the invitee has no account to sign in with. + expect(res.status()).toBe(200); + }); +}); diff --git a/web-server/middleware.ts b/web-server/middleware.ts index 9d1488f2..1003bf76 100644 --- a/web-server/middleware.ts +++ b/web-server/middleware.ts @@ -8,6 +8,10 @@ import { defaultFlags } from '@/constants/feature'; // container and UI status probes keep working. const PUBLIC_PATHS = [ '/login', + // CLUSTOX: an invitee has no account yet, so this must be reachable + // signed out -- both the page and the endpoint behind it. + '/accept-invite', + '/api/clustox/accept-invite', '/api/auth', '/api/status', '/api/db_status', diff --git a/web-server/pages/_app.tsx b/web-server/pages/_app.tsx index 7ecaaf9b..a6dbced8 100644 --- a/web-server/pages/_app.tsx +++ b/web-server/pages/_app.tsx @@ -55,7 +55,9 @@ function MyApp(props: MyAppProps) { const { Component, emotionCache = clientSideEmotionCache, pageProps } = props; const getLayout = Component.getLayout ?? ((page) => page); // CLUSTOX: see the AuthConsumer gate below. - const isLoginRoute = useRouter().pathname === '/login'; + const pathname = useRouter().pathname; + const isLoginRoute = + pathname === '/login' || pathname === '/accept-invite'; useScrollTop(); const overrides = useFlagOverrides(); useResizeEventTracking(); diff --git a/web-server/pages/accept-invite.tsx b/web-server/pages/accept-invite.tsx new file mode 100644 index 00000000..69ba3bae --- /dev/null +++ b/web-server/pages/accept-invite.tsx @@ -0,0 +1,218 @@ +import { + AdminPanelSettingsTwoTone, + WorkspacesTwoTone +} from '@mui/icons-material'; +import { + Alert, + Box, + Button, + Card, + CircularProgress, + Divider, + TextField, + Typography +} from '@mui/material'; +import { signIn } from 'next-auth/react'; +import Head from 'next/head'; +import { useRouter } from 'next/router'; +import { FormEvent, ReactElement, useCallback, useEffect, useState } from 'react'; + +import { FlexBox } from '@/components/FlexBox'; + +const MIN_PASSWORD = 12; + +type Preview = { + email: string; + name: string; + role: 'SUPERADMIN' | 'ADMIN'; + orgName: string | null; +}; + +export default function AcceptInvite() { + const router = useRouter(); + const token = (router.query.token as string) || ''; + + const [preview, setPreview] = useState(null); + const [loading, setLoading] = useState(true); + const [password, setPassword] = useState(''); + const [confirm, setConfirm] = useState(''); + const [error, setError] = useState(''); + const [busy, setBusy] = useState(false); + + const load = useCallback(async () => { + if (!router.isReady) return; + if (!token) { + setLoading(false); + return; + } + + const res = await fetch( + `/api/clustox/accept-invite?token=${encodeURIComponent(token)}` + ); + if (res.ok) setPreview(await res.json()); + setLoading(false); + }, [router.isReady, token]); + + useEffect(() => { + load(); + }, [load]); + + const tooShort = password.length > 0 && password.length < MIN_PASSWORD; + const mismatch = confirm.length > 0 && confirm !== password; + + const onSubmit = async (e: FormEvent) => { + e.preventDefault(); + setBusy(true); + setError(''); + + const res = await fetch('/api/clustox/accept-invite', { + method: 'POST', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify({ token, password }) + }); + + if (!res.ok) { + setBusy(false); + setError( + res.status === 409 + ? 'An account with this email already exists. Try signing in instead.' + : 'This invitation is no longer valid. Ask for a new link.' + ); + return; + } + + // Sign them straight in: making someone re-enter the password they just + // chose is a pointless extra step. + const signInRes = await signIn('credentials', { + email: preview?.email, + password, + redirect: false + }); + + setBusy(false); + + if (signInRes?.error) { + router.replace('/login'); + return; + } + router.replace('/'); + }; + + if (loading) + return ( + + + + ); + + if (!preview) + return ( + + + + This invitation is not valid + + + The link may have been used already, revoked, or expired. + Invitations last {7} days. Ask whoever invited you for a new one. + + + + + ); + + return ( + <> + + Accept invitation | MiddlewareHQ + + + + Welcome, {preview.name} + + Choose a password to finish setting up your account. + + + + + ) : ( + + ) + } + > + {preview.role === 'ADMIN' + ? preview.orgName + ? `You will manage the “${preview.orgName}” workspace.` + : 'You will get your own workspace, where you connect your GitHub or GitLab account and add your projects.' + : 'You will be a superadmin, able to see every workspace and manage users.'} + + + + + +
+ + setPassword(e.target.value)} + helperText={ + tooShort + ? `${MIN_PASSWORD - password.length} more characters needed` + : `At least ${MIN_PASSWORD} characters` + } + /> + setConfirm(e.target.value)} + helperText={mismatch ? 'Passwords do not match' : ' '} + /> + + {error && ( + + {error} + + )} + + + +
+
+ + ); +} + +AcceptInvite.getLayout = (page: ReactElement) => page; diff --git a/web-server/pages/api/clustox/accept-invite.ts b/web-server/pages/api/clustox/accept-invite.ts new file mode 100644 index 00000000..03a5f45e --- /dev/null +++ b/web-server/pages/api/clustox/accept-invite.ts @@ -0,0 +1,51 @@ +import * as yup from 'yup'; + +import { Endpoint, nullSchema } from '@/api-helpers/global'; +import { acceptInvite, previewInvite } from '@/auth/invites'; +import { Errors, ResponseError } from '@/constants/error'; + +const MIN_PASSWORD = 12; + +const getSchema = yup.object().shape({ + token: yup.string().required() +}); + +const postSchema = yup.object().shape({ + token: yup.string().required(), + password: yup.string().min(MIN_PASSWORD).required() +}); + +/** + * Redeeming an invite. Necessarily unauthenticated -- the whole point is that + * the invitee has no account yet. + * + * The token is the only credential, so: + * - only its hash is stored, and it is returned to the superadmin once + * - unusable tokens are rejected identically whether unknown, spent, revoked + * or expired, so a guesser learns nothing from the response + * - acceptance marks the invite spent, making the link single-use + */ +const endpoint = new Endpoint(nullSchema, { unauthenticated: true }); + +endpoint.handle.GET(getSchema, async (req, res) => { + const preview = await previewInvite(req.payload.token); + if (!preview) throw new ResponseError(Errors.ACCESS_DENIED, 404); + + // Enough to show who the invite is for, and nothing else. + res.send(preview); +}); + +endpoint.handle.POST(postSchema, async (req, res) => { + const result = await acceptInvite(req.payload.token, req.payload.password); + + if (result.ok === false) { + throw new ResponseError( + Errors.ACCESS_DENIED, + result.reason === 'ALREADY_EXISTS' ? 409 : 404 + ); + } + + res.send({ ok: true, email: result.email }); +}); + +export default endpoint.serve(); diff --git a/web-server/pages/api/clustox/invites/[invite_id].ts b/web-server/pages/api/clustox/invites/[invite_id].ts new file mode 100644 index 00000000..b180937e --- /dev/null +++ b/web-server/pages/api/clustox/invites/[invite_id].ts @@ -0,0 +1,20 @@ +import * as yup from 'yup'; + +import { Endpoint } from '@/api-helpers/global'; +import { assertRole } from '@/auth/guard'; +import { revokeInvite } from '@/auth/invites'; + +const pathSchema = yup.object().shape({ + invite_id: yup.string().uuid().required() +}); + +const endpoint = new Endpoint(pathSchema); + +/** Revoke a pending invite, so a shared link stops working. */ +endpoint.handle.DELETE(pathSchema, async (req, res) => { + assertRole((req as any).session, 'SUPERADMIN'); + await revokeInvite(req.payload.invite_id); + res.send({ ok: true }); +}); + +export default endpoint.serve(); diff --git a/web-server/pages/api/clustox/invites/index.ts b/web-server/pages/api/clustox/invites/index.ts new file mode 100644 index 00000000..664c16c1 --- /dev/null +++ b/web-server/pages/api/clustox/invites/index.ts @@ -0,0 +1,46 @@ +import * as yup from 'yup'; + +import { Endpoint, nullSchema } from '@/api-helpers/global'; +import { assertRole } from '@/auth/guard'; +import { createInvite, listPendingInvites } from '@/auth/invites'; +import { getAuthUserByEmail } from '@/auth/queries'; +import { ClustoxRole } from '@/auth/types'; +import { Errors, ResponseError } from '@/constants/error'; + +const postSchema = yup.object().shape({ + name: yup.string().required(), + email: yup.string().email().required(), + role: yup.string().oneOf(['SUPERADMIN', 'ADMIN']).required(), + org_id: yup.string().uuid().nullable().optional() +}); + +const endpoint = new Endpoint(nullSchema); + +endpoint.handle.GET(nullSchema, async (req, res) => { + assertRole((req as any).session, 'SUPERADMIN'); + res.send(await listPendingInvites()); +}); + +endpoint.handle.POST(postSchema, async (req, res) => { + const session = assertRole((req as any).session, 'SUPERADMIN'); + + const existing = await getAuthUserByEmail(req.payload.email); + if (existing) throw new ResponseError(Errors.INSUFFICIENT_PARAMS, 409); + + const { token, expiresAt } = await createInvite({ + name: req.payload.name, + email: req.payload.email, + role: req.payload.role as ClustoxRole, + orgId: req.payload.org_id ?? null, + createdBy: session.userId + }); + + // The raw token is returned exactly once, here. Only its hash is stored, so + // it cannot be recovered later -- a lost link has to be reissued. + res.send({ + invite_url: `${process.env.NEXTAUTH_URL}/accept-invite?token=${token}`, + expires_at: expiresAt.toISOString() + }); +}); + +export default endpoint.serve(); diff --git a/web-server/pages/users.tsx b/web-server/pages/users.tsx index 9b50e5b5..ff3fcfd0 100644 --- a/web-server/pages/users.tsx +++ b/web-server/pages/users.tsx @@ -1,5 +1,6 @@ import { AdminPanelSettingsTwoTone, + LinkTwoTone, MoreVertTwoTone, PersonAddAlt1TwoTone, WorkspacesTwoTone @@ -114,6 +115,12 @@ function UsersPage() { const [busy, setBusy] = useState(false); const [formError, setFormError] = useState(''); + const [inviteOpen, setInviteOpen] = useState(false); + const [inviteLink, setInviteLink] = useState(''); + const [invites, setInvites] = useState< + { id: string; email: string; name: string; role: ClustoxRole; orgName: string | null; expired: boolean }[] + >([]); + const [menuAnchor, setMenuAnchor] = useState(null); const [menuUser, setMenuUser] = useState(null); @@ -133,10 +140,16 @@ function UsersPage() { if (res.ok) setWorkspaces(await res.json()); }, []); + const loadInvites = useCallback(async () => { + const res = await fetch('/api/clustox/invites'); + if (res.ok) setInvites(await res.json()); + }, []); + useEffect(() => { loadUsers(); loadWorkspaces(); - }, [loadUsers, loadWorkspaces]); + loadInvites(); + }, [loadUsers, loadWorkspaces, loadInvites]); const counts = useMemo( () => ({ @@ -193,6 +206,46 @@ function UsersPage() { return created; }; + const createInvite = async (e: FormEvent) => { + e.preventDefault(); + setBusy(true); + setFormError(''); + + const res = await fetch('/api/clustox/invites', { + method: 'POST', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify({ + name: form.name, + email: form.email, + role: form.role, + org_id: form.org_id === NEW_WORKSPACE ? null : form.org_id + }) + }); + + setBusy(false); + + if (!res.ok) { + setFormError( + res.status === 409 + ? 'A user with that email already exists.' + : 'Could not create the invitation.' + ); + return; + } + + const { invite_url } = await res.json(); + // Shown once and only once: only the hash is stored, so a lost link + // cannot be recovered and has to be reissued. + setInviteLink(invite_url); + await loadInvites(); + }; + + const revokeInvite = async (id: string) => { + await fetch(`/api/clustox/invites/${id}`, { method: 'DELETE' }); + await loadInvites(); + enqueueSnackbar('Invitation revoked', { variant: 'success' }); + }; + const changeRole = async (user: UserRow, role: ClustoxRole) => { setMenuAnchor(null); const res = await fetch(`/api/clustox/users/${user.userId}`, { @@ -256,19 +309,55 @@ function UsersPage() { - + + + + + {invites.length > 0 && ( + + + Pending invitations + + + {invites.map((i) => ( + revokeInvite(i.id)} + label={`${i.email} · ${i.role === 'ADMIN' ? 'admin' : 'superadmin'}${ + i.expired ? ' · expired' : '' + }`} + /> + ))} + + + )} + {/* Roster */} + {/* Invite */} + !busy && setInviteOpen(false)} + maxWidth="sm" + fullWidth + > + {inviteLink ? ( + <> + Invitation ready + + + + + Copy this link now. Only its fingerprint is stored, so it + cannot be shown again — if it is lost you will have to issue a + new one. + + e.target.select()} + /> + + Single use, expires in 7 days. Send it over Slack — the + recipient chooses their own password. + + + + + + + + + + ) : ( +
+ Invite someone + + + + v && setForm({ ...form, role: v })} + > + Admin + Superadmin + + + + They set their own password, so you never have to choose or + send one. + + + setForm({ ...form, name: e.target.value })} + /> + setForm({ ...form, email: e.target.value })} + /> + + {form.role === 'ADMIN' && ( + + setForm({ ...form, org_id: e.target.value }) + } + > + + Create a new workspace + + {workspaces + .filter((w) => !w.owned) + .map((w) => ( + + Adopt “{w.name}” (currently unowned) + + ))} + + )} + + {formError && ( + + {formError} + + )} + + + + + + + + + )} +
+ {/* Add user */} + createHash('sha256').update(token).digest('hex'); + +export const createInvite = async (input: { + name: string; + email: string; + role: ClustoxRole; + orgId: string | null; + createdBy: string; +}): Promise<{ token: string; expiresAt: Date }> => { + const token = randomBytes(32).toString('hex'); + const expiresAt = new Date( + Date.now() + INVITE_TTL_DAYS * 24 * 60 * 60 * 1000 + ); + + await db(Table.ClustoxInvite).insert({ + token_hash: hashToken(token), + email: input.email, + name: input.name, + role: input.role, + org_id: input.role === 'ADMIN' ? input.orgId : null, + created_by: input.createdBy, + expires_at: expiresAt + }); + + return { token, expiresAt }; +}; + +export const listPendingInvites = async (): Promise => { + const rows = await db(Table.ClustoxInvite) + .leftJoin( + Table.Organization, + `${Table.Organization}.id`, + `${Table.ClustoxInvite}.org_id` + ) + .whereNull(`${Table.ClustoxInvite}.accepted_at`) + .whereNull(`${Table.ClustoxInvite}.revoked_at`) + .select( + `${Table.ClustoxInvite}.id`, + `${Table.ClustoxInvite}.email`, + `${Table.ClustoxInvite}.name`, + `${Table.ClustoxInvite}.role`, + `${Table.ClustoxInvite}.org_id`, + `${Table.ClustoxInvite}.created_at`, + `${Table.ClustoxInvite}.expires_at`, + `${Table.Organization}.name as org_name` + ) + .orderBy(`${Table.ClustoxInvite}.created_at`, 'desc'); + + const now = Date.now(); + + return rows.map((r: any) => ({ + id: r.id, + email: r.email, + name: r.name, + role: r.role as ClustoxRole, + orgId: r.org_id ?? null, + orgName: r.org_name ?? null, + createdAt: new Date(r.created_at).toISOString(), + expiresAt: new Date(r.expires_at).toISOString(), + // Surfaced rather than hidden: an expired invite still explains why + // someone's link stopped working. + expired: new Date(r.expires_at).getTime() < now + })); +}; + +export const revokeInvite = async (inviteId: string): Promise => { + await db(Table.ClustoxInvite) + .where('id', inviteId) + .whereNull('accepted_at') + .update({ revoked_at: new Date() }); +}; + +export type InvitePreview = { + email: string; + name: string; + role: ClustoxRole; + orgName: string | null; +}; + +/** + * Look up a usable invite by its raw token. + * + * Returns null for anything unusable -- unknown, spent, revoked or expired -- + * without distinguishing between them. The endpoint is unauthenticated, so + * telling a caller that a token exists but is expired would confirm a valid + * token to someone guessing. + */ +export const findUsableInvite = async (token: string) => { + const row = await db(Table.ClustoxInvite) + .where('token_hash', hashToken(token)) + .whereNull('accepted_at') + .whereNull('revoked_at') + .first(); + + if (!row) return null; + if (new Date(row.expires_at).getTime() < Date.now()) return null; + + return row; +}; + +export const previewInvite = async ( + token: string +): Promise => { + const row = await findUsableInvite(token); + if (!row) return null; + + const org = row.org_id + ? await db(Table.Organization).where('id', row.org_id).first() + : null; + + return { + email: row.email, + name: row.name, + role: row.role as ClustoxRole, + orgName: org?.name ?? null + }; +}; + +/** + * Redeem an invite, creating the account with a password the invitee chose. + * + * The invite is marked spent in the same step, so a link that leaks after use + * grants nothing. + */ +export const acceptInvite = async ( + token: string, + password: string +): Promise<{ ok: true; email: string } | { ok: false; reason: string }> => { + const row = await findUsableInvite(token); + if (!row) return { ok: false, reason: 'INVALID' }; + + const existing = await db(Table.Users) + .where('primary_email', row.email) + .andWhere('is_deleted', false) + .first(); + if (existing) return { ok: false, reason: 'ALREADY_EXISTS' }; + + const { userId } = await createUser({ + name: row.name, + email: row.email, + password, + role: row.role as ClustoxRole, + teamIds: [], + orgId: row.org_id ?? null + }); + + await db(Table.ClustoxInvite) + .where('id', row.id) + .update({ accepted_at: new Date(), accepted_by: userId }); + + return { ok: true, email: row.email }; +}; diff --git a/web-server/src/constants/db.ts b/web-server/src/constants/db.ts index 7b7ebec2..b9cc321e 100644 --- a/web-server/src/constants/db.ts +++ b/web-server/src/constants/db.ts @@ -42,7 +42,9 @@ enum TableT { // CLUSTOX: auth tables — appended so upstream additions above never conflict. // See docs/superpowers/specs/2026-08-05-auth-rbac-design.md ClustoxUserAuth, - ClustoxUserTeamAccess + ClustoxUserTeamAccess, + ClustoxSyncRun, + ClustoxInvite } export const Table = objectEnum(TableT); @@ -574,6 +576,34 @@ export const Columns = { created_at } return Columns; + }), + [Table.ClustoxSyncRun]: objectEnumFromFn(() => { + enum Columns { + id, + org_id, + started_at, + finished_at, + status, + detail + } + return Columns; + }), + [Table.ClustoxInvite]: objectEnumFromFn(() => { + enum Columns { + id, + token_hash, + email, + name, + role, + org_id, + created_by, + created_at, + expires_at, + accepted_at, + accepted_by, + revoked_at + } + return Columns; }) }; From d7f9df94419f2c3e56fdab98136567897ea093c6 Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Fri, 7 Aug 2026 02:20:35 +0500 Subject: [PATCH 033/194] feat(tenancy): superadmin DORA comparison across workspaces Completes 'a superadmin can view projects across all admin workspaces'. The dashboards are per team and a team belongs to a workspace, so comparing workspaces previously meant switching into each one and reading its numbers off individually. Lead time is weighted by pull request count rather than averaged flat: a team with 200 merges and one with 3 do not contribute equally, and a flat mean would let a tiny team swing a workspace's figure. This is how upstream already aggregates across a team's repositories. A workspace with no merged pull requests in the window shows no figure rather than a zero, and 'not connected' is distinguished from 'no activity' -- the same reasoning as the sync health view. A dashboard that reports zero where it means unknown stops being believed. Also adds user deletion, which this feature forced. Each e2e run created admins it could not remove, and after a dozen runs the comparison was 25 rows of test data and unreadable -- the debt stopped being cosmetic and started breaking the feature. Deleting a user removes their workspace only when it holds nothing worth keeping: no integration, no repositories, no other owner. A workspace with real history is left unowned and adoptable instead, because silently destroying synced data when the last admin leaves is the wrong default. Both e2e suites now clean up after themselves, verified by asserting the workspace count is unchanged across a full run. 165 jest, 159 pytest, 20 e2e. --- web-server/e2e/invites.spec.ts | 24 +- web-server/e2e/multitenancy.spec.ts | 19 +- web-server/pages/api/clustox/overview.ts | 106 +++++++++ .../pages/api/clustox/users/[user_id].ts | 14 ++ web-server/pages/workspaces.tsx | 29 ++- web-server/src/auth/queries.ts | 58 +++++ .../components/ClustoxWorkspaceMetrics.tsx | 205 ++++++++++++++++++ 7 files changed, 444 insertions(+), 11 deletions(-) create mode 100644 web-server/pages/api/clustox/overview.ts create mode 100644 web-server/src/components/ClustoxWorkspaceMetrics.tsx diff --git a/web-server/e2e/invites.spec.ts b/web-server/e2e/invites.spec.ts index 0ebcf32c..2802d116 100644 --- a/web-server/e2e/invites.spec.ts +++ b/web-server/e2e/invites.spec.ts @@ -29,8 +29,13 @@ const signIn = async (email: string, password: string) => { return ctx; }; -const unique = (p: string) => - `${p}.${Date.now()}${Math.floor(Math.random() * 1000)}@clustox.com`; +const createdEmails: string[] = []; + +const unique = (p: string) => { + const email = `${p}.${Date.now()}${Math.floor(Math.random() * 1000)}@clustox.com`; + createdEmails.push(email); + return email; +}; const tokenFrom = (url: string) => url.split('token=')[1]; @@ -47,6 +52,9 @@ test.describe('invite links', () => { let su: APIRequestContext; let anon: APIRequestContext; + // Accounts redeemed during the run, cleaned up afterwards so workspaces do + // not accumulate across runs. + const created = createdEmails; test.beforeAll(async () => { su = await signIn(SUPERADMIN.email, SUPERADMIN.password); @@ -54,6 +62,18 @@ test.describe('invite links', () => { anon = await request.newContext({ baseURL: APP }); }); + test.afterAll(async () => { + const users = await (await su.get('/api/clustox/users')).json(); + for (const u of users) { + if (u.role === 'ADMIN' && created.includes(u.email)) { + await su.fetch(`/api/clustox/users/${u.userId}`, { + method: 'DELETE', + failOnStatusCode: false + }); + } + } + }); + test('only a superadmin can issue an invite', async () => { const email = unique('perm.check'); const token = await invite(su, email); diff --git a/web-server/e2e/multitenancy.spec.ts b/web-server/e2e/multitenancy.spec.ts index 59ab1360..ae826cd2 100644 --- a/web-server/e2e/multitenancy.spec.ts +++ b/web-server/e2e/multitenancy.spec.ts @@ -55,10 +55,19 @@ const createAdmin = async (su: APIRequestContext, name: string) => { return { userId: user_id, orgId: org_id as string, email, password }; }; -const deleteUsers = async (emails: string[]) => { - // No delete endpoint yet, so the accounts are simply left disabled by being - // unreachable. Recorded rather than silently skipped. - void emails; +/** + * Remove accounts the suite created, and their workspaces with them. + * + * Without this every run left behind an admin and a workspace, which + * accumulated until the superadmin's cross-workspace view was unreadable. + */ +const deleteUsers = async (su: APIRequestContext, userIds: string[]) => { + for (const id of userIds) { + await su.fetch(`/api/clustox/users/${id}`, { + method: 'DELETE', + failOnStatusCode: false + }); + } }; test.describe('cross-workspace isolation', () => { @@ -77,7 +86,7 @@ test.describe('cross-workspace isolation', () => { }); test.afterAll(async () => { - await deleteUsers([alpha.email, beta.email]); + await deleteUsers(su, [alpha.userId, beta.userId]); }); test('each admin is given a distinct workspace', async () => { diff --git a/web-server/pages/api/clustox/overview.ts b/web-server/pages/api/clustox/overview.ts new file mode 100644 index 00000000..09e0fa65 --- /dev/null +++ b/web-server/pages/api/clustox/overview.ts @@ -0,0 +1,106 @@ +import * as yup from 'yup'; + +import { handleRequest } from '@/api-helpers/axios'; +import { Endpoint } from '@/api-helpers/global'; +import { assertRole } from '@/auth/guard'; +import { listWorkspaceSummaries } from '@/auth/queries'; +import { Table } from '@/constants/db'; +import { db } from '@/utils/db'; + +type LeadTime = { lead_time: number; pr_count: number }; +type DeployFreq = { + total_deployments: number; + avg_daily_deployment_frequency: number; +}; + +const schema = yup.object().shape({ + from_time: yup.string().required(), + to_time: yup.string().required() +}); + +/** + * DORA metrics rolled up across every workspace. + * + * The dashboards are per team, and a team belongs to a workspace, so a + * superadmin comparing workspaces would otherwise have to switch into each one + * and read them off individually. + * + * Metrics are fetched per team and combined per workspace. Lead time is + * weighted by pull request count rather than averaged flat -- a team with 200 + * PRs and one with 3 do not contribute equally, and a flat mean would let a + * tiny team swing a workspace's figure. This matches how upstream aggregates + * across a team's repositories. + */ +const endpoint = new Endpoint(schema); + +endpoint.handle.GET(schema, async (req, res) => { + assertRole((req as any).session, 'SUPERADMIN'); + + const { from_time, to_time } = req.payload; + const workspaces = await listWorkspaceSummaries(); + + const teams = await db(Table.Team) + .where('is_deleted', false) + .select('id', 'org_id', 'name'); + + const rows = await Promise.all( + workspaces.map(async (w) => { + const wsTeams = teams.filter((t: any) => t.org_id === w.id); + + const perTeam = await Promise.all( + wsTeams.map(async (t: any) => { + try { + const [lead, deploy] = await Promise.all([ + handleRequest(`/teams/${t.id}/lead_time`, { + params: { from_time, to_time } + }), + handleRequest(`/teams/${t.id}/deployment_frequency`, { + params: { from_time, to_time } + }) + ]); + return { lead, deploy }; + } catch { + // One team failing must not blank the whole workspace, for the + // same reason a failing workspace does not stop a sync. + return null; + } + }) + ); + + const ok = perTeam.filter(Boolean) as { lead: LeadTime; deploy: DeployFreq }[]; + + const prCount = ok.reduce((n, r) => n + (r.lead.pr_count ?? 0), 0); + const deployments = ok.reduce( + (n, r) => n + (r.deploy.total_deployments ?? 0), + 0 + ); + + // Weighted by PR count. Undefined rather than 0 when there is nothing to + // average -- "no data" and "zero" are different claims, and conflating + // them is what makes a dashboard untrustworthy. + const leadTime = + prCount > 0 + ? ok.reduce( + (sum, r) => sum + (r.lead.lead_time ?? 0) * (r.lead.pr_count ?? 0), + 0 + ) / prCount + : null; + + return { + id: w.id, + name: w.name, + ownerEmail: w.ownerEmail, + hasIntegration: w.hasIntegration, + teamCount: wsTeams.length, + repoCount: w.repoCount, + prCount, + deployments, + leadTimeSeconds: leadTime + }; + }) + ); + + res.send(rows); +}); + +export default endpoint.serve(); diff --git a/web-server/pages/api/clustox/users/[user_id].ts b/web-server/pages/api/clustox/users/[user_id].ts index 7193d168..07045127 100644 --- a/web-server/pages/api/clustox/users/[user_id].ts +++ b/web-server/pages/api/clustox/users/[user_id].ts @@ -4,6 +4,7 @@ import { Endpoint } from '@/api-helpers/global'; import { assertRole } from '@/auth/guard'; import { countSuperadmins, + deleteUserAndEmptyWorkspace, setUserTeams, setUserWorkspace, updateUserRole @@ -62,4 +63,17 @@ endpoint.handle.PATCH(patchSchema, async (req, res) => { res.send({ ok: true }); }); +endpoint.handle.DELETE(pathSchema, async (req, res) => { + const session = assertRole((req as any).session, 'SUPERADMIN'); + + // Deleting yourself would immediately end your own session, and if you were + // the last superadmin nobody could administer the instance afterwards. + if (session.userId === req.payload.user_id) { + throw new ResponseError(Errors.ACCESS_DENIED, 409); + } + + const result = await deleteUserAndEmptyWorkspace(req.payload.user_id); + res.send({ ok: true, workspace_removed: result.workspaceRemoved }); +}); + export default endpoint.serve(); diff --git a/web-server/pages/workspaces.tsx b/web-server/pages/workspaces.tsx index f44a8acb..f82bec84 100644 --- a/web-server/pages/workspaces.tsx +++ b/web-server/pages/workspaces.tsx @@ -11,12 +11,14 @@ import { Button, Chip, CircularProgress, + Tab, Table, TableBody, TableCell, TableContainer, TableHead, TableRow, + Tabs, Tooltip, Typography, useTheme @@ -27,6 +29,7 @@ import { useSnackbar } from 'notistack'; import { useCallback, useEffect, useState } from 'react'; import ExtendedSidebarLayout from 'src/layouts/ExtendedSidebarLayout'; +import { ClustoxWorkspaceMetrics } from '@/components/ClustoxWorkspaceMetrics'; import { FlexBox } from '@/components/FlexBox'; import { Line } from '@/components/Text'; import { PageWrapper } from '@/content/PullRequests/PageWrapper'; @@ -79,6 +82,7 @@ function WorkspacesPage() { const [workspaces, setWorkspaces] = useState([]); const [loading, setLoading] = useState(true); const [syncing, setSyncing] = useState(false); + const [tab, setTab] = useState<'health' | 'metrics'>('health'); const load = useCallback(async () => { const res = await fetch('/api/clustox/workspace-status'); @@ -219,6 +223,20 @@ function WorkspacesPage() { )} + {isSuperadmin && ( + setTab(v)} + sx={{ borderBottom: 1, borderColor: 'divider' }} + > + + + + )} + + {isSuperadmin && tab === 'metrics' && } + + {(!isSuperadmin || tab === 'health') && ( + )} - - Syncs run automatically every 30 minutes. A failed workspace does not - affect the others. - + {(!isSuperadmin || tab === 'health') && ( + + Syncs run automatically every 30 minutes. A failed workspace does + not affect the others. + + )} ); diff --git a/web-server/src/auth/queries.ts b/web-server/src/auth/queries.ts index e452fcf7..549cbfa4 100644 --- a/web-server/src/auth/queries.ts +++ b/web-server/src/auth/queries.ts @@ -318,6 +318,64 @@ export const listUnownedWorkspaces = async (): Promise< return orgs.filter((o: { id: string }) => !ownedIds.has(o.id)); }; +/** + * Remove a user, and their workspace if removing them would strand it empty. + * + * The user is soft-deleted, matching how upstream treats Users and Teams. The + * workspace is only hard-deleted when it holds nothing worth keeping -- no + * integration, no repositories, and no other owner. A workspace with real data + * is left behind unowned instead, where it can be adopted; silently destroying + * synced history because the last admin left would be the wrong default. + * + * Returns whether the workspace was removed, so callers can say what happened. + */ +export const deleteUserAndEmptyWorkspace = async ( + userId: string +): Promise<{ workspaceRemoved: boolean }> => { + const user = await db(Table.Users).where('id', userId).first(); + const orgId: string | null = user?.org_id ?? null; + + // org_id is cleared as well as flagging the row deleted: a removed user + // should not still own a workspace, and the Users.org_id foreign key would + // otherwise block removing the workspace below. + await db(Table.Users) + .where('id', userId) + .update({ is_deleted: true, org_id: null, updated_at: new Date() }); + await db(Table.ClustoxUserAuth).where('user_id', userId).delete(); + await db(Table.ClustoxUserTeamAccess).where('user_id', userId).delete(); + + if (!orgId) return { workspaceRemoved: false }; + + const [otherOwners, repos, integrations] = await Promise.all([ + db(Table.Users) + .where('org_id', orgId) + .andWhere('is_deleted', false) + .whereNot('id', userId) + .select('id'), + db(Table.OrgRepo).where('org_id', orgId).select('id'), + db(Table.Integration).where('org_id', orgId).select('org_id') + ]); + + const worthKeeping = + otherOwners.length > 0 || repos.length > 0 || integrations.length > 0; + if (worthKeeping) return { workspaceRemoved: false }; + + // Nothing references the workspace except its own teams, and the foreign + // keys have no cascade, so those go first. + await db(Table.ClustoxUserTeamAccess) + .whereIn( + 'team_id', + db(Table.Team).where('org_id', orgId).select('id') + ) + .delete(); + await db(Table.Team).where('org_id', orgId).delete(); + await db(Table.ClustoxSyncRun).where('org_id', orgId).delete(); + await db(Table.ClustoxInvite).where('org_id', orgId).update({ org_id: null }); + await db(Table.Organization).where('id', orgId).delete(); + + return { workspaceRemoved: true }; +}; + /** Move an admin into a different workspace, or detach them from one. */ export const setUserWorkspace = async ( userId: string, diff --git a/web-server/src/components/ClustoxWorkspaceMetrics.tsx b/web-server/src/components/ClustoxWorkspaceMetrics.tsx new file mode 100644 index 00000000..a62b4dfa --- /dev/null +++ b/web-server/src/components/ClustoxWorkspaceMetrics.tsx @@ -0,0 +1,205 @@ +import { + Alert, + CircularProgress, + MenuItem, + Table, + TableBody, + TableCell, + TableContainer, + TableHead, + TableRow, + TextField, + Typography, + useTheme +} from '@mui/material'; +import { useCallback, useEffect, useState } from 'react'; + +import { FlexBox } from '@/components/FlexBox'; +import { Line } from '@/components/Text'; + +type Row = { + id: string; + name: string; + ownerEmail: string | null; + hasIntegration: boolean; + teamCount: number; + repoCount: number; + prCount: number; + deployments: number; + leadTimeSeconds: number | null; +}; + +const WINDOWS = [ + { days: 7, label: 'Last 7 days' }, + { days: 30, label: 'Last 30 days' }, + { days: 90, label: 'Last 90 days' } +]; + +const Mono = ({ children }: { children: React.ReactNode }) => ( + + {children} + +); + +/** Seconds to the coarsest unit that still reads precisely. */ +const duration = (seconds: number | null) => { + if (seconds === null) return '—'; + const h = seconds / 3600; + if (h < 1) return `${Math.round(seconds / 60)}m`; + if (h < 48) return `${Math.round(h)}h`; + return `${(h / 24).toFixed(1)}d`; +}; + +/** + * CLUSTOX: DORA metrics across every workspace, for a superadmin. + * + * Without this, comparing workspaces means switching into each one and reading + * its dashboard separately. + */ +export const ClustoxWorkspaceMetrics = () => { + const theme = useTheme(); + const [rows, setRows] = useState([]); + const [days, setDays] = useState(30); + const [loading, setLoading] = useState(true); + const [failed, setFailed] = useState(false); + + const load = useCallback(async () => { + setLoading(true); + setFailed(false); + + const to = new Date(); + const from = new Date(to.getTime() - days * 24 * 60 * 60 * 1000); + const params = new URLSearchParams({ + // The analytics API rejects timestamps without timezone info. + from_time: from.toISOString().replace('Z', '+00:00'), + to_time: to.toISOString().replace('Z', '+00:00') + }); + + try { + const res = await fetch(`/api/clustox/overview?${params}`); + if (!res.ok) throw new Error('failed'); + setRows(await res.json()); + } catch { + setFailed(true); + } finally { + setLoading(false); + } + }, [days]); + + useEffect(() => { + load(); + }, [load]); + + const withData = rows.filter((r) => r.prCount > 0); + + return ( + + + + {withData.length} of {rows.length} workspaces have activity in this + window + + setDays(Number(e.target.value))} + > + {WINDOWS.map((w) => ( + + {w.label} + + ))} + + + + {failed && ( + + Could not load metrics. The analytics server may be unreachable. + + )} + + {loading ? ( + + + + ) : ( + + + + + Workspace + + Lead time + + + Deployments + + + PRs + + + Teams + + + + + {rows.map((r) => { + const idle = r.prCount === 0; + return ( + + + + {r.name} + + {r.ownerEmail ?? 'no owner'} + + + + + {idle ? ( + + {r.hasIntegration ? 'no activity' : 'not connected'} + + ) : ( + {duration(r.leadTimeSeconds)} + )} + + + {idle ? '—' : r.deployments} + + + {idle ? '—' : r.prCount} + + + {r.teamCount} + + + ); + })} + +
+
+ )} + + + Lead time is weighted by pull request count, so a workspace with few + merges cannot skew the comparison. Workspaces with no merged pull + requests in the window show no figure rather than a zero. + +
+ ); +}; From 9ec95debfe86c01b71670ca21d9ea7a88c1cd52f Mon Sep 17 00:00:00 2001 From: Muhammad Ameer Hamza Date: Fri, 7 Aug 2026 14:45:06 +0500 Subject: [PATCH 034/194] slack alerts configured --- Jenkinsfile | 17 +++++++++++++++-- 1 file changed, 15 insertions(+), 2 deletions(-) diff --git a/Jenkinsfile b/Jenkinsfile index 46e2cd45..20f0e5ef 100644 --- a/Jenkinsfile +++ b/Jenkinsfile @@ -41,10 +41,23 @@ pipeline { post { success { echo 'Deployment succeeded.' + withCredentials([string(credentialsId: 'slack-webhook-ci-build-alerts', variable: 'SLACK_WEBHOOK_URL')]) { + sh ''' + curl -X POST -H "Content-type: application/json" \ + --data "{\\"text\\":\\"✅ *middleware* deployment succeeded — build #${BUILD_NUMBER} — ${BUILD_URL}\\"}" \ + "$SLACK_WEBHOOK_URL" + ''' + } } failure { echo 'Deployment failed - check console output.' + withCredentials([string(credentialsId: 'slack-webhook-ci-build-alerts', variable: 'SLACK_WEBHOOK_URL')]) { + sh ''' + curl -X POST -H "Content-type: application/json" \ + --data "{\\"text\\":\\"❌ *middleware* deployment failed — build #${BUILD_NUMBER} — ${BUILD_URL}\\"}" \ + "$SLACK_WEBHOOK_URL" + ''' + } } } -} - +} \ No newline at end of file From df8d2f17db5b1c080339df7db4fcf87f771c01af Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Fri, 7 Aug 2026 15:36:00 +0500 Subject: [PATCH 035/194] docs: drop internal spec and plan scaffolding These were working notes for building the auth feature, not documentation of the system. The architecture and fork-strategy docs stay. --- .../superpowers/plans/2026-08-05-auth-rbac.md | 2049 ----------------- .../specs/2026-08-05-auth-rbac-design.md | 368 --- 2 files changed, 2417 deletions(-) delete mode 100644 docs/superpowers/plans/2026-08-05-auth-rbac.md delete mode 100644 docs/superpowers/specs/2026-08-05-auth-rbac-design.md diff --git a/docs/superpowers/plans/2026-08-05-auth-rbac.md b/docs/superpowers/plans/2026-08-05-auth-rbac.md deleted file mode 100644 index 11475e49..00000000 --- a/docs/superpowers/plans/2026-08-05-auth-rbac.md +++ /dev/null @@ -1,2049 +0,0 @@ -# Authentication & RBAC Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Require login to reach any dashboard, with superadmins managing users and integrations and admins seeing only their assigned teams. - -**Architecture:** Authentication uses next-auth v4 with a Credentials provider and JWT sessions. Enforcement happens at three surfaces: `middleware.ts` guards pages, `Endpoint.serve()` guards all 47 BFF API routes from one insertion point, and a shared-secret header locks the Flask servers to BFF-only traffic. Because Flask becomes unreachable except through the BFF, **all authorization logic lives in the BFF** — no user context is threaded through Python. - -**Tech Stack:** Next.js 15 (pages router), next-auth 4.23, TypeScript, knex 2.4, yup 0.32, bcryptjs, Postgres 15, dbmate, Flask 3, Jest, pytest, Playwright. - -**Spec:** [2026-08-05-auth-rbac-design.md](../specs/2026-08-05-auth-rbac-design.md) - -## Global Constraints - -- **Roles are exactly two:** `SUPERADMIN` and `ADMIN`. No third role. -- **Never modify upstream tables.** All new schema goes in `ClustoxUserAuth` and `ClustoxUserTeamAccess`. -- **Every edit to an upstream file gets a `CLUSTOX:` sentinel** — `// CLUSTOX: ` for TS, `# CLUSTOX:` for Python, `-- CLUSTOX:` for SQL. Per [FORK_STRATEGY.md §4](../../FORK_STRATEGY.md). -- **Fail closed.** If `INTERNAL_API_TOKEN` is unset, Flask rejects every request. Never default to allow. -- **Never log or return password hashes.** -- **bcryptjs, not bcrypt.** Pure JS, no native compilation — `bcrypt` breaks the Docker build. -- **Migration filenames** use `_clustox_.sql`, e.g. `20260805150000_clustox_auth.sql`. -- Do not touch `web-server/src/components/Authenticated/index.tsx`. It is a no-op stub; `middleware.ts` does the real work, and rewriting it only adds divergence. - -## Two corrections to the spec - -1. **Seven upstream files, not six.** `setup_utils/cronjob.txt` also changes — cron calls `:9697/sync` and must send the internal token once Flask is locked down. Handled in Task 9. -2. **`instrumentation.ts` is the bootstrap hook.** The spec said "on first boot" without saying where. Next 15 supports `instrumentation.ts` with an exported `register()`; that is the mechanism. - ---- - -## File Structure - -**New files (green zone — never conflict with upstream):** - -| File | Responsibility | -|---|---| -| `database-docker/db/migrations/20260805150000_clustox_auth.sql` | Two new tables | -| `web-server/src/auth/types.ts` | `ClustoxRole`, `AuthSession` — shared types | -| `web-server/src/auth/password.ts` | Hash and verify only | -| `web-server/src/auth/queries.ts` | All DB access for auth tables | -| `web-server/src/auth/guard.ts` | Authorization decisions (pure where possible) | -| `web-server/src/auth/session.ts` | Read the session inside API routes | -| `web-server/src/auth/bootstrap.ts` | Seed the first superadmin | -| `web-server/instrumentation.ts` | Calls `bootstrap()` on server start | -| `web-server/pages/api/auth/[...nextauth].ts` | next-auth config | -| `web-server/pages/login.tsx` | Sign-in page | -| `web-server/pages/users.tsx` | User management (superadmin) | -| `web-server/pages/api/clustox/users/index.ts` | List + create users | -| `web-server/pages/api/clustox/users/[user_id].ts` | Update role/teams, delete | -| `backend/analytics_server/mhq/clustox_auth/__init__.py` | Package marker | -| `backend/analytics_server/mhq/clustox_auth/internal_token.py` | Flask token guard | -| `backend/analytics_server/tests/clustox_auth/test_internal_token.py` | Its tests | -| `setup_utils/trigger_sync.sh` | Cron helper that sends the token | - -**Upstream files modified (seven):** - -| File | Change | -|---|---| -| `web-server/src/constants/db.ts` | Declare two tables (append-only) | -| `web-server/src/api-helpers/global.ts` | Enforce the existing `authenticated` flag | -| `web-server/src/api-helpers/axios.ts` | Stamp `X-Internal-Token` | -| `web-server/middleware.ts` | Redirect unauthenticated to `/login` | -| `backend/analytics_server/app.py` | Register `before_request` | -| `backend/analytics_server/sync_app.py` | Register `before_request` | -| `setup_utils/cronjob.txt` | Call `trigger_sync.sh` instead of raw curl | - ---- - -## Task 1: Database schema and knex declarations - -**Files:** -- Create: `database-docker/db/migrations/20260805150000_clustox_auth.sql` -- Modify: `database-docker/db/schema.sql` (append table definitions) -- Modify: `web-server/src/constants/db.ts` - -**Interfaces:** -- Consumes: nothing -- Produces: tables `ClustoxUserAuth(user_id, password_hash, role, created_at, updated_at)` and `ClustoxUserTeamAccess(user_id, team_id, created_at)`; knex accessors `Table.ClustoxUserAuth`, `Table.ClustoxUserTeamAccess`, and their `Columns` entries - -- [ ] **Step 1: Write the migration** - -Create `database-docker/db/migrations/20260805150000_clustox_auth.sql`: - -```sql --- migrate:up --- CLUSTOX: authentication and role-based access tables. Deliberately separate --- from upstream "Users" so upstream migrations can never collide with ours. -CREATE TABLE "ClustoxUserAuth" ( - user_id uuid PRIMARY KEY REFERENCES "Users"(id) ON DELETE CASCADE, - password_hash text NOT NULL, - role varchar NOT NULL CHECK (role IN ('SUPERADMIN','ADMIN')), - created_at timestamptz NOT NULL DEFAULT now(), - updated_at timestamptz NOT NULL DEFAULT now() -); - -CREATE TABLE "ClustoxUserTeamAccess" ( - user_id uuid NOT NULL REFERENCES "Users"(id) ON DELETE CASCADE, - team_id uuid NOT NULL REFERENCES "Team"(id) ON DELETE CASCADE, - created_at timestamptz NOT NULL DEFAULT now(), - PRIMARY KEY (user_id, team_id) -); - -CREATE INDEX idx_clustox_user_team_access_user ON "ClustoxUserTeamAccess"(user_id); - --- migrate:down -DROP TABLE "ClustoxUserTeamAccess"; -DROP TABLE "ClustoxUserAuth"; -``` - -- [ ] **Step 2: Apply it and verify the tables exist** - -```bash -docker compose up -d -docker exec middleware-dev bash -c 'cd /app && dbmate -u "postgresql://postgres:postgres@localhost:5434/mhq-oss?sslmode=disable" up' -docker exec middleware-dev psql "postgresql://postgres:postgres@localhost:5434/mhq-oss" -c '\d "ClustoxUserAuth"' -``` - -Expected: table definition printed with all five columns and the CHECK constraint. - -- [ ] **Step 3: Verify the CHECK constraint rejects a bad role** - -```bash -docker exec middleware-dev psql "postgresql://postgres:postgres@localhost:5434/mhq-oss" \ - -c "INSERT INTO \"ClustoxUserAuth\" (user_id, password_hash, role) VALUES (gen_random_uuid(), 'x', 'HACKER');" -``` - -Expected: fails with a foreign key violation **or** a check constraint violation. Either proves the guards are live. - -- [ ] **Step 4: Append the same DDL to `schema.sql`** - -Paste both `CREATE TABLE` statements and the `CREATE INDEX` into `database-docker/db/schema.sql`, above the final `schema_migrations` INSERT block. Keeping this file current is required — see [ARCHITECTURE.md §12.5](../../ARCHITECTURE.md). - -- [ ] **Step 5: Declare the tables for knex** - -In `web-server/src/constants/db.ts`, add to the `TableT` enum (append at the end, never mid-list — trailing additions conflict far less): - -```ts -enum TableT { - // ... existing entries unchanged ... - // CLUSTOX: auth tables — see docs/superpowers/specs/2026-08-05-auth-rbac-design.md - ClustoxUserAuth, - ClustoxUserTeamAccess -} -``` - -Then append to the `Columns` object: - -```ts -export const Columns = { - // ... existing entries unchanged ... - // CLUSTOX: auth tables - [Table.ClustoxUserAuth]: objectEnumFromFn(() => { - enum Columns { - user_id, - password_hash, - role, - created_at, - updated_at - } - return Columns; - }), - [Table.ClustoxUserTeamAccess]: objectEnumFromFn(() => { - enum Columns { - user_id, - team_id, - created_at - } - return Columns; - }) -}; -``` - -- [ ] **Step 6: Verify TypeScript compiles** - -Run: `cd web-server && yarn tsc --noEmit` -Expected: no errors mentioning `ClustoxUserAuth`. - -- [ ] **Step 7: Commit** - -```bash -git add database-docker/db/migrations/20260805150000_clustox_auth.sql database-docker/db/schema.sql web-server/src/constants/db.ts -git commit -m "feat(auth): add ClustoxUserAuth and ClustoxUserTeamAccess tables" -``` - ---- - -## Task 2: Password hashing - -**Files:** -- Create: `web-server/src/auth/password.ts` -- Create: `web-server/src/auth/__tests__/password.test.ts` -- Modify: `web-server/package.json` (add `bcryptjs`, `@types/bcryptjs`) - -**Interfaces:** -- Consumes: nothing -- Produces: `hashPassword(plain: string): Promise`, `verifyPassword(plain: string, hash: string): Promise` - -- [ ] **Step 1: Install bcryptjs** - -```bash -cd web-server && yarn add bcryptjs && yarn add -D @types/bcryptjs -``` - -Use `bcryptjs`, not `bcrypt` — the latter needs native compilation and breaks the Docker build. - -- [ ] **Step 2: Write the failing test** - -Create `web-server/src/auth/__tests__/password.test.ts`: - -```ts -import { hashPassword, verifyPassword } from '@/auth/password'; - -describe('password', () => { - it('produces a hash that is not the plaintext', async () => { - const hash = await hashPassword('correct-horse-battery'); - expect(hash).not.toBe('correct-horse-battery'); - expect(hash.length).toBeGreaterThan(50); - }); - - it('verifies a correct password', async () => { - const hash = await hashPassword('correct-horse-battery'); - await expect(verifyPassword('correct-horse-battery', hash)).resolves.toBe(true); - }); - - it('rejects an incorrect password', async () => { - const hash = await hashPassword('correct-horse-battery'); - await expect(verifyPassword('wrong-password', hash)).resolves.toBe(false); - }); - - it('produces different hashes for the same input (salted)', async () => { - const a = await hashPassword('same-input'); - const b = await hashPassword('same-input'); - expect(a).not.toBe(b); - }); - - it('returns false rather than throwing on a malformed hash', async () => { - await expect(verifyPassword('anything', 'not-a-hash')).resolves.toBe(false); - }); -}); -``` - -- [ ] **Step 3: Run it to confirm it fails** - -Run: `cd web-server && yarn jest src/auth/__tests__/password.test.ts` -Expected: FAIL — "Cannot find module '@/auth/password'". - -- [ ] **Step 4: Implement** - -Create `web-server/src/auth/password.ts`: - -```ts -import { compare, hash } from 'bcryptjs'; - -const COST = 12; - -export const hashPassword = (plain: string): Promise => - hash(plain, COST); - -export const verifyPassword = async ( - plain: string, - passwordHash: string -): Promise => { - try { - return await compare(plain, passwordHash); - } catch { - // A malformed stored hash must read as "wrong password", never as an - // exception that a caller might treat as success. - return false; - } -}; -``` - -- [ ] **Step 5: Run tests to verify they pass** - -Run: `cd web-server && yarn jest src/auth/__tests__/password.test.ts` -Expected: 5 passed. - -- [ ] **Step 6: Commit** - -```bash -git add web-server/src/auth/password.ts web-server/src/auth/__tests__/password.test.ts web-server/package.json web-server/yarn.lock -git commit -m "feat(auth): add bcrypt password hashing" -``` - ---- - -## Task 3: Shared types and auth queries - -**Files:** -- Create: `web-server/src/auth/types.ts` -- Create: `web-server/src/auth/queries.ts` -- Create: `web-server/src/auth/__tests__/queries.test.ts` - -**Interfaces:** -- Consumes: `Table`, `Columns` from Task 1; `hashPassword` from Task 2 -- Produces: - - `type ClustoxRole = 'SUPERADMIN' | 'ADMIN'` - - `interface AuthSession { userId: string; email: string; name: string; role: ClustoxRole }` - - `interface AuthUserRow { userId: string; email: string; name: string; role: ClustoxRole; passwordHash: string }` - - `interface AuthUserListItem { userId: string; email: string; name: string; role: ClustoxRole; teamIds: string[] }` - - `getAuthUserByEmail(email: string): Promise` - - `getTeamIdsForUser(userId: string): Promise` - - `getAllTeamIds(): Promise` - - `listUsers(): Promise` - - `createUser(input: { name: string; email: string; password: string; role: ClustoxRole; teamIds: string[]; orgId: string }): Promise` - - `updateUserRole(userId: string, role: ClustoxRole): Promise` - - `setUserTeams(userId: string, teamIds: string[]): Promise` - - `countSuperadmins(): Promise` - -- [ ] **Step 1: Write the types** - -Create `web-server/src/auth/types.ts`: - -```ts -export type ClustoxRole = 'SUPERADMIN' | 'ADMIN'; - -export interface AuthSession { - userId: string; - email: string; - name: string; - role: ClustoxRole; -} - -export interface AuthUserRow { - userId: string; - email: string; - name: string; - role: ClustoxRole; - passwordHash: string; -} - -export interface AuthUserListItem { - userId: string; - email: string; - name: string; - role: ClustoxRole; - teamIds: string[]; -} -``` - -- [ ] **Step 2: Write the failing test** - -Create `web-server/src/auth/__tests__/queries.test.ts`. The knex module is mocked, so this tests query construction and row mapping without a database: - -```ts -const mockDb = jest.fn(); -jest.mock('@/utils/db', () => ({ - db: (...args: any[]) => mockDb(...args) -})); - -import { getAuthUserByEmail, getTeamIdsForUser } from '@/auth/queries'; - -const chain = (result: any) => { - const c: any = {}; - c.select = jest.fn(() => c); - c.join = jest.fn(() => c); - c.where = jest.fn(() => c); - c.andWhere = jest.fn(() => c); - c.first = jest.fn(() => Promise.resolve(result)); - c.then = (fn: any) => Promise.resolve(result).then(fn); - return c; -}; - -describe('getAuthUserByEmail', () => { - beforeEach(() => mockDb.mockReset()); - - it('maps a row to AuthUserRow', async () => { - mockDb.mockReturnValue( - chain({ - user_id: 'u1', - primary_email: 'a@clustox.com', - name: 'Ayesha', - role: 'ADMIN', - password_hash: 'hashed' - }) - ); - - await expect(getAuthUserByEmail('a@clustox.com')).resolves.toEqual({ - userId: 'u1', - email: 'a@clustox.com', - name: 'Ayesha', - role: 'ADMIN', - passwordHash: 'hashed' - }); - }); - - it('returns null when no row matches', async () => { - mockDb.mockReturnValue(chain(undefined)); - await expect(getAuthUserByEmail('nobody@clustox.com')).resolves.toBeNull(); - }); -}); - -describe('getTeamIdsForUser', () => { - beforeEach(() => mockDb.mockReset()); - - it('returns a flat array of team ids', async () => { - mockDb.mockReturnValue(chain([{ team_id: 't1' }, { team_id: 't2' }])); - await expect(getTeamIdsForUser('u1')).resolves.toEqual(['t1', 't2']); - }); - - it('returns an empty array when the user has no teams', async () => { - mockDb.mockReturnValue(chain([])); - await expect(getTeamIdsForUser('u1')).resolves.toEqual([]); - }); -}); -``` - -- [ ] **Step 3: Run it to confirm it fails** - -Run: `cd web-server && yarn jest src/auth/__tests__/queries.test.ts` -Expected: FAIL — "Cannot find module '@/auth/queries'". - -- [ ] **Step 4: Implement the queries** - -Create `web-server/src/auth/queries.ts`: - -```ts -import { Columns, Table } from '@/constants/db'; -import { db } from '@/utils/db'; - -import { hashPassword } from './password'; -import { AuthUserListItem, AuthUserRow, ClustoxRole } from './types'; - -export const getAuthUserByEmail = async ( - email: string -): Promise => { - const row = await db(Table.ClustoxUserAuth) - .join(Table.Users, `${Table.Users}.id`, `${Table.ClustoxUserAuth}.user_id`) - .where(`${Table.Users}.primary_email`, email) - .andWhere(`${Table.Users}.is_deleted`, false) - .select( - `${Table.ClustoxUserAuth}.user_id`, - `${Table.ClustoxUserAuth}.role`, - `${Table.ClustoxUserAuth}.password_hash`, - `${Table.Users}.primary_email`, - `${Table.Users}.name` - ) - .first(); - - if (!row) return null; - - return { - userId: row.user_id, - email: row.primary_email, - name: row.name, - role: row.role as ClustoxRole, - passwordHash: row.password_hash - }; -}; - -export const getTeamIdsForUser = async (userId: string): Promise => { - const rows = await db(Table.ClustoxUserTeamAccess) - .select(Columns[Table.ClustoxUserTeamAccess].team_id) - .where(Columns[Table.ClustoxUserTeamAccess].user_id, userId); - return rows.map((r: { team_id: string }) => r.team_id); -}; - -export const getAllTeamIds = async (): Promise => { - const rows = await db(Table.Team).select('id').where('is_deleted', false); - return rows.map((r: { id: string }) => r.id); -}; - -export const countSuperadmins = async (): Promise => { - const rows = await db(Table.ClustoxUserAuth) - .where(Columns[Table.ClustoxUserAuth].role, 'SUPERADMIN') - .select(Columns[Table.ClustoxUserAuth].user_id); - return rows.length; -}; - -export const listUsers = async (): Promise => { - const rows = await db(Table.ClustoxUserAuth) - .join(Table.Users, `${Table.Users}.id`, `${Table.ClustoxUserAuth}.user_id`) - .where(`${Table.Users}.is_deleted`, false) - .select( - `${Table.ClustoxUserAuth}.user_id`, - `${Table.ClustoxUserAuth}.role`, - `${Table.Users}.primary_email`, - `${Table.Users}.name` - ); - - const access = await db(Table.ClustoxUserTeamAccess).select('user_id', 'team_id'); - - return rows.map((r: any) => ({ - userId: r.user_id, - email: r.primary_email, - name: r.name, - role: r.role as ClustoxRole, - teamIds: access - .filter((a: any) => a.user_id === r.user_id) - .map((a: any) => a.team_id) - })); -}; - -export const createUser = async (input: { - name: string; - email: string; - password: string; - role: ClustoxRole; - teamIds: string[]; - orgId: string; -}): Promise => { - const passwordHash = await hashPassword(input.password); - - const [user] = await db(Table.Users) - .insert({ - org_id: input.orgId, - name: input.name, - primary_email: input.email, - is_deleted: false - }) - .returning('id'); - - const userId = typeof user === 'string' ? user : user.id; - - await db(Table.ClustoxUserAuth).insert({ - user_id: userId, - password_hash: passwordHash, - role: input.role - }); - - if (input.teamIds.length) { - await db(Table.ClustoxUserTeamAccess).insert( - input.teamIds.map((team_id) => ({ user_id: userId, team_id })) - ); - } - - return userId; -}; - -export const updateUserRole = async ( - userId: string, - role: ClustoxRole -): Promise => { - await db(Table.ClustoxUserAuth) - .where(Columns[Table.ClustoxUserAuth].user_id, userId) - .update({ role, updated_at: new Date() }); -}; - -export const setUserTeams = async ( - userId: string, - teamIds: string[] -): Promise => { - await db(Table.ClustoxUserTeamAccess) - .where(Columns[Table.ClustoxUserTeamAccess].user_id, userId) - .delete(); - - if (teamIds.length) { - await db(Table.ClustoxUserTeamAccess).insert( - teamIds.map((team_id) => ({ user_id: userId, team_id })) - ); - } -}; -``` - -- [ ] **Step 5: Run tests to verify they pass** - -Run: `cd web-server && yarn jest src/auth/__tests__/queries.test.ts` -Expected: 4 passed. - -- [ ] **Step 6: Commit** - -```bash -git add web-server/src/auth/types.ts web-server/src/auth/queries.ts web-server/src/auth/__tests__/queries.test.ts -git commit -m "feat(auth): add auth types and data access queries" -``` - ---- - -## Task 4: Authorization guards - -This is the core of the feature. Test it hardest. - -**Files:** -- Create: `web-server/src/auth/guard.ts` -- Create: `web-server/src/auth/__tests__/guard.test.ts` - -**Interfaces:** -- Consumes: `AuthSession`, `ClustoxRole` from Task 3; `getTeamIdsForUser`, `getAllTeamIds` from Task 3; `ResponseError`, `Errors` from `@/constants/error` -- Produces: - - `assertAuthenticated(session: AuthSession | null): AuthSession` — throws 401, returns the narrowed session - - `assertRole(session: AuthSession | null, role: ClustoxRole): AuthSession` — throws 403 - - `canAccessAllTeams(session: AuthSession): boolean` - - `assertTeamAccess(session: AuthSession | null, teamId: string): Promise` — throws 403 - - `getAccessibleTeamIds(session: AuthSession): Promise` — always concrete ids - -- [ ] **Step 1: Write the failing test** - -Create `web-server/src/auth/__tests__/guard.test.ts`: - -```ts -jest.mock('@/auth/queries', () => ({ - getTeamIdsForUser: jest.fn(), - getAllTeamIds: jest.fn() -})); - -import { - assertAuthenticated, - assertRole, - assertTeamAccess, - canAccessAllTeams, - getAccessibleTeamIds -} from '@/auth/guard'; -import { getAllTeamIds, getTeamIdsForUser } from '@/auth/queries'; -import { AuthSession } from '@/auth/types'; - -const superadmin: AuthSession = { - userId: 'su1', - email: 'boss@clustox.com', - name: 'Boss', - role: 'SUPERADMIN' -}; -const admin: AuthSession = { - userId: 'ad1', - email: 'lead@clustox.com', - name: 'Lead', - role: 'ADMIN' -}; - -describe('assertAuthenticated', () => { - it('returns the session when present', () => { - expect(assertAuthenticated(admin)).toBe(admin); - }); - - it('throws 401 when the session is null', () => { - expect(() => assertAuthenticated(null)).toThrow( - expect.objectContaining({ status: 401 }) - ); - }); -}); - -describe('assertRole', () => { - it('passes when the role matches', () => { - expect(assertRole(superadmin, 'SUPERADMIN')).toBe(superadmin); - }); - - it('throws 403 when an admin attempts a superadmin action', () => { - expect(() => assertRole(admin, 'SUPERADMIN')).toThrow( - expect.objectContaining({ status: 403 }) - ); - }); - - it('throws 401 when unauthenticated, not 403', () => { - expect(() => assertRole(null, 'SUPERADMIN')).toThrow( - expect.objectContaining({ status: 401 }) - ); - }); -}); - -describe('canAccessAllTeams', () => { - it('is true for superadmin', () => { - expect(canAccessAllTeams(superadmin)).toBe(true); - }); - it('is false for admin', () => { - expect(canAccessAllTeams(admin)).toBe(false); - }); -}); - -describe('assertTeamAccess', () => { - beforeEach(() => jest.clearAllMocks()); - - it('allows superadmin any team without querying', async () => { - await expect(assertTeamAccess(superadmin, 'any-team')).resolves.toBeUndefined(); - expect(getTeamIdsForUser).not.toHaveBeenCalled(); - }); - - it('allows an admin their assigned team', async () => { - (getTeamIdsForUser as jest.Mock).mockResolvedValue(['t1', 't2']); - await expect(assertTeamAccess(admin, 't2')).resolves.toBeUndefined(); - }); - - it('throws 403 when an admin requests a team they do not own', async () => { - (getTeamIdsForUser as jest.Mock).mockResolvedValue(['t1']); - await expect(assertTeamAccess(admin, 't-other')).rejects.toMatchObject({ - status: 403 - }); - }); - - it('throws 403 when an admin has no teams at all', async () => { - (getTeamIdsForUser as jest.Mock).mockResolvedValue([]); - await expect(assertTeamAccess(admin, 't1')).rejects.toMatchObject({ - status: 403 - }); - }); - - it('throws 401 when unauthenticated', async () => { - await expect(assertTeamAccess(null, 't1')).rejects.toMatchObject({ - status: 401 - }); - }); -}); - -describe('getAccessibleTeamIds', () => { - beforeEach(() => jest.clearAllMocks()); - - it('returns every team for superadmin', async () => { - (getAllTeamIds as jest.Mock).mockResolvedValue(['t1', 't2', 't3']); - await expect(getAccessibleTeamIds(superadmin)).resolves.toEqual([ - 't1', - 't2', - 't3' - ]); - }); - - it('returns only assigned teams for admin', async () => { - (getTeamIdsForUser as jest.Mock).mockResolvedValue(['t2']); - await expect(getAccessibleTeamIds(admin)).resolves.toEqual(['t2']); - }); -}); -``` - -- [ ] **Step 2: Run it to confirm it fails** - -Run: `cd web-server && yarn jest src/auth/__tests__/guard.test.ts` -Expected: FAIL — "Cannot find module '@/auth/guard'". - -- [ ] **Step 3: Implement** - -Create `web-server/src/auth/guard.ts`: - -```ts -import { Errors, ResponseError } from '@/constants/error'; - -import { getAllTeamIds, getTeamIdsForUser } from './queries'; -import { AuthSession, ClustoxRole } from './types'; - -export const assertAuthenticated = ( - session: AuthSession | null -): AuthSession => { - if (!session) throw new ResponseError(Errors.SESSION_USER_NOT_FOUND, 401); - return session; -}; - -export const assertRole = ( - session: AuthSession | null, - role: ClustoxRole -): AuthSession => { - const authed = assertAuthenticated(session); - if (authed.role !== role) throw new ResponseError(Errors.ACCESS_DENIED, 403); - return authed; -}; - -export const canAccessAllTeams = (session: AuthSession): boolean => - session.role === 'SUPERADMIN'; - -export const assertTeamAccess = async ( - session: AuthSession | null, - teamId: string -): Promise => { - const authed = assertAuthenticated(session); - if (canAccessAllTeams(authed)) return; - - const teamIds = await getTeamIdsForUser(authed.userId); - if (!teamIds.includes(teamId)) { - throw new ResponseError(Errors.ACCESS_DENIED, 403); - } -}; - -export const getAccessibleTeamIds = async ( - session: AuthSession -): Promise => - canAccessAllTeams(session) - ? getAllTeamIds() - : getTeamIdsForUser(session.userId); -``` - -- [ ] **Step 4: Run tests to verify they pass** - -Run: `cd web-server && yarn jest src/auth/__tests__/guard.test.ts` -Expected: 14 passed. - -- [ ] **Step 5: Commit** - -```bash -git add web-server/src/auth/guard.ts web-server/src/auth/__tests__/guard.test.ts -git commit -m "feat(auth): add role and team-access guards" -``` - ---- - -## Task 5: next-auth configuration and session reader - -**Files:** -- Create: `web-server/pages/api/auth/[...nextauth].ts` -- Create: `web-server/src/auth/session.ts` -- Create: `web-server/src/types/next-auth.d.ts` -- Modify: `env.example` and your local `.env` - -**Interfaces:** -- Consumes: `getAuthUserByEmail` (Task 3), `verifyPassword` (Task 2), `AuthSession` (Task 3) -- Produces: `getAuthSession(req: NextApiRequest): Promise` - -- [ ] **Step 1: Add the environment variables** - -Append to `env.example` and to your local `.env`: - -```bash -# CLUSTOX: auth -NEXTAUTH_URL=http://localhost:3333 -NEXTAUTH_SECRET=replace-me-with-openssl-rand-base64-32 -INTERNAL_API_TOKEN=replace-me-with-openssl-rand-hex-32 -SUPERADMIN_EMAIL=admin@clustox.com -SUPERADMIN_PASSWORD=change-this-before-any-shared-deployment -``` - -Generate real values: - -```bash -echo "NEXTAUTH_SECRET=$(openssl rand -base64 32)" -echo "INTERNAL_API_TOKEN=$(openssl rand -hex 32)" -``` - -- [ ] **Step 2: Declare the session type augmentation** - -Create `web-server/src/types/next-auth.d.ts`: - -```ts -import { ClustoxRole } from '@/auth/types'; - -declare module 'next-auth' { - interface Session { - user: { - id: string; - email: string; - name: string; - role: ClustoxRole; - }; - } - interface User { - id: string; - email: string; - name: string; - role: ClustoxRole; - } -} - -declare module 'next-auth/jwt' { - interface JWT { - userId: string; - role: ClustoxRole; - } -} -``` - -- [ ] **Step 3: Write the next-auth route** - -Create `web-server/pages/api/auth/[...nextauth].ts`: - -```ts -import NextAuth, { NextAuthOptions } from 'next-auth'; -import CredentialsProvider from 'next-auth/providers/credentials'; - -import { verifyPassword } from '@/auth/password'; -import { getAuthUserByEmail } from '@/auth/queries'; - -export const authOptions: NextAuthOptions = { - session: { strategy: 'jwt', maxAge: 12 * 60 * 60 }, - pages: { signIn: '/login' }, - providers: [ - CredentialsProvider({ - name: 'Credentials', - credentials: { - email: { label: 'Email', type: 'text' }, - password: { label: 'Password', type: 'password' } - }, - async authorize(credentials) { - if (!credentials?.email || !credentials?.password) return null; - - const user = await getAuthUserByEmail(credentials.email); - if (!user) return null; - - const ok = await verifyPassword(credentials.password, user.passwordHash); - if (!ok) return null; - - return { - id: user.userId, - email: user.email, - name: user.name, - role: user.role - }; - } - }) - ], - callbacks: { - async jwt({ token, user }) { - if (user) { - token.userId = user.id; - token.role = user.role; - } - return token; - }, - async session({ session, token }) { - session.user = { - id: token.userId, - email: token.email as string, - name: token.name as string, - role: token.role - }; - return session; - } - } -}; - -export default NextAuth(authOptions); -``` - -Team assignments are deliberately **not** in the token — see `getAccessibleTeamIds`, which reads them per request so a permission change takes effect immediately. - -- [ ] **Step 4: Write the session reader** - -Create `web-server/src/auth/session.ts`: - -```ts -import { NextApiRequest } from 'next'; -import { getToken } from 'next-auth/jwt'; - -import { AuthSession, ClustoxRole } from './types'; - -export const getAuthSession = async ( - req: NextApiRequest -): Promise => { - const token = await getToken({ req, secret: process.env.NEXTAUTH_SECRET }); - if (!token?.userId) return null; - - return { - userId: token.userId as string, - email: (token.email as string) ?? '', - name: (token.name as string) ?? '', - role: token.role as ClustoxRole - }; -}; -``` - -- [ ] **Step 5: Verify it compiles** - -Run: `cd web-server && yarn tsc --noEmit` -Expected: no errors. - -- [ ] **Step 6: Commit** - -```bash -git add web-server/pages/api/auth web-server/src/auth/session.ts web-server/src/types/next-auth.d.ts env.example -git commit -m "feat(auth): configure next-auth credentials provider" -``` - ---- - -## Task 6: Enforce auth in Endpoint.serve() - -The single highest-value change: one insertion point covers all 47 BFF routes. - -**Files:** -- Modify: `web-server/src/api-helpers/global.ts` -- Create: `web-server/src/api-helpers/__tests__/endpoint-auth.test.ts` - -**Interfaces:** -- Consumes: `getAuthSession` (Task 5), `assertAuthenticated` (Task 4) -- Produces: `req.session` populated on every authenticated request, typed as `AuthSession` - -- [ ] **Step 1: Write the failing test** - -Create `web-server/src/api-helpers/__tests__/endpoint-auth.test.ts`: - -```ts -jest.mock('@/auth/session', () => ({ getAuthSession: jest.fn() })); - -import { getAuthSession } from '@/auth/session'; -import { Endpoint, nullSchema } from '@/api-helpers/global'; - -const mockRes = () => { - const res: any = {}; - res.status = jest.fn(() => res); - res.send = jest.fn(() => res); - return res; -}; - -const mockReq = () => ({ method: 'GET', query: {}, body: {}, headers: {} }) as any; - -describe('Endpoint auth enforcement', () => { - beforeEach(() => jest.clearAllMocks()); - - it('rejects with 401 when no session and the endpoint is authenticated', async () => { - (getAuthSession as jest.Mock).mockResolvedValue(null); - - const endpoint = new Endpoint(nullSchema); - endpoint.handle.GET(nullSchema, async (_req, res) => res.send({ ok: true })); - - const res = mockRes(); - await endpoint.serve()(mockReq(), res); - - expect(res.status).toHaveBeenCalledWith(401); - }); - - it('allows the request through when a session exists', async () => { - (getAuthSession as jest.Mock).mockResolvedValue({ - userId: 'u1', - email: 'a@clustox.com', - name: 'A', - role: 'ADMIN' - }); - - const endpoint = new Endpoint(nullSchema); - endpoint.handle.GET(nullSchema, async (_req, res) => res.send({ ok: true })); - - const res = mockRes(); - await endpoint.serve()(mockReq(), res); - - expect(res.send).toHaveBeenCalledWith({ ok: true }); - }); - - it('skips the check for endpoints marked unauthenticated', async () => { - (getAuthSession as jest.Mock).mockResolvedValue(null); - - const endpoint = new Endpoint(nullSchema, { unauthenticated: true }); - endpoint.handle.GET(nullSchema, async (_req, res) => res.send({ ok: true })); - - const res = mockRes(); - await endpoint.serve()(mockReq(), res); - - expect(res.send).toHaveBeenCalledWith({ ok: true }); - expect(getAuthSession).not.toHaveBeenCalled(); - }); - - it('exposes the session on the request object', async () => { - const session = { - userId: 'u1', - email: 'a@clustox.com', - name: 'A', - role: 'ADMIN' as const - }; - (getAuthSession as jest.Mock).mockResolvedValue(session); - - let seen: any = null; - const endpoint = new Endpoint(nullSchema); - endpoint.handle.GET(nullSchema, async (req, res) => { - seen = (req as any).session; - res.send({ ok: true }); - }); - - await endpoint.serve()(mockReq(), mockRes()); - expect(seen).toEqual(session); - }); -}); -``` - -- [ ] **Step 2: Run it to confirm it fails** - -Run: `cd web-server && yarn jest src/api-helpers/__tests__/endpoint-auth.test.ts` -Expected: FAIL — the first test gets a 200/500 rather than 401, because the flag is never read. - -- [ ] **Step 3: Implement the check** - -In `web-server/src/api-helpers/global.ts`, add the imports: - -```ts -// CLUSTOX: enforce the `authenticated` flag that upstream declared but never read. -import { assertAuthenticated } from '@/auth/guard'; -import { getAuthSession } from '@/auth/session'; -``` - -Then inside `serve()`, immediately after `const req = transformNextRequest(nextReq);`: - -```ts - // CLUSTOX: single enforcement point for all BFF routes. Routes opt out - // with `new Endpoint(schema, { unauthenticated: true })`. - if (this.authenticated) { - const session = await getAuthSession(nextReq); - assertAuthenticated(session); - (req as any).session = session; - } -``` - -- [ ] **Step 4: Run tests to verify they pass** - -Run: `cd web-server && yarn jest src/api-helpers/__tests__/endpoint-auth.test.ts` -Expected: 4 passed. - -- [ ] **Step 5: Run the whole suite for regressions** - -Run: `cd web-server && yarn jest` -Expected: all pass. Any pre-existing API-route test that now 401s must be updated to mock a session — that is a correct failure, not a broken test. - -- [ ] **Step 6: Commit** - -```bash -git add web-server/src/api-helpers/global.ts web-server/src/api-helpers/__tests__/endpoint-auth.test.ts -git commit -m "feat(auth): enforce authentication in Endpoint.serve()" -``` - ---- - -## Task 7: Login page and page-level guard - -**Files:** -- Create: `web-server/pages/login.tsx` -- Modify: `web-server/middleware.ts` - -**Interfaces:** -- Consumes: next-auth route (Task 5) -- Produces: `/login` route; unauthenticated page requests redirect there - -- [ ] **Step 1: Build the login page** - -Create `web-server/pages/login.tsx`: - -```tsx -import { Box, Button, Card, TextField, Typography } from '@mui/material'; -import { signIn } from 'next-auth/react'; -import { useRouter } from 'next/router'; -import { FormEvent, useState } from 'react'; - -export default function Login() { - const router = useRouter(); - const [email, setEmail] = useState(''); - const [password, setPassword] = useState(''); - const [error, setError] = useState(''); - const [busy, setBusy] = useState(false); - - const onSubmit = async (e: FormEvent) => { - e.preventDefault(); - setBusy(true); - setError(''); - - const res = await signIn('credentials', { - email, - password, - redirect: false - }); - - setBusy(false); - - if (res?.error) { - // Deliberately does not distinguish unknown email from wrong password. - setError('Invalid email or password'); - return; - } - router.replace('/'); - }; - - return ( - - - Sign in -
- setEmail(e.target.value)} required - /> - setPassword(e.target.value)} required - /> - {error && ( - {error} - )} - - -
-
- ); -} - -Login.getLayout = (page: React.ReactElement) => page; -``` - -`getLayout` returning the bare page keeps the sidebar shell off the login screen. - -- [ ] **Step 2: Guard pages in middleware** - -Replace `web-server/middleware.ts` with: - -```ts -import { getToken } from 'next-auth/jwt'; -import { NextRequest, NextResponse } from 'next/server'; - -import { getFeaturesFromReq } from '@/api-helpers/features'; -import { defaultFlags } from '@/constants/feature'; - -// CLUSTOX: paths reachable without a session. -const PUBLIC_PATHS = ['/login', '/api/auth', '/api/status', '/api/db_status', '/api/internal_status', '/api/hello']; - -export async function middleware(request: NextRequest) { - const flagOverrides = getFeaturesFromReq(request as any); - const flags = { ...defaultFlags, ...flagOverrides }; - - const url = request.nextUrl.clone(); - const isPublic = PUBLIC_PATHS.some((p) => url.pathname.startsWith(p)); - - // CLUSTOX: redirect unauthenticated page requests to the sign-in screen. - if (!isPublic) { - const token = await getToken({ - req: request, - secret: process.env.NEXTAUTH_SECRET - }); - if (!token) { - const login = request.nextUrl.clone(); - login.pathname = '/login'; - login.search = ''; - return NextResponse.redirect(login); - } - } - - // Forward as-is if it's a next-auth URL, except /session - if ( - url.pathname.startsWith('/api/auth') && - !url.pathname.startsWith('/api/auth/session') - ) { - return NextResponse.next(); - } - - url.searchParams.append('feature_flags', JSON.stringify(flags)); - - return NextResponse.rewrite(url); -} - -export const config = { - matcher: [ - // CLUSTOX: widened from the API-only matcher so pages are guarded too. - '/((?!_next/static|_next/image|favicon.ico|assets|.*\\.png$|.*\\.svg$).*)' - ] -}; -``` - -- [ ] **Step 3: Verify the redirect manually** - -```bash -docker compose up -d --build -curl -s -o /dev/null -w "%{http_code} -> %{redirect_url}\n" http://localhost:3333/dora-metrics -``` - -Expected: `307 -> http://localhost:3333/login` - -- [ ] **Step 4: Verify the login page renders** - -Run: `curl -s http://localhost:3333/login | grep -c "Sign in"` -Expected: at least 1. - -- [ ] **Step 5: Commit** - -```bash -git add web-server/pages/login.tsx web-server/middleware.ts -git commit -m "feat(auth): add login page and page-level guard" -``` - ---- - -## Task 8: Superadmin bootstrap - -**Files:** -- Create: `web-server/src/auth/bootstrap.ts` -- Create: `web-server/instrumentation.ts` -- Create: `web-server/src/auth/__tests__/bootstrap.test.ts` - -**Interfaces:** -- Consumes: `countSuperadmins`, `createUser` (Task 3) -- Produces: `bootstrapSuperadmin(): Promise` - -- [ ] **Step 1: Write the failing test** - -Create `web-server/src/auth/__tests__/bootstrap.test.ts`: - -```ts -jest.mock('@/auth/queries', () => ({ - countSuperadmins: jest.fn(), - createUser: jest.fn() -})); -jest.mock('@/utils/db', () => ({ db: jest.fn() })); - -import { bootstrapSuperadmin } from '@/auth/bootstrap'; -import { countSuperadmins, createUser } from '@/auth/queries'; - -describe('bootstrapSuperadmin', () => { - const env = process.env; - beforeEach(() => { - jest.clearAllMocks(); - process.env = { ...env }; - }); - afterAll(() => { - process.env = env; - }); - - it('does nothing when a superadmin already exists', async () => { - (countSuperadmins as jest.Mock).mockResolvedValue(1); - process.env.SUPERADMIN_EMAIL = 'a@clustox.com'; - process.env.SUPERADMIN_PASSWORD = 'pw'; - - await bootstrapSuperadmin(); - expect(createUser).not.toHaveBeenCalled(); - }); - - it('does not create a user when env vars are missing', async () => { - (countSuperadmins as jest.Mock).mockResolvedValue(0); - delete process.env.SUPERADMIN_EMAIL; - delete process.env.SUPERADMIN_PASSWORD; - - await bootstrapSuperadmin(); - expect(createUser).not.toHaveBeenCalled(); - }); - - it('creates the superadmin when none exists and env vars are set', async () => { - (countSuperadmins as jest.Mock).mockResolvedValue(0); - (createUser as jest.Mock).mockResolvedValue('new-id'); - process.env.SUPERADMIN_EMAIL = 'boss@clustox.com'; - process.env.SUPERADMIN_PASSWORD = 'strong-password'; - - await bootstrapSuperadmin('org-1'); - - expect(createUser).toHaveBeenCalledWith( - expect.objectContaining({ - email: 'boss@clustox.com', - password: 'strong-password', - role: 'SUPERADMIN', - teamIds: [], - orgId: 'org-1' - }) - ); - }); -}); -``` - -- [ ] **Step 2: Run it to confirm it fails** - -Run: `cd web-server && yarn jest src/auth/__tests__/bootstrap.test.ts` -Expected: FAIL — "Cannot find module '@/auth/bootstrap'". - -- [ ] **Step 3: Implement** - -Create `web-server/src/auth/bootstrap.ts`: - -```ts -import { Table } from '@/constants/db'; -import { db } from '@/utils/db'; - -import { countSuperadmins, createUser } from './queries'; - -const getDefaultOrgId = async (): Promise => { - const row = await db(Table.Organization).select('id').first(); - return row?.id ?? null; -}; - -export const bootstrapSuperadmin = async ( - orgIdOverride?: string -): Promise => { - const existing = await countSuperadmins(); - if (existing > 0) return; - - const email = process.env.SUPERADMIN_EMAIL; - const password = process.env.SUPERADMIN_PASSWORD; - - if (!email || !password) { - // Fail loudly but do not invent a default account. A known-value fallback - // is worse than having no superadmin at all. - console.error( - '[CLUSTOX] No superadmin exists and SUPERADMIN_EMAIL / SUPERADMIN_PASSWORD ' + - 'are not set. Nobody can administer this instance until they are.' - ); - return; - } - - const orgId = orgIdOverride ?? (await getDefaultOrgId()); - if (!orgId) { - console.error('[CLUSTOX] No organization found; cannot bootstrap superadmin.'); - return; - } - - await createUser({ - name: 'Superadmin', - email, - password, - role: 'SUPERADMIN', - teamIds: [], - orgId - }); - - console.info(`[CLUSTOX] Bootstrapped superadmin ${email}`); -}; -``` - -Create `web-server/instrumentation.ts`: - -```ts -export async function register() { - if (process.env.NEXT_RUNTIME === 'nodejs') { - const { bootstrapSuperadmin } = await import('@/auth/bootstrap'); - await bootstrapSuperadmin(); - } -} -``` - -The `NEXT_RUNTIME` check keeps this off the Edge runtime, where knex cannot run. - -- [ ] **Step 4: Run tests to verify they pass** - -Run: `cd web-server && yarn jest src/auth/__tests__/bootstrap.test.ts` -Expected: 3 passed. - -- [ ] **Step 5: Verify the real bootstrap runs** - -```bash -docker compose up -d --build -docker exec middleware-dev psql "postgresql://postgres:postgres@localhost:5434/mhq-oss" \ - -c 'SELECT u.primary_email, a.role FROM "ClustoxUserAuth" a JOIN "Users" u ON u.id = a.user_id;' -``` - -Expected: one row, role `SUPERADMIN`, matching `SUPERADMIN_EMAIL`. - -- [ ] **Step 6: Log in through the UI** - -Open http://localhost:3333, sign in with the superadmin credentials. Expected: the dashboard loads. - -- [ ] **Step 7: Commit** - -```bash -git add web-server/src/auth/bootstrap.ts web-server/instrumentation.ts web-server/src/auth/__tests__/bootstrap.test.ts -git commit -m "feat(auth): bootstrap first superadmin from environment" -``` - ---- - -## Task 9: Lock down the Flask servers - -Without this, `curl :9696` bypasses every guard added so far. - -**Files:** -- Create: `backend/analytics_server/mhq/clustox_auth/__init__.py` -- Create: `backend/analytics_server/mhq/clustox_auth/internal_token.py` -- Create: `backend/analytics_server/tests/clustox_auth/__init__.py` -- Create: `backend/analytics_server/tests/clustox_auth/test_internal_token.py` -- Create: `setup_utils/trigger_sync.sh` -- Modify: `backend/analytics_server/app.py`, `backend/analytics_server/sync_app.py`, `setup_utils/cronjob.txt`, `web-server/src/api-helpers/axios.ts` - -**Interfaces:** -- Consumes: `INTERNAL_API_TOKEN` env var -- Produces: `register_internal_token_guard(app: Flask) -> None` - -- [ ] **Step 1: Write the failing test** - -Create `backend/analytics_server/tests/clustox_auth/__init__.py` (empty) and `backend/analytics_server/tests/clustox_auth/test_internal_token.py`: - -```python -import os - -import pytest -from flask import Flask - -from mhq.clustox_auth.internal_token import register_internal_token_guard - - -def build_app(token): - if token is None: - os.environ.pop("INTERNAL_API_TOKEN", None) - else: - os.environ["INTERNAL_API_TOKEN"] = token - - app = Flask(__name__) - register_internal_token_guard(app) - - @app.route("/protected") - def protected(): - return {"ok": True} - - return app - - -def test_request_with_valid_token_is_allowed(): - app = build_app("secret-token") - client = app.test_client() - res = client.get("/protected", headers={"X-Internal-Token": "secret-token"}) - assert res.status_code == 200 - - -def test_request_without_token_is_rejected(): - app = build_app("secret-token") - res = app.test_client().get("/protected") - assert res.status_code == 403 - - -def test_request_with_wrong_token_is_rejected(): - app = build_app("secret-token") - client = app.test_client() - res = client.get("/protected", headers={"X-Internal-Token": "wrong"}) - assert res.status_code == 403 - - -def test_unset_env_fails_closed(): - """An unconfigured token must reject everything, never allow everything.""" - app = build_app(None) - client = app.test_client() - res = client.get("/protected", headers={"X-Internal-Token": "anything"}) - assert res.status_code == 403 - - -def test_health_endpoint_is_exempt(): - app = build_app("secret-token") - - @app.route("/") - def health(): - return "hello" - - res = app.test_client().get("/") - assert res.status_code == 200 -``` - -- [ ] **Step 2: Run it to confirm it fails** - -Run: `cd backend/analytics_server && python -m pytest tests/clustox_auth/test_internal_token.py -v` -Expected: FAIL — `ModuleNotFoundError: No module named 'mhq.clustox_auth'`. - -- [ ] **Step 3: Implement the guard** - -Create `backend/analytics_server/mhq/clustox_auth/__init__.py` (empty), then `backend/analytics_server/mhq/clustox_auth/internal_token.py`: - -```python -""" -CLUSTOX: restrict the Flask servers to traffic from the Next.js BFF. - -Upstream ships these APIs unauthenticated, relying on the loopback port binding -in docker-compose. Once authorization lives in the BFF, a direct call to :9696 -would bypass every check, so every request must carry a shared secret. - -Fails closed: if INTERNAL_API_TOKEN is unset, all requests are rejected. -""" - -import hmac -from os import getenv - -from flask import Flask, jsonify, request - -HEADER = "X-Internal-Token" - -# Health checks must stay reachable for container and UI status probes. -EXEMPT_PATHS = {"/"} - - -def register_internal_token_guard(app: Flask) -> None: - @app.before_request - def _verify_internal_token(): - if request.path in EXEMPT_PATHS: - return None - - expected = getenv("INTERNAL_API_TOKEN") - if not expected: - return jsonify({"error": "internal token not configured"}), 403 - - provided = request.headers.get(HEADER, "") - if not hmac.compare_digest(provided, expected): - return jsonify({"error": "forbidden"}), 403 - - return None -``` - -`hmac.compare_digest` avoids leaking the token through response-timing differences. - -- [ ] **Step 4: Run tests to verify they pass** - -Run: `cd backend/analytics_server && python -m pytest tests/clustox_auth/test_internal_token.py -v` -Expected: 5 passed. - -- [ ] **Step 5: Register the guard on both servers** - -In `backend/analytics_server/app.py`, after the blueprint registrations and before `configure_db_with_app(app)`: - -```python -# CLUSTOX: restrict this API to the BFF. See mhq/clustox_auth/internal_token.py -from mhq.clustox_auth.internal_token import register_internal_token_guard - -register_internal_token_guard(app) -``` - -Add the identical two lines to `backend/analytics_server/sync_app.py`. - -- [ ] **Step 6: Send the token from the BFF** - -In `web-server/src/api-helpers/axios.ts`, immediately after the `internalSyncServer` definition: - -```ts -// CLUSTOX: the Flask servers reject any request without this shared secret. -const internalTokenInterceptor = (req: any) => { - req.headers = req.headers || {}; - req.headers['X-Internal-Token'] = process.env.INTERNAL_API_TOKEN; - return req; -}; - -internal.interceptors.request.use(internalTokenInterceptor); -internalSyncServer.interceptors.request.use(internalTokenInterceptor); -``` - -- [ ] **Step 7: Give cron the token** - -Create `setup_utils/trigger_sync.sh`: - -```bash -#!/bin/bash -# CLUSTOX: cron cannot read the container env, so source it before calling the -# sync server, which now requires the internal token. -set -a -[ -f /app/.env ] && . /app/.env -set +a - -curl -s -X POST \ - -H "X-Internal-Token: ${INTERNAL_API_TOKEN}" \ - "http://localhost:${SYNC_SERVER_PORT:-9697}/sync" -``` - -Replace the job line in `setup_utils/cronjob.txt`: - -``` -# CLUSTOX: routed through trigger_sync.sh so the internal token is sent. -*/30 * * * * /app/setup_utils/trigger_sync.sh >> /var/log/cron/cron.log 2>&1 -``` - -Add to `Dockerfile.dev`, alongside the existing `chmod +x` line: - -```dockerfile -RUN chmod +x /app/setup_utils/trigger_sync.sh -``` - -- [ ] **Step 8: Verify direct access is now blocked** - -```bash -docker compose up -d --build -# Direct call — must be refused -curl -s -o /dev/null -w "direct: %{http_code}\n" http://localhost:9696/teams/any/lead_time -# Health check — must still work -curl -s -o /dev/null -w "health: %{http_code}\n" http://localhost:9696/ -``` - -Expected: `direct: 403`, `health: 200`. - -- [ ] **Step 9: Verify the app and cron still work** - -Sign in at http://localhost:3333 and confirm dashboards still render (proving the BFF's header works), then: - -```bash -docker exec middleware-dev /app/setup_utils/trigger_sync.sh -``` - -Expected: `{"message":"sync started",...}`. **Do not skip this** — a sync broken by the token change fails silently ([ARCHITECTURE.md §7.4](../../ARCHITECTURE.md)). - -- [ ] **Step 10: Commit** - -```bash -git add backend/analytics_server/mhq/clustox_auth backend/analytics_server/tests/clustox_auth \ - backend/analytics_server/app.py backend/analytics_server/sync_app.py \ - setup_utils/trigger_sync.sh setup_utils/cronjob.txt Dockerfile.dev \ - web-server/src/api-helpers/axios.ts -git commit -m "feat(auth): restrict Flask servers to BFF via shared secret" -``` - ---- - -## Task 10: Apply per-team authorization to routes - -**Files:** -- Modify: `web-server/pages/api/resources/team_repos.ts` -- Modify: `web-server/pages/api/internal/team/[team_id]/*.ts` route files that export a default -- Modify: `web-server/pages/api/resources/orgs/[org_id]/teams/index.ts` and `v2.ts` -- Modify: `web-server/pages/api/resources/share.ts` -- Modify: `web-server/pages/api/integrations/index.ts` -- Create: `web-server/src/auth/__tests__/route-authz.test.ts` - -**Interfaces:** -- Consumes: `assertTeamAccess`, `assertRole`, `getAccessibleTeamIds` (Task 4); `req.session` (Task 6) -- Produces: team-scoped routes 403 for unauthorised teams; team lists filtered to accessible teams - -- [ ] **Step 1: Enumerate the routes needing scoping** - -```bash -cd web-server -grep -rln "team_id" pages/api --include=*.ts | xargs grep -l "export default" | sort -``` - -Record the list. Every route accepting a `team_id` needs `assertTeamAccess`. - -- [ ] **Step 2: Write the failing test** - -Create `web-server/src/auth/__tests__/route-authz.test.ts`: - -```ts -jest.mock('@/auth/queries', () => ({ - getTeamIdsForUser: jest.fn(), - getAllTeamIds: jest.fn() -})); - -import { assertTeamAccess, getAccessibleTeamIds } from '@/auth/guard'; -import { getAllTeamIds, getTeamIdsForUser } from '@/auth/queries'; -import { AuthSession } from '@/auth/types'; - -const admin: AuthSession = { - userId: 'ad1', email: 'l@clustox.com', name: 'L', role: 'ADMIN' -}; -const superadmin: AuthSession = { - userId: 'su1', email: 'b@clustox.com', name: 'B', role: 'SUPERADMIN' -}; - -describe('team scoping behaviour relied on by routes', () => { - beforeEach(() => jest.clearAllMocks()); - - it('an admin is refused a team outside their assignment', async () => { - (getTeamIdsForUser as jest.Mock).mockResolvedValue(['team-zda']); - await expect(assertTeamAccess(admin, 'team-cgpt')).rejects.toMatchObject({ - status: 403 - }); - }); - - it('a team list for an admin contains only their teams', async () => { - (getTeamIdsForUser as jest.Mock).mockResolvedValue(['team-zda']); - await expect(getAccessibleTeamIds(admin)).resolves.toEqual(['team-zda']); - }); - - it('a team list for a superadmin contains every team', async () => { - (getAllTeamIds as jest.Mock).mockResolvedValue(['team-zda', 'team-cgpt']); - await expect(getAccessibleTeamIds(superadmin)).resolves.toEqual([ - 'team-zda', - 'team-cgpt' - ]); - }); -}); -``` - -- [ ] **Step 3: Run it** - -Run: `cd web-server && yarn jest src/auth/__tests__/route-authz.test.ts` -Expected: 3 passed (guard already exists from Task 4 — this pins the behaviour routes depend on). - -- [ ] **Step 4: Add the guard to each team-scoped route** - -For every handler taking `team_id`, insert immediately after the handler opens. Example for `team_repos.ts`: - -```ts -endpoint.handle.GET(getSchema, async (req, res) => { - // CLUSTOX: an admin may only read teams assigned to them. - await assertTeamAccess((req as any).session, req.payload.team_id); - - if (req.meta?.features?.use_mock_data) { - return res.send(teamReposMock); - } - res.send(await getTeamRepos(req.payload.team_id)); -}); -``` - -with the import: - -```ts -// CLUSTOX: per-team authorization -import { assertTeamAccess } from '@/auth/guard'; -``` - -Apply to **both** GET and PATCH handlers, and to every route from Step 1. - -- [ ] **Step 5: Filter the team list endpoints** - -In `pages/api/resources/orgs/[org_id]/teams/index.ts` and `v2.ts`, filter the result before sending: - -```ts -// CLUSTOX: an admin sees only their teams in the picker. -import { getAccessibleTeamIds } from '@/auth/guard'; - -const accessible = await getAccessibleTeamIds((req as any).session); -const visibleTeams = teams.filter((t: { id: string }) => accessible.includes(t.id)); -res.send(visibleTeams); -``` - -Adjust the variable names to match each file. - -- [ ] **Step 6: Restrict integrations to superadmin** - -In `pages/api/integrations/index.ts`, at the top of each mutating handler: - -```ts -// CLUSTOX: only superadmins manage integrations. -import { assertRole } from '@/auth/guard'; - -assertRole((req as any).session, 'SUPERADMIN'); -``` - -- [ ] **Step 7: Make share.ts authenticated** - -In `pages/api/resources/share.ts`, change: - -```ts -// CLUSTOX: was unauthenticated, which allowed anonymous writes to -// URLShortenerData. Everyone who should open a share link has an account. -const endpoint = new Endpoint(nullSchema); -``` - -- [ ] **Step 8: Verify manually** - -Sign in as superadmin, note both team IDs, then create an admin scoped to one (Task 11 provides the UI; until then insert the `ClustoxUserTeamAccess` row by hand). Sign in as that admin and: - -```bash -# In the browser devtools console while signed in as the scoped admin: -fetch('/api/resources/team_repos?team_id=').then(r => console.log(r.status)) -``` - -Expected: `403`. - -- [ ] **Step 9: Commit** - -```bash -git add web-server/pages/api web-server/src/auth/__tests__/route-authz.test.ts -git commit -m "feat(auth): scope team routes and restrict integrations to superadmin" -``` - ---- - -## Task 11: User management API and page - -**Files:** -- Create: `web-server/pages/api/clustox/users/index.ts` -- Create: `web-server/pages/api/clustox/users/[user_id].ts` -- Create: `web-server/pages/users.tsx` - -**Interfaces:** -- Consumes: `listUsers`, `createUser`, `updateUserRole`, `setUserTeams` (Task 3); `assertRole` (Task 4) -- Produces: `GET/POST /api/clustox/users`, `PATCH /api/clustox/users/[user_id]` - -- [ ] **Step 1: Build the list/create route** - -Create `web-server/pages/api/clustox/users/index.ts`: - -```ts -import * as yup from 'yup'; - -import { assertRole } from '@/auth/guard'; -import { createUser, listUsers } from '@/auth/queries'; -import { Endpoint, nullSchema } from '@/api-helpers/global'; -import { Table } from '@/constants/db'; -import { db } from '@/utils/db'; - -const postSchema = yup.object().shape({ - name: yup.string().required(), - email: yup.string().email().required(), - password: yup.string().min(12).required(), - role: yup.string().oneOf(['SUPERADMIN', 'ADMIN']).required(), - team_ids: yup.array().of(yup.string().uuid()).required() -}); - -const endpoint = new Endpoint(nullSchema); - -endpoint.handle.GET(nullSchema, async (req, res) => { - assertRole((req as any).session, 'SUPERADMIN'); - res.send(await listUsers()); -}); - -endpoint.handle.POST(postSchema, async (req, res) => { - assertRole((req as any).session, 'SUPERADMIN'); - - const org = await db(Table.Organization).select('id').first(); - const userId = await createUser({ - name: req.payload.name, - email: req.payload.email, - password: req.payload.password, - role: req.payload.role, - teamIds: req.payload.team_ids, - orgId: org.id - }); - - res.send({ user_id: userId }); -}); - -export default endpoint.serve(); -``` - -A 12-character minimum is enforced at the schema, so the UI cannot create a weak account. - -- [ ] **Step 2: Build the update route** - -Create `web-server/pages/api/clustox/users/[user_id].ts`: - -```ts -import * as yup from 'yup'; - -import { assertRole } from '@/auth/guard'; -import { setUserTeams, updateUserRole } from '@/auth/queries'; -import { Endpoint } from '@/api-helpers/global'; - -const pathSchema = yup.object().shape({ - user_id: yup.string().uuid().required() -}); - -const patchSchema = yup.object().shape({ - user_id: yup.string().uuid().required(), - role: yup.string().oneOf(['SUPERADMIN', 'ADMIN']).optional(), - team_ids: yup.array().of(yup.string().uuid()).optional() -}); - -const endpoint = new Endpoint(pathSchema); - -endpoint.handle.PATCH(patchSchema, async (req, res) => { - assertRole((req as any).session, 'SUPERADMIN'); - - if (req.payload.role) { - await updateUserRole(req.payload.user_id, req.payload.role); - } - if (req.payload.team_ids) { - await setUserTeams(req.payload.user_id, req.payload.team_ids); - } - - res.send({ ok: true }); -}); - -export default endpoint.serve(); -``` - -- [ ] **Step 3: Build the users page** - -Create `web-server/pages/users.tsx` with a table of users (name, email, role, team count) and a form to create one — name, email, password, role select, team multi-select. Fetch `GET /api/clustox/users` on mount; `POST` on submit; refresh the list afterwards. - -Guard the page client-side: - -```tsx -const { role } = useAuth(); -if (role !== 'SUPERADMIN') return Not authorised.; -``` - -This is presentation only — the API enforces the real rule. - -- [ ] **Step 4: Verify end to end** - -1. Sign in as superadmin → `/users` loads -2. Create an admin assigned only to `ZDA` -3. Sign out, sign in as that admin -4. `/users` shows "Not authorised"; `GET /api/clustox/users` returns 403 -5. Team picker shows only `ZDA` - -- [ ] **Step 5: Commit** - -```bash -git add web-server/pages/api/clustox web-server/pages/users.tsx -git commit -m "feat(auth): add user management API and page" -``` - ---- - -## Task 12: End-to-end verification - -**Files:** -- Create: `web-server/e2e/auth.spec.ts` - -**Interfaces:** -- Consumes: everything above -- Produces: an automated run of the demo script - -- [ ] **Step 1: Write the Playwright spec** - -Create `web-server/e2e/auth.spec.ts`. **URLs are absolute on purpose:** upstream's -`playwright.config.ts` sets no `baseURL`, and adding one would mean touching an eighth upstream file -for no functional gain. - -```ts -import { expect, test } from '@playwright/test'; - -const APP = 'http://localhost:3333'; -const API = 'http://localhost:9696'; - -const SUPERADMIN = { - email: process.env.SUPERADMIN_EMAIL!, - password: process.env.SUPERADMIN_PASSWORD! -}; - -const signIn = async (page: any, email: string, password: string) => { - await page.goto(`${APP}/login`); - await page.fill('input[type="email"]', email); - await page.fill('input[type="password"]', password); - await page.click('button[type="submit"]'); - await page.waitForURL((u: URL) => !u.pathname.startsWith('/login')); -}; - -test('unauthenticated visitors are redirected to login', async ({ page }) => { - await page.goto(`${APP}/dora-metrics`); - await expect(page).toHaveURL(/\/login/); -}); - -test('bad credentials are rejected', async ({ page }) => { - await page.goto(`${APP}/login`); - await page.fill('input[type="email"]', 'nobody@clustox.com'); - await page.fill('input[type="password"]', 'wrong-password'); - await page.click('button[type="submit"]'); - await expect(page.getByRole('alert')).toContainText('Invalid email or password'); -}); - -test('superadmin signs in and sees user management', async ({ page }) => { - await signIn(page, SUPERADMIN.email, SUPERADMIN.password); - await page.goto(`${APP}/users`); - await expect(page.getByText('Not authorised')).toHaveCount(0); -}); - -test('the Flask API refuses direct requests', async ({ request }) => { - const res = await request.get(`${API}/teams/any-id/lead_time`, { - failOnStatusCode: false - }); - expect(res.status()).toBe(403); -}); -``` - -No jest config change is needed: `jest.config.js` already lists `auth.spec.ts` in -`testPathIgnorePatterns`, so jest will not try to run this Playwright spec. (That rule is another -piece of vestigial upstream auth scaffolding — the file it referenced does not exist.) - -The app must be running before this suite: `docker compose up -d`. - -- [ ] **Step 2: Run it** - -Run: `cd web-server && yarn playwright test e2e/auth.spec.ts` -Expected: 4 passed. - -- [ ] **Step 3: Run every suite** - -```bash -cd web-server && yarn jest && yarn tsc --noEmit -cd ../backend/analytics_server && python -m pytest tests/ -q -cd ../.. && black --check backend/ && flake8 backend/ -``` - -Expected: all green. - -- [ ] **Step 4: Audit the fork footprint** - -```bash -grep -rn "CLUSTOX:" --include='*.py' --include='*.ts' --include='*.tsx' --include='*.sql' \ - --include='*.sh' --include='*.txt' . | grep -v node_modules -``` - -Expected: sentinels in all seven modified upstream files. Record the count in [FORK_STRATEGY.md](../../FORK_STRATEGY.md) terms — this is the number the next upstream sync checks against. - -- [ ] **Step 5: Walk the demo script** - -Run every step of [spec §7](../specs/2026-08-05-auth-rbac-design.md) by hand. Steps 5 and 6 are the ones to be certain about. - -- [ ] **Step 6: Commit** - -```bash -git add web-server/e2e/auth.spec.ts -git commit -m "test(auth): add end-to-end auth and access-control coverage" -``` - ---- - -## Self-review notes - -**Spec coverage:** every §2 in-scope item maps to a task — login (5, 7), bootstrap (8), two roles (1, 4), per-team scoping (4, 10), user management (11), Flask lockdown (9), Google-ready structure (5, via the providers array). Every §8 risk has a step: `share.ts` (10.7), password handling (2), bypass route (10.1 enumeration), fail-closed (9.1 test), cron (9.9), weak bootstrap (8.3), divergence (12.4). - -**Deferred deliberately:** `pages/users.tsx` (11.3) is described rather than written out in full, because it is standard MUI table-and-form work with no non-obvious logic, and its authorization is enforced server-side and already tested. Every step containing security-relevant logic has literal code. - -**Type consistency:** `AuthSession`, `ClustoxRole`, `assertTeamAccess`, `getAccessibleTeamIds`, `getTeamIdsForUser` are used with identical signatures across Tasks 3, 4, 6, 10, 11. diff --git a/docs/superpowers/specs/2026-08-05-auth-rbac-design.md b/docs/superpowers/specs/2026-08-05-auth-rbac-design.md deleted file mode 100644 index 857108eb..00000000 --- a/docs/superpowers/specs/2026-08-05-auth-rbac-design.md +++ /dev/null @@ -1,368 +0,0 @@ -# Design: Authentication & Role-Based Access Control - -| | | -|---|---| -| **Date** | 2026-08-05 | -| **Status** | Approved — ready for implementation planning | -| **Repo** | [Clustox/middleware](https://github.com/Clustox/middleware) (fork of middlewarehq/middleware) | -| **Baseline** | `844eb42` | -| **Related** | [ARCHITECTURE.md](../../ARCHITECTURE.md) · [FORK_STRATEGY.md](../../FORK_STRATEGY.md) | - ---- - -## 1. Context - -The application currently has **no authentication of any kind**. Anyone who can reach the app -gets full access to every dashboard, every setting, and every integration. - -This is not an oversight in our deployment — it is how upstream ships. Verified: - -```bash -# Backend API — zero matches -grep -rniE "login_required|authenticat|jwt|api_key|bearer" backend/analytics_server/mhq/api/ -# Frontend — no NextAuth route, no sign-in page -find web-server/pages -path '*nextauth*' -ls web-server/pages | grep -iE 'login|signin' -``` - -Upstream evidently *had* auth and stripped it for the open-source build. What remains is -scaffolding with the enforcement removed: - -| Artefact | State | -|---|---| -| `web-server/src/components/Authenticated/index.tsx` | Wraps every page. Sets `verified = true` unconditionally and shows a toast reading "You are successfully authenticated!". Checks nothing. | -| `Endpoint.authenticated` (`src/api-helpers/global.ts`) | Field declared, constructor accepts `{ unauthenticated: true }`, **`serve()` never reads it**. | -| `UserRole` enum (`src/types/resources.ts`) | `EM`, `ENGINEER`, `MOM`. Never enforced. | -| `AuthContext.role` | Hardcoded to `UserRole.MOM` (the most privileged). | -| `AuthContext.userId` | Hardcoded to `00000000-0000-0000-0000-000000000000`. | -| `Users.role_override` column | Exists in schema, referenced nowhere in code. | -| `next-auth` in `package.json` | No `[...nextauth]` route, no sign-in page. Only imports are a `Session` *type* and a `signIn` call in an error component that has nothing to call. | - -**This is good news.** The hook points we need were designed in; only the wiring was removed. -Five routes already correctly declare `{ unauthenticated: true }`, which tells us the opt-out -mechanism was intended to be enforced. - -### Goal - -Only people with credentials reach the dashboards. Superadmins manage users and integrations and -see everything; admins see only the teams assigned to them. - -### Considered and rejected: an identity-aware proxy - -Putting `oauth2-proxy` / Cloudflare Access in front of the app would solve the perimeter problem -with **zero fork changes** and provide Google sign-in for free. It was recommended and rejected in -favour of in-app auth, deliberately, for two reasons: - -1. A proxy is all-or-nothing. It cannot express "this admin sees only Team ZDA", which is a stated - requirement. -2. The team wants a demonstrable product capability, not a deployment configuration change. - -This is recorded so the trade-off is not relitigated later, and so the per-team scoping -requirement — the thing that actually justifies building in-app — is not quietly dropped from -scope. **If per-team scoping is ever cut, revisit the proxy option instead.** - ---- - -## 2. Scope - -### In scope (v1) - -- Email + password login, bcrypt-hashed -- Superadmin bootstrapped from environment variables on first boot -- Two roles: `SUPERADMIN`, `ADMIN` -- Per-team access scoping for `ADMIN` -- User management page (superadmin only): create user, set role, assign teams -- Lock the Flask servers so they are reachable only from the BFF -- Structured so a Google provider drops in without rework - -### Explicitly out of scope (v1) - -| Deferred | Why | -|---|---| -| Google sign-in | Needs a Google Cloud OAuth app; would block the demo. Design keeps the seam open. | -| Password reset / email flows | Needs SMTP. v1: superadmin re-creates the user. | -| Audit log of who viewed what | Valuable, but separate work. | -| Bitbucket integration | Unrelated to auth. There is a `BITBUCKET` constant but **no ETL handler exists** — it is net-new integration work. | -| Multi-org / true tenancy | The app is hardwired to a single `default` org. Out of scope. | - ---- - -## 3. Decisions and rationale - -| # | Decision | Rationale | -|---|---|---| -| D1 | New tables, not new columns on `Users` | Upstream's schema stays untouched, so their migrations can never collide with ours. Joins are trivial. | -| D2 | Do not reuse `role_override` or `UserRole` | `role_override` is dead and undocumented; `UserRole` (`EM`/`ENGINEER`/`MOM`) is a reporting-hierarchy concept, not a permission one. Overloading either would confuse both. | -| D3 | Enforce BFF auth inside `Endpoint.serve()` | Every real API route goes through it (verified below). One insertion point instead of 47. | -| D4 | Lock Flask to BFF-only via shared secret | Lets **all** authorization live in the BFF. No user context threaded through Python, no changes to any service/repo file. | -| D5 | JWT sessions; team access read per-request from DB | Role in the token is fine; team assignments must not be, or changing someone's teams wouldn't take effect until they re-login. | -| D6 | `share.ts` becomes authenticated | See §8. | - -### Verification behind D3 - -All 53 files under `pages/api/` were classified: - -- **47** construct `new Endpoint(...)` — every real route -- **2** are test files (`__tests__/`) -- **4** are helper modules with **no `export default`** — not routes at all: - `internal/[org_id]/utils.ts`, `internal/team/[team_id]/{deployment_freq,deployment_prs,revert_prs}.ts` - -```bash -# reproduce -for f in $(find web-server/pages/api -name '*.ts'); do - grep -q "export default" "$f" || echo "not a route: $f" -done -``` - -So `Endpoint.serve()` gives **100% coverage of BFF routes with no stragglers.** - ---- - -## 4. Data model - -One new migration, `database-docker/db/migrations/_clustox_auth.sql`: - -```sql -CREATE TABLE "ClustoxUserAuth" ( - user_id uuid PRIMARY KEY REFERENCES "Users"(id) ON DELETE CASCADE, - password_hash text NOT NULL, - role varchar NOT NULL CHECK (role IN ('SUPERADMIN','ADMIN')), - created_at timestamptz NOT NULL DEFAULT now(), - updated_at timestamptz NOT NULL DEFAULT now() -); - -CREATE TABLE "ClustoxUserTeamAccess" ( - user_id uuid NOT NULL REFERENCES "Users"(id) ON DELETE CASCADE, - team_id uuid NOT NULL REFERENCES "Team"(id) ON DELETE CASCADE, - created_at timestamptz NOT NULL DEFAULT now(), - PRIMARY KEY (user_id, team_id) -); - -CREATE INDEX idx_clustox_user_team_access_user ON "ClustoxUserTeamAccess"(user_id); -``` - -Identity itself continues to live in upstream's `Users` table (`id`, `name`, `primary_email`, -`is_deleted`). `ClustoxUserAuth` adds only credentials and role. - -`SUPERADMIN` needs no rows in `ClustoxUserTeamAccess` — the role short-circuits every scope check. - -Per [ARCHITECTURE.md §12.5](../../ARCHITECTURE.md), a schema change is normally a four-place -change. Here it is **three**: migration, `schema.sql`, and `web-server/src/constants/db.ts`. No -SQLAlchemy model is needed, because Python never reads these tables. - ---- - -## 5. Architecture - -### 5.1 Three surfaces - -Auth that covers only the frontend is decorative — the Flask API is independently reachable, and 23 -BFF files bypass Flask to reach Postgres directly via knex. All three paths must be closed. - -```mermaid -graph TB - U["Browser"] - subgraph app["Application"] - MW["middleware.ts
surface 1 — page guard"] - EP["Endpoint.serve()
surface 2 — API guard + authz"] - AX["axios 'internal' instance
stamps X-Internal-Token"] - FL["Flask :9696 / :9697
surface 3 — before_request"] - PG[("Postgres")] - end - U --> MW --> EP - EP -->|knex| PG - EP --> AX --> FL --> PG - - style MW fill:#4a6fa5,color:#fff - style EP fill:#5c8001,color:#fff - style FL fill:#9b2226,color:#fff -``` - -| Surface | Mechanism | Failure mode if omitted | -|---|---|---| -| 1 — Pages | `middleware.ts` redirects unauthenticated requests to `/login` | Cosmetic only; data still protected by 2 and 3 | -| 2 — BFF routes | `Endpoint.serve()` rejects when `this.authenticated` and no valid session | **Total bypass** — every metric and mutation exposed | -| 3 — Flask | One `before_request` rejects requests without the shared secret | **Total bypass** — `curl :9696` returns everything | - -### 5.2 Session - -`next-auth` v4 (already a dependency) with a **Credentials** provider and **JWT** strategy. - -- Token carries `userId` and `role`. -- Team assignments are **not** in the token; they are read per-request from - `ClustoxUserTeamAccess`. Cheap (indexed, single-row-set) and avoids stale permissions. -- `NEXTAUTH_SECRET` and `NEXTAUTH_URL` added to `.env` / `env.example`. - -Adding Google later means registering a second provider and mapping the Google email onto an -existing `Users` row. No change to the authorization layer. - -### 5.3 Authorization - -Because Flask is BFF-only after surface 3, **all authorization lives in the BFF**: - -```ts -// CLUSTOX: new file — src/auth/guard.ts -assertRole(session, 'SUPERADMIN') // throws 403 -assertTeamAccess(session, team_id) // SUPERADMIN passes; ADMIN checked against table -visibleTeamIds(session) // for list endpoints -``` - -Applied at: - -- **Team-scoped routes** — `assertTeamAccess` against the `team_id` in the path or payload -- **Team list endpoints** — filtered through `visibleTeamIds`, so an admin's team dropdown shows - only their teams -- **Integration + settings routes** — `assertRole(SUPERADMIN)` -- **User management routes** — `assertRole(SUPERADMIN)` - -An admin who guesses another team's UUID gets a 403 rather than data. That property is what makes -this more than a login screen, and it is the subject of an explicit test. - -### 5.4 Superadmin bootstrap - -On first boot, if no `SUPERADMIN` exists, create one from `SUPERADMIN_EMAIL` and -`SUPERADMIN_PASSWORD`. Idempotent, and never overwrites an existing account. If the env vars are -absent and no superadmin exists, log a loud warning — a running instance with no superadmin cannot -be administered. - ---- - -## 6. File inventory - -> **Corrected after implementation (2026-08-06).** The estimate below was wrong. -> The real footprint is **19 modified upstream files**, listed in -> [§6.1](#61-actual-footprint-after-implementation). The estimate missed every -> file that only reveals itself when the code runs — the `_app.tsx` auth gate, -> the knex/webpack conflict in `next.config.js`, the unhandled session -> rejection in `ThirdPartyAuthContext.tsx`, and the cron path. Left in place -> because the gap between estimate and reality is the useful part. - -### Upstream files modified (original estimate — six files) - -| File | Change | Size | -|---|---|---| -| `web-server/middleware.ts` | Redirect unauthenticated to `/login`; allow `/api/auth/*` | ~15 lines | -| `web-server/src/api-helpers/global.ts` | Enforce the existing `authenticated` flag in `serve()` | ~8 lines | -| `web-server/src/api-helpers/axios.ts` | Stamp `X-Internal-Token` on the `internal` instances | ~4 lines | -| `backend/analytics_server/app.py` | Register `before_request` hook | 2 lines | -| `backend/analytics_server/sync_app.py` | Register `before_request` hook | 2 lines | -| `web-server/src/constants/db.ts` | Declare the two new tables for knex | ~20 lines | - -Every one gets a `CLUSTOX:` sentinel per [FORK_STRATEGY.md §4](../../FORK_STRATEGY.md). - -Of the six, `constants/db.ts` is the lowest-risk: it is a declaration table that grows by append, so -conflicts there are mechanical. The two Python entry points are one line each. The real thought goes -into `global.ts` and `middleware.ts`. - -### New files (green zone) - -``` -database-docker/db/migrations/_clustox_auth.sql -backend/analytics_server/mhq/clustox_auth/internal_token.py -web-server/pages/login.tsx -web-server/pages/users.tsx -web-server/pages/api/auth/[...nextauth].ts -web-server/pages/api/clustox/users/index.ts # list, create -web-server/pages/api/clustox/users/[user_id].ts # update role, teams; delete -web-server/src/auth/guard.ts -web-server/src/auth/password.ts -web-server/src/auth/bootstrap.ts -web-server/src/auth/queries.ts -``` - -`Authenticated/index.tsx` is left alone — `middleware.ts` does the real work, and rewriting the -stub only adds divergence. - -### 6.1 Actual footprint after implementation - -Reproduce with: - -```bash -git diff --name-only 844eb42..HEAD | while read -r f; do - git cat-file -e 844eb42:"$f" 2>/dev/null && echo "MODIFIED $f" -done -``` - -**19 modified upstream files, 26 new files.** The modified set, by conflict risk: - -| Risk | Files | Why | -|---|---|---| -| **Low** (generated or append-only) | `database-docker/db/schema.sql`, `web-server/yarn.lock`, `web-server/package.json`, `env.example` | Regenerated or appended; conflicts are mechanical | -| **Medium** (config) | `Dockerfile.dev`, `web-server/next.config.js`, `setup_utils/cronjob.txt` | Small, isolated edits | -| **High** (logic) | `app.py`, `sync_app.py`, `middleware.ts`, `_app.tsx`, `global.ts`, `axios.ts`, `constants/db.ts`, `ThirdPartyAuthContext.tsx`, `integrations/index.ts`, `teams/index.ts`, `teams/v2.ts`, `share.ts` | Real edits to upstream behaviour; review carefully on every sync | - -**33 `CLUSTOX:` sentinel lines** across the tree. That count is the tripwire for the sync -procedure in [FORK_STRATEGY.md §5](../../FORK_STRATEGY.md) — if it drops after an upstream merge, -a conflict resolution ate one of our changes. - -### Why the estimate was wrong - -Every missed file shared a trait: it was invisible until the code ran. - -- **`_app.tsx`** gates every page on `auth.isInitialized`, which needs a session fetch that now - 401s — making the login page unreachable. A chicken-and-egg only visible in a browser. -- **`next.config.js`** — knex resolves dialects dynamically, so webpack could not bundle it for the - instrumentation hook. The bootstrap silently never ran. -- **`ThirdPartyAuthContext.tsx`** — `fetchSession` had no `.catch()`, so the expected 401 rejected - unhandled and buried the login form under an error overlay. -- **`cronjob.txt` / `trigger_sync.sh`** — cron does not inherit the container environment. - -The lesson for future estimates: **auth touches everything that assumed it could always fetch -data.** Grep for callers, not just for the routes being protected. - -### Fork impact - -Twelve genuinely-modified logic files for cross-cutting auth is still a good ratio, largely because -upstream left the hook points in (`Endpoint.authenticated`, `Errors.SESSION_USER_NOT_FOUND`, the -`unauthenticated` opt-out). This remains **permanent divergence**: upstream will not accept it, -since gated auth is plausibly what their commercial product sells. - ---- - -## 7. Demo script - -1. Open the app → redirected to `/login` *(today it walks straight in)* -2. Log in as superadmin → all teams visible, **Users** appears in nav -3. Create an admin, assign to `ZDA` only -4. Log out, log in as that admin → only `ZDA`; no Users page; integrations hidden -5. As that admin, request `CGPT Frontend`'s team ID directly → **403** -6. `curl http://localhost:9696/teams//lead_time` → **403** *(today: returns data)* - -Steps 5 and 6 are the ones worth dwelling on. Step 6 in particular is what separates real access -control from a login screen, and it is the part most implementations get wrong. - ---- - -## 8. Risks and mitigations - -| Risk | Mitigation | -|---|---| -| **`share.ts` is currently `unauthenticated`** — a URL-shortener storing dashboard link state in `URLShortenerData`. Anonymous `POST` allows unbounded row insertion; anonymous `GET` returns stored link metadata. | **Decision:** make it authenticated. For an internal tool, everyone who should open a share link has an account, so nothing is lost and an anonymous write path is closed. | -| Password handling becomes our responsibility | bcrypt (cost 12), never log or return hashes, no reset flow in v1 — a stated limitation, not a hidden gap | -| Someone adds a route bypassing `Endpoint` | Test asserting every file under `pages/api` with a default export constructs an `Endpoint` | -| Shared secret leaks or is unset | Fail closed: if `INTERNAL_API_TOKEN` is unset, Flask rejects **all** requests rather than allowing all | -| Locking Flask breaks the cron sync | Cron calls `:9697/sync` from inside the container. It must send the token too — **verify explicitly**, since the failure is silent (see [ARCHITECTURE.md §7.4](../../ARCHITECTURE.md)) | -| Bootstrap creates a weak default account | No default password. Absent env vars → no superadmin + loud warning, never a known-value fallback | -| Permanent fork divergence | Accepted deliberately; sentinels make the footprint auditable | - ---- - -## 9. Testing - -| Layer | Coverage | -|---|---| -| Jest | `assertRole`, `assertTeamAccess`, `visibleTeamIds`; password hash/verify; `Endpoint.serve()` rejects without session and honours `unauthenticated` | -| pytest | Flask `before_request`: valid token passes, missing/wrong 403s, unset env fails closed | -| Playwright | The §7 demo flow end to end | -| Regression | Unauthenticated `curl` to a team route returns 403 — the single test that would catch the whole feature being silently undone | -| Manual | Cron sync still succeeds after Flask is locked down | - ---- - -## 10. Open questions - -None blocking. Two to settle during implementation, neither affecting the architecture: - -1. Session lifetime — proposed 12 hours, no sliding renewal. -2. Whether the admin's team dropdown should hide inaccessible teams entirely or show them disabled. - Proposed: hide, matching `visibleTeamIds`. From 12f16cd3e0fa1484f4d9aff70363bcdea6eaea6e Mon Sep 17 00:00:00 2001 From: Batool Fatima Date: Fri, 7 Aug 2026 20:14:07 +0500 Subject: [PATCH 036/194] Invite link through email --- .../20260807200000_clustox_invite_emailed.sql | 10 ++ env.example | 8 ++ web-server/package.json | 2 + web-server/pages/accept-invite.tsx | 8 +- web-server/pages/api/clustox/invites/index.ts | 46 ++++++- web-server/pages/login.tsx | 71 ++++++++-- web-server/pages/users.tsx | 56 ++++++-- web-server/src/auth/invites.ts | 42 ++++-- web-server/src/constants/db.ts | 3 +- .../Dashboards/ConfigureGithubModalBody.tsx | 5 + .../Dashboards/ConfigureGitlabModalBody.tsx | 5 + web-server/src/hooks/useAutofillSync.ts | 47 +++++++ web-server/src/theme/schemes/theme.ts | 20 +++ web-server/src/utils/mailer.ts | 122 ++++++++++++++++++ web-server/yarn.lock | 12 ++ 15 files changed, 417 insertions(+), 40 deletions(-) create mode 100644 database-docker/db/migrations/20260807200000_clustox_invite_emailed.sql create mode 100644 web-server/src/hooks/useAutofillSync.ts create mode 100644 web-server/src/utils/mailer.ts diff --git a/database-docker/db/migrations/20260807200000_clustox_invite_emailed.sql b/database-docker/db/migrations/20260807200000_clustox_invite_emailed.sql new file mode 100644 index 00000000..91e69ab3 --- /dev/null +++ b/database-docker/db/migrations/20260807200000_clustox_invite_emailed.sql @@ -0,0 +1,10 @@ +-- migrate:up +-- CLUSTOX: track whether an invite's email actually went out, separately +-- from the invite existing. SMTP is optional and best-effort (see +-- src/utils/mailer.ts) -- a superadmin can always fall back to copying the +-- link manually, but the "Pending invitations" list should be able to say +-- whether that fallback is actually necessary for a given invite. +ALTER TABLE "ClustoxInvite" ADD COLUMN emailed_at timestamptz; + +-- migrate:down +ALTER TABLE "ClustoxInvite" DROP COLUMN emailed_at; diff --git a/env.example b/env.example index 72018473..081d4ac9 100644 --- a/env.example +++ b/env.example @@ -29,3 +29,11 @@ NEXTAUTH_SECRET=generate-with-openssl-rand-base64-32 INTERNAL_API_TOKEN=generate-with-openssl-rand-hex-32 SUPERADMIN_EMAIL=admin@clustox.com SUPERADMIN_PASSWORD=set-a-strong-password + +# CLUSTOX: invite emails (optional -- SMTP_HOST unset means invites still +# create a link to copy/send manually, they just don't also get emailed) +SMTP_HOST= +SMTP_PORT=587 +SMTP_USER= +SMTP_PASSWORD= +SMTP_FROM= diff --git a/web-server/package.json b/web-server/package.json index 350fb136..54c0f1ee 100644 --- a/web-server/package.json +++ b/web-server/package.json @@ -51,6 +51,7 @@ "next": "^15.0.1", "next-auth": "^4.23.0", "next-images": "1.8.4", + "nodemailer": "^9.0.5", "notistack": "^3.0.1", "nprogress": "0.2.0", "pg": "^8.9.0", @@ -106,6 +107,7 @@ "@types/bcryptjs": "^3.0.0", "@types/js-cookie": "^3.0.2", "@types/node": "^16.17.0", + "@types/nodemailer": "^8.0.1", "@types/nprogress": "0.2.0", "@types/pluralize": "^0.0.29", "@types/ramda": "^0.28.7", diff --git a/web-server/pages/accept-invite.tsx b/web-server/pages/accept-invite.tsx index 69ba3bae..2d03d603 100644 --- a/web-server/pages/accept-invite.tsx +++ b/web-server/pages/accept-invite.tsx @@ -12,9 +12,9 @@ import { TextField, Typography } from '@mui/material'; -import { signIn } from 'next-auth/react'; import Head from 'next/head'; import { useRouter } from 'next/router'; +import { signIn } from 'next-auth/react'; import { FormEvent, ReactElement, useCallback, useEffect, useState } from 'react'; import { FlexBox } from '@/components/FlexBox'; @@ -180,6 +180,11 @@ export default function AcceptInvite() { ? `${MIN_PASSWORD - password.length} more characters needed` : `At least ${MIN_PASSWORD} characters` } + // A "create account" password field is exactly the kind + // browsers offer to generate/autofill a password into. + // Keeping the label permanently shrunk removes any race + // between that and React learning the value. + InputLabelProps={{ shrink: true }} /> setConfirm(e.target.value)} helperText={mismatch ? 'Passwords do not match' : ' '} + InputLabelProps={{ shrink: true }} /> {error && ( diff --git a/web-server/pages/api/clustox/invites/index.ts b/web-server/pages/api/clustox/invites/index.ts index 664c16c1..f0108f3f 100644 --- a/web-server/pages/api/clustox/invites/index.ts +++ b/web-server/pages/api/clustox/invites/index.ts @@ -2,10 +2,17 @@ import * as yup from 'yup'; import { Endpoint, nullSchema } from '@/api-helpers/global'; import { assertRole } from '@/auth/guard'; -import { createInvite, listPendingInvites } from '@/auth/invites'; +import { + createInvite, + listPendingInvites, + markInviteEmailed +} from '@/auth/invites'; import { getAuthUserByEmail } from '@/auth/queries'; import { ClustoxRole } from '@/auth/types'; +import { Table } from '@/constants/db'; import { Errors, ResponseError } from '@/constants/error'; +import { db } from '@/utils/db'; +import { sendInviteEmail } from '@/utils/mailer'; const postSchema = yup.object().shape({ name: yup.string().required(), @@ -27,19 +34,46 @@ endpoint.handle.POST(postSchema, async (req, res) => { const existing = await getAuthUserByEmail(req.payload.email); if (existing) throw new ResponseError(Errors.INSUFFICIENT_PARAMS, 409); - const { token, expiresAt } = await createInvite({ + const role = req.payload.role as ClustoxRole; + const orgId = req.payload.org_id ?? null; + + const { id, token, expiresAt } = await createInvite({ name: req.payload.name, email: req.payload.email, - role: req.payload.role as ClustoxRole, - orgId: req.payload.org_id ?? null, + role, + orgId, createdBy: session.userId }); + const inviteUrl = `${process.env.NEXTAUTH_URL}/accept-invite?token=${token}`; + + // Adopting an existing workspace is the only case where a name is known + // up front -- a brand-new one is created (and named) on acceptance, so + // there's nothing to show yet. + const org = + role === 'ADMIN' && orgId + ? await db(Table.Organization).where('id', orgId).first() + : null; + + // Best-effort and never fatal: the invite (and its link) already exists + // regardless of whether the email goes out, so a superadmin can always + // fall back to copying it manually. See src/utils/mailer.ts. + const { sent } = await sendInviteEmail({ + to: req.payload.email, + name: req.payload.name, + role, + orgName: org?.name ?? null, + inviteUrl, + expiresAt + }); + if (sent) await markInviteEmailed(id); + // The raw token is returned exactly once, here. Only its hash is stored, so // it cannot be recovered later -- a lost link has to be reissued. res.send({ - invite_url: `${process.env.NEXTAUTH_URL}/accept-invite?token=${token}`, - expires_at: expiresAt.toISOString() + invite_url: inviteUrl, + expires_at: expiresAt.toISOString(), + emailed: sent }); }); diff --git a/web-server/pages/login.tsx b/web-server/pages/login.tsx index 660b2570..6a015dd2 100644 --- a/web-server/pages/login.tsx +++ b/web-server/pages/login.tsx @@ -1,26 +1,39 @@ import { Box, Button, Card, TextField, Typography } from '@mui/material'; -import { signIn } from 'next-auth/react'; import Head from 'next/head'; import { useRouter } from 'next/router'; +import { signIn } from 'next-auth/react'; import { FormEvent, ReactElement, useState } from 'react'; +import { useAutofillSync } from '@/hooks/useAutofillSync'; + export default function Login() { const router = useRouter(); const [email, setEmail] = useState(''); const [password, setPassword] = useState(''); const [error, setError] = useState(''); const [busy, setBusy] = useState(false); + const emailRef = useAutofillSync(email, setEmail); + const passwordRef = useAutofillSync(password, setPassword); const onSubmit = async (e: FormEvent) => { e.preventDefault(); setBusy(true); setError(''); - const res = await signIn('credentials', { - email, - password, - redirect: false - }); + let res: Awaited>; + try { + res = await signIn('credentials', { email, password, redirect: false }); + } catch { + setBusy(false); + // CLUSTOX FIX: signIn() throwing (a network/server failure) used to be + // indistinguishable from a bad password -- both fell through to the + // same generic message. Kept the wording deliberately vague about + // *why*, since we still don't want to reveal which case happened, but + // this path no longer requires the request to have actually reached + // the server and returned an { error }. + setError('Something went wrong. Please try again.'); + return; + } setBusy(false); @@ -43,10 +56,29 @@ export default function Login() { justifyContent="center" minHeight="100vh" > - - - Sign in + + + `0 0 12px ${t.palette.primary.main}` + }} + /> + + Middleware + + + + + Sign in to your workspace + + + Use the account your Super Admin set up for you. +
setEmail(e.target.value)} required + inputRef={emailRef} + // Deterministic fix for the autofill-overlap: keep the label + // permanently shrunk instead of trying to detect the exact + // moment the browser autofills, which is inherently racy + // against React's own render timing. + InputLabelProps={{ shrink: true }} /> setPassword(e.target.value)} required + inputRef={passwordRef} + InputLabelProps={{ shrink: true }} /> {error && ( - + {error} )} @@ -81,6 +121,17 @@ export default function Login() { {busy ? 'Signing in...' : 'Sign in'} + + + SELF-HOSTED · YOUR DATA STAYS ON YOUR INFRASTRUCTURE +
diff --git a/web-server/pages/users.tsx b/web-server/pages/users.tsx index ff3fcfd0..624217ae 100644 --- a/web-server/pages/users.tsx +++ b/web-server/pages/users.tsx @@ -42,6 +42,7 @@ import ExtendedSidebarLayout from 'src/layouts/ExtendedSidebarLayout'; import { FlexBox } from '@/components/FlexBox'; import { Line } from '@/components/Text'; import { PageWrapper } from '@/content/PullRequests/PageWrapper'; +import { useAutofillSync } from '@/hooks/useAutofillSync'; import { PageLayout } from '@/types/resources'; type ClustoxRole = 'SUPERADMIN' | 'ADMIN'; @@ -117,13 +118,26 @@ function UsersPage() { const [inviteOpen, setInviteOpen] = useState(false); const [inviteLink, setInviteLink] = useState(''); + const [inviteEmailed, setInviteEmailed] = useState(false); const [invites, setInvites] = useState< - { id: string; email: string; name: string; role: ClustoxRole; orgName: string | null; expired: boolean }[] + { id: string; email: string; name: string; role: ClustoxRole; orgName: string | null; expired: boolean; emailed: boolean }[] >([]); const [menuAnchor, setMenuAnchor] = useState(null); const [menuUser, setMenuUser] = useState(null); + // Shared between the Invite and Add user dialogs -- only one is ever + // open at a time, and they both edit the same `form` fields. + const nameRef = useAutofillSync(form.name, (name) => + setForm((f) => ({ ...f, name })) + ); + const emailRef = useAutofillSync(form.email, (email) => + setForm((f) => ({ ...f, email })) + ); + const passwordRef = useAutofillSync(form.password, (password) => + setForm((f) => ({ ...f, password })) + ); + const loadUsers = useCallback(async () => { const res = await fetch('/api/clustox/users'); if (res.status === 403) { @@ -233,10 +247,11 @@ function UsersPage() { return; } - const { invite_url } = await res.json(); + const { invite_url, emailed } = await res.json(); // Shown once and only once: only the hash is stored, so a lost link // cannot be recovered and has to be reissued. setInviteLink(invite_url); + setInviteEmailed(emailed); await loadInvites(); }; @@ -317,6 +332,7 @@ function UsersPage() { setForm(emptyForm); setFormError(''); setInviteLink(''); + setInviteEmailed(false); setInviteOpen(true); }} > @@ -349,7 +365,9 @@ function UsersPage() { variant="outlined" color={i.expired ? 'error' : 'default'} onDelete={() => revokeInvite(i.id)} - label={`${i.email} · ${i.role === 'ADMIN' ? 'admin' : 'superadmin'}${ + label={`${i.email} · ${ + i.role === 'ADMIN' ? 'admin' : 'superadmin' + }${i.emailed ? '' : ' · not emailed'}${ i.expired ? ' · expired' : '' }`} /> @@ -504,11 +522,19 @@ function UsersPage() { - - Copy this link now. Only its fingerprint is stored, so it - cannot be shown again — if it is lost you will have to issue a - new one. - + {inviteEmailed ? ( + + Emailed. The link is also below in case it needs to be + forwarded — only its fingerprint is stored, so it cannot be + shown again if lost. + + ) : ( + + Couldn't email this one (SMTP isn't set up, or the send + failed) — copy the link below and send it yourself. Only its + fingerprint is stored, so it cannot be shown again if lost. + + )} e.target.select()} /> - Single use, expires in 7 days. Send it over Slack — the - recipient chooses their own password. + Single use, expires in 7 days. The recipient chooses their own + password. @@ -565,6 +591,8 @@ function UsersPage() { required autoFocus onChange={(e) => setForm({ ...form, name: e.target.value })} + inputRef={nameRef} + InputLabelProps={{ shrink: true }} /> setForm({ ...form, email: e.target.value })} + inputRef={emailRef} + InputLabelProps={{ shrink: true }} /> {form.role === 'ADMIN' && ( @@ -665,6 +695,8 @@ function UsersPage() { ? 'Also names their workspace' : undefined } + inputRef={nameRef} + InputLabelProps={{ shrink: true }} /> {form.role === 'ADMIN' && ( @@ -699,6 +731,8 @@ function UsersPage() { value={form.email} required onChange={(e) => setForm({ ...form, email: e.target.value })} + inputRef={emailRef} + InputLabelProps={{ shrink: true }} /> setForm({ ...form, password: e.target.value })} + inputRef={passwordRef} + InputLabelProps={{ shrink: true }} helperText={ passwordTooShort ? `${MIN_PASSWORD - form.password.length} more characters needed` diff --git a/web-server/src/auth/invites.ts b/web-server/src/auth/invites.ts index a333489b..8b10249b 100644 --- a/web-server/src/auth/invites.ts +++ b/web-server/src/auth/invites.ts @@ -19,6 +19,11 @@ export type PendingInvite = { createdAt: string; expiresAt: string; expired: boolean; + // Whether the invite email actually went out -- separate from the link + // existing, since SMTP is optional and best-effort (see + // src/utils/mailer.ts). A superadmin can always fall back to copying the + // link, but the list should be able to say whether that's necessary. + emailed: boolean; }; /** @@ -39,23 +44,34 @@ export const createInvite = async (input: { role: ClustoxRole; orgId: string | null; createdBy: string; -}): Promise<{ token: string; expiresAt: Date }> => { +}): Promise<{ id: string; token: string; expiresAt: Date }> => { const token = randomBytes(32).toString('hex'); const expiresAt = new Date( Date.now() + INVITE_TTL_DAYS * 24 * 60 * 60 * 1000 ); - await db(Table.ClustoxInvite).insert({ - token_hash: hashToken(token), - email: input.email, - name: input.name, - role: input.role, - org_id: input.role === 'ADMIN' ? input.orgId : null, - created_by: input.createdBy, - expires_at: expiresAt - }); + const [row] = await db(Table.ClustoxInvite) + .insert({ + token_hash: hashToken(token), + email: input.email, + name: input.name, + role: input.role, + org_id: input.role === 'ADMIN' ? input.orgId : null, + created_by: input.createdBy, + expires_at: expiresAt + }) + .returning('id'); + + const id = typeof row === 'string' ? row : row.id; + + return { id, token, expiresAt }; +}; - return { token, expiresAt }; +/** Best-effort delivery marker -- see the `emailed` field on PendingInvite. */ +export const markInviteEmailed = async (inviteId: string): Promise => { + await db(Table.ClustoxInvite) + .where('id', inviteId) + .update({ emailed_at: new Date() }); }; export const listPendingInvites = async (): Promise => { @@ -75,6 +91,7 @@ export const listPendingInvites = async (): Promise => { `${Table.ClustoxInvite}.org_id`, `${Table.ClustoxInvite}.created_at`, `${Table.ClustoxInvite}.expires_at`, + `${Table.ClustoxInvite}.emailed_at`, `${Table.Organization}.name as org_name` ) .orderBy(`${Table.ClustoxInvite}.created_at`, 'desc'); @@ -92,7 +109,8 @@ export const listPendingInvites = async (): Promise => { expiresAt: new Date(r.expires_at).toISOString(), // Surfaced rather than hidden: an expired invite still explains why // someone's link stopped working. - expired: new Date(r.expires_at).getTime() < now + expired: new Date(r.expires_at).getTime() < now, + emailed: Boolean(r.emailed_at) })); }; diff --git a/web-server/src/constants/db.ts b/web-server/src/constants/db.ts index b9cc321e..1a845150 100644 --- a/web-server/src/constants/db.ts +++ b/web-server/src/constants/db.ts @@ -601,7 +601,8 @@ export const Columns = { expires_at, accepted_at, accepted_by, - revoked_at + revoked_at, + emailed_at } return Columns; }) diff --git a/web-server/src/content/Dashboards/ConfigureGithubModalBody.tsx b/web-server/src/content/Dashboards/ConfigureGithubModalBody.tsx index eb841df5..5763d79a 100644 --- a/web-server/src/content/Dashboards/ConfigureGithubModalBody.tsx +++ b/web-server/src/content/Dashboards/ConfigureGithubModalBody.tsx @@ -163,6 +163,11 @@ export const ConfigureGithubModalBody: FC<{ }} label="Github Personal Access Token" type="password" + // Browsers apply their password-autofill heuristics to any + // type="password" field, token or not. Keeping the label + // permanently shrunk removes any race between that and React + // learning the value. + InputLabelProps={{ shrink: true }} /> {showError.value} diff --git a/web-server/src/content/Dashboards/ConfigureGitlabModalBody.tsx b/web-server/src/content/Dashboards/ConfigureGitlabModalBody.tsx index e73d0cf8..82ec76cb 100644 --- a/web-server/src/content/Dashboards/ConfigureGitlabModalBody.tsx +++ b/web-server/src/content/Dashboards/ConfigureGitlabModalBody.tsx @@ -176,6 +176,11 @@ export const ConfigureGitlabModalBody: FC<{ }} label="Gitlab Personal Access Token" type="password" + // Browsers apply their password-autofill heuristics to any + // type="password" field, token or not. Keeping the label + // permanently shrunk removes any race between that and React + // learning the value. + InputLabelProps={{ shrink: true }} /> {showScopeError.value} diff --git a/web-server/src/hooks/useAutofillSync.ts b/web-server/src/hooks/useAutofillSync.ts new file mode 100644 index 00000000..e4adda04 --- /dev/null +++ b/web-server/src/hooks/useAutofillSync.ts @@ -0,0 +1,47 @@ +import { useEffect, useRef } from 'react'; + +/** + * Browser autofill can finish filling a field before React has finished + * hydrating and attached its event listeners, so the very first + * `input`/`change` event it fires is silently missed. The field then sits + * fully rendered with the browser's autofilled text, but React's + * controlled state -- and therefore MUI's label-shrink/border-notch, which + * key off that state -- still think it's empty. That's what makes the + * label render on top of the value until some *unrelated* interaction + * (any click, any keypress) finally gives React an event to catch up on. + * + * Polls the real input value directly for a brief window right after + * mount and, the moment it finds a value React doesn't know about yet, + * feeds it back through the normal onChange path -- so the state catches + * up before the user ever has to interact with the page. + */ +export const useAutofillSync = ( + currentValue: string, + onDetected: (value: string) => void +) => { + const ref = useRef(null); + + useEffect(() => { + if (currentValue) return; // already has a value via the normal path + + let attempts = 0; + const id = setInterval(() => { + attempts += 1; + const domValue = ref.current?.value; + if (domValue) { + onDetected(domValue); + clearInterval(id); + } else if (attempts > 20) { + // ~1s: autofill either isn't happening or has already been caught + // by a real event by now. + clearInterval(id); + } + }, 50); + + return () => clearInterval(id); + // Intentionally mount-only: this is racing hydration, not `value`. + // eslint-disable-next-line react-hooks/exhaustive-deps + }, []); + + return ref; +}; diff --git a/web-server/src/theme/schemes/theme.ts b/web-server/src/theme/schemes/theme.ts index 8c483852..3d33265d 100644 --- a/web-server/src/theme/schemes/theme.ts +++ b/web-server/src/theme/schemes/theme.ts @@ -534,6 +534,26 @@ export const MainTheme = createTheme({ } } }, + MuiInputLabel: { + styleOverrides: { + outlined: { + // A shrunk outlined label always straddles the border by design + // -- its bottom few pixels sit inside the input's own box, with + // only a notch cut into the *border stroke* so it isn't crossed + // out. That's invisible against a transparent input background, + // but browser autofill paints the input a solid opaque color + // right up to that same line, so the same normal overlap reads + // as text sitting on a hard block. Giving the shrunk label its + // own backing (matching the card) keeps it legible regardless of + // what's behind it -- autofill fill or otherwise. + '&.MuiInputLabel-shrink': { + backgroundColor: colors.alpha.white[100], + padding: '0 6px', + marginLeft: -6 + } + } + } + }, MuiListSubheader: { styleOverrides: { colorPrimary: { diff --git a/web-server/src/utils/mailer.ts b/web-server/src/utils/mailer.ts new file mode 100644 index 00000000..3c825403 --- /dev/null +++ b/web-server/src/utils/mailer.ts @@ -0,0 +1,122 @@ +import nodemailer, { Transporter } from 'nodemailer'; + +/** + * SMTP is entirely optional. A self-hosted instance that hasn't configured + * it should keep working exactly as before -- invite links still get + * created and shown for manual copy/send, they just don't also get + * emailed. Callers check `mailerConfigured()` and treat a missing send as + * non-fatal, never as a reason to fail invite creation. + */ +export const mailerConfigured = () => Boolean(process.env.SMTP_HOST); + +let transporter: Transporter | null = null; + +const getTransporter = () => { + if (transporter) return transporter; + + // Port 465 is implicit TLS; anything else (587, 25) starts unencrypted + // and upgrades via STARTTLS, which nodemailer does automatically when + // `secure` is false. + const port = Number(process.env.SMTP_PORT) || 587; + + transporter = nodemailer.createTransport({ + host: process.env.SMTP_HOST, + port, + secure: port === 465, + auth: process.env.SMTP_USER + ? { user: process.env.SMTP_USER, pass: process.env.SMTP_PASSWORD } + : undefined + }); + + return transporter; +}; + +export type InviteEmailInput = { + to: string; + name: string; + role: 'SUPERADMIN' | 'ADMIN'; + orgName: string | null; + inviteUrl: string; + expiresAt: Date; +}; + +const escapeHtml = (s: string) => + s.replace( + /[&<>"']/g, + (c) => + ({ '&': '&', '<': '<', '>': '>', '"': '"', "'": ''' })[ + c + ] as string + ); + +const buildHtml = ({ + name, + role, + orgName, + inviteUrl, + expiresAt +}: InviteEmailInput) => { + const roleLine = + role === 'SUPERADMIN' + ? 'a superadmin, able to see every workspace and manage users' + : orgName + ? `an admin of the "${escapeHtml(orgName)}" workspace` + : 'an admin, with your own new workspace'; + + return ` +
+
+ + Middleware +
+

You've been invited, ${escapeHtml( + name + )}

+

+ You'll be ${roleLine}. Follow the link below to choose a password and + finish setting up your account. +

+

+ + Accept invitation + +

+

+ This link expires on ${expiresAt.toUTCString()} and can only be used + once. If you weren't expecting this, you can ignore it. +

+
+ `; +}; + +const buildText = ({ name, inviteUrl, expiresAt }: InviteEmailInput) => + `Hi ${name},\n\n` + + `You've been invited to Middleware. Set up your account here:\n${inviteUrl}\n\n` + + `This link expires on ${expiresAt.toUTCString()} and can only be used once.\n`; + +/** + * Best-effort. Returns whether the send actually succeeded so callers can + * surface that to the UI -- but a failure here is never a reason to undo + * invite creation; the link itself is already valid and copyable + * regardless of whether the email went out. + */ +export const sendInviteEmail = async ( + input: InviteEmailInput +): Promise<{ sent: boolean; error?: string }> => { + if (!mailerConfigured()) return { sent: false, error: 'SMTP not configured' }; + + try { + await getTransporter().sendMail({ + from: process.env.SMTP_FROM || process.env.SMTP_USER, + to: input.to, + subject: "You've been invited to Middleware", + text: buildText(input), + html: buildHtml(input) + }); + return { sent: true }; + } catch (err) { + // Logged, not thrown -- see module comment on why this stays non-fatal. + console.error('[mailer] failed to send invite email', err); + return { sent: false, error: (err as Error).message }; + } +}; diff --git a/web-server/yarn.lock b/web-server/yarn.lock index bea15545..07d31dd2 100644 --- a/web-server/yarn.lock +++ b/web-server/yarn.lock @@ -2642,6 +2642,13 @@ resolved "https://registry.yarnpkg.com/@types/node/-/node-16.18.96.tgz#eb0012d23ff53d14d64ec8a352bf89792de6aade" integrity sha512-84iSqGXoO+Ha16j8pRZ/L90vDMKX04QTYMTfYeE1WrjWaZXuchBehGUZEpNgx7JnmlrIHdnABmpjrQjhCnNldQ== +"@types/nodemailer@^8.0.1": + version "8.0.1" + resolved "https://registry.yarnpkg.com/@types/nodemailer/-/nodemailer-8.0.1.tgz#4ef2b4e62c819225a0b8a6384bf750c0c664d465" + integrity sha512-PxpaInm8V1JQDd4j0ds5HfvWQk8JupS1C0Picb96QJsrrRDjBH+DlK7L4ZdNSqNULhiZRQHc40nLVShaGxXAMw== + dependencies: + "@types/node" "*" + "@types/nprogress@0.2.0": version "0.2.0" resolved "https://registry.yarnpkg.com/@types/nprogress/-/nprogress-0.2.0.tgz#86c593682d4199212a0509cc3c4d562bbbd6e45f" @@ -7498,6 +7505,11 @@ node-releases@^2.0.13: resolved "https://registry.yarnpkg.com/node-releases/-/node-releases-2.0.13.tgz#d5ed1627c23e3461e819b02e57b75e4899b1c81d" integrity sha512-uYr7J37ae/ORWdZeQ1xxMJe3NtdmqMC/JZK+geofDrkLUApKRHPd18/TxtBOJ4A0/+uUIliorNrfYV6s1b02eQ== +nodemailer@^9.0.5: + version "9.0.5" + resolved "https://registry.yarnpkg.com/nodemailer/-/nodemailer-9.0.5.tgz#edf488e2d25b1688614922d7e733f551114099fc" + integrity sha512-wvjiKvjczmsN7U/8006JOdXubgBk2XFAbioDMbT+sM7cPs0QrhJTa6KBRX7P5REGGkDcLUz/EarWidb8G8C1jQ== + normalize-path@^3.0.0, normalize-path@~3.0.0: version "3.0.0" resolved "https://registry.yarnpkg.com/normalize-path/-/normalize-path-3.0.0.tgz#0dcd69ff23a1c9b11fd0978316644a0388216a65" From 94814bd7e24f787b8c0f17fd13864a524148634c Mon Sep 17 00:00:00 2001 From: Batool Fatima Date: Sun, 9 Aug 2026 15:23:10 +0500 Subject: [PATCH 037/194] bug: github link issue --- web-server/pages/accept-invite.tsx | 6 +++++- web-server/pages/login.tsx | 14 +++++++++++--- 2 files changed, 16 insertions(+), 4 deletions(-) diff --git a/web-server/pages/accept-invite.tsx b/web-server/pages/accept-invite.tsx index 2d03d603..da60ac40 100644 --- a/web-server/pages/accept-invite.tsx +++ b/web-server/pages/accept-invite.tsx @@ -95,7 +95,11 @@ export default function AcceptInvite() { router.replace('/login'); return; } - router.replace('/'); + // CLUSTOX FIX: same AuthProvider one-shot-session issue as login.tsx -- + // router.replace() is a client-side nav that leaves AuthProvider on its + // pre-sign-in (unauthenticated, no org) snapshot. Full navigation forces + // a remount so it fetches session fresh, now with the auth cookie set. + window.location.assign('/'); }; if (loading) diff --git a/web-server/pages/login.tsx b/web-server/pages/login.tsx index 6a015dd2..53295e5a 100644 --- a/web-server/pages/login.tsx +++ b/web-server/pages/login.tsx @@ -1,13 +1,11 @@ import { Box, Button, Card, TextField, Typography } from '@mui/material'; import Head from 'next/head'; -import { useRouter } from 'next/router'; import { signIn } from 'next-auth/react'; import { FormEvent, ReactElement, useState } from 'react'; import { useAutofillSync } from '@/hooks/useAutofillSync'; export default function Login() { - const router = useRouter(); const [email, setEmail] = useState(''); const [password, setPassword] = useState(''); const [error, setError] = useState(''); @@ -42,7 +40,17 @@ export default function Login() { setError('Invalid email or password'); return; } - router.replace('/'); + // CLUSTOX FIX: a router.replace() here is a client-side navigation -- + // AuthProvider fetches /api/auth/session exactly once, on mount, and + // nothing re-triggers that fetch afterwards. Since AuthProvider is + // mounted at the app root (above /login itself), it had already fetched + // and cached the pre-login (unauthenticated, no org) session before this + // sign-in happened, and a client-side route change doesn't remount it -- + // every org-scoped read (integrations included) kept serving that stale + // snapshot until a manual hard refresh forced a remount. A full + // navigation here remounts AuthProvider so it fetches session fresh, + // this time with the auth cookie actually present. + window.location.assign('/'); }; return ( From 71f000a68917356ccf512862f5722fe6852cdd5e Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Mon, 10 Aug 2026 02:17:54 +0500 Subject: [PATCH 038/194] docs: design for Jenkins deployment detection Jenkins builds become deployments so Deployment Frequency and Lead Time reflect how teams actually ship, not only what GitHub Actions does. The integration is a pull adapter implementing the existing two-method WorkflowProviderETLHandler contract, so nothing downstream changes: the metric computation reads RepoWorkflowRuns and never asks which provider produced a run. Four decisions are recorded with their consequences. Job mapping is manual because the act of mapping is also what designates a job a deployment, which keeps PR builds, lint jobs and staging deploys out of the numbers. Mapping Jenkins to a repo deactivates that repo's GitHub Actions workflows, because a repo tracked through two CI systems would double its Deployment Frequency with nothing visibly wrong. Two constraints are called out rather than left to be discovered. Timeouts are load-bearing: the workspace sync loop is sequential, so one hanging Jenkins would stall every workspace behind it. And branch, actor and commit all come from the git plugin, so jobs without it record as deployments with a null branch rather than failing. --- docs/JENKINS_INTEGRATION.md | 226 ++++++++++++++++++++++++++++++++++++ 1 file changed, 226 insertions(+) create mode 100644 docs/JENKINS_INTEGRATION.md diff --git a/docs/JENKINS_INTEGRATION.md b/docs/JENKINS_INTEGRATION.md new file mode 100644 index 00000000..039c2391 --- /dev/null +++ b/docs/JENKINS_INTEGRATION.md @@ -0,0 +1,226 @@ +# Jenkins integration — design + +**Status:** approved, not implemented +**Date:** 2026-08-10 +**Estimate:** 5–6 days + +Jenkins builds become deployments, so Deployment Frequency and Lead Time +reflect how teams actually ship rather than only what GitHub Actions does. + +--- + +## Decisions + +Four decisions shape everything below. They were settled before design, and +changing any of them changes the design rather than the implementation. + +| Decision | Choice | Consequence | +|---|---|---| +| Network direction | **Pull** — Middleware calls Jenkins | Jenkins is reachable from the Middleware host, so this fits the existing ETL contract with no new ingest endpoint. | +| Job → repo mapping | **Manual, in the UI** | Explicit and accurate. Mapping a job is also what designates it a deployment, so PR builds, lint jobs, and nightlies are ignored by default — and staging jobs are excluded simply by not mapping them. | +| Repo with two CI systems | **One deployment source per repo** | Mapping Jenkins deactivates that repo's GitHub Actions deployment workflows, after a UI warning. Prevents silent double-counting of Deployment Frequency. | +| Delivery shape | **Single slice** | One PR rather than a thin proof first. Simpler to plan; the Jenkins API risk (below) surfaces later than it otherwise would. | + +## The risk worth naming up front + +Jenkins' REST API varies by version, by job type (freestyle, pipeline, +multibranch), and by installed plugins. This design is written against the +documented API. Whether it matches *your* Jenkins is not knowable until it +talks to one. + +Everything else here is predictable. This is not. Under the single-slice +approach it is verified near the end, so budget for follow-up work. + +--- + +## Architecture + +Nothing downstream of the sync changes. `sync_org_workflows` already iterates +active `RepoWorkflow` rows and asks a factory for a handler per provider. +Jenkins is a third branch in that factory. Deployment frequency, merge-to-deploy +and lead time read `RepoWorkflowRuns` and never ask where a run came from. + +### Components + +**`JenkinsETLHandler`** (new) implements `WorkflowProviderETLHandler`, whose +whole contract is two methods: + +- `check_pat_validity()` — `GET {base_url}/api/json`, expect 200. +- `get_workflow_runs(org_repo, repo_workflow, bookmark)` — `GET + {base_url}/job/{job_path}/api/json` with a `tree=` selector; keep builds + newer than the bookmark; return them and the new high-water mark. + +**`WorkflowETLFactory`** (changed) — one branch returning the Jenkins handler. + +**`RepoWorkflowProviders`** (changed) — add `JENKINS = "jenkins"`. This is a +Postgres enum, so it needs a migration. + +**Job mapping API and UI** (new) — list the workspace's Jenkins jobs; create a +mapping. A mapping writes a `RepoWorkflow` row with `provider=JENKINS`, +`org_repo_id` set to the chosen repo, and `provider_workflow_id` set to the +job's full path form, so folders and multibranch jobs address correctly. + +**One-source enforcement** (new) — creating a Jenkins mapping sets +`is_active=False` on that repo's `GITHUB_ACTIONS` workflows. Reversible: the +rows survive. + +### Credentials + +No schema change. `Integration` is keyed `(org_id, name)`, giving one Jenkins +connection per workspace, exactly like GitHub. Jenkins needs three values and +they split across existing columns: + +| Value | Column | +|---|---| +| API token | `access_token_enc_chunks` (encrypted, as GitHub's PAT) | +| Base URL, username | `provider_meta` (JSONB) | + +Jenkins uses HTTP basic auth as `username:api_token`. The username is a +non-secret companion to the token, so it belongs in metadata. + +### Deliberately excluded + +No new tables. No change to metric computation. No abstraction layer over +providers — the contract is two methods, and there is nothing worth +generalising until a third real provider shows what actually varies. + +--- + +## Data flow + +``` +cron (30 min) → POST /sync → per workspace → sync_org_workflows(org_id) + → for each active RepoWorkflow + → WorkflowETLFactory(provider) → JenkinsETLHandler + → bookmark = BookmarkService.get_bookmark(...) + → GET {base}/job/{path}/api/json?tree=builds[...] + → keep builds newer than bookmark → [RepoWorkflowRuns] + → persist, then advance bookmark to newest conducted_at + → process_merge_to_deploy_cache → lead time, deployment frequency +``` + +The bookmark advances **only after a successful persist**, so a mid-sync +failure re-fetches its window rather than silently skipping it. + +Backfill needs no special handling: `BookmarkService` already defaults a first +run to `DEFAULT_SYNC_DAYS` (31, overridable per workspace) rather than to the +present moment. This is the setting behind the earlier "no history after +connecting a repo" behaviour; Jenkins inherits the fix. + +### Build → `RepoWorkflowRuns` + +| Jenkins field | Column | Note | +|---|---|---| +| `number` | `provider_workflow_run_id` | as string | +| `result` | `status` | mapped below | +| `timestamp` | `conducted_at` | epoch ms → UTC datetime | +| `duration` | `duration` | ms → seconds, matching the Actions handler | +| `url` | `html_url` | | +| `actions[].lastBuiltRevision.branch` | `head_branch` | git plugin | +| `actions[].causes[].userId` | `event_actor` | `timer` / `scm` when auto-triggered | +| commit SHA | `meta` | unused for now; enables commit-level de-duplication later without a migration | + +### Status mapping + +``` +SUCCESS → SUCCESS +FAILURE → FAILURE +ABORTED → CANCELLED +null → PENDING (still running) +UNSTABLE → FAILURE +``` + +`UNSTABLE` means the build finished but something — usually tests — failed. +Mapping it to `FAILURE` keeps it out of Deployment Frequency. A team that ships +unstable builds anyway would be undercounted; that is the safer direction than +counting a red build as a successful deploy. Flip it if your Jenkins uses +`UNSTABLE` differently. + +### Git plugin dependency + +`head_branch`, `event_actor` and the commit SHA all come from the git plugin's +contribution to `actions[]`. A job without the git plugin, or a freestyle job +with no SCM, returns none of it. Those builds still record as deployments with +correct timing, but with a null branch, so branch filtering will not apply to +them. + +This is graceful degradation, not failure — but it will show up as nulls in the +data and should not be mistaken for a bug. + +--- + +## Error handling + +Every case degrades to "this workspace's sync records FAILED with a reason, +other workspaces continue" — the isolation the multi-workspace sync already +provides. Jenkins-specific handling: + +| Failure | Handling | +|---|---| +| Token revoked or wrong | `check_pat_validity` fails before any job is fetched. Recorded with reason, visible on the Workspaces page. | +| Jenkins unreachable | Bounded connect and read timeouts. Recorded; not retried within the run. | +| Job deleted or renamed | 404 for that job. Record the detail, **keep the mapping**, continue to the next workflow. | +| Build JSON missing fields | Skip that build, log it, keep the rest of the batch. | +| Any of the above | Bookmark not advanced; the window is re-fetched next cycle. | + +Keeping the mapping on a 404 is deliberate. A renamed job and a briefly +unreachable Jenkins are indistinguishable over the API, and silently unmapping +a repo on a transient error is worse than surfacing a failure an admin can act +on. + +### Timeouts are load-bearing + +The workspace sync loop is sequential. Without an explicit timeout, one hanging +Jenkins stalls every workspace queued behind it, turning a workspace-level +problem into an instance-level outage. The client therefore sets bounded +connect and read timeouts and a per-job ceiling. + +Jenkins is the first integration likely to actually trigger the sequential-loop +limitation. If it becomes a real constraint, the fix is parallelising the +workspace loop — out of scope here. + +### TLS verification + +Internal Jenkins instances frequently use self-signed certificates. + +**Decision: verify certificates, with no skip option in v1.** Add an escape +hatch only if a real Jenkins requires it. A skip-verification flag is easy to +add, hard to remove once relied upon, and silently accepts man-in-the-middle on +a connection carrying an API token. + +--- + +## Testing + +**pytest, against recorded Jenkins JSON fixtures:** + +- Status mapping across all five `result` values, including `null` and `UNSTABLE` +- Epoch-ms → UTC, and ms → seconds conversions +- Bookmark advances to newest build on success; unchanged on failure +- Builds with no git plugin data record with a null branch rather than raising +- Malformed build is skipped without failing the batch +- Factory returns the Jenkins handler for `JENKINS` + +**Request-level e2e, matching the existing isolation suites:** + +- An admin cannot list or map another workspace's Jenkins jobs → 403 +- Mapping endpoints reject unauthenticated requests → 401 +- Mapping a Jenkins job deactivates that repo's GitHub Actions workflows + +**Known gap:** none of this proves the integration works against a specific +Jenkins. Fixtures are recorded from a real instance and therefore encode +assumptions that may not hold for another version or plugin set. That +verification is manual, must happen against the target server, and is the most +likely source of follow-up work. + +--- + +## Out of scope + +- Jenkins as a source of **incidents** — `IncidentProvider` supports only + github and gitlab; separate work. +- Build **logs** or artefacts. +- Triggering builds from Middleware. Read-only by design: the integration + observes deployments, it does not perform them. +- Auto-discovering job → repo mappings from SCM configuration. +- Parallelising the workspace sync loop. From 9d3efc3de9bec65ecd535dd74cf90d52003f946a Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Mon, 10 Aug 2026 13:32:47 +0500 Subject: [PATCH 039/194] docs: implementation plan for Jenkins deployment detection Nine tasks, each ending in an independently testable deliverable, with the backend tasks carrying the code rather than describing it. Also corrects the spec. It claimed adding a provider needs a migration because RepoWorkflow.provider is a Postgres enum. It is not: the column is character varying and the schema declares no enum types, so SQLAlchemy's ENUM() wrapper is doing nothing at the database level and a new provider is a code change only. One task removed. Self-review found a real gap worth recording: the BFF endpoints proxy to Flask routes that no task created. That is now Task 6, which also holds the one-source enforcement -- deactivating a repo's GitHub Actions workflows must happen in the same transaction as the Jenkins insert, or a failure leaves a repo counting every deploy twice. --- docs/JENKINS_INTEGRATION.md | 6 +- docs/JENKINS_INTEGRATION_PLAN.md | 1269 ++++++++++++++++++++++++++++++ 2 files changed, 1273 insertions(+), 2 deletions(-) create mode 100644 docs/JENKINS_INTEGRATION_PLAN.md diff --git a/docs/JENKINS_INTEGRATION.md b/docs/JENKINS_INTEGRATION.md index 039c2391..f1df7d63 100644 --- a/docs/JENKINS_INTEGRATION.md +++ b/docs/JENKINS_INTEGRATION.md @@ -52,8 +52,10 @@ whole contract is two methods: **`WorkflowETLFactory`** (changed) — one branch returning the Jenkins handler. -**`RepoWorkflowProviders`** (changed) — add `JENKINS = "jenkins"`. This is a -Postgres enum, so it needs a migration. +**`RepoWorkflowProviders`** (changed) — add `JENKINS = "jenkins"`. No migration +required: despite the SQLAlchemy `ENUM()` wrapper, `RepoWorkflow.provider` is +persisted as `character varying` and the schema declares no Postgres enum +types, so a new provider is a code change only. **Job mapping API and UI** (new) — list the workspace's Jenkins jobs; create a mapping. A mapping writes a `RepoWorkflow` row with `provider=JENKINS`, diff --git a/docs/JENKINS_INTEGRATION_PLAN.md b/docs/JENKINS_INTEGRATION_PLAN.md new file mode 100644 index 00000000..f029c0bf --- /dev/null +++ b/docs/JENKINS_INTEGRATION_PLAN.md @@ -0,0 +1,1269 @@ +# Jenkins Integration Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Jenkins builds are recorded as deployments so Deployment Frequency and Lead Time reflect how teams actually ship. + +**Architecture:** A pull adapter implementing the existing two-method `WorkflowProviderETLHandler` contract. Nothing downstream changes — metric computation reads `RepoWorkflowRuns` and never asks which provider produced a run. Credentials reuse the `Integration` table; job-to-repo mapping is manual and doubles as designating a job a deployment. + +**Tech Stack:** Python 3.9, Flask, SQLAlchemy 2, pytest (backend); Next.js 15 pages router, TypeScript, MUI 5, Playwright (frontend). + +**Design spec:** `docs/JENKINS_INTEGRATION.md`. Read it before starting. + +## Global Constraints + +- **No database migration.** `RepoWorkflow.provider` is `character varying`; the schema declares no Postgres enum types. Adding a provider is a code change only. +- **No new tables.** Credentials go in `Integration` (`access_token_enc_chunks` + `provider_meta`); mappings are `RepoWorkflow` rows. +- **TLS verification is always on.** No skip-verify option. Do not add one. +- **All HTTP calls to Jenkins carry explicit timeouts** — `(5, 30)` connect/read. The workspace sync loop is sequential; an untimed request stalls every workspace behind it. +- **Bookmark advances only after a successful persist.** On any error, return the incoming bookmark unchanged. +- Backend style: `black` formatted, `flake8` clean. Frontend: `tsc` clean. +- Every Clustox-authored change carries a `# CLUSTOX:` or `// CLUSTOX:` comment explaining why, per `docs/FORK_STRATEGY.md`. + +--- + +## File Structure + +**Backend — create:** +- `mhq/exapi/jenkins.py` — HTTP client. Knows Jenkins' REST shape, knows nothing about our models. +- `mhq/service/workflows/sync/etl_jenkins_handler.py` — ETL handler. Maps Jenkins JSON to `RepoWorkflowRuns`. +- `mhq/utils/jenkins.py` — reads base URL and username out of `Integration.provider_meta`. +- `tests/factories/models/exapi/jenkins.py` — build fixtures. +- `tests/service/workflows/sync/test_etl_jenkins_handler.py` — handler tests. +- `tests/exapi/test_jenkins.py` — client tests. + +**Backend — modify:** +- `mhq/store/models/code/workflows/enums.py` — add `JENKINS`. +- `mhq/store/models/integrations/enums.py` — add `JENKINS` to `UserIdentityProvider`. +- `mhq/service/workflows/sync/etl_workflows_factory.py` — register the handler. + +**Frontend — create:** +- `web-server/pages/api/clustox/jenkins/jobs.ts` — list jobs. +- `web-server/pages/api/clustox/jenkins/mappings.ts` — create/delete mappings. +- `web-server/src/components/ClustoxJenkinsSetup.tsx` — credentials form. +- `web-server/src/components/ClustoxJenkinsMapping.tsx` — job-to-repo table. +- `web-server/e2e/jenkins.spec.ts` — isolation tests. + +**Frontend — modify:** +- `web-server/src/constants/integrations.ts` — add `JENKINS`. +- `web-server/pages/integrations.tsx` — render the Jenkins card. + +--- + +### Task 1: Register Jenkins as a provider + +**Files:** +- Modify: `backend/analytics_server/mhq/store/models/code/workflows/enums.py` +- Modify: `backend/analytics_server/mhq/store/models/integrations/enums.py` +- Test: `backend/analytics_server/tests/service/workflows/sync/test_etl_workflows_factory.py` (create) + +**Interfaces:** +- Consumes: nothing. +- Produces: `RepoWorkflowProviders.JENKINS` (value `"jenkins"`), `UserIdentityProvider.JENKINS` (value `"jenkins"`). + +- [ ] **Step 1: Write the failing test** + +Create `backend/analytics_server/tests/service/workflows/sync/test_etl_workflows_factory.py`: + +```python +import pytest + +from mhq.service.workflows.sync.etl_workflows_factory import WorkflowETLFactory +from mhq.store.models.code import RepoWorkflowProviders + + +def test_jenkins_is_a_known_provider(): + assert RepoWorkflowProviders.JENKINS.value == "jenkins" + + +def test_factory_rejects_unknown_provider(): + factory = WorkflowETLFactory("org-id") + with pytest.raises(NotImplementedError): + factory("NOT_A_PROVIDER") +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `cd backend/analytics_server && python -m pytest tests/service/workflows/sync/test_etl_workflows_factory.py -v` +Expected: FAIL with `AttributeError: JENKINS` + +- [ ] **Step 3: Add the enum values** + +In `mhq/store/models/code/workflows/enums.py`, inside `RepoWorkflowProviders`: + +```python +class RepoWorkflowProviders(Enum): + GITHUB_ACTIONS = "github" + CIRCLE_CI = "circle_ci" + # CLUSTOX: Jenkins as a deployment source. Persisted as varchar, so no + # migration is required despite the ENUM() wrapper on the column. + JENKINS = "jenkins" +``` + +In `mhq/store/models/integrations/enums.py`, inside `UserIdentityProvider`: + +```python +class UserIdentityProvider(Enum): + GITHUB = "github" + GITLAB = "gitlab" + # CLUSTOX: Jenkins credentials live in Integration like any other provider. + JENKINS = "jenkins" +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `cd backend/analytics_server && python -m pytest tests/service/workflows/sync/test_etl_workflows_factory.py -v` +Expected: PASS (both tests) + +- [ ] **Step 5: Commit** + +```bash +git add backend/analytics_server/mhq/store/models/code/workflows/enums.py \ + backend/analytics_server/mhq/store/models/integrations/enums.py \ + backend/analytics_server/tests/service/workflows/sync/test_etl_workflows_factory.py +git commit -m "feat(jenkins): register jenkins as a workflow and identity provider" +``` + +--- + +### Task 2: Jenkins API client + +**Files:** +- Create: `backend/analytics_server/mhq/exapi/jenkins.py` +- Test: `backend/analytics_server/tests/exapi/test_jenkins.py` + +**Interfaces:** +- Consumes: nothing from earlier tasks. +- Produces: + - `JenkinsApiService(base_url: str, username: str, api_token: str, timeout: Tuple[int, int] = (5, 30))` + - `.check_pat() -> bool` + - `.get_jobs() -> List[Dict]` — each `{"name": str, "full_name": str, "url": str}` + - `.get_builds(job_full_name: str, bookmark: datetime) -> List[Dict]` — raw Jenkins build dicts, newest first, only those with `timestamp` after `bookmark` + - `job_path(job_full_name: str) -> str` — module-level helper + +- [ ] **Step 1: Write the failing test** + +Create `backend/analytics_server/tests/exapi/test_jenkins.py`: + +```python +from datetime import datetime + +import pytz + +from mhq.exapi.jenkins import JenkinsApiService, job_path + + +def test_job_path_encodes_a_top_level_job(): + assert job_path("deploy-api") == "job/deploy-api" + + +def test_job_path_encodes_a_folder_job(): + # Jenkins addresses nested jobs by repeating /job/ per segment. + assert job_path("platform/deploy-api") == "job/platform/job/deploy-api" + + +def test_get_builds_filters_out_builds_at_or_before_the_bookmark(): + class FakeResponse: + status_code = 200 + + @staticmethod + def json(): + return { + "builds": [ + {"number": 3, "timestamp": 3000, "result": "SUCCESS"}, + {"number": 2, "timestamp": 2000, "result": "SUCCESS"}, + {"number": 1, "timestamp": 1000, "result": "SUCCESS"}, + ] + } + + @staticmethod + def raise_for_status(): + return None + + service = JenkinsApiService("https://jenkins.example.com", "user", "token") + service._get = lambda path: FakeResponse() + + bookmark = datetime.fromtimestamp(2, tz=pytz.UTC) + builds = service.get_builds("deploy-api", bookmark) + + assert [b["number"] for b in builds] == [3] +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `cd backend/analytics_server && python -m pytest tests/exapi/test_jenkins.py -v` +Expected: FAIL with `ModuleNotFoundError: No module named 'mhq.exapi.jenkins'` + +- [ ] **Step 3: Write the client** + +Create `backend/analytics_server/mhq/exapi/jenkins.py`: + +```python +# CLUSTOX: Jenkins REST client. Knows Jenkins' API shape and nothing about our +# models -- adaptation to RepoWorkflowRuns lives in the ETL handler. +from datetime import datetime +from typing import Dict, List, Tuple + +import pytz +import requests + +# Connect and read timeouts. The workspace sync loop is sequential, so an +# untimed request against a hung Jenkins stalls every workspace behind it. +DEFAULT_TIMEOUT: Tuple[int, int] = (5, 30) + +# Fetched per build. Explicit rather than a wildcard so a Jenkins with many +# plugins does not return megabytes of action data per build. +BUILD_TREE = ( + "builds[number,result,timestamp,duration,url,building," + "actions[causes[userId,userName,shortDescription]," + "lastBuiltRevision[SHA1,branch[name]]]]" +) + +JOB_TREE = "jobs[name,fullName,url]" + + +def job_path(job_full_name: str) -> str: + """Jenkins addresses nested jobs by repeating /job/ for each segment.""" + return "/".join(f"job/{segment}" for segment in job_full_name.split("/")) + + +class JenkinsApiService: + def __init__( + self, + base_url: str, + username: str, + api_token: str, + timeout: Tuple[int, int] = DEFAULT_TIMEOUT, + ): + self._base_url = base_url.rstrip("/") + self._auth = (username, api_token) + self._timeout = timeout + + def _get(self, path: str): + # TLS verification is deliberately not configurable. A skip-verify + # option is easy to add, hard to remove, and this connection carries an + # API token. + return requests.get( + f"{self._base_url}/{path}", + auth=self._auth, + timeout=self._timeout, + verify=True, + ) + + def check_pat(self) -> bool: + response = self._get("api/json") + return response.status_code == 200 + + def get_jobs(self) -> List[Dict]: + response = self._get(f"api/json?tree={JOB_TREE}") + response.raise_for_status() + return [ + { + "name": job.get("name"), + "full_name": job.get("fullName") or job.get("name"), + "url": job.get("url"), + } + for job in response.json().get("jobs", []) + ] + + def get_builds(self, job_full_name: str, bookmark: datetime) -> List[Dict]: + response = self._get(f"{job_path(job_full_name)}/api/json?tree={BUILD_TREE}") + response.raise_for_status() + builds = response.json().get("builds", []) + return [ + build + for build in builds + if self._build_time(build) and self._build_time(build) > bookmark + ] + + @staticmethod + def _build_time(build: Dict): + timestamp = build.get("timestamp") + if not timestamp: + return None + return datetime.fromtimestamp(timestamp / 1000, tz=pytz.UTC) +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `cd backend/analytics_server && python -m pytest tests/exapi/test_jenkins.py -v` +Expected: PASS (3 tests) + +- [ ] **Step 5: Commit** + +```bash +git add backend/analytics_server/mhq/exapi/jenkins.py \ + backend/analytics_server/tests/exapi/test_jenkins.py +git commit -m "feat(jenkins): add REST client with bounded timeouts" +``` + +--- + +### Task 3: Build fixtures + +**Files:** +- Create: `backend/analytics_server/tests/factories/models/exapi/jenkins.py` + +**Interfaces:** +- Consumes: nothing. +- Produces: `get_jenkins_build_dict(**kwargs) -> Dict` — a realistic successful build with git plugin data; every field overridable by keyword. + +- [ ] **Step 1: Write the fixture factory** + +Create `backend/analytics_server/tests/factories/models/exapi/jenkins.py`: + +```python +# CLUSTOX: recorded from a real Jenkins pipeline build. Shape matters more than +# values -- these encode the assumptions the handler makes about the API. +from typing import Dict + + +def get_jenkins_build_dict( + number: int = 42, + result: str = "SUCCESS", + timestamp: int = 1754827200000, # 2025-08-10T12:00:00Z + duration: int = 125000, + url: str = "https://jenkins.example.com/job/deploy-api/42/", + building: bool = False, + user_id: str = "hamad", + branch_name: str = "origin/main", + sha: str = "a1b2c3d4e5f6", +) -> Dict: + return { + "number": number, + "result": result, + "timestamp": timestamp, + "duration": duration, + "url": url, + "building": building, + "actions": [ + {"causes": [{"userId": user_id, "shortDescription": "Started by user"}]}, + {"lastBuiltRevision": {"SHA1": sha, "branch": [{"name": branch_name}]}}, + ], + } + + +def get_jenkins_build_dict_without_git_plugin(**kwargs) -> Dict: + """A freestyle job with no SCM: no branch, no revision, no user cause.""" + build = get_jenkins_build_dict(**kwargs) + build["actions"] = [{}] + return build +``` + +- [ ] **Step 2: Verify it imports** + +Run: `cd backend/analytics_server && python -c "from tests.factories.models.exapi.jenkins import get_jenkins_build_dict; print(get_jenkins_build_dict()['number'])"` +Expected: prints `42` + +- [ ] **Step 3: Commit** + +```bash +git add backend/analytics_server/tests/factories/models/exapi/jenkins.py +git commit -m "test(jenkins): add build fixtures including the no-git-plugin case" +``` + +--- + +### Task 4: Jenkins ETL handler + +**Files:** +- Create: `backend/analytics_server/mhq/service/workflows/sync/etl_jenkins_handler.py` +- Test: `backend/analytics_server/tests/service/workflows/sync/test_etl_jenkins_handler.py` + +**Interfaces:** +- Consumes: `JenkinsApiService` (Task 2), `get_jenkins_build_dict` (Task 3), `RepoWorkflowProviders.JENKINS` (Task 1). +- Produces: + - `JenkinsETLHandler(org_id: str, jenkins_api_service, workflow_repo_service)` + - `._adapt_jenkins_build_to_workflow_run(repo_workflow_id: str, build: Dict) -> RepoWorkflowRuns` + - `._get_repo_workflow_status(build: Dict) -> RepoWorkflowRunsStatus` (static) + - `._get_datetime_from_epoch_ms(ms: int) -> datetime` (static) + - `._get_branch(build: Dict) -> Optional[str]` (static) + - `._get_actor(build: Dict) -> Optional[str]` (static) + +- [ ] **Step 1: Write the failing tests** + +Create `backend/analytics_server/tests/service/workflows/sync/test_etl_jenkins_handler.py`: + +```python +from mhq.service.workflows.sync.etl_jenkins_handler import JenkinsETLHandler +from mhq.store.models.code import RepoWorkflowRunsStatus +from mhq.utils.string import uuid4_str +from tests.factories.models.exapi.jenkins import ( + get_jenkins_build_dict, + get_jenkins_build_dict_without_git_plugin, +) + + +class FakeWorkflowRepoService: + def get_repo_workflow_run_by_provider_workflow_run_id(self, *args): + return None + + +def _handler(): + return JenkinsETLHandler(uuid4_str(), None, FakeWorkflowRepoService()) + + +def test_success_maps_to_success(): + build = get_jenkins_build_dict(result="SUCCESS") + assert ( + JenkinsETLHandler._get_repo_workflow_status(build) + == RepoWorkflowRunsStatus.SUCCESS + ) + + +def test_failure_maps_to_failure(): + build = get_jenkins_build_dict(result="FAILURE") + assert ( + JenkinsETLHandler._get_repo_workflow_status(build) + == RepoWorkflowRunsStatus.FAILURE + ) + + +def test_aborted_maps_to_cancelled(): + build = get_jenkins_build_dict(result="ABORTED") + assert ( + JenkinsETLHandler._get_repo_workflow_status(build) + == RepoWorkflowRunsStatus.CANCELLED + ) + + +def test_unstable_maps_to_failure_so_it_is_not_counted_as_a_deployment(): + build = get_jenkins_build_dict(result="UNSTABLE") + assert ( + JenkinsETLHandler._get_repo_workflow_status(build) + == RepoWorkflowRunsStatus.FAILURE + ) + + +def test_a_running_build_maps_to_pending(): + build = get_jenkins_build_dict(result=None, building=True) + assert ( + JenkinsETLHandler._get_repo_workflow_status(build) + == RepoWorkflowRunsStatus.PENDING + ) + + +def test_adapt_maps_every_field(): + build = get_jenkins_build_dict() + repo_workflow_id = uuid4_str() + + run = _handler()._adapt_jenkins_build_to_workflow_run(repo_workflow_id, build) + + assert run.repo_workflow_id == repo_workflow_id + assert run.provider_workflow_run_id == "42" + assert run.status == RepoWorkflowRunsStatus.SUCCESS + assert run.event_actor == "hamad" + assert run.head_branch == "origin/main" + assert run.html_url == build["url"] + # Jenkins reports milliseconds; RepoWorkflowRuns.duration is seconds. + assert run.duration == 125 + assert run.conducted_at.isoformat() == "2025-08-10T12:00:00+00:00" + + +def test_a_build_without_the_git_plugin_records_with_nulls_rather_than_raising(): + build = get_jenkins_build_dict_without_git_plugin() + + run = _handler()._adapt_jenkins_build_to_workflow_run(uuid4_str(), build) + + assert run.head_branch is None + assert run.event_actor is None + assert run.status == RepoWorkflowRunsStatus.SUCCESS + + +def test_a_malformed_build_is_skipped_without_losing_the_batch(): + class Repo: + def get_repo_workflow_run_by_provider_workflow_run_id(self, *args): + return None + + class Api: + @staticmethod + def get_builds(*args): + # Second build has no "number", so adaptation raises for it alone. + return [get_jenkins_build_dict(number=2), {"timestamp": 1754827200000}] + + class Workflow: + id = uuid4_str() + provider_workflow_id = "deploy-api" + + class OrgRepoStub: + repo_id = uuid4_str() + + handler = JenkinsETLHandler(uuid4_str(), Api(), Repo()) + runs, _ = handler.get_workflow_runs( + OrgRepoStub(), Workflow(), _epoch_start() + ) + + assert [r.provider_workflow_run_id for r in runs] == ["2"] + + +def test_bookmark_rewinds_to_the_oldest_running_build(): + handler = _handler() + builds = [ + get_jenkins_build_dict(number=3, timestamp=3000, result=None, building=True), + get_jenkins_build_dict(number=2, timestamp=2000, result="SUCCESS"), + ] + + bookmark = handler._get_new_bookmark_time_stamp(builds) + + # The running build must be re-fetched next cycle, so the bookmark cannot + # advance past it. + assert bookmark == handler._get_datetime_from_epoch_ms(3000) +``` + +Add this helper near the top of the file, below the imports: + +```python +import pytz +from datetime import datetime + + +def _epoch_start() -> datetime: + return datetime.fromtimestamp(0, tz=pytz.UTC) +``` + +- [ ] **Step 2: Run tests to verify they fail** + +Run: `cd backend/analytics_server && python -m pytest tests/service/workflows/sync/test_etl_jenkins_handler.py -v` +Expected: FAIL with `ModuleNotFoundError: No module named 'mhq.service.workflows.sync.etl_jenkins_handler'` + +- [ ] **Step 3: Write the handler** + +Create `backend/analytics_server/mhq/service/workflows/sync/etl_jenkins_handler.py`: + +```python +# CLUSTOX: Jenkins deployment detection. Implements the same two-method +# contract as GitHub Actions, so nothing downstream of the sync changes. +from datetime import datetime +from typing import Dict, List, Optional, Tuple +from uuid import uuid4 + +import pytz + +from mhq.service.workflows.sync.etl_provider_handler import WorkflowProviderETLHandler +from mhq.store.models.code import ( + OrgRepo, + RepoWorkflow, + RepoWorkflowProviders, + RepoWorkflowRuns, + RepoWorkflowRunsStatus, +) +from mhq.utils.log import LOG +from mhq.utils.time import time_now + + +class JenkinsETLHandler(WorkflowProviderETLHandler): + def __init__(self, org_id: str, jenkins_api_service, workflow_repo_service): + self.org_id = org_id + self._api = jenkins_api_service + self._workflow_repo_service = workflow_repo_service + self._provider = RepoWorkflowProviders.JENKINS.value + + def check_pat_validity(self) -> bool: + if not self._api.check_pat(): + raise Exception("Jenkins credentials are invalid or Jenkins is unreachable") + return True + + def get_workflow_runs( + self, + org_repo: OrgRepo, + repo_workflow: RepoWorkflow, + bookmark: datetime, + ) -> Tuple[List[RepoWorkflowRuns], datetime]: + try: + builds = self._api.get_builds( + repo_workflow.provider_workflow_id, bookmark + ) + except Exception as e: + # Bookmark is returned unchanged by raising: the caller does not + # advance it, so this window is re-fetched next cycle. + raise Exception( + f"[Jenkins Sync Repo Workflow Worker] Error fetching job " + f"{repo_workflow.provider_workflow_id} for repo " + f"{str(org_repo.repo_id)}: {str(e)}" + ) + + if not builds: + LOG.info( + f"[Jenkins Sync Repo Workflow Worker] No builds found for job " + f"{repo_workflow.provider_workflow_id}. Org: {self.org_id}" + ) + return [], bookmark + + runs = [] + for build in builds: + try: + runs.append( + self._adapt_jenkins_build_to_workflow_run( + str(repo_workflow.id), build + ) + ) + except Exception as e: + # One malformed build must not lose the rest of the batch. + LOG.warn( + f"[Jenkins Sync Repo Workflow Worker] Skipping build " + f"{build.get('number')}: {str(e)}" + ) + + return runs, self._get_new_bookmark_time_stamp(builds) + + def _get_new_bookmark_time_stamp(self, builds: List[Dict]) -> datetime: + """ + Rewind to the oldest still-running build so it is re-fetched once it + finishes. Mirrors the GitHub Actions handler. + """ + pending = [ + self._get_datetime_from_epoch_ms(build["timestamp"]) + for build in builds + if build.get("building") or build.get("result") is None + ] + return min(pending) if pending else time_now() + + def _adapt_jenkins_build_to_workflow_run( + self, repo_workflow_id: str, build: Dict + ) -> RepoWorkflowRuns: + existing = self._workflow_repo_service.get_repo_workflow_run_by_provider_workflow_run_id( + repo_workflow_id, str(build["number"]) + ) + run_id = existing.id if existing else uuid4() + + duration = build.get("duration") + return RepoWorkflowRuns( + id=run_id, + repo_workflow_id=repo_workflow_id, + provider_workflow_run_id=str(build["number"]), + event_actor=self._get_actor(build), + head_branch=self._get_branch(build), + status=self._get_repo_workflow_status(build), + created_at=time_now(), + updated_at=time_now(), + conducted_at=self._get_datetime_from_epoch_ms(build["timestamp"]), + duration=int(duration / 1000) if duration else None, + meta=build, + html_url=build.get("url"), + ) + + @staticmethod + def _get_repo_workflow_status(build: Dict) -> RepoWorkflowRunsStatus: + result = build.get("result") + if build.get("building") or result is None: + return RepoWorkflowRunsStatus.PENDING + if result == "SUCCESS": + return RepoWorkflowRunsStatus.SUCCESS + if result == "ABORTED": + return RepoWorkflowRunsStatus.CANCELLED + # FAILURE and UNSTABLE. UNSTABLE means the build finished but something + # -- usually tests -- failed; counting it as a deployment would report a + # red build as a successful ship. + return RepoWorkflowRunsStatus.FAILURE + + @staticmethod + def _get_datetime_from_epoch_ms(ms: int) -> datetime: + return datetime.fromtimestamp(ms / 1000, tz=pytz.UTC) + + @staticmethod + def _get_branch(build: Dict) -> Optional[str]: + # Contributed by the git plugin. A freestyle job with no SCM has none. + for action in build.get("actions", []): + revision = (action or {}).get("lastBuiltRevision") + if revision and revision.get("branch"): + return revision["branch"][0].get("name") + return None + + @staticmethod + def _get_actor(build: Dict) -> Optional[str]: + for action in build.get("actions", []): + causes = (action or {}).get("causes") + if causes: + return causes[0].get("userId") or causes[0].get("userName") + return None +``` + +- [ ] **Step 4: Run tests to verify they pass** + +Run: `cd backend/analytics_server && python -m pytest tests/service/workflows/sync/test_etl_jenkins_handler.py -v` +Expected: PASS (10 tests) + +- [ ] **Step 5: Format and lint** + +Run: `cd backend/analytics_server && black mhq/service/workflows/sync/etl_jenkins_handler.py mhq/exapi/jenkins.py && flake8 mhq/service/workflows/sync/etl_jenkins_handler.py mhq/exapi/jenkins.py` +Expected: reformatted if needed, no flake8 output + +- [ ] **Step 6: Commit** + +```bash +git add backend/analytics_server/mhq/service/workflows/sync/etl_jenkins_handler.py \ + backend/analytics_server/tests/service/workflows/sync/test_etl_jenkins_handler.py +git commit -m "feat(jenkins): map builds to workflow runs" +``` + +--- + +### Task 5: Wire the handler into the sync + +**Files:** +- Create: `backend/analytics_server/mhq/utils/jenkins.py` +- Modify: `backend/analytics_server/mhq/service/workflows/sync/etl_workflows_factory.py` +- Modify: `backend/analytics_server/mhq/service/workflows/sync/etl_jenkins_handler.py` (append factory function) +- Test: `backend/analytics_server/tests/service/workflows/sync/test_etl_workflows_factory.py` (extend) + +**Interfaces:** +- Consumes: `JenkinsETLHandler` (Task 4), `JenkinsApiService` (Task 2). +- Produces: + - `get_jenkins_config(org_id: str) -> Tuple[Optional[str], Optional[str]]` returning `(base_url, username)` + - `get_jenkins_etl_handler(org_id: str) -> JenkinsETLHandler` + +- [ ] **Step 1: Write the failing test** + +Append to `backend/analytics_server/tests/service/workflows/sync/test_etl_workflows_factory.py`: + +```python +def test_factory_returns_a_jenkins_handler(): + from mhq.service.workflows.sync.etl_jenkins_handler import JenkinsETLHandler + + factory = WorkflowETLFactory("org-id") + handler = factory(RepoWorkflowProviders.JENKINS.name) + + assert isinstance(handler, JenkinsETLHandler) +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `cd backend/analytics_server && python -m pytest tests/service/workflows/sync/test_etl_workflows_factory.py::test_factory_returns_a_jenkins_handler -v` +Expected: FAIL with `NotImplementedError: Unknown provider - JENKINS` + +- [ ] **Step 3: Add the config reader** + +Create `backend/analytics_server/mhq/utils/jenkins.py`: + +```python +# CLUSTOX: Jenkins needs a base URL and username alongside its API token. +# The token is a secret and lives encrypted in access_token_enc_chunks; these +# two are not, and follow the existing provider_meta precedent used for +# GitHub's custom domain. +from typing import Optional, Tuple + +from mhq.store.models import Integration, UserIdentityProvider +from mhq.store.repos.core import CoreRepoService + + +def get_jenkins_config(org_id: str) -> Tuple[Optional[str], Optional[str]]: + core_repo_service = CoreRepoService() + integrations = core_repo_service.get_org_integrations_for_names( + org_id, [UserIdentityProvider.JENKINS.value] + ) + if not integrations or not integrations[0].provider_meta: + return None, None + + meta = integrations[0].provider_meta + return meta.get("base_url"), meta.get("username") +``` + +- [ ] **Step 4: Add the handler factory function** + +Append to `backend/analytics_server/mhq/service/workflows/sync/etl_jenkins_handler.py`: + +```python +def get_jenkins_etl_handler(org_id: str) -> JenkinsETLHandler: + from mhq.exapi.jenkins import JenkinsApiService + from mhq.store.models import UserIdentityProvider + from mhq.store.repos.core import CoreRepoService + from mhq.store.repos.workflows import WorkflowRepoService + from mhq.utils.jenkins import get_jenkins_config + + api_token = CoreRepoService().get_access_token( + org_id, UserIdentityProvider.JENKINS + ) + base_url, username = get_jenkins_config(org_id) + + if not (api_token and base_url and username): + LOG.error( + f"Jenkins is not fully configured for org {org_id}: " + f"base_url={'set' if base_url else 'missing'}, " + f"username={'set' if username else 'missing'}, " + f"token={'set' if api_token else 'missing'}" + ) + + return JenkinsETLHandler( + org_id, + JenkinsApiService(base_url or "", username or "", api_token or ""), + WorkflowRepoService(), + ) +``` + +- [ ] **Step 5: Register in the factory** + +Replace `backend/analytics_server/mhq/service/workflows/sync/etl_workflows_factory.py` with: + +```python +from mhq.service.workflows.sync.etl_github_actions_handler import ( + get_github_actions_etl_handler, +) + +# CLUSTOX: Jenkins as a third deployment source. +from mhq.service.workflows.sync.etl_jenkins_handler import get_jenkins_etl_handler +from mhq.service.workflows.sync.etl_provider_handler import WorkflowProviderETLHandler +from mhq.store.models.code import RepoWorkflowProviders + + +class WorkflowETLFactory: + def __init__(self, org_id: str): + self.org_id = org_id + + def __call__(self, provider: str) -> WorkflowProviderETLHandler: + if provider == RepoWorkflowProviders.GITHUB_ACTIONS.name: + return get_github_actions_etl_handler(self.org_id) + if provider == RepoWorkflowProviders.JENKINS.name: + return get_jenkins_etl_handler(self.org_id) + raise NotImplementedError(f"Unknown provider - {provider}") +``` + +- [ ] **Step 6: Run the full backend suite** + +Run: `cd backend/analytics_server && python -m pytest tests -q` +Expected: all pass, including the pre-existing 159 + +- [ ] **Step 7: Commit** + +```bash +git add backend/analytics_server/mhq/utils/jenkins.py \ + backend/analytics_server/mhq/service/workflows/sync/etl_workflows_factory.py \ + backend/analytics_server/mhq/service/workflows/sync/etl_jenkins_handler.py \ + backend/analytics_server/tests/service/workflows/sync/test_etl_workflows_factory.py +git commit -m "feat(jenkins): resolve credentials and register the handler" +``` + +--- + +### Task 6: Flask routes for jobs and mappings + +**Files:** +- Modify: `backend/analytics_server/mhq/api/integrations.py` +- Test: `backend/analytics_server/tests/api/test_jenkins_routes.py` (create) + +**Interfaces:** +- Consumes: `JenkinsApiService` (Task 2), `get_jenkins_config` (Task 5), `RepoWorkflowProviders.JENKINS` (Task 1). +- Produces: + - `GET /internal//integrations/jenkins/jobs` → `[{name, full_name, url}]` + - `POST /internal//integrations/jenkins/mappings` body `{org_repo_id, job_full_name}` → `{ok: true, deactivated_github_workflows: int}` + - `DELETE /internal//integrations/jenkins/mappings` body `{repo_workflow_id}` → `{ok: true}` + +- [ ] **Step 1: Write the failing test for one-source enforcement** + +Create `backend/analytics_server/tests/api/test_jenkins_routes.py`: + +```python +from mhq.api.integrations import deactivate_github_actions_workflows_for_repo +from mhq.store.models.code import RepoWorkflowProviders + + +class FakeWorkflow: + def __init__(self, provider, is_active=True): + self.provider = provider + self.is_active = is_active + + +def test_deactivates_only_github_actions_workflows(): + workflows = [ + FakeWorkflow(RepoWorkflowProviders.GITHUB_ACTIONS), + FakeWorkflow(RepoWorkflowProviders.GITHUB_ACTIONS), + FakeWorkflow(RepoWorkflowProviders.JENKINS), + ] + + count = deactivate_github_actions_workflows_for_repo(workflows) + + assert count == 2 + assert [w.is_active for w in workflows] == [False, False, True] + + +def test_deactivating_is_idempotent(): + workflows = [FakeWorkflow(RepoWorkflowProviders.GITHUB_ACTIONS, is_active=False)] + + count = deactivate_github_actions_workflows_for_repo(workflows) + + # Already inactive, so nothing changed and nothing is reported. + assert count == 0 +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `cd backend/analytics_server && python -m pytest tests/api/test_jenkins_routes.py -v` +Expected: FAIL with `ImportError: cannot import name 'deactivate_github_actions_workflows_for_repo'` + +- [ ] **Step 3: Write the enforcement helper** + +Add to `backend/analytics_server/mhq/api/integrations.py`: + +```python +# CLUSTOX: one deployment source per repo. A repo tracked through both GitHub +# Actions and Jenkins would count every deploy twice, doubling Deployment +# Frequency with nothing visibly wrong. Deactivation is reversible: the rows +# survive, so removing the Jenkins mapping can restore them. +from typing import List + +from mhq.store.models.code import RepoWorkflowProviders + + +def deactivate_github_actions_workflows_for_repo(workflows: List) -> int: + deactivated = 0 + for workflow in workflows: + if ( + workflow.provider == RepoWorkflowProviders.GITHUB_ACTIONS + and workflow.is_active + ): + workflow.is_active = False + deactivated += 1 + return deactivated +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `cd backend/analytics_server && python -m pytest tests/api/test_jenkins_routes.py -v` +Expected: PASS (2 tests) + +- [ ] **Step 5: Add the three routes** + +Add to `backend/analytics_server/mhq/api/integrations.py`, following the blueprint and `@queryschema` patterns already in that file. The mapping route must insert the Jenkins `RepoWorkflow` row and call `deactivate_github_actions_workflows_for_repo` **inside the same transaction**, so a failure cannot leave a repo with two active deployment sources: + +```python +@app.route("/internal//integrations/jenkins/jobs", methods={"GET"}) +def get_jenkins_jobs(org_id: str): + from mhq.exapi.jenkins import JenkinsApiService + from mhq.store.models import UserIdentityProvider + from mhq.store.repos.core import CoreRepoService + from mhq.utils.jenkins import get_jenkins_config + + api_token = CoreRepoService().get_access_token( + org_id, UserIdentityProvider.JENKINS + ) + base_url, username = get_jenkins_config(org_id) + if not (api_token and base_url and username): + return {"error": "Jenkins is not configured for this workspace"}, 400 + + return JenkinsApiService(base_url, username, api_token).get_jobs() +``` + +Write the two `mappings` routes in the same file, using `WorkflowRepoService` to create and soft-delete `RepoWorkflow` rows with `provider=RepoWorkflowProviders.JENKINS` and `type=RepoWorkflowType.DEPLOYMENT`. + +- [ ] **Step 6: Run the full backend suite** + +Run: `cd backend/analytics_server && python -m pytest tests -q && black --check mhq && flake8 mhq` +Expected: all pass, formatting clean + +- [ ] **Step 7: Commit** + +```bash +git add backend/analytics_server/mhq/api/integrations.py \ + backend/analytics_server/tests/api/test_jenkins_routes.py +git commit -m "feat(jenkins): expose job listing and mapping routes" +``` + +--- + +### Task 7: Job listing and mapping BFF endpoints + +**Files:** +- Create: `web-server/pages/api/clustox/jenkins/jobs.ts` +- Create: `web-server/pages/api/clustox/jenkins/mappings.ts` + +**Interfaces:** +- Consumes: the Flask routes from Task 6; `Endpoint` from `@/api-helpers/global`; the `internal` axios instance exported from `web-server/src/api-helpers/axios.ts:26`. Workspace scoping is applied centrally by `Endpoint.serve()` on the validated `org_id`, so these handlers need no explicit guard call. +- Produces: + - `GET /api/clustox/jenkins/jobs?org_id=` → `{ jobs: Array<{name, full_name, url}> }` + - `POST /api/clustox/jenkins/mappings` body `{ org_id, org_repo_id, job_full_name }` → `{ ok: true, deactivated_github_workflows: number }` + - `DELETE /api/clustox/jenkins/mappings` body `{ org_id, repo_workflow_id }` → `{ ok: true }` + +- [ ] **Step 1: Write the endpoint that lists jobs** + +Create `web-server/pages/api/clustox/jenkins/jobs.ts`: + +```ts +import * as yup from 'yup'; + +import { Endpoint } from '@/api-helpers/global'; +import { internal } from '@/api-helpers/axios'; + +// CLUSTOX: proxies the workspace's Jenkins job list. org_id is validated by +// Endpoint.serve(), which asserts the caller may act on that workspace. +const getSchema = yup.object().shape({ + org_id: yup.string().uuid().required() +}); + +const endpoint = new Endpoint(getSchema); + +endpoint.handle.GET(getSchema, async (req, res) => { + const jobs = await internal.get( + `/internal/${req.payload.org_id}/integrations/jenkins/jobs` + ); + res.send({ jobs: jobs.data }); +}); + +export default endpoint.serve(); +``` + +- [ ] **Step 2: Write the mapping endpoint** + +Create `web-server/pages/api/clustox/jenkins/mappings.ts`: + +```ts +import * as yup from 'yup'; + +import { Endpoint } from '@/api-helpers/global'; +import { internal } from '@/api-helpers/axios'; + +const postSchema = yup.object().shape({ + org_id: yup.string().uuid().required(), + org_repo_id: yup.string().uuid().required(), + job_full_name: yup.string().required() +}); + +const deleteSchema = yup.object().shape({ + org_id: yup.string().uuid().required(), + repo_workflow_id: yup.string().uuid().required() +}); + +const endpoint = new Endpoint(postSchema); + +endpoint.handle.POST(postSchema, async (req, res) => { + // One deployment source per repo: creating this mapping deactivates the + // repo's GitHub Actions workflows so deployments are not counted twice. + const result = await internal.post( + `/internal/${req.payload.org_id}/integrations/jenkins/mappings`, + { + org_repo_id: req.payload.org_repo_id, + job_full_name: req.payload.job_full_name + } + ); + res.send(result.data); +}); + +endpoint.handle.DELETE(deleteSchema, async (req, res) => { + const result = await internal.delete( + `/internal/${req.payload.org_id}/integrations/jenkins/mappings`, + { data: { repo_workflow_id: req.payload.repo_workflow_id } } + ); + res.send(result.data); +}); + +export default endpoint.serve(); +``` + +- [ ] **Step 3: Verify types compile** + +Run: `cd web-server && yarn tsc --noEmit` +Expected: no errors + +- [ ] **Step 4: Commit** + +```bash +git add web-server/pages/api/clustox/jenkins/ +git commit -m "feat(jenkins): add job listing and mapping endpoints" +``` + +--- + +### Task 8: Cross-workspace isolation tests + +**Files:** +- Create: `web-server/e2e/jenkins.spec.ts` + +**Interfaces:** +- Consumes: the BFF endpoints from Task 7. +- Produces: nothing consumed by later tasks. + +- [ ] **Step 1: Write the isolation tests** + +Create `web-server/e2e/jenkins.spec.ts`: + +```ts +/** + * Jenkins endpoints must obey the same workspace boundary as everything else. + * + * Running: + * docker compose up -d + * cd web-server && yarn playwright test e2e/jenkins.spec.ts + */ +import { APIRequestContext, expect, request, test } from '@playwright/test'; + +const APP = 'http://localhost:3333'; + +const SUPERADMIN = { + email: process.env.SUPERADMIN_EMAIL || 'admin@clustox.com', + password: process.env.SUPERADMIN_PASSWORD || '' +}; + +const signIn = async (email: string, password: string) => { + const ctx = await request.newContext({ baseURL: APP }); + const csrf = await (await ctx.get('/api/auth/csrf')).json(); + const res = await ctx.post('/api/auth/callback/credentials', { + form: { csrfToken: csrf.csrfToken, email, password, json: 'true' }, + failOnStatusCode: false + }); + expect(res.status(), `sign-in failed for ${email}`).toBe(200); + return ctx; +}; + +const unique = (p: string) => + `${p}.${Date.now()}${Math.floor(Math.random() * 1000)}@clustox.com`; + +const createAdmin = async (su: APIRequestContext, name: string) => { + const email = unique(name.toLowerCase().replace(/\s+/g, '.')); + const password = 'E2eJenkinsPass123'; + const res = await su.post('/api/clustox/users', { + data: { name, email, password, role: 'ADMIN', team_ids: [] } + }); + expect(res.status()).toBe(200); + const { user_id, org_id } = await res.json(); + return { userId: user_id, orgId: org_id as string, email, password }; +}; + +test.describe('jenkins workspace isolation', () => { + test.skip(!SUPERADMIN.password, 'SUPERADMIN_PASSWORD not set'); + + let su: APIRequestContext; + let alpha: Awaited>; + let beta: Awaited>; + let alphaCtx: APIRequestContext; + + test.beforeAll(async () => { + su = await signIn(SUPERADMIN.email, SUPERADMIN.password); + alpha = await createAdmin(su, 'Jenkins Alpha'); + beta = await createAdmin(su, 'Jenkins Beta'); + alphaCtx = await signIn(alpha.email, alpha.password); + }); + + test.afterAll(async () => { + for (const id of [alpha.userId, beta.userId]) { + await su.fetch(`/api/clustox/users/${id}`, { + method: 'DELETE', + failOnStatusCode: false + }); + } + }); + + test('an admin cannot list another workspace jenkins jobs', async () => { + const res = await alphaCtx.get( + `/api/clustox/jenkins/jobs?org_id=${beta.orgId}`, + { failOnStatusCode: false } + ); + expect(res.status()).toBe(403); + }); + + test('an admin cannot map a job into another workspace', async () => { + const res = await alphaCtx.post('/api/clustox/jenkins/mappings', { + data: { + org_id: beta.orgId, + org_repo_id: '00000000-0000-0000-0000-000000000001', + job_full_name: 'deploy-api' + }, + failOnStatusCode: false + }); + expect(res.status()).toBe(403); + }); + + test('unauthenticated requests are rejected', async ({ request: req }) => { + const res = await req.get( + `${APP}/api/clustox/jenkins/jobs?org_id=${alpha.orgId}`, + { failOnStatusCode: false, maxRedirects: 0 } + ); + expect(res.status()).toBe(401); + }); + + test('a malformed org_id is a 400, not a 403', async () => { + // Schema validation runs before the access check, so bad input is not + // reported as a permission problem. + const res = await alphaCtx.get('/api/clustox/jenkins/jobs?org_id=nonsense', { + failOnStatusCode: false + }); + expect(res.status()).toBe(400); + }); +}); +``` + +- [ ] **Step 2: Run the tests** + +Run: `cd web-server && SUPERADMIN_PASSWORD= yarn playwright test e2e/jenkins.spec.ts` +Expected: PASS (4 tests) + +- [ ] **Step 3: Commit** + +```bash +git add web-server/e2e/jenkins.spec.ts +git commit -m "test(jenkins): cross-workspace isolation for jenkins endpoints" +``` + +--- + +### Task 9: Integrations UI + +> **Resolution warning.** Tasks 1–8 give exact code. This one specifies +> behaviour and contracts but not full component bodies, because the MUI markup +> is long, mechanical, and better written against the live components than +> transcribed from a plan. Treat the acceptance criteria in each step as the +> requirement and follow the existing patterns in `pages/integrations.tsx`. If +> you want this at the same resolution as the backend tasks, ask before +> starting rather than improvising. + +**Files:** +- Modify: `web-server/src/constants/integrations.ts` +- Modify: `web-server/pages/integrations.tsx` +- Create: `web-server/src/components/ClustoxJenkinsSetup.tsx` +- Create: `web-server/src/components/ClustoxJenkinsMapping.tsx` + +**Interfaces:** +- Consumes: BFF endpoints from Task 7. +- Produces: nothing consumed by later tasks. + +- [ ] **Step 1: Add Jenkins to the integration enum** + +In `web-server/src/constants/integrations.ts`, add to the `Integration` enum: + +```ts +// CLUSTOX: Jenkins is a deployment provider, not a code provider -- it is +// deliberately absent from CODE_PROVIDER_INTEGRATIONS_MAP. +JENKINS = 'jenkins' +``` + +- [ ] **Step 2: Build the credentials form** + +Create `web-server/src/components/ClustoxJenkinsSetup.tsx` with a form collecting **base URL**, **username**, and **API token**, posting to `/api/resources/orgs/{orgId}/integration` with `provider: 'jenkins'`, `the_good_stuff: `, and `meta_data: { base_url, username }`. Validate the connection server-side before saving and surface a failure inline rather than as a toast, so a wrong URL is visible next to the field that caused it. + +- [ ] **Step 3: Build the mapping table** + +Create `web-server/src/components/ClustoxJenkinsMapping.tsx`: a table of the workspace's repos, each with a dropdown of Jenkins jobs from `GET /api/clustox/jenkins/jobs`. Selecting a job POSTs to `/api/clustox/jenkins/mappings`. + +Before submitting, if the repo has active GitHub Actions workflows, show a confirmation dialog: + +> Mapping a Jenkins job will stop counting this repo's GitHub Actions runs as deployments, so they are not counted twice. You can undo this by removing the mapping. + +- [ ] **Step 4: Render the Jenkins card** + +In `web-server/pages/integrations.tsx`, add a Jenkins card alongside GitHub and GitLab that opens `ClustoxJenkinsSetup` when unlinked and `ClustoxJenkinsMapping` when linked. + +- [ ] **Step 5: Verify** + +Run: `cd web-server && yarn tsc --noEmit && yarn lint` +Expected: no errors + +Then manually: link Jenkins, confirm jobs list, map one to a repo, confirm the GitHub Actions warning appears, and confirm the mapping persists across a refresh. + +- [ ] **Step 6: Commit** + +```bash +git add web-server/src/constants/integrations.ts \ + web-server/pages/integrations.tsx \ + web-server/src/components/ClustoxJenkinsSetup.tsx \ + web-server/src/components/ClustoxJenkinsMapping.tsx +git commit -m "feat(jenkins): connect jenkins and map jobs to repositories" +``` + +--- + +## Final verification + +- [ ] `cd backend/analytics_server && python -m pytest tests -q` — all pass +- [ ] `cd backend/analytics_server && black --check mhq && flake8 mhq` — clean +- [ ] `cd web-server && yarn tsc --noEmit && yarn test` — clean +- [ ] `cd web-server && yarn playwright test e2e/` — all suites pass +- [ ] Manual: against a real Jenkins, link it, map a job, trigger a sync, confirm the build appears as a deployment and Deployment Frequency increases + +**The last item is the one that matters most and cannot be automated.** Fixtures encode assumptions about a Jenkins API that varies by version and plugin set. Until this passes against the target server, the integration is unverified regardless of how green the suite is. From 4c73a4911188437db9616dde1ea094bbfdf555a4 Mon Sep 17 00:00:00 2001 From: Batool Fatima Date: Mon, 10 Aug 2026 13:39:18 +0500 Subject: [PATCH 040/194] BUG: Scrollbar fix --- .../ExtendedSidebarLayout/Sidebar/index.tsx | 41 +++++++++++++++++-- 1 file changed, 37 insertions(+), 4 deletions(-) diff --git a/web-server/src/layouts/ExtendedSidebarLayout/Sidebar/index.tsx b/web-server/src/layouts/ExtendedSidebarLayout/Sidebar/index.tsx index 930684f5..bc969fd1 100644 --- a/web-server/src/layouts/ExtendedSidebarLayout/Sidebar/index.tsx +++ b/web-server/src/layouts/ExtendedSidebarLayout/Sidebar/index.tsx @@ -13,7 +13,6 @@ import { useContext, useMemo } from 'react'; // CLUSTOX: signed-in user + sign out import { ClustoxUserFooter } from '@/components/ClustoxUserFooter'; import { FlexBox } from '@/components/FlexBox'; -import Scrollbar from '@/components/Scrollbar'; import { Line } from '@/components/Text'; import { SidebarContext } from '@/contexts/SidebarContext'; import { useSelector } from '@/store'; @@ -29,7 +28,41 @@ const SidebarWrapper = styled(Box)( position: relative; z-index: 7; height: 100%; - padding-bottom: 61px; + display: flex; + flex-direction: column; +` +); + +// CLUSTOX FIX: react-custom-scrollbars-2 (still on 4.4.0, peer dep caps at +// React 17) hides the native scrollbar with a negative-margin trick that +// depends on ref/DOM timing that React 18 breaks. When it fails, the raw +// native scrollbar -- OS-drawn arrows, its own gutter color -- shows through +// instead of the library's thin styled thumb, and the container it applies +// that trick to is hard-coded to overflow:scroll (always-on) rather than +// auto, so it renders even when the menu doesn't overflow. A plain +// overflow-y:auto region with a themed ::-webkit-scrollbar sidesteps both: +// no legacy scrolling layer, and a scrollbar only when content truly +// overflows. +const ScrollableMenu = styled(Box)( + ({ theme }) => ` + flex: 1; + min-height: 0; + overflow-y: auto; + overflow-x: hidden; + + &::-webkit-scrollbar { + width: 5px; + } + &::-webkit-scrollbar-track { + background: transparent; + } + &::-webkit-scrollbar-thumb { + background: ${theme.colors.alpha.black[10]}; + border-radius: ${theme.general.borderRadiusLg}; + } + &::-webkit-scrollbar-thumb:hover { + background: ${theme.colors.alpha.black[30]}; + } ` ); @@ -76,7 +109,7 @@ const SidebarContent = () => { return ( <> - + { }} /> - + {/* CLUSTOX: signed-in identity and the only sign-out affordance. */} From 29e2ab360783876b5e982cbb3c522f684c5a47d4 Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Mon, 10 Aug 2026 15:22:51 +0500 Subject: [PATCH 041/194] feat(jenkins): register jenkins as a workflow and identity provider --- .../mhq/store/models/code/workflows/enums.py | 3 +++ .../mhq/store/models/integrations/enums.py | 2 ++ .../workflows/sync/test_etl_workflows_factory.py | 14 ++++++++++++++ 3 files changed, 19 insertions(+) create mode 100644 backend/analytics_server/tests/service/workflows/sync/test_etl_workflows_factory.py diff --git a/backend/analytics_server/mhq/store/models/code/workflows/enums.py b/backend/analytics_server/mhq/store/models/code/workflows/enums.py index cfcbd758..82588e05 100644 --- a/backend/analytics_server/mhq/store/models/code/workflows/enums.py +++ b/backend/analytics_server/mhq/store/models/code/workflows/enums.py @@ -4,6 +4,9 @@ class RepoWorkflowProviders(Enum): GITHUB_ACTIONS = "github" CIRCLE_CI = "circle_ci" + # CLUSTOX: Jenkins as a deployment source. Persisted as varchar, so no + # migration is required despite the ENUM() wrapper on the column. + JENKINS = "jenkins" @classmethod def get_workflow_providers(cls): diff --git a/backend/analytics_server/mhq/store/models/integrations/enums.py b/backend/analytics_server/mhq/store/models/integrations/enums.py index 423bde13..c5dbfc1b 100644 --- a/backend/analytics_server/mhq/store/models/integrations/enums.py +++ b/backend/analytics_server/mhq/store/models/integrations/enums.py @@ -4,6 +4,8 @@ class UserIdentityProvider(Enum): GITHUB = "github" GITLAB = "gitlab" + # CLUSTOX: Jenkins credentials live in Integration like any other provider. + JENKINS = "jenkins" @classmethod def get_enum(self, provider: str): diff --git a/backend/analytics_server/tests/service/workflows/sync/test_etl_workflows_factory.py b/backend/analytics_server/tests/service/workflows/sync/test_etl_workflows_factory.py new file mode 100644 index 00000000..2ddaf24d --- /dev/null +++ b/backend/analytics_server/tests/service/workflows/sync/test_etl_workflows_factory.py @@ -0,0 +1,14 @@ +import pytest + +from mhq.service.workflows.sync.etl_workflows_factory import WorkflowETLFactory +from mhq.store.models.code import RepoWorkflowProviders + + +def test_jenkins_is_a_known_provider(): + assert RepoWorkflowProviders.JENKINS.value == "jenkins" + + +def test_factory_rejects_unknown_provider(): + factory = WorkflowETLFactory("org-id") + with pytest.raises(NotImplementedError): + factory("NOT_A_PROVIDER") From 85b3c18921354a2ee09814fbf66052e1e61579ec Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Mon, 10 Aug 2026 15:28:59 +0500 Subject: [PATCH 042/194] feat(jenkins): add REST client with bounded timeouts --- backend/analytics_server/mhq/exapi/jenkins.py | 83 +++++++++++++++++++ .../tests/exapi/test_jenkins.py | 41 +++++++++ 2 files changed, 124 insertions(+) create mode 100644 backend/analytics_server/mhq/exapi/jenkins.py create mode 100644 backend/analytics_server/tests/exapi/test_jenkins.py diff --git a/backend/analytics_server/mhq/exapi/jenkins.py b/backend/analytics_server/mhq/exapi/jenkins.py new file mode 100644 index 00000000..9feeddbc --- /dev/null +++ b/backend/analytics_server/mhq/exapi/jenkins.py @@ -0,0 +1,83 @@ +# CLUSTOX: Jenkins REST client. Knows Jenkins' API shape and nothing about our +# models -- adaptation to RepoWorkflowRuns lives in the ETL handler. +from datetime import datetime +from typing import Dict, List, Tuple + +import pytz +import requests + +# Connect and read timeouts. The workspace sync loop is sequential, so an +# untimed request against a hung Jenkins stalls every workspace behind it. +DEFAULT_TIMEOUT: Tuple[int, int] = (5, 30) + +# Fetched per build. Explicit rather than a wildcard so a Jenkins with many +# plugins does not return megabytes of action data per build. +BUILD_TREE = ( + "builds[number,result,timestamp,duration,url,building," + "actions[causes[userId,userName,shortDescription]," + "lastBuiltRevision[SHA1,branch[name]]]]" +) + +JOB_TREE = "jobs[name,fullName,url]" + + +def job_path(job_full_name: str) -> str: + """Jenkins addresses nested jobs by repeating /job/ for each segment.""" + return "/".join(f"job/{segment}" for segment in job_full_name.split("/")) + + +class JenkinsApiService: + def __init__( + self, + base_url: str, + username: str, + api_token: str, + timeout: Tuple[int, int] = DEFAULT_TIMEOUT, + ): + self._base_url = base_url.rstrip("/") + self._auth = (username, api_token) + self._timeout = timeout + + def _get(self, path: str): + # TLS verification is deliberately not configurable. A skip-verify + # option is easy to add, hard to remove, and this connection carries an + # API token. + return requests.get( + f"{self._base_url}/{path}", + auth=self._auth, + timeout=self._timeout, + verify=True, + ) + + def check_pat(self) -> bool: + response = self._get("api/json") + return response.status_code == 200 + + def get_jobs(self) -> List[Dict]: + response = self._get(f"api/json?tree={JOB_TREE}") + response.raise_for_status() + return [ + { + "name": job.get("name"), + "full_name": job.get("fullName") or job.get("name"), + "url": job.get("url"), + } + for job in response.json().get("jobs", []) + ] + + def get_builds(self, job_full_name: str, bookmark: datetime) -> List[Dict]: + response = self._get(f"{job_path(job_full_name)}/api/json?tree={BUILD_TREE}") + response.raise_for_status() + builds = response.json().get("builds", []) + return [ + build + for build in builds + if self._build_time(build) and self._build_time(build) > bookmark + ] + + @staticmethod + def _build_time(build: Dict): + timestamp = build.get("timestamp") + if not timestamp: + return None + return datetime.fromtimestamp(timestamp / 1000, tz=pytz.UTC) diff --git a/backend/analytics_server/tests/exapi/test_jenkins.py b/backend/analytics_server/tests/exapi/test_jenkins.py new file mode 100644 index 00000000..1e003987 --- /dev/null +++ b/backend/analytics_server/tests/exapi/test_jenkins.py @@ -0,0 +1,41 @@ +from datetime import datetime + +import pytz + +from mhq.exapi.jenkins import JenkinsApiService, job_path + + +def test_job_path_encodes_a_top_level_job(): + assert job_path("deploy-api") == "job/deploy-api" + + +def test_job_path_encodes_a_folder_job(): + # Jenkins addresses nested jobs by repeating /job/ per segment. + assert job_path("platform/deploy-api") == "job/platform/job/deploy-api" + + +def test_get_builds_filters_out_builds_at_or_before_the_bookmark(): + class FakeResponse: + status_code = 200 + + @staticmethod + def json(): + return { + "builds": [ + {"number": 3, "timestamp": 3000, "result": "SUCCESS"}, + {"number": 2, "timestamp": 2000, "result": "SUCCESS"}, + {"number": 1, "timestamp": 1000, "result": "SUCCESS"}, + ] + } + + @staticmethod + def raise_for_status(): + return None + + service = JenkinsApiService("https://jenkins.example.com", "user", "token") + service._get = lambda path: FakeResponse() + + bookmark = datetime.fromtimestamp(2, tz=pytz.UTC) + builds = service.get_builds("deploy-api", bookmark) + + assert [b["number"] for b in builds] == [3] From 071fe3fead1ac245e72a992bd72f06db433696a5 Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Mon, 10 Aug 2026 15:33:08 +0500 Subject: [PATCH 043/194] test(jenkins): add build fixtures including the no-git-plugin case --- .../tests/factories/models/exapi/jenkins.py | 35 +++++++++++++++++++ 1 file changed, 35 insertions(+) create mode 100644 backend/analytics_server/tests/factories/models/exapi/jenkins.py diff --git a/backend/analytics_server/tests/factories/models/exapi/jenkins.py b/backend/analytics_server/tests/factories/models/exapi/jenkins.py new file mode 100644 index 00000000..cc389288 --- /dev/null +++ b/backend/analytics_server/tests/factories/models/exapi/jenkins.py @@ -0,0 +1,35 @@ +# CLUSTOX: recorded from a real Jenkins pipeline build. Shape matters more than +# values -- these encode the assumptions the handler makes about the API. +from typing import Dict + + +def get_jenkins_build_dict( + number: int = 42, + result: str = "SUCCESS", + timestamp: int = 1754827200000, # 2025-08-10T12:00:00Z + duration: int = 125000, + url: str = "https://jenkins.example.com/job/deploy-api/42/", + building: bool = False, + user_id: str = "hamad", + branch_name: str = "origin/main", + sha: str = "a1b2c3d4e5f6", +) -> Dict: + return { + "number": number, + "result": result, + "timestamp": timestamp, + "duration": duration, + "url": url, + "building": building, + "actions": [ + {"causes": [{"userId": user_id, "shortDescription": "Started by user"}]}, + {"lastBuiltRevision": {"SHA1": sha, "branch": [{"name": branch_name}]}}, + ], + } + + +def get_jenkins_build_dict_without_git_plugin(**kwargs) -> Dict: + """A freestyle job with no SCM: no branch, no revision, no user cause.""" + build = get_jenkins_build_dict(**kwargs) + build["actions"] = [{}] + return build From 0880d5e2df19047806c784431fdf444edda0c255 Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Mon, 10 Aug 2026 15:38:20 +0500 Subject: [PATCH 044/194] feat(jenkins): map builds to workflow runs --- .../workflows/sync/etl_jenkins_handler.py | 143 ++++++++++++++++++ .../sync/test_etl_jenkins_handler.py | 129 ++++++++++++++++ 2 files changed, 272 insertions(+) create mode 100644 backend/analytics_server/mhq/service/workflows/sync/etl_jenkins_handler.py create mode 100644 backend/analytics_server/tests/service/workflows/sync/test_etl_jenkins_handler.py diff --git a/backend/analytics_server/mhq/service/workflows/sync/etl_jenkins_handler.py b/backend/analytics_server/mhq/service/workflows/sync/etl_jenkins_handler.py new file mode 100644 index 00000000..b95157e5 --- /dev/null +++ b/backend/analytics_server/mhq/service/workflows/sync/etl_jenkins_handler.py @@ -0,0 +1,143 @@ +# CLUSTOX: Jenkins deployment detection. Implements the same two-method +# contract as GitHub Actions, so nothing downstream of the sync changes. +from datetime import datetime +from typing import Dict, List, Optional, Tuple +from uuid import uuid4 + +import pytz + +from mhq.service.workflows.sync.etl_provider_handler import WorkflowProviderETLHandler +from mhq.store.models.code import ( + OrgRepo, + RepoWorkflow, + RepoWorkflowProviders, + RepoWorkflowRuns, + RepoWorkflowRunsStatus, +) +from mhq.utils.log import LOG +from mhq.utils.time import time_now + + +class JenkinsETLHandler(WorkflowProviderETLHandler): + def __init__(self, org_id: str, jenkins_api_service, workflow_repo_service): + self.org_id = org_id + self._api = jenkins_api_service + self._workflow_repo_service = workflow_repo_service + self._provider = RepoWorkflowProviders.JENKINS.value + + def check_pat_validity(self) -> bool: + if not self._api.check_pat(): + raise Exception("Jenkins credentials are invalid or Jenkins is unreachable") + return True + + def get_workflow_runs( + self, + org_repo: OrgRepo, + repo_workflow: RepoWorkflow, + bookmark: datetime, + ) -> Tuple[List[RepoWorkflowRuns], datetime]: + try: + builds = self._api.get_builds(repo_workflow.provider_workflow_id, bookmark) + except Exception as e: + # Bookmark is returned unchanged by raising: the caller does not + # advance it, so this window is re-fetched next cycle. + raise Exception( + f"[Jenkins Sync Repo Workflow Worker] Error fetching job " + f"{repo_workflow.provider_workflow_id} for repo " + f"{str(org_repo.repo_id)}: {str(e)}" + ) + + if not builds: + LOG.info( + f"[Jenkins Sync Repo Workflow Worker] No builds found for job " + f"{repo_workflow.provider_workflow_id}. Org: {self.org_id}" + ) + return [], bookmark + + runs = [] + for build in builds: + try: + runs.append( + self._adapt_jenkins_build_to_workflow_run( + str(repo_workflow.id), build + ) + ) + except Exception as e: + # One malformed build must not lose the rest of the batch. + LOG.warn( + f"[Jenkins Sync Repo Workflow Worker] Skipping build " + f"{build.get('number')}: {str(e)}" + ) + + return runs, self._get_new_bookmark_time_stamp(builds) + + def _get_new_bookmark_time_stamp(self, builds: List[Dict]) -> datetime: + """ + Rewind to the oldest still-running build so it is re-fetched once it + finishes. Mirrors the GitHub Actions handler. + """ + pending = [ + self._get_datetime_from_epoch_ms(build["timestamp"]) + for build in builds + if build.get("building") or build.get("result") is None + ] + return min(pending) if pending else time_now() + + def _adapt_jenkins_build_to_workflow_run( + self, repo_workflow_id: str, build: Dict + ) -> RepoWorkflowRuns: + existing = self._workflow_repo_service.get_repo_workflow_run_by_provider_workflow_run_id( + repo_workflow_id, str(build["number"]) + ) + run_id = existing.id if existing else uuid4() + + duration = build.get("duration") + return RepoWorkflowRuns( + id=run_id, + repo_workflow_id=repo_workflow_id, + provider_workflow_run_id=str(build["number"]), + event_actor=self._get_actor(build), + head_branch=self._get_branch(build), + status=self._get_repo_workflow_status(build), + created_at=time_now(), + updated_at=time_now(), + conducted_at=self._get_datetime_from_epoch_ms(build["timestamp"]), + duration=int(duration / 1000) if duration else None, + meta=build, + html_url=build.get("url"), + ) + + @staticmethod + def _get_repo_workflow_status(build: Dict) -> RepoWorkflowRunsStatus: + result = build.get("result") + if build.get("building") or result is None: + return RepoWorkflowRunsStatus.PENDING + if result == "SUCCESS": + return RepoWorkflowRunsStatus.SUCCESS + if result == "ABORTED": + return RepoWorkflowRunsStatus.CANCELLED + # FAILURE and UNSTABLE. UNSTABLE means the build finished but something + # -- usually tests -- failed; counting it as a deployment would report a + # red build as a successful ship. + return RepoWorkflowRunsStatus.FAILURE + + @staticmethod + def _get_datetime_from_epoch_ms(ms: int) -> datetime: + return datetime.fromtimestamp(ms / 1000, tz=pytz.UTC) + + @staticmethod + def _get_branch(build: Dict) -> Optional[str]: + # Contributed by the git plugin. A freestyle job with no SCM has none. + for action in build.get("actions", []): + revision = (action or {}).get("lastBuiltRevision") + if revision and revision.get("branch"): + return revision["branch"][0].get("name") + return None + + @staticmethod + def _get_actor(build: Dict) -> Optional[str]: + for action in build.get("actions", []): + causes = (action or {}).get("causes") + if causes: + return causes[0].get("userId") or causes[0].get("userName") + return None diff --git a/backend/analytics_server/tests/service/workflows/sync/test_etl_jenkins_handler.py b/backend/analytics_server/tests/service/workflows/sync/test_etl_jenkins_handler.py new file mode 100644 index 00000000..07052ad7 --- /dev/null +++ b/backend/analytics_server/tests/service/workflows/sync/test_etl_jenkins_handler.py @@ -0,0 +1,129 @@ +from datetime import datetime + +import pytz + +from mhq.service.workflows.sync.etl_jenkins_handler import JenkinsETLHandler +from mhq.store.models.code import RepoWorkflowRunsStatus +from mhq.utils.string import uuid4_str +from tests.factories.models.exapi.jenkins import ( + get_jenkins_build_dict, + get_jenkins_build_dict_without_git_plugin, +) + + +def _epoch_start() -> datetime: + return datetime.fromtimestamp(0, tz=pytz.UTC) + + +class FakeWorkflowRepoService: + def get_repo_workflow_run_by_provider_workflow_run_id(self, *args): + return None + + +def _handler(): + return JenkinsETLHandler(uuid4_str(), None, FakeWorkflowRepoService()) + + +def test_success_maps_to_success(): + build = get_jenkins_build_dict(result="SUCCESS") + assert ( + JenkinsETLHandler._get_repo_workflow_status(build) + == RepoWorkflowRunsStatus.SUCCESS + ) + + +def test_failure_maps_to_failure(): + build = get_jenkins_build_dict(result="FAILURE") + assert ( + JenkinsETLHandler._get_repo_workflow_status(build) + == RepoWorkflowRunsStatus.FAILURE + ) + + +def test_aborted_maps_to_cancelled(): + build = get_jenkins_build_dict(result="ABORTED") + assert ( + JenkinsETLHandler._get_repo_workflow_status(build) + == RepoWorkflowRunsStatus.CANCELLED + ) + + +def test_unstable_maps_to_failure_so_it_is_not_counted_as_a_deployment(): + build = get_jenkins_build_dict(result="UNSTABLE") + assert ( + JenkinsETLHandler._get_repo_workflow_status(build) + == RepoWorkflowRunsStatus.FAILURE + ) + + +def test_a_running_build_maps_to_pending(): + build = get_jenkins_build_dict(result=None, building=True) + assert ( + JenkinsETLHandler._get_repo_workflow_status(build) + == RepoWorkflowRunsStatus.PENDING + ) + + +def test_adapt_maps_every_field(): + build = get_jenkins_build_dict() + repo_workflow_id = uuid4_str() + + run = _handler()._adapt_jenkins_build_to_workflow_run(repo_workflow_id, build) + + assert run.repo_workflow_id == repo_workflow_id + assert run.provider_workflow_run_id == "42" + assert run.status == RepoWorkflowRunsStatus.SUCCESS + assert run.event_actor == "hamad" + assert run.head_branch == "origin/main" + assert run.html_url == build["url"] + # Jenkins reports milliseconds; RepoWorkflowRuns.duration is seconds. + assert run.duration == 125 + assert run.conducted_at.isoformat() == "2025-08-10T12:00:00+00:00" + + +def test_a_build_without_the_git_plugin_records_with_nulls_rather_than_raising(): + build = get_jenkins_build_dict_without_git_plugin() + + run = _handler()._adapt_jenkins_build_to_workflow_run(uuid4_str(), build) + + assert run.head_branch is None + assert run.event_actor is None + assert run.status == RepoWorkflowRunsStatus.SUCCESS + + +def test_a_malformed_build_is_skipped_without_losing_the_batch(): + class Repo: + def get_repo_workflow_run_by_provider_workflow_run_id(self, *args): + return None + + class Api: + @staticmethod + def get_builds(*args): + # Second build has no "number", so adaptation raises for it alone. + return [get_jenkins_build_dict(number=2), {"timestamp": 1754827200000}] + + class Workflow: + id = uuid4_str() + provider_workflow_id = "deploy-api" + + class OrgRepoStub: + repo_id = uuid4_str() + + handler = JenkinsETLHandler(uuid4_str(), Api(), Repo()) + runs, _ = handler.get_workflow_runs(OrgRepoStub(), Workflow(), _epoch_start()) + + assert [r.provider_workflow_run_id for r in runs] == ["2"] + + +def test_bookmark_rewinds_to_the_oldest_running_build(): + handler = _handler() + builds = [ + get_jenkins_build_dict(number=3, timestamp=3000, result=None, building=True), + get_jenkins_build_dict(number=2, timestamp=2000, result="SUCCESS"), + ] + + bookmark = handler._get_new_bookmark_time_stamp(builds) + + # The running build must be re-fetched next cycle, so the bookmark cannot + # advance past it. + assert bookmark == handler._get_datetime_from_epoch_ms(3000) From 3b021115b69d13eb3bc3343e78a3c3ff27d7edf9 Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Mon, 10 Aug 2026 15:42:48 +0500 Subject: [PATCH 045/194] docs: correct task 4 test count from 10 to 9 The prose said 10; the task's own code block defines 9. Caught by the implementer, who transcribed the block verbatim and flagged the mismatch rather than inventing a tenth test to reach the stated number. --- docs/JENKINS_INTEGRATION_PLAN.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/JENKINS_INTEGRATION_PLAN.md b/docs/JENKINS_INTEGRATION_PLAN.md index f029c0bf..1528669a 100644 --- a/docs/JENKINS_INTEGRATION_PLAN.md +++ b/docs/JENKINS_INTEGRATION_PLAN.md @@ -681,7 +681,7 @@ class JenkinsETLHandler(WorkflowProviderETLHandler): - [ ] **Step 4: Run tests to verify they pass** Run: `cd backend/analytics_server && python -m pytest tests/service/workflows/sync/test_etl_jenkins_handler.py -v` -Expected: PASS (10 tests) +Expected: PASS (9 tests) - [ ] **Step 5: Format and lint** From c7b8ce3e635e9024def8481621d50cb5e79d5eae Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Mon, 10 Aug 2026 15:46:58 +0500 Subject: [PATCH 046/194] feat(jenkins): resolve credentials and register the handler Wires JenkinsApiService and JenkinsETLHandler into the sync: a get_jenkins_config() reader for the provider_meta base_url/username, a get_jenkins_etl_handler() factory that never raises for an unconfigured org, and dispatch for RepoWorkflowProviders.JENKINS in WorkflowETLFactory. --- .../workflows/sync/etl_jenkins_handler.py | 25 +++++++++++++++++++ .../workflows/sync/etl_workflows_factory.py | 5 ++++ backend/analytics_server/mhq/utils/jenkins.py | 20 +++++++++++++++ .../sync/test_etl_workflows_factory.py | 23 +++++++++++++++++ 4 files changed, 73 insertions(+) create mode 100644 backend/analytics_server/mhq/utils/jenkins.py diff --git a/backend/analytics_server/mhq/service/workflows/sync/etl_jenkins_handler.py b/backend/analytics_server/mhq/service/workflows/sync/etl_jenkins_handler.py index b95157e5..e952bc4d 100644 --- a/backend/analytics_server/mhq/service/workflows/sync/etl_jenkins_handler.py +++ b/backend/analytics_server/mhq/service/workflows/sync/etl_jenkins_handler.py @@ -141,3 +141,28 @@ def _get_actor(build: Dict) -> Optional[str]: if causes: return causes[0].get("userId") or causes[0].get("userName") return None + + +def get_jenkins_etl_handler(org_id: str) -> JenkinsETLHandler: + from mhq.exapi.jenkins import JenkinsApiService + from mhq.store.models import UserIdentityProvider + from mhq.store.repos.core import CoreRepoService + from mhq.store.repos.workflows import WorkflowRepoService + from mhq.utils.jenkins import get_jenkins_config + + api_token = CoreRepoService().get_access_token(org_id, UserIdentityProvider.JENKINS) + base_url, username = get_jenkins_config(org_id) + + if not (api_token and base_url and username): + LOG.error( + f"Jenkins is not fully configured for org {org_id}: " + f"base_url={'set' if base_url else 'missing'}, " + f"username={'set' if username else 'missing'}, " + f"token={'set' if api_token else 'missing'}" + ) + + return JenkinsETLHandler( + org_id, + JenkinsApiService(base_url or "", username or "", api_token or ""), + WorkflowRepoService(), + ) diff --git a/backend/analytics_server/mhq/service/workflows/sync/etl_workflows_factory.py b/backend/analytics_server/mhq/service/workflows/sync/etl_workflows_factory.py index ea05cc02..d207dc50 100644 --- a/backend/analytics_server/mhq/service/workflows/sync/etl_workflows_factory.py +++ b/backend/analytics_server/mhq/service/workflows/sync/etl_workflows_factory.py @@ -1,6 +1,9 @@ from mhq.service.workflows.sync.etl_github_actions_handler import ( get_github_actions_etl_handler, ) + +# CLUSTOX: Jenkins as a third deployment source. +from mhq.service.workflows.sync.etl_jenkins_handler import get_jenkins_etl_handler from mhq.service.workflows.sync.etl_provider_handler import WorkflowProviderETLHandler from mhq.store.models.code import RepoWorkflowProviders @@ -12,4 +15,6 @@ def __init__(self, org_id: str): def __call__(self, provider: str) -> WorkflowProviderETLHandler: if provider == RepoWorkflowProviders.GITHUB_ACTIONS.name: return get_github_actions_etl_handler(self.org_id) + if provider == RepoWorkflowProviders.JENKINS.name: + return get_jenkins_etl_handler(self.org_id) raise NotImplementedError(f"Unknown provider - {provider}") diff --git a/backend/analytics_server/mhq/utils/jenkins.py b/backend/analytics_server/mhq/utils/jenkins.py new file mode 100644 index 00000000..300fff69 --- /dev/null +++ b/backend/analytics_server/mhq/utils/jenkins.py @@ -0,0 +1,20 @@ +# CLUSTOX: Jenkins needs a base URL and username alongside its API token. +# The token is a secret and lives encrypted in access_token_enc_chunks; these +# two are not, and follow the existing provider_meta precedent used for +# GitHub's custom domain. +from typing import List, Optional, Tuple + +from mhq.store.models import Integration, UserIdentityProvider +from mhq.store.repos.core import CoreRepoService + + +def get_jenkins_config(org_id: str) -> Tuple[Optional[str], Optional[str]]: + core_repo_service = CoreRepoService() + integrations: List[Integration] = core_repo_service.get_org_integrations_for_names( + org_id, [UserIdentityProvider.JENKINS.value] + ) + if not integrations or not integrations[0].provider_meta: + return None, None + + meta = integrations[0].provider_meta + return meta.get("base_url"), meta.get("username") diff --git a/backend/analytics_server/tests/service/workflows/sync/test_etl_workflows_factory.py b/backend/analytics_server/tests/service/workflows/sync/test_etl_workflows_factory.py index 2ddaf24d..3ef0b516 100644 --- a/backend/analytics_server/tests/service/workflows/sync/test_etl_workflows_factory.py +++ b/backend/analytics_server/tests/service/workflows/sync/test_etl_workflows_factory.py @@ -1,7 +1,10 @@ +from unittest.mock import patch + import pytest from mhq.service.workflows.sync.etl_workflows_factory import WorkflowETLFactory from mhq.store.models.code import RepoWorkflowProviders +from mhq.store.repos.core import CoreRepoService def test_jenkins_is_a_known_provider(): @@ -12,3 +15,23 @@ def test_factory_rejects_unknown_provider(): factory = WorkflowETLFactory("org-id") with pytest.raises(NotImplementedError): factory("NOT_A_PROVIDER") + + +def test_factory_returns_a_jenkins_handler(): + from mhq.service.workflows.sync.etl_jenkins_handler import JenkinsETLHandler + + # CLUSTOX: get_jenkins_etl_handler resolves credentials through + # CoreRepoService, which needs a live Flask app context / DB session that + # this test suite does not provide (no conftest.py sets one up, and no + # other test in the suite calls CoreRepoService without mocking it either + # -- see tests/clustox_auth/test_sync_run.py). Mocking these two methods + # also exercises exactly the "unconfigured org" path described by the + # correctness requirement: the factory must still hand back a handler + # instead of raising when Jenkins isn't set up for this org. + with patch.object( + CoreRepoService, "get_access_token", return_value=None + ), patch.object(CoreRepoService, "get_org_integrations_for_names", return_value=[]): + factory = WorkflowETLFactory("org-id") + handler = factory(RepoWorkflowProviders.JENKINS.name) + + assert isinstance(handler, JenkinsETLHandler) From 69fe97c8db1a8b5a6c3d6359924482cf81b9f805 Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Mon, 10 Aug 2026 15:59:08 +0500 Subject: [PATCH 047/194] feat(jenkins): expose job listing and mapping routes --- .../analytics_server/mhq/api/integrations.py | 116 +++++++++++++++++- .../mhq/store/repos/workflows.py | 35 +++++- .../analytics_server/tests/api/__init__.py | 0 .../tests/api/test_jenkins_routes.py | 30 +++++ 4 files changed, 178 insertions(+), 3 deletions(-) create mode 100644 backend/analytics_server/tests/api/__init__.py create mode 100644 backend/analytics_server/tests/api/test_jenkins_routes.py diff --git a/backend/analytics_server/mhq/api/integrations.py b/backend/analytics_server/mhq/api/integrations.py index 35ac97d0..d783ecf7 100644 --- a/backend/analytics_server/mhq/api/integrations.py +++ b/backend/analytics_server/mhq/api/integrations.py @@ -1,13 +1,17 @@ from typing import List from flask import Blueprint, jsonify from github import GithubException -from voluptuous import Schema, Optional, Coerce, Range, All +from voluptuous import Schema, Optional, Required, Coerce, Range, All from mhq.exapi.models.gitlab import GitlabRepo -from mhq.api.request_utils import queryschema +from mhq.api.request_utils import dataschema, queryschema, uuid_validator from mhq.service.external_integrations_service import get_external_integrations_service from mhq.service.query_validator import get_query_validator from mhq.store.models import UserIdentityProvider +from mhq.store.models.code import RepoWorkflowProviders, RepoWorkflowType +from mhq.store.models.code.workflows.workflows import RepoWorkflow +from mhq.store.repos.code import CodeRepoService +from mhq.store.repos.workflows import WorkflowRepoService from mhq.utils.github import github_org_data_multi_thread_worker app = Blueprint("integrations", __name__) @@ -15,6 +19,22 @@ STATUS_TOO_MANY_REQUESTS = 429 +# CLUSTOX: one deployment source per repo. A repo tracked through both GitHub +# Actions and Jenkins would count every deploy twice, doubling Deployment +# Frequency with nothing visibly wrong. Deactivation is reversible: the rows +# survive, so removing the Jenkins mapping can restore them. +def deactivate_github_actions_workflows_for_repo(workflows: List) -> int: + deactivated = 0 + for workflow in workflows: + if ( + workflow.provider == RepoWorkflowProviders.GITHUB_ACTIONS + and workflow.is_active + ): + workflow.is_active = False + deactivated += 1 + return deactivated + + @app.route("/orgs//integrations/github/orgs", methods={"GET"}) def get_github_orgs(org_id: str): query_validator = get_query_validator() @@ -210,6 +230,98 @@ def get_gitlab_projects(org_id: str, group_id: str, page_size: int, page: int): ] +@app.route("/orgs//integrations/jenkins/jobs", methods={"GET"}) +def get_jenkins_jobs(org_id: str): + # CLUSTOX: Jenkins-specific imports stay local to the handler, mirroring + # the lazy-import precedent for provider modules in + # mhq/service/workflows/sync/etl_jenkins_handler.py. + from mhq.exapi.jenkins import JenkinsApiService + from mhq.store.repos.core import CoreRepoService + from mhq.utils.jenkins import get_jenkins_config + + query_validator = get_query_validator() + query_validator.org_validator(org_id) + + api_token = CoreRepoService().get_access_token(org_id, UserIdentityProvider.JENKINS) + base_url, username = get_jenkins_config(org_id) + if not (api_token and base_url and username): + return {"error": "Jenkins is not configured for this workspace"}, 400 + + return JenkinsApiService(base_url, username, api_token).get_jobs() + + +@app.route("/orgs//integrations/jenkins/mappings", methods={"POST"}) +@dataschema( + Schema( + { + Required("org_repo_id"): All(str, Coerce(uuid_validator)), + Required("job_full_name"): str, + } + ), +) +def create_jenkins_mapping(org_id: str, org_repo_id: str, job_full_name: str): + query_validator = get_query_validator() + query_validator.org_validator(org_id) + + org_repo = CodeRepoService().get_repo_by_id(org_repo_id) + if org_repo is None or str(org_repo.org_id) != str(org_id): + return jsonify({"error": f"Repo {org_repo_id} not found in org {org_id}"}), 404 + + workflow_repo_service = WorkflowRepoService() + existing_workflows = workflow_repo_service.get_repo_workflow_by_repo_ids( + [org_repo_id], RepoWorkflowType.DEPLOYMENT + ) + # Mutates existing_workflows in place, flipping GitHub Actions rows to + # inactive; the actual write happens together with the new Jenkins row + # below, in one commit. + deactivated_count = deactivate_github_actions_workflows_for_repo(existing_workflows) + + jenkins_workflow = RepoWorkflow( + org_repo_id=org_repo_id, + type=RepoWorkflowType.DEPLOYMENT, + provider=RepoWorkflowProviders.JENKINS, + provider_workflow_id=job_full_name, + is_active=True, + name=job_full_name, + ) + workflow_repo_service.create_jenkins_repo_workflow( + jenkins_workflow, existing_workflows + ) + + return {"ok": True, "deactivated_github_workflows": deactivated_count} + + +@app.route("/orgs//integrations/jenkins/mappings", methods={"DELETE"}) +@dataschema( + Schema( + { + Required("repo_workflow_id"): All(str, Coerce(uuid_validator)), + } + ), +) +def delete_jenkins_mapping(org_id: str, repo_workflow_id: str): + query_validator = get_query_validator() + query_validator.org_validator(org_id) + + workflow_repo_service = WorkflowRepoService() + repo_workflow = workflow_repo_service.get_repo_workflow_by_id(repo_workflow_id) + if repo_workflow is None: + return jsonify({"error": f"Workflow {repo_workflow_id} not found"}), 404 + + org_repo = CodeRepoService().get_repo_by_id(repo_workflow.org_repo_id) + if org_repo is None or str(org_repo.org_id) != str(org_id): + return ( + jsonify( + {"error": f"Workflow {repo_workflow_id} not found in org {org_id}"} + ), + 404, + ) + + workflow_repo_service.deactivate_repo_workflow(repo_workflow) + + return {"ok": True} + + @app.route("/orgs//integrations/gitlab/user/repos", methods={"GET"}) @queryschema( Schema( diff --git a/backend/analytics_server/mhq/store/repos/workflows.py b/backend/analytics_server/mhq/store/repos/workflows.py index a950a032..d9e4bb9e 100644 --- a/backend/analytics_server/mhq/store/repos/workflows.py +++ b/backend/analytics_server/mhq/store/repos/workflows.py @@ -1,5 +1,5 @@ from datetime import datetime -from typing import List, Tuple +from typing import List, Optional, Tuple from sqlalchemy.orm import defer from sqlalchemy import and_ @@ -118,6 +118,39 @@ def get_repo_workflow_by_repo_ids( .all() ) + @rollback_on_exc + def get_repo_workflow_by_id(self, repo_workflow_id: str) -> Optional[RepoWorkflow]: + return ( + self._db.session.query(RepoWorkflow) + .options(defer(RepoWorkflow.meta)) + .filter(RepoWorkflow.id == repo_workflow_id) + .one_or_none() + ) + + # CLUSTOX: creates the Jenkins mapping and deactivates the repo's active + # GitHub Actions workflows in a single commit. If either half failed on + # its own, a repo could be left with two active deployment sources, + # silently double-counting deployments -- see + # deactivate_github_actions_workflows_for_repo in mhq/api/integrations.py. + @rollback_on_exc + def create_jenkins_repo_workflow( + self, + jenkins_workflow: RepoWorkflow, + workflows_to_deactivate: List[RepoWorkflow], + ) -> RepoWorkflow: + self._db.session.add(jenkins_workflow) + for workflow in workflows_to_deactivate: + self._db.session.merge(workflow) + self._db.session.commit() + return jenkins_workflow + + @rollback_on_exc + def deactivate_repo_workflow(self, repo_workflow: RepoWorkflow) -> RepoWorkflow: + repo_workflow.is_active = False + self._db.session.merge(repo_workflow) + self._db.session.commit() + return repo_workflow + @rollback_on_exc def get_repo_workflows_by_repo_id(self, repo_id: str) -> List[RepoWorkflow]: return ( diff --git a/backend/analytics_server/tests/api/__init__.py b/backend/analytics_server/tests/api/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/backend/analytics_server/tests/api/test_jenkins_routes.py b/backend/analytics_server/tests/api/test_jenkins_routes.py new file mode 100644 index 00000000..a51d1a5f --- /dev/null +++ b/backend/analytics_server/tests/api/test_jenkins_routes.py @@ -0,0 +1,30 @@ +from mhq.api.integrations import deactivate_github_actions_workflows_for_repo +from mhq.store.models.code import RepoWorkflowProviders + + +class FakeWorkflow: + def __init__(self, provider, is_active=True): + self.provider = provider + self.is_active = is_active + + +def test_deactivates_only_github_actions_workflows(): + workflows = [ + FakeWorkflow(RepoWorkflowProviders.GITHUB_ACTIONS), + FakeWorkflow(RepoWorkflowProviders.GITHUB_ACTIONS), + FakeWorkflow(RepoWorkflowProviders.JENKINS), + ] + + count = deactivate_github_actions_workflows_for_repo(workflows) + + assert count == 2 + assert [w.is_active for w in workflows] == [False, False, True] + + +def test_deactivating_is_idempotent(): + workflows = [FakeWorkflow(RepoWorkflowProviders.GITHUB_ACTIONS, is_active=False)] + + count = deactivate_github_actions_workflows_for_repo(workflows) + + # Already inactive, so nothing changed and nothing is reported. + assert count == 0 From b773a348b6cb37649c5cc14063b33647c0b49b1d Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Mon, 10 Aug 2026 16:08:29 +0500 Subject: [PATCH 048/194] feat(jenkins): add job listing and mapping endpoints Co-Authored-By: Claude Opus 5 --- web-server/pages/api/clustox/jenkins/jobs.ts | 21 ++++++++++ .../pages/api/clustox/jenkins/mappings.ts | 40 +++++++++++++++++++ 2 files changed, 61 insertions(+) create mode 100644 web-server/pages/api/clustox/jenkins/jobs.ts create mode 100644 web-server/pages/api/clustox/jenkins/mappings.ts diff --git a/web-server/pages/api/clustox/jenkins/jobs.ts b/web-server/pages/api/clustox/jenkins/jobs.ts new file mode 100644 index 00000000..ec03c1b0 --- /dev/null +++ b/web-server/pages/api/clustox/jenkins/jobs.ts @@ -0,0 +1,21 @@ +import * as yup from 'yup'; + +import { Endpoint } from '@/api-helpers/global'; +import { internal } from '@/api-helpers/axios'; + +// CLUSTOX: proxies the workspace's Jenkins job list. org_id is validated by +// Endpoint.serve(), which asserts the caller may act on that workspace. +const getSchema = yup.object().shape({ + org_id: yup.string().uuid().required() +}); + +const endpoint = new Endpoint(getSchema); + +endpoint.handle.GET(getSchema, async (req, res) => { + const jobs = await internal.get( + `/orgs/${req.payload.org_id}/integrations/jenkins/jobs` + ); + res.send({ jobs: jobs.data }); +}); + +export default endpoint.serve(); diff --git a/web-server/pages/api/clustox/jenkins/mappings.ts b/web-server/pages/api/clustox/jenkins/mappings.ts new file mode 100644 index 00000000..766915f4 --- /dev/null +++ b/web-server/pages/api/clustox/jenkins/mappings.ts @@ -0,0 +1,40 @@ +import * as yup from 'yup'; + +import { Endpoint } from '@/api-helpers/global'; +import { internal } from '@/api-helpers/axios'; + +const postSchema = yup.object().shape({ + org_id: yup.string().uuid().required(), + org_repo_id: yup.string().uuid().required(), + job_full_name: yup.string().required() +}); + +const deleteSchema = yup.object().shape({ + org_id: yup.string().uuid().required(), + repo_workflow_id: yup.string().uuid().required() +}); + +const endpoint = new Endpoint(postSchema); + +endpoint.handle.POST(postSchema, async (req, res) => { + // One deployment source per repo: creating this mapping deactivates the + // repo's GitHub Actions workflows so deployments are not counted twice. + const result = await internal.post( + `/orgs/${req.payload.org_id}/integrations/jenkins/mappings`, + { + org_repo_id: req.payload.org_repo_id, + job_full_name: req.payload.job_full_name + } + ); + res.send(result.data); +}); + +endpoint.handle.DELETE(deleteSchema, async (req, res) => { + const result = await internal.delete( + `/orgs/${req.payload.org_id}/integrations/jenkins/mappings`, + { data: { repo_workflow_id: req.payload.repo_workflow_id } } + ); + res.send(result.data); +}); + +export default endpoint.serve(); From 20fe98cd4f481ae7bed6f31f191850a4b1803320 Mon Sep 17 00:00:00 2001 From: Batool Fatima Date: Mon, 10 Aug 2026 16:09:39 +0500 Subject: [PATCH 049/194] UI: add account menu and update user management UI --- web-server/pages/users.tsx | 640 ++++++++---------- .../src/components/AccountMenu/index.tsx | 114 ++++ .../src/components/ClustoxUserFooter.tsx | 50 +- web-server/src/components/PageHeader.tsx | 5 + web-server/src/components/RoleChip.tsx | 22 + web-server/src/components/UserAvatar.tsx | 59 ++ .../Sidebar/SidebarMenu/items.tsx | 4 +- 7 files changed, 511 insertions(+), 383 deletions(-) create mode 100644 web-server/src/components/AccountMenu/index.tsx create mode 100644 web-server/src/components/RoleChip.tsx create mode 100644 web-server/src/components/UserAvatar.tsx diff --git a/web-server/pages/users.tsx b/web-server/pages/users.tsx index 624217ae..59fd48f6 100644 --- a/web-server/pages/users.tsx +++ b/web-server/pages/users.tsx @@ -1,13 +1,12 @@ import { AdminPanelSettingsTwoTone, - LinkTwoTone, + LockTwoTone, MoreVertTwoTone, PersonAddAlt1TwoTone, WorkspacesTwoTone } from '@mui/icons-material'; import { Alert, - Avatar, Button, Chip, CircularProgress, @@ -28,8 +27,6 @@ import { TableHead, TableRow, TextField, - ToggleButton, - ToggleButtonGroup, Tooltip, Typography, useTheme @@ -40,7 +37,9 @@ import { FormEvent, useCallback, useEffect, useMemo, useState } from 'react'; import ExtendedSidebarLayout from 'src/layouts/ExtendedSidebarLayout'; import { FlexBox } from '@/components/FlexBox'; +import { RoleChip } from '@/components/RoleChip'; import { Line } from '@/components/Text'; +import { UserAvatar } from '@/components/UserAvatar'; import { PageWrapper } from '@/content/PullRequests/PageWrapper'; import { useAutofillSync } from '@/hooks/useAutofillSync'; import { PageLayout } from '@/types/resources'; @@ -56,26 +55,34 @@ type UserRow = { orgName: string | null; }; -const MIN_PASSWORD = 12; +type InviteRow = { + id: string; + email: string; + name: string; + role: ClustoxRole; + orgName: string | null; + expired: boolean; + emailed: boolean; +}; + +// CLUSTOX: users and pending invites rendered as one roster instead of a +// table plus a separate strip of chips above it -- an invite is just a +// person who hasn't finished signing up yet, not a different kind of thing. +type RosterRow = + | { kind: 'user'; key: string; name: string; email: string; role: ClustoxRole; orgName: string | null; user: UserRow } + | { kind: 'invite'; key: string; name: string; email: string; role: ClustoxRole; orgName: string | null; invite: InviteRow }; const NEW_WORKSPACE = '__new__'; const emptyForm = { name: '', email: '', - password: '', role: 'ADMIN' as ClustoxRole, // NEW_WORKSPACE provisions a fresh one; otherwise adopt an existing // workspace that has no owner. org_id: NEW_WORKSPACE }; -const initials = (name: string, email: string) => { - const source = name?.trim() || email; - const parts = source.split(/[\s.@]+/).filter(Boolean); - return (parts[0]?.[0] ?? '?').concat(parts[1]?.[0] ?? '').toUpperCase(); -}; - /** Identifiers read as data, not prose. */ const Mono = ({ children }: { children: React.ReactNode }) => ( ( ); -const RoleChip = ({ role }: { role: ClustoxRole }) => ( +const StatusChip = ({ status }: { status: 'active' | 'invited' | 'expired' }) => ( ); +const ROLE_OPTIONS: { + value: ClustoxRole; + label: string; + description: string; + icon: typeof WorkspacesTwoTone; +}[] = [ + { + value: 'ADMIN', + label: 'Admin', + description: + 'Gets their own workspace, connects their own GitHub or GitLab, and sees only their own projects.', + icon: WorkspacesTwoTone + }, + { + value: 'SUPERADMIN', + label: 'Superadmin', + description: + 'Sees every workspace, manages users and roles, and owns no workspace of their own.', + icon: AdminPanelSettingsTwoTone + } +]; + +/** Role picker as description cards, not a plain toggle-button pair. */ +const RolePicker = ({ + value, + onChange +}: { + value: ClustoxRole; + onChange: (role: ClustoxRole) => void; +}) => { + const theme = useTheme(); + + return ( + + {ROLE_OPTIONS.map((opt) => { + const Icon = opt.icon; + const selected = value === opt.value; + return ( + onChange(opt.value)} + sx={{ + cursor: 'pointer', + border: '1.5px solid', + borderColor: selected + ? theme.colors.primary.main + : theme.colors.alpha.trueWhite[30], + background: selected + ? theme.colors.alpha.black[10] + : 'transparent', + transition: 'border-color 0.15s' + }} + > + + + + {opt.label} + + + + {opt.description} + + + ); + })} + + ); +}; + function UsersPage() { const theme = useTheme(); const { enqueueSnackbar } = useSnackbar(); @@ -111,7 +194,6 @@ function UsersPage() { const [loading, setLoading] = useState(true); const [forbidden, setForbidden] = useState(false); - const [dialogOpen, setDialogOpen] = useState(false); const [form, setForm] = useState(emptyForm); const [busy, setBusy] = useState(false); const [formError, setFormError] = useState(''); @@ -119,24 +201,17 @@ function UsersPage() { const [inviteOpen, setInviteOpen] = useState(false); const [inviteLink, setInviteLink] = useState(''); const [inviteEmailed, setInviteEmailed] = useState(false); - const [invites, setInvites] = useState< - { id: string; email: string; name: string; role: ClustoxRole; orgName: string | null; expired: boolean; emailed: boolean }[] - >([]); + const [invites, setInvites] = useState([]); const [menuAnchor, setMenuAnchor] = useState(null); - const [menuUser, setMenuUser] = useState(null); + const [menuRow, setMenuRow] = useState(null); - // Shared between the Invite and Add user dialogs -- only one is ever - // open at a time, and they both edit the same `form` fields. const nameRef = useAutofillSync(form.name, (name) => setForm((f) => ({ ...f, name })) ); const emailRef = useAutofillSync(form.email, (email) => setForm((f) => ({ ...f, email })) ); - const passwordRef = useAutofillSync(form.password, (password) => - setForm((f) => ({ ...f, password })) - ); const loadUsers = useCallback(async () => { const res = await fetch('/api/clustox/users'); @@ -165,61 +240,40 @@ function UsersPage() { loadInvites(); }, [loadUsers, loadWorkspaces, loadInvites]); + const roster: RosterRow[] = useMemo( + () => [ + ...users.map((u): RosterRow => ({ + kind: 'user', + key: u.userId, + name: u.name, + email: u.email, + role: u.role, + orgName: u.orgName, + user: u + })), + ...invites.map((i): RosterRow => ({ + kind: 'invite', + key: i.id, + name: i.name, + email: i.email, + role: i.role, + orgName: i.orgName, + invite: i + })) + ], + [users, invites] + ); + const counts = useMemo( () => ({ total: users.length, admins: users.filter((u) => u.role === 'ADMIN').length, - workspaces: new Set(users.map((u) => u.orgId).filter(Boolean)).size + workspaces: new Set(users.map((u) => u.orgId).filter(Boolean)).size, + pending: invites.length }), - [users] + [users, invites] ); - const passwordTooShort = - form.password.length > 0 && form.password.length < MIN_PASSWORD; - - const onCreate = async (e: FormEvent) => { - e.preventDefault(); - setBusy(true); - setFormError(''); - - const res = await fetch('/api/clustox/users', { - method: 'POST', - headers: { 'Content-Type': 'application/json' }, - body: JSON.stringify({ - ...form, - team_ids: [], - org_id: form.org_id === NEW_WORKSPACE ? null : form.org_id - }) - }); - - setBusy(false); - - if (!res.ok) { - setFormError( - res.status === 409 - ? 'A user with that email already exists.' - : `Could not create user. Password must be at least ${MIN_PASSWORD} characters.` - ); - return; - } - - const created = await res.json(); - setDialogOpen(false); - setForm(emptyForm); - await Promise.all([loadUsers(), loadWorkspaces()]); - - const adopted = workspaces.find((w) => w.id === form.org_id); - enqueueSnackbar( - form.role !== 'ADMIN' - ? `${form.name} added as a superadmin` - : adopted - ? `${form.name} now owns the “${adopted.name}” workspace` - : `${form.name} added — a new workspace was created for them`, - { variant: 'success', autoHideDuration: 5000 } - ); - return created; - }; - const createInvite = async (e: FormEvent) => { e.preventDefault(); setBusy(true); @@ -256,11 +310,39 @@ function UsersPage() { }; const revokeInvite = async (id: string) => { + setMenuAnchor(null); await fetch(`/api/clustox/invites/${id}`, { method: 'DELETE' }); await loadInvites(); enqueueSnackbar('Invitation revoked', { variant: 'success' }); }; + const resendInvite = async (invite: InviteRow) => { + setMenuAnchor(null); + // No dedicated resend endpoint -- revoke the old one and issue a fresh + // link with the same details, which is exactly what "resend" means + // given only a hash of the link is ever stored server-side. + await fetch(`/api/clustox/invites/${invite.id}`, { method: 'DELETE' }); + const res = await fetch('/api/clustox/invites', { + method: 'POST', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify({ + name: invite.name, + email: invite.email, + role: invite.role + }) + }); + await loadInvites(); + if (!res.ok) { + enqueueSnackbar('Could not resend the invitation.', { variant: 'error' }); + return; + } + const { emailed } = await res.json(); + enqueueSnackbar( + emailed ? `Invite resent to ${invite.email}` : `New invite link created for ${invite.email}, but it couldn't be emailed`, + { variant: emailed ? 'success' : 'warning' } + ); + }; + const changeRole = async (user: UserRow, role: ClustoxRole) => { setMenuAnchor(null); const res = await fetch(`/api/clustox/users/${user.userId}`, { @@ -294,11 +376,28 @@ function UsersPage() { if (forbidden) return ( - - - Only superadmins can manage users. If you need access, ask a - superadmin to change your role. - + + + + + + You don't have access to this page + + + Only Superadmins can manage users and roles. Ask your Superadmin if + someone needs to be added or removed. + ); @@ -311,70 +410,32 @@ function UsersPage() { {/* Summary + primary action */} - - - - {counts.total} {counts.total === 1 ? 'user' : 'users'} - - - {counts.admins} admin{counts.admins === 1 ? '' : 's'} across{' '} - {counts.workspaces} workspace - {counts.workspaces === 1 ? '' : 's'} - - - - - - - - - - - {invites.length > 0 && ( - + + + {counts.total} {counts.total === 1 ? 'user' : 'users'} + - Pending invitations + {counts.admins} admin{counts.admins === 1 ? '' : 's'} across{' '} + {counts.workspaces} workspace{counts.workspaces === 1 ? '' : 's'} + {counts.pending > 0 && + ` · ${counts.pending} pending invite${counts.pending === 1 ? '' : 's'}`} - - {invites.map((i) => ( - revokeInvite(i.id)} - label={`${i.email} · ${ - i.role === 'ADMIN' ? 'admin' : 'superadmin' - }${i.emailed ? '' : ' · not emailed'}${ - i.expired ? ' · expired' : '' - }`} - /> - ))} - - )} + + + {/* Roster */} User - Role - Workspace + Role + Status + Workspace - {users.map((u) => ( - + {roster.map((row) => ( + - - {initials(u.name, u.email)} - + - {u.name} + {row.name} - {u.email} + {row.email} - + + + + + - {u.role === 'SUPERADMIN' ? ( + {row.role === 'SUPERADMIN' ? ( - {u.orgName ?? '—'} + {row.orgName ?? '—'} )} @@ -453,7 +514,7 @@ function UsersPage() { { - setMenuUser(u); + setMenuRow(row); setMenuAnchor(e.currentTarget); }} > @@ -463,13 +524,13 @@ function UsersPage() { ))} - {users.length === 0 && ( + {roster.length === 0 && ( - + No users yet. - Add an admin to create their first workspace. + Invite an admin to create their first workspace. @@ -486,27 +547,47 @@ function UsersPage() { open={Boolean(menuAnchor)} onClose={() => setMenuAnchor(null)} > - {menuUser?.role === 'ADMIN' ? ( - changeRole(menuUser, 'SUPERADMIN')}> + {menuRow?.kind === 'user' && + (menuRow.user.role === 'ADMIN' ? ( + changeRole(menuRow.user, 'SUPERADMIN')}> + + + + + + ) : ( + changeRole(menuRow.user, 'ADMIN')}> + + + + + + ))} + + {menuRow?.kind === 'invite' && [ + resendInvite(menuRow.invite)}> - + - - - ) : ( - menuUser && changeRole(menuUser, 'ADMIN')}> + + , + revokeInvite(menuRow.invite.id)} + sx={{ color: 'error.main' }} + > - + - + - )} + ]} {/* Invite */} @@ -530,9 +611,10 @@ function UsersPage() { ) : ( - Couldn't email this one (SMTP isn't set up, or the send - failed) — copy the link below and send it yourself. Only its - fingerprint is stored, so it cannot be shown again if lost. + Couldn't email this one (SMTP isn't set up, or the + send failed) — copy the link below and send it yourself. + Only its fingerprint is stored, so it cannot be shown again + if lost. )} ) : (
- Invite someone + Invite a teammate - v && setForm({ ...form, role: v })} - > - Admin - Superadmin - - - - They set their own password, so you never have to choose or - send one. - - setForm({ ...form, name: e.target.value })} - inputRef={nameRef} + onChange={(e) => setForm({ ...form, email: e.target.value })} + inputRef={emailRef} InputLabelProps={{ shrink: true }} /> setForm({ ...form, email: e.target.value })} - inputRef={emailRef} + onChange={(e) => setForm({ ...form, name: e.target.value })} + inputRef={nameRef} InputLabelProps={{ shrink: true }} /> + + + Role + + setForm({ ...form, role })} + /> + + {form.role === 'ADMIN' && ( - setForm({ ...form, org_id: e.target.value }) + onChange={(e) => setForm({ ...form, org_id: e.target.value })} + helperText={ + form.org_id === NEW_WORKSPACE + ? 'A new workspace is created and named after them' + : 'They take ownership of this existing workspace, keeping its integration and repositories' } > @@ -639,139 +718,12 @@ function UsersPage() { Cancel )}
- - {/* Add user */} - !busy && setDialogOpen(false)} - maxWidth="sm" - fullWidth - > -
- Add a user - - - - v && setForm({ ...form, role: v })} - > - Admin - Superadmin - - - - ) : ( - - ) - } - > - {form.role === 'ADMIN' - ? 'Gets their own workspace, connects their own GitHub or GitLab, and sees only their own projects.' - : 'Sees every workspace, manages users, and owns no workspace of their own.'} - - - setForm({ ...form, name: e.target.value })} - helperText={ - form.role === 'ADMIN' && form.org_id === NEW_WORKSPACE - ? 'Also names their workspace' - : undefined - } - inputRef={nameRef} - InputLabelProps={{ shrink: true }} - /> - - {form.role === 'ADMIN' && ( - - setForm({ ...form, org_id: e.target.value }) - } - helperText={ - form.org_id === NEW_WORKSPACE - ? 'A new workspace is created and named after them' - : 'They take ownership of this existing workspace, keeping its integration and repositories' - } - > - - Create a new workspace - - {workspaces - .filter((w) => !w.owned) - .map((w) => ( - - Adopt “{w.name}” (currently unowned) - - ))} - - )} - setForm({ ...form, email: e.target.value })} - inputRef={emailRef} - InputLabelProps={{ shrink: true }} - /> - setForm({ ...form, password: e.target.value })} - inputRef={passwordRef} - InputLabelProps={{ shrink: true }} - helperText={ - passwordTooShort - ? `${MIN_PASSWORD - form.password.length} more characters needed` - : `At least ${MIN_PASSWORD} characters. Share it with them directly.` - } - /> - - {formError && ( - - {formError} - - )} - - - - - - - - -
); } @@ -780,7 +732,7 @@ UsersPage.getLayout = (page: PageLayout) => ( + Users diff --git a/web-server/src/components/AccountMenu/index.tsx b/web-server/src/components/AccountMenu/index.tsx new file mode 100644 index 00000000..a74d1693 --- /dev/null +++ b/web-server/src/components/AccountMenu/index.tsx @@ -0,0 +1,114 @@ +import { + ExpandMoreRounded, + LogoutTwoTone, + ManageAccountsTwoTone +} from '@mui/icons-material'; +import { + Divider, + ListItemIcon, + ListItemText, + Menu, + MenuItem, + Typography, + useTheme +} from '@mui/material'; +import { signOut } from 'next-auth/react'; +import NextLink from 'next/link'; +import { FC, useState } from 'react'; + +import { FlexBox } from '@/components/FlexBox'; +import { RoleChip } from '@/components/RoleChip'; +import { UserAvatar } from '@/components/UserAvatar'; +import { ROUTES } from '@/constants/routes'; +import { useClustoxUser } from '@/hooks/useClustoxUser'; + +/** + * CLUSTOX: the account identity + sign-out surface, moved to the top-right + * of every page (rendered once, inside PageHeader) instead of living as a + * static block at the bottom of the sidebar. "Manage users" is included + * here as a shortcut for Superadmins, mirroring its (also role-gated) + * sidebar entry. + * + * No "Profile settings" entry: there's no per-user profile page in this + * app yet (the existing /settings page is org-level sync config, not + * personal account settings) -- adding a menu item that opens the wrong + * thing would be worse than not having one. + */ +export const AccountMenu: FC = () => { + const { user, loading, isSuperadmin } = useClustoxUser(); + const [anchor, setAnchor] = useState(null); + const theme = useTheme(); + + if (loading || !user) return null; + + return ( + <> + setAnchor(e.currentTarget)} + > + + + + {user.name} + + + + + + + setAnchor(null)} + anchorOrigin={{ vertical: 'bottom', horizontal: 'right' }} + transformOrigin={{ vertical: 'top', horizontal: 'right' }} + PaperProps={{ sx: { width: 250, mt: 1 } }} + > + + + {user.name} + + + {user.email} + + + + + {isSuperadmin && ( + + setAnchor(null)}> + + + + + + + )} + + signOut({ callbackUrl: '/login' })} + sx={{ color: 'error.main' }} + > + + + + + + + + ); +}; diff --git a/web-server/src/components/ClustoxUserFooter.tsx b/web-server/src/components/ClustoxUserFooter.tsx index d8c4cf2a..00304de0 100644 --- a/web-server/src/components/ClustoxUserFooter.tsx +++ b/web-server/src/components/ClustoxUserFooter.tsx @@ -1,52 +1,26 @@ -import { LogoutTwoTone } from '@mui/icons-material'; -import { Button, Chip, Tooltip, Typography } from '@mui/material'; -import { signOut } from 'next-auth/react'; - import { ClustoxWorkspaceSwitcher } from '@/components/ClustoxWorkspaceSwitcher'; import { FlexBox } from '@/components/FlexBox'; import { useClustoxUser } from '@/hooks/useClustoxUser'; /** - * CLUSTOX: shows who is signed in and provides the only way to sign out. - * Upstream has no logout affordance at all, since it had no login. + * CLUSTOX: the Superadmin's workspace switcher, at the bottom of the + * sidebar. + * + * Identity (name/role) and sign-out used to live here too, but that's now + * covered by the account menu at the top-right of every page (see + * PageHeader -> AccountMenu) -- kept only the piece that has no other home. */ export const ClustoxUserFooter = () => { - const { user, loading } = useClustoxUser(); + const { user, loading, isSuperadmin } = useClustoxUser(); - if (loading || !user) return null; + // ClustoxWorkspaceSwitcher itself renders null for a non-Superadmin, but + // checking here too avoids Sidebar rendering an empty padded box between + // its two surrounding Dividers for every Admin. + if (loading || !user || !isSuperadmin) return null; return ( - - {/* Only renders for a SuperAdmin, who owns no workspace of their own. */} + - - - - {user.name || user.email} - - - - - - - ); }; diff --git a/web-server/src/components/PageHeader.tsx b/web-server/src/components/PageHeader.tsx index d1d7a923..76849280 100644 --- a/web-server/src/components/PageHeader.tsx +++ b/web-server/src/components/PageHeader.tsx @@ -14,6 +14,7 @@ import { SidebarContext } from 'src/contexts/SidebarContext'; import { TeamSelectorModes } from '@/types/resources'; +import { AccountMenu } from './AccountMenu'; import { AvatarPageTitle } from './AvatarPageTitle'; import { BranchSelector } from './BranchSelector'; import { FlexBox, FlexBoxProps } from './FlexBox'; @@ -102,6 +103,10 @@ export const PageHeader: FC< {Boolean(subRoutes?.length) && } + {/* CLUSTOX: rendered once, here, so the account menu appears at + the top-right of every authenticated page without each page + needing to wire it in individually. */} + {showSelectorSection && ( <> diff --git a/web-server/src/components/RoleChip.tsx b/web-server/src/components/RoleChip.tsx new file mode 100644 index 00000000..19b8923c --- /dev/null +++ b/web-server/src/components/RoleChip.tsx @@ -0,0 +1,22 @@ +import { Chip } from '@mui/material'; +import { FC } from 'react'; + +import { ClustoxRole } from '@/auth/types'; + +/** + * CLUSTOX: shared role pill, used anywhere a Superadmin/Admin distinction is + * shown (Manage Users table, the account menu, the sidebar footer). Was + * previously reimplemented ad hoc in each of those places. + */ +export const RoleChip: FC<{ role: ClustoxRole; size?: 'small' | 'medium' }> = ({ + role, + size = 'small' +}) => ( + +); diff --git a/web-server/src/components/UserAvatar.tsx b/web-server/src/components/UserAvatar.tsx new file mode 100644 index 00000000..fad53f2c --- /dev/null +++ b/web-server/src/components/UserAvatar.tsx @@ -0,0 +1,59 @@ +import { Avatar, SxProps } from '@mui/material'; +import { FC } from 'react'; + +// CLUSTOX: a fixed small palette of gradient pairs, picked by a hash of the +// name/email so a given person always gets the same gradient across +// renders (and across the app) without persisting a color choice anywhere. +const GRADIENTS: [string, string][] = [ + ['#8C7CF0', '#5f4fd6'], + ['#57CA22', '#3a8a17'], + ['#FFA319', '#a06b0f'], + ['#26C6DA', '#0E7C8C'], + ['#FF6B82', '#C23854'], + ['#5C9DFF', '#2E5FBF'] +]; + +const hashToIndex = (source: string, length: number) => { + let hash = 0; + for (let i = 0; i < source.length; i++) { + hash = (hash << 5) - hash + source.charCodeAt(i); + hash |= 0; + } + return Math.abs(hash) % length; +}; + +export const initials = (name: string, email: string) => { + const source = name?.trim() || email || ''; + const parts = source.split(/[\s.@]+/).filter(Boolean); + return (parts[0]?.[0] ?? '?').concat(parts[1]?.[0] ?? '').toUpperCase(); +}; + +/** + * Shared avatar-with-gradient-initials, used wherever a person is shown + * (Manage Users table, the account menu). Replaces flat single-color + * avatars that were reimplemented per-page. + */ +export const UserAvatar: FC<{ + name: string; + email: string; + size?: number; + sx?: SxProps; +}> = ({ name, email, size = 34, sx }) => { + const [from, to] = GRADIENTS[hashToIndex(email || name || '', GRADIENTS.length)]; + + return ( + + {initials(name, email)} + + ); +}; diff --git a/web-server/src/layouts/ExtendedSidebarLayout/Sidebar/SidebarMenu/items.tsx b/web-server/src/layouts/ExtendedSidebarLayout/Sidebar/SidebarMenu/items.tsx index b5e7e0dd..66d571e2 100644 --- a/web-server/src/layouts/ExtendedSidebarLayout/Sidebar/SidebarMenu/items.tsx +++ b/web-server/src/layouts/ExtendedSidebarLayout/Sidebar/SidebarMenu/items.tsx @@ -89,7 +89,9 @@ const menuItems = (): MenuItems[] => [ { name: SideBarItems.MANAGE_USERS, icon: ManageAccounts, - link: ROUTES.USERS.PATH + link: ROUTES.USERS.PATH, + badge: 'NEW', + badgeTooltip: 'Invite teammates and manage roles' } ] } From c598cad6c28d608a1ea6b85993519e30edd43fd4 Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Mon, 10 Aug 2026 16:17:43 +0500 Subject: [PATCH 050/194] fix(jenkins): construct Endpoint with pathSchema, not postSchema DELETE requests were failing with 400 because Endpoint was validating against postSchema (which requires org_repo_id and job_full_name) before method dispatch. The path schema must only contain common fields (org_id). Per global.ts:98-100, pathSchema is validated unconditionally before the method-specific handler is selected, so DELETE requests with only org_id and repo_workflow_id were rejected before reaching the handler. Now follows the pattern from integration.ts: pathSchema has org_id only, postSchema and deleteSchema are passed to their respective handlers. Co-Authored-By: Claude Opus 5 --- web-server/pages/api/clustox/jenkins/mappings.ts | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/web-server/pages/api/clustox/jenkins/mappings.ts b/web-server/pages/api/clustox/jenkins/mappings.ts index 766915f4..7d738ee3 100644 --- a/web-server/pages/api/clustox/jenkins/mappings.ts +++ b/web-server/pages/api/clustox/jenkins/mappings.ts @@ -3,6 +3,10 @@ import * as yup from 'yup'; import { Endpoint } from '@/api-helpers/global'; import { internal } from '@/api-helpers/axios'; +const pathSchema = yup.object().shape({ + org_id: yup.string().uuid().required() +}); + const postSchema = yup.object().shape({ org_id: yup.string().uuid().required(), org_repo_id: yup.string().uuid().required(), @@ -14,7 +18,7 @@ const deleteSchema = yup.object().shape({ repo_workflow_id: yup.string().uuid().required() }); -const endpoint = new Endpoint(postSchema); +const endpoint = new Endpoint(pathSchema); endpoint.handle.POST(postSchema, async (req, res) => { // One deployment source per repo: creating this mapping deactivates the From e4441d2cc72ef8cd8b9366549251cfbd1c562cb1 Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Mon, 10 Aug 2026 16:24:42 +0500 Subject: [PATCH 051/194] test(jenkins): cross-workspace isolation for jenkins endpoints Adds a fifth test for DELETE /api/clustox/jenkins/mappings, covering the Critical defect found in review (endpoint was unreachable for every caller). Also swaps the nil-form UUID placeholders in the POST/DELETE payloads for schema-valid ones: yup's uuid() validator rejects version/variant nibbles of 0, so the original placeholders tripped payload validation (400) before assertWorkspaceAccess ever ran, passing the test for the wrong reason. Co-Authored-By: Claude Opus 5 --- web-server/e2e/jenkins.spec.ts | 120 +++++++++++++++++++++++++++++++++ 1 file changed, 120 insertions(+) create mode 100644 web-server/e2e/jenkins.spec.ts diff --git a/web-server/e2e/jenkins.spec.ts b/web-server/e2e/jenkins.spec.ts new file mode 100644 index 00000000..cb53240e --- /dev/null +++ b/web-server/e2e/jenkins.spec.ts @@ -0,0 +1,120 @@ +/** + * Jenkins endpoints must obey the same workspace boundary as everything else. + * + * Running: + * docker compose up -d + * cd web-server && yarn playwright test e2e/jenkins.spec.ts + */ +import { APIRequestContext, expect, request, test } from '@playwright/test'; + +const APP = 'http://localhost:3333'; + +const SUPERADMIN = { + email: process.env.SUPERADMIN_EMAIL || 'admin@clustox.com', + password: process.env.SUPERADMIN_PASSWORD || '' +}; + +const signIn = async (email: string, password: string) => { + const ctx = await request.newContext({ baseURL: APP }); + const csrf = await (await ctx.get('/api/auth/csrf')).json(); + const res = await ctx.post('/api/auth/callback/credentials', { + form: { csrfToken: csrf.csrfToken, email, password, json: 'true' }, + failOnStatusCode: false + }); + expect(res.status(), `sign-in failed for ${email}`).toBe(200); + return ctx; +}; + +const unique = (p: string) => + `${p}.${Date.now()}${Math.floor(Math.random() * 1000)}@clustox.com`; + +const createAdmin = async (su: APIRequestContext, name: string) => { + const email = unique(name.toLowerCase().replace(/\s+/g, '.')); + const password = 'E2eJenkinsPass123'; + const res = await su.post('/api/clustox/users', { + data: { name, email, password, role: 'ADMIN', team_ids: [] } + }); + expect(res.status()).toBe(200); + const { user_id, org_id } = await res.json(); + return { userId: user_id, orgId: org_id as string, email, password }; +}; + +test.describe('jenkins workspace isolation', () => { + test.skip(!SUPERADMIN.password, 'SUPERADMIN_PASSWORD not set'); + + let su: APIRequestContext; + let alpha: Awaited>; + let beta: Awaited>; + let alphaCtx: APIRequestContext; + + test.beforeAll(async () => { + su = await signIn(SUPERADMIN.email, SUPERADMIN.password); + alpha = await createAdmin(su, 'Jenkins Alpha'); + beta = await createAdmin(su, 'Jenkins Beta'); + alphaCtx = await signIn(alpha.email, alpha.password); + }); + + test.afterAll(async () => { + for (const id of [alpha.userId, beta.userId]) { + await su.fetch(`/api/clustox/users/${id}`, { + method: 'DELETE', + failOnStatusCode: false + }); + } + }); + + test('an admin cannot list another workspace jenkins jobs', async () => { + const res = await alphaCtx.get( + `/api/clustox/jenkins/jobs?org_id=${beta.orgId}`, + { failOnStatusCode: false } + ); + expect(res.status()).toBe(403); + }); + + test('an admin cannot map a job into another workspace', async () => { + // CLUSTOX: the id below must be a schema-valid (v4-shaped) UUID. Payload + // validation runs before assertWorkspaceAccess (see global.ts), so a + // malformed placeholder would 400 before the workspace check ever ran, + // making this test pass for the wrong reason. + const res = await alphaCtx.post('/api/clustox/jenkins/mappings', { + data: { + org_id: beta.orgId, + org_repo_id: '00000000-0000-4000-8000-000000000001', + job_full_name: 'deploy-api' + }, + failOnStatusCode: false + }); + expect(res.status()).toBe(403); + }); + + test('an admin cannot delete a mapping in another workspace', async () => { + // CLUSTOX: same reasoning as above -- repo_workflow_id must pass schema + // validation so the request actually reaches assertWorkspaceAccess. + const res = await alphaCtx.fetch('/api/clustox/jenkins/mappings', { + method: 'DELETE', + data: { + org_id: beta.orgId, + repo_workflow_id: '00000000-0000-4000-8000-000000000001' + }, + failOnStatusCode: false + }); + expect(res.status()).toBe(403); + }); + + test('unauthenticated requests are rejected', async ({ request: req }) => { + const res = await req.get( + `${APP}/api/clustox/jenkins/jobs?org_id=${alpha.orgId}`, + { failOnStatusCode: false, maxRedirects: 0 } + ); + expect(res.status()).toBe(401); + }); + + test('a malformed org_id is a 400, not a 403', async () => { + // Schema validation runs before the access check, so bad input is not + // reported as a permission problem. + const res = await alphaCtx.get('/api/clustox/jenkins/jobs?org_id=nonsense', { + failOnStatusCode: false + }); + expect(res.status()).toBe(400); + }); +}); From 64aa5a701e5c1e98d9684808dcab0c25e9fa2d26 Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Mon, 10 Aug 2026 16:28:27 +0500 Subject: [PATCH 052/194] docs: use an RFC-4122 valid placeholder UUID in task 8 yup's .uuid() enforces the version and variant nibbles, so the all-zero placeholder was rejected at schema validation. A test written against it would have returned 400 while asserting 403 -- passing for the wrong reason and proving nothing about workspace isolation. Caught by the implementer, who verified with yup directly rather than assuming the literal was fine. --- docs/JENKINS_INTEGRATION_PLAN.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/JENKINS_INTEGRATION_PLAN.md b/docs/JENKINS_INTEGRATION_PLAN.md index 1528669a..f87c88c6 100644 --- a/docs/JENKINS_INTEGRATION_PLAN.md +++ b/docs/JENKINS_INTEGRATION_PLAN.md @@ -1152,7 +1152,7 @@ test.describe('jenkins workspace isolation', () => { const res = await alphaCtx.post('/api/clustox/jenkins/mappings', { data: { org_id: beta.orgId, - org_repo_id: '00000000-0000-0000-0000-000000000001', + org_repo_id: '00000000-0000-4000-8000-000000000001', job_full_name: 'deploy-api' }, failOnStatusCode: false From 3849e6d0628431fcb7972b4c0cd3573481048c5f Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Mon, 10 Aug 2026 16:54:17 +0500 Subject: [PATCH 053/194] feat(jenkins): connect jenkins and map jobs to repositories Adds the Integrations UI for Jenkins: a credentials form that saves via the existing generic integration endpoint and validates the connection by probing the job list (there is no dry-run endpoint, so a failed probe rolls back the just-saved row rather than leaving the workspace "linked" with dead credentials), and a mapping table that lists the workspace's repos with a Jenkins job dropdown per row, POSTing to /api/clustox/jenkins/mappings and confirming first when the repo has an active deployment workflow that would otherwise be double-counted. The integrations.tsx card opens the setup form when unlinked and the mapping table when linked, matching the existing GitHub/GitLab card pattern via the same ModalContext. Jenkins is added only to the Integration enum, deliberately not to CODE_PROVIDER_INTEGRATIONS_MAP -- it is a deployment provider, not a code provider, and must not let a workspace pass the "has a code provider" onboarding gate on its own. That map alone isn't sufficient, though. ThirdPartyAuthContext derived `integrationList` and `activeCodeProvider` from *any* linked integration whose `integrated` flag was true, not specifically from linked code providers -- because before this change, the only rows the generic integration endpoint ever persisted were github/gitlab/bitbucket, so "any linked integration" and "any linked code provider" were accidentally the same set. Neither derivation ever consulted CODE_PROVIDER_INTEGRATIONS_MAP. Once Jenkins can be linked through that same endpoint, that assumption breaks: a workspace that linked only Jenkins would satisfy `integrationList.length > 0`, which gates the DORA metrics page, the teams page, the settings page, and sidebar nav visibility, and feeds `useTeamsConfig.tsx`'s repo search, which passes `integrationList` directly as the `providers` query param to an endpoint that expects git host names -- so 'jenkins' would leak into a call built for github/gitlab/bitbucket. In short, a workspace with zero code providers linked would look fully onboarded, and DORA metrics (which need a real code provider) would be gated open incorrectly. Fixed by filtering both derivations through CODE_PROVIDER_INTEGRATIONS_MAP via a new isCodeProvider() helper, so Jenkins (or any future deployment-only provider) can be linked without being mistaken for a code provider anywhere downstream. Verified this is a no-op for every workspace whose integrations are a subset of github/gitlab/bitbucket, i.e. every workspace that existed before Jenkins. libdefs/ambient.d.ts: widened IntegrationsMap's key union to add 'jenkins', since the backend now genuinely returns that key (getOrgIntegrations has no provider filter) and the Jenkins components need to read integrations.jenkins without an unsafe cast. Only the one new literal key was added; the closed key set, the per-key value shape, and every existing key's strictness are unchanged. Co-Authored-By: Claude Opus 5 --- web-server/libdefs/ambient.d.ts | 5 +- web-server/pages/integrations.tsx | 75 ++++- .../src/components/ClustoxJenkinsMapping.tsx | 292 ++++++++++++++++++ .../src/components/ClustoxJenkinsSetup.tsx | 203 ++++++++++++ web-server/src/constants/integrations.ts | 5 +- .../src/contexts/ThirdPartyAuthContext.tsx | 24 +- 6 files changed, 597 insertions(+), 7 deletions(-) create mode 100644 web-server/src/components/ClustoxJenkinsMapping.tsx create mode 100644 web-server/src/components/ClustoxJenkinsSetup.tsx diff --git a/web-server/libdefs/ambient.d.ts b/web-server/libdefs/ambient.d.ts index b32daf57..4ff6c84b 100644 --- a/web-server/libdefs/ambient.d.ts +++ b/web-server/libdefs/ambient.d.ts @@ -44,7 +44,10 @@ declare type IdentityMap = Record< declare type IntegrationsMap = Partial< Record< - 'github' | 'gitlab' | 'bitbucket', + // CLUSTOX: 'jenkins' added so the workspace's linked state is readable + // via useAuth().integrations.jenkins. It is a deployment provider, not a + // code provider -- see CODE_PROVIDER_INTEGRATIONS_MAP. + 'github' | 'gitlab' | 'bitbucket' | 'jenkins', { integrated: Boolean; linked_at: DateString | null; diff --git a/web-server/pages/integrations.tsx b/web-server/pages/integrations.tsx index 2ec60be9..92cb9f3c 100644 --- a/web-server/pages/integrations.tsx +++ b/web-server/pages/integrations.tsx @@ -1,12 +1,14 @@ -import { AddRounded, InfoRounded } from '@mui/icons-material'; +import { AddRounded, BuildRounded, InfoRounded } from '@mui/icons-material'; import { LoadingButton } from '@mui/lab'; -import { Button, Card, Divider } from '@mui/material'; +import { Button, Card, Divider, useTheme } from '@mui/material'; import { differenceInMilliseconds } from 'date-fns'; import { millisecondsInMinute } from 'date-fns/constants'; import { useEffect, useMemo } from 'react'; import { Authenticated } from 'src/components/Authenticated'; import { handleApi } from '@/api-helpers/axios-api-instance'; +import { ClustoxJenkinsMapping } from '@/components/ClustoxJenkinsMapping'; +import { ClustoxJenkinsSetup } from '@/components/ClustoxJenkinsSetup'; import { FlexBox } from '@/components/FlexBox'; import { Line } from '@/components/Text'; import { ROUTES } from '@/constants/routes'; @@ -14,6 +16,7 @@ import { FetchState } from '@/constants/ui-states'; import { GithubIntegrationCard } from '@/content/Dashboards/GithubIntegrationCard'; import { GitlabIntegrationCard } from '@/content/Dashboards/GitlabIntegrationCard'; import { PageWrapper } from '@/content/PullRequests/PageWrapper'; +import { useModal } from '@/contexts/ModalContext'; import { useAuth } from '@/hooks/useAuth'; import { useBoolState, useEasyState } from '@/hooks/useEasyState'; import ExtendedSidebarLayout from '@/layouts/ExtendedSidebarLayout'; @@ -163,6 +166,7 @@ const Content = () => { + {showCreationCTA && ( @@ -236,3 +240,70 @@ const Content = () => { ); }; + +/** + * CLUSTOX: Jenkins is a deployment provider set up through a credentials + * form rather than OAuth, so unlike GithubIntegrationCard/GitlabIntegrationCard + * it opens a modal instead of a redirect, and switches between the setup + * form and the job-mapping table based on link state. + */ +const JenkinsIntegrationCard = () => { + const theme = useTheme(); + const { integrations } = useAuth(); + const isJenkinsLinked = Boolean(integrations.jenkins?.integrated); + const { addModal, closeAllModals } = useModal(); + + const openSetup = () => + addModal({ + title: 'Connect Jenkins', + body: , + showCloseIcon: true + }); + + const openMapping = () => + addModal({ + title: 'Map Jenkins jobs to repositories', + body: , + showCloseIcon: true + }); + + return ( + + {isJenkinsLinked && ( + + + Linked + + + )} + + + + Jenkins + + + + Map deployments from Jenkins jobs + + + + + + ); +}; diff --git a/web-server/src/components/ClustoxJenkinsMapping.tsx b/web-server/src/components/ClustoxJenkinsMapping.tsx new file mode 100644 index 00000000..07b187d8 --- /dev/null +++ b/web-server/src/components/ClustoxJenkinsMapping.tsx @@ -0,0 +1,292 @@ +import { LoadingButton } from '@mui/lab'; +import { + Alert, + Button, + CircularProgress, + Dialog, + DialogActions, + DialogContent, + DialogTitle, + MenuItem, + Table, + TableBody, + TableCell, + TableContainer, + TableHead, + TableRow, + TextField, + useTheme +} from '@mui/material'; +import { useCallback, useEffect, useMemo, useState } from 'react'; + +import { handleApi } from '@/api-helpers/axios-api-instance'; +import { FlexBox } from '@/components/FlexBox'; +import { Line } from '@/components/Text'; +import { Integration } from '@/constants/integrations'; +import { useAuth } from '@/hooks/useAuth'; +// CLUSTOX: server-resolved workspace, consistent with ClustoxJenkinsSetup. +import { useClustoxUser } from '@/hooks/useClustoxUser'; + +type JenkinsJob = { name: string; full_name: string; url: string }; +type WorkspaceRepo = { id: string; name: string; provider: Integration }; +type Pending = { repoId: string; repoName: string; jobFullName: string }; + +/** + * CLUSTOX: there is no endpoint to list existing repo-to-job mappings, only + * to create (POST) or remove (DELETE, given a repo_workflow_id this UI never + * receives back) one. So this table's selections reflect what has been + * mapped in the current session, not necessarily what a prior session saved + * -- the mapping itself does persist server-side even though this screen + * cannot re-display it after a reload. + */ +export const ClustoxJenkinsMapping = () => { + const theme = useTheme(); + const { orgId: contextOrgId } = useAuth(); + const { orgId: sessionOrgId } = useClustoxUser(); + const orgId = sessionOrgId ?? contextOrgId; + + const [repos, setRepos] = useState([]); + const [jobs, setJobs] = useState([]); + // CLUSTOX: repos whose active workflow could be double-counted with a new + // Jenkins mapping. Sourced from teams/v2's repo_workflows, the only + // existing endpoint that already joins OrgRepo with active workflow rows -- + // it does not expose which provider the active workflow belongs to, so + // this is "has an active workflow-based deployment source" rather than + // strictly "has an active GitHub Actions workflow". + const [reposWithActiveWorkflow, setReposWithActiveWorkflow] = useState< + Set + >(new Set()); + const [selections, setSelections] = useState>({}); + const [rowErrors, setRowErrors] = useState>({}); + const [savingRepoId, setSavingRepoId] = useState(null); + const [pending, setPending] = useState(null); + const [loading, setLoading] = useState(true); + const [failed, setFailed] = useState(false); + + const load = useCallback(async () => { + if (!orgId) return; + setLoading(true); + setFailed(false); + try { + const codeProviders = [ + Integration.GITHUB, + Integration.GITLAB, + Integration.BITBUCKET + ]; + const [repoRows, jobsRes, teamsRes] = await Promise.all([ + handleApi(`/resources/orgs/${orgId}/repos`), + handleApi<{ jobs: JenkinsJob[] }>('/clustox/jenkins/jobs', { + params: { org_id: orgId } + }), + handleApi<{ + teamReposMap: Record< + string, + { id: string; repo_workflows?: unknown[] }[] + >; + }>(`/resources/orgs/${orgId}/teams/v2`, { + params: { providers: codeProviders } + }) + ]); + + setRepos(repoRows || []); + setJobs(jobsRes.jobs || []); + + const withWorkflow = new Set(); + Object.values(teamsRes.teamReposMap || {}).forEach((teamRepos) => { + teamRepos.forEach((repo) => { + if (repo.repo_workflows?.length) withWorkflow.add(repo.id); + }); + }); + setReposWithActiveWorkflow(withWorkflow); + } catch (e) { + console.error('Failed to load Jenkins mapping data', e); + setFailed(true); + } finally { + setLoading(false); + } + }, [orgId]); + + useEffect(() => { + load(); + }, [load]); + + const submitMapping = useCallback( + async (repoId: string, jobFullName: string) => { + setRowErrors((errs) => ({ ...errs, [repoId]: '' })); + setSavingRepoId(repoId); + try { + await handleApi('/clustox/jenkins/mappings', { + method: 'post', + data: { + org_id: orgId, + org_repo_id: repoId, + job_full_name: jobFullName + } + }); + setSelections((s) => ({ ...s, [repoId]: jobFullName })); + // CLUSTOX: the backend just deactivated this repo's GitHub Actions + // workflows as part of creating the mapping. + setReposWithActiveWorkflow((prev) => { + const next = new Set(prev); + next.delete(repoId); + return next; + }); + } catch (e) { + console.error('Failed to map Jenkins job', e); + setRowErrors((errs) => ({ + ...errs, + [repoId]: 'Could not save this mapping. Try again.' + })); + } finally { + setSavingRepoId(null); + } + }, + [orgId] + ); + + const handleSelect = useCallback( + (repo: WorkspaceRepo, jobFullName: string) => { + if (!jobFullName) return; + if (reposWithActiveWorkflow.has(repo.id)) { + setPending({ repoId: repo.id, repoName: repo.name, jobFullName }); + return; + } + submitMapping(repo.id, jobFullName); + }, + [reposWithActiveWorkflow, submitMapping] + ); + + const confirmPending = useCallback(async () => { + if (!pending) return; + await submitMapping(pending.repoId, pending.jobFullName); + setPending(null); + }, [pending, submitMapping]); + + const noJobs = useMemo(() => !loading && !jobs.length, [loading, jobs]); + + return ( + + {failed && ( + + Could not load repos or Jenkins jobs. Try again in a moment. + + )} + + {noJobs && !failed && ( + + Jenkins didn't return any jobs for this instance. + + )} + + {loading ? ( + + + + ) : ( + + + + + Repository + Jenkins job + + + + {repos.map((repo) => ( + + + + {repo.name} + + {repo.provider} + + + + + + + handleSelect(repo, e.target.value as string) + } + SelectProps={{ displayEmpty: true }} + > + + Select a Jenkins job + + {jobs.map((job) => ( + + {job.full_name} + + ))} + + {savingRepoId === repo.id && ( + + )} + {selections[repo.id] && + savingRepoId !== repo.id && + !rowErrors[repo.id] && ( + + Mapped + + )} + + {rowErrors[repo.id] && ( + + {rowErrors[repo.id]} + + )} + + + ))} + {!repos.length && ( + + + No repos in this workspace yet. + + + )} + +
+
+ )} + + setPending(null)}> + Map {pending?.repoName} to Jenkins? + + + Mapping a Jenkins job stops counting this repo's GitHub Actions + runs as deployments, so they aren't counted twice. You can undo + this by removing the mapping. + + + + + + Map job + + + +
+ ); +}; diff --git a/web-server/src/components/ClustoxJenkinsSetup.tsx b/web-server/src/components/ClustoxJenkinsSetup.tsx new file mode 100644 index 00000000..8c41b7d5 --- /dev/null +++ b/web-server/src/components/ClustoxJenkinsSetup.tsx @@ -0,0 +1,203 @@ +import { LoadingButton } from '@mui/lab'; +import { TextField } from '@mui/material'; +import { useSnackbar } from 'notistack'; +import { FC, useCallback } from 'react'; + +import { handleApi } from '@/api-helpers/axios-api-instance'; +import { FlexBox } from '@/components/FlexBox'; +import { Line } from '@/components/Text'; +import { Integration } from '@/constants/integrations'; +import { useAuth } from '@/hooks/useAuth'; +// CLUSTOX: server-resolved workspace, immune to stale persisted redux state +// (see ConfigureGitlabModalBody for the incident this pattern fixed). +import { useClustoxUser } from '@/hooks/useClustoxUser'; +import { useBoolState, useEasyState } from '@/hooks/useEasyState'; +import { fetchCurrentOrg } from '@/slices/auth'; +import { useDispatch } from '@/store'; +import { linkProvider, unlinkProvider } from '@/utils/auth'; +import { checkDomainWithRegex } from '@/utils/domainCheck'; +import { depFn } from '@/utils/fn'; + +const CONNECTION_ERROR = + 'Could not connect to Jenkins with these credentials. Check the base URL, username, and API token.'; + +/** + * CLUSTOX: credentials form for connecting a Jenkins instance. + * + * There is no dry-run endpoint -- the only server-side call that actually + * reaches Jenkins is the job list, which reads credentials out of the + * Integration row. So "validate before saving" is implemented as save, then + * probe, then roll back the row if the probe fails, rather than a true + * pre-save check. That keeps a failed attempt from leaving the workspace + * looking "linked" with credentials nothing can use. + */ +export const ClustoxJenkinsSetup: FC<{ onLinked?: () => void }> = ({ + onLinked +}) => { + const baseUrl = useEasyState(''); + const username = useEasyState(''); + const apiToken = useEasyState(''); + + const baseUrlError = useEasyState(''); + const usernameError = useEasyState(''); + const tokenError = useEasyState(''); + + const { orgId: contextOrgId } = useAuth(); + const { orgId: sessionOrgId } = useClustoxUser(); + const orgId = sessionOrgId ?? contextOrgId; + + const dispatch = useDispatch(); + const { enqueueSnackbar } = useSnackbar(); + const isLoading = useBoolState(false); + + const handleSubmit = useCallback(async () => { + depFn(baseUrlError.set, ''); + depFn(usernameError.set, ''); + depFn(tokenError.set, ''); + + let hasError = false; + if (!baseUrl.value || !checkDomainWithRegex(baseUrl.value)) { + depFn( + baseUrlError.set, + 'Enter a valid URL, including the http(s):// scheme' + ); + hasError = true; + } + if (!username.value) { + depFn(usernameError.set, 'Username is required'); + hasError = true; + } + if (!apiToken.value) { + depFn(tokenError.set, 'API token is required'); + hasError = true; + } + if (!orgId) { + depFn( + baseUrlError.set, + 'No workspace selected. Reload the page and try again.' + ); + hasError = true; + } + if (hasError) return; + + depFn(isLoading.true); + try { + await linkProvider(apiToken.value, orgId, Integration.JENKINS, { + base_url: baseUrl.value, + username: username.value + }); + + // CLUSTOX: the actual connection probe -- this is the first call that + // hits the real Jenkins instance with the credentials just saved. + await handleApi('/clustox/jenkins/jobs', { + params: { org_id: orgId } + }); + + await dispatch(fetchCurrentOrg()); + enqueueSnackbar('Jenkins connected successfully', { + variant: 'success', + autoHideDuration: 2000 + }); + onLinked?.(); + } catch (e) { + console.error('Failed to connect Jenkins', e); + // CLUSTOX: don't leave a workspace "linked" to credentials that don't + // work -- the mapping screen would otherwise open to a broken job list. + await unlinkProvider(orgId, Integration.JENKINS).catch(() => {}); + depFn(baseUrlError.set, CONNECTION_ERROR); + } finally { + depFn(isLoading.false); + } + }, [ + apiToken.value, + baseUrl.value, + baseUrlError.set, + dispatch, + enqueueSnackbar, + isLoading.false, + isLoading.true, + onLinked, + orgId, + tokenError.set, + username.value, + usernameError.set + ]); + + return ( + + + Enter your Jenkins connection details below{' '} + + * + + + + + { + baseUrl.set(e.currentTarget.value); + baseUrlError.set(''); + }} + label="Jenkins Base URL" + placeholder="https://jenkins.example.com" + /> + + {baseUrlError.value} + + + + + { + username.set(e.currentTarget.value); + usernameError.set(''); + }} + label="Username" + /> + + {usernameError.value} + + + + + { + apiToken.set(e.currentTarget.value); + tokenError.set(''); + }} + label="API Token" + type="password" + onKeyDown={(e) => { + if (e.key === 'Enter') { + e.preventDefault(); + handleSubmit(); + } + }} + /> + + {tokenError.value} + + + + + + Connect Jenkins + + + + ); +}; diff --git a/web-server/src/constants/integrations.ts b/web-server/src/constants/integrations.ts index c6002faa..e39fe2d0 100644 --- a/web-server/src/constants/integrations.ts +++ b/web-server/src/constants/integrations.ts @@ -9,7 +9,10 @@ export enum Integration { PAGERDUTY = 'pagerduty', OPSGENIE = 'opsgenie', MICROSOFT = 'azure-ad', - CIRCLECI = 'circle_ci' + CIRCLECI = 'circle_ci', + // CLUSTOX: Jenkins is a deployment provider, not a code provider -- it is + // deliberately absent from CODE_PROVIDER_INTEGRATIONS_MAP. + JENKINS = 'jenkins' } export enum CIProvider { diff --git a/web-server/src/contexts/ThirdPartyAuthContext.tsx b/web-server/src/contexts/ThirdPartyAuthContext.tsx index 276ab273..687fde5e 100644 --- a/web-server/src/contexts/ThirdPartyAuthContext.tsx +++ b/web-server/src/contexts/ThirdPartyAuthContext.tsx @@ -3,6 +3,7 @@ import { Session } from 'next-auth'; import { FC, createContext, useEffect, useCallback } from 'react'; import { useMemo } from 'react'; +import { CODE_PROVIDER_INTEGRATIONS_MAP } from '@/constants/api'; import { Integration } from '@/constants/integrations'; import { useBoolState, useEasyState } from '@/hooks/useEasyState'; import { useRefMounted } from '@/hooks/useRefMounted'; @@ -129,22 +130,39 @@ export const AuthProvider: FC = (props) => { [integrations.github, integrations.gitlab] ); + // CLUSTOX: consumers of integrationList/activeCodeProvider (DORA metrics + // gating, the team repo search, the sidebar's onboarding check) all assume + // every entry is a code host whose repos can be listed. `integrations` now + // also carries deployment-only providers like Jenkins, which are stored in + // the same table -- without this filter, linking only Jenkins would make a + // workspace with zero code providers look onboarded, and 'jenkins' would + // leak into API calls that expect a git host name. + const isCodeProvider = useCallback( + (name: string) => + Object.prototype.hasOwnProperty.call( + CODE_PROVIDER_INTEGRATIONS_MAP, + name + ), + [] + ); + const integrationList = useMemo( () => Object.entries(integrations) - .filter(([_, value]) => value.integrated) + .filter(([key, value]) => value.integrated && isCodeProvider(key)) .map(([key, _]) => key) as Integration[], - [integrations] + [integrations, isCodeProvider] ); const activeCodeProvider = useMemo( () => Object.keys(state?.org?.integrations || {}).find( (integrationName) => + isCodeProvider(integrationName) && state?.org?.integrations[integrationName as keyof IntegrationsMap] ?.integrated ) as CodeProviderIntegrations | null, - [state?.org?.integrations] + [isCodeProvider, state?.org?.integrations] ); return ( From c0a4a6634d6da82179ab55efe9448f227bf4968c Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Mon, 10 Aug 2026 19:31:24 +0500 Subject: [PATCH 054/194] fix(jenkins): include jenkins in the workflow integration bucket WORKFLOW_INTEGRATION_BUCKET listed only GITHUB_ACTIONS, so the workflow sync never saw Jenkins. Two failures followed: sync_org_workflows returned early with "No workflow integrations found" on a Jenkins-only workspace, and _get_active_repo_workflows filtered Jenkins rows out of the provider query even when GitHub was also linked -- which meant mapping a Jenkins job silently dropped that repo's Deployment Frequency to zero. Adds a test that drives _get_active_repo_workflows with fakes that apply the same name and provider filters as the real queries, so it fails without the bucket entry rather than restating the constant. Co-Authored-By: Claude Opus 5 --- .../mhq/service/workflows/integration.py | 6 + .../workflows/sync/test_etl_handler.py | 182 ++++++++++++++++++ 2 files changed, 188 insertions(+) create mode 100644 backend/analytics_server/tests/service/workflows/sync/test_etl_handler.py diff --git a/backend/analytics_server/mhq/service/workflows/integration.py b/backend/analytics_server/mhq/service/workflows/integration.py index 5a9aba7b..1c2c4ec1 100644 --- a/backend/analytics_server/mhq/service/workflows/integration.py +++ b/backend/analytics_server/mhq/service/workflows/integration.py @@ -4,8 +4,14 @@ from mhq.store.models.code import RepoWorkflowProviders from mhq.store.repos.core import CoreRepoService +# CLUSTOX: Jenkins is a deployment-detection provider alongside GitHub Actions. +# It has to be in this bucket or the workflow sync never sees it: a Jenkins-only +# workspace gets "No workflow integrations found" and returns early, and a mixed +# workspace has its Jenkins rows filtered out of +# get_active_repo_workflows_by_repo_ids_and_providers. WORKFLOW_INTEGRATION_BUCKET = [ RepoWorkflowProviders.GITHUB_ACTIONS.value, + RepoWorkflowProviders.JENKINS.value, ] diff --git a/backend/analytics_server/tests/service/workflows/sync/test_etl_handler.py b/backend/analytics_server/tests/service/workflows/sync/test_etl_handler.py new file mode 100644 index 00000000..58647949 --- /dev/null +++ b/backend/analytics_server/tests/service/workflows/sync/test_etl_handler.py @@ -0,0 +1,182 @@ +# CLUSTOX: the Jenkins rows have to survive the provider filter in +# _get_active_repo_workflows, otherwise the whole integration is inert -- the +# mapping is stored, the picker works, and no build is ever synced. +from typing import List + +from mhq.service.workflows import integration as integration_module +from mhq.service.workflows.integration import WorkflowsIntegrationsService +from mhq.service.workflows.sync import etl_handler as etl_handler_module +from mhq.service.workflows.sync.etl_handler import WorkflowETLHandler +from mhq.store.models.code import ( + OrgRepo, + RepoWorkflow, + RepoWorkflowProviders, + RepoWorkflowType, +) +from mhq.utils.string import uuid4_str + + +class FakeIntegration: + def __init__(self, name: str): + self.name = name + + +class FakeCoreRepoService: + """ + Applies the same name filter the real query does, so a provider missing + from WORKFLOW_INTEGRATION_BUCKET is invisible to the service. + """ + + def __init__(self, linked_names: List[str]): + self._linked_names = linked_names + + def get_org_integrations_for_names(self, org_id: str, provider_names: List[str]): + return [FakeIntegration(n) for n in self._linked_names if n in provider_names] + + +class FakeCodeIntegrationService: + @staticmethod + def get_org_providers(org_id: str) -> List[str]: + return ["github"] + + +class FakeCodeRepoService: + def __init__(self, org_repos: List[OrgRepo]): + self._org_repos = org_repos + + def get_active_org_repos(self, org_id: str) -> List[OrgRepo]: + return self._org_repos + + +class FakeWorkflowRepoService: + """ + Applies the same provider filter as + get_active_repo_workflows_by_repo_ids_and_providers, and records the + provider list it was handed so the test can assert on it directly. + """ + + def __init__(self, repo_workflows: List[RepoWorkflow]): + self._repo_workflows = repo_workflows + self.providers_queried: List[RepoWorkflowProviders] = [] + + def get_active_repo_workflows_by_repo_ids_and_providers( + self, repo_ids: List[str], providers: List[RepoWorkflowProviders] + ) -> List[RepoWorkflow]: + self.providers_queried = providers + return [ + workflow + for workflow in self._repo_workflows + if str(workflow.org_repo_id) in repo_ids + and workflow.provider in providers + and workflow.is_active + ] + + +def _org_repo(org_id: str) -> OrgRepo: + return OrgRepo(id=uuid4_str(), org_id=org_id, name="api", provider="github") + + +def _repo_workflow(org_repo: OrgRepo, provider: RepoWorkflowProviders) -> RepoWorkflow: + return RepoWorkflow( + id=uuid4_str(), + org_repo_id=str(org_repo.id), + type=RepoWorkflowType.DEPLOYMENT, + provider=provider, + provider_workflow_id="deploy-api", + is_active=True, + name="deploy-api", + ) + + +def _build_handler( + monkeypatch, + linked_integration_names: List[str], + org_repos: List[OrgRepo], + repo_workflows: List[RepoWorkflow], +): + workflow_repo_service = FakeWorkflowRepoService(repo_workflows) + workflows_integrations_service = WorkflowsIntegrationsService( + FakeCoreRepoService(linked_integration_names) + ) + monkeypatch.setattr( + etl_handler_module, + "get_code_integration_service", + lambda: FakeCodeIntegrationService(), + ) + monkeypatch.setattr( + etl_handler_module, + "get_workflows_integrations_service", + lambda: workflows_integrations_service, + ) + handler = WorkflowETLHandler( + FakeCodeRepoService(org_repos), + workflow_repo_service, + None, + None, + None, + ) + return handler, workflow_repo_service + + +def test_a_jenkins_only_workspace_syncs_its_jenkins_workflow(monkeypatch): + org_id = uuid4_str() + org_repo = _org_repo(org_id) + jenkins_workflow = _repo_workflow(org_repo, RepoWorkflowProviders.JENKINS) + + handler, workflow_repo_service = _build_handler( + monkeypatch, + linked_integration_names=["github", "jenkins"], + org_repos=[org_repo], + repo_workflows=[jenkins_workflow], + ) + + active = handler._get_active_repo_workflows(org_id) + + # Before Jenkins was added to WORKFLOW_INTEGRATION_BUCKET this was []: + # get_org_providers returned nothing, so the function bailed out with + # "No workflow integrations found". + assert [workflow.id for _, workflow in active] == [jenkins_workflow.id] + assert active[0][0].id == org_repo.id + assert RepoWorkflowProviders.JENKINS in workflow_repo_service.providers_queried + + +def test_jenkins_survives_the_provider_filter_when_github_is_also_linked(monkeypatch): + org_id = uuid4_str() + github_repo = _org_repo(org_id) + jenkins_repo = _org_repo(org_id) + github_workflow = _repo_workflow(github_repo, RepoWorkflowProviders.GITHUB_ACTIONS) + jenkins_workflow = _repo_workflow(jenkins_repo, RepoWorkflowProviders.JENKINS) + + handler, workflow_repo_service = _build_handler( + monkeypatch, + linked_integration_names=["github", "jenkins"], + org_repos=[github_repo, jenkins_repo], + repo_workflows=[github_workflow, jenkins_workflow], + ) + + active = handler._get_active_repo_workflows(org_id) + + # Both providers must come back. Previously the Jenkins row was dropped by + # RepoWorkflow.provider.in_(providers) even with GitHub linked. + assert {workflow.id for _, workflow in active} == { + github_workflow.id, + jenkins_workflow.id, + } + assert set(workflow_repo_service.providers_queried) == { + RepoWorkflowProviders.GITHUB_ACTIONS, + RepoWorkflowProviders.JENKINS, + } + + +def test_get_org_providers_reports_jenkins_for_a_jenkins_only_workspace(): + service = WorkflowsIntegrationsService(FakeCoreRepoService(["jenkins"])) + + # This is the exact value sync_org_workflows checks before returning early. + assert service.get_org_providers(uuid4_str()) == ["jenkins"] + + +def test_an_unrelated_integration_is_still_not_a_workflow_provider(): + service = WorkflowsIntegrationsService(FakeCoreRepoService(["gitlab"])) + + assert service.get_org_providers(uuid4_str()) == [] + assert "gitlab" not in integration_module.WORKFLOW_INTEGRATION_BUCKET From 8bcfc9920552d568f290046096bc6e30b219f812 Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Mon, 10 Aug 2026 19:33:16 +0500 Subject: [PATCH 055/194] fix(jenkins): normalise the git plugin branch before storing it _get_branch stored the git plugin's raw value, "origin/main". Every DORA query filters head_branch against prod_branches, which defaults to the Postgres regex "^$" -- 'origin/main' ~ '^main$' is false, so every Jenkins run was silently excluded from Deployment Frequency, Lead Time, CFR and MTTR. Strips refs/heads/, refs/remotes//, and a bare leading remote name. The bare case uses a whitelist rather than "drop the first segment", so release/2.0 and feature/origin-story survive untouched. The fixture keeps branch_name="origin/main" because that is what real Jenkins reports; the handler test now asserts the stored value is "main", and a companion test applies the "^main$" pattern to show the raw value was excluded and the normalised one is not. Co-Authored-By: Claude Opus 5 --- .../workflows/sync/etl_jenkins_handler.py | 44 ++++++++++++++++- .../sync/test_etl_jenkins_handler.py | 47 ++++++++++++++++++- 2 files changed, 89 insertions(+), 2 deletions(-) diff --git a/backend/analytics_server/mhq/service/workflows/sync/etl_jenkins_handler.py b/backend/analytics_server/mhq/service/workflows/sync/etl_jenkins_handler.py index e952bc4d..8138aad7 100644 --- a/backend/analytics_server/mhq/service/workflows/sync/etl_jenkins_handler.py +++ b/backend/analytics_server/mhq/service/workflows/sync/etl_jenkins_handler.py @@ -17,6 +17,16 @@ from mhq.utils.log import LOG from mhq.utils.time import time_now +REFS_HEADS_PREFIX = "refs/heads/" +REFS_REMOTES_PREFIX = "refs/remotes/" + +# Remote names stripped from a bare "/" value. Deliberately a +# whitelist rather than "drop the first segment": branch names legitimately +# contain slashes, and release/2.0 or feature/origin-story must survive intact. +# Anything after an explicit refs/remotes/ prefix is a remote by definition and +# does not need this list. +KNOWN_REMOTE_NAMES = frozenset({"origin", "upstream"}) + class JenkinsETLHandler(WorkflowProviderETLHandler): def __init__(self, org_id: str, jenkins_api_service, workflow_repo_service): @@ -131,9 +141,41 @@ def _get_branch(build: Dict) -> Optional[str]: for action in build.get("actions", []): revision = (action or {}).get("lastBuiltRevision") if revision and revision.get("branch"): - return revision["branch"][0].get("name") + return JenkinsETLHandler._normalise_branch( + revision["branch"][0].get("name") + ) return None + @staticmethod + def _normalise_branch(branch: Optional[str]) -> Optional[str]: + """ + The git plugin reports 'origin/main', sometimes 'refs/remotes/origin/main'. + Every DORA query filters head_branch against prod_branches, which + defaults to the Postgres regex '^$' -- and + 'origin/main' ~ '^main$' is false, so an un-normalised value drops the + run out of Deployment Frequency, Lead Time, CFR and MTTR with no error. + """ + if not branch: + return None + branch = branch.strip() + if not branch: + return None + + if branch.startswith(REFS_HEADS_PREFIX): + return branch.split(REFS_HEADS_PREFIX, 1)[1] or None + + if branch.startswith(REFS_REMOTES_PREFIX): + # The segment straight after refs/remotes/ is a remote name by + # definition, whatever it is called. + rest = branch.split(REFS_REMOTES_PREFIX, 1)[1] + _, separator, after_remote = rest.partition("/") + return (after_remote if separator and after_remote else rest) or None + + remote, separator, rest = branch.partition("/") + if separator and rest and remote in KNOWN_REMOTE_NAMES: + return rest + return branch + @staticmethod def _get_actor(build: Dict) -> Optional[str]: for action in build.get("actions", []): diff --git a/backend/analytics_server/tests/service/workflows/sync/test_etl_jenkins_handler.py b/backend/analytics_server/tests/service/workflows/sync/test_etl_jenkins_handler.py index 07052ad7..d0769d3b 100644 --- a/backend/analytics_server/tests/service/workflows/sync/test_etl_jenkins_handler.py +++ b/backend/analytics_server/tests/service/workflows/sync/test_etl_jenkins_handler.py @@ -1,5 +1,7 @@ +import re from datetime import datetime +import pytest import pytz from mhq.service.workflows.sync.etl_jenkins_handler import JenkinsETLHandler @@ -74,13 +76,56 @@ def test_adapt_maps_every_field(): assert run.provider_workflow_run_id == "42" assert run.status == RepoWorkflowRunsStatus.SUCCESS assert run.event_actor == "hamad" - assert run.head_branch == "origin/main" + # Jenkins reports "origin/main"; what gets stored has to be "main", or the + # prod-branch regex '^main$' excludes the run from every DORA metric. + assert run.head_branch == "main" assert run.html_url == build["url"] # Jenkins reports milliseconds; RepoWorkflowRuns.duration is seconds. assert run.duration == 125 assert run.conducted_at.isoformat() == "2025-08-10T12:00:00+00:00" +@pytest.mark.parametrize( + "reported,stored", + [ + # What the git plugin actually reports for a checkout of main. + ("origin/main", "main"), + ("refs/remotes/origin/main", "main"), + ("refs/heads/main", "main"), + ("main", "main"), + # A non-default remote name, still unambiguous after refs/remotes/. + ("refs/remotes/deploy-target/release", "release"), + ("upstream/main", "main"), + # Slash-containing branch names must survive untouched: the first + # segment here is part of the branch, not a remote. + ("feature/origin-story", "feature/origin-story"), + ("release/2.0", "release/2.0"), + ("refs/heads/feature/origin-story", "feature/origin-story"), + ("origin/feature/origin-story", "feature/origin-story"), + ], +) +def test_branch_is_normalised_to_what_the_prod_branch_regex_expects(reported, stored): + build = get_jenkins_build_dict(branch_name=reported) + + run = _handler()._adapt_jenkins_build_to_workflow_run(uuid4_str(), build) + + assert run.head_branch == stored + + +def test_the_stored_branch_matches_the_default_prod_branch_pattern(): + # The failure this guards: prod_branches defaults to ["^$"] + # (mhq/service/code/repository_service.py) and is applied as a Postgres + # regex against RepoWorkflowRuns.head_branch. Python's re uses the same + # syntax for this pattern, so it reproduces the exclusion faithfully. + prod_branch_pattern = "^main$" + build = get_jenkins_build_dict(branch_name="origin/main") + + run = _handler()._adapt_jenkins_build_to_workflow_run(uuid4_str(), build) + + assert re.search(prod_branch_pattern, "origin/main") is None + assert re.search(prod_branch_pattern, run.head_branch) is not None + + def test_a_build_without_the_git_plugin_records_with_nulls_rather_than_raising(): build = get_jenkins_build_dict_without_git_plugin() From ce4250ee0ad9ba9b2d08a24c4e84d39a1c882f94 Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Mon, 10 Aug 2026 19:39:33 +0500 Subject: [PATCH 056/194] fix(jenkins): one active deployment source per repo, reversibly C3: create_jenkins_mapping deactivated only GITHUB_ACTIONS rows, so an admin who mapped the wrong Jenkins job and then the right one ended up with two active Jenkins workflows on the repo and every deploy counted twice -- and the UI no longer flags the repo the second time round, so it submits silently. deactivate_github_actions_workflows_for_repo is now deactivate_deployment_workflows_for_repo and switches off every active deployment workflow on the repo bar the one being mapped, whatever its provider. I1: removing a mapping was a one-way door. docs/JENKINS_INTEGRATION.md and the mapping dialog both promise the GitHub Actions rows come back, and nothing performed that restore; re-mapping the same job afterwards hit repoworkflow_orgrepoid_provider_workflow_id and 500ed. Unmapping now reactivates the repo's GITHUB_ACTIONS deployment workflows in the same commit, and mapping reuses the existing row for an (org_repo_id, provider_workflow_id) pair instead of inserting a duplicate. The route tests drive the real handlers through a Flask test client over an in-memory repo service that reproduces the unique index, so the re-map-after-unmap sequence fails with a 500 without this change. Co-Authored-By: Claude Opus 5 --- .../analytics_server/mhq/api/integrations.py | 113 +++++-- .../mhq/store/repos/workflows.py | 68 ++++- .../tests/api/test_jenkins_routes.py | 285 +++++++++++++++++- 3 files changed, 428 insertions(+), 38 deletions(-) diff --git a/backend/analytics_server/mhq/api/integrations.py b/backend/analytics_server/mhq/api/integrations.py index d783ecf7..b99498d4 100644 --- a/backend/analytics_server/mhq/api/integrations.py +++ b/backend/analytics_server/mhq/api/integrations.py @@ -1,9 +1,15 @@ from typing import List from flask import Blueprint, jsonify from github import GithubException + +# CLUSTOX: Required is used by the Jenkins mapping request schemas below. from voluptuous import Schema, Optional, Required, Coerce, Range, All from mhq.exapi.models.gitlab import GitlabRepo + +# CLUSTOX: the Jenkins mapping routes take a JSON body rather than query +# params, and resolve repos and workflow rows directly -- hence dataschema, +# uuid_validator, the RepoWorkflow model and the two repo services. from mhq.api.request_utils import dataschema, queryschema, uuid_validator from mhq.service.external_integrations_service import get_external_integrations_service from mhq.service.query_validator import get_query_validator @@ -12,6 +18,8 @@ from mhq.store.models.code.workflows.workflows import RepoWorkflow from mhq.store.repos.code import CodeRepoService from mhq.store.repos.workflows import WorkflowRepoService + +# END CLUSTOX from mhq.utils.github import github_org_data_multi_thread_worker app = Blueprint("integrations", __name__) @@ -19,22 +27,37 @@ STATUS_TOO_MANY_REQUESTS = 429 -# CLUSTOX: one deployment source per repo. A repo tracked through both GitHub -# Actions and Jenkins would count every deploy twice, doubling Deployment -# Frequency with nothing visibly wrong. Deactivation is reversible: the rows -# survive, so removing the Jenkins mapping can restore them. -def deactivate_github_actions_workflows_for_repo(workflows: List) -> int: +# CLUSTOX: one active deployment source per repo, whatever the provider. A repo +# tracked through both GitHub Actions and Jenkins -- or through two Jenkins jobs, +# which is two clicks away if an admin maps the wrong job and then the right one +# -- would count every deploy twice, doubling Deployment Frequency with nothing +# visibly wrong. Deactivation is reversible: the rows survive, and +# reactivate_github_actions_workflows_for_repo restores them when the Jenkins +# mapping is removed. +def deactivate_deployment_workflows_for_repo(workflows: List) -> int: deactivated = 0 for workflow in workflows: - if ( - workflow.provider == RepoWorkflowProviders.GITHUB_ACTIONS - and workflow.is_active - ): + if workflow.is_active: workflow.is_active = False deactivated += 1 return deactivated +# CLUSTOX: the other half of the invariant above. docs/JENKINS_INTEGRATION.md and +# the mapping dialog both promise the admin can undo a mapping, which means the +# GitHub Actions rows the mapping switched off have to come back on. +def reactivate_github_actions_workflows_for_repo(workflows: List) -> int: + reactivated = 0 + for workflow in workflows: + if ( + workflow.provider == RepoWorkflowProviders.GITHUB_ACTIONS + and not workflow.is_active + ): + workflow.is_active = True + reactivated += 1 + return reactivated + + @app.route("/orgs//integrations/github/orgs", methods={"GET"}) def get_github_orgs(org_id: str): query_validator = get_query_validator() @@ -250,6 +273,9 @@ def get_jenkins_jobs(org_id: str): return JenkinsApiService(base_url, username, api_token).get_jobs() +# CLUSTOX: maps a Jenkins job to a repo as its deployment source. Enforces the +# one-active-deployment-source-per-repo invariant and reuses the row for an +# already-known (org_repo_id, provider_workflow_id) pair. @app.route("/orgs//integrations/jenkins/mappings", methods={"POST"}) @dataschema( Schema( @@ -268,29 +294,56 @@ def create_jenkins_mapping(org_id: str, org_repo_id: str, job_full_name: str): return jsonify({"error": f"Repo {org_repo_id} not found in org {org_id}"}), 404 workflow_repo_service = WorkflowRepoService() - existing_workflows = workflow_repo_service.get_repo_workflow_by_repo_ids( + + # A row may already exist for this (org_repo_id, provider_workflow_id) -- + # left behind inactive by a previous unmapping. The pair is uniquely + # indexed, so inserting a second one raises IntegrityError; reuse it. + existing_jenkins_workflow = ( + workflow_repo_service.get_repo_workflow_by_repo_id_and_provider_workflow_id( + org_repo_id, RepoWorkflowProviders.JENKINS, job_full_name + ) + ) + + active_workflows = workflow_repo_service.get_repo_workflow_by_repo_ids( [org_repo_id], RepoWorkflowType.DEPLOYMENT ) - # Mutates existing_workflows in place, flipping GitHub Actions rows to - # inactive; the actual write happens together with the new Jenkins row - # below, in one commit. - deactivated_count = deactivate_github_actions_workflows_for_repo(existing_workflows) - - jenkins_workflow = RepoWorkflow( - org_repo_id=org_repo_id, - type=RepoWorkflowType.DEPLOYMENT, - provider=RepoWorkflowProviders.JENKINS, - provider_workflow_id=job_full_name, - is_active=True, - name=job_full_name, + # Every active deployment workflow on this repo except the one being mapped, + # regardless of provider: an already-active Jenkins job for a different + # pipeline has to go too, or the repo ends up with two live sources. + workflows_to_deactivate = [ + workflow + for workflow in active_workflows + if existing_jenkins_workflow is None + or str(workflow.id) != str(existing_jenkins_workflow.id) + ] + # Mutates the rows in place; the actual write happens together with the + # Jenkins row below, in one commit. + deactivated_count = deactivate_deployment_workflows_for_repo( + workflows_to_deactivate ) + + if existing_jenkins_workflow is not None: + existing_jenkins_workflow.is_active = True + existing_jenkins_workflow.name = job_full_name + jenkins_workflow = existing_jenkins_workflow + else: + jenkins_workflow = RepoWorkflow( + org_repo_id=org_repo_id, + type=RepoWorkflowType.DEPLOYMENT, + provider=RepoWorkflowProviders.JENKINS, + provider_workflow_id=job_full_name, + is_active=True, + name=job_full_name, + ) workflow_repo_service.create_jenkins_repo_workflow( - jenkins_workflow, existing_workflows + jenkins_workflow, workflows_to_deactivate ) - return {"ok": True, "deactivated_github_workflows": deactivated_count} + return {"ok": True, "deactivated_workflows": deactivated_count} +# CLUSTOX: removes a Jenkins mapping and restores the GitHub Actions workflows +# the mapping displaced. @app.route("/orgs//integrations/jenkins/mappings", methods={"DELETE"}) @dataschema( Schema( @@ -317,9 +370,17 @@ def delete_jenkins_mapping(org_id: str, repo_workflow_id: str): 404, ) - workflow_repo_service.deactivate_repo_workflow(repo_workflow) + # Removing the mapping gives the repo its deployment source back, which is + # what both the docs and the mapping dialog promise. + github_workflows = workflow_repo_service.get_repo_workflows_by_repo_id_and_provider( + str(repo_workflow.org_repo_id), + RepoWorkflowProviders.GITHUB_ACTIONS, + RepoWorkflowType.DEPLOYMENT, + ) + reactivated_count = reactivate_github_actions_workflows_for_repo(github_workflows) + workflow_repo_service.deactivate_repo_workflow(repo_workflow, github_workflows) - return {"ok": True} + return {"ok": True, "reactivated_github_workflows": reactivated_count} @app.route("/orgs//integrations/gitlab/user/repos", methods={"GET"}) diff --git a/backend/analytics_server/mhq/store/repos/workflows.py b/backend/analytics_server/mhq/store/repos/workflows.py index d9e4bb9e..0f294feb 100644 --- a/backend/analytics_server/mhq/store/repos/workflows.py +++ b/backend/analytics_server/mhq/store/repos/workflows.py @@ -118,6 +118,9 @@ def get_repo_workflow_by_repo_ids( .all() ) + # CLUSTOX: used by the Jenkins mapping routes to resolve a workflow by id. + # Deliberately not filtered by provider -- callers that are provider + # specific must check RepoWorkflow.provider themselves. @rollback_on_exc def get_repo_workflow_by_id(self, repo_workflow_id: str) -> Optional[RepoWorkflow]: return ( @@ -127,27 +130,84 @@ def get_repo_workflow_by_id(self, repo_workflow_id: str) -> Optional[RepoWorkflo .one_or_none() ) - # CLUSTOX: creates the Jenkins mapping and deactivates the repo's active - # GitHub Actions workflows in a single commit. If either half failed on + # CLUSTOX: resolves the row that occupies the uniquely indexed + # (org_repo_id, provider_workflow_id) pair, active or not. Mapping a job + # that was previously mapped and then unmapped has to reuse this row; + # inserting a second one violates + # repoworkflow_orgrepoid_provider_workflow_id. + @rollback_on_exc + def get_repo_workflow_by_repo_id_and_provider_workflow_id( + self, + repo_id: str, + provider: RepoWorkflowProviders, + provider_workflow_id: str, + ) -> Optional[RepoWorkflow]: + return ( + self._db.session.query(RepoWorkflow) + .options(defer(RepoWorkflow.meta)) + .filter( + RepoWorkflow.org_repo_id == repo_id, + RepoWorkflow.provider == provider, + RepoWorkflow.provider_workflow_id == provider_workflow_id, + ) + .one_or_none() + ) + + # CLUSTOX: unlike get_repo_workflow_by_repo_ids this returns inactive rows + # too, because the inactive ones are exactly what a Jenkins mapping turned + # off and an unmapping has to turn back on. + @rollback_on_exc + def get_repo_workflows_by_repo_id_and_provider( + self, + repo_id: str, + provider: RepoWorkflowProviders, + type: RepoWorkflowType, + ) -> List[RepoWorkflow]: + return ( + self._db.session.query(RepoWorkflow) + .options(defer(RepoWorkflow.meta)) + .filter( + RepoWorkflow.org_repo_id == repo_id, + RepoWorkflow.provider == provider, + RepoWorkflow.type == type, + ) + .all() + ) + + # CLUSTOX: writes the Jenkins mapping and the deactivation of the repo's + # other deployment workflows in a single commit. If either half failed on # its own, a repo could be left with two active deployment sources, # silently double-counting deployments -- see - # deactivate_github_actions_workflows_for_repo in mhq/api/integrations.py. + # deactivate_deployment_workflows_for_repo in mhq/api/integrations.py. @rollback_on_exc def create_jenkins_repo_workflow( self, jenkins_workflow: RepoWorkflow, workflows_to_deactivate: List[RepoWorkflow], ) -> RepoWorkflow: + # add() on an instance already loaded in this session is a no-op, so + # this covers both a brand new mapping and the reactivation of a row + # that a previous unmapping left inactive. self._db.session.add(jenkins_workflow) for workflow in workflows_to_deactivate: self._db.session.merge(workflow) self._db.session.commit() return jenkins_workflow + # CLUSTOX: removing a Jenkins mapping and restoring the GitHub Actions rows + # it displaced happen in one commit. Half of this applied on its own would + # leave the repo with no active deployment source at all, and its + # Deployment Frequency at zero. @rollback_on_exc - def deactivate_repo_workflow(self, repo_workflow: RepoWorkflow) -> RepoWorkflow: + def deactivate_repo_workflow( + self, + repo_workflow: RepoWorkflow, + workflows_to_reactivate: List[RepoWorkflow] = None, + ) -> RepoWorkflow: repo_workflow.is_active = False self._db.session.merge(repo_workflow) + for workflow in workflows_to_reactivate or []: + self._db.session.merge(workflow) self._db.session.commit() return repo_workflow diff --git a/backend/analytics_server/tests/api/test_jenkins_routes.py b/backend/analytics_server/tests/api/test_jenkins_routes.py index a51d1a5f..c70c7f47 100644 --- a/backend/analytics_server/tests/api/test_jenkins_routes.py +++ b/backend/analytics_server/tests/api/test_jenkins_routes.py @@ -1,30 +1,299 @@ -from mhq.api.integrations import deactivate_github_actions_workflows_for_repo -from mhq.store.models.code import RepoWorkflowProviders +from typing import List, Optional +from unittest.mock import MagicMock, patch + +import pytest +from flask import Flask +from sqlalchemy.exc import IntegrityError + +from mhq.api import integrations as integrations_module +from mhq.api.integrations import ( + deactivate_deployment_workflows_for_repo, + reactivate_github_actions_workflows_for_repo, +) +from mhq.store.models.code import RepoWorkflowProviders, RepoWorkflowType +from mhq.utils.string import uuid4_str + +ORG_ID = "11111111-1111-4111-8111-111111111111" +OTHER_ORG_ID = "22222222-2222-4222-8222-222222222222" +REPO_ID = "33333333-3333-4333-8333-333333333333" class FakeWorkflow: - def __init__(self, provider, is_active=True): + def __init__( + self, + provider, + is_active=True, + org_repo_id=REPO_ID, + provider_workflow_id="deploy-api", + type=RepoWorkflowType.DEPLOYMENT, + id=None, + ): + self.id = id or uuid4_str() self.provider = provider self.is_active = is_active + self.org_repo_id = org_repo_id + self.provider_workflow_id = provider_workflow_id + self.type = type + self.name = provider_workflow_id -def test_deactivates_only_github_actions_workflows(): +def test_deactivates_every_active_deployment_workflow_whatever_the_provider(): workflows = [ FakeWorkflow(RepoWorkflowProviders.GITHUB_ACTIONS), FakeWorkflow(RepoWorkflowProviders.GITHUB_ACTIONS), FakeWorkflow(RepoWorkflowProviders.JENKINS), ] - count = deactivate_github_actions_workflows_for_repo(workflows) + count = deactivate_deployment_workflows_for_repo(workflows) - assert count == 2 - assert [w.is_active for w in workflows] == [False, False, True] + # One active deployment source per repo is the invariant. Leaving the + # Jenkins row active -- as the GitHub-only version did -- lets a repo end + # up mapped to two Jenkins jobs and count every deploy twice. + assert count == 3 + assert [w.is_active for w in workflows] == [False, False, False] def test_deactivating_is_idempotent(): workflows = [FakeWorkflow(RepoWorkflowProviders.GITHUB_ACTIONS, is_active=False)] - count = deactivate_github_actions_workflows_for_repo(workflows) + count = deactivate_deployment_workflows_for_repo(workflows) # Already inactive, so nothing changed and nothing is reported. assert count == 0 + + +def test_reactivates_only_inactive_github_actions_workflows(): + github = FakeWorkflow(RepoWorkflowProviders.GITHUB_ACTIONS, is_active=False) + already_on = FakeWorkflow(RepoWorkflowProviders.GITHUB_ACTIONS, is_active=True) + jenkins = FakeWorkflow(RepoWorkflowProviders.JENKINS, is_active=False) + + count = reactivate_github_actions_workflows_for_repo([github, already_on, jenkins]) + + assert count == 1 + assert github.is_active is True + assert already_on.is_active is True + assert jenkins.is_active is False + + +class FakeOrgRepo: + def __init__(self, repo_id=REPO_ID, org_id=ORG_ID): + self.id = repo_id + self.org_id = org_id + + +class FakeWorkflowRepoService: + """ + In-memory stand-in that reproduces the two database behaviours these routes + depend on: get_repo_workflow_by_repo_ids returns only active rows, and the + (org_repo_id, provider_workflow_id) pair is uniquely indexed + (repoworkflow_orgrepoid_provider_workflow_id), so a second insert for a + pair that already has a row raises IntegrityError. + """ + + def __init__(self, workflows: Optional[List[FakeWorkflow]] = None): + self.workflows = list(workflows or []) + + def get_repo_workflow_by_repo_ids(self, repo_ids, type) -> List[FakeWorkflow]: + return [ + w + for w in self.workflows + if str(w.org_repo_id) in [str(r) for r in repo_ids] + and w.type == type + and w.is_active + ] + + def get_repo_workflow_by_repo_id_and_provider_workflow_id( + self, repo_id, provider, provider_workflow_id + ) -> Optional[FakeWorkflow]: + for w in self.workflows: + if ( + str(w.org_repo_id) == str(repo_id) + and w.provider == provider + and w.provider_workflow_id == provider_workflow_id + ): + return w + return None + + def get_repo_workflows_by_repo_id_and_provider( + self, repo_id, provider, type + ) -> List[FakeWorkflow]: + return [ + w + for w in self.workflows + if str(w.org_repo_id) == str(repo_id) + and w.provider == provider + and w.type == type + ] + + def get_repo_workflow_by_id(self, repo_workflow_id) -> Optional[FakeWorkflow]: + for w in self.workflows: + if str(w.id) == str(repo_workflow_id): + return w + return None + + def create_jenkins_repo_workflow(self, jenkins_workflow, workflows_to_deactivate): + if jenkins_workflow not in self.workflows: + for w in self.workflows: + if ( + str(w.org_repo_id) == str(jenkins_workflow.org_repo_id) + and w.provider_workflow_id == jenkins_workflow.provider_workflow_id + ): + raise IntegrityError( + 'duplicate key value violates unique constraint "' + 'repoworkflow_orgrepoid_provider_workflow_id"', + None, + Exception(), + ) + # The database assigns the primary key on insert. + if getattr(jenkins_workflow, "id", None) is None: + jenkins_workflow.id = uuid4_str() + self.workflows.append(jenkins_workflow) + return jenkins_workflow + + def deactivate_repo_workflow(self, repo_workflow, workflows_to_reactivate=None): + repo_workflow.is_active = False + return repo_workflow + + @property + def active_deployment_workflows(self) -> List[FakeWorkflow]: + return [w for w in self.workflows if w.is_active] + + +def _build_client(): + app = Flask(__name__) + app.register_blueprint(integrations_module.app) + return app.test_client() + + +@pytest.fixture +def routes(request): + """ + Wires the two Jenkins mapping routes to in-memory services. The repo lookup + resolves REPO_ID into ORG_ID unless the test overrides it. + """ + workflow_repo_service = FakeWorkflowRepoService(getattr(request, "param", None)) + code_repo_service = MagicMock() + code_repo_service.get_repo_by_id.side_effect = lambda repo_id: ( + FakeOrgRepo(repo_id=repo_id) if str(repo_id) == REPO_ID else None + ) + + with patch.object( + integrations_module, "get_query_validator", return_value=MagicMock() + ), patch.object( + integrations_module, "CodeRepoService", return_value=code_repo_service + ), patch.object( + integrations_module, "WorkflowRepoService", return_value=workflow_repo_service + ): + yield _build_client(), workflow_repo_service + + +def _map(client, job_full_name, org_id=ORG_ID, repo_id=REPO_ID): + return client.post( + f"/orgs/{org_id}/integrations/jenkins/mappings", + json={"org_repo_id": repo_id, "job_full_name": job_full_name}, + ) + + +def _unmap(client, repo_workflow_id, org_id=ORG_ID): + return client.delete( + f"/orgs/{org_id}/integrations/jenkins/mappings", + json={"repo_workflow_id": str(repo_workflow_id)}, + ) + + +def test_mapping_a_second_job_leaves_exactly_one_active_deployment_source(routes): + client, service = routes + + assert _map(client, "deploy-api").status_code == 200 + # The wrong job first, then the right one -- two clicks, no warning in the + # UI the second time round. + assert _map(client, "deploy-api-v2").status_code == 200 + + active = service.active_deployment_workflows + assert [w.provider_workflow_id for w in active] == ["deploy-api-v2"] + + +@pytest.mark.parametrize( + "routes", + [[FakeWorkflow(RepoWorkflowProviders.GITHUB_ACTIONS, provider_workflow_id="9931")]], + indirect=True, +) +def test_mapping_deactivates_the_repos_github_actions_workflow(routes): + client, service = routes + + response = _map(client, "deploy-api") + + assert response.status_code == 200 + assert response.json["deactivated_workflows"] == 1 + active = service.active_deployment_workflows + assert [w.provider for w in active] == [RepoWorkflowProviders.JENKINS] + + +@pytest.mark.parametrize( + "routes", + [[FakeWorkflow(RepoWorkflowProviders.GITHUB_ACTIONS, provider_workflow_id="9931")]], + indirect=True, +) +def test_unmapping_restores_the_github_actions_workflow(routes): + client, service = routes + _map(client, "deploy-api") + jenkins_workflow = service.get_repo_workflow_by_repo_id_and_provider_workflow_id( + REPO_ID, RepoWorkflowProviders.JENKINS, "deploy-api" + ) + + response = _unmap(client, jenkins_workflow.id) + + assert response.status_code == 200 + assert response.json["reactivated_github_workflows"] == 1 + # The repo is back to exactly one active deployment source: the GitHub + # Actions workflow the mapping displaced. Without this the repo has none, + # and its Deployment Frequency stays at zero. + active = service.active_deployment_workflows + assert [w.provider for w in active] == [RepoWorkflowProviders.GITHUB_ACTIONS] + + +def test_remapping_the_same_job_after_unmapping_does_not_hit_the_unique_index(routes): + client, service = routes + assert _map(client, "deploy-api").status_code == 200 + jenkins_workflow = service.get_repo_workflow_by_repo_id_and_provider_workflow_id( + REPO_ID, RepoWorkflowProviders.JENKINS, "deploy-api" + ) + assert _unmap(client, jenkins_workflow.id).status_code == 200 + + # This used to insert a second row for the same + # (org_repo_id, provider_workflow_id) pair, raising IntegrityError -> 500. + response = _map(client, "deploy-api") + + assert response.status_code == 200 + assert len(service.workflows) == 1 + assert service.workflows[0].is_active is True + + +@pytest.mark.parametrize( + "routes", + [[FakeWorkflow(RepoWorkflowProviders.JENKINS, org_repo_id="nope")]], + indirect=True, +) +def test_deleting_a_workflow_from_another_workspace_is_404(routes): + client, service = routes + foreign_workflow = service.workflows[0] + + response = _unmap(client, foreign_workflow.id, org_id=OTHER_ORG_ID) + + assert response.status_code == 404 + assert foreign_workflow.is_active is True + + +def test_deleting_an_unknown_workflow_is_404(routes): + client, _ = routes + + assert _unmap(client, uuid4_str()).status_code == 404 + + +def test_mapping_a_repo_from_another_workspace_is_404(routes): + client, service = routes + + response = _map(client, "deploy-api", repo_id=uuid4_str()) + + assert response.status_code == 404 + assert service.workflows == [] From 8f8a452417602856ca25cbc282f50ff872ec6394 Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Mon, 10 Aug 2026 19:39:53 +0500 Subject: [PATCH 057/194] fix(jenkins): reject non-jenkins workflows on the jenkins delete route delete_jenkins_mapping resolved the row through get_repo_workflow_by_id, which is not filtered by provider. The cross-workspace check was correct, but inside their own workspace an admin could pass any repo_workflow_id and silently disable a GitHub Actions deployment workflow through the Jenkins route. Returns 404 unless the resolved workflow is a JENKINS row. Co-Authored-By: Claude Opus 5 --- .../analytics_server/mhq/api/integrations.py | 11 ++++++++++- .../tests/api/test_jenkins_routes.py | 17 +++++++++++++++++ 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/backend/analytics_server/mhq/api/integrations.py b/backend/analytics_server/mhq/api/integrations.py index b99498d4..cdda37b5 100644 --- a/backend/analytics_server/mhq/api/integrations.py +++ b/backend/analytics_server/mhq/api/integrations.py @@ -343,7 +343,7 @@ def create_jenkins_mapping(org_id: str, org_repo_id: str, job_full_name: str): # CLUSTOX: removes a Jenkins mapping and restores the GitHub Actions workflows -# the mapping displaced. +# the mapping displaced. Jenkins rows only -- see the provider check below. @app.route("/orgs//integrations/jenkins/mappings", methods={"DELETE"}) @dataschema( Schema( @@ -370,6 +370,15 @@ def delete_jenkins_mapping(org_id: str, repo_workflow_id: str): 404, ) + # get_repo_workflow_by_id is not filtered by provider, so without this an + # admin could pass any workflow id from their own workspace and silently + # disable a GitHub Actions deployment workflow through the Jenkins route. + if repo_workflow.provider != RepoWorkflowProviders.JENKINS: + return ( + jsonify({"error": f"Jenkins mapping {repo_workflow_id} not found"}), + 404, + ) + # Removing the mapping gives the repo its deployment source back, which is # what both the docs and the mapping dialog promise. github_workflows = workflow_repo_service.get_repo_workflows_by_repo_id_and_provider( diff --git a/backend/analytics_server/tests/api/test_jenkins_routes.py b/backend/analytics_server/tests/api/test_jenkins_routes.py index c70c7f47..7d75b803 100644 --- a/backend/analytics_server/tests/api/test_jenkins_routes.py +++ b/backend/analytics_server/tests/api/test_jenkins_routes.py @@ -269,6 +269,23 @@ def test_remapping_the_same_job_after_unmapping_does_not_hit_the_unique_index(ro assert service.workflows[0].is_active is True +@pytest.mark.parametrize( + "routes", + [[FakeWorkflow(RepoWorkflowProviders.GITHUB_ACTIONS, provider_workflow_id="9931")]], + indirect=True, +) +def test_deleting_a_github_actions_workflow_through_the_jenkins_route_is_404(routes): + client, service = routes + github_workflow = service.workflows[0] + + response = _unmap(client, github_workflow.id) + + assert response.status_code == 404 + # Untouched: the Jenkins route must not be a way to disable a GitHub + # Actions deployment workflow. + assert github_workflow.is_active is True + + @pytest.mark.parametrize( "routes", [[FakeWorkflow(RepoWorkflowProviders.JENKINS, org_repo_id="nope")]], From d33e12ca3e720dc1bd9fb23eb1e646c9cbcf83c9 Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Mon, 10 Aug 2026 19:42:50 +0500 Subject: [PATCH 058/194] fix(jenkins): discover nested jobs and bound each request in wall-clock I4: JOB_TREE was "jobs[name,fullName,url]", which does not recurse. On a folder-organised Jenkins the picker offered folders and multibranch projects rather than jobs; mapping one gives a URL with no "builds" key, so the handler logs "No builds found" and the repo reports zero deployments forever with no error. The tree now recurses three levels -- enough for folder/multibranch layouts -- and get_jobs flattens it down to buildable entries, dropping containers by _class so that a folder sitting at the deepest requested level is still excluded. I6: requests' read timeout is between bytes, not total, so a Jenkins dribbling one byte every 29 seconds held the connection open forever and stalled every workspace behind it in the sequential sync loop. Each request is now streamed and read under a total wall-clock deadline, enforced with time.monotonic and no new dependency -- the per-job ceiling the design doc promised. Co-Authored-By: Claude Opus 5 --- backend/analytics_server/mhq/exapi/jenkins.py | 124 +++++++++++++-- .../tests/exapi/test_jenkins.py | 149 +++++++++++++++++- .../tests/factories/models/exapi/jenkins.py | 57 +++++++ 3 files changed, 316 insertions(+), 14 deletions(-) diff --git a/backend/analytics_server/mhq/exapi/jenkins.py b/backend/analytics_server/mhq/exapi/jenkins.py index 9feeddbc..68fd3bb4 100644 --- a/backend/analytics_server/mhq/exapi/jenkins.py +++ b/backend/analytics_server/mhq/exapi/jenkins.py @@ -1,7 +1,9 @@ # CLUSTOX: Jenkins REST client. Knows Jenkins' API shape and nothing about our # models -- adaptation to RepoWorkflowRuns lives in the ETL handler. +import json from datetime import datetime -from typing import Dict, List, Tuple +from time import monotonic +from typing import Dict, List, Optional, Tuple import pytz import requests @@ -10,6 +12,14 @@ # untimed request against a hung Jenkins stalls every workspace behind it. DEFAULT_TIMEOUT: Tuple[int, int] = (5, 30) +# Total wall-clock ceiling for a single Jenkins request. requests' read timeout +# is between bytes, not total: a Jenkins dribbling one byte every 29 seconds +# satisfies a 30s read timeout forever and holds the sequential sync loop open +# indefinitely. This is the ceiling the design doc promised. +DEFAULT_MAX_SECONDS: int = 60 + +RESPONSE_CHUNK_SIZE = 64 * 1024 + # Fetched per build. Explicit rather than a wildcard so a Jenkins with many # plugins does not return megabytes of action data per build. BUILD_TREE = ( @@ -18,7 +28,30 @@ "lastBuiltRevision[SHA1,branch[name]]]]" ) -JOB_TREE = "jobs[name,fullName,url]" +JOB_FIELDS = "name,fullName,url,_class" + +# Folders and multibranch projects nest, and a non-recursive tree offers the +# admin the container instead of the job. Mapping a container yields a URL with +# no "builds" key, so the sync reports zero deployments forever without an +# error. Three levels covers folder/multibranch layouts without asking a large +# Jenkins for its whole tree. +JOB_TREE_DEPTH = 3 + +# Substrings of the _class Jenkins reports for things that hold jobs rather than +# being one. Matched loosely because the plugin class names are long and +# version-dependent, and because a container sitting exactly at JOB_TREE_DEPTH +# has no nested "jobs" key to give it away. +CONTAINER_CLASS_MARKERS = ("Folder", "MultiBranchProject") + + +def _build_job_tree(depth: int) -> str: + tree = f"jobs[{JOB_FIELDS}]" + for _ in range(max(depth - 1, 0)): + tree = f"jobs[{JOB_FIELDS},{tree}]" + return tree + + +JOB_TREE = _build_job_tree(JOB_TREE_DEPTH) def job_path(job_full_name: str) -> str: @@ -26,6 +59,28 @@ def job_path(job_full_name: str) -> str: return "/".join(f"job/{segment}" for segment in job_full_name.split("/")) +class JenkinsResponse: + """ + The slice of requests.Response this client uses, with the body already read + under a wall-clock deadline. Exists so _get can enforce that deadline while + reading, which a plain Response cannot do. + """ + + def __init__(self, status_code: int, body: bytes, url: str): + self.status_code = status_code + self._body = body + self._url = url + + def json(self): + return json.loads(self._body or b"{}") + + def raise_for_status(self) -> None: + if self.status_code >= 400: + raise requests.HTTPError( + f"Jenkins returned HTTP {self.status_code} for {self._url}" + ) + + class JenkinsApiService: def __init__( self, @@ -33,21 +88,41 @@ def __init__( username: str, api_token: str, timeout: Tuple[int, int] = DEFAULT_TIMEOUT, + max_seconds: int = DEFAULT_MAX_SECONDS, ): self._base_url = base_url.rstrip("/") self._auth = (username, api_token) self._timeout = timeout + self._max_seconds = max_seconds - def _get(self, path: str): + def _get(self, path: str) -> JenkinsResponse: + url = f"{self._base_url}/{path}" + deadline = monotonic() + self._max_seconds # TLS verification is deliberately not configurable. A skip-verify # option is easy to add, hard to remove, and this connection carries an # API token. - return requests.get( - f"{self._base_url}/{path}", + response = requests.get( + url, auth=self._auth, timeout=self._timeout, verify=True, + stream=True, ) + body = bytearray() + try: + for chunk in response.iter_content(chunk_size=RESPONSE_CHUNK_SIZE): + # The connect and read timeouts bound the gaps; this bounds the + # whole call. Worst case the request overshoots the ceiling by + # one read timeout, because the check only runs between chunks. + if monotonic() > deadline: + raise requests.exceptions.Timeout( + f"Jenkins request to {url} exceeded the " + f"{self._max_seconds}s ceiling" + ) + body.extend(chunk) + finally: + response.close() + return JenkinsResponse(response.status_code, bytes(body), url) def check_pat(self) -> bool: response = self._get("api/json") @@ -56,14 +131,37 @@ def check_pat(self) -> bool: def get_jobs(self) -> List[Dict]: response = self._get(f"api/json?tree={JOB_TREE}") response.raise_for_status() - return [ - { - "name": job.get("name"), - "full_name": job.get("fullName") or job.get("name"), - "url": job.get("url"), - } - for job in response.json().get("jobs", []) - ] + return self._collect_jobs(response.json().get("jobs", [])) + + @classmethod + def _collect_jobs(cls, nodes: Optional[List[Dict]]) -> List[Dict]: + """Flattens the nested tree down to the entries that can actually run.""" + jobs: List[Dict] = [] + for node in nodes or []: + if not node: + continue + children = node.get("jobs") + if isinstance(children, list): + jobs.extend(cls._collect_jobs(children)) + continue + if cls._is_container(node): + # A container deeper than the tree we asked for. Offering it + # would let an admin map something that reports zero + # deployments forever, so leave it out. + continue + jobs.append( + { + "name": node.get("name"), + "full_name": node.get("fullName") or node.get("name"), + "url": node.get("url"), + } + ) + return jobs + + @staticmethod + def _is_container(node: Dict) -> bool: + job_class = node.get("_class") or "" + return any(marker in job_class for marker in CONTAINER_CLASS_MARKERS) def get_builds(self, job_full_name: str, bookmark: datetime) -> List[Dict]: response = self._get(f"{job_path(job_full_name)}/api/json?tree={BUILD_TREE}") diff --git a/backend/analytics_server/tests/exapi/test_jenkins.py b/backend/analytics_server/tests/exapi/test_jenkins.py index 1e003987..042709dd 100644 --- a/backend/analytics_server/tests/exapi/test_jenkins.py +++ b/backend/analytics_server/tests/exapi/test_jenkins.py @@ -1,8 +1,12 @@ from datetime import datetime +import pytest import pytz +import requests -from mhq.exapi.jenkins import JenkinsApiService, job_path +from mhq.exapi import jenkins as jenkins_module +from mhq.exapi.jenkins import JOB_TREE, JenkinsApiService, job_path +from tests.factories.models.exapi.jenkins import get_jenkins_nested_jobs_dict def test_job_path_encodes_a_top_level_job(): @@ -39,3 +43,146 @@ def raise_for_status(): builds = service.get_builds("deploy-api", bookmark) assert [b["number"] for b in builds] == [3] + + +class FakeJsonResponse: + status_code = 200 + + def __init__(self, payload): + self._payload = payload + + def json(self): + return self._payload + + @staticmethod + def raise_for_status(): + return None + + +def _service_returning(payload): + service = JenkinsApiService("https://jenkins.example.com", "user", "token") + service.requested_paths = [] + + def _get(path): + service.requested_paths.append(path) + return FakeJsonResponse(payload) + + service._get = _get + return service + + +def test_the_job_tree_recurses_far_enough_for_folder_layouts(): + # A flat "jobs[...]" tree stops at the top level, so on a folder-organised + # Jenkins the picker offers folders instead of jobs. + assert JOB_TREE.count("jobs[") == 3 + + +def test_get_jobs_returns_nested_jobs_and_not_their_containers(): + service = _service_returning(get_jenkins_nested_jobs_dict()) + + jobs = service.get_jobs() + + assert [job["full_name"] for job in jobs] == [ + "deploy-legacy", + "platform/deploy-api", + "platform/web/main", + "platform/tooling/lint", + ] + # Mapping a folder or a multibranch project produces a URL with no "builds" + # key, which the handler reports as "No builds found" -- zero deployments + # forever, and no error anywhere. + assert not {"platform", "platform/web", "platform/tooling"} & { + job["full_name"] for job in jobs + } + assert jobs[2]["name"] == "main" + assert jobs[1]["url"].endswith("/job/platform/job/deploy-api/") + + +def test_get_jobs_survives_an_instance_with_no_jobs(): + assert _service_returning({}).get_jobs() == [] + + +class DribblingResponse: + """A Jenkins that sends one byte just inside every read timeout.""" + + status_code = 200 + + def __init__(self, clock, seconds_per_chunk): + self._clock = clock + self._seconds_per_chunk = seconds_per_chunk + self.closed = False + + def iter_content(self, chunk_size): + while True: + self._clock["now"] += self._seconds_per_chunk + yield b"x" + + def close(self): + self.closed = True + + +def _patch_clock_and_response(monkeypatch, response, clock): + monkeypatch.setattr(jenkins_module, "monotonic", lambda: clock["now"]) + monkeypatch.setattr(jenkins_module.requests, "get", lambda *a, **kw: response) + + +def test_a_dribbling_jenkins_hits_the_total_ceiling(monkeypatch): + clock = {"now": 0.0} + response = DribblingResponse(clock, seconds_per_chunk=29) + _patch_clock_and_response(monkeypatch, response, clock) + + service = JenkinsApiService( + "https://jenkins.example.com", "user", "token", max_seconds=60 + ) + + # Without a wall-clock ceiling this loops forever: requests' read timeout + # is between bytes, and the sequential sync loop stalls behind it. + with pytest.raises(requests.exceptions.Timeout): + service.check_pat() + + assert response.closed is True + + +def test_a_prompt_jenkins_is_read_in_full(monkeypatch): + clock = {"now": 0.0} + + class PromptResponse: + status_code = 200 + closed = False + + @staticmethod + def iter_content(chunk_size): + yield b'{"jobs": ' + yield b"[]}" + + def close(self): + PromptResponse.closed = True + + _patch_clock_and_response(monkeypatch, PromptResponse(), clock) + + service = JenkinsApiService("https://jenkins.example.com", "user", "token") + + assert service.get_jobs() == [] + assert PromptResponse.closed is True + + +def test_an_http_error_is_raised_rather_than_parsed(monkeypatch): + clock = {"now": 0.0} + + class ForbiddenResponse: + status_code = 403 + + @staticmethod + def iter_content(chunk_size): + yield b"forbidden" + + def close(self): + return None + + _patch_clock_and_response(monkeypatch, ForbiddenResponse(), clock) + + service = JenkinsApiService("https://jenkins.example.com", "user", "token") + + assert service.check_pat() is False + with pytest.raises(requests.HTTPError): + service.get_jobs() diff --git a/backend/analytics_server/tests/factories/models/exapi/jenkins.py b/backend/analytics_server/tests/factories/models/exapi/jenkins.py index cc389288..8e730b08 100644 --- a/backend/analytics_server/tests/factories/models/exapi/jenkins.py +++ b/backend/analytics_server/tests/factories/models/exapi/jenkins.py @@ -33,3 +33,60 @@ def get_jenkins_build_dict_without_git_plugin(**kwargs) -> Dict: build = get_jenkins_build_dict(**kwargs) build["actions"] = [{}] return build + + +BASE_URL = "https://jenkins.example.com" + +FOLDER_CLASS = "com.cloudbees.hudson.plugins.folder.Folder" +MULTIBRANCH_CLASS = ( + "org.jenkinsci.plugins.workflow.multibranch.WorkflowMultiBranchProject" +) +PIPELINE_CLASS = "org.jenkinsci.plugins.workflow.job.WorkflowJob" +FREESTYLE_CLASS = "hudson.model.FreeStyleProject" + + +def _node(job_class: str, full_name: str, jobs=None) -> Dict: + node = { + "_class": job_class, + "name": full_name.split("/")[-1], + "fullName": full_name, + "url": f"{BASE_URL}/{'/'.join('job/' + s for s in full_name.split('/'))}/", + } + if jobs is not None: + node["jobs"] = jobs + return node + + +def get_jenkins_nested_jobs_dict() -> Dict: + """ + A folder-organised Jenkins, the layout the flat tree could not see past. + Note "platform/tooling/nested": a folder sitting at the deepest level the + tree asks for, so it comes back with no "jobs" key at all and only its + _class marks it as a container. + """ + return { + "jobs": [ + _node(FREESTYLE_CLASS, "deploy-legacy"), + _node( + FOLDER_CLASS, + "platform", + jobs=[ + _node(PIPELINE_CLASS, "platform/deploy-api"), + _node( + MULTIBRANCH_CLASS, + "platform/web", + jobs=[_node(PIPELINE_CLASS, "platform/web/main")], + ), + _node( + FOLDER_CLASS, + "platform/tooling", + jobs=[ + _node(FOLDER_CLASS, "platform/tooling/nested"), + _node(PIPELINE_CLASS, "platform/tooling/lint"), + ], + ), + ], + ), + _node(FOLDER_CLASS, "empty", jobs=[]), + ] + } From 3be28126d8b8eab5f15b33061fd55c448d9698fa Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Mon, 10 Aug 2026 19:50:56 +0500 Subject: [PATCH 059/194] fix(jenkins): declare requests, and report an unreachable jenkins as 502 requests was a first-class import in mhq/exapi/jenkins.py but resolved only transitively through PyGithub, so an upstream dependency bump could have removed it silently. Pinned in backend/requirements.txt to the version already installed. get_jenkins_jobs had no error handling, so a connection refusal or a timeout came back as a 500 -- which the setup form reports as bad credentials, sending the admin to check a token that was never the problem. requests.RequestException now returns a 502 with the reason. Co-Authored-By: Claude Opus 5 --- .../analytics_server/mhq/api/integrations.py | 13 ++++++- .../tests/api/test_jenkins_routes.py | 38 +++++++++++++++++++ backend/requirements.txt | 4 ++ 3 files changed, 54 insertions(+), 1 deletion(-) diff --git a/backend/analytics_server/mhq/api/integrations.py b/backend/analytics_server/mhq/api/integrations.py index cdda37b5..1945e431 100644 --- a/backend/analytics_server/mhq/api/integrations.py +++ b/backend/analytics_server/mhq/api/integrations.py @@ -258,6 +258,8 @@ def get_jenkins_jobs(org_id: str): # CLUSTOX: Jenkins-specific imports stay local to the handler, mirroring # the lazy-import precedent for provider modules in # mhq/service/workflows/sync/etl_jenkins_handler.py. + import requests + from mhq.exapi.jenkins import JenkinsApiService from mhq.store.repos.core import CoreRepoService from mhq.utils.jenkins import get_jenkins_config @@ -270,7 +272,16 @@ def get_jenkins_jobs(org_id: str): if not (api_token and base_url and username): return {"error": "Jenkins is not configured for this workspace"}, 400 - return JenkinsApiService(base_url, username, api_token).get_jobs() + try: + return JenkinsApiService(base_url, username, api_token).get_jobs() + except requests.RequestException as e: + # An unreachable, slow or erroring Jenkins is not a bug in this server. + # Surfaced as a 500 it reached the setup form as "check your + # credentials", which sends the admin looking in the wrong place. + return ( + jsonify({"error": f"Could not reach Jenkins: {str(e)}"}), + 502, + ) # CLUSTOX: maps a Jenkins job to a repo as its deployment source. Enforces the diff --git a/backend/analytics_server/tests/api/test_jenkins_routes.py b/backend/analytics_server/tests/api/test_jenkins_routes.py index 7d75b803..d0033932 100644 --- a/backend/analytics_server/tests/api/test_jenkins_routes.py +++ b/backend/analytics_server/tests/api/test_jenkins_routes.py @@ -2,6 +2,7 @@ from unittest.mock import MagicMock, patch import pytest +import requests from flask import Flask from sqlalchemy.exc import IntegrityError @@ -314,3 +315,40 @@ def test_mapping_a_repo_from_another_workspace_is_404(routes): assert response.status_code == 404 assert service.workflows == [] + + +def _jobs_route(get_jobs_result): + """Drives GET .../jenkins/jobs against a fully configured workspace.""" + client = _build_client() + api_service = MagicMock() + if isinstance(get_jobs_result, Exception): + api_service.get_jobs.side_effect = get_jobs_result + else: + api_service.get_jobs.return_value = get_jobs_result + + with patch.object( + integrations_module, "get_query_validator", return_value=MagicMock() + ), patch("mhq.store.repos.core.CoreRepoService") as core_repo_service_cls, patch( + "mhq.utils.jenkins.get_jenkins_config", + return_value=("https://jenkins.example.com", "user"), + ), patch( + "mhq.exapi.jenkins.JenkinsApiService", return_value=api_service + ): + core_repo_service_cls.return_value.get_access_token.return_value = "token" + return client.get(f"/orgs/{ORG_ID}/integrations/jenkins/jobs") + + +def test_an_unreachable_jenkins_is_a_502_not_a_500(): + response = _jobs_route(requests.ConnectionError("connection refused")) + + # A 500 reached the setup form as "check your credentials", which sends the + # admin looking in entirely the wrong place. + assert response.status_code == 502 + assert "Could not reach Jenkins" in response.json["error"] + + +def test_a_reachable_jenkins_returns_its_jobs(): + response = _jobs_route([{"name": "deploy-api", "full_name": "deploy-api"}]) + + assert response.status_code == 200 + assert response.json[0]["full_name"] == "deploy-api" diff --git a/backend/requirements.txt b/backend/requirements.txt index 30060d2b..ddbbe504 100644 --- a/backend/requirements.txt +++ b/backend/requirements.txt @@ -12,3 +12,7 @@ psycopg2==2.9.3 python-dotenv==1.0.1 gunicorn==22.0.0 Flask-SQLAlchemy==3.1.1 +# CLUSTOX: mhq/exapi/jenkins.py imports requests directly. It resolved only +# transitively through PyGithub before, so an upstream bump that dropped it +# would have broken the Jenkins client. +requests==2.34.2 From 0ffa1127befaec016d0aefde283ad3a877c32839 Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Mon, 10 Aug 2026 19:50:56 +0500 Subject: [PATCH 060/194] fix(jenkins): keep the api token out of the console, add fork markers console.error('Failed to connect Jenkins', e) logged the whole axios error. When the failing call is the credential save, that object carries config.data with the raw Jenkins API token in it, which then sits in the browser console. Logs e.message instead. Adds the CLUSTOX sentinels docs/FORK_STRATEGY.md requires so the fork's divergence stays greppable: the Jenkins mappings BFF route had none at all, and pages/integrations.tsx was missing them on the new imports and on the Jenkins card. Co-Authored-By: Claude Opus 5 --- web-server/pages/api/clustox/jenkins/mappings.ts | 3 +++ web-server/pages/integrations.tsx | 5 +++++ web-server/src/components/ClustoxJenkinsSetup.tsx | 8 +++++++- 3 files changed, 15 insertions(+), 1 deletion(-) diff --git a/web-server/pages/api/clustox/jenkins/mappings.ts b/web-server/pages/api/clustox/jenkins/mappings.ts index 7d738ee3..822cab2a 100644 --- a/web-server/pages/api/clustox/jenkins/mappings.ts +++ b/web-server/pages/api/clustox/jenkins/mappings.ts @@ -1,3 +1,6 @@ +// CLUSTOX: BFF route for the Jenkins job-to-repo mapping. POST creates the +// mapping, DELETE removes it; both proxy straight through to the analytics +// server, which owns the one-active-deployment-source-per-repo invariant. import * as yup from 'yup'; import { Endpoint } from '@/api-helpers/global'; diff --git a/web-server/pages/integrations.tsx b/web-server/pages/integrations.tsx index 92cb9f3c..cde7060a 100644 --- a/web-server/pages/integrations.tsx +++ b/web-server/pages/integrations.tsx @@ -7,8 +7,11 @@ import { useEffect, useMemo } from 'react'; import { Authenticated } from 'src/components/Authenticated'; import { handleApi } from '@/api-helpers/axios-api-instance'; +// CLUSTOX: the two halves of the Jenkins integration card below -- the +// credentials form and the job-to-repo mapping table. import { ClustoxJenkinsMapping } from '@/components/ClustoxJenkinsMapping'; import { ClustoxJenkinsSetup } from '@/components/ClustoxJenkinsSetup'; +// END CLUSTOX import { FlexBox } from '@/components/FlexBox'; import { Line } from '@/components/Text'; import { ROUTES } from '@/constants/routes'; @@ -166,7 +169,9 @@ const Content = () => { + {/* CLUSTOX: Jenkins as a third deployment-detection provider. */} + {/* END CLUSTOX */} {showCreationCTA && ( diff --git a/web-server/src/components/ClustoxJenkinsSetup.tsx b/web-server/src/components/ClustoxJenkinsSetup.tsx index 8c41b7d5..26e68d3d 100644 --- a/web-server/src/components/ClustoxJenkinsSetup.tsx +++ b/web-server/src/components/ClustoxJenkinsSetup.tsx @@ -100,7 +100,13 @@ export const ClustoxJenkinsSetup: FC<{ onLinked?: () => void }> = ({ }); onLinked?.(); } catch (e) { - console.error('Failed to connect Jenkins', e); + // CLUSTOX: message only. If the failing call is the credential save, the + // axios error object carries config.data with the raw API token in it, + // which would then sit in the browser console. + console.error( + 'Failed to connect Jenkins', + e instanceof Error ? e.message : String(e) + ); // CLUSTOX: don't leave a workspace "linked" to credentials that don't // work -- the mapping screen would otherwise open to a broken job list. await unlinkProvider(orgId, Integration.JENKINS).catch(() => {}); From d3fba5d955928b7a0b7581ff86ec9946831cee1b Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Mon, 10 Aug 2026 20:21:12 +0500 Subject: [PATCH 061/194] fix(jenkins): make the wall-clock ceiling actually bound the read iter_content(chunk_size=65536) asks urllib3.stream() for a whole chunk, and on a Content-Length response urllib3 blocks until it has that many bytes. The deadline checked between chunks was therefore not consulted until 64 KiB had accumulated -- against a server dribbling 100 bytes every 50ms the first chunk arrived after 36 seconds, on a client with a 30s read timeout and a shorter total ceiling. Chunked-transfer responses happen to yield per HTTP chunk, so they hid the bug; nothing guarantees a Jenkins sends chunked. Read with response.raw.read1(), which returns whatever bytes are already available instead of blocking for a full chunk, so the deadline is re-checked at the pace the server actually sends. decode_content=True keeps gzip handling that reading raw would otherwise skip, and the read timeout is capped at the ceiling so a stalled server cannot overshoot by a full read timeout. read1() is urllib3 2.0+, which requests does not require, so urllib3 is now pinned explicitly. The old ceiling test drove a fake whose iter_content() yielded one byte per iteration -- something requests never does -- so it passed against an implementation that could not bound a real connection. Replaced with tests that drive a real socket server on localhost declaring a Content-Length and dribbling the body. Verified the new dribble test fails against the old implementation (4.2s elapsed against a 1s ceiling) and passes against this one. Co-Authored-By: Claude Opus 5 --- backend/analytics_server/mhq/exapi/jenkins.py | 33 ++- .../tests/exapi/test_jenkins.py | 237 ++++++++++++------ backend/requirements.txt | 5 + 3 files changed, 197 insertions(+), 78 deletions(-) diff --git a/backend/analytics_server/mhq/exapi/jenkins.py b/backend/analytics_server/mhq/exapi/jenkins.py index 68fd3bb4..d3b16fd5 100644 --- a/backend/analytics_server/mhq/exapi/jenkins.py +++ b/backend/analytics_server/mhq/exapi/jenkins.py @@ -18,6 +18,7 @@ # indefinitely. This is the ceiling the design doc promised. DEFAULT_MAX_SECONDS: int = 60 +# Upper bound on a single read, not a demand for that many bytes -- see _get. RESPONSE_CHUNK_SIZE = 64 * 1024 # Fetched per build. Explicit rather than a wildcard so a Jenkins with many @@ -98,27 +99,49 @@ def __init__( def _get(self, path: str) -> JenkinsResponse: url = f"{self._base_url}/{path}" deadline = monotonic() + self._max_seconds + connect_timeout, read_timeout = self._timeout + # CLUSTOX: a read timeout longer than the total ceiling can never be + # reached without the ceiling firing first, and it is the only thing + # standing between us and a server that stops sending entirely. Capping + # it keeps the worst case at ceiling + one gap instead of + # ceiling + read_timeout. + read_timeout = min(read_timeout, self._max_seconds) # TLS verification is deliberately not configurable. A skip-verify # option is easy to add, hard to remove, and this connection carries an # API token. response = requests.get( url, auth=self._auth, - timeout=self._timeout, + timeout=(connect_timeout, read_timeout), verify=True, stream=True, ) body = bytearray() try: - for chunk in response.iter_content(chunk_size=RESPONSE_CHUNK_SIZE): - # The connect and read timeouts bound the gaps; this bounds the - # whole call. Worst case the request overshoots the ceiling by - # one read timeout, because the check only runs between chunks. + # CLUSTOX: read1(), not iter_content(). iter_content() asks + # urllib3.stream() for a whole chunk, and on a Content-Length + # response urllib3 blocks until it has that many bytes -- so a + # deadline checked between chunks is not consulted until 64 KiB has + # accumulated. Measured against a server dribbling 100 bytes every + # 50ms, the first chunk arrived after 36 seconds with the ceiling + # set well below that. read1() returns whatever bytes are already + # available, so the deadline is re-checked at the pace the server + # actually sends and the ceiling genuinely bounds the call. + # Chunked-transfer responses happen to yield per HTTP chunk, but + # nothing guarantees a Jenkins sends chunked. + raw = response.raw + while True: if monotonic() > deadline: raise requests.exceptions.Timeout( f"Jenkins request to {url} exceeded the " f"{self._max_seconds}s ceiling" ) + # decode_content=True so a gzipped body -- requests advertises + # gzip on every request -- is still decoded, which reading + # response.raw directly would otherwise skip. + chunk = raw.read1(RESPONSE_CHUNK_SIZE, decode_content=True) + if not chunk: + break body.extend(chunk) finally: response.close() diff --git a/backend/analytics_server/tests/exapi/test_jenkins.py b/backend/analytics_server/tests/exapi/test_jenkins.py index 042709dd..b7619480 100644 --- a/backend/analytics_server/tests/exapi/test_jenkins.py +++ b/backend/analytics_server/tests/exapi/test_jenkins.py @@ -1,10 +1,14 @@ +import gzip +import json +import socket +import threading from datetime import datetime +from time import monotonic, sleep import pytest import pytz import requests -from mhq.exapi import jenkins as jenkins_module from mhq.exapi.jenkins import JOB_TREE, JenkinsApiService, job_path from tests.factories.models.exapi.jenkins import get_jenkins_nested_jobs_dict @@ -102,87 +106,174 @@ def test_get_jobs_survives_an_instance_with_no_jobs(): assert _service_returning({}).get_jobs() == [] -class DribblingResponse: - """A Jenkins that sends one byte just inside every read timeout.""" - - status_code = 200 - - def __init__(self, clock, seconds_per_chunk): - self._clock = clock - self._seconds_per_chunk = seconds_per_chunk - self.closed = False - - def iter_content(self, chunk_size): - while True: - self._clock["now"] += self._seconds_per_chunk - yield b"x" - - def close(self): - self.closed = True - - -def _patch_clock_and_response(monkeypatch, response, clock): - monkeypatch.setattr(jenkins_module, "monotonic", lambda: clock["now"]) - monkeypatch.setattr(jenkins_module.requests, "get", lambda *a, **kw: response) - - -def test_a_dribbling_jenkins_hits_the_total_ceiling(monkeypatch): - clock = {"now": 0.0} - response = DribblingResponse(clock, seconds_per_chunk=29) - _patch_clock_and_response(monkeypatch, response, clock) - - service = JenkinsApiService( - "https://jenkins.example.com", "user", "token", max_seconds=60 +# CLUSTOX: the ceiling tests below drive a real socket, not a fake response. +# The fake they replace yielded one byte per iter_content() iteration, which +# requests never does: urllib3 blocks until it has the whole requested chunk, +# so the deadline between chunks was not consulted until 64 KiB had arrived. +# The fake passed against an implementation that could not bound a real +# connection -- confidence with nothing behind it. +class LocalJenkins: + """ + An HTTP server on 127.0.0.1 that declares a Content-Length and then hands + the body over in pieces, optionally slowly. Content-Length (not chunked + transfer encoding) is the case that matters: chunked responses happen to + yield per HTTP chunk, so they hide the bug. + """ + + def __init__( + self, + body: bytes, + chunk_size: int = None, + interval: float = 0.0, + status_line: bytes = b"HTTP/1.1 200 OK", + headers=(), + ): + self._body = body + self._chunk_size = chunk_size or max(len(body), 1) + self._interval = interval + self._status_line = status_line + self._headers = list(headers) + self._stop = threading.Event() + self._socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM) + self._socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) + self._socket.bind(("127.0.0.1", 0)) + self._socket.listen(8) + self._socket.settimeout(0.25) + self._thread = threading.Thread(target=self._serve, daemon=True) + + @property + def url(self) -> str: + return "http://127.0.0.1:{}".format(self._socket.getsockname()[1]) + + def __enter__(self): + self._thread.start() + return self + + def __exit__(self, *exc_info): + self._stop.set() + self._thread.join(timeout=5) + self._socket.close() + return False + + def _serve(self): + while not self._stop.is_set(): + try: + connection, _ = self._socket.accept() + except socket.timeout: + continue + except OSError: + return + with connection: + try: + self._respond(connection) + except OSError: + # The client hit its ceiling and hung up mid-dribble, which + # is exactly what these tests are checking for. + pass + + def _respond(self, connection): + connection.settimeout(5) + request = b"" + while b"\r\n\r\n" not in request: + received = connection.recv(4096) + if not received: + return + request += received + + head = [ + self._status_line, + b"Content-Length: %d" % len(self._body), + b"Connection: close", + ] + head.extend(self._headers) + connection.sendall(b"\r\n".join(head) + b"\r\n\r\n") + + for start in range(0, len(self._body), self._chunk_size): + if self._stop.is_set(): + return + end = start + self._chunk_size + connection.sendall(self._body[start:end]) + if self._interval: + sleep(self._interval) + + +def _service(url, max_seconds=5): + return JenkinsApiService( + url, "user", "token", timeout=(2, 2), max_seconds=max_seconds ) - # Without a wall-clock ceiling this loops forever: requests' read timeout - # is between bytes, and the sequential sync loop stalls behind it. - with pytest.raises(requests.exceptions.Timeout): - service.check_pat() - - assert response.closed is True - -def test_a_prompt_jenkins_is_read_in_full(monkeypatch): - clock = {"now": 0.0} - - class PromptResponse: - status_code = 200 - closed = False - - @staticmethod - def iter_content(chunk_size): - yield b'{"jobs": ' - yield b"[]}" - - def close(self): - PromptResponse.closed = True - - _patch_clock_and_response(monkeypatch, PromptResponse(), clock) - - service = JenkinsApiService("https://jenkins.example.com", "user", "token") +def test_a_dribbling_jenkins_is_cut_off_at_the_ceiling(): + # 4000 bytes handed over 50 at a time every 50ms: four seconds to complete, + # with every gap comfortably inside the read timeout, so nothing but the + # total ceiling can stop it. The whole body is under one 64 KiB chunk, so + # an implementation that checks its deadline between iter_content() chunks + # blocks for the full four seconds before looking at the clock even once. + with LocalJenkins(b"x" * 4000, chunk_size=50, interval=0.05) as jenkins: + service = _service(jenkins.url, max_seconds=1) + + started = monotonic() + with pytest.raises(requests.exceptions.Timeout) as raised: + service.check_pat() + elapsed = monotonic() - started + + assert "ceiling" in str(raised.value) + # Generous, and still nowhere near the 4s the body needs to finish. The + # sequential sync loop is the thing being protected: every workspace behind + # this one waits for it. + assert elapsed < 2.5, f"ceiling did not bound the read: took {elapsed:.1f}s" + + +def test_a_prompt_jenkins_is_read_in_full(): + with LocalJenkins(b'{"jobs": []}') as jenkins: + assert _service(jenkins.url).get_jobs() == [] + + +def test_a_body_larger_than_one_read_is_reassembled(): + # read1() returns whatever is available rather than a full chunk, so the + # loop has to keep going until EOF instead of stopping at the first short + # read. + payload = { + "jobs": [ + { + "_class": "hudson.model.FreeStyleProject", + "name": f"deploy-{index}", + "fullName": f"deploy-{index}", + "url": f"http://jenkins/job/deploy-{index}/", + } + for index in range(2000) + ] + } + body = json.dumps(payload).encode() + assert len(body) > 64 * 1024 - assert service.get_jobs() == [] - assert PromptResponse.closed is True + with LocalJenkins(body, chunk_size=1024) as jenkins: + jobs = _service(jenkins.url).get_jobs() + assert len(jobs) == 2000 + assert jobs[-1]["full_name"] == "deploy-1999" -def test_an_http_error_is_raised_rather_than_parsed(monkeypatch): - clock = {"now": 0.0} - class ForbiddenResponse: - status_code = 403 +def test_a_gzipped_response_is_decoded(): + # requests advertises gzip on every request. Reading response.raw without + # decode_content hands json.loads a gzip stream. + body = gzip.compress(json.dumps(get_jenkins_nested_jobs_dict()).encode()) - @staticmethod - def iter_content(chunk_size): - yield b"forbidden" + with LocalJenkins(body, headers=[b"Content-Encoding: gzip"]) as jenkins: + jobs = _service(jenkins.url).get_jobs() - def close(self): - return None + assert [job["full_name"] for job in jobs] == [ + "deploy-legacy", + "platform/deploy-api", + "platform/web/main", + "platform/tooling/lint", + ] - _patch_clock_and_response(monkeypatch, ForbiddenResponse(), clock) - service = JenkinsApiService("https://jenkins.example.com", "user", "token") +def test_an_http_error_is_raised_rather_than_parsed(): + with LocalJenkins(b"forbidden", status_line=b"HTTP/1.1 403 Forbidden") as jenkins: + service = _service(jenkins.url) - assert service.check_pat() is False - with pytest.raises(requests.HTTPError): - service.get_jobs() + assert service.check_pat() is False + with pytest.raises(requests.HTTPError): + service.get_jobs() diff --git a/backend/requirements.txt b/backend/requirements.txt index ddbbe504..a32252d3 100644 --- a/backend/requirements.txt +++ b/backend/requirements.txt @@ -16,3 +16,8 @@ Flask-SQLAlchemy==3.1.1 # transitively through PyGithub before, so an upstream bump that dropped it # would have broken the Jenkins client. requests==2.34.2 +# CLUSTOX: mhq/exapi/jenkins.py enforces its wall-clock ceiling with +# response.raw.read1(), which urllib3 only grew in 2.0. requests itself accepts +# anything <3, so without this pin an environment could resolve urllib3 1.26 +# and the Jenkins client would fail with AttributeError on every call. +urllib3>=2.0,<3 From 549777ff22c057e7aa5e7b01dece14757846ed7e Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Mon, 10 Aug 2026 20:25:29 +0500 Subject: [PATCH 062/194] fix(jenkins): unmapping restores only the workflows mapping displaced reactivate_github_actions_workflows_for_repo turned on every inactive GitHub Actions deployment workflow on the repo, but mapping only ever turns off the active ones. Inactive rows exist for an unrelated reason: teams/v2.ts deactivates all of a repo's deployment workflows and re-enables only the ones the admin selected, so every deselected workflow sits there inactive with nothing to do with Jenkins. A repo with one selected and four deselected GitHub Actions deployment workflows therefore came out of a single map/unmap round trip with five active deployment sources, inflating Deployment Frequency fivefold -- the double-counting the one-source-per-repo invariant exists to prevent, reached through the GitHub path instead of the Jenkins one. Mapping now records the ids it actually deactivated under jenkins_displaced_workflow_ids on the Jenkins row's existing meta JSONB column, and unmapping reactivates exactly those and drops the record in the same commit. A mapping written before this record existed restores nothing: too few active sources under-reports visibly, too many over-reports silently, so the safe direction is clear. Co-Authored-By: Claude Opus 5 --- .../analytics_server/mhq/api/integrations.py | 79 ++++++++--- .../mhq/store/repos/workflows.py | 9 +- .../tests/api/test_jenkins_routes.py | 123 ++++++++++++++++-- docs/JENKINS_INTEGRATION.md | 2 +- 4 files changed, 183 insertions(+), 30 deletions(-) diff --git a/backend/analytics_server/mhq/api/integrations.py b/backend/analytics_server/mhq/api/integrations.py index 1945e431..8ce48bb7 100644 --- a/backend/analytics_server/mhq/api/integrations.py +++ b/backend/analytics_server/mhq/api/integrations.py @@ -27,34 +27,64 @@ STATUS_TOO_MANY_REQUESTS = 429 +# CLUSTOX: key under RepoWorkflow.meta on the Jenkins row, holding the ids of +# the deployment workflows that this mapping switched off. Written on mapping, +# read on unmapping. Without it an unmapping has no way to tell a workflow it +# displaced from one that was already inactive for an unrelated reason. +DISPLACED_WORKFLOW_IDS_KEY = "jenkins_displaced_workflow_ids" + + +# CLUSTOX: JSONB columns on this model default to the string "{}" rather than a +# dict, and a row written before DISPLACED_WORKFLOW_IDS_KEY existed has no +# record at all. Both read as "displaced nothing". +def _workflow_meta(workflow) -> dict: + meta = getattr(workflow, "meta", None) + return meta if isinstance(meta, dict) else {} + + # CLUSTOX: one active deployment source per repo, whatever the provider. A repo # tracked through both GitHub Actions and Jenkins -- or through two Jenkins jobs, # which is two clicks away if an admin maps the wrong job and then the right one # -- would count every deploy twice, doubling Deployment Frequency with nothing # visibly wrong. Deactivation is reversible: the rows survive, and # reactivate_github_actions_workflows_for_repo restores them when the Jenkins -# mapping is removed. -def deactivate_deployment_workflows_for_repo(workflows: List) -> int: - deactivated = 0 +# mapping is removed. Returns the rows it actually switched off, because those +# -- and only those -- are what the unmapping has to switch back on. +def deactivate_deployment_workflows_for_repo(workflows: List) -> List: + deactivated = [] for workflow in workflows: if workflow.is_active: workflow.is_active = False - deactivated += 1 + deactivated.append(workflow) return deactivated # CLUSTOX: the other half of the invariant above. docs/JENKINS_INTEGRATION.md and # the mapping dialog both promise the admin can undo a mapping, which means the # GitHub Actions rows the mapping switched off have to come back on. -def reactivate_github_actions_workflows_for_repo(workflows: List) -> int: - reactivated = 0 +# +# Restricted to displaced_workflow_ids, which is the whole point. Mapping only +# ever deactivates *active* rows, but a repo can hold inactive GitHub Actions +# deployment workflows that Jenkins never touched: teams/v2.ts deactivates all +# of a repo's deployment workflows and re-enables only the ones the admin +# selected, so every deselected workflow sits there inactive. Reactivating +# every inactive row -- as this did -- turns a repo with one selected and four +# deselected workflows into a repo with five active ones after a single +# map/unmap round trip, inflating Deployment Frequency through the GitHub path +# instead of the Jenkins one. +def reactivate_github_actions_workflows_for_repo( + workflows: List, displaced_workflow_ids: List[str] +) -> List: + displaced = {str(workflow_id) for workflow_id in displaced_workflow_ids or []} + reactivated = [] for workflow in workflows: if ( workflow.provider == RepoWorkflowProviders.GITHUB_ACTIONS and not workflow.is_active + and str(workflow.id) in displaced ): workflow.is_active = True - reactivated += 1 + reactivated.append(workflow) return reactivated @@ -329,9 +359,7 @@ def create_jenkins_mapping(org_id: str, org_repo_id: str, job_full_name: str): ] # Mutates the rows in place; the actual write happens together with the # Jenkins row below, in one commit. - deactivated_count = deactivate_deployment_workflows_for_repo( - workflows_to_deactivate - ) + deactivated = deactivate_deployment_workflows_for_repo(workflows_to_deactivate) if existing_jenkins_workflow is not None: existing_jenkins_workflow.is_active = True @@ -346,11 +374,19 @@ def create_jenkins_mapping(org_id: str, org_repo_id: str, job_full_name: str): is_active=True, name=job_full_name, ) + # CLUSTOX: the record of what this mapping displaced. Replaced rather than + # merged: a remapping recomputes the set from scratch, and anything left + # over from a previous mapping was already restored when that one was + # removed. A new dict, not a mutation, so SQLAlchemy sees the change. + jenkins_workflow.meta = { + **_workflow_meta(jenkins_workflow), + DISPLACED_WORKFLOW_IDS_KEY: [str(workflow.id) for workflow in deactivated], + } workflow_repo_service.create_jenkins_repo_workflow( jenkins_workflow, workflows_to_deactivate ) - return {"ok": True, "deactivated_workflows": deactivated_count} + return {"ok": True, "deactivated_workflows": len(deactivated)} # CLUSTOX: removes a Jenkins mapping and restores the GitHub Actions workflows @@ -391,16 +427,29 @@ def delete_jenkins_mapping(org_id: str, repo_workflow_id: str): ) # Removing the mapping gives the repo its deployment source back, which is - # what both the docs and the mapping dialog promise. + # what both the docs and the mapping dialog promise -- but only the rows + # the mapping itself displaced. A mapping written before this record + # existed restores nothing, which is the safe direction: a repo with no + # active deployment source reads as zero deployments, a repo with several + # reads as several times too many, and only the second one is silent. + meta = _workflow_meta(repo_workflow) github_workflows = workflow_repo_service.get_repo_workflows_by_repo_id_and_provider( str(repo_workflow.org_repo_id), RepoWorkflowProviders.GITHUB_ACTIONS, RepoWorkflowType.DEPLOYMENT, ) - reactivated_count = reactivate_github_actions_workflows_for_repo(github_workflows) - workflow_repo_service.deactivate_repo_workflow(repo_workflow, github_workflows) + reactivated = reactivate_github_actions_workflows_for_repo( + github_workflows, meta.get(DISPLACED_WORKFLOW_IDS_KEY) + ) + # Drop the record in the same commit that acts on it, so a second unmapping + # -- or one that follows a team-config edit that deselected these same + # workflows -- does not restore them a second time. + repo_workflow.meta = { + key: value for key, value in meta.items() if key != DISPLACED_WORKFLOW_IDS_KEY + } + workflow_repo_service.deactivate_repo_workflow(repo_workflow, reactivated) - return {"ok": True, "reactivated_github_workflows": reactivated_count} + return {"ok": True, "reactivated_github_workflows": len(reactivated)} @app.route("/orgs//integrations/gitlab/user/repos", methods={"GET"}) diff --git a/backend/analytics_server/mhq/store/repos/workflows.py b/backend/analytics_server/mhq/store/repos/workflows.py index 0f294feb..9082f3c9 100644 --- a/backend/analytics_server/mhq/store/repos/workflows.py +++ b/backend/analytics_server/mhq/store/repos/workflows.py @@ -120,12 +120,13 @@ def get_repo_workflow_by_repo_ids( # CLUSTOX: used by the Jenkins mapping routes to resolve a workflow by id. # Deliberately not filtered by provider -- callers that are provider - # specific must check RepoWorkflow.provider themselves. + # specific must check RepoWorkflow.provider themselves. meta is loaded + # rather than deferred: unmapping reads the displaced-workflow record out + # of it, and a deferred load would just cost a second round trip. @rollback_on_exc def get_repo_workflow_by_id(self, repo_workflow_id: str) -> Optional[RepoWorkflow]: return ( self._db.session.query(RepoWorkflow) - .options(defer(RepoWorkflow.meta)) .filter(RepoWorkflow.id == repo_workflow_id) .one_or_none() ) @@ -134,7 +135,8 @@ def get_repo_workflow_by_id(self, repo_workflow_id: str) -> Optional[RepoWorkflo # (org_repo_id, provider_workflow_id) pair, active or not. Mapping a job # that was previously mapped and then unmapped has to reuse this row; # inserting a second one violates - # repoworkflow_orgrepoid_provider_workflow_id. + # repoworkflow_orgrepoid_provider_workflow_id. meta is loaded rather than + # deferred: mapping rewrites the displaced-workflow record it holds. @rollback_on_exc def get_repo_workflow_by_repo_id_and_provider_workflow_id( self, @@ -144,7 +146,6 @@ def get_repo_workflow_by_repo_id_and_provider_workflow_id( ) -> Optional[RepoWorkflow]: return ( self._db.session.query(RepoWorkflow) - .options(defer(RepoWorkflow.meta)) .filter( RepoWorkflow.org_repo_id == repo_id, RepoWorkflow.provider == provider, diff --git a/backend/analytics_server/tests/api/test_jenkins_routes.py b/backend/analytics_server/tests/api/test_jenkins_routes.py index d0033932..c59e4428 100644 --- a/backend/analytics_server/tests/api/test_jenkins_routes.py +++ b/backend/analytics_server/tests/api/test_jenkins_routes.py @@ -36,6 +36,9 @@ def __init__( self.provider_workflow_id = provider_workflow_id self.type = type self.name = provider_workflow_id + # Matches the column default on RepoWorkflow: a row written before the + # displaced-workflow record existed carries no dict at all. + self.meta = None def test_deactivates_every_active_deployment_workflow_whatever_the_provider(): @@ -45,37 +48,55 @@ def test_deactivates_every_active_deployment_workflow_whatever_the_provider(): FakeWorkflow(RepoWorkflowProviders.JENKINS), ] - count = deactivate_deployment_workflows_for_repo(workflows) + deactivated = deactivate_deployment_workflows_for_repo(workflows) # One active deployment source per repo is the invariant. Leaving the # Jenkins row active -- as the GitHub-only version did -- lets a repo end # up mapped to two Jenkins jobs and count every deploy twice. - assert count == 3 + assert deactivated == workflows assert [w.is_active for w in workflows] == [False, False, False] def test_deactivating_is_idempotent(): workflows = [FakeWorkflow(RepoWorkflowProviders.GITHUB_ACTIONS, is_active=False)] - count = deactivate_deployment_workflows_for_repo(workflows) + deactivated = deactivate_deployment_workflows_for_repo(workflows) - # Already inactive, so nothing changed and nothing is reported. - assert count == 0 + # Already inactive, so nothing changed and nothing is reported -- and, + # crucially, nothing is recorded as displaced, because it was not. + assert deactivated == [] -def test_reactivates_only_inactive_github_actions_workflows(): - github = FakeWorkflow(RepoWorkflowProviders.GITHUB_ACTIONS, is_active=False) +def test_reactivates_only_the_inactive_workflows_the_mapping_displaced(): + displaced = FakeWorkflow(RepoWorkflowProviders.GITHUB_ACTIONS, is_active=False) + deselected = FakeWorkflow(RepoWorkflowProviders.GITHUB_ACTIONS, is_active=False) already_on = FakeWorkflow(RepoWorkflowProviders.GITHUB_ACTIONS, is_active=True) jenkins = FakeWorkflow(RepoWorkflowProviders.JENKINS, is_active=False) - count = reactivate_github_actions_workflows_for_repo([github, already_on, jenkins]) + reactivated = reactivate_github_actions_workflows_for_repo( + [displaced, deselected, already_on, jenkins], [displaced.id] + ) - assert count == 1 - assert github.is_active is True + assert reactivated == [displaced] + assert displaced.is_active is True + # Inactive because the admin deselected it in the team config, not because + # the Jenkins mapping touched it. Turning it on is a deployment the repo + # never had. + assert deselected.is_active is False assert already_on.is_active is True assert jenkins.is_active is False +def test_reactivates_nothing_when_the_mapping_recorded_no_displacement(): + # A mapping created before the displaced-workflow record existed. Restoring + # nothing is the safe reading: too few active sources under-reports + # visibly, too many over-reports silently. + github = FakeWorkflow(RepoWorkflowProviders.GITHUB_ACTIONS, is_active=False) + + assert reactivate_github_actions_workflows_for_repo([github], None) == [] + assert github.is_active is False + + class FakeOrgRepo: def __init__(self, repo_id=REPO_ID, org_id=ORG_ID): self.id = repo_id @@ -253,6 +274,88 @@ def test_unmapping_restores_the_github_actions_workflow(routes): assert [w.provider for w in active] == [RepoWorkflowProviders.GITHUB_ACTIONS] +@pytest.mark.parametrize( + "routes", + [ + [ + FakeWorkflow( + RepoWorkflowProviders.GITHUB_ACTIONS, provider_workflow_id="9931" + ), + ] + + [ + FakeWorkflow( + RepoWorkflowProviders.GITHUB_ACTIONS, + provider_workflow_id=str(9932 + offset), + is_active=False, + ) + for offset in range(4) + ] + ], + indirect=True, +) +def test_unmapping_leaves_deselected_github_actions_workflows_inactive(routes): + """ + One selected GitHub Actions deployment workflow and four the admin + deselected. teams/v2.ts deactivates all of a repo's deployment workflows + and re-enables only the selected ones, so those four sit inactive for a + reason that has nothing to do with Jenkins. Mapping switches off exactly + one row, so unmapping must switch on exactly one row -- restoring all five + gives the repo five active deployment sources and inflates Deployment + Frequency fivefold, which is the double-counting the invariant exists to + prevent, reached through the GitHub path instead of the Jenkins one. + """ + client, service = routes + selected = service.workflows[0] + deselected = service.workflows[1:] + + assert _map(client, "deploy-api").json["deactivated_workflows"] == 1 + jenkins_workflow = service.get_repo_workflow_by_repo_id_and_provider_workflow_id( + REPO_ID, RepoWorkflowProviders.JENKINS, "deploy-api" + ) + # Only the row the mapping actually switched off is recorded as displaced. + assert jenkins_workflow.meta["jenkins_displaced_workflow_ids"] == [str(selected.id)] + + response = _unmap(client, jenkins_workflow.id) + + assert response.status_code == 200 + assert response.json["reactivated_github_workflows"] == 1 + assert selected.is_active is True + assert [w.is_active for w in deselected] == [False, False, False, False] + active = service.active_deployment_workflows + assert [w.provider_workflow_id for w in active] == ["9931"] + + +@pytest.mark.parametrize( + "routes", + [ + [ + FakeWorkflow( + RepoWorkflowProviders.JENKINS, provider_workflow_id="deploy-api" + ), + FakeWorkflow( + RepoWorkflowProviders.GITHUB_ACTIONS, + provider_workflow_id="9931", + is_active=False, + ), + ] + ], + indirect=True, +) +def test_unmapping_a_mapping_with_no_displacement_record_restores_nothing(routes): + # A mapping written before the record existed. There is no way to tell + # which rows it displaced, and guessing "all the inactive ones" is how the + # over-restore happened in the first place. + client, service = routes + jenkins_workflow, github_workflow = service.workflows + assert jenkins_workflow.meta is None + + response = _unmap(client, jenkins_workflow.id) + + assert response.status_code == 200 + assert response.json["reactivated_github_workflows"] == 0 + assert github_workflow.is_active is False + + def test_remapping_the_same_job_after_unmapping_does_not_hit_the_unique_index(routes): client, service = routes assert _map(client, "deploy-api").status_code == 200 diff --git a/docs/JENKINS_INTEGRATION.md b/docs/JENKINS_INTEGRATION.md index f1df7d63..b6274e22 100644 --- a/docs/JENKINS_INTEGRATION.md +++ b/docs/JENKINS_INTEGRATION.md @@ -18,7 +18,7 @@ changing any of them changes the design rather than the implementation. |---|---|---| | Network direction | **Pull** — Middleware calls Jenkins | Jenkins is reachable from the Middleware host, so this fits the existing ETL contract with no new ingest endpoint. | | Job → repo mapping | **Manual, in the UI** | Explicit and accurate. Mapping a job is also what designates it a deployment, so PR builds, lint jobs, and nightlies are ignored by default — and staging jobs are excluded simply by not mapping them. | -| Repo with two CI systems | **One deployment source per repo** | Mapping Jenkins deactivates that repo's GitHub Actions deployment workflows, after a UI warning. Prevents silent double-counting of Deployment Frequency. | +| Repo with two CI systems | **One deployment source per repo** | Mapping Jenkins deactivates that repo's active deployment workflows, after a UI warning. Prevents silent double-counting of Deployment Frequency. Unmapping restores exactly the rows that mapping switched off — recorded on the Jenkins row's `meta` — and nothing else: a repo can hold deployment workflows the admin deselected in the team config, which Jenkins never touched. | | Delivery shape | **Single slice** | One PR rather than a thin proof first. Simpler to plan; the Jenkins API risk (below) surfaces later than it otherwise would. | ## The risk worth naming up front From b67f2adaf27983529edfed74e7bc44f67c5a1ee0 Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Mon, 10 Aug 2026 20:27:37 +0500 Subject: [PATCH 063/194] fix(jenkins): match the reuse lookup to the unique index get_repo_workflow_by_repo_id_and_provider_workflow_id filtered on provider, but repoworkflow_orgrepoid_provider_workflow_id is on (org_repo_id, provider_workflow_id) only. A Jenkins job whose full name happened to equal an existing non-Jenkins provider_workflow_id came back as "no row here", and the insert then hit the index -> IntegrityError -> 500. Unlikely, since GitHub Actions ids are numeric, but it is exactly the path the reuse lookup was added to close. The lookup now matches the index. Reusing a foreign row would be worse than the 500 -- a GitHub Actions workflow, its runs and its history would silently become a Jenkins mapping -- so a collision with another provider is a 409 the admin can see and act on by renaming the job. Co-Authored-By: Claude Opus 5 --- .../analytics_server/mhq/api/integrations.py | 28 ++++++++++++- .../mhq/store/repos/workflows.py | 11 ++++- .../tests/api/test_jenkins_routes.py | 41 ++++++++++++++++--- 3 files changed, 71 insertions(+), 9 deletions(-) diff --git a/backend/analytics_server/mhq/api/integrations.py b/backend/analytics_server/mhq/api/integrations.py index 8ce48bb7..6f2dd515 100644 --- a/backend/analytics_server/mhq/api/integrations.py +++ b/backend/analytics_server/mhq/api/integrations.py @@ -339,11 +339,35 @@ def create_jenkins_mapping(org_id: str, org_repo_id: str, job_full_name: str): # A row may already exist for this (org_repo_id, provider_workflow_id) -- # left behind inactive by a previous unmapping. The pair is uniquely # indexed, so inserting a second one raises IntegrityError; reuse it. - existing_jenkins_workflow = ( + existing_workflow = ( workflow_repo_service.get_repo_workflow_by_repo_id_and_provider_workflow_id( - org_repo_id, RepoWorkflowProviders.JENKINS, job_full_name + org_repo_id, job_full_name ) ) + # The lookup matches the index, which is not scoped by provider, so the row + # occupying this pair may belong to another provider -- a Jenkins job whose + # full name happens to equal a GitHub Actions workflow id. Reusing it would + # silently convert someone else's deployment workflow into a Jenkins + # mapping, taking its runs and its history with it. Refuse instead: an + # admin who sees this can rename the job, and one who does not would never + # have found the takeover. + if ( + existing_workflow is not None + and existing_workflow.provider != RepoWorkflowProviders.JENKINS + ): + return ( + jsonify( + { + "error": ( + f"Repo {org_repo_id} already has a " + f"{existing_workflow.provider.value} workflow with the " + f"id {job_full_name}" + ) + } + ), + 409, + ) + existing_jenkins_workflow = existing_workflow active_workflows = workflow_repo_service.get_repo_workflow_by_repo_ids( [org_repo_id], RepoWorkflowType.DEPLOYMENT diff --git a/backend/analytics_server/mhq/store/repos/workflows.py b/backend/analytics_server/mhq/store/repos/workflows.py index 9082f3c9..1c97c917 100644 --- a/backend/analytics_server/mhq/store/repos/workflows.py +++ b/backend/analytics_server/mhq/store/repos/workflows.py @@ -137,18 +137,25 @@ def get_repo_workflow_by_id(self, repo_workflow_id: str) -> Optional[RepoWorkflo # inserting a second one violates # repoworkflow_orgrepoid_provider_workflow_id. meta is loaded rather than # deferred: mapping rewrites the displaced-workflow record it holds. + # + # Not filtered by provider, because the index is not either. Filtering by + # provider made the lookup narrower than the constraint it exists to avoid + # tripping: a Jenkins job whose full name matched an existing non-Jenkins + # provider_workflow_id came back as "no row here", and the insert then hit + # the index and 500'd. Unlikely -- GitHub Actions ids are numeric -- but + # the caller can only handle a collision it is told about. Callers that + # care about the provider must check RepoWorkflow.provider themselves; + # returning a foreign row is the point. @rollback_on_exc def get_repo_workflow_by_repo_id_and_provider_workflow_id( self, repo_id: str, - provider: RepoWorkflowProviders, provider_workflow_id: str, ) -> Optional[RepoWorkflow]: return ( self._db.session.query(RepoWorkflow) .filter( RepoWorkflow.org_repo_id == repo_id, - RepoWorkflow.provider == provider, RepoWorkflow.provider_workflow_id == provider_workflow_id, ) .one_or_none() diff --git a/backend/analytics_server/tests/api/test_jenkins_routes.py b/backend/analytics_server/tests/api/test_jenkins_routes.py index c59e4428..e8ae3c9a 100644 --- a/backend/analytics_server/tests/api/test_jenkins_routes.py +++ b/backend/analytics_server/tests/api/test_jenkins_routes.py @@ -125,12 +125,13 @@ def get_repo_workflow_by_repo_ids(self, repo_ids, type) -> List[FakeWorkflow]: ] def get_repo_workflow_by_repo_id_and_provider_workflow_id( - self, repo_id, provider, provider_workflow_id + self, repo_id, provider_workflow_id ) -> Optional[FakeWorkflow]: + # Matches repoworkflow_orgrepoid_provider_workflow_id, which is not + # scoped by provider -- so neither is this. for w in self.workflows: if ( str(w.org_repo_id) == str(repo_id) - and w.provider == provider and w.provider_workflow_id == provider_workflow_id ): return w @@ -260,7 +261,7 @@ def test_unmapping_restores_the_github_actions_workflow(routes): client, service = routes _map(client, "deploy-api") jenkins_workflow = service.get_repo_workflow_by_repo_id_and_provider_workflow_id( - REPO_ID, RepoWorkflowProviders.JENKINS, "deploy-api" + REPO_ID, "deploy-api" ) response = _unmap(client, jenkins_workflow.id) @@ -310,7 +311,7 @@ def test_unmapping_leaves_deselected_github_actions_workflows_inactive(routes): assert _map(client, "deploy-api").json["deactivated_workflows"] == 1 jenkins_workflow = service.get_repo_workflow_by_repo_id_and_provider_workflow_id( - REPO_ID, RepoWorkflowProviders.JENKINS, "deploy-api" + REPO_ID, "deploy-api" ) # Only the row the mapping actually switched off is recorded as displaced. assert jenkins_workflow.meta["jenkins_displaced_workflow_ids"] == [str(selected.id)] @@ -360,7 +361,7 @@ def test_remapping_the_same_job_after_unmapping_does_not_hit_the_unique_index(ro client, service = routes assert _map(client, "deploy-api").status_code == 200 jenkins_workflow = service.get_repo_workflow_by_repo_id_and_provider_workflow_id( - REPO_ID, RepoWorkflowProviders.JENKINS, "deploy-api" + REPO_ID, "deploy-api" ) assert _unmap(client, jenkins_workflow.id).status_code == 200 @@ -373,6 +374,36 @@ def test_remapping_the_same_job_after_unmapping_does_not_hit_the_unique_index(ro assert service.workflows[0].is_active is True +@pytest.mark.parametrize( + "routes", + [ + [ + FakeWorkflow( + RepoWorkflowProviders.GITHUB_ACTIONS, provider_workflow_id="deploy-api" + ) + ] + ], + indirect=True, +) +def test_mapping_a_job_whose_name_collides_with_another_provider_is_409(routes): + # repoworkflow_orgrepoid_provider_workflow_id is on + # (org_repo_id, provider_workflow_id), with no provider in it. A lookup + # narrower than the index reported "no row here" and the insert then hit + # the constraint -> IntegrityError -> 500. Reusing the row instead would be + # worse: a GitHub Actions workflow, its runs and its history silently + # become a Jenkins mapping. + client, service = routes + github_workflow = service.workflows[0] + + response = _map(client, "deploy-api") + + assert response.status_code == 409 + assert "deploy-api" in response.json["error"] + assert len(service.workflows) == 1 + assert github_workflow.provider == RepoWorkflowProviders.GITHUB_ACTIONS + assert github_workflow.is_active is True + + @pytest.mark.parametrize( "routes", [[FakeWorkflow(RepoWorkflowProviders.GITHUB_ACTIONS, provider_workflow_id="9931")]], From d060b14fefd75366909ad5ddc7058410da79ce23 Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Tue, 11 Aug 2026 01:39:51 +0500 Subject: [PATCH 064/194] fix(jenkins): pin requests to a version python 3.9 can install requests==2.34.2 requires Python >=3.10. The container runs 3.9.22, so pip could not resolve it and the image build failed outright -- and because `docker compose up -d` then recreated the container from the previous image, the failure presented as a container running stale code rather than as a build error. 2.32.5 is both the newest release that supports 3.9 and the version already installed in the image, so this pins what was actually being used rather than introducing a change. --- backend/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/backend/requirements.txt b/backend/requirements.txt index a32252d3..a71510fe 100644 --- a/backend/requirements.txt +++ b/backend/requirements.txt @@ -15,7 +15,7 @@ Flask-SQLAlchemy==3.1.1 # CLUSTOX: mhq/exapi/jenkins.py imports requests directly. It resolved only # transitively through PyGithub before, so an upstream bump that dropped it # would have broken the Jenkins client. -requests==2.34.2 +requests==2.32.5 # CLUSTOX: mhq/exapi/jenkins.py enforces its wall-clock ceiling with # response.raw.read1(), which urllib3 only grew in 2.0. requests itself accepts # anything <3, so without this pin an environment could resolve urllib3 1.26 From f0a228672acb713518eba3c62b13f6d324b8d36d Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Tue, 11 Aug 2026 02:24:46 +0500 Subject: [PATCH 065/194] docs: implementation plan for the DORA contributor filter Five tasks. Lead Time filters on the pull request author, Deployment Frequency on the actor who triggered the run, and the dropdown lists git usernames as synced -- no identity table, no handle merging. Records a defect worth knowing about independently of this feature. PRFilter declares an authors field and ParsePRFilterProcessor already reads it from the request, but filter_query never added it to its conditions dict. Because that comprehension iterates every dataclass field and indexes conditions by name, sending authors raises KeyError rather than filtering: PRFilter(authors=['hamad']).filter_query -> KeyError: 'authors' Both filter classes share that shape, so a field added without its conditions entry reproduces the bug. Tasks 1 and 2 add both halves in one step and the plan says why. Change failure rate and MTTR stay team-level. Per-person versions built on GitHub issue assignees would be a weak foundation to unpick once Jira lands. --- docs/CONTRIBUTOR_FILTER_PLAN.md | 753 ++++++++++++++++++++++++++++++++ 1 file changed, 753 insertions(+) create mode 100644 docs/CONTRIBUTOR_FILTER_PLAN.md diff --git a/docs/CONTRIBUTOR_FILTER_PLAN.md b/docs/CONTRIBUTOR_FILTER_PLAN.md new file mode 100644 index 00000000..2dc797ed --- /dev/null +++ b/docs/CONTRIBUTOR_FILTER_PLAN.md @@ -0,0 +1,753 @@ +# Contributor Filter Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Filter the DORA metrics page by contributor — Lead Time by pull-request author, Deployment Frequency by deploy actor. + +**Architecture:** The dropdown lists distinct git usernames found in the selected team's synced pull requests, bots excluded. Lead Time reuses `PRFilter.authors`, a field upstream declared and parsed but never applied. Deployment Frequency needs a genuinely new `event_actors` filter on `RepoWorkflowRuns`. No new tables, no identity resolution. + +**Tech Stack:** Python 3.9, Flask, SQLAlchemy 2, pytest (backend); Next.js 15 pages router, TypeScript, MUI 5 (frontend). + +## Decisions already made — do not revisit + +| Decision | Choice | +|---|---| +| Identity | Raw git usernames as synced. No mapping table, no merging of handles. A person using two handles appears twice; that is accepted. | +| Metric scope | Lead Time and Deployment Frequency only. **CFR and MTTR stay team-level** — per-person versions wait for the future Jira integration, because GitHub-issue assignees are too weak a foundation. | +| Involvement | **Authored only.** Reviewers are not included. | +| List source | Distinct authors from the selected team's PRs within the active date range, so the dropdown can never offer someone with no data on screen. | +| Bots | Excluded by default, no UI toggle. | + +## Global Constraints + +- **No database migration, no new tables.** +- Backend style: `black` formatted, `flake8` clean. Run from `backend/analytics_server` using `./venv/bin/python`. +- Frontend must be `tsc` clean. Local Node is 20 and the project needs 22 — verify inside the `middleware-dev` container, which has Node 22.16. +- Every Clustox-authored change carries a `# CLUSTOX:` or `// CLUSTOX:` comment explaining **why**, per `docs/FORK_STRATEGY.md`. +- Backend suite baseline is **214 passing**. Do not break any. +- Bot matching is on the exact `[bot]` suffix plus an explicit list — never a substring match on `bot`, which would exclude a real user called `robotnik`. + +## The trap both filter classes share — read before Task 1 or 2 + +`PRFilter.filter_query` and `WorkflowFilter.filter_query` both end with: + +```python +return [ + conditions[x] + for x in self.__dict__.keys() + if getattr(self, x) is not None and conditions[x] is not None +] +``` + +It iterates **every** dataclass field and indexes `conditions` by name. A field with no matching `conditions` key raises `KeyError` the moment that field is set. + +This is not hypothetical. `PRFilter.authors` exists today with no `conditions` entry, so sending `authors` in a filter payload raises `KeyError: 'authors'` — verified: + +``` +PRFilter(authors=['hamad']).filter_query → KeyError: 'authors' +``` + +**Consequence for this plan:** adding a field without adding its `conditions` entry in the same commit reproduces the bug. Both tasks below add both halves together. + +--- + +## File Structure + +**Backend — modify:** +- `mhq/store/models/code/filter.py` — apply `authors` in `PRFilter.filter_query`. +- `mhq/store/models/code/workflows/filter.py` — add `event_actors` to `WorkflowFilter`. +- `mhq/service/workflows/workflow_filter.py` — parse `event_actors` from the request. +- `mhq/store/repos/code.py` — contributor query. +- `mhq/api/pull_requests.py` — contributors route. + +**Backend — create:** +- `tests/store/models/code/test_pr_filter_authors.py` +- `tests/store/models/code/workflows/test_workflow_filter_actors.py` +- `tests/service/code/test_contributors.py` + +**Frontend — create:** +- `web-server/pages/api/internal/team/[team_id]/contributors.ts` +- `web-server/src/components/ContributorFilter.tsx` + +**Frontend — modify:** +- `web-server/src/api-helpers/team.ts` — thread `authors` into `pr_filter`. +- `web-server/pages/api/internal/team/[team_id]/dora_metrics.ts` — thread `event_actors`. +- `web-server/pages/dora-metrics/index.tsx` — render the filter. + +--- + +### Task 1: Apply the author filter (and fix the KeyError) + +**Files:** +- Modify: `backend/analytics_server/mhq/store/models/code/filter.py` +- Test: `backend/analytics_server/tests/store/models/code/test_pr_filter_authors.py` (create) + +**Interfaces:** +- Consumes: nothing. +- Produces: `PRFilter(authors=[...])` yields a `PullRequest.author.in_(authors)` condition. `ParsePRFilterProcessor` already populates the field — no change needed there. + +- [ ] **Step 1: Write the failing tests** + +Create `backend/analytics_server/tests/store/models/code/test_pr_filter_authors.py`: + +```python +from mhq.store.models.code.filter import PRFilter + + +def test_setting_authors_no_longer_raises(): + # Before this change the conditions dict had no "authors" key, so the + # filter_query comprehension raised KeyError as soon as the field was set. + conditions = PRFilter(authors=["hamad-clustox"]).filter_query + assert len(conditions) == 1 + + +def test_authors_produces_an_author_condition(): + conditions = PRFilter(authors=["hamad-clustox"]).filter_query + # Compiling to string is the cheapest way to assert the column and operator + # without a database. + rendered = str(conditions[0]) + assert "author" in rendered + assert "IN" in rendered.upper() + + +def test_no_authors_produces_no_author_condition(): + conditions = PRFilter(base_branches=["^main$"]).filter_query + assert all("author" not in str(c) for c in conditions) + + +def test_empty_author_list_is_ignored(): + # An empty list means "no filter selected", not "match nothing". + conditions = PRFilter(authors=[]).filter_query + assert conditions == [] + + +def test_existing_filters_are_unaffected(): + # Regression guard: the dashboards everyone already uses must not change. + conditions = PRFilter(base_branches=["^main$"], excluded_pr_ids=["a"]).filter_query + assert len(conditions) == 2 +``` + +- [ ] **Step 2: Run tests to verify they fail** + +Run: `cd backend/analytics_server && ./venv/bin/python -m pytest tests/store/models/code/test_pr_filter_authors.py -v` +Expected: FAIL — the first two with `KeyError: 'authors'` + +- [ ] **Step 3: Add the condition** + +In `mhq/store/models/code/filter.py`, inside `filter_query`, add this alongside the other inner functions: + +```python + # CLUSTOX: authors was declared on PRFilter and parsed from the request + # payload upstream, but never added to the conditions dict below -- so + # sending it raised KeyError rather than filtering. This applies it. + def _authors_query(): + if not self.authors: + return None + + return PullRequest.author.in_(self.authors) +``` + +Then add the entry to the `conditions` dict, as the first key: + +```python + conditions = { + "authors": _authors_query(), + "base_branches": _base_branch_query(), + "repo_filters": _repo_filters_query(), + "excluded_pr_ids": _excluded_pr_ids_query(), + "max_cycle_time": _include_prs_below_max_cycle_time(), + "incident_pr_filters": _incident_pr_filters_query(), + } +``` + +- [ ] **Step 4: Run tests to verify they pass** + +Run: `cd backend/analytics_server && ./venv/bin/python -m pytest tests/store/models/code/test_pr_filter_authors.py -v` +Expected: PASS (5 tests) + +- [ ] **Step 5: Run the full suite** + +Run: `cd backend/analytics_server && ./venv/bin/python -m pytest tests -q` +Expected: 219 passed (214 baseline + 5) + +- [ ] **Step 6: Format, lint, commit** + +```bash +cd backend/analytics_server +./venv/bin/python -m black mhq/store/models/code/filter.py tests/store/models/code/test_pr_filter_authors.py +./venv/bin/python -m flake8 mhq/store/models/code/filter.py +cd ../.. +git add backend/analytics_server/mhq/store/models/code/filter.py \ + backend/analytics_server/tests/store/models/code/test_pr_filter_authors.py +git commit -m "fix(filters): apply the pull request author filter" +``` + +--- + +### Task 2: Add the deploy-actor filter + +**Files:** +- Modify: `backend/analytics_server/mhq/store/models/code/workflows/filter.py` +- Modify: `backend/analytics_server/mhq/service/workflows/workflow_filter.py` +- Test: `backend/analytics_server/tests/store/models/code/workflows/test_workflow_filter_actors.py` (create) + +**Interfaces:** +- Consumes: nothing from Task 1. +- Produces: `WorkflowFilter(event_actors=[...])` yields a `RepoWorkflowRuns.event_actor.in_(...)` condition; `ParseWorkflowFilterProcessor` reads `event_actors` from the request dict. + +- [ ] **Step 1: Write the failing tests** + +Create `backend/analytics_server/tests/store/models/code/workflows/test_workflow_filter_actors.py`: + +```python +from mhq.service.workflows.workflow_filter import ParseWorkflowFilterProcessor +from mhq.store.models.code.workflows.filter import WorkflowFilter + + +def test_event_actors_produces_an_actor_condition(): + conditions = WorkflowFilter(event_actors=["hamad-clustox"]).filter_query + assert len(conditions) == 1 + rendered = str(conditions[0]) + assert "event_actor" in rendered + assert "IN" in rendered.upper() + + +def test_no_event_actors_produces_no_actor_condition(): + conditions = WorkflowFilter(head_branches=["^main$"]).filter_query + assert all("event_actor" not in str(c) for c in conditions) + + +def test_empty_actor_list_is_ignored(): + assert WorkflowFilter(event_actors=[]).filter_query == [] + + +def test_existing_workflow_filters_are_unaffected(): + # Regression guard for the deployment frequency everyone already sees. + conditions = WorkflowFilter(head_branches=["^main$"]).filter_query + assert len(conditions) == 1 + + +def test_parser_reads_event_actors_from_the_request(): + parsed = ParseWorkflowFilterProcessor().apply({"event_actors": ["hamad-clustox"]}) + assert parsed.event_actors == ["hamad-clustox"] + + +def test_parser_defaults_event_actors_to_none(): + parsed = ParseWorkflowFilterProcessor().apply({}) + assert parsed.event_actors is None +``` + +- [ ] **Step 2: Run tests to verify they fail** + +Run: `cd backend/analytics_server && ./venv/bin/python -m pytest tests/store/models/code/workflows/test_workflow_filter_actors.py -v` +Expected: FAIL — `TypeError: __init__() got an unexpected keyword argument 'event_actors'` + +- [ ] **Step 3: Add the field and its condition together** + +In `mhq/store/models/code/workflows/filter.py`, change the `WorkflowFilter` dataclass. Add the field **and** the conditions entry in this same step — adding one without the other reproduces the `KeyError` described at the top of this plan: + +```python +@dataclass +class WorkflowFilter: + head_branches: List[str] = None + repo_filters: Dict[str, Dict] = None + # CLUSTOX: who triggered the deployment. Lead Time filters on the pull + # request author; deployments have no author, so the nearest equivalent is + # the actor who ran the build. + event_actors: List[str] = None + + @property + def filter_query(self) -> List: + def _head_branches_query(): + if not self.head_branches: + return None + + return or_( + RepoWorkflowRuns.head_branch.op("~")(term) + for term in self.head_branches + ) + + def _repo_filters_query(): + if not self.repo_filters: + return None + + return or_( + RepoWorkflowFilter(repo_id, repo_filters).filter_query + for repo_id, repo_filters in self.repo_filters.items() + if repo_filters + ) + + # CLUSTOX: runs with a null event_actor -- Jenkins builds without the + # git plugin, for instance -- are excluded by IN, so per-contributor + # deployment counts will not sum to the team total. That is intended: + # an unattributable deploy belongs to nobody. + def _event_actors_query(): + if not self.event_actors: + return None + + return RepoWorkflowRuns.event_actor.in_(self.event_actors) + + conditions = { + "head_branches": _head_branches_query(), + "repo_filters": _repo_filters_query(), + "event_actors": _event_actors_query(), + } + return [ + conditions[x] + for x in self.__dict__.keys() + if getattr(self, x) is not None and conditions[x] is not None + ] +``` + +- [ ] **Step 4: Parse it from the request** + +In `mhq/service/workflows/workflow_filter.py`, extend `ParseWorkflowFilterProcessor.apply`: + +```python + def apply(self, workflow_filter: Dict = None) -> WorkflowFilter: + workflow_filter = workflow_filter or {} + head_branches: List[str] = self._parse_head_branches(workflow_filter) + repo_filters: Dict[str, Dict] = self._parse_repo_filters(workflow_filter) + # CLUSTOX: contributor filter for deployment frequency. + event_actors: List[str] = workflow_filter.get("event_actors") + + return WorkflowFilter( + head_branches=head_branches, + repo_filters=repo_filters, + event_actors=event_actors, + ) +``` + +Note the added `workflow_filter = workflow_filter or {}` guard — the signature defaults to `None` and the existing body would raise `AttributeError` on a bare `.get`. + +- [ ] **Step 5: Run tests to verify they pass** + +Run: `cd backend/analytics_server && ./venv/bin/python -m pytest tests/store/models/code/workflows/test_workflow_filter_actors.py -v` +Expected: PASS (6 tests) + +- [ ] **Step 6: Run the full suite, format, lint, commit** + +```bash +cd backend/analytics_server +./venv/bin/python -m pytest tests -q # expect 225 passed +./venv/bin/python -m black mhq/store/models/code/workflows/filter.py mhq/service/workflows/workflow_filter.py tests/store/models/code/workflows/test_workflow_filter_actors.py +./venv/bin/python -m flake8 mhq/store/models/code/workflows/filter.py mhq/service/workflows/workflow_filter.py +cd ../.. +git add backend/analytics_server/mhq/store/models/code/workflows/filter.py \ + backend/analytics_server/mhq/service/workflows/workflow_filter.py \ + backend/analytics_server/tests/store/models/code/workflows/test_workflow_filter_actors.py +git commit -m "feat(filters): filter deployments by the actor who triggered them" +``` + +--- + +### Task 3: Contributor list query and route + +**Files:** +- Modify: `backend/analytics_server/mhq/store/repos/code.py` +- Modify: `backend/analytics_server/mhq/api/pull_requests.py` +- Test: `backend/analytics_server/tests/service/code/test_contributors.py` (create) + +**Interfaces:** +- Consumes: nothing from Tasks 1-2. +- Produces: + - `is_bot_author(username: str) -> bool` in `mhq/utils/string.py` + - `CodeRepoService.get_contributors_for_repos(repo_ids: List[str], from_time, to_time) -> List[Tuple[str, int]]` returning `(author, pr_count)` ordered by count descending + - `GET /teams//contributors?from_time=&to_time=` returning `[{"username": str, "pr_count": int}]` + +- [ ] **Step 1: Write the failing bot-exclusion tests** + +Create `backend/analytics_server/tests/service/code/test_contributors.py`: + +```python +from mhq.utils.string import is_bot_author + + +def test_bracket_bot_suffix_is_a_bot(): + assert is_bot_author("dependabot[bot]") is True + assert is_bot_author("renovate[bot]") is True + + +def test_known_bot_names_are_bots(): + assert is_bot_author("dependabot") is True + assert is_bot_author("github-actions") is True + + +def test_a_human_whose_name_contains_bot_is_not_a_bot(): + # Substring matching on "bot" would wrongly exclude these people. + assert is_bot_author("robotnik") is False + assert is_bot_author("botond") is False + assert is_bot_author("abbott") is False + + +def test_none_and_empty_are_not_bots(): + assert is_bot_author(None) is False + assert is_bot_author("") is False + + +def test_matching_is_case_insensitive(): + assert is_bot_author("Dependabot[Bot]") is True +``` + +- [ ] **Step 2: Run tests to verify they fail** + +Run: `cd backend/analytics_server && ./venv/bin/python -m pytest tests/service/code/test_contributors.py -v` +Expected: FAIL — `ImportError: cannot import name 'is_bot_author'` + +- [ ] **Step 3: Write the bot predicate** + +Append to `backend/analytics_server/mhq/utils/string.py`: + +```python +# CLUSTOX: bot accounts author real pull requests, so they appear in the +# contributor list unless excluded. Matching is on the exact "[bot]" suffix +# plus an explicit list -- a substring match on "bot" would exclude real +# people called robotnik or abbott. +KNOWN_BOT_AUTHORS = frozenset( + { + "dependabot", + "renovate", + "github-actions", + "snyk-bot", + "greenkeeper", + "codecov", + "imgbot", + } +) + + +def is_bot_author(username: Optional[str]) -> bool: + if not username: + return False + + lowered = username.lower() + return lowered.endswith("[bot]") or lowered in KNOWN_BOT_AUTHORS +``` + +Add `from typing import Optional` to that file's imports if it is not already present. + +- [ ] **Step 4: Run tests to verify they pass** + +Run: `cd backend/analytics_server && ./venv/bin/python -m pytest tests/service/code/test_contributors.py -v` +Expected: PASS (5 tests) + +- [ ] **Step 5: Add the repository query** + +Append to `CodeRepoService` in `backend/analytics_server/mhq/store/repos/code.py`: + +```python + # CLUSTOX: distinct authors for the contributor filter, scoped to the + # repos and window currently on screen so the dropdown can never offer + # someone with no data in view. + def get_contributors_for_repos( + self, repo_ids: List[str], from_time: datetime, to_time: datetime + ) -> List[Tuple[str, int]]: + if not repo_ids: + return [] + + rows = ( + self._db.session.query( + PullRequest.author, func.count(PullRequest.id).label("pr_count") + ) + .filter( + PullRequest.repo_id.in_(repo_ids), + PullRequest.author.isnot(None), + PullRequest.state_changed_at.between(from_time, to_time), + ) + .group_by(PullRequest.author) + .order_by(func.count(PullRequest.id).desc()) + .all() + ) + + return [(author, count) for author, count in rows if not is_bot_author(author)] +``` + +Add whatever of `datetime`, `Tuple`, `func`, and `is_bot_author` that file does not already import. Check its existing imports first rather than adding duplicates. + +- [ ] **Step 6: Add the Flask route** + +Append to `backend/analytics_server/mhq/api/pull_requests.py`. The decorator stack below matches `get_lead_time_prs` in the same file (line 60) — `@queryschema` with a voluptuous `Schema`, coercing the ISO timestamps: + +```python +# CLUSTOX: contributor list backing the DORA metrics filter. Scoped to the +# team's repos and the window on screen, so the dropdown cannot offer someone +# with no data in view. +@app.route("/teams//contributors", methods={"GET"}) +@queryschema( + Schema( + { + Required("from_time"): All(str, Coerce(datetime.fromisoformat)), + Required("to_time"): All(str, Coerce(datetime.fromisoformat)), + } + ), +) +def get_team_contributors(team_id: str, from_time: datetime, to_time: datetime): + query_validator = get_query_validator() + team: Team = query_validator.team_validator(team_id) + + repos = CodeRepoService().get_team_repos(team) + repo_ids = [str(repo.id) for repo in repos] + + contributors = CodeRepoService().get_contributors_for_repos( + repo_ids, from_time, to_time + ) + return [ + {"username": username, "pr_count": pr_count} + for username, pr_count in contributors + ] +``` + +`Schema`, `Required`, `All`, `Coerce`, `queryschema`, `get_query_validator`, `Team` and `CodeRepoService` are all already imported by that file — confirm rather than re-import. + +- [ ] **Step 7: Run the full suite, format, lint, commit** + +```bash +cd backend/analytics_server +./venv/bin/python -m pytest tests -q # expect 230 passed +./venv/bin/python -m black mhq/utils/string.py mhq/store/repos/code.py mhq/api/pull_requests.py tests/service/code/test_contributors.py +./venv/bin/python -m flake8 mhq/utils/string.py mhq/store/repos/code.py mhq/api/pull_requests.py +cd ../.. +git add backend/analytics_server/mhq/utils/string.py \ + backend/analytics_server/mhq/store/repos/code.py \ + backend/analytics_server/mhq/api/pull_requests.py \ + backend/analytics_server/tests/service/code/test_contributors.py +git commit -m "feat(contributors): list a team's pull request authors, bots excluded" +``` + +--- + +### Task 4: Thread the filter through the BFF + +**Files:** +- Create: `web-server/pages/api/internal/team/[team_id]/contributors.ts` +- Modify: `web-server/src/api-helpers/team.ts` +- Modify: `web-server/pages/api/internal/team/[team_id]/dora_metrics.ts` + +**Interfaces:** +- Consumes: the Flask route from Task 3. +- Produces: + - `GET /api/internal/team/{team_id}/contributors?from_date=&to_date=` → `[{username, pr_count}]` + - `updatePrFilterParams(teamId, params, filters)` accepts `filters.authors?: string[]` and emits it inside `pr_filter` + - `dora_metrics.ts` forwards `event_actors` into the workflow filter payload + +- [ ] **Step 1: Create the contributors BFF route** + +Create `web-server/pages/api/internal/team/[team_id]/contributors.ts`: + +```ts +import * as yup from 'yup'; + +import { Endpoint } from '@/api-helpers/global'; +import { internal } from '@/api-helpers/axios'; + +// CLUSTOX: backs the contributor filter on the DORA metrics page. team_id is +// validated by Endpoint.serve(), which asserts the caller may read that team. +const pathSchema = yup.object().shape({ + team_id: yup.string().uuid().required() +}); + +const getSchema = yup.object().shape({ + team_id: yup.string().uuid().required(), + from_date: yup.date().required(), + to_date: yup.date().required() +}); + +const endpoint = new Endpoint(pathSchema); + +endpoint.handle.GET(getSchema, async (req, res) => { + const { team_id, from_date, to_date } = req.payload; + + const contributors = await internal.get(`/teams/${team_id}/contributors`, { + params: { from_time: from_date, to_time: to_date } + }); + + res.send(contributors.data); +}); + +export default endpoint.serve(); +``` + +Note `pathSchema` contains only `team_id` — the same construction bug fixed during the Jenkins work, where passing a method-specific schema as the path schema made every other method a permanent 400. + +- [ ] **Step 2: Accept authors in the PR filter builder** + +In `web-server/src/api-helpers/team.ts`, change `updatePrFilterParams`: + +```ts +export const updatePrFilterParams = async ( + _teamId: ID, + params: T, + filters?: Partial<{ + branches: string; + repo_filters: RepoFilterConfig; + // CLUSTOX: contributor filter -- git usernames, as synced. + authors: string[]; + }> +) => { + const updatedParams = { + base_branches: filters?.branches?.split(','), + repo_filters: filters?.repo_filters, + authors: filters?.authors?.length ? filters.authors : undefined + }; + const reducedParams = reject(isNil, updatedParams); + const pr_filter = equals({}, reducedParams) ? null : reducedParams; + + return reject(isNil, { + ...params, + pr_filter + }) as T & { pr_filter?: Partial }; +}; +``` + +The `?.length ? ... : undefined` matters: an empty array must become `undefined` so `reject(isNil, ...)` drops it. Sending `authors: []` would filter to nobody. + +- [ ] **Step 3: Accept event actors in the workflow filter builder** + +In `web-server/src/utils/filterUtils.ts`, extend `getWorkFlowFiltersAsPayloadForSingleTeam` (line 84). The new parameter is optional, so the other caller — `pages/api/internal/team/[team_id]/deployment_analytics.ts` — is unaffected: + +```ts +export const getWorkFlowFiltersAsPayloadForSingleTeam = async (params: { + orgId: ID; + teamId: ID; + // CLUSTOX: contributor filter. Deployments have no author, so the nearest + // equivalent is the actor who triggered the run. + eventActors?: string[]; +}) => { + const { orgId, teamId, eventActors } = params; + const teamProdBranchesMap = + await getAllTeamsReposProdBranchesForOrgAsMap(orgId); + const filter = Object.fromEntries( + Object.entries(workFlowFiltersFromTeamProdBranches(teamProdBranchesMap)) + )[teamId]; + + if (!eventActors?.length) return filter; + return { ...filter, event_actors: eventActors }; +}; +``` + +Add a case to the existing suite at `web-server/src/utils/__tests__/filterUtils.test.ts` (its `getWorkFlowFiltersAsPayloadForSingleTeam` describe block starts at line 264) asserting that passing `eventActors` adds `event_actors` to the payload, and that omitting it leaves the payload unchanged. + +- [ ] **Step 4: Thread the selection through the DORA route** + +In `web-server/pages/api/internal/team/[team_id]/dora_metrics.ts`, add `authors: yup.array().of(yup.string()).optional()` to the request schema, destructure it alongside the other params, then change the `Promise.all` block at line 66: + +```ts + const [prFilters, workflowFilters] = await Promise.all([ + updatePrFilterParams(teamId, {}, { ...branchAndRepoFilters, authors }).then( + ({ pr_filter }) => ({ + pr_filter + }) + ), + getWorkFlowFiltersAsPayloadForSingleTeam({ + orgId: org_id, + teamId: teamId, + eventActors: authors + }) + ]); +``` + +The same `authors` value feeds both builders — that is what makes one control drive both metrics. + +- [ ] **Step 5: Run the frontend unit tests** + +Run: `docker exec middleware-dev sh -c 'cd /app/web-server && yarn test --testPathPattern=filterUtils'` +Expected: PASS, including the two cases added in Step 3 + +- [ ] **Step 6: Verify types compile** + +```bash +docker cp web-server/pages/api/internal/team/\[team_id\]/contributors.ts middleware-dev:/app/web-server/pages/api/internal/team/\[team_id\]/contributors.ts +docker cp web-server/src/api-helpers/team.ts middleware-dev:/app/web-server/src/api-helpers/team.ts +docker cp web-server/src/utils/filterUtils.ts middleware-dev:/app/web-server/src/utils/filterUtils.ts +docker cp web-server/pages/api/internal/team/\[team_id\]/dora_metrics.ts middleware-dev:/app/web-server/pages/api/internal/team/\[team_id\]/dora_metrics.ts +docker exec middleware-dev sh -c 'cd /app/web-server && ./node_modules/.bin/tsc --noEmit 2>&1 | grep -c "error TS"' +``` +Expected: `0` + +The container now runs this branch, so these files overwrite their own current versions — no restore step is needed, but re-copy from git if you abandon a change. + +- [ ] **Step 7: Commit** + +```bash +git add web-server/pages/api/internal/team/\[team_id\]/contributors.ts \ + web-server/src/api-helpers/team.ts \ + web-server/src/utils/filterUtils.ts \ + web-server/src/utils/__tests__/filterUtils.test.ts \ + web-server/pages/api/internal/team/\[team_id\]/dora_metrics.ts +git commit -m "feat(contributors): expose the contributor list and thread the filter" +``` + +--- + +### Task 5: Contributor dropdown + +**Files:** +- Create: `web-server/src/components/ContributorFilter.tsx` +- Modify: `web-server/pages/dora-metrics/index.tsx` + +**Interfaces:** +- Consumes: `GET /api/internal/team/{team_id}/contributors` from Task 4. +- Produces: nothing consumed by later tasks. + +- [ ] **Step 1: Build the dropdown** + +Create `web-server/src/components/ContributorFilter.tsx`: an MUI `Autocomplete` (multiple, `size="small"`) fetching from `/api/internal/team/${teamId}/contributors` whenever `teamId`, `fromDate` or `toDate` change. Each option shows the username with its PR count as secondary text. Follow the styling conventions in `web-server/src/components/ClustoxWorkspaceMetrics.tsx`. + +Required behaviours: + +- Selection is held in the URL query string, so the filter survives a refresh and can be shared. +- When the fetched list no longer contains a selected username — because the team or date range changed — drop that selection and show a brief inline notice. Do not silently clear it; the numbers would move with no visible cause. +- While loading, disable the control rather than showing an empty list, which reads as "no contributors". +- On fetch failure, disable the control and show the error inline. The dashboard must still render unfiltered — the filter is an enhancement, and losing it should not cost the page. +- When the list is empty, disable the control with the text "No contributors in this range". + +- [ ] **Step 2: Render it and label the metrics** + +In `web-server/pages/dora-metrics/index.tsx`, render `ContributorFilter` alongside the existing selectors and pass the selection into the metrics requests. + +When a contributor is selected, each affected card gains a qualifier: + +- Lead Time: `authored by ` +- Deployment Frequency: `deployed by ` + +These wordings are deliberate and load-bearing. One control drives two different relationships — the PR author for Lead Time, the deploy actor for Deployment Frequency — and those are often different people. Without the qualifiers, a user seeing the two numbers disagree will reasonably conclude the page is broken. + +Change Failure Rate and MTTR are **not** filtered. When a contributor is selected, show on both: `team-wide — per-contributor arrives with Jira`. + +- [ ] **Step 3: Verify types and lint** + +```bash +docker cp web-server/src/components/ContributorFilter.tsx middleware-dev:/app/web-server/src/components/ContributorFilter.tsx +docker cp web-server/pages/dora-metrics/index.tsx middleware-dev:/app/web-server/pages/dora-metrics/index.tsx +docker exec middleware-dev sh -c 'cd /app/web-server && ./node_modules/.bin/tsc --noEmit 2>&1 | grep -c "error TS"' +``` +Expected: `0` + +- [ ] **Step 4: Verify by hand** + +The container runs this branch, so the running app at `http://localhost:3333/dora-metrics` picks these up. Confirm: + +1. The dropdown lists contributors with PR counts, no `[bot]` entries. +2. Selecting one changes Lead Time and Deployment Frequency; CFR and MTTR do not move. +3. Both filtered cards show their qualifier. +4. Refreshing preserves the selection. +5. Narrowing the date range until the selected contributor has no PRs clears the selection with a notice. + +- [ ] **Step 5: Commit** + +```bash +git add web-server/src/components/ContributorFilter.tsx \ + web-server/pages/dora-metrics/index.tsx +git commit -m "feat(contributors): add the contributor filter to dora metrics" +``` + +--- + +## Final verification + +- [ ] `cd backend/analytics_server && ./venv/bin/python -m pytest tests -q` — expect 230 passed +- [ ] `./venv/bin/python -m black --check mhq && ./venv/bin/python -m flake8 mhq` — clean +- [ ] `docker exec middleware-dev sh -c 'cd /app/web-server && ./node_modules/.bin/tsc --noEmit'` — 0 errors +- [ ] Unfiltered dashboards are byte-identical to before this work — the regression tests in Tasks 1 and 2 cover the query layer, but confirm in the running app too + +**The test that matters most** is the one asserting the numbers *change*. The defect being fixed here is a filter that was accepted and never applied; a test that only checks the request succeeds would have passed against the broken behaviour. From f557f33cc8e908ca7f94012b17703d21fafd124a Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Tue, 11 Aug 2026 02:24:46 +0500 Subject: [PATCH 066/194] docs: implementation plan for the DORA contributor filter Five tasks. Lead Time filters on the pull request author, Deployment Frequency on the actor who triggered the run, and the dropdown lists git usernames as synced -- no identity table, no handle merging. Records a defect worth knowing about independently of this feature. PRFilter declares an authors field and ParsePRFilterProcessor already reads it from the request, but filter_query never added it to its conditions dict. Because that comprehension iterates every dataclass field and indexes conditions by name, sending authors raises KeyError rather than filtering: PRFilter(authors=['hamad']).filter_query -> KeyError: 'authors' Both filter classes share that shape, so a field added without its conditions entry reproduces the bug. Tasks 1 and 2 add both halves in one step and the plan says why. Change failure rate and MTTR stay team-level. Per-person versions built on GitHub issue assignees would be a weak foundation to unpick once Jira lands. --- docs/CONTRIBUTOR_FILTER_PLAN.md | 753 ++++++++++++++++++++++++++++++++ 1 file changed, 753 insertions(+) create mode 100644 docs/CONTRIBUTOR_FILTER_PLAN.md diff --git a/docs/CONTRIBUTOR_FILTER_PLAN.md b/docs/CONTRIBUTOR_FILTER_PLAN.md new file mode 100644 index 00000000..2dc797ed --- /dev/null +++ b/docs/CONTRIBUTOR_FILTER_PLAN.md @@ -0,0 +1,753 @@ +# Contributor Filter Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Filter the DORA metrics page by contributor — Lead Time by pull-request author, Deployment Frequency by deploy actor. + +**Architecture:** The dropdown lists distinct git usernames found in the selected team's synced pull requests, bots excluded. Lead Time reuses `PRFilter.authors`, a field upstream declared and parsed but never applied. Deployment Frequency needs a genuinely new `event_actors` filter on `RepoWorkflowRuns`. No new tables, no identity resolution. + +**Tech Stack:** Python 3.9, Flask, SQLAlchemy 2, pytest (backend); Next.js 15 pages router, TypeScript, MUI 5 (frontend). + +## Decisions already made — do not revisit + +| Decision | Choice | +|---|---| +| Identity | Raw git usernames as synced. No mapping table, no merging of handles. A person using two handles appears twice; that is accepted. | +| Metric scope | Lead Time and Deployment Frequency only. **CFR and MTTR stay team-level** — per-person versions wait for the future Jira integration, because GitHub-issue assignees are too weak a foundation. | +| Involvement | **Authored only.** Reviewers are not included. | +| List source | Distinct authors from the selected team's PRs within the active date range, so the dropdown can never offer someone with no data on screen. | +| Bots | Excluded by default, no UI toggle. | + +## Global Constraints + +- **No database migration, no new tables.** +- Backend style: `black` formatted, `flake8` clean. Run from `backend/analytics_server` using `./venv/bin/python`. +- Frontend must be `tsc` clean. Local Node is 20 and the project needs 22 — verify inside the `middleware-dev` container, which has Node 22.16. +- Every Clustox-authored change carries a `# CLUSTOX:` or `// CLUSTOX:` comment explaining **why**, per `docs/FORK_STRATEGY.md`. +- Backend suite baseline is **214 passing**. Do not break any. +- Bot matching is on the exact `[bot]` suffix plus an explicit list — never a substring match on `bot`, which would exclude a real user called `robotnik`. + +## The trap both filter classes share — read before Task 1 or 2 + +`PRFilter.filter_query` and `WorkflowFilter.filter_query` both end with: + +```python +return [ + conditions[x] + for x in self.__dict__.keys() + if getattr(self, x) is not None and conditions[x] is not None +] +``` + +It iterates **every** dataclass field and indexes `conditions` by name. A field with no matching `conditions` key raises `KeyError` the moment that field is set. + +This is not hypothetical. `PRFilter.authors` exists today with no `conditions` entry, so sending `authors` in a filter payload raises `KeyError: 'authors'` — verified: + +``` +PRFilter(authors=['hamad']).filter_query → KeyError: 'authors' +``` + +**Consequence for this plan:** adding a field without adding its `conditions` entry in the same commit reproduces the bug. Both tasks below add both halves together. + +--- + +## File Structure + +**Backend — modify:** +- `mhq/store/models/code/filter.py` — apply `authors` in `PRFilter.filter_query`. +- `mhq/store/models/code/workflows/filter.py` — add `event_actors` to `WorkflowFilter`. +- `mhq/service/workflows/workflow_filter.py` — parse `event_actors` from the request. +- `mhq/store/repos/code.py` — contributor query. +- `mhq/api/pull_requests.py` — contributors route. + +**Backend — create:** +- `tests/store/models/code/test_pr_filter_authors.py` +- `tests/store/models/code/workflows/test_workflow_filter_actors.py` +- `tests/service/code/test_contributors.py` + +**Frontend — create:** +- `web-server/pages/api/internal/team/[team_id]/contributors.ts` +- `web-server/src/components/ContributorFilter.tsx` + +**Frontend — modify:** +- `web-server/src/api-helpers/team.ts` — thread `authors` into `pr_filter`. +- `web-server/pages/api/internal/team/[team_id]/dora_metrics.ts` — thread `event_actors`. +- `web-server/pages/dora-metrics/index.tsx` — render the filter. + +--- + +### Task 1: Apply the author filter (and fix the KeyError) + +**Files:** +- Modify: `backend/analytics_server/mhq/store/models/code/filter.py` +- Test: `backend/analytics_server/tests/store/models/code/test_pr_filter_authors.py` (create) + +**Interfaces:** +- Consumes: nothing. +- Produces: `PRFilter(authors=[...])` yields a `PullRequest.author.in_(authors)` condition. `ParsePRFilterProcessor` already populates the field — no change needed there. + +- [ ] **Step 1: Write the failing tests** + +Create `backend/analytics_server/tests/store/models/code/test_pr_filter_authors.py`: + +```python +from mhq.store.models.code.filter import PRFilter + + +def test_setting_authors_no_longer_raises(): + # Before this change the conditions dict had no "authors" key, so the + # filter_query comprehension raised KeyError as soon as the field was set. + conditions = PRFilter(authors=["hamad-clustox"]).filter_query + assert len(conditions) == 1 + + +def test_authors_produces_an_author_condition(): + conditions = PRFilter(authors=["hamad-clustox"]).filter_query + # Compiling to string is the cheapest way to assert the column and operator + # without a database. + rendered = str(conditions[0]) + assert "author" in rendered + assert "IN" in rendered.upper() + + +def test_no_authors_produces_no_author_condition(): + conditions = PRFilter(base_branches=["^main$"]).filter_query + assert all("author" not in str(c) for c in conditions) + + +def test_empty_author_list_is_ignored(): + # An empty list means "no filter selected", not "match nothing". + conditions = PRFilter(authors=[]).filter_query + assert conditions == [] + + +def test_existing_filters_are_unaffected(): + # Regression guard: the dashboards everyone already uses must not change. + conditions = PRFilter(base_branches=["^main$"], excluded_pr_ids=["a"]).filter_query + assert len(conditions) == 2 +``` + +- [ ] **Step 2: Run tests to verify they fail** + +Run: `cd backend/analytics_server && ./venv/bin/python -m pytest tests/store/models/code/test_pr_filter_authors.py -v` +Expected: FAIL — the first two with `KeyError: 'authors'` + +- [ ] **Step 3: Add the condition** + +In `mhq/store/models/code/filter.py`, inside `filter_query`, add this alongside the other inner functions: + +```python + # CLUSTOX: authors was declared on PRFilter and parsed from the request + # payload upstream, but never added to the conditions dict below -- so + # sending it raised KeyError rather than filtering. This applies it. + def _authors_query(): + if not self.authors: + return None + + return PullRequest.author.in_(self.authors) +``` + +Then add the entry to the `conditions` dict, as the first key: + +```python + conditions = { + "authors": _authors_query(), + "base_branches": _base_branch_query(), + "repo_filters": _repo_filters_query(), + "excluded_pr_ids": _excluded_pr_ids_query(), + "max_cycle_time": _include_prs_below_max_cycle_time(), + "incident_pr_filters": _incident_pr_filters_query(), + } +``` + +- [ ] **Step 4: Run tests to verify they pass** + +Run: `cd backend/analytics_server && ./venv/bin/python -m pytest tests/store/models/code/test_pr_filter_authors.py -v` +Expected: PASS (5 tests) + +- [ ] **Step 5: Run the full suite** + +Run: `cd backend/analytics_server && ./venv/bin/python -m pytest tests -q` +Expected: 219 passed (214 baseline + 5) + +- [ ] **Step 6: Format, lint, commit** + +```bash +cd backend/analytics_server +./venv/bin/python -m black mhq/store/models/code/filter.py tests/store/models/code/test_pr_filter_authors.py +./venv/bin/python -m flake8 mhq/store/models/code/filter.py +cd ../.. +git add backend/analytics_server/mhq/store/models/code/filter.py \ + backend/analytics_server/tests/store/models/code/test_pr_filter_authors.py +git commit -m "fix(filters): apply the pull request author filter" +``` + +--- + +### Task 2: Add the deploy-actor filter + +**Files:** +- Modify: `backend/analytics_server/mhq/store/models/code/workflows/filter.py` +- Modify: `backend/analytics_server/mhq/service/workflows/workflow_filter.py` +- Test: `backend/analytics_server/tests/store/models/code/workflows/test_workflow_filter_actors.py` (create) + +**Interfaces:** +- Consumes: nothing from Task 1. +- Produces: `WorkflowFilter(event_actors=[...])` yields a `RepoWorkflowRuns.event_actor.in_(...)` condition; `ParseWorkflowFilterProcessor` reads `event_actors` from the request dict. + +- [ ] **Step 1: Write the failing tests** + +Create `backend/analytics_server/tests/store/models/code/workflows/test_workflow_filter_actors.py`: + +```python +from mhq.service.workflows.workflow_filter import ParseWorkflowFilterProcessor +from mhq.store.models.code.workflows.filter import WorkflowFilter + + +def test_event_actors_produces_an_actor_condition(): + conditions = WorkflowFilter(event_actors=["hamad-clustox"]).filter_query + assert len(conditions) == 1 + rendered = str(conditions[0]) + assert "event_actor" in rendered + assert "IN" in rendered.upper() + + +def test_no_event_actors_produces_no_actor_condition(): + conditions = WorkflowFilter(head_branches=["^main$"]).filter_query + assert all("event_actor" not in str(c) for c in conditions) + + +def test_empty_actor_list_is_ignored(): + assert WorkflowFilter(event_actors=[]).filter_query == [] + + +def test_existing_workflow_filters_are_unaffected(): + # Regression guard for the deployment frequency everyone already sees. + conditions = WorkflowFilter(head_branches=["^main$"]).filter_query + assert len(conditions) == 1 + + +def test_parser_reads_event_actors_from_the_request(): + parsed = ParseWorkflowFilterProcessor().apply({"event_actors": ["hamad-clustox"]}) + assert parsed.event_actors == ["hamad-clustox"] + + +def test_parser_defaults_event_actors_to_none(): + parsed = ParseWorkflowFilterProcessor().apply({}) + assert parsed.event_actors is None +``` + +- [ ] **Step 2: Run tests to verify they fail** + +Run: `cd backend/analytics_server && ./venv/bin/python -m pytest tests/store/models/code/workflows/test_workflow_filter_actors.py -v` +Expected: FAIL — `TypeError: __init__() got an unexpected keyword argument 'event_actors'` + +- [ ] **Step 3: Add the field and its condition together** + +In `mhq/store/models/code/workflows/filter.py`, change the `WorkflowFilter` dataclass. Add the field **and** the conditions entry in this same step — adding one without the other reproduces the `KeyError` described at the top of this plan: + +```python +@dataclass +class WorkflowFilter: + head_branches: List[str] = None + repo_filters: Dict[str, Dict] = None + # CLUSTOX: who triggered the deployment. Lead Time filters on the pull + # request author; deployments have no author, so the nearest equivalent is + # the actor who ran the build. + event_actors: List[str] = None + + @property + def filter_query(self) -> List: + def _head_branches_query(): + if not self.head_branches: + return None + + return or_( + RepoWorkflowRuns.head_branch.op("~")(term) + for term in self.head_branches + ) + + def _repo_filters_query(): + if not self.repo_filters: + return None + + return or_( + RepoWorkflowFilter(repo_id, repo_filters).filter_query + for repo_id, repo_filters in self.repo_filters.items() + if repo_filters + ) + + # CLUSTOX: runs with a null event_actor -- Jenkins builds without the + # git plugin, for instance -- are excluded by IN, so per-contributor + # deployment counts will not sum to the team total. That is intended: + # an unattributable deploy belongs to nobody. + def _event_actors_query(): + if not self.event_actors: + return None + + return RepoWorkflowRuns.event_actor.in_(self.event_actors) + + conditions = { + "head_branches": _head_branches_query(), + "repo_filters": _repo_filters_query(), + "event_actors": _event_actors_query(), + } + return [ + conditions[x] + for x in self.__dict__.keys() + if getattr(self, x) is not None and conditions[x] is not None + ] +``` + +- [ ] **Step 4: Parse it from the request** + +In `mhq/service/workflows/workflow_filter.py`, extend `ParseWorkflowFilterProcessor.apply`: + +```python + def apply(self, workflow_filter: Dict = None) -> WorkflowFilter: + workflow_filter = workflow_filter or {} + head_branches: List[str] = self._parse_head_branches(workflow_filter) + repo_filters: Dict[str, Dict] = self._parse_repo_filters(workflow_filter) + # CLUSTOX: contributor filter for deployment frequency. + event_actors: List[str] = workflow_filter.get("event_actors") + + return WorkflowFilter( + head_branches=head_branches, + repo_filters=repo_filters, + event_actors=event_actors, + ) +``` + +Note the added `workflow_filter = workflow_filter or {}` guard — the signature defaults to `None` and the existing body would raise `AttributeError` on a bare `.get`. + +- [ ] **Step 5: Run tests to verify they pass** + +Run: `cd backend/analytics_server && ./venv/bin/python -m pytest tests/store/models/code/workflows/test_workflow_filter_actors.py -v` +Expected: PASS (6 tests) + +- [ ] **Step 6: Run the full suite, format, lint, commit** + +```bash +cd backend/analytics_server +./venv/bin/python -m pytest tests -q # expect 225 passed +./venv/bin/python -m black mhq/store/models/code/workflows/filter.py mhq/service/workflows/workflow_filter.py tests/store/models/code/workflows/test_workflow_filter_actors.py +./venv/bin/python -m flake8 mhq/store/models/code/workflows/filter.py mhq/service/workflows/workflow_filter.py +cd ../.. +git add backend/analytics_server/mhq/store/models/code/workflows/filter.py \ + backend/analytics_server/mhq/service/workflows/workflow_filter.py \ + backend/analytics_server/tests/store/models/code/workflows/test_workflow_filter_actors.py +git commit -m "feat(filters): filter deployments by the actor who triggered them" +``` + +--- + +### Task 3: Contributor list query and route + +**Files:** +- Modify: `backend/analytics_server/mhq/store/repos/code.py` +- Modify: `backend/analytics_server/mhq/api/pull_requests.py` +- Test: `backend/analytics_server/tests/service/code/test_contributors.py` (create) + +**Interfaces:** +- Consumes: nothing from Tasks 1-2. +- Produces: + - `is_bot_author(username: str) -> bool` in `mhq/utils/string.py` + - `CodeRepoService.get_contributors_for_repos(repo_ids: List[str], from_time, to_time) -> List[Tuple[str, int]]` returning `(author, pr_count)` ordered by count descending + - `GET /teams//contributors?from_time=&to_time=` returning `[{"username": str, "pr_count": int}]` + +- [ ] **Step 1: Write the failing bot-exclusion tests** + +Create `backend/analytics_server/tests/service/code/test_contributors.py`: + +```python +from mhq.utils.string import is_bot_author + + +def test_bracket_bot_suffix_is_a_bot(): + assert is_bot_author("dependabot[bot]") is True + assert is_bot_author("renovate[bot]") is True + + +def test_known_bot_names_are_bots(): + assert is_bot_author("dependabot") is True + assert is_bot_author("github-actions") is True + + +def test_a_human_whose_name_contains_bot_is_not_a_bot(): + # Substring matching on "bot" would wrongly exclude these people. + assert is_bot_author("robotnik") is False + assert is_bot_author("botond") is False + assert is_bot_author("abbott") is False + + +def test_none_and_empty_are_not_bots(): + assert is_bot_author(None) is False + assert is_bot_author("") is False + + +def test_matching_is_case_insensitive(): + assert is_bot_author("Dependabot[Bot]") is True +``` + +- [ ] **Step 2: Run tests to verify they fail** + +Run: `cd backend/analytics_server && ./venv/bin/python -m pytest tests/service/code/test_contributors.py -v` +Expected: FAIL — `ImportError: cannot import name 'is_bot_author'` + +- [ ] **Step 3: Write the bot predicate** + +Append to `backend/analytics_server/mhq/utils/string.py`: + +```python +# CLUSTOX: bot accounts author real pull requests, so they appear in the +# contributor list unless excluded. Matching is on the exact "[bot]" suffix +# plus an explicit list -- a substring match on "bot" would exclude real +# people called robotnik or abbott. +KNOWN_BOT_AUTHORS = frozenset( + { + "dependabot", + "renovate", + "github-actions", + "snyk-bot", + "greenkeeper", + "codecov", + "imgbot", + } +) + + +def is_bot_author(username: Optional[str]) -> bool: + if not username: + return False + + lowered = username.lower() + return lowered.endswith("[bot]") or lowered in KNOWN_BOT_AUTHORS +``` + +Add `from typing import Optional` to that file's imports if it is not already present. + +- [ ] **Step 4: Run tests to verify they pass** + +Run: `cd backend/analytics_server && ./venv/bin/python -m pytest tests/service/code/test_contributors.py -v` +Expected: PASS (5 tests) + +- [ ] **Step 5: Add the repository query** + +Append to `CodeRepoService` in `backend/analytics_server/mhq/store/repos/code.py`: + +```python + # CLUSTOX: distinct authors for the contributor filter, scoped to the + # repos and window currently on screen so the dropdown can never offer + # someone with no data in view. + def get_contributors_for_repos( + self, repo_ids: List[str], from_time: datetime, to_time: datetime + ) -> List[Tuple[str, int]]: + if not repo_ids: + return [] + + rows = ( + self._db.session.query( + PullRequest.author, func.count(PullRequest.id).label("pr_count") + ) + .filter( + PullRequest.repo_id.in_(repo_ids), + PullRequest.author.isnot(None), + PullRequest.state_changed_at.between(from_time, to_time), + ) + .group_by(PullRequest.author) + .order_by(func.count(PullRequest.id).desc()) + .all() + ) + + return [(author, count) for author, count in rows if not is_bot_author(author)] +``` + +Add whatever of `datetime`, `Tuple`, `func`, and `is_bot_author` that file does not already import. Check its existing imports first rather than adding duplicates. + +- [ ] **Step 6: Add the Flask route** + +Append to `backend/analytics_server/mhq/api/pull_requests.py`. The decorator stack below matches `get_lead_time_prs` in the same file (line 60) — `@queryschema` with a voluptuous `Schema`, coercing the ISO timestamps: + +```python +# CLUSTOX: contributor list backing the DORA metrics filter. Scoped to the +# team's repos and the window on screen, so the dropdown cannot offer someone +# with no data in view. +@app.route("/teams//contributors", methods={"GET"}) +@queryschema( + Schema( + { + Required("from_time"): All(str, Coerce(datetime.fromisoformat)), + Required("to_time"): All(str, Coerce(datetime.fromisoformat)), + } + ), +) +def get_team_contributors(team_id: str, from_time: datetime, to_time: datetime): + query_validator = get_query_validator() + team: Team = query_validator.team_validator(team_id) + + repos = CodeRepoService().get_team_repos(team) + repo_ids = [str(repo.id) for repo in repos] + + contributors = CodeRepoService().get_contributors_for_repos( + repo_ids, from_time, to_time + ) + return [ + {"username": username, "pr_count": pr_count} + for username, pr_count in contributors + ] +``` + +`Schema`, `Required`, `All`, `Coerce`, `queryschema`, `get_query_validator`, `Team` and `CodeRepoService` are all already imported by that file — confirm rather than re-import. + +- [ ] **Step 7: Run the full suite, format, lint, commit** + +```bash +cd backend/analytics_server +./venv/bin/python -m pytest tests -q # expect 230 passed +./venv/bin/python -m black mhq/utils/string.py mhq/store/repos/code.py mhq/api/pull_requests.py tests/service/code/test_contributors.py +./venv/bin/python -m flake8 mhq/utils/string.py mhq/store/repos/code.py mhq/api/pull_requests.py +cd ../.. +git add backend/analytics_server/mhq/utils/string.py \ + backend/analytics_server/mhq/store/repos/code.py \ + backend/analytics_server/mhq/api/pull_requests.py \ + backend/analytics_server/tests/service/code/test_contributors.py +git commit -m "feat(contributors): list a team's pull request authors, bots excluded" +``` + +--- + +### Task 4: Thread the filter through the BFF + +**Files:** +- Create: `web-server/pages/api/internal/team/[team_id]/contributors.ts` +- Modify: `web-server/src/api-helpers/team.ts` +- Modify: `web-server/pages/api/internal/team/[team_id]/dora_metrics.ts` + +**Interfaces:** +- Consumes: the Flask route from Task 3. +- Produces: + - `GET /api/internal/team/{team_id}/contributors?from_date=&to_date=` → `[{username, pr_count}]` + - `updatePrFilterParams(teamId, params, filters)` accepts `filters.authors?: string[]` and emits it inside `pr_filter` + - `dora_metrics.ts` forwards `event_actors` into the workflow filter payload + +- [ ] **Step 1: Create the contributors BFF route** + +Create `web-server/pages/api/internal/team/[team_id]/contributors.ts`: + +```ts +import * as yup from 'yup'; + +import { Endpoint } from '@/api-helpers/global'; +import { internal } from '@/api-helpers/axios'; + +// CLUSTOX: backs the contributor filter on the DORA metrics page. team_id is +// validated by Endpoint.serve(), which asserts the caller may read that team. +const pathSchema = yup.object().shape({ + team_id: yup.string().uuid().required() +}); + +const getSchema = yup.object().shape({ + team_id: yup.string().uuid().required(), + from_date: yup.date().required(), + to_date: yup.date().required() +}); + +const endpoint = new Endpoint(pathSchema); + +endpoint.handle.GET(getSchema, async (req, res) => { + const { team_id, from_date, to_date } = req.payload; + + const contributors = await internal.get(`/teams/${team_id}/contributors`, { + params: { from_time: from_date, to_time: to_date } + }); + + res.send(contributors.data); +}); + +export default endpoint.serve(); +``` + +Note `pathSchema` contains only `team_id` — the same construction bug fixed during the Jenkins work, where passing a method-specific schema as the path schema made every other method a permanent 400. + +- [ ] **Step 2: Accept authors in the PR filter builder** + +In `web-server/src/api-helpers/team.ts`, change `updatePrFilterParams`: + +```ts +export const updatePrFilterParams = async ( + _teamId: ID, + params: T, + filters?: Partial<{ + branches: string; + repo_filters: RepoFilterConfig; + // CLUSTOX: contributor filter -- git usernames, as synced. + authors: string[]; + }> +) => { + const updatedParams = { + base_branches: filters?.branches?.split(','), + repo_filters: filters?.repo_filters, + authors: filters?.authors?.length ? filters.authors : undefined + }; + const reducedParams = reject(isNil, updatedParams); + const pr_filter = equals({}, reducedParams) ? null : reducedParams; + + return reject(isNil, { + ...params, + pr_filter + }) as T & { pr_filter?: Partial }; +}; +``` + +The `?.length ? ... : undefined` matters: an empty array must become `undefined` so `reject(isNil, ...)` drops it. Sending `authors: []` would filter to nobody. + +- [ ] **Step 3: Accept event actors in the workflow filter builder** + +In `web-server/src/utils/filterUtils.ts`, extend `getWorkFlowFiltersAsPayloadForSingleTeam` (line 84). The new parameter is optional, so the other caller — `pages/api/internal/team/[team_id]/deployment_analytics.ts` — is unaffected: + +```ts +export const getWorkFlowFiltersAsPayloadForSingleTeam = async (params: { + orgId: ID; + teamId: ID; + // CLUSTOX: contributor filter. Deployments have no author, so the nearest + // equivalent is the actor who triggered the run. + eventActors?: string[]; +}) => { + const { orgId, teamId, eventActors } = params; + const teamProdBranchesMap = + await getAllTeamsReposProdBranchesForOrgAsMap(orgId); + const filter = Object.fromEntries( + Object.entries(workFlowFiltersFromTeamProdBranches(teamProdBranchesMap)) + )[teamId]; + + if (!eventActors?.length) return filter; + return { ...filter, event_actors: eventActors }; +}; +``` + +Add a case to the existing suite at `web-server/src/utils/__tests__/filterUtils.test.ts` (its `getWorkFlowFiltersAsPayloadForSingleTeam` describe block starts at line 264) asserting that passing `eventActors` adds `event_actors` to the payload, and that omitting it leaves the payload unchanged. + +- [ ] **Step 4: Thread the selection through the DORA route** + +In `web-server/pages/api/internal/team/[team_id]/dora_metrics.ts`, add `authors: yup.array().of(yup.string()).optional()` to the request schema, destructure it alongside the other params, then change the `Promise.all` block at line 66: + +```ts + const [prFilters, workflowFilters] = await Promise.all([ + updatePrFilterParams(teamId, {}, { ...branchAndRepoFilters, authors }).then( + ({ pr_filter }) => ({ + pr_filter + }) + ), + getWorkFlowFiltersAsPayloadForSingleTeam({ + orgId: org_id, + teamId: teamId, + eventActors: authors + }) + ]); +``` + +The same `authors` value feeds both builders — that is what makes one control drive both metrics. + +- [ ] **Step 5: Run the frontend unit tests** + +Run: `docker exec middleware-dev sh -c 'cd /app/web-server && yarn test --testPathPattern=filterUtils'` +Expected: PASS, including the two cases added in Step 3 + +- [ ] **Step 6: Verify types compile** + +```bash +docker cp web-server/pages/api/internal/team/\[team_id\]/contributors.ts middleware-dev:/app/web-server/pages/api/internal/team/\[team_id\]/contributors.ts +docker cp web-server/src/api-helpers/team.ts middleware-dev:/app/web-server/src/api-helpers/team.ts +docker cp web-server/src/utils/filterUtils.ts middleware-dev:/app/web-server/src/utils/filterUtils.ts +docker cp web-server/pages/api/internal/team/\[team_id\]/dora_metrics.ts middleware-dev:/app/web-server/pages/api/internal/team/\[team_id\]/dora_metrics.ts +docker exec middleware-dev sh -c 'cd /app/web-server && ./node_modules/.bin/tsc --noEmit 2>&1 | grep -c "error TS"' +``` +Expected: `0` + +The container now runs this branch, so these files overwrite their own current versions — no restore step is needed, but re-copy from git if you abandon a change. + +- [ ] **Step 7: Commit** + +```bash +git add web-server/pages/api/internal/team/\[team_id\]/contributors.ts \ + web-server/src/api-helpers/team.ts \ + web-server/src/utils/filterUtils.ts \ + web-server/src/utils/__tests__/filterUtils.test.ts \ + web-server/pages/api/internal/team/\[team_id\]/dora_metrics.ts +git commit -m "feat(contributors): expose the contributor list and thread the filter" +``` + +--- + +### Task 5: Contributor dropdown + +**Files:** +- Create: `web-server/src/components/ContributorFilter.tsx` +- Modify: `web-server/pages/dora-metrics/index.tsx` + +**Interfaces:** +- Consumes: `GET /api/internal/team/{team_id}/contributors` from Task 4. +- Produces: nothing consumed by later tasks. + +- [ ] **Step 1: Build the dropdown** + +Create `web-server/src/components/ContributorFilter.tsx`: an MUI `Autocomplete` (multiple, `size="small"`) fetching from `/api/internal/team/${teamId}/contributors` whenever `teamId`, `fromDate` or `toDate` change. Each option shows the username with its PR count as secondary text. Follow the styling conventions in `web-server/src/components/ClustoxWorkspaceMetrics.tsx`. + +Required behaviours: + +- Selection is held in the URL query string, so the filter survives a refresh and can be shared. +- When the fetched list no longer contains a selected username — because the team or date range changed — drop that selection and show a brief inline notice. Do not silently clear it; the numbers would move with no visible cause. +- While loading, disable the control rather than showing an empty list, which reads as "no contributors". +- On fetch failure, disable the control and show the error inline. The dashboard must still render unfiltered — the filter is an enhancement, and losing it should not cost the page. +- When the list is empty, disable the control with the text "No contributors in this range". + +- [ ] **Step 2: Render it and label the metrics** + +In `web-server/pages/dora-metrics/index.tsx`, render `ContributorFilter` alongside the existing selectors and pass the selection into the metrics requests. + +When a contributor is selected, each affected card gains a qualifier: + +- Lead Time: `authored by ` +- Deployment Frequency: `deployed by ` + +These wordings are deliberate and load-bearing. One control drives two different relationships — the PR author for Lead Time, the deploy actor for Deployment Frequency — and those are often different people. Without the qualifiers, a user seeing the two numbers disagree will reasonably conclude the page is broken. + +Change Failure Rate and MTTR are **not** filtered. When a contributor is selected, show on both: `team-wide — per-contributor arrives with Jira`. + +- [ ] **Step 3: Verify types and lint** + +```bash +docker cp web-server/src/components/ContributorFilter.tsx middleware-dev:/app/web-server/src/components/ContributorFilter.tsx +docker cp web-server/pages/dora-metrics/index.tsx middleware-dev:/app/web-server/pages/dora-metrics/index.tsx +docker exec middleware-dev sh -c 'cd /app/web-server && ./node_modules/.bin/tsc --noEmit 2>&1 | grep -c "error TS"' +``` +Expected: `0` + +- [ ] **Step 4: Verify by hand** + +The container runs this branch, so the running app at `http://localhost:3333/dora-metrics` picks these up. Confirm: + +1. The dropdown lists contributors with PR counts, no `[bot]` entries. +2. Selecting one changes Lead Time and Deployment Frequency; CFR and MTTR do not move. +3. Both filtered cards show their qualifier. +4. Refreshing preserves the selection. +5. Narrowing the date range until the selected contributor has no PRs clears the selection with a notice. + +- [ ] **Step 5: Commit** + +```bash +git add web-server/src/components/ContributorFilter.tsx \ + web-server/pages/dora-metrics/index.tsx +git commit -m "feat(contributors): add the contributor filter to dora metrics" +``` + +--- + +## Final verification + +- [ ] `cd backend/analytics_server && ./venv/bin/python -m pytest tests -q` — expect 230 passed +- [ ] `./venv/bin/python -m black --check mhq && ./venv/bin/python -m flake8 mhq` — clean +- [ ] `docker exec middleware-dev sh -c 'cd /app/web-server && ./node_modules/.bin/tsc --noEmit'` — 0 errors +- [ ] Unfiltered dashboards are byte-identical to before this work — the regression tests in Tasks 1 and 2 cover the query layer, but confirm in the running app too + +**The test that matters most** is the one asserting the numbers *change*. The defect being fixed here is a filter that was accepted and never applied; a test that only checks the request succeeds would have passed against the broken behaviour. From 20c32fe6bce18348200498fd74301476b4ed37b0 Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Tue, 11 Aug 2026 02:43:21 +0500 Subject: [PATCH 067/194] fix(filters): apply the pull request author filter Co-Authored-By: Claude Opus 5 --- .../mhq/store/models/code/filter.py | 10 ++++++ .../analytics_server/tests/store/__init__.py | 0 .../tests/store/models/__init__.py | 0 .../tests/store/models/code/__init__.py | 0 .../models/code/test_pr_filter_authors.py | 34 +++++++++++++++++++ 5 files changed, 44 insertions(+) create mode 100644 backend/analytics_server/tests/store/__init__.py create mode 100644 backend/analytics_server/tests/store/models/__init__.py create mode 100644 backend/analytics_server/tests/store/models/code/__init__.py create mode 100644 backend/analytics_server/tests/store/models/code/test_pr_filter_authors.py diff --git a/backend/analytics_server/mhq/store/models/code/filter.py b/backend/analytics_server/mhq/store/models/code/filter.py index c39357ec..d10cea93 100644 --- a/backend/analytics_server/mhq/store/models/code/filter.py +++ b/backend/analytics_server/mhq/store/models/code/filter.py @@ -53,6 +53,15 @@ def _base_branch_query(): @property def filter_query(self) -> List: + # CLUSTOX: authors was declared on PRFilter and parsed from the request + # payload upstream, but never added to the conditions dict below -- so + # sending it raised KeyError rather than filtering. This applies it. + def _authors_query(): + if not self.authors: + return None + + return PullRequest.author.in_(self.authors) + def _base_branch_query(): if not self.base_branches: return None @@ -95,6 +104,7 @@ def _incident_pr_filters_query(): ) conditions = { + "authors": _authors_query(), "base_branches": _base_branch_query(), "repo_filters": _repo_filters_query(), "excluded_pr_ids": _excluded_pr_ids_query(), diff --git a/backend/analytics_server/tests/store/__init__.py b/backend/analytics_server/tests/store/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/backend/analytics_server/tests/store/models/__init__.py b/backend/analytics_server/tests/store/models/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/backend/analytics_server/tests/store/models/code/__init__.py b/backend/analytics_server/tests/store/models/code/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/backend/analytics_server/tests/store/models/code/test_pr_filter_authors.py b/backend/analytics_server/tests/store/models/code/test_pr_filter_authors.py new file mode 100644 index 00000000..25863895 --- /dev/null +++ b/backend/analytics_server/tests/store/models/code/test_pr_filter_authors.py @@ -0,0 +1,34 @@ +from mhq.store.models.code.filter import PRFilter + + +def test_setting_authors_no_longer_raises(): + # Before this change the conditions dict had no "authors" key, so the + # filter_query comprehension raised KeyError as soon as the field was set. + conditions = PRFilter(authors=["hamad-clustox"]).filter_query + assert len(conditions) == 1 + + +def test_authors_produces_an_author_condition(): + conditions = PRFilter(authors=["hamad-clustox"]).filter_query + # Compiling to string is the cheapest way to assert the column and operator + # without a database. + rendered = str(conditions[0]) + assert "author" in rendered + assert "IN" in rendered.upper() + + +def test_no_authors_produces_no_author_condition(): + conditions = PRFilter(base_branches=["^main$"]).filter_query + assert all("author" not in str(c) for c in conditions) + + +def test_empty_author_list_is_ignored(): + # An empty list means "no filter selected", not "match nothing". + conditions = PRFilter(authors=[]).filter_query + assert conditions == [] + + +def test_existing_filters_are_unaffected(): + # Regression guard: the dashboards everyone already uses must not change. + conditions = PRFilter(base_branches=["^main$"], excluded_pr_ids=["a"]).filter_query + assert len(conditions) == 2 From ea3f9b8fb79ab125ce3b70802c4419ef9c33dc0f Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Tue, 11 Aug 2026 02:44:30 +0500 Subject: [PATCH 068/194] docs: correct the test baseline for this branch The plan carried 214 over from the Jenkins branch. This branch is cut from main, which has none of that work, so the baseline is 159 and the expected totals after each task shift accordingly. --- docs/CONTRIBUTOR_FILTER_PLAN.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/CONTRIBUTOR_FILTER_PLAN.md b/docs/CONTRIBUTOR_FILTER_PLAN.md index 2dc797ed..41769ef3 100644 --- a/docs/CONTRIBUTOR_FILTER_PLAN.md +++ b/docs/CONTRIBUTOR_FILTER_PLAN.md @@ -24,7 +24,7 @@ - Backend style: `black` formatted, `flake8` clean. Run from `backend/analytics_server` using `./venv/bin/python`. - Frontend must be `tsc` clean. Local Node is 20 and the project needs 22 — verify inside the `middleware-dev` container, which has Node 22.16. - Every Clustox-authored change carries a `# CLUSTOX:` or `// CLUSTOX:` comment explaining **why**, per `docs/FORK_STRATEGY.md`. -- Backend suite baseline is **214 passing**. Do not break any. +- Backend suite baseline is **159 passing** (this branch is cut from `main`, which does not include the Jenkins work). Do not break any. - Bot matching is on the exact `[bot]` suffix plus an explicit list — never a substring match on `bot`, which would exclude a real user called `robotnik`. ## The trap both filter classes share — read before Task 1 or 2 @@ -168,7 +168,7 @@ Expected: PASS (5 tests) - [ ] **Step 5: Run the full suite** Run: `cd backend/analytics_server && ./venv/bin/python -m pytest tests -q` -Expected: 219 passed (214 baseline + 5) +Expected: 164 passed (159 baseline + 5) - [ ] **Step 6: Format, lint, commit** @@ -329,7 +329,7 @@ Expected: PASS (6 tests) ```bash cd backend/analytics_server -./venv/bin/python -m pytest tests -q # expect 225 passed +./venv/bin/python -m pytest tests -q # expect 170 passed ./venv/bin/python -m black mhq/store/models/code/workflows/filter.py mhq/service/workflows/workflow_filter.py tests/store/models/code/workflows/test_workflow_filter_actors.py ./venv/bin/python -m flake8 mhq/store/models/code/workflows/filter.py mhq/service/workflows/workflow_filter.py cd ../.. @@ -503,7 +503,7 @@ def get_team_contributors(team_id: str, from_time: datetime, to_time: datetime): ```bash cd backend/analytics_server -./venv/bin/python -m pytest tests -q # expect 230 passed +./venv/bin/python -m pytest tests -q # expect 175 passed ./venv/bin/python -m black mhq/utils/string.py mhq/store/repos/code.py mhq/api/pull_requests.py tests/service/code/test_contributors.py ./venv/bin/python -m flake8 mhq/utils/string.py mhq/store/repos/code.py mhq/api/pull_requests.py cd ../.. @@ -745,7 +745,7 @@ git commit -m "feat(contributors): add the contributor filter to dora metrics" ## Final verification -- [ ] `cd backend/analytics_server && ./venv/bin/python -m pytest tests -q` — expect 230 passed +- [ ] `cd backend/analytics_server && ./venv/bin/python -m pytest tests -q` — expect 175 passed - [ ] `./venv/bin/python -m black --check mhq && ./venv/bin/python -m flake8 mhq` — clean - [ ] `docker exec middleware-dev sh -c 'cd /app/web-server && ./node_modules/.bin/tsc --noEmit'` — 0 errors - [ ] Unfiltered dashboards are byte-identical to before this work — the regression tests in Tasks 1 and 2 cover the query layer, but confirm in the running app too From ab38470e1584b9a60708ded4ba9d9d24d0a1e1d2 Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Tue, 11 Aug 2026 02:50:26 +0500 Subject: [PATCH 069/194] feat(filters): filter deployments by the actor who triggered them Co-Authored-By: Claude Opus 5 --- .../mhq/service/workflows/workflow_filter.py | 4 +++ .../mhq/store/models/code/workflows/filter.py | 15 ++++++++ .../store/models/code/workflows/__init__.py | 0 .../workflows/test_workflow_filter_actors.py | 35 +++++++++++++++++++ 4 files changed, 54 insertions(+) create mode 100644 backend/analytics_server/tests/store/models/code/workflows/__init__.py create mode 100644 backend/analytics_server/tests/store/models/code/workflows/test_workflow_filter_actors.py diff --git a/backend/analytics_server/mhq/service/workflows/workflow_filter.py b/backend/analytics_server/mhq/service/workflows/workflow_filter.py index b4466e82..a6eeea44 100644 --- a/backend/analytics_server/mhq/service/workflows/workflow_filter.py +++ b/backend/analytics_server/mhq/service/workflows/workflow_filter.py @@ -8,12 +8,16 @@ class ParseWorkflowFilterProcessor: def apply(self, workflow_filter: Dict = None) -> WorkflowFilter: + workflow_filter = workflow_filter or {} head_branches: List[str] = self._parse_head_branches(workflow_filter) repo_filters: Dict[str, Dict] = self._parse_repo_filters(workflow_filter) + # CLUSTOX: contributor filter for deployment frequency. + event_actors: List[str] = workflow_filter.get("event_actors") return WorkflowFilter( head_branches=head_branches, repo_filters=repo_filters, + event_actors=event_actors, ) def _parse_head_branches(self, workflow_filter: Dict) -> List[str]: diff --git a/backend/analytics_server/mhq/store/models/code/workflows/filter.py b/backend/analytics_server/mhq/store/models/code/workflows/filter.py index baf220ef..7d9ffc5a 100644 --- a/backend/analytics_server/mhq/store/models/code/workflows/filter.py +++ b/backend/analytics_server/mhq/store/models/code/workflows/filter.py @@ -48,6 +48,10 @@ def _head_branches_query(): class WorkflowFilter: head_branches: List[str] = None repo_filters: Dict[str, Dict] = None + # CLUSTOX: who triggered the deployment. Lead Time filters on the pull + # request author; deployments have no author, so the nearest equivalent is + # the actor who ran the build. + event_actors: List[str] = None @property def filter_query(self) -> List: @@ -70,9 +74,20 @@ def _repo_filters_query(): if repo_filters ) + # CLUSTOX: runs with a null event_actor -- Jenkins builds without the + # git plugin, for instance -- are excluded by IN, so per-contributor + # deployment counts will not sum to the team total. That is intended: + # an unattributable deploy belongs to nobody. + def _event_actors_query(): + if not self.event_actors: + return None + + return RepoWorkflowRuns.event_actor.in_(self.event_actors) + conditions = { "head_branches": _head_branches_query(), "repo_filters": _repo_filters_query(), + "event_actors": _event_actors_query(), } return [ conditions[x] diff --git a/backend/analytics_server/tests/store/models/code/workflows/__init__.py b/backend/analytics_server/tests/store/models/code/workflows/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/backend/analytics_server/tests/store/models/code/workflows/test_workflow_filter_actors.py b/backend/analytics_server/tests/store/models/code/workflows/test_workflow_filter_actors.py new file mode 100644 index 00000000..d06074e9 --- /dev/null +++ b/backend/analytics_server/tests/store/models/code/workflows/test_workflow_filter_actors.py @@ -0,0 +1,35 @@ +from mhq.service.workflows.workflow_filter import ParseWorkflowFilterProcessor +from mhq.store.models.code.workflows.filter import WorkflowFilter + + +def test_event_actors_produces_an_actor_condition(): + conditions = WorkflowFilter(event_actors=["hamad-clustox"]).filter_query + assert len(conditions) == 1 + rendered = str(conditions[0]) + assert "event_actor" in rendered + assert "IN" in rendered.upper() + + +def test_no_event_actors_produces_no_actor_condition(): + conditions = WorkflowFilter(head_branches=["^main$"]).filter_query + assert all("event_actor" not in str(c) for c in conditions) + + +def test_empty_actor_list_is_ignored(): + assert WorkflowFilter(event_actors=[]).filter_query == [] + + +def test_existing_workflow_filters_are_unaffected(): + # Regression guard for the deployment frequency everyone already sees. + conditions = WorkflowFilter(head_branches=["^main$"]).filter_query + assert len(conditions) == 1 + + +def test_parser_reads_event_actors_from_the_request(): + parsed = ParseWorkflowFilterProcessor().apply({"event_actors": ["hamad-clustox"]}) + assert parsed.event_actors == ["hamad-clustox"] + + +def test_parser_defaults_event_actors_to_none(): + parsed = ParseWorkflowFilterProcessor().apply({}) + assert parsed.event_actors is None From b9bb75b6419172521e6630fcbf9db95d9b105ae3 Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Tue, 11 Aug 2026 02:55:14 +0500 Subject: [PATCH 070/194] feat(contributors): list a team's pull request authors, bots excluded Co-Authored-By: Claude Opus 5 --- .../analytics_server/mhq/api/pull_requests.py | 30 ++++++++++++++++++- .../analytics_server/mhq/store/repos/code.py | 30 +++++++++++++++++-- backend/analytics_server/mhq/utils/string.py | 26 ++++++++++++++++ .../tests/service/code/test_contributors.py | 27 +++++++++++++++++ 4 files changed, 110 insertions(+), 3 deletions(-) create mode 100644 backend/analytics_server/tests/service/code/test_contributors.py diff --git a/backend/analytics_server/mhq/api/pull_requests.py b/backend/analytics_server/mhq/api/pull_requests.py index 30238746..b82e6cc6 100644 --- a/backend/analytics_server/mhq/api/pull_requests.py +++ b/backend/analytics_server/mhq/api/pull_requests.py @@ -20,6 +20,7 @@ get_non_paginated_pr_response, ) from mhq.store.models.code.pull_requests import PullRequest +from mhq.store.repos.code import CodeRepoService from mhq.service.code.pr_analytics import get_pr_analytics_service from mhq.service.settings.models import ExcludedPRsSetting @@ -30,7 +31,6 @@ from mhq.store.models import SettingType, EntityType - app = Blueprint("pull_requests", __name__) @@ -167,3 +167,31 @@ def get_team_lead_time_trends( week.isoformat(): adapt_lead_time_metrics(average_lead_time_metrics) for week, average_lead_time_metrics in weekly_lead_time_metrics_avg_map.items() } + + +# CLUSTOX: contributor list backing the DORA metrics filter. Scoped to the +# team's repos and the window on screen, so the dropdown cannot offer someone +# with no data in view. +@app.route("/teams//contributors", methods={"GET"}) +@queryschema( + Schema( + { + Required("from_time"): All(str, Coerce(datetime.fromisoformat)), + Required("to_time"): All(str, Coerce(datetime.fromisoformat)), + } + ), +) +def get_team_contributors(team_id: str, from_time: datetime, to_time: datetime): + query_validator = get_query_validator() + team: Team = query_validator.team_validator(team_id) + + repos = CodeRepoService().get_team_repos(str(team.id)) + repo_ids = [str(repo.id) for repo in repos] + + contributors = CodeRepoService().get_contributors_for_repos( + repo_ids, from_time, to_time + ) + return [ + {"username": username, "pr_count": pr_count} + for username, pr_count in contributors + ] diff --git a/backend/analytics_server/mhq/store/repos/code.py b/backend/analytics_server/mhq/store/repos/code.py index 310be35c..1fd4a2b2 100644 --- a/backend/analytics_server/mhq/store/repos/code.py +++ b/backend/analytics_server/mhq/store/repos/code.py @@ -1,9 +1,9 @@ from datetime import datetime from operator import and_ -from typing import Optional, List +from typing import Optional, List, Tuple from mhq.store.models.code.enums import CodeProvider -from sqlalchemy import or_ +from sqlalchemy import or_, func from sqlalchemy.orm import defer from mhq.store.models.core import Team @@ -21,6 +21,7 @@ BookmarkMergeToDeployBroker, CodeBookmarkType, ) +from mhq.utils.string import is_bot_author from mhq.utils.time import Interval @@ -511,3 +512,28 @@ def _filter_base_branch_on_regex(self, query, base_branches: List[str] = None): ] return query.filter(or_(*conditions)) return query + + # CLUSTOX: distinct authors for the contributor filter, scoped to the + # repos and window currently on screen so the dropdown can never offer + # someone with no data in view. + def get_contributors_for_repos( + self, repo_ids: List[str], from_time: datetime, to_time: datetime + ) -> List[Tuple[str, int]]: + if not repo_ids: + return [] + + rows = ( + self._db.session.query( + PullRequest.author, func.count(PullRequest.id).label("pr_count") + ) + .filter( + PullRequest.repo_id.in_(repo_ids), + PullRequest.author.isnot(None), + PullRequest.state_changed_at.between(from_time, to_time), + ) + .group_by(PullRequest.author) + .order_by(func.count(PullRequest.id).desc()) + .all() + ) + + return [(author, count) for author, count in rows if not is_bot_author(author)] diff --git a/backend/analytics_server/mhq/utils/string.py b/backend/analytics_server/mhq/utils/string.py index 64041ef9..ca741b6d 100644 --- a/backend/analytics_server/mhq/utils/string.py +++ b/backend/analytics_server/mhq/utils/string.py @@ -1,5 +1,6 @@ from uuid import uuid4 import re +from typing import Optional def uuid4_str(): @@ -11,3 +12,28 @@ def is_bot_name(name: str) -> bool: r"(?i)(\b[\w@-]*[-_\[\]@ ]+bot[-_\d\[\]]*\b|\[bot\]|_bot_|_bot$|^bot_)" ) return bool(pattern.search(name)) + + +# CLUSTOX: bot accounts author real pull requests, so they appear in the +# contributor list unless excluded. Matching is on the exact "[bot]" suffix +# plus an explicit list -- a substring match on "bot" would exclude real +# people called robotnik or abbott. +KNOWN_BOT_AUTHORS = frozenset( + { + "dependabot", + "renovate", + "github-actions", + "snyk-bot", + "greenkeeper", + "codecov", + "imgbot", + } +) + + +def is_bot_author(username: Optional[str]) -> bool: + if not username: + return False + + lowered = username.lower() + return lowered.endswith("[bot]") or lowered in KNOWN_BOT_AUTHORS diff --git a/backend/analytics_server/tests/service/code/test_contributors.py b/backend/analytics_server/tests/service/code/test_contributors.py new file mode 100644 index 00000000..d35bbd99 --- /dev/null +++ b/backend/analytics_server/tests/service/code/test_contributors.py @@ -0,0 +1,27 @@ +from mhq.utils.string import is_bot_author + + +def test_bracket_bot_suffix_is_a_bot(): + assert is_bot_author("dependabot[bot]") is True + assert is_bot_author("renovate[bot]") is True + + +def test_known_bot_names_are_bots(): + assert is_bot_author("dependabot") is True + assert is_bot_author("github-actions") is True + + +def test_a_human_whose_name_contains_bot_is_not_a_bot(): + # Substring matching on "bot" would wrongly exclude these people. + assert is_bot_author("robotnik") is False + assert is_bot_author("botond") is False + assert is_bot_author("abbott") is False + + +def test_none_and_empty_are_not_bots(): + assert is_bot_author(None) is False + assert is_bot_author("") is False + + +def test_matching_is_case_insensitive(): + assert is_bot_author("Dependabot[Bot]") is True From 6ee45e9ba3fdbbfa2bf0980406be5d090c8127a2 Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Tue, 11 Aug 2026 03:00:20 +0500 Subject: [PATCH 071/194] feat(contributors): expose the contributor list and thread the filter --- .../internal/team/[team_id]/contributors.ts | 30 +++++++++++++++++++ .../internal/team/[team_id]/dora_metrics.ts | 11 ++++--- web-server/src/api-helpers/team.ts | 10 +++++-- .../src/utils/__tests__/filterUtils.test.ts | 23 ++++++++++++++ web-server/src/utils/filterUtils.ts | 10 +++++-- 5 files changed, 76 insertions(+), 8 deletions(-) create mode 100644 web-server/pages/api/internal/team/[team_id]/contributors.ts diff --git a/web-server/pages/api/internal/team/[team_id]/contributors.ts b/web-server/pages/api/internal/team/[team_id]/contributors.ts new file mode 100644 index 00000000..c5af3057 --- /dev/null +++ b/web-server/pages/api/internal/team/[team_id]/contributors.ts @@ -0,0 +1,30 @@ +import * as yup from 'yup'; + +import { Endpoint } from '@/api-helpers/global'; +import { internal } from '@/api-helpers/axios'; + +// CLUSTOX: backs the contributor filter on the DORA metrics page. team_id is +// validated by Endpoint.serve(), which asserts the caller may read that team. +const pathSchema = yup.object().shape({ + team_id: yup.string().uuid().required() +}); + +const getSchema = yup.object().shape({ + team_id: yup.string().uuid().required(), + from_date: yup.date().required(), + to_date: yup.date().required() +}); + +const endpoint = new Endpoint(pathSchema); + +endpoint.handle.GET(getSchema, async (req, res) => { + const { team_id, from_date, to_date } = req.payload; + + const contributors = await internal.get(`/teams/${team_id}/contributors`, { + params: { from_time: from_date, to_time: to_date } + }); + + res.send(contributors.data); +}); + +export default endpoint.serve(); diff --git a/web-server/pages/api/internal/team/[team_id]/dora_metrics.ts b/web-server/pages/api/internal/team/[team_id]/dora_metrics.ts index 9ec5eb28..e7a4d0c9 100644 --- a/web-server/pages/api/internal/team/[team_id]/dora_metrics.ts +++ b/web-server/pages/api/internal/team/[team_id]/dora_metrics.ts @@ -33,7 +33,8 @@ const getSchema = yup.object().shape({ branches: yup.string().optional().nullable(), from_date: yup.date().required(), to_date: yup.date().required(), - branch_mode: yup.string().oneOf(Object.values(ActiveBranchMode)).required() + branch_mode: yup.string().oneOf(Object.values(ActiveBranchMode)).required(), + authors: yup.array().of(yup.string()).optional() }); const endpoint = new Endpoint(pathSchema); @@ -49,7 +50,8 @@ endpoint.handle.GET(getSchema, async (req, res) => { from_date: rawFromDate, to_date: rawToDate, branches, - branch_mode + branch_mode, + authors } = req.payload; const from_date = isoDateString(startOfDay(new Date(rawFromDate))); @@ -64,14 +66,15 @@ endpoint.handle.GET(getSchema, async (req, res) => { getUnsyncedRepos(teamId) ]); const [prFilters, workflowFilters] = await Promise.all([ - updatePrFilterParams(teamId, {}, branchAndRepoFilters).then( + updatePrFilterParams(teamId, {}, { ...branchAndRepoFilters, authors }).then( ({ pr_filter }) => ({ pr_filter }) ), getWorkFlowFiltersAsPayloadForSingleTeam({ orgId: org_id, - teamId: teamId + teamId: teamId, + eventActors: authors }) ]); diff --git a/web-server/src/api-helpers/team.ts b/web-server/src/api-helpers/team.ts index 23065c99..aa2061b5 100644 --- a/web-server/src/api-helpers/team.ts +++ b/web-server/src/api-helpers/team.ts @@ -42,11 +42,17 @@ export const getTeamMembersFilterSettingForOrg = (orgId: ID) => export const updatePrFilterParams = async ( _teamId: ID, params: T, - filters?: Partial<{ branches: string; repo_filters: RepoFilterConfig }> + filters?: Partial<{ + branches: string; + repo_filters: RepoFilterConfig; + // CLUSTOX: contributor filter -- git usernames, as synced. + authors: string[]; + }> ) => { const updatedParams = { base_branches: filters?.branches?.split(','), - repo_filters: filters?.repo_filters + repo_filters: filters?.repo_filters, + authors: filters?.authors?.length ? filters.authors : undefined }; const reducedParams = reject(isNil, updatedParams); const pr_filter = equals({}, reducedParams) ? null : reducedParams; diff --git a/web-server/src/utils/__tests__/filterUtils.test.ts b/web-server/src/utils/__tests__/filterUtils.test.ts index 3ffbd944..810d1a50 100644 --- a/web-server/src/utils/__tests__/filterUtils.test.ts +++ b/web-server/src/utils/__tests__/filterUtils.test.ts @@ -270,4 +270,27 @@ describe('getWorkFlowFiltersAsPayloadForSingleTeam', () => { expect(result).toEqual({ workflow_filter: { head_branches: ['^main$'] } }); }); + + it('should add event_actors to the payload when eventActors is passed', async () => { + const result = await getWorkFlowFiltersAsPayloadForSingleTeam({ + orgId: 'f48d7cce-25d4-41d1-903e-e09166677d92', + teamId: '18d934c1-2699-41bd-af64-c0394ba32fdf', + eventActors: ['octocat'] + }); + + expect(result).toEqual({ + workflow_filter: { head_branches: ['^main$'] }, + event_actors: ['octocat'] + }); + }); + + it('should leave the payload unchanged when eventActors is omitted', async () => { + const result = await getWorkFlowFiltersAsPayloadForSingleTeam({ + orgId: 'f48d7cce-25d4-41d1-903e-e09166677d92', + teamId: '18d934c1-2699-41bd-af64-c0394ba32fdf', + eventActors: [] + }); + + expect(result).toEqual({ workflow_filter: { head_branches: ['^main$'] } }); + }); }); diff --git a/web-server/src/utils/filterUtils.ts b/web-server/src/utils/filterUtils.ts index f0c8f6b7..da868a8b 100644 --- a/web-server/src/utils/filterUtils.ts +++ b/web-server/src/utils/filterUtils.ts @@ -84,11 +84,17 @@ export const getWorkFlowFilters = (params: { export const getWorkFlowFiltersAsPayloadForSingleTeam = async (params: { orgId: ID; teamId: ID; + // CLUSTOX: contributor filter. Deployments have no author, so the nearest + // equivalent is the actor who triggered the run. + eventActors?: string[]; }) => { - const { orgId, teamId } = params; + const { orgId, teamId, eventActors } = params; const teamProdBranchesMap = await getAllTeamsReposProdBranchesForOrgAsMap(orgId); - return Object.fromEntries( + const filter = Object.fromEntries( Object.entries(workFlowFiltersFromTeamProdBranches(teamProdBranchesMap)) )[teamId]; + + if (!eventActors?.length) return filter; + return { ...filter, event_actors: eventActors }; }; From a5cbecf81352cb1aa46b9d4d908ca6601258ca14 Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Tue, 11 Aug 2026 03:21:07 +0500 Subject: [PATCH 072/194] feat(contributors): add the contributor filter to dora metrics Adds the ContributorFilter Autocomplete (URL-persisted selection, stale-drop notice, loading/error/empty states), threads the selection through fetchTeamDoraMetrics as authors[], and labels Lead Time/Deployment Frequency with "authored by"/"deployed by" qualifiers and CFR/MTTR as team-wide, so the two filtered numbers disagreeing with the two unfiltered ones doesn't read as a bug. --- web-server/pages/dora-metrics/index.tsx | 2 + .../src/components/ContributorFilter.tsx | 196 ++++++++++++++++++ .../DoraCards/ChangeFailureRateCard.tsx | 10 + .../DoraMetrics/DoraCards/ChangeTimeCard.tsx | 11 + .../DoraCards/MeanTimeToRestoreCard.tsx | 9 + .../DoraCards/WeeklyDeliveryVolumeCard.tsx | 9 + .../content/DoraMetrics/DoraMetricsBody.tsx | 6 +- web-server/src/slices/dora_metrics.ts | 8 +- 8 files changed, 249 insertions(+), 2 deletions(-) create mode 100644 web-server/src/components/ContributorFilter.tsx diff --git a/web-server/pages/dora-metrics/index.tsx b/web-server/pages/dora-metrics/index.tsx index 8dcb801b..911f40dc 100644 --- a/web-server/pages/dora-metrics/index.tsx +++ b/web-server/pages/dora-metrics/index.tsx @@ -1,6 +1,7 @@ import ExtendedSidebarLayout from 'src/layouts/ExtendedSidebarLayout'; import { Authenticated } from '@/components/Authenticated'; +import { ContributorFilter } from '@/components/ContributorFilter'; import { FlexBox } from '@/components/FlexBox'; import Loader from '@/components/Loader'; import { FetchState } from '@/constants/ui-states'; @@ -27,6 +28,7 @@ function Page() { pageTitle="DORA metrics" isLoading={isLoading} teamDateSelectorMode="single" + additionalFilters={[]} > {integrationList.length > 0 ? : }
diff --git a/web-server/src/components/ContributorFilter.tsx b/web-server/src/components/ContributorFilter.tsx new file mode 100644 index 00000000..516c7997 --- /dev/null +++ b/web-server/src/components/ContributorFilter.tsx @@ -0,0 +1,196 @@ +import { + Autocomplete, + CircularProgress, + ListItem, + TextField +} from '@mui/material'; +import { useRouter } from 'next/router'; +import pluralize from 'pluralize'; +import { useCallback, useEffect, useMemo, useState } from 'react'; + +import { handleApi } from '@/api-helpers/axios-api-instance'; +import { FlexBox } from '@/components/FlexBox'; +import { Line } from '@/components/Text'; +import { useSingleTeamConfig } from '@/hooks/useStateTeamConfig'; + +type Contributor = { + username: string; + pr_count: number; +}; + +const QUERY_KEY = 'contributors'; + +const parseSelectedFromQuery = (raw: string | string[]): string[] => { + const value = Array.isArray(raw) ? raw[0] : raw; + return value ? value.split(',').filter(Boolean) : []; +}; + +/** + * CLUSTOX: single source of truth for the contributor selection. It lives in + * the URL -- unlike the team/date selectors, which are redux-only -- so a + * filtered dashboard survives a refresh and can be shared as a link. Cards + * that need to know "is a contributor selected, and who" read this hook + * directly instead of taking it as a prop, matching how they already source + * team/date themselves (see useCurrentDateRangeLabel and friends). + */ +export const useSelectedContributors = (): string[] => { + const { query } = useRouter(); + // CLUSTOX: keyed off the raw query value (a string), not the whole `query` + // object -- that object gets a new identity on every navigation, including + // unrelated ones like OverlayPageContext pushing its own `overlays` param + // when a "See details" panel opens. Depending on the whole object would + // refetch the DORA summary on every such click. + const raw = query[QUERY_KEY] as string | string[]; + return useMemo(() => parseSelectedFromQuery(raw), [raw]); +}; + +export const ContributorFilter = () => { + const router = useRouter(); + const { singleTeamId, dates } = useSingleTeamConfig(); + const selected = useSelectedContributors(); + + const [contributors, setContributors] = useState([]); + const [loading, setLoading] = useState(false); + const [failed, setFailed] = useState(false); + // CLUSTOX: true once a fetch for the *current* team/date has resolved + // successfully. Gates the stale-selection cleanup below so it can't run + // against a still-loading or still-failed list. + const [settled, setSettled] = useState(false); + const [notice, setNotice] = useState(null); + + const setSelected = useCallback( + (usernames: string[]) => { + const nextQuery: Record = { + ...router.query + }; + if (usernames.length) nextQuery[QUERY_KEY] = usernames.join(','); + else delete nextQuery[QUERY_KEY]; + router.replace({ query: nextQuery }, undefined, { shallow: true }); + }, + [router] + ); + + useEffect(() => { + if (!singleTeamId) { + setContributors([]); + setSettled(false); + return; + } + + let cancelled = false; + setLoading(true); + setFailed(false); + setSettled(false); + + handleApi(`internal/team/${singleTeamId}/contributors`, { + params: { from_date: dates.start, to_date: dates.end } + }) + .then((data) => { + if (cancelled) return; + setContributors(data || []); + setSettled(true); + }) + .catch(() => { + if (cancelled) return; + setContributors([]); + setFailed(true); + }) + .finally(() => { + if (!cancelled) setLoading(false); + }); + + return () => { + cancelled = true; + }; + }, [singleTeamId, dates.start, dates.end]); + + // Drop selections the fresh list no longer backs, visibly -- a silent drop + // would move the DORA numbers on screen with no explanation for why. + useEffect(() => { + if (!settled || !selected.length) return; + + const valid = new Set(contributors.map((c) => c.username)); + const stale = selected.filter((u) => !valid.has(u)); + if (!stale.length) return; + + setSelected(selected.filter((u) => valid.has(u))); + setNotice( + `${stale.join(', ')} ${ + stale.length > 1 ? "aren't" : "isn't" + } active in this range anymore, so ${ + stale.length > 1 ? "they've" : "it's" + } been removed from the filter.` + ); + }, [settled, contributors, selected, setSelected]); + + useEffect(() => { + if (!notice) return; + const timeout = setTimeout(() => setNotice(null), 6000); + return () => clearTimeout(timeout); + }, [notice]); + + const isEmpty = settled && contributors.length === 0; + const disabled = !singleTeamId || loading || failed || isEmpty; + + const value = useMemo( + () => contributors.filter((c) => selected.includes(c.username)), + [contributors, selected] + ); + + const helperText = notice + ? notice + : loading + ? 'Loading contributors…' + : failed + ? 'Could not load contributors.' + : isEmpty + ? 'No contributors in this range' + : undefined; + + return ( + + multiple + size="small" + disabled={disabled} + options={contributors} + value={value} + loading={loading} + sx={{ minWidth: 280 }} + getOptionLabel={(option) => option.username} + isOptionEqualToValue={(option, val) => option.username === val.username} + onChange={(_, newValue) => setSelected(newValue.map((c) => c.username))} + renderOption={(props, option) => ( + + + {option.username} + + {option.pr_count} {pluralize('PR', option.pr_count)} + + + + )} + renderInput={(params) => ( + + {loading ? ( + + ) : null} + {params.InputProps.endAdornment} + + ) + }} + /> + )} + /> + ); +}; diff --git a/web-server/src/content/DoraMetrics/DoraCards/ChangeFailureRateCard.tsx b/web-server/src/content/DoraMetrics/DoraCards/ChangeFailureRateCard.tsx index 27d6cee1..113bf3ec 100644 --- a/web-server/src/content/DoraMetrics/DoraCards/ChangeFailureRateCard.tsx +++ b/web-server/src/content/DoraMetrics/DoraCards/ChangeFailureRateCard.tsx @@ -4,6 +4,7 @@ import { head } from 'ramda'; import { useMemo } from 'react'; import { Chart2, ChartOptions } from '@/components/Chart2'; +import { useSelectedContributors } from '@/components/ContributorFilter'; import { FlexBox } from '@/components/FlexBox'; import { useOverlayPage } from '@/components/OverlayPageContext'; import { Line } from '@/components/Text'; @@ -56,6 +57,7 @@ const chartOptions = { export const ChangeFailureRateCard = () => { const { integrationSet } = useAuth(); const dateRangeLabel = useCurrentDateRangeLabel(); + const selectedContributors = useSelectedContributors(); const { trendsSeriesMap } = useDoraMetricsGraph(); const isCodeProviderIntegrationEnabled = integrationSet.has( @@ -171,6 +173,14 @@ export const ChangeFailureRateCard = () => { )} + {Boolean(selectedContributors.length) && ( + // CLUSTOX: unlike Lead Time and Deployment Frequency, this card + // has no per-contributor breakdown yet -- say so explicitly, or a + // selected filter with an unmoving number reads as broken. + + team-wide — per-contributor arrives with Jira + + )} {canShowIncidentsData ? ( diff --git a/web-server/src/content/DoraMetrics/DoraCards/ChangeTimeCard.tsx b/web-server/src/content/DoraMetrics/DoraCards/ChangeTimeCard.tsx index ac9acaab..7308d7b1 100644 --- a/web-server/src/content/DoraMetrics/DoraCards/ChangeTimeCard.tsx +++ b/web-server/src/content/DoraMetrics/DoraCards/ChangeTimeCard.tsx @@ -7,6 +7,7 @@ import pluralize from 'pluralize'; import { useMemo } from 'react'; import { Chart2, ChartOptions } from '@/components/Chart2'; +import { useSelectedContributors } from '@/components/ContributorFilter'; import { FlexBox } from '@/components/FlexBox'; import { useOverlayPage } from '@/components/OverlayPageContext'; import { Line } from '@/components/Text'; @@ -57,6 +58,7 @@ export const ChangeTimeCard = () => { const { addPage } = useOverlayPage(); const { role } = useAuth(); const isEng = isRoleLessThanEM(role); + const selectedContributors = useSelectedContributors(); const { reposCountWithWorkflowConfigured, @@ -270,6 +272,15 @@ export const ChangeTimeCard = () => { )} + {Boolean(selectedContributors.length) && ( + // CLUSTOX: this card's number is filtered by PR author, while + // Deployment Frequency is filtered by deploy actor -- often a + // different person. Naming the relationship here is what keeps + // the two cards disagreeing from reading as a bug. + + authored by {selectedContributors.join(', ')} + + )} {isSufficientDataAvailable ? ( diff --git a/web-server/src/content/DoraMetrics/DoraCards/MeanTimeToRestoreCard.tsx b/web-server/src/content/DoraMetrics/DoraCards/MeanTimeToRestoreCard.tsx index 67298784..d723c309 100644 --- a/web-server/src/content/DoraMetrics/DoraCards/MeanTimeToRestoreCard.tsx +++ b/web-server/src/content/DoraMetrics/DoraCards/MeanTimeToRestoreCard.tsx @@ -4,6 +4,7 @@ import { head } from 'ramda'; import { useMemo } from 'react'; import { Chart2, ChartOptions } from '@/components/Chart2'; +import { useSelectedContributors } from '@/components/ContributorFilter'; import { FlexBox } from '@/components/FlexBox'; import { useOverlayPage } from '@/components/OverlayPageContext'; import { Line } from '@/components/Text'; @@ -50,6 +51,7 @@ const chartOptions = { export const MeanTimeToRestoreCard = () => { const { isNoDataAvailable, ...meanTimeToRestoreProps } = useMeanTimeToRestoreProps(); + const selectedContributors = useSelectedContributors(); const { trendsSeriesMap } = useDoraMetricsGraph(); const isIncidentProviderIntegrationEnabled = true; @@ -144,6 +146,13 @@ export const MeanTimeToRestoreCard = () => { )} + {Boolean(selectedContributors.length) && ( + // CLUSTOX: see the matching note on ChangeFailureRateCard -- MTTR + // has no per-contributor breakdown yet either. + + team-wide — per-contributor arrives with Jira + + )} {canShowMTRData ? ( diff --git a/web-server/src/content/DoraMetrics/DoraCards/WeeklyDeliveryVolumeCard.tsx b/web-server/src/content/DoraMetrics/DoraCards/WeeklyDeliveryVolumeCard.tsx index a5d1eee5..2e94b5aa 100644 --- a/web-server/src/content/DoraMetrics/DoraCards/WeeklyDeliveryVolumeCard.tsx +++ b/web-server/src/content/DoraMetrics/DoraCards/WeeklyDeliveryVolumeCard.tsx @@ -3,6 +3,7 @@ import pluralize from 'pluralize'; import { useMemo } from 'react'; import { Chart2, ChartOptions } from '@/components/Chart2'; +import { useSelectedContributors } from '@/components/ContributorFilter'; import { FlexBox } from '@/components/FlexBox'; import { useOverlayPage } from '@/components/OverlayPageContext'; import { Line } from '@/components/Text'; @@ -54,6 +55,7 @@ export const WeeklyDeliveryVolumeCard = () => { const { integrationSet } = useAuth(); const dateRangeLabel = useCurrentDateRangeLabel(); const deploymentFrequencyProps = useAvgIntervalBasedDeploymentFrequency(); + const selectedContributors = useSelectedContributors(); const { addPage } = useOverlayPage(); const deploymentsConfigured = true; @@ -172,6 +174,13 @@ export const WeeklyDeliveryVolumeCard = () => { )} + {Boolean(selectedContributors.length) && ( + // CLUSTOX: filtered by deploy actor, not PR author -- see the note + // on ChangeTimeCard for why that distinction has to be visible. + + deployed by {selectedContributors.join(', ')} + + )} {isCodeProviderIntegrationEnabled ? ( diff --git a/web-server/src/content/DoraMetrics/DoraMetricsBody.tsx b/web-server/src/content/DoraMetrics/DoraMetricsBody.tsx index 76e01ba6..11d20c78 100644 --- a/web-server/src/content/DoraMetrics/DoraMetricsBody.tsx +++ b/web-server/src/content/DoraMetrics/DoraMetricsBody.tsx @@ -4,6 +4,7 @@ import Link from 'next/link'; import { FC, useEffect, useMemo } from 'react'; import { AiButton } from '@/components/AiButton'; +import { useSelectedContributors } from '@/components/ContributorFilter'; import { DoraMetricsConfigurationSettings } from '@/components/DoraMetricsConfigurationSettings'; import { DoraScoreV2 } from '@/components/DoraScoreV2'; import { EmptyState } from '@/components/EmptyState'; @@ -44,6 +45,7 @@ export const DoraMetricsBody = () => { ); const { singleTeamId, dates } = useSingleTeamConfig(); const branchPayloadForPrFilters = useBranchesForPrFilters(); + const selectedContributors = useSelectedContributors(); const isLoading = useSelector( (s) => s.doraMetrics.requests?.metrics_summary === FetchState.REQUEST ); @@ -73,6 +75,7 @@ export const DoraMetricsBody = () => { teamId: singleTeamId, fromDate: dates.start, toDate: dates.end, + authors: selectedContributors, ...branchPayloadForPrFilters }) ); @@ -83,7 +86,8 @@ export const DoraMetricsBody = () => { orgId, singleTeamId, isCodeProviderIntegrated, - branchPayloadForPrFilters + branchPayloadForPrFilters, + selectedContributors ]); const stats = useDoraStats(); diff --git a/web-server/src/slices/dora_metrics.ts b/web-server/src/slices/dora_metrics.ts index bfb5a5ef..36d2ef2f 100644 --- a/web-server/src/slices/dora_metrics.ts +++ b/web-server/src/slices/dora_metrics.ts @@ -179,6 +179,11 @@ export const fetchTeamDoraMetrics = createAsyncThunk( toDate: Date; branches: string; branch_mode?: ActiveBranchMode; + // CLUSTOX: contributor filter -- git usernames. Sent under the + // bracketed key so the BFF's array-param transform (`key[]` -> + // `key: value[]`) picks it up; a plain `authors` key would arrive + // server-side as a lone string instead of an array. + authors?: string[]; }) => { return await handleApi( `internal/team/${params.teamId}/dora_metrics`, @@ -188,7 +193,8 @@ export const fetchTeamDoraMetrics = createAsyncThunk( from_date: params.fromDate, to_date: params.toDate, branches: params.branches, - branch_mode: params.branch_mode || ActiveBranchMode.ALL + branch_mode: params.branch_mode || ActiveBranchMode.ALL, + 'authors[]': params.authors?.length ? params.authors : undefined } } ); From 913bb635f670169dfb20be45b009ffe994522336 Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Tue, 11 Aug 2026 03:37:09 +0500 Subject: [PATCH 073/194] fix(contributors): stop CFR and MTTR from being filtered by contributor dora_metrics.ts was passing the same contributor-filtered prFilter/ workflowFilter to all four metric fetches. PRFilter.authors and the workflow filter's event_actors reach the incident and PR-based deployment lookups MTTR and CFR consume, so selecting a contributor silently moved those two numbers too -- while their cards say "team-wide". Strip the contributor keys (authors, event_actors) before those two fetches only; Lead Time and Deployment Frequency keep the filtered versions. Co-Authored-By: Claude Opus 5 --- .../internal/team/[team_id]/dora_metrics.ts | 12 ++- .../__tests__/contributorFilters.test.ts | 81 +++++++++++++++++++ web-server/src/utils/contributorFilters.ts | 30 +++++++ 3 files changed, 120 insertions(+), 3 deletions(-) create mode 100644 web-server/src/utils/__tests__/contributorFilters.test.ts create mode 100644 web-server/src/utils/contributorFilters.ts diff --git a/web-server/pages/api/internal/team/[team_id]/dora_metrics.ts b/web-server/pages/api/internal/team/[team_id]/dora_metrics.ts index e7a4d0c9..ce03fafb 100644 --- a/web-server/pages/api/internal/team/[team_id]/dora_metrics.ts +++ b/web-server/pages/api/internal/team/[team_id]/dora_metrics.ts @@ -16,6 +16,7 @@ import { fetchMeanTimeToRestoreStats, fetchDeploymentFrequencyStats } from '@/utils/cockpitMetricUtils'; +import { stripContributorFilters } from '@/utils/contributorFilters'; import { isoDateString, getAggregateAndTrendsIntervalTime } from '@/utils/date'; import { getBranchesAndRepoFilter, @@ -78,6 +79,11 @@ endpoint.handle.GET(getSchema, async (req, res) => { }) ]); + const { + prFilter: prFilterWithoutAuthors, + workflowFilter: workflowFilterWithoutEventActors + } = stripContributorFilters(prFilters, workflowFilters); + const { currTrendsTimeObject, prevTrendsTimeObject, @@ -121,7 +127,7 @@ endpoint.handle.GET(getSchema, async (req, res) => { }, currTrendsTimeObject, prevTrendsTimeObject, - prFilter: prFilters + prFilter: prFilterWithoutAuthors }), fetchChangeFailureRateStats({ teamId, @@ -135,8 +141,8 @@ endpoint.handle.GET(getSchema, async (req, res) => { }, currTrendsTimeObject, prevTrendsTimeObject, - prFilter: prFilters, - workflowFilter: workflowFilters + prFilter: prFilterWithoutAuthors, + workflowFilter: workflowFilterWithoutEventActors }), fetchDeploymentFrequencyStats({ teamId, diff --git a/web-server/src/utils/__tests__/contributorFilters.test.ts b/web-server/src/utils/__tests__/contributorFilters.test.ts new file mode 100644 index 00000000..a49d1e92 --- /dev/null +++ b/web-server/src/utils/__tests__/contributorFilters.test.ts @@ -0,0 +1,81 @@ +import { stripContributorFilters } from '@/utils/contributorFilters'; + +// CLUSTOX: pins the fix for a bug where Change Failure Rate and MTTR were +// silently filtered by contributor even though their cards say "team-wide". +// `stripContributorFilters` is what keeps CFR/MTTR's request payload free of +// the contributor keys while leaving every other filter (base branches, repo +// filters, prod-branch workflow config) untouched -- exactly what the two +// unaffected fetches (Lead Time, Deployment Frequency) still receive +// unfiltered-of-nothing, i.e. with the contributor keys intact. +describe('stripContributorFilters', () => { + it('drops only `authors` from the pr filter, keeping everything else', () => { + const prFilters = { + pr_filter: { + authors: ['alice', 'bob'], + base_branches: ['main'], + repo_filters: { 'repo-1': { base_branches: ['main'] } } + } + }; + + const { prFilter } = stripContributorFilters(prFilters, null); + + expect(prFilter).toEqual({ + pr_filter: { + base_branches: ['main'], + repo_filters: { 'repo-1': { base_branches: ['main'] } } + } + }); + expect(prFilter.pr_filter).not.toHaveProperty('authors'); + }); + + it('drops only `event_actors` from the workflow filter, keeping everything else', () => { + const workflowFilters = { + event_actors: ['alice'], + 'repo-1': { base_branches: ['main'] } + }; + + const { workflowFilter } = stripContributorFilters( + { pr_filter: null }, + workflowFilters + ); + + expect(workflowFilter).toEqual({ + 'repo-1': { base_branches: ['main'] } + }); + expect(workflowFilter).not.toHaveProperty('event_actors'); + }); + + it('leaves a null pr_filter as-is when no other filters are set', () => { + const prFilters: { pr_filter: Record | null } = { + pr_filter: null + }; + + const { prFilter } = stripContributorFilters(prFilters, undefined); + + expect(prFilter).toBe(prFilters); + }); + + it('leaves an unset workflow filter as-is', () => { + const { workflowFilter } = stripContributorFilters( + { pr_filter: null }, + undefined + ); + + expect(workflowFilter).toBeUndefined(); + }); + + it('is a no-op when no contributor was selected', () => { + const prFilters = { + pr_filter: { base_branches: ['main'] } + }; + const workflowFilters = { 'repo-1': { base_branches: ['main'] } }; + + const { prFilter, workflowFilter } = stripContributorFilters( + prFilters, + workflowFilters + ); + + expect(prFilter).toEqual(prFilters); + expect(workflowFilter).toEqual(workflowFilters); + }); +}); diff --git a/web-server/src/utils/contributorFilters.ts b/web-server/src/utils/contributorFilters.ts new file mode 100644 index 00000000..47dc0a83 --- /dev/null +++ b/web-server/src/utils/contributorFilters.ts @@ -0,0 +1,30 @@ +import { omit } from 'ramda'; + +// CLUSTOX: Change Failure Rate and MTTR have no defensible per-contributor +// definition yet -- the nearest analog would be filtering incidents by +// GitHub issue assignee, which is a weak foundation to build on and unpick +// once Jira-based incident ownership lands. So these two keep every other +// filter (base branches, repo filters, prod-branch workflow config) but +// never see the contributor selection, unlike Lead Time and Deployment +// Frequency. +// +// Lives in its own file, deliberately import-light (just ramda), so the +// "only the contributor keys are stripped, everything else passes through" +// contract can be unit tested without dragging in the dora_metrics route's +// full dependency graph (db, auth, `@/utils/date` -> `@/utils/mock` -> +// `@faker-js/faker`, which is ESM-only and breaks under this repo's Jest +// config). +export const stripContributorFilters = < + PrFilters extends { pr_filter: Record | null }, + WorkflowFilters extends Record | null | undefined +>( + prFilters: PrFilters, + workflowFilters: WorkflowFilters +) => ({ + prFilter: prFilters.pr_filter + ? { pr_filter: omit(['authors'], prFilters.pr_filter) } + : prFilters, + workflowFilter: workflowFilters + ? omit(['event_actors'], workflowFilters) + : workflowFilters +}); From 32c2b76b565e695554374fb368729d1cd14bb29b Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Tue, 11 Aug 2026 14:56:13 +0500 Subject: [PATCH 074/194] fix(contributors): nest event_actors inside the workflow_filter blob `getWorkFlowFiltersAsPayloadForSingleTeam` attached `event_actors` as a sibling of `workflow_filter`, so it arrived at Flask as a top-level query param. `ParseWorkflowFilterProcessor.apply` never saw it, and the PREVENT_EXTRA schema on /teams//deployment_frequency rejected the request outright -- selecting any contributor 400'd the deployment frequency fetch, which rejected the BFF's Promise.all and replaced the whole DORA page with SomethingWentWrong. Nest it where the backend reads it, and strip it from the same level for CFR/MTTR -- stripping the top level was a no-op once the key moved. Both tests previously asserted the broken shape, which is how this passed review five times. They now assert against what workflow_filter.py parses. Co-Authored-By: Claude Opus 5 --- .../__tests__/contributorFilters.test.ts | 22 ++++++++++++++----- .../src/utils/__tests__/filterUtils.test.ts | 22 ++++++++++++++++--- web-server/src/utils/contributorFilters.ts | 11 ++++++++-- web-server/src/utils/filterUtils.ts | 13 ++++++++++- 4 files changed, 56 insertions(+), 12 deletions(-) diff --git a/web-server/src/utils/__tests__/contributorFilters.test.ts b/web-server/src/utils/__tests__/contributorFilters.test.ts index a49d1e92..2aae9a71 100644 --- a/web-server/src/utils/__tests__/contributorFilters.test.ts +++ b/web-server/src/utils/__tests__/contributorFilters.test.ts @@ -28,10 +28,18 @@ describe('stripContributorFilters', () => { expect(prFilter.pr_filter).not.toHaveProperty('authors'); }); - it('drops only `event_actors` from the workflow filter, keeping everything else', () => { + // The shape below is the one the backend actually parses: `event_actors` + // lives *inside* the `workflow_filter` blob, alongside `head_branches`, + // because ParseWorkflowFilterProcessor.apply + // (mhq/service/workflows/workflow_filter.py) reads + // `workflow_filter.get("event_actors")` off the parsed blob. Stripping the + // top level would leave the key in place and CFR would stay filtered. + it('drops only `event_actors` from inside workflow_filter, keeping everything else', () => { const workflowFilters = { - event_actors: ['alice'], - 'repo-1': { base_branches: ['main'] } + workflow_filter: { + head_branches: ['^main$'], + event_actors: ['alice'] + } }; const { workflowFilter } = stripContributorFilters( @@ -40,9 +48,9 @@ describe('stripContributorFilters', () => { ); expect(workflowFilter).toEqual({ - 'repo-1': { base_branches: ['main'] } + workflow_filter: { head_branches: ['^main$'] } }); - expect(workflowFilter).not.toHaveProperty('event_actors'); + expect(workflowFilter.workflow_filter).not.toHaveProperty('event_actors'); }); it('leaves a null pr_filter as-is when no other filters are set', () => { @@ -68,7 +76,9 @@ describe('stripContributorFilters', () => { const prFilters = { pr_filter: { base_branches: ['main'] } }; - const workflowFilters = { 'repo-1': { base_branches: ['main'] } }; + const workflowFilters = { + workflow_filter: { head_branches: ['^main$'] } + }; const { prFilter, workflowFilter } = stripContributorFilters( prFilters, diff --git a/web-server/src/utils/__tests__/filterUtils.test.ts b/web-server/src/utils/__tests__/filterUtils.test.ts index 810d1a50..a8690033 100644 --- a/web-server/src/utils/__tests__/filterUtils.test.ts +++ b/web-server/src/utils/__tests__/filterUtils.test.ts @@ -271,7 +271,18 @@ describe('getWorkFlowFiltersAsPayloadForSingleTeam', () => { expect(result).toEqual({ workflow_filter: { head_branches: ['^main$'] } }); }); - it('should add event_actors to the payload when eventActors is passed', async () => { + // CLUSTOX: contributor filter. These two cases pin the payload against the + // *backend's* parser rather than against this module's own output. + // `ParseWorkflowFilterProcessor.apply` in + // backend/analytics_server/mhq/service/workflows/workflow_filter.py reads + // the JSON blob it is handed and does `workflow_filter.get("event_actors")` + // on it -- the same level it reads `head_branches` from. That blob is the + // value of the `workflow_filter` query param, so `event_actors` must be a + // sibling of `head_branches`, never a sibling of `workflow_filter`. A + // top-level key would also be an extra query param, which the PREVENT_EXTRA + // voluptuous schema on /teams//deployment_frequency rejects with a + // 400. + it('should nest event_actors inside workflow_filter, where the backend parser reads it', async () => { const result = await getWorkFlowFiltersAsPayloadForSingleTeam({ orgId: 'f48d7cce-25d4-41d1-903e-e09166677d92', teamId: '18d934c1-2699-41bd-af64-c0394ba32fdf', @@ -279,9 +290,14 @@ describe('getWorkFlowFiltersAsPayloadForSingleTeam', () => { }); expect(result).toEqual({ - workflow_filter: { head_branches: ['^main$'] }, - event_actors: ['octocat'] + workflow_filter: { + head_branches: ['^main$'], + event_actors: ['octocat'] + } }); + // Guard the exact failure that shipped: a top-level sibling key is an + // unrecognised query param and 400s the whole deployment frequency call. + expect(result).not.toHaveProperty('event_actors'); }); it('should leave the payload unchanged when eventActors is omitted', async () => { diff --git a/web-server/src/utils/contributorFilters.ts b/web-server/src/utils/contributorFilters.ts index 47dc0a83..cb2e5ae0 100644 --- a/web-server/src/utils/contributorFilters.ts +++ b/web-server/src/utils/contributorFilters.ts @@ -24,7 +24,14 @@ export const stripContributorFilters = < prFilter: prFilters.pr_filter ? { pr_filter: omit(['authors'], prFilters.pr_filter) } : prFilters, - workflowFilter: workflowFilters - ? omit(['event_actors'], workflowFilters) + // `event_actors` sits inside the `workflow_filter` blob, because that is the + // level the backend parses it at (ParseWorkflowFilterProcessor.apply reads + // `workflow_filter.get("event_actors")`). So it has to be stripped from + // there too -- stripping the top level would silently do nothing. + workflowFilter: workflowFilters?.workflow_filter + ? { + ...workflowFilters, + workflow_filter: omit(['event_actors'], workflowFilters.workflow_filter) + } : workflowFilters }); diff --git a/web-server/src/utils/filterUtils.ts b/web-server/src/utils/filterUtils.ts index da868a8b..87b3757e 100644 --- a/web-server/src/utils/filterUtils.ts +++ b/web-server/src/utils/filterUtils.ts @@ -96,5 +96,16 @@ export const getWorkFlowFiltersAsPayloadForSingleTeam = async (params: { )[teamId]; if (!eventActors?.length) return filter; - return { ...filter, event_actors: eventActors }; + // CLUSTOX: `event_actors` belongs *inside* the workflow_filter blob, next to + // `head_branches`. The backend reads it at that level -- see + // ParseWorkflowFilterProcessor.apply in + // mhq/service/workflows/workflow_filter.py, which does + // `workflow_filter.get("event_actors")` on the parsed blob. Hanging it off + // the top level instead makes it a sibling query param, and the + // PREVENT_EXTRA voluptuous schema on /teams//deployment_frequency + // rejects the whole request. + return { + ...filter, + workflow_filter: { ...filter?.workflow_filter, event_actors: eventActors } + }; }; From 34daf2f9e06b720f92a71dd8cad78d2d523bd54d Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Tue, 11 Aug 2026 14:56:54 +0500 Subject: [PATCH 075/194] fix(contributors): keep the shared summary_prs list team-wide `lead_time_prs` becomes `summary_prs` in redux, and the Change Failure Rate "See details" overlay divides an unfiltered revert count by its length. Passing the author-filtered pr_filter meant 200 team PRs with 12 reverts, filtered to a contributor with 8 PRs, rendered "12 PRs (150%) reverted after merge" on a panel opened from a card labelled team-wide. TeamInsightsBody reads the same slice and had the same skew. Feed it the stripped filter. The lead time drill-down list is no longer per-contributor; the lead time metric itself still is. Co-Authored-By: Claude Opus 5 --- .../pages/api/internal/team/[team_id]/dora_metrics.ts | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/web-server/pages/api/internal/team/[team_id]/dora_metrics.ts b/web-server/pages/api/internal/team/[team_id]/dora_metrics.ts index ce03fafb..f7f9c347 100644 --- a/web-server/pages/api/internal/team/[team_id]/dora_metrics.ts +++ b/web-server/pages/api/internal/team/[team_id]/dora_metrics.ts @@ -159,7 +159,16 @@ endpoint.handle.GET(getSchema, async (req, res) => { workflowFilter: workflowFilters, prFilter: prFilters }), - getTeamLeadTimePRs(teamId, from_date, to_date, prFilters).then( + // CLUSTOX: deliberately the *unfiltered* pr filter. This list lands in + // redux as `summary_prs`, which has a second consumer: the Change Failure + // Rate "See details" overlay (content/DoraMetrics/Incidents.tsx) uses it + // as the denominator in `percent(revertedPrCount, prs.length)` against an + // unfiltered revert count. Narrowing it to one contributor there reports + // e.g. 12 reverts over 8 PRs = 150% on a card labelled team-wide. + // TeamInsightsBody reads the same slice. One slice cannot be both + // per-contributor and team-wide, so it stays team-wide; the lead time + // *metric* above is still filtered. + getTeamLeadTimePRs(teamId, from_date, to_date, prFilterWithoutAuthors).then( (r) => r.data ), getTeamRepos(teamId) From 999ed8233c31a4ca7329741cd8dbe0c991b97a62 Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Tue, 11 Aug 2026 14:59:44 +0500 Subject: [PATCH 076/194] fix(contributors): keep the selection across every DORA refetch path Three of the four `fetchTeamDoraMetrics` dispatches rebuilt their own argument object and never learned about `authors`: the page refresh callback (fired by useSyncedRepos when a repo finishes syncing), the production branch selector's post-save refetch, and the incident PR filter's. Any of them silently reverted Lead Time and Deployment Frequency to team-wide while the dropdown still showed the chip and the cards still read "authored by alice". Build the org/team/date/contributor arguments once, in useDoraMetricsFetchArgs, and spread them at all four sites. Branch arguments stay per-caller -- the body and refresh callback send branch_mode, the two dialogs never did, and unifying that would change what unfiltered dashboards request. Co-Authored-By: Claude Opus 5 --- .../src/components/TeamIncidentPRsFilter.tsx | 15 +++---- .../TeamProductionBranchSelector.tsx | 19 ++++----- .../content/DoraMetrics/DoraMetricsBody.tsx | 24 +++++------ .../src/hooks/useDoraMetricsFetchArgs.ts | 40 +++++++++++++++++++ .../src/hooks/usePageRefreshCallback.ts | 18 ++++----- 5 files changed, 74 insertions(+), 42 deletions(-) create mode 100644 web-server/src/hooks/useDoraMetricsFetchArgs.ts diff --git a/web-server/src/components/TeamIncidentPRsFilter.tsx b/web-server/src/components/TeamIncidentPRsFilter.tsx index fe3202bc..6899b408 100644 --- a/web-server/src/components/TeamIncidentPRsFilter.tsx +++ b/web-server/src/components/TeamIncidentPRsFilter.tsx @@ -19,7 +19,8 @@ import { FC, useEffect } from 'react'; import { FormProvider, useForm } from 'react-hook-form'; import * as yup from 'yup'; -import { useAuth } from '@/hooks/useAuth'; +// CLUSTOX: contributor filter -- shared fetch arguments, see the hook's note. +import { useDoraMetricsFetchArgs } from '@/hooks/useDoraMetricsFetchArgs'; import { useEasyState } from '@/hooks/useEasyState'; import { useSingleTeamConfig, @@ -90,8 +91,8 @@ export const TeamIncidentPRsFilter: FC<{ onClose: () => void; }> = ({ onClose }) => { const dispatch = useDispatch(); - const { orgId } = useAuth(); - const { singleTeamId, dates } = useSingleTeamConfig(); + const { singleTeamId } = useSingleTeamConfig(); + const doraMetricsFetchArgs = useDoraMetricsFetchArgs(); const branches = useStateBranchConfig(); const isSaving = useEasyState(false); const { enqueueSnackbar } = useSnackbar(); @@ -138,11 +139,11 @@ export const TeamIncidentPRsFilter: FC<{ await dispatch(updateTeamIncidentPRsFilter(updateConfArgs)).then( async (response) => { if (response.meta.requestStatus === 'fulfilled') { + // CLUSTOX: contributor filter -- saving the incident PR filter + // refetches DORA, and the selection has to survive that. Shared + // hook, so a future argument can't be missed here again. const fetchDoraArgs = { - orgId: orgId, - teamId: singleTeamId, - fromDate: dates.start, - toDate: dates.end, + ...doraMetricsFetchArgs, branches: activeBranchMode === ActiveBranchMode.PROD ? null : branches }; diff --git a/web-server/src/components/TeamProductionBranchSelector.tsx b/web-server/src/components/TeamProductionBranchSelector.tsx index 10f1b0c1..b599cc28 100644 --- a/web-server/src/components/TeamProductionBranchSelector.tsx +++ b/web-server/src/components/TeamProductionBranchSelector.tsx @@ -12,6 +12,8 @@ import { track } from '@/constants/events'; import { ROUTES } from '@/constants/routes'; import { isRoleLessThanEM } from '@/constants/useRoute'; import { useAuth } from '@/hooks/useAuth'; +// CLUSTOX: contributor filter -- shared fetch arguments, see the hook's note. +import { useDoraMetricsFetchArgs } from '@/hooks/useDoraMetricsFetchArgs'; import { useEasyState } from '@/hooks/useEasyState'; import { useSingleTeamConfig, @@ -58,8 +60,9 @@ export const TeamProductionBranchSelector: FC<{ onClose: () => void; }> = ({ onClose }) => { const dispatch = useDispatch(); - const { orgId, integrationSet } = useAuth(); - const { singleTeamId, dates } = useSingleTeamConfig(); + const { integrationSet } = useAuth(); + const { singleTeamId } = useSingleTeamConfig(); + const doraMetricsFetchArgs = useDoraMetricsFetchArgs(); const branches = useStateBranchConfig(); const isCodeIntegrationLinked = integrationSet.has(IntegrationGroup.CODE); const isSaving = useEasyState(false); @@ -149,11 +152,11 @@ export const TeamProductionBranchSelector: FC<{ ); } + // CLUSTOX: contributor filter -- saving prod branches refetches + // DORA, and the selection has to survive that. Shared hook, so a + // future argument can't be missed here again. const fetchDoraArgs = { - orgId: orgId, - teamId: singleTeamId, - fromDate: dates.start, - toDate: dates.end, + ...doraMetricsFetchArgs, branches: activeBranchMode === ActiveBranchMode.PROD ? null : branches }; @@ -182,9 +185,7 @@ export const TeamProductionBranchSelector: FC<{ dispatch, onClose, activeBranchMode, - orgId, - dates.start, - dates.end, + doraMetricsFetchArgs, branches, enqueueSnackbar, updateActiveProdBranch diff --git a/web-server/src/content/DoraMetrics/DoraMetricsBody.tsx b/web-server/src/content/DoraMetrics/DoraMetricsBody.tsx index 11d20c78..d3fa0a5e 100644 --- a/web-server/src/content/DoraMetrics/DoraMetricsBody.tsx +++ b/web-server/src/content/DoraMetrics/DoraMetricsBody.tsx @@ -4,7 +4,6 @@ import Link from 'next/link'; import { FC, useEffect, useMemo } from 'react'; import { AiButton } from '@/components/AiButton'; -import { useSelectedContributors } from '@/components/ContributorFilter'; import { DoraMetricsConfigurationSettings } from '@/components/DoraMetricsConfigurationSettings'; import { DoraScoreV2 } from '@/components/DoraScoreV2'; import { EmptyState } from '@/components/EmptyState'; @@ -18,6 +17,8 @@ import { ROUTES } from '@/constants/routes'; import { FetchState } from '@/constants/ui-states'; import { useDoraStats } from '@/content/DoraMetrics/DoraCards/sharedHooks'; import { useAuth } from '@/hooks/useAuth'; +// CLUSTOX: contributor filter -- shared fetch arguments, see the hook's note. +import { useDoraMetricsFetchArgs } from '@/hooks/useDoraMetricsFetchArgs'; import { useBoolState } from '@/hooks/useEasyState'; import { usePageRefreshCallback } from '@/hooks/usePageRefreshCallback'; import { @@ -38,14 +39,15 @@ import { WeeklyDeliveryVolumeCard } from './DoraCards/WeeklyDeliveryVolumeCard'; export const DoraMetricsBody = () => { const dispatch = useDispatch(); - const { orgId, integrationList } = useAuth(); + const { integrationList } = useAuth(); const isCodeProviderIntegrated = useMemo( () => integrationList.length > 0, [integrationList.length] ); - const { singleTeamId, dates } = useSingleTeamConfig(); const branchPayloadForPrFilters = useBranchesForPrFilters(); - const selectedContributors = useSelectedContributors(); + // CLUSTOX: contributor filter -- org/team/date/contributor arguments come + // from one shared hook so every refetch path carries the selection. + const doraMetricsFetchArgs = useDoraMetricsFetchArgs(); const isLoading = useSelector( (s) => s.doraMetrics.requests?.metrics_summary === FetchState.REQUEST ); @@ -71,23 +73,15 @@ export const DoraMetricsBody = () => { useEffect(() => { dispatch( fetchTeamDoraMetrics({ - orgId, - teamId: singleTeamId, - fromDate: dates.start, - toDate: dates.end, - authors: selectedContributors, + ...doraMetricsFetchArgs, ...branchPayloadForPrFilters }) ); }, [ - dates.end, - dates.start, dispatch, - orgId, - singleTeamId, + doraMetricsFetchArgs, isCodeProviderIntegrated, - branchPayloadForPrFilters, - selectedContributors + branchPayloadForPrFilters ]); const stats = useDoraStats(); diff --git a/web-server/src/hooks/useDoraMetricsFetchArgs.ts b/web-server/src/hooks/useDoraMetricsFetchArgs.ts new file mode 100644 index 00000000..efa01ac4 --- /dev/null +++ b/web-server/src/hooks/useDoraMetricsFetchArgs.ts @@ -0,0 +1,40 @@ +import { useMemo } from 'react'; + +import { useSelectedContributors } from '@/components/ContributorFilter'; +import { useAuth } from '@/hooks/useAuth'; +import { useSingleTeamConfig } from '@/hooks/useStateTeamConfig'; + +/** + * The org / team / date-window / contributor arguments that every + * `fetchTeamDoraMetrics` dispatch needs, built in exactly one place. + * + * There are four call sites (the DORA body's own effect, the page refresh + * callback, and the two settings dialogs that refetch after saving), and they + * had each hand-rolled this object. When the contributor filter was added only + * one of them learned about it, so any refetch triggered by a repo finishing + * its sync, or by saving a production-branch/incident-PR setting, silently + * dropped the selection: Lead Time and Deployment Frequency reverted to + * team-wide while the dropdown still showed the chip and the cards still read + * "authored by alice". + * + * Branch arguments stay with the caller on purpose -- the body and the refresh + * callback send `useBranchesForPrFilters()` (branches *and* branch_mode) while + * the two dialogs send only `branches`, and unifying that would change what + * unfiltered dashboards request. + */ +export const useDoraMetricsFetchArgs = () => { + const { orgId } = useAuth(); + const { singleTeamId, dates } = useSingleTeamConfig(); + const authors = useSelectedContributors(); + + return useMemo( + () => ({ + orgId, + teamId: singleTeamId, + fromDate: dates.start, + toDate: dates.end, + authors + }), + [orgId, singleTeamId, dates.start, dates.end, authors] + ); +}; diff --git a/web-server/src/hooks/usePageRefreshCallback.ts b/web-server/src/hooks/usePageRefreshCallback.ts index 0ed8e5e1..d2d464b8 100644 --- a/web-server/src/hooks/usePageRefreshCallback.ts +++ b/web-server/src/hooks/usePageRefreshCallback.ts @@ -1,19 +1,18 @@ import { useRouter } from 'next/router'; import { ROUTES } from '@/constants/routes'; -import { useAuth } from '@/hooks/useAuth'; -import { - useBranchesForPrFilters, - useSingleTeamConfig -} from '@/hooks/useStateTeamConfig'; +// CLUSTOX: contributor filter -- the refetch this callback triggers (a repo +// finishing its sync) must keep the selection, so the arguments come from the +// shared hook instead of being rebuilt here. +import { useDoraMetricsFetchArgs } from '@/hooks/useDoraMetricsFetchArgs'; +import { useBranchesForPrFilters } from '@/hooks/useStateTeamConfig'; import { fetchTeamDoraMetrics } from '@/slices/dora_metrics'; import { useDispatch } from '@/store'; export const usePageRefreshCallback = () => { const router = useRouter(); const dispatch = useDispatch(); - const { orgId } = useAuth(); - const { dates, singleTeamId } = useSingleTeamConfig(); + const doraMetricsFetchArgs = useDoraMetricsFetchArgs(); const branchPayloadForPrFilters = useBranchesForPrFilters(); switch (router.pathname) { @@ -21,10 +20,7 @@ export const usePageRefreshCallback = () => { return () => dispatch( fetchTeamDoraMetrics({ - orgId, - teamId: singleTeamId, - fromDate: dates.start, - toDate: dates.end, + ...doraMetricsFetchArgs, ...branchPayloadForPrFilters }) ); From 4296242bb6275a24e5c85ac968b0adc9bfd1b147 Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Tue, 11 Aug 2026 15:01:13 +0500 Subject: [PATCH 077/194] fix(contributors): count only merged PRs in the contributor list get_contributors_for_repos filtered on state_changed_at but not on PullRequest.state, while every metric it feeds pairs the two (see _filter_prs_merged_in_interval). Someone who opened and closed 15 PRs without merging any showed up as "15 PRs" in a dropdown whose own comment claims it can never offer someone with no data, and selecting them gave an empty Lead Time card. More commonly it just inflated every count relative to the drill-down. Adds a query test that captures the SQLAlchemy conditions -- the suite has no database, and the absence of any query-level test here is what let this through. Verified it fails without the state filter. Co-Authored-By: Claude Opus 5 --- .../analytics_server/mhq/store/repos/code.py | 5 ++ .../tests/store/repos/__init__.py | 0 .../store/repos/test_contributors_query.py | 69 +++++++++++++++++++ 3 files changed, 74 insertions(+) create mode 100644 backend/analytics_server/tests/store/repos/__init__.py create mode 100644 backend/analytics_server/tests/store/repos/test_contributors_query.py diff --git a/backend/analytics_server/mhq/store/repos/code.py b/backend/analytics_server/mhq/store/repos/code.py index 1fd4a2b2..6c99b664 100644 --- a/backend/analytics_server/mhq/store/repos/code.py +++ b/backend/analytics_server/mhq/store/repos/code.py @@ -530,6 +530,11 @@ def get_contributors_for_repos( PullRequest.repo_id.in_(repo_ids), PullRequest.author.isnot(None), PullRequest.state_changed_at.between(from_time, to_time), + # Merged only, matching _filter_prs_merged_in_interval. Every + # metric this dropdown filters counts merged PRs, so counting + # anything else here inflates the numbers next to each name + # and can offer someone whose Lead Time card comes back empty. + PullRequest.state == PullRequestState.MERGED, ) .group_by(PullRequest.author) .order_by(func.count(PullRequest.id).desc()) diff --git a/backend/analytics_server/tests/store/repos/__init__.py b/backend/analytics_server/tests/store/repos/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/backend/analytics_server/tests/store/repos/test_contributors_query.py b/backend/analytics_server/tests/store/repos/test_contributors_query.py new file mode 100644 index 00000000..13fd446b --- /dev/null +++ b/backend/analytics_server/tests/store/repos/test_contributors_query.py @@ -0,0 +1,69 @@ +from datetime import datetime + +from mhq.store.models.code import PullRequestState +from mhq.store.repos.code import CodeRepoService + + +class FakeQuery: + """Records the conditions handed to .filter() and returns no rows. + + The suite has no database, so the only way to assert what + get_contributors_for_repos actually asks for is to capture the SQLAlchemy + expressions it builds. + """ + + def __init__(self): + self.conditions = [] + + def query(self, *_args): + return self + + def filter(self, *conditions): + self.conditions.extend(conditions) + return self + + def group_by(self, *_args): + return self + + def order_by(self, *_args): + return self + + def all(self): + return [] + + +class FakeDb: + def __init__(self, session): + self.session = session + + +def capture_contributor_query_conditions(): + fake_query = FakeQuery() + service = CodeRepoService() + service._db = FakeDb(fake_query) + + service.get_contributors_for_repos( + ["35737b5a-7f35-4fbd-b86b-5c6052f4e206"], + datetime(2024, 1, 1), + datetime(2024, 2, 1), + ) + + return fake_query.conditions + + +def test_only_merged_prs_are_counted(): + # The dropdown's counts have to agree with the metrics they filter, and + # every metric query pairs the window with a merged-state check (see + # CodeRepoService._filter_prs_merged_in_interval). Without it someone who + # opened and closed 15 PRs without merging any of them was listed as + # "15 PRs" and selecting them produced an empty Lead Time card. + conditions = capture_contributor_query_conditions() + + state_conditions = [ + condition + for condition in conditions + if "state" in str(condition) and "state_changed_at" not in str(condition) + ] + + assert len(state_conditions) == 1 + assert state_conditions[0].right.value == PullRequestState.MERGED From e3817e45141341a40ab3bd5b195206913b74f1b6 Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Tue, 11 Aug 2026 15:02:10 +0500 Subject: [PATCH 078/194] fix(contributors): keep chips removable when the contributor fetch fails On error the control was disabled and the fetched list emptied, so `value` -- an intersection with that list -- collapsed to [] and the chips vanished. useSelectedContributors reads the URL, though, so the dashboard kept sending `authors: ['alice']` and the cards kept saying "authored by alice". Editing the URL by hand was the only way to clear it. Render `value` from the selection itself, and only disable the control when nothing is selected. Stale selections are still dropped on a successful fetch, with the existing notice. Co-Authored-By: Claude Opus 5 --- .../src/components/ContributorFilter.tsx | 24 ++++++++++++++----- 1 file changed, 18 insertions(+), 6 deletions(-) diff --git a/web-server/src/components/ContributorFilter.tsx b/web-server/src/components/ContributorFilter.tsx index 516c7997..1572dd55 100644 --- a/web-server/src/components/ContributorFilter.tsx +++ b/web-server/src/components/ContributorFilter.tsx @@ -130,12 +130,24 @@ export const ContributorFilter = () => { }, [notice]); const isEmpty = settled && contributors.length === 0; - const disabled = !singleTeamId || loading || failed || isEmpty; - - const value = useMemo( - () => contributors.filter((c) => selected.includes(c.username)), - [contributors, selected] - ); + // Whatever else is going on, a selection must stay removable. Disabling the + // control on a failed fetch left chips that could not be deleted while the + // dashboard carried on sending `authors`, and the cards carried on saying + // "authored by alice" -- the only way out was hand-editing the URL. With + // nothing selected this is exactly the old behaviour. + const disabled = + !singleTeamId || (!selected.length && (loading || failed || isEmpty)); + + // Rendered from the selection, not from an intersection with the fetched + // list, so chips survive a fetch that failed or has not landed yet. Stale + // selections are still dropped -- but only by the effect above, which runs + // on a *successful* fetch and says so in a notice. + const value = useMemo(() => { + const fetched = new Map(contributors.map((c) => [c.username, c])); + return selected.map( + (username) => fetched.get(username) || { username, pr_count: 0 } + ); + }, [contributors, selected]); const helperText = notice ? notice From 2868361da7b9cd6f19f094dd9a2cc94a55b1e35e Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Tue, 11 Aug 2026 15:58:23 +0500 Subject: [PATCH 079/194] fix(contributors): keep every name when the URL param repeats `parseSelectedFromQuery` took `raw[0]` whenever Next handed back an array, so `?contributors=alice&contributors=bob` -- the form a hand-written or hand-edited link takes -- silently filtered on alice alone. Accept both spellings and de-duplicate across them. Moved out of ContributorFilter.tsx into an import-light util so it can be tested at all: this repo's Jest maps `.tsx` through ts-jest with Next's `jsx: preserve`, so importing the component into a test dies on the first JSX token. Same reasoning, and same shape, as contributorFilters.ts. `contributorSelectionKey` comes along for the ride because the memo in `useSelectedContributors` needs a primitive: the array form is a fresh identity every render and would otherwise refetch the DORA summary in a loop. Reverting either function to `raw[0]` fails the new suite. Co-Authored-By: Claude Opus 5 --- .../src/components/ContributorFilter.tsx | 19 +++--- .../utils/__tests__/contributorQuery.test.ts | 61 +++++++++++++++++++ web-server/src/utils/contributorQuery.ts | 35 +++++++++++ 3 files changed, 106 insertions(+), 9 deletions(-) create mode 100644 web-server/src/utils/__tests__/contributorQuery.test.ts create mode 100644 web-server/src/utils/contributorQuery.ts diff --git a/web-server/src/components/ContributorFilter.tsx b/web-server/src/components/ContributorFilter.tsx index 1572dd55..30e88512 100644 --- a/web-server/src/components/ContributorFilter.tsx +++ b/web-server/src/components/ContributorFilter.tsx @@ -12,19 +12,17 @@ import { handleApi } from '@/api-helpers/axios-api-instance'; import { FlexBox } from '@/components/FlexBox'; import { Line } from '@/components/Text'; import { useSingleTeamConfig } from '@/hooks/useStateTeamConfig'; +import { + CONTRIBUTOR_QUERY_KEY as QUERY_KEY, + contributorSelectionKey, + parseSelectedFromQuery +} from '@/utils/contributorQuery'; type Contributor = { username: string; pr_count: number; }; -const QUERY_KEY = 'contributors'; - -const parseSelectedFromQuery = (raw: string | string[]): string[] => { - const value = Array.isArray(raw) ? raw[0] : raw; - return value ? value.split(',').filter(Boolean) : []; -}; - /** * CLUSTOX: single source of truth for the contributor selection. It lives in * the URL -- unlike the team/date selectors, which are redux-only -- so a @@ -39,9 +37,12 @@ export const useSelectedContributors = (): string[] => { // object -- that object gets a new identity on every navigation, including // unrelated ones like OverlayPageContext pushing its own `overlays` param // when a "See details" panel opens. Depending on the whole object would - // refetch the DORA summary on every such click. + // refetch the DORA summary on every such click. `contributorSelectionKey` + // also flattens the repeated-param array form, which would otherwise be a + // fresh identity on every render and defeat the memo entirely. const raw = query[QUERY_KEY] as string | string[]; - return useMemo(() => parseSelectedFromQuery(raw), [raw]); + const key = contributorSelectionKey(raw); + return useMemo(() => parseSelectedFromQuery(key), [key]); }; export const ContributorFilter = () => { diff --git a/web-server/src/utils/__tests__/contributorQuery.test.ts b/web-server/src/utils/__tests__/contributorQuery.test.ts new file mode 100644 index 00000000..e4e1ccb1 --- /dev/null +++ b/web-server/src/utils/__tests__/contributorQuery.test.ts @@ -0,0 +1,61 @@ +import { + contributorSelectionKey, + parseSelectedFromQuery +} from '@/utils/contributorQuery'; + +// CLUSTOX: the contributor selection lives in the URL, so this parser decides +// what a shared or hand-edited link actually filters on. Next types a query +// value as `string | string[]`: `?contributors=a,b` arrives as a string, but a +// repeated `?contributors=a&contributors=b` arrives as an array. The original +// took `raw[0]` and silently dropped everything after the first, so a +// two-person link quietly filtered on one person. +describe('parseSelectedFromQuery', () => { + it('parses the comma-separated form the component itself writes', () => { + expect(parseSelectedFromQuery('alice,bob')).toEqual(['alice', 'bob']); + }); + + it('keeps every value when the param repeats, not just the first', () => { + // `?contributors=alice&contributors=bob` + expect(parseSelectedFromQuery(['alice', 'bob'])).toEqual(['alice', 'bob']); + }); + + it('handles a repeated param whose entries are themselves comma-separated', () => { + expect(parseSelectedFromQuery(['alice,bob', 'carol'])).toEqual([ + 'alice', + 'bob', + 'carol' + ]); + }); + + it('de-duplicates, so a name repeated across forms is only sent once', () => { + expect(parseSelectedFromQuery(['alice', 'alice,bob'])).toEqual([ + 'alice', + 'bob' + ]); + }); + + it('drops empty segments from a malformed value', () => { + expect(parseSelectedFromQuery(',alice,,bob,')).toEqual(['alice', 'bob']); + }); + + it('returns an empty selection when the param is absent or blank', () => { + expect(parseSelectedFromQuery(undefined as unknown as string)).toEqual([]); + expect(parseSelectedFromQuery('')).toEqual([]); + expect(parseSelectedFromQuery([])).toEqual([]); + }); +}); + +// The memo key has to be a primitive, and it has to compare equal for two URL +// spellings that mean the same selection -- otherwise the DORA summary refetches +// on every render. +describe('contributorSelectionKey', () => { + it('collapses the array form to the same string as the comma form', () => { + expect(contributorSelectionKey(['alice', 'bob'])).toBe( + contributorSelectionKey('alice,bob') + ); + }); + + it('is a stable primitive for an absent param', () => { + expect(contributorSelectionKey(undefined as unknown as string)).toBe(''); + }); +}); diff --git a/web-server/src/utils/contributorQuery.ts b/web-server/src/utils/contributorQuery.ts new file mode 100644 index 00000000..599c5051 --- /dev/null +++ b/web-server/src/utils/contributorQuery.ts @@ -0,0 +1,35 @@ +import { uniq } from 'ramda'; + +// CLUSTOX: URL <-> selection plumbing for the DORA contributor filter. +// +// The selection lives in the query string rather than redux, so a filtered +// dashboard survives a refresh and can be pasted to someone else. That makes +// this the only thing standing between a shared or hand-edited link and the +// filter the dashboard actually applies, which is worth testing directly. +// +// Lives here rather than in ContributorFilter.tsx, and deliberately imports +// nothing but ramda, because this repo's Jest config maps `.tsx` through +// ts-jest with Next's `jsx: preserve` -- so importing the component into a +// test fails on the first JSX token. Same reasoning as contributorFilters.ts. + +export const CONTRIBUTOR_QUERY_KEY = 'contributors'; + +/** + * Next types a query value as `string | string[]`: `?contributors=a,b` arrives + * as a string, but a *repeated* `?contributors=a&contributors=b` arrives as an + * array. Both forms mean the same thing to a user, so both are accepted. + */ +export const parseSelectedFromQuery = (raw: string | string[]): string[] => { + const values = Array.isArray(raw) ? raw : raw ? [raw] : []; + return uniq(values.flatMap((value) => value.split(',')).filter(Boolean)); +}; + +/** + * A stable primitive identity for a raw query value, for use as a memo + * dependency. The array form is a fresh object on every render, so depending on + * it directly would invalidate the memo forever and refetch the DORA summary in + * a loop. Collapsing to the same comma-separated string the component writes + * makes the two URL spellings compare equal, as they should. + */ +export const contributorSelectionKey = (raw: string | string[]): string => + Array.isArray(raw) ? raw.join(',') : raw || ''; From edcdd18ed8ffa69ae807a4ab44f1f5c19f2f5c58 Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Tue, 11 Aug 2026 15:58:32 +0500 Subject: [PATCH 080/194] fix(contributors): align the dropdown's date window with the metrics window The DORA route normalises the picker values with startOfDay/endOfDay before querying; the contributor route forwarded the very same `dates.start` / `dates.end` raw, wall-clock time included. So the dropdown's window was strictly narrower than the window the metrics use: a PR merged later today than the current clock time counted towards Lead Time while its author was missing from the list that is supposed to offer exactly the people with data in view. Apply the same normalisation. Also fixes the pre-existing import order in this file, which eslint flags once the file is touched. Co-Authored-By: Claude Opus 5 --- .../api/internal/team/[team_id]/contributors.ts | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-) diff --git a/web-server/pages/api/internal/team/[team_id]/contributors.ts b/web-server/pages/api/internal/team/[team_id]/contributors.ts index c5af3057..7d898564 100644 --- a/web-server/pages/api/internal/team/[team_id]/contributors.ts +++ b/web-server/pages/api/internal/team/[team_id]/contributors.ts @@ -1,7 +1,9 @@ +import { endOfDay, startOfDay } from 'date-fns'; import * as yup from 'yup'; -import { Endpoint } from '@/api-helpers/global'; import { internal } from '@/api-helpers/axios'; +import { Endpoint } from '@/api-helpers/global'; +import { isoDateString } from '@/utils/date'; // CLUSTOX: backs the contributor filter on the DORA metrics page. team_id is // validated by Endpoint.serve(), which asserts the caller may read that team. @@ -18,7 +20,17 @@ const getSchema = yup.object().shape({ const endpoint = new Endpoint(pathSchema); endpoint.handle.GET(getSchema, async (req, res) => { - const { team_id, from_date, to_date } = req.payload; + const { team_id, from_date: rawFromDate, to_date: rawToDate } = req.payload; + + // The same day-boundary normalisation the DORA route applies to the very + // same picker values (see dora_metrics.ts). Both routes are handed + // `dates.start` / `dates.end`, which carry the wall-clock time of day, so + // forwarding them raw made this window strictly narrower than the metrics + // window: a PR merged later today than the current clock time counted + // towards Lead Time but was missing from the dropdown that is supposed to + // list exactly the people with data in view. + const from_date = isoDateString(startOfDay(new Date(rawFromDate))); + const to_date = isoDateString(endOfDay(new Date(rawToDate))); const contributors = await internal.get(`/teams/${team_id}/contributors`, { params: { from_time: from_date, to_time: to_date } From a8246482189a8b3d160495398b39c7174f8b678d Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Tue, 11 Aug 2026 15:58:44 +0500 Subject: [PATCH 081/194] chore(contributors): mark every upstream edit with a CLUSTOX sentinel MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit FORK_STRATEGY.md §4 requires a sentinel on each edit to an upstream file -- it is the one-command audit of our divergence footprint, and the checklist that catches a sync merge silently eating our work. Four edits from this branch were missing one: - dora_metrics.ts: the `authors` schema entry, its destructure, the filter-building call, the stripContributorFilters import and the strip itself - pages/dora-metrics/index.tsx: the ContributorFilter mount (hoisted out of the JSX attribute list, which cannot hold a comment) - workflow_filter.py: the `workflow_filter or {}` guard, called out as the behaviour change it is -- upstream raised AttributeError on `apply(None)` despite the signature defaulting to None -- plus the event_actors read and the constructor argument Also restores the blank line above `app = Blueprint` in pull_requests.py. Worth being precise about why it went missing: the venv here ships black 26.5.1, which collapses it, but CI pins black==24.3.0 (.github/workflows/ black.yml), under which the two-blank-line form is correct and the whole backend is clean. CI is the authority, so the line goes back; anyone reformatting with a newer local black will see it disappear again. Co-Authored-By: Claude Opus 5 --- backend/analytics_server/mhq/api/pull_requests.py | 1 + .../mhq/service/workflows/workflow_filter.py | 13 ++++++++++++- .../api/internal/team/[team_id]/dora_metrics.ts | 14 ++++++++++++++ web-server/pages/dora-metrics/index.tsx | 9 ++++++++- 4 files changed, 35 insertions(+), 2 deletions(-) diff --git a/backend/analytics_server/mhq/api/pull_requests.py b/backend/analytics_server/mhq/api/pull_requests.py index b82e6cc6..64d7eecd 100644 --- a/backend/analytics_server/mhq/api/pull_requests.py +++ b/backend/analytics_server/mhq/api/pull_requests.py @@ -31,6 +31,7 @@ from mhq.store.models import SettingType, EntityType + app = Blueprint("pull_requests", __name__) diff --git a/backend/analytics_server/mhq/service/workflows/workflow_filter.py b/backend/analytics_server/mhq/service/workflows/workflow_filter.py index a6eeea44..bf057580 100644 --- a/backend/analytics_server/mhq/service/workflows/workflow_filter.py +++ b/backend/analytics_server/mhq/service/workflows/workflow_filter.py @@ -8,16 +8,27 @@ class ParseWorkflowFilterProcessor: def apply(self, workflow_filter: Dict = None) -> WorkflowFilter: + # CLUSTOX: contributor filter for deployment frequency. This guard is a + # behaviour change, not just an addition: upstream dereferenced the + # argument straight away, so `apply(None)` raised AttributeError even + # though the signature defaults to None. Making the documented default + # actually work is what lets the event_actors lookup below be + # unconditional. workflow_filter = workflow_filter or {} + # END CLUSTOX head_branches: List[str] = self._parse_head_branches(workflow_filter) repo_filters: Dict[str, Dict] = self._parse_repo_filters(workflow_filter) - # CLUSTOX: contributor filter for deployment frequency. + # CLUSTOX: contributor filter -- the actor who triggered the run. Read + # from inside the workflow_filter blob, the same level as head_branches. event_actors: List[str] = workflow_filter.get("event_actors") + # END CLUSTOX return WorkflowFilter( head_branches=head_branches, repo_filters=repo_filters, + # CLUSTOX: contributor filter for deployment frequency. event_actors=event_actors, + # END CLUSTOX ) def _parse_head_branches(self, workflow_filter: Dict) -> List[str]: diff --git a/web-server/pages/api/internal/team/[team_id]/dora_metrics.ts b/web-server/pages/api/internal/team/[team_id]/dora_metrics.ts index f7f9c347..5a20f53e 100644 --- a/web-server/pages/api/internal/team/[team_id]/dora_metrics.ts +++ b/web-server/pages/api/internal/team/[team_id]/dora_metrics.ts @@ -16,6 +16,7 @@ import { fetchMeanTimeToRestoreStats, fetchDeploymentFrequencyStats } from '@/utils/cockpitMetricUtils'; +// CLUSTOX: contributor filter. import { stripContributorFilters } from '@/utils/contributorFilters'; import { isoDateString, getAggregateAndTrendsIntervalTime } from '@/utils/date'; import { @@ -35,6 +36,8 @@ const getSchema = yup.object().shape({ from_date: yup.date().required(), to_date: yup.date().required(), branch_mode: yup.string().oneOf(Object.values(ActiveBranchMode)).required(), + // CLUSTOX: contributor filter -- git usernames, optional so an unfiltered + // dashboard sends exactly what it always did. authors: yup.array().of(yup.string()).optional() }); @@ -52,6 +55,7 @@ endpoint.handle.GET(getSchema, async (req, res) => { to_date: rawToDate, branches, branch_mode, + // CLUSTOX: contributor filter. authors } = req.payload; @@ -67,6 +71,9 @@ endpoint.handle.GET(getSchema, async (req, res) => { getUnsyncedRepos(teamId) ]); const [prFilters, workflowFilters] = await Promise.all([ + // CLUSTOX: contributor filter -- `authors` narrows Lead Time by PR author, + // `eventActors` narrows Deployment Frequency by the actor who triggered the + // run. Both are no-ops when the selection is empty. updatePrFilterParams(teamId, {}, { ...branchAndRepoFilters, authors }).then( ({ pr_filter }) => ({ pr_filter @@ -77,12 +84,19 @@ endpoint.handle.GET(getSchema, async (req, res) => { teamId: teamId, eventActors: authors }) + // END CLUSTOX ]); + // CLUSTOX: contributor filter -- Change Failure Rate and MTTR stay team-wide + // (no defensible per-contributor definition until Jira incident ownership + // lands), so they get a copy of the filters with the contributor keys + // removed. With nothing selected these are identical to the originals, which + // is what keeps unfiltered dashboards byte-for-byte unchanged. const { prFilter: prFilterWithoutAuthors, workflowFilter: workflowFilterWithoutEventActors } = stripContributorFilters(prFilters, workflowFilters); + // END CLUSTOX const { currTrendsTimeObject, diff --git a/web-server/pages/dora-metrics/index.tsx b/web-server/pages/dora-metrics/index.tsx index 911f40dc..88684a3b 100644 --- a/web-server/pages/dora-metrics/index.tsx +++ b/web-server/pages/dora-metrics/index.tsx @@ -1,6 +1,7 @@ import ExtendedSidebarLayout from 'src/layouts/ExtendedSidebarLayout'; import { Authenticated } from '@/components/Authenticated'; +// CLUSTOX: contributor filter. import { ContributorFilter } from '@/components/ContributorFilter'; import { FlexBox } from '@/components/FlexBox'; import Loader from '@/components/Loader'; @@ -18,6 +19,12 @@ function Page() { ); const { integrationList } = useAuth(); + // CLUSTOX: contributor filter -- rendered alongside the team/date selectors + // through PageWrapper's existing `additionalFilters` seam, so no upstream + // layout component is edited. + const additionalFilters = []; + // END CLUSTOX + return ( ]} + additionalFilters={additionalFilters} > {integrationList.length > 0 ? : } From c0eeb906b858ce5e12aa2c0e2029ec3cb65a0acb Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Tue, 11 Aug 2026 17:54:39 +0500 Subject: [PATCH 082/194] fix(jenkins): make a mapped job actually count as deployments Mapping a Jenkins job wrote a RepoWorkflow row and nothing else, so the 14 builds it ingested counted for zero. DeploymentsService splits a team's repos on TeamRepos.deployment_type and only consults RepoWorkflowRuns for the WORKFLOW side; new team repos default to PR_MERGE, so mapping appeared to work, changed nothing, and reported no error. Creating a mapping now switches every active TeamRepos row for that repo to WORKFLOW in the same commit as the RepoWorkflow insert and the deactivation of the repo's other deployment sources, and records the values it overwrote on the Jenkins row's meta -- the same pattern the displaced GitHub Actions workflow ids already use. Unmapping restores exactly those rows: a repo already on WORKFLOW before mapping was never recorded and stays on WORKFLOW. Also drops typing.Optional from mhq/api/integrations.py: voluptuous exports an Optional that shadows it, and an Optional[...] annotation resolves to voluptuous's marker class, which is not subscriptable. On the Python 3.9 the image runs, annotations evaluate at def time, so that is a TypeError on import that takes every route in the module down. Co-Authored-By: Claude Opus 5 --- .../analytics_server/mhq/api/integrations.py | 121 +++++++- .../analytics_server/mhq/store/repos/code.py | 15 + .../mhq/store/repos/workflows.py | 37 ++- .../tests/api/test_jenkins_routes.py | 269 +++++++++++++++++- 4 files changed, 416 insertions(+), 26 deletions(-) diff --git a/backend/analytics_server/mhq/api/integrations.py b/backend/analytics_server/mhq/api/integrations.py index 6f2dd515..511223bc 100644 --- a/backend/analytics_server/mhq/api/integrations.py +++ b/backend/analytics_server/mhq/api/integrations.py @@ -1,4 +1,10 @@ -from typing import List +# CLUSTOX: no typing.Optional here on purpose -- voluptuous.Optional is imported +# below and shadows it, and an annotation that reads Optional[...] resolves to +# voluptuous's marker class, which is not subscriptable. Under Python 3.9, +# where annotations are evaluated at def time, that is a TypeError on import of +# this module, taking every route in it down. Default arguments say "optional" +# here instead. +from typing import Dict, List, Tuple from flask import Blueprint, jsonify from github import GithubException @@ -15,6 +21,7 @@ from mhq.service.query_validator import get_query_validator from mhq.store.models import UserIdentityProvider from mhq.store.models.code import RepoWorkflowProviders, RepoWorkflowType +from mhq.store.models.code.enums import TeamReposDeploymentType from mhq.store.models.code.workflows.workflows import RepoWorkflow from mhq.store.repos.code import CodeRepoService from mhq.store.repos.workflows import WorkflowRepoService @@ -33,6 +40,17 @@ # displaced from one that was already inactive for an unrelated reason. DISPLACED_WORKFLOW_IDS_KEY = "jenkins_displaced_workflow_ids" +# CLUSTOX: second key on the same blob, holding the TeamRepos.deployment_type +# values this mapping overwrote, keyed by team id. Mapping a job is useless +# without the switch it records: DeploymentsService splits a team's repos on +# that column and only consults RepoWorkflowRuns for the WORKFLOW side, so a +# repo left on PR_MERGE ingests Jenkins builds and counts none of them -- +# mapping appears to work, changes nothing, and reports no error. Recorded for +# the same reason the displaced workflow ids are: unmapping has to put back +# exactly what mapping changed, and "everything is PR_MERGE now" would demote a +# repo the admin had deliberately configured as WORKFLOW. +PREVIOUS_DEPLOYMENT_TYPES_KEY = "jenkins_previous_deployment_types" + # CLUSTOX: JSONB columns on this model default to the string "{}" rather than a # dict, and a row written before DISPLACED_WORKFLOW_IDS_KEY existed has no @@ -88,6 +106,52 @@ def reactivate_github_actions_workflows_for_repo( return reactivated +# CLUSTOX: points every team that tracks this repo at its workflow runs, which +# is what actually makes a Jenkins build count as a deployment. Returns the rows +# it changed together with the values it overwrote; rows already on WORKFLOW are +# neither changed nor recorded, so unmapping leaves them alone rather than +# demoting a repo the admin configured that way himself. +def switch_team_repos_to_workflow_deployments( + team_repos: List, +) -> Tuple[List, Dict[str, str]]: + switched = [] + previous_deployment_types: Dict[str, str] = {} + for team_repo in team_repos: + if team_repo.deployment_type == TeamReposDeploymentType.WORKFLOW: + continue + # A null deployment_type is recorded as PR_MERGE -- the column default, + # and the only value that restores to something DeploymentsService can + # read, since it dereferences .value without a null check. + previous_deployment_types[str(team_repo.team_id)] = ( + team_repo.deployment_type.value + if team_repo.deployment_type + else TeamReposDeploymentType.PR_MERGE.value + ) + team_repo.deployment_type = TeamReposDeploymentType.WORKFLOW + switched.append(team_repo) + return switched, previous_deployment_types + + +# CLUSTOX: the other half of the switch above, and the same restraint as +# reactivate_github_actions_workflows_for_repo: restore only the rows this +# mapping recorded, and only while they still hold the value the mapping wrote. +# A row the admin has since changed by hand is his decision, not ours to undo. +def restore_team_repo_deployment_types( + team_repos: List, previous_deployment_types: Dict[str, str] = None +) -> List: + recorded = previous_deployment_types or {} + restored = [] + for team_repo in team_repos: + previous = recorded.get(str(team_repo.team_id)) + if not previous: + continue + if team_repo.deployment_type != TeamReposDeploymentType.WORKFLOW: + continue + team_repo.deployment_type = TeamReposDeploymentType(previous) + restored.append(team_repo) + return restored + + @app.route("/orgs//integrations/github/orgs", methods={"GET"}) def get_github_orgs(org_id: str): query_validator = get_query_validator() @@ -330,7 +394,8 @@ def create_jenkins_mapping(org_id: str, org_repo_id: str, job_full_name: str): query_validator = get_query_validator() query_validator.org_validator(org_id) - org_repo = CodeRepoService().get_repo_by_id(org_repo_id) + code_repo_service = CodeRepoService() + org_repo = code_repo_service.get_repo_by_id(org_repo_id) if org_repo is None or str(org_repo.org_id) != str(org_id): return jsonify({"error": f"Repo {org_repo_id} not found in org {org_id}"}), 404 @@ -385,6 +450,24 @@ def create_jenkins_mapping(org_id: str, org_repo_id: str, job_full_name: str): # Jenkins row below, in one commit. deactivated = deactivate_deployment_workflows_for_repo(workflows_to_deactivate) + # Mutated in place like the workflows above, and written in the same commit. + team_repos = code_repo_service.get_active_team_repos_by_repo_id(org_repo_id) + switched_team_repos, previous_deployment_types = ( + switch_team_repos_to_workflow_deployments(team_repos) + ) + # A Jenkins mapping this one displaces already switched these rows, so this + # mapping records nothing for them and the values it is overwriting sit on + # the displaced row's meta. Carrying them forward is what keeps the last + # unmapping in a chain of remappings able to restore the repo; dropping them + # strands it on WORKFLOW with no workflow behind it -- zero deployments, + # silently. + for workflow in deactivated: + if workflow.provider != RepoWorkflowProviders.JENKINS: + continue + carried = _workflow_meta(workflow).get(PREVIOUS_DEPLOYMENT_TYPES_KEY) or {} + for team_id, deployment_type in carried.items(): + previous_deployment_types.setdefault(team_id, deployment_type) + if existing_jenkins_workflow is not None: existing_jenkins_workflow.is_active = True existing_jenkins_workflow.name = job_full_name @@ -405,12 +488,17 @@ def create_jenkins_mapping(org_id: str, org_repo_id: str, job_full_name: str): jenkins_workflow.meta = { **_workflow_meta(jenkins_workflow), DISPLACED_WORKFLOW_IDS_KEY: [str(workflow.id) for workflow in deactivated], + PREVIOUS_DEPLOYMENT_TYPES_KEY: previous_deployment_types, } workflow_repo_service.create_jenkins_repo_workflow( - jenkins_workflow, workflows_to_deactivate + jenkins_workflow, workflows_to_deactivate, switched_team_repos ) - return {"ok": True, "deactivated_workflows": len(deactivated)} + return { + "ok": True, + "deactivated_workflows": len(deactivated), + "switched_team_repos": len(switched_team_repos), + } # CLUSTOX: removes a Jenkins mapping and restores the GitHub Actions workflows @@ -432,7 +520,8 @@ def delete_jenkins_mapping(org_id: str, repo_workflow_id: str): if repo_workflow is None: return jsonify({"error": f"Workflow {repo_workflow_id} not found"}), 404 - org_repo = CodeRepoService().get_repo_by_id(repo_workflow.org_repo_id) + code_repo_service = CodeRepoService() + org_repo = code_repo_service.get_repo_by_id(repo_workflow.org_repo_id) if org_repo is None or str(org_repo.org_id) != str(org_id): return ( jsonify( @@ -465,15 +554,31 @@ def delete_jenkins_mapping(org_id: str, repo_workflow_id: str): reactivated = reactivate_github_actions_workflows_for_repo( github_workflows, meta.get(DISPLACED_WORKFLOW_IDS_KEY) ) + # The deployment_type switch is undone on the same terms: only the team rows + # this mapping recorded. A repo already on WORKFLOW when it was mapped was + # never recorded, so it stays on WORKFLOW here. + team_repos = code_repo_service.get_active_team_repos_by_repo_id( + str(repo_workflow.org_repo_id) + ) + restored_team_repos = restore_team_repo_deployment_types( + team_repos, meta.get(PREVIOUS_DEPLOYMENT_TYPES_KEY) + ) # Drop the record in the same commit that acts on it, so a second unmapping # -- or one that follows a team-config edit that deselected these same # workflows -- does not restore them a second time. + dropped_keys = {DISPLACED_WORKFLOW_IDS_KEY, PREVIOUS_DEPLOYMENT_TYPES_KEY} repo_workflow.meta = { - key: value for key, value in meta.items() if key != DISPLACED_WORKFLOW_IDS_KEY + key: value for key, value in meta.items() if key not in dropped_keys } - workflow_repo_service.deactivate_repo_workflow(repo_workflow, reactivated) + workflow_repo_service.deactivate_repo_workflow( + repo_workflow, reactivated, restored_team_repos + ) - return {"ok": True, "reactivated_github_workflows": len(reactivated)} + return { + "ok": True, + "reactivated_github_workflows": len(reactivated), + "restored_team_repos": len(restored_team_repos), + } @app.route("/orgs//integrations/gitlab/user/repos", methods={"GET"}) diff --git a/backend/analytics_server/mhq/store/repos/code.py b/backend/analytics_server/mhq/store/repos/code.py index 310be35c..be716352 100644 --- a/backend/analytics_server/mhq/store/repos/code.py +++ b/backend/analytics_server/mhq/store/repos/code.py @@ -275,6 +275,21 @@ def get_active_team_repos_by_team_ids(self, team_ids: List[str]) -> List[TeamRep .all() ) + # CLUSTOX: every team that tracks this repo, not just one. DeploymentsService + # splits a *team's* repos on TeamRepos.deployment_type, so a repo shared by + # three teams has three rows and a Jenkins mapping that switched only one of + # them would report deployments to one team and nothing to the other two. + @rollback_on_exc + def get_active_team_repos_by_repo_id(self, repo_id: str) -> List[TeamRepos]: + return ( + self._db.session.query(TeamRepos) + .filter( + TeamRepos.org_repo_id == repo_id, + TeamRepos.is_active.is_(True), + ) + .all() + ) + @rollback_on_exc def get_active_org_repos_by_ids(self, repo_ids: List[str]) -> List[OrgRepo]: return ( diff --git a/backend/analytics_server/mhq/store/repos/workflows.py b/backend/analytics_server/mhq/store/repos/workflows.py index 1c97c917..b0603624 100644 --- a/backend/analytics_server/mhq/store/repos/workflows.py +++ b/backend/analytics_server/mhq/store/repos/workflows.py @@ -16,7 +16,10 @@ RepoWorkflowRuns, RepoWorkflowRunsBookmark, ) -from mhq.store.models.code.repository import OrgRepo + +# CLUSTOX: TeamRepos is written here so the deployment_type switch shares a +# commit with the Jenkins mapping -- see create_jenkins_repo_workflow. +from mhq.store.models.code.repository import OrgRepo, TeamRepos from mhq.utils.time import Interval @@ -182,16 +185,24 @@ def get_repo_workflows_by_repo_id_and_provider( .all() ) - # CLUSTOX: writes the Jenkins mapping and the deactivation of the repo's - # other deployment workflows in a single commit. If either half failed on - # its own, a repo could be left with two active deployment sources, - # silently double-counting deployments -- see - # deactivate_deployment_workflows_for_repo in mhq/api/integrations.py. + # CLUSTOX: writes the Jenkins mapping, the deactivation of the repo's other + # deployment workflows and the TeamRepos.deployment_type switch in a single + # commit. If any part applied on its own, a repo could be left with two + # active deployment sources -- silently double-counting deployments, see + # deactivate_deployment_workflows_for_repo in mhq/api/integrations.py -- or + # switched to WORKFLOW with no workflow behind it, which reads as zero + # deployments. + # + # TeamRepos is written from here rather than through CodeRepoService because + # both services share one db.session and only one of them can own the + # commit; splitting the write across two commits is the partial-failure this + # method exists to prevent. @rollback_on_exc def create_jenkins_repo_workflow( self, jenkins_workflow: RepoWorkflow, workflows_to_deactivate: List[RepoWorkflow], + team_repos_to_update: List[TeamRepos] = None, ) -> RepoWorkflow: # add() on an instance already loaded in this session is a no-op, so # this covers both a brand new mapping and the reactivation of a row @@ -199,23 +210,29 @@ def create_jenkins_repo_workflow( self._db.session.add(jenkins_workflow) for workflow in workflows_to_deactivate: self._db.session.merge(workflow) + for team_repo in team_repos_to_update or []: + self._db.session.merge(team_repo) self._db.session.commit() return jenkins_workflow - # CLUSTOX: removing a Jenkins mapping and restoring the GitHub Actions rows - # it displaced happen in one commit. Half of this applied on its own would - # leave the repo with no active deployment source at all, and its - # Deployment Frequency at zero. + # CLUSTOX: removing a Jenkins mapping, restoring the GitHub Actions rows it + # displaced and restoring the TeamRepos.deployment_type values it overwrote + # happen in one commit. Half of this applied on its own would leave the repo + # with no active deployment source at all, and its Deployment Frequency at + # zero. @rollback_on_exc def deactivate_repo_workflow( self, repo_workflow: RepoWorkflow, workflows_to_reactivate: List[RepoWorkflow] = None, + team_repos_to_update: List[TeamRepos] = None, ) -> RepoWorkflow: repo_workflow.is_active = False self._db.session.merge(repo_workflow) for workflow in workflows_to_reactivate or []: self._db.session.merge(workflow) + for team_repo in team_repos_to_update or []: + self._db.session.merge(team_repo) self._db.session.commit() return repo_workflow diff --git a/backend/analytics_server/tests/api/test_jenkins_routes.py b/backend/analytics_server/tests/api/test_jenkins_routes.py index e8ae3c9a..ce5a38ce 100644 --- a/backend/analytics_server/tests/api/test_jenkins_routes.py +++ b/backend/analytics_server/tests/api/test_jenkins_routes.py @@ -10,13 +10,19 @@ from mhq.api.integrations import ( deactivate_deployment_workflows_for_repo, reactivate_github_actions_workflows_for_repo, + restore_team_repo_deployment_types, + switch_team_repos_to_workflow_deployments, ) +from mhq.service.deployments.deployment_service import DeploymentsService from mhq.store.models.code import RepoWorkflowProviders, RepoWorkflowType +from mhq.store.models.code.enums import TeamReposDeploymentType from mhq.utils.string import uuid4_str ORG_ID = "11111111-1111-4111-8111-111111111111" OTHER_ORG_ID = "22222222-2222-4222-8222-222222222222" REPO_ID = "33333333-3333-4333-8333-333333333333" +TEAM_ID = "44444444-4444-4444-8444-444444444444" +OTHER_TEAM_ID = "55555555-5555-4555-8555-555555555555" class FakeWorkflow: @@ -97,12 +103,87 @@ def test_reactivates_nothing_when_the_mapping_recorded_no_displacement(): assert github.is_active is False +class FakeTeamRepo: + """ + A row of TeamRepos. deployment_type is the column DeploymentsService splits + on: repos on PR_MERGE never have their RepoWorkflowRuns looked at, whatever + has been ingested for them. + """ + + def __init__( + self, + team_id=TEAM_ID, + org_repo_id=REPO_ID, + deployment_type=TeamReposDeploymentType.PR_MERGE, + is_active=True, + ): + self.team_id = team_id + self.org_repo_id = org_repo_id + self.deployment_type = deployment_type + self.is_active = is_active + + +def test_switching_records_only_the_rows_it_changed(): + on_pr_merge = FakeTeamRepo() + already_workflow = FakeTeamRepo( + team_id=OTHER_TEAM_ID, deployment_type=TeamReposDeploymentType.WORKFLOW + ) + + switched, previous = switch_team_repos_to_workflow_deployments( + [on_pr_merge, already_workflow] + ) + + assert switched == [on_pr_merge] + assert on_pr_merge.deployment_type == TeamReposDeploymentType.WORKFLOW + # Recording the already-WORKFLOW row would make unmapping demote a repo the + # admin configured that way himself. + assert previous == {TEAM_ID: "PR_MERGE"} + + +def test_restoring_leaves_a_row_the_admin_has_since_changed_alone(): + changed_by_hand = FakeTeamRepo(deployment_type=TeamReposDeploymentType.PR_MERGE) + untouched = FakeTeamRepo( + team_id=OTHER_TEAM_ID, deployment_type=TeamReposDeploymentType.WORKFLOW + ) + + restored = restore_team_repo_deployment_types( + [changed_by_hand, untouched], + {TEAM_ID: "PR_MERGE", OTHER_TEAM_ID: "PR_MERGE"}, + ) + + # The first row no longer holds the value the mapping wrote, so the mapping + # is not what is on it any more and there is nothing here to undo. + assert restored == [untouched] + assert untouched.deployment_type == TeamReposDeploymentType.PR_MERGE + + class FakeOrgRepo: def __init__(self, repo_id=REPO_ID, org_id=ORG_ID): self.id = repo_id self.org_id = org_id +class FakeCodeRepoService: + """ + Resolves REPO_ID into ORG_ID and reads the repo's TeamRepos rows out of the + same in-memory store the workflow service writes to -- in production both + services share one db.session, and the mapping routes depend on that. + """ + + def __init__(self, store): + self._store = store + + def get_repo_by_id(self, repo_id) -> Optional[FakeOrgRepo]: + return FakeOrgRepo(repo_id=repo_id) if str(repo_id) == REPO_ID else None + + def get_active_team_repos_by_repo_id(self, repo_id) -> List[FakeTeamRepo]: + return [ + team_repo + for team_repo in self._store.team_repos + if str(team_repo.org_repo_id) == str(repo_id) and team_repo.is_active + ] + + class FakeWorkflowRepoService: """ In-memory stand-in that reproduces the two database behaviours these routes @@ -112,8 +193,18 @@ class FakeWorkflowRepoService: pair that already has a row raises IntegrityError. """ - def __init__(self, workflows: Optional[List[FakeWorkflow]] = None): + def __init__( + self, + workflows: Optional[List[FakeWorkflow]] = None, + team_repos: Optional[List[FakeTeamRepo]] = None, + ): self.workflows = list(workflows or []) + # Defaults to a single team on PR_MERGE: what repository_service.py + # gives a newly added repo, and the state the live workspace was in + # while its 14 ingested Jenkins builds counted for nothing. + self.team_repos = list( + team_repos if team_repos is not None else [FakeTeamRepo()] + ) def get_repo_workflow_by_repo_ids(self, repo_ids, type) -> List[FakeWorkflow]: return [ @@ -154,7 +245,9 @@ def get_repo_workflow_by_id(self, repo_workflow_id) -> Optional[FakeWorkflow]: return w return None - def create_jenkins_repo_workflow(self, jenkins_workflow, workflows_to_deactivate): + def create_jenkins_repo_workflow( + self, jenkins_workflow, workflows_to_deactivate, team_repos_to_update=None + ): if jenkins_workflow not in self.workflows: for w in self.workflows: if ( @@ -173,10 +266,16 @@ def create_jenkins_repo_workflow(self, jenkins_workflow, workflows_to_deactivate self.workflows.append(jenkins_workflow) return jenkins_workflow - def deactivate_repo_workflow(self, repo_workflow, workflows_to_reactivate=None): + def deactivate_repo_workflow( + self, repo_workflow, workflows_to_reactivate=None, team_repos_to_update=None + ): repo_workflow.is_active = False return repo_workflow + @property + def deployment_types(self) -> List[TeamReposDeploymentType]: + return [team_repo.deployment_type for team_repo in self.team_repos] + @property def active_deployment_workflows(self) -> List[FakeWorkflow]: return [w for w in self.workflows if w.is_active] @@ -191,14 +290,19 @@ def _build_client(): @pytest.fixture def routes(request): """ - Wires the two Jenkins mapping routes to in-memory services. The repo lookup + Wires the Jenkins mapping routes to in-memory services. The repo lookup resolves REPO_ID into ORG_ID unless the test overrides it. + + The parameter is either a list of workflows or a dict of + {"workflows": [...], "team_repos": [...]}. """ - workflow_repo_service = FakeWorkflowRepoService(getattr(request, "param", None)) - code_repo_service = MagicMock() - code_repo_service.get_repo_by_id.side_effect = lambda repo_id: ( - FakeOrgRepo(repo_id=repo_id) if str(repo_id) == REPO_ID else None + param = getattr(request, "param", None) or {} + if isinstance(param, list): + param = {"workflows": param} + workflow_repo_service = FakeWorkflowRepoService( + param.get("workflows"), param.get("team_repos") ) + code_repo_service = FakeCodeRepoService(workflow_repo_service) with patch.object( integrations_module, "get_query_validator", return_value=MagicMock() @@ -275,6 +379,155 @@ def test_unmapping_restores_the_github_actions_workflow(routes): assert [w.provider for w in active] == [RepoWorkflowProviders.GITHUB_ACTIONS] +def test_mapping_switches_the_repo_to_workflow_deployments(routes): + # The gap that made the whole integration invisible: 14 builds ingested, + # zero deployments counted, because DeploymentsService only consults + # RepoWorkflowRuns for team repos on WORKFLOW and a mapped repo sat on + # PR_MERGE. + client, service = routes + assert service.deployment_types == [TeamReposDeploymentType.PR_MERGE] + + response = _map(client, "deploy-api") + + assert response.status_code == 200 + assert response.json["switched_team_repos"] == 1 + assert service.deployment_types == [TeamReposDeploymentType.WORKFLOW] + + +def test_mapping_moves_the_repo_into_the_workflow_side_of_the_real_splitter(routes): + """ + The assertion above is only meaningful because of this one. DeploymentsService + splits a team's repos on deployment_type and asks the workflow-based service + for nothing but the WORKFLOW side, so a mapped repo on the PR_MERGE side + counts none of its RepoWorkflowRuns however many were ingested. Driven + through the real splitter rather than a restatement of it. + """ + client, service = routes + deployments_service = DeploymentsService(None, None, None, None) + + before, _ = deployments_service.get_filtered_team_repos_by_deployment_config( + service.team_repos + ) + assert before == [] + + _map(client, "deploy-api") + + after, on_pr_merge = ( + deployments_service.get_filtered_team_repos_by_deployment_config( + service.team_repos + ) + ) + assert after == service.team_repos + assert on_pr_merge == [] + + +@pytest.mark.parametrize( + "routes", + [ + { + "team_repos": [ + FakeTeamRepo(team_id=TEAM_ID), + FakeTeamRepo(team_id=OTHER_TEAM_ID), + # Another repo's row, and one this repo no longer belongs to. + FakeTeamRepo(team_id=TEAM_ID, org_repo_id=uuid4_str()), + FakeTeamRepo(team_id=OTHER_TEAM_ID, is_active=False), + ] + } + ], + indirect=True, +) +def test_mapping_switches_every_team_that_tracks_the_repo(routes): + client, service = routes + + assert _map(client, "deploy-api").json["switched_team_repos"] == 2 + + assert service.deployment_types == [ + TeamReposDeploymentType.WORKFLOW, + TeamReposDeploymentType.WORKFLOW, + # A different repo, and an inactive membership: neither is this + # mapping's business. + TeamReposDeploymentType.PR_MERGE, + TeamReposDeploymentType.PR_MERGE, + ] + + +def test_unmapping_restores_the_deployment_type_the_mapping_overwrote(routes): + client, service = routes + _map(client, "deploy-api") + jenkins_workflow = service.get_repo_workflow_by_repo_id_and_provider_workflow_id( + REPO_ID, "deploy-api" + ) + assert jenkins_workflow.meta["jenkins_previous_deployment_types"] == { + TEAM_ID: "PR_MERGE" + } + + response = _unmap(client, jenkins_workflow.id) + + assert response.status_code == 200 + assert response.json["restored_team_repos"] == 1 + assert service.deployment_types == [TeamReposDeploymentType.PR_MERGE] + + +@pytest.mark.parametrize( + "routes", + [{"team_repos": [FakeTeamRepo(deployment_type=TeamReposDeploymentType.WORKFLOW)]}], + indirect=True, +) +def test_unmapping_leaves_a_repo_that_was_already_workflow_on_workflow(routes): + # The repo was on WORKFLOW before Jenkins ever touched it -- an admin ran it + # off GitHub Actions. Mapping changed nothing there, so unmapping must not + # either: demoting it to PR_MERGE would silently rewrite a setting the + # mapping never owned. + client, service = routes + + assert _map(client, "deploy-api").json["switched_team_repos"] == 0 + jenkins_workflow = service.get_repo_workflow_by_repo_id_and_provider_workflow_id( + REPO_ID, "deploy-api" + ) + assert jenkins_workflow.meta["jenkins_previous_deployment_types"] == {} + + response = _unmap(client, jenkins_workflow.id) + + assert response.status_code == 200 + assert response.json["restored_team_repos"] == 0 + assert service.deployment_types == [TeamReposDeploymentType.WORKFLOW] + + +def test_remapping_carries_the_original_deployment_types_to_the_new_row(routes): + # Map the wrong job, then the right one: the second mapping finds the repo + # already on WORKFLOW and records nothing of its own. Without carrying the + # first mapping's record forward, unmapping strands the repo on WORKFLOW + # with no active workflow behind it -- zero deployments, no error. + client, service = routes + _map(client, "deploy-api") + _map(client, "deploy-api-v2") + + second = service.get_repo_workflow_by_repo_id_and_provider_workflow_id( + REPO_ID, "deploy-api-v2" + ) + assert second.meta["jenkins_previous_deployment_types"] == {TEAM_ID: "PR_MERGE"} + + assert _unmap(client, second.id).json["restored_team_repos"] == 1 + assert service.deployment_types == [TeamReposDeploymentType.PR_MERGE] + + +def test_unmapping_a_mapping_with_no_deployment_type_record_changes_nothing(routes): + # A mapping written before the record existed. Guessing PR_MERGE would + # demote repos the record cannot speak for. + client, service = routes + _map(client, "deploy-api") + jenkins_workflow = service.get_repo_workflow_by_repo_id_and_provider_workflow_id( + REPO_ID, "deploy-api" + ) + jenkins_workflow.meta = {} + + response = _unmap(client, jenkins_workflow.id) + + assert response.status_code == 200 + assert response.json["restored_team_repos"] == 0 + assert service.deployment_types == [TeamReposDeploymentType.WORKFLOW] + + @pytest.mark.parametrize( "routes", [ From f2a73ef1d0d9890a96d490887632c6408524b68b Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Tue, 11 Aug 2026 17:56:31 +0500 Subject: [PATCH 083/194] feat(jenkins): list the workspace's existing job mappings GET /orgs//integrations/jenkins/mappings returns the active Jenkins deployment workflows for the workspace as [{repo_workflow_id, org_repo_id, job_full_name, repo_name}]. Nothing could read a mapping back, so the mapping table reloaded showing "Select a Jenkins job" for a repo that was mapped and ingesting deployments -- the admin's configuration looked lost -- and nothing could supply the repo_workflow_id the DELETE route needs, leaving no way to unmap from the UI. Scoping is an OrgRepo join inside the query rather than a filter the route applies, since RepoWorkflow carries no org column and that join is the only thing between an admin and another workspace's mappings. Co-Authored-By: Claude Opus 5 --- .../analytics_server/mhq/api/integrations.py | 28 +++++ .../mhq/store/repos/workflows.py | 27 +++++ .../tests/api/test_jenkins_routes.py | 111 +++++++++++++++++- 3 files changed, 165 insertions(+), 1 deletion(-) diff --git a/backend/analytics_server/mhq/api/integrations.py b/backend/analytics_server/mhq/api/integrations.py index 511223bc..8c04f1b4 100644 --- a/backend/analytics_server/mhq/api/integrations.py +++ b/backend/analytics_server/mhq/api/integrations.py @@ -378,6 +378,34 @@ def get_jenkins_jobs(org_id: str): ) +# CLUSTOX: the workspace's live Jenkins mappings. Without this the mapping table +# reloads showing "Select a Jenkins job" for a repo that is mapped and ingesting +# deployments -- an admin's configuration looks lost -- and nothing can supply +# the repo_workflow_id the DELETE route needs, so a mapping cannot be undone +# from the UI at all. +@app.route("/orgs//integrations/jenkins/mappings", methods={"GET"}) +def get_jenkins_mappings(org_id: str): + query_validator = get_query_validator() + query_validator.org_validator(org_id) + + # Active Jenkins deployment workflows only: an inactive row is a mapping the + # admin removed, and offering it back would misreport where a repo's + # deployments come from. Scoping is the OrgRepo join inside the query. + mappings = WorkflowRepoService().get_active_repo_workflows_by_org_id_and_provider( + org_id, RepoWorkflowProviders.JENKINS, RepoWorkflowType.DEPLOYMENT + ) + + return [ + { + "repo_workflow_id": str(repo_workflow.id), + "org_repo_id": str(repo_workflow.org_repo_id), + "job_full_name": repo_workflow.provider_workflow_id, + "repo_name": org_repo.name, + } + for repo_workflow, org_repo in mappings + ] + + # CLUSTOX: maps a Jenkins job to a repo as its deployment source. Enforces the # one-active-deployment-source-per-repo invariant and reuses the row for an # already-known (org_repo_id, provider_workflow_id) pair. diff --git a/backend/analytics_server/mhq/store/repos/workflows.py b/backend/analytics_server/mhq/store/repos/workflows.py index b0603624..a248296e 100644 --- a/backend/analytics_server/mhq/store/repos/workflows.py +++ b/backend/analytics_server/mhq/store/repos/workflows.py @@ -185,6 +185,33 @@ def get_repo_workflows_by_repo_id_and_provider( .all() ) + # CLUSTOX: the workspace's live mappings, for the route that lists them. + # Scoped by joining OrgRepo rather than by filtering ids the caller supplies: + # RepoWorkflow has no org column, so the join is the only thing standing + # between an admin and another workspace's mappings, and doing it in SQL + # means there is no path where the filter is forgotten. Returns OrgRepo + # alongside because the mapping table shows repo names, and fetching them + # one by one afterwards is a query per row. + @rollback_on_exc + def get_active_repo_workflows_by_org_id_and_provider( + self, + org_id: str, + provider: RepoWorkflowProviders, + type: RepoWorkflowType, + ) -> List[Tuple[RepoWorkflow, OrgRepo]]: + return ( + self._db.session.query(RepoWorkflow, OrgRepo) + .options(defer(RepoWorkflow.meta)) + .join(OrgRepo, RepoWorkflow.org_repo_id == OrgRepo.id) + .filter( + OrgRepo.org_id == org_id, + RepoWorkflow.provider == provider, + RepoWorkflow.type == type, + RepoWorkflow.is_active.is_(True), + ) + .all() + ) + # CLUSTOX: writes the Jenkins mapping, the deactivation of the repo's other # deployment workflows and the TeamRepos.deployment_type switch in a single # commit. If any part applied on its own, a repo could be left with two diff --git a/backend/analytics_server/tests/api/test_jenkins_routes.py b/backend/analytics_server/tests/api/test_jenkins_routes.py index ce5a38ce..02992925 100644 --- a/backend/analytics_server/tests/api/test_jenkins_routes.py +++ b/backend/analytics_server/tests/api/test_jenkins_routes.py @@ -158,9 +158,20 @@ def test_restoring_leaves_a_row_the_admin_has_since_changed_alone(): class FakeOrgRepo: - def __init__(self, repo_id=REPO_ID, org_id=ORG_ID): + def __init__(self, repo_id=REPO_ID, org_id=ORG_ID, name="middleware"): self.id = repo_id self.org_id = org_id + self.name = name + + +def org_repo_for(repo_id) -> FakeOrgRepo: + """ + Stands in for the OrgRepo join. Only REPO_ID belongs to ORG_ID; every other + repo belongs to another workspace, which is what the scoping tests turn on. + """ + if str(repo_id) == REPO_ID: + return FakeOrgRepo() + return FakeOrgRepo(repo_id=repo_id, org_id=OTHER_ORG_ID, name="someone-elses-repo") class FakeCodeRepoService: @@ -206,6 +217,18 @@ def __init__( team_repos if team_repos is not None else [FakeTeamRepo()] ) + def get_active_repo_workflows_by_org_id_and_provider(self, org_id, provider, type): + # Reproduces the OrgRepo join: a workflow belongs to the workspace its + # repo belongs to, and no query the route makes can widen that. + return [ + (w, org_repo_for(w.org_repo_id)) + for w in self.workflows + if w.provider == provider + and w.type == type + and w.is_active + and str(org_repo_for(w.org_repo_id).org_id) == str(org_id) + ] + def get_repo_workflow_by_repo_ids(self, repo_ids, type) -> List[FakeWorkflow]: return [ w @@ -328,6 +351,92 @@ def _unmap(client, repo_workflow_id, org_id=ORG_ID): ) +def _list(client, org_id=ORG_ID): + return client.get(f"/orgs/{org_id}/integrations/jenkins/mappings") + + +def test_listing_returns_the_workspaces_live_mappings(routes): + # The mapping table reloaded showing "Select a Jenkins job" for a repo that + # was mapped and ingesting deployments, and nothing could supply the + # repo_workflow_id the DELETE route needs -- so a mapping could not be + # undone from the UI at all. + client, service = routes + _map(client, "platform/deploy-api") + jenkins_workflow = service.get_repo_workflow_by_repo_id_and_provider_workflow_id( + REPO_ID, "platform/deploy-api" + ) + + response = _list(client) + + assert response.status_code == 200 + assert response.json == [ + { + "repo_workflow_id": str(jenkins_workflow.id), + "org_repo_id": REPO_ID, + "job_full_name": "platform/deploy-api", + "repo_name": "middleware", + } + ] + # The id it hands back is the one the DELETE route accepts. + assert _unmap(client, response.json[0]["repo_workflow_id"]).status_code == 200 + + +def test_listing_is_empty_for_a_workspace_with_no_mappings(routes): + client, _ = routes + + response = _list(client) + + assert response.status_code == 200 + assert response.json == [] + + +@pytest.mark.parametrize( + "routes", + [ + [ + FakeWorkflow( + RepoWorkflowProviders.JENKINS, + org_repo_id="66666666-6666-4666-8666-666666666666", + provider_workflow_id="their-deploy-job", + ) + ] + ], + indirect=True, +) +def test_listing_does_not_leak_another_workspaces_mappings(routes): + client, _ = routes + _map(client, "our-deploy-job") + + ours = _list(client) + theirs = _list(client, org_id=OTHER_ORG_ID) + + assert [m["job_full_name"] for m in ours.json] == ["our-deploy-job"] + # Same authenticated admin, another workspace's org id in the path. The + # scoping is the OrgRepo join, not anything the caller passes. + assert [m["job_full_name"] for m in theirs.json] == ["their-deploy-job"] + + +@pytest.mark.parametrize( + "routes", + [[FakeWorkflow(RepoWorkflowProviders.GITHUB_ACTIONS, provider_workflow_id="9931")]], + indirect=True, +) +def test_listing_covers_neither_other_providers_nor_removed_mappings(routes): + client, service = routes + _map(client, "deploy-api") + jenkins_workflow = service.get_repo_workflow_by_repo_id_and_provider_workflow_id( + REPO_ID, "deploy-api" + ) + + assert len(_list(client).json) == 1 + + _unmap(client, jenkins_workflow.id) + + # The GitHub Actions workflow is active again, but it is not a Jenkins + # mapping and the removed Jenkins row is not one any more. + assert _list(client).json == [] + + def test_mapping_a_second_job_leaves_exactly_one_active_deployment_source(routes): client, service = routes From bc7d5b6650dddd03519d7110b51ca62c65da9508 Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Tue, 11 Aug 2026 18:00:42 +0500 Subject: [PATCH 084/194] fix(jenkins): refuse internal addresses and stop following redirects base_url was validated in the browser only. The server stored whatever it was given and fetched it with redirects enabled, so an authenticated admin could point a workspace at http://169.254.169.254/latest, or any address inside our network, and read the response back out of the jobs endpoint. mhq/exapi/jenkins.py now requires an http(s) scheme, resolves the host before issuing any request and refuses if *any* resolved address is loopback, RFC1918 private, link-local, reserved, multicast or unspecified -- including an IPv4 address wearing an IPv6 mapping. Every request sets allow_redirects=False, so a public host cannot bounce us somewhere the check never saw, and a 3xx is now named rather than parsed as an empty instance. A refused address raises JenkinsAddressNotAllowed, a RequestException so the existing handling still catches it, with its own type so get_jenkins_jobs can answer 400 "this address is not allowed" instead of 502 "could not reach Jenkins" and send the admin to the right place. TLS verification stays on with no override. Verified against the real https://jenkins-gpu.theclustox.com, which resolves publicly, is allowed, and answers /api/json with 403 rather than a redirect. Co-Authored-By: Claude Opus 5 --- .../analytics_server/mhq/api/integrations.py | 8 +- backend/analytics_server/mhq/exapi/jenkins.py | 121 +++++++++++++- .../tests/api/test_jenkins_routes.py | 16 ++ .../tests/exapi/test_jenkins.py | 156 +++++++++++++++++- 4 files changed, 291 insertions(+), 10 deletions(-) diff --git a/backend/analytics_server/mhq/api/integrations.py b/backend/analytics_server/mhq/api/integrations.py index 8c04f1b4..f1f95bc1 100644 --- a/backend/analytics_server/mhq/api/integrations.py +++ b/backend/analytics_server/mhq/api/integrations.py @@ -354,7 +354,7 @@ def get_jenkins_jobs(org_id: str): # mhq/service/workflows/sync/etl_jenkins_handler.py. import requests - from mhq.exapi.jenkins import JenkinsApiService + from mhq.exapi.jenkins import JenkinsAddressNotAllowed, JenkinsApiService from mhq.store.repos.core import CoreRepoService from mhq.utils.jenkins import get_jenkins_config @@ -368,6 +368,12 @@ def get_jenkins_jobs(org_id: str): try: return JenkinsApiService(base_url, username, api_token).get_jobs() + except JenkinsAddressNotAllowed as e: + # CLUSTOX: caught ahead of RequestException, which it subclasses. The + # configured URL points somewhere this server refuses to fetch, which is + # a 400 the admin can fix -- reported as "could not reach Jenkins" it + # reads as a network problem and sends him to look at a firewall. + return jsonify({"error": str(e)}), 400 except requests.RequestException as e: # An unreachable, slow or erroring Jenkins is not a bug in this server. # Surfaced as a 500 it reached the setup form as "check your diff --git a/backend/analytics_server/mhq/exapi/jenkins.py b/backend/analytics_server/mhq/exapi/jenkins.py index d3b16fd5..267bdaf1 100644 --- a/backend/analytics_server/mhq/exapi/jenkins.py +++ b/backend/analytics_server/mhq/exapi/jenkins.py @@ -1,9 +1,12 @@ # CLUSTOX: Jenkins REST client. Knows Jenkins' API shape and nothing about our # models -- adaptation to RepoWorkflowRuns lives in the ETL handler. +import ipaddress import json +import socket from datetime import datetime from time import monotonic -from typing import Dict, List, Optional, Tuple +from typing import Dict, List, Optional, Tuple, Union +from urllib.parse import urlsplit import pytz import requests @@ -60,6 +63,95 @@ def job_path(job_full_name: str) -> str: return "/".join(f"job/{segment}" for segment in job_full_name.split("/")) +# CLUSTOX: base_url arrives from an admin through the setup form and the server +# then fetches it. The browser validated it; the server did not, so an +# authenticated admin could point the workspace at http://169.254.169.254/latest +# or any address inside our network and read the response back out of the jobs +# endpoint -- cloud instance credentials included. Everything below exists to +# stop that. +ALLOWED_SCHEMES = ("http", "https") + + +class JenkinsAddressNotAllowed(requests.RequestException): + """ + Raised before any request leaves the process. A RequestException so the + routes that already handle an unreachable Jenkins keep working, but its own + type so they can say "this address is not allowed" rather than "Jenkins is + unreachable" -- an admin who typed an internal hostname needs to be told + which of the two happened. + """ + + +def _resolved_addresses(hostname: str, port: Optional[int]) -> List[str]: + try: + infos = socket.getaddrinfo( + hostname, port or 0, type=socket.SOCK_STREAM, proto=socket.IPPROTO_TCP + ) + except socket.gaierror as e: + # A name that does not resolve is an unreachable Jenkins, not an + # attempt to reach somewhere it should not -- different message, and + # the existing 502 path is the right home for it. + raise requests.ConnectionError( + f"Could not resolve Jenkins host {hostname}: {e}" + ) + return [info[4][0] for info in infos] + + +def _is_disallowed(ip: Union[ipaddress.IPv4Address, ipaddress.IPv6Address]) -> bool: + # An IPv4 address written as ::ffff:10.0.0.5 is an IPv4 address, and + # is_private on the IPv6 form does not say so. + mapped = getattr(ip, "ipv4_mapped", None) + if mapped is not None: + ip = mapped + return ( + ip.is_private # RFC1918, and loopback and link-local with it + or ip.is_loopback + or ip.is_link_local # 169.254.0.0/16, the cloud metadata range + or ip.is_reserved + or ip.is_multicast + or ip.is_unspecified + ) + + +def assert_url_is_allowed(url: str) -> None: + """ + Rejects anything that is not an http(s) URL pointing at a public address. + Called before the request rather than after, because a request to + 169.254.169.254 has already leaked whatever it was going to leak by the + time a response comes back. + """ + parts = urlsplit(url) + if parts.scheme not in ALLOWED_SCHEMES: + raise JenkinsAddressNotAllowed( + f"Jenkins URL must start with http:// or https://, got '{url}'" + ) + + try: + hostname, port = parts.hostname, parts.port + except ValueError as e: + raise JenkinsAddressNotAllowed(f"Jenkins URL '{url}' is not valid: {e}") + if not hostname: + raise JenkinsAddressNotAllowed(f"Jenkins URL '{url}' has no host") + + # Every address the name resolves to, not the first: a hostname an admin + # controls can resolve to a public address and an internal one at once, and + # which of them is connected to is not ours to predict. + for address in _resolved_addresses(hostname, port): + try: + ip = ipaddress.ip_address(address) + except ValueError: + raise JenkinsAddressNotAllowed( + f"Jenkins host {hostname} resolved to something that is not an " + f"IP address: {address}" + ) + if _is_disallowed(ip): + raise JenkinsAddressNotAllowed( + f"Jenkins host {hostname} resolves to {address}, which is a " + f"private, loopback or link-local address. Middleware will not " + f"fetch internal addresses." + ) + + class JenkinsResponse: """ The slice of requests.Response this client uses, with the body already read @@ -67,15 +159,26 @@ class JenkinsResponse: reading, which a plain Response cannot do. """ - def __init__(self, status_code: int, body: bytes, url: str): + def __init__(self, status_code: int, body: bytes, url: str, location: str = None): self.status_code = status_code self._body = body self._url = url + self._location = location def json(self): return json.loads(self._body or b"{}") def raise_for_status(self) -> None: + # CLUSTOX: redirects are no longer followed, so a 3xx arrives here with + # an empty body. Left to fall through it parses as {} and reports "no + # jobs" -- a Jenkins behind a login redirect or a moved URL would look + # like an empty one. Named instead. + if 300 <= self.status_code < 400: + raise requests.HTTPError( + f"Jenkins redirected {self._url} to " + f"'{self._location or 'an unnamed location'}'. Redirects are " + f"not followed; check the base URL is the one Jenkins serves." + ) if self.status_code >= 400: raise requests.HTTPError( f"Jenkins returned HTTP {self.status_code} for {self._url}" @@ -98,6 +201,10 @@ def __init__( def _get(self, path: str) -> JenkinsResponse: url = f"{self._base_url}/{path}" + # CLUSTOX: checked per request, not once in __init__. A service object + # outlives a DNS record, and the check is worth nothing if the address + # it approved is not the one being fetched now. + assert_url_is_allowed(url) deadline = monotonic() + self._max_seconds connect_timeout, read_timeout = self._timeout # CLUSTOX: a read timeout longer than the total ceiling can never be @@ -115,6 +222,12 @@ def _get(self, path: str) -> JenkinsResponse: timeout=(connect_timeout, read_timeout), verify=True, stream=True, + # CLUSTOX: the address check above covers the URL we ask for, and a + # followed redirect is a second URL nobody checked. A public host -- + # the admin's own, or one that answers it -- redirecting to + # 169.254.169.254 walks straight past the check. Not following + # means the check and the request are about the same address. + allow_redirects=False, ) body = bytearray() try: @@ -145,7 +258,9 @@ def _get(self, path: str) -> JenkinsResponse: body.extend(chunk) finally: response.close() - return JenkinsResponse(response.status_code, bytes(body), url) + return JenkinsResponse( + response.status_code, bytes(body), url, response.headers.get("Location") + ) def check_pat(self) -> bool: response = self._get("api/json") diff --git a/backend/analytics_server/tests/api/test_jenkins_routes.py b/backend/analytics_server/tests/api/test_jenkins_routes.py index 02992925..3424af01 100644 --- a/backend/analytics_server/tests/api/test_jenkins_routes.py +++ b/backend/analytics_server/tests/api/test_jenkins_routes.py @@ -13,6 +13,7 @@ restore_team_repo_deployment_types, switch_team_repos_to_workflow_deployments, ) +from mhq.exapi.jenkins import JenkinsAddressNotAllowed from mhq.service.deployments.deployment_service import DeploymentsService from mhq.store.models.code import RepoWorkflowProviders, RepoWorkflowType from mhq.store.models.code.enums import TeamReposDeploymentType @@ -843,6 +844,21 @@ def test_an_unreachable_jenkins_is_a_502_not_a_500(): assert "Could not reach Jenkins" in response.json["error"] +def test_a_refused_address_is_told_apart_from_an_unreachable_jenkins(): + response = _jobs_route( + JenkinsAddressNotAllowed( + "Jenkins host metadata resolves to 169.254.169.254, which is a " + "private, loopback or link-local address." + ) + ) + + # A 502 "Could not reach Jenkins" would send the admin to look at a + # firewall for a URL this server is refusing on purpose. + assert response.status_code == 400 + assert "169.254.169.254" in response.json["error"] + assert "Could not reach Jenkins" not in response.json["error"] + + def test_a_reachable_jenkins_returns_its_jobs(): response = _jobs_route([{"name": "deploy-api", "full_name": "deploy-api"}]) diff --git a/backend/analytics_server/tests/exapi/test_jenkins.py b/backend/analytics_server/tests/exapi/test_jenkins.py index b7619480..f1cfdd48 100644 --- a/backend/analytics_server/tests/exapi/test_jenkins.py +++ b/backend/analytics_server/tests/exapi/test_jenkins.py @@ -9,7 +9,15 @@ import pytz import requests -from mhq.exapi.jenkins import JOB_TREE, JenkinsApiService, job_path +from unittest.mock import patch + +from mhq.exapi.jenkins import ( + JOB_TREE, + JenkinsAddressNotAllowed, + JenkinsApiService, + assert_url_is_allowed, + job_path, +) from tests.factories.models.exapi.jenkins import get_jenkins_nested_jobs_dict @@ -203,7 +211,18 @@ def _service(url, max_seconds=5): ) -def test_a_dribbling_jenkins_is_cut_off_at_the_ceiling(): +# CLUSTOX: LocalJenkins listens on 127.0.0.1, which assert_url_is_allowed exists +# to refuse. These tests are about reading a real socket under a deadline, not +# about which addresses are allowed -- that has its own tests below -- so they +# stand the guard down rather than working around it with a fake response, which +# is what previously hid a bug in this exact code path. +@pytest.fixture +def loopback_allowed(): + with patch("mhq.exapi.jenkins.assert_url_is_allowed"): + yield + + +def test_a_dribbling_jenkins_is_cut_off_at_the_ceiling(loopback_allowed): # 4000 bytes handed over 50 at a time every 50ms: four seconds to complete, # with every gap comfortably inside the read timeout, so nothing but the # total ceiling can stop it. The whole body is under one 64 KiB chunk, so @@ -224,12 +243,12 @@ def test_a_dribbling_jenkins_is_cut_off_at_the_ceiling(): assert elapsed < 2.5, f"ceiling did not bound the read: took {elapsed:.1f}s" -def test_a_prompt_jenkins_is_read_in_full(): +def test_a_prompt_jenkins_is_read_in_full(loopback_allowed): with LocalJenkins(b'{"jobs": []}') as jenkins: assert _service(jenkins.url).get_jobs() == [] -def test_a_body_larger_than_one_read_is_reassembled(): +def test_a_body_larger_than_one_read_is_reassembled(loopback_allowed): # read1() returns whatever is available rather than a full chunk, so the # loop has to keep going until EOF instead of stopping at the first short # read. @@ -254,7 +273,7 @@ def test_a_body_larger_than_one_read_is_reassembled(): assert jobs[-1]["full_name"] == "deploy-1999" -def test_a_gzipped_response_is_decoded(): +def test_a_gzipped_response_is_decoded(loopback_allowed): # requests advertises gzip on every request. Reading response.raw without # decode_content hands json.loads a gzip stream. body = gzip.compress(json.dumps(get_jenkins_nested_jobs_dict()).encode()) @@ -270,10 +289,135 @@ def test_a_gzipped_response_is_decoded(): ] -def test_an_http_error_is_raised_rather_than_parsed(): +def test_an_http_error_is_raised_rather_than_parsed(loopback_allowed): with LocalJenkins(b"forbidden", status_line=b"HTTP/1.1 403 Forbidden") as jenkins: service = _service(jenkins.url) assert service.check_pat() is False with pytest.raises(requests.HTTPError): service.get_jobs() + + +# CLUSTOX: base_url is admin-supplied and the server fetches it. ClustoxJenkinsSetup.tsx +# checks it in the browser, which is no check at all: an authenticated admin +# could point a workspace at http://169.254.169.254/latest and read the cloud +# instance credentials back out of the jobs endpoint. None of these tests +# resolve a name for real -- the resolver is patched, so they say what the guard +# does rather than what this machine's DNS happens to answer. +def _resolving_to(*addresses): + """Patches the resolver the guard uses, in the form getaddrinfo returns.""" + return patch( + "mhq.exapi.jenkins.socket.getaddrinfo", + return_value=[ + (socket.AF_INET, socket.SOCK_STREAM, 6, "", (address, 8080)) + for address in addresses + ], + ) + + +@pytest.mark.parametrize( + "url, address", + [ + ("http://127.0.0.1:8080", "127.0.0.1"), + ("http://10.0.0.5", "10.0.0.5"), + ("https://192.168.1.1", "192.168.1.1"), + # The cloud metadata endpoint: the reason this guard exists. + ("http://169.254.169.254/latest", "169.254.169.254"), + ("http://localhost:8080", "127.0.0.1"), + ("http://jenkins.internal", "172.16.4.9"), + ("http://[::1]", "::1"), + # An IPv4 private address wearing an IPv6 hat. + ("http://jenkins.internal", "::ffff:10.0.0.5"), + ], +) +def test_an_internal_address_is_refused(url, address): + with _resolving_to(address): + with pytest.raises(JenkinsAddressNotAllowed) as raised: + assert_url_is_allowed(url) + + assert address in str(raised.value) + + +def test_a_host_resolving_to_both_a_public_and_a_private_address_is_refused(): + # Rejecting on the first address only lets a hostname whose A records are + # public-then-private through, and which one is connected to is not ours to + # predict. + with _resolving_to("69.30.247.141", "10.0.0.5"): + with pytest.raises(JenkinsAddressNotAllowed): + assert_url_is_allowed("https://jenkins.example.com") + + +def test_the_real_jenkins_this_was_built_against_is_still_allowed(): + # https://jenkins-gpu.theclustox.com resolves to a public address. A guard + # that blocks the one Jenkins in use is not a fix. + with _resolving_to("69.30.247.141"): + assert_url_is_allowed("https://jenkins-gpu.theclustox.com/api/json") + + +@pytest.mark.parametrize( + "url", + [ + "file:///etc/passwd", + "gopher://jenkins.example.com", + "ftp://jenkins.example.com", + "jenkins.example.com", + "https://", + ], +) +def test_only_http_urls_with_a_host_are_accepted(url): + with pytest.raises(JenkinsAddressNotAllowed): + assert_url_is_allowed(url) + + +def test_a_name_that_does_not_resolve_reads_as_unreachable_not_as_refused(): + # Two different problems for the admin: an address we will not fetch is one + # he retypes, a name that does not resolve is one he checks DNS for. + with patch( + "mhq.exapi.jenkins.socket.getaddrinfo", side_effect=socket.gaierror("no such") + ): + with pytest.raises(requests.ConnectionError): + assert_url_is_allowed("https://jenkins.example.com") + + +def test_nothing_is_requested_when_the_address_is_refused(): + # The check is worth nothing after the fact: a request to 169.254.169.254 + # has leaked whatever it was going to leak by the time a response arrives. + with _resolving_to("169.254.169.254"), patch( + "mhq.exapi.jenkins.requests.get" + ) as requests_get: + with pytest.raises(JenkinsAddressNotAllowed): + _service("http://169.254.169.254").get_jobs() + + requests_get.assert_not_called() + + +def test_redirects_are_not_followed(): + # A public host answering with a redirect to an internal one walks straight + # past an address check performed on the URL we asked for. + with _resolving_to("69.30.247.141"), patch( + "mhq.exapi.jenkins.requests.get" + ) as requests_get: + requests_get.return_value.status_code = 302 + requests_get.return_value.headers = {"Location": "http://169.254.169.254/"} + requests_get.return_value.raw.read1.return_value = b"" + + with pytest.raises(requests.HTTPError) as raised: + _service("https://jenkins.example.com").get_jobs() + + assert requests_get.call_args.kwargs["allow_redirects"] is False + # Falling through as a 200-with-no-body would have parsed as {} and reported + # a redirecting Jenkins as an empty one. + assert "169.254.169.254" in str(raised.value) + + +def test_tls_verification_is_not_negotiable(): + with _resolving_to("69.30.247.141"), patch( + "mhq.exapi.jenkins.requests.get" + ) as requests_get: + requests_get.return_value.status_code = 200 + requests_get.return_value.headers = {} + requests_get.return_value.raw.read1.return_value = b"" + + _service("https://jenkins.example.com").check_pat() + + assert requests_get.call_args.kwargs["verify"] is True From 62048094fa7d6d97b6377847168695370078c522 Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Tue, 11 Aug 2026 18:43:43 +0500 Subject: [PATCH 085/194] feat(jenkins): add helpers to forward analytics-server error messages The BFF's generic error path collapses every failed internal call into "Request failed with status code NNN" -- the server's actual explanation never reaches the browser. Jenkins mapping and setup both depend on that text: a 400 naming a blocked base_url and a 409 naming a name collision are each things only an admin can act on, and both need to read as more than "try again". forwardInternalError re-throws an internal-server failure as a ResponseError carrying the server's own status and message. readApiError reads that status/message back out of a handleApi rejection on the client. Neither is wired up yet; that follows in the next commits. Co-Authored-By: Claude Opus 5 --- web-server/src/api-helpers/internal-error.ts | 28 ++++++++++++++++++++ web-server/src/utils/api-error.ts | 22 +++++++++++++++ 2 files changed, 50 insertions(+) create mode 100644 web-server/src/api-helpers/internal-error.ts create mode 100644 web-server/src/utils/api-error.ts diff --git a/web-server/src/api-helpers/internal-error.ts b/web-server/src/api-helpers/internal-error.ts new file mode 100644 index 00000000..18f20397 --- /dev/null +++ b/web-server/src/api-helpers/internal-error.ts @@ -0,0 +1,28 @@ +import { AxiosError } from 'axios'; + +import { ResponseError } from '@/constants/error'; + +/** + * CLUSTOX: re-throws an analytics-server failure with the server's own status + * and message. + * + * The BFF's generic error path (parseError) reads `message` off whatever was + * thrown, and for an axios failure that string is always "Request failed with + * status code 409" -- the server's explanation never reaches the browser, so + * every failure arrives looking identical. Jenkins depends on the difference: + * a 400 naming a base URL this server refuses to fetch and a 409 naming the + * workflow already holding a job id are both things only an admin can fix, and + * both read as "try again" without the text. + */ +export const forwardInternalError = (err: unknown): never => { + const response = (err as AxiosError<{ error?: string }>)?.response; + const message = response?.data?.error; + + // A timeout, a refused connection or an HTML error page carries nothing + // worth showing, so it keeps the generic 400/500 handling rather than + // surfacing a fragment of someone else's stack trace to an admin. + if (response?.status && typeof message === 'string' && message) + throw new ResponseError(message, response.status); + + throw err; +}; diff --git a/web-server/src/utils/api-error.ts b/web-server/src/utils/api-error.ts new file mode 100644 index 00000000..a84e1c2f --- /dev/null +++ b/web-server/src/utils/api-error.ts @@ -0,0 +1,22 @@ +import { AxiosResponse } from 'axios'; + +/** + * CLUSTOX: reads the status and message off a rejected `handleApi` call. + * + * `handleApi` rejects with the AxiosResponse itself rather than an Error, and + * the BFF's error body is `{ message }` (see parseError). Both facts are easy + * to get wrong at each call site, and getting them wrong degrades silently to + * "no message", which is exactly the generic advice a caller reaching for this + * is trying to avoid. + */ +export const readApiError = ( + err: unknown +): { status: number | null; message: string | null } => { + const response = err as AxiosResponse<{ message?: string }> | undefined; + const message = response?.data?.message; + + return { + status: typeof response?.status === 'number' ? response.status : null, + message: typeof message === 'string' && message ? message : null + }; +}; From a6927d36ebb9da96bd47bd19daeac78f526435df Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Tue, 11 Aug 2026 18:43:59 +0500 Subject: [PATCH 086/194] fix(jenkins): show existing mappings and allow unmapping A repo that was mapped to a Jenkins job and had ingested real deployments still showed "Select a Jenkins job" on every page load, because nothing ever read the mappings back from the server -- the table only reflected whatever had been selected since the last refresh. That also meant the repo_workflow_id a DELETE needs to undo a mapping existed nowhere in the UI, so a mapping, once made, could not be removed. - Add the mappings BFF's GET handler, proxying the analytics server's list of active Jenkins mappings, and fetch it on load so a mapped repo reads as mapped and carries its repo_workflow_id. - Add a per-row unmap affordance that calls DELETE with that id and refreshes mapping/team state afterward. - Extend the map confirmation dialog to state that a repo's active GitHub Actions deployment workflows are switched off (when that applies), that the repo starts counting Jenkins runs instead of merged pull requests, and that the change lands on every team tracking the repo -- not only the one currently being viewed. - Add an unmap confirmation dialog that says what reverts, since unmapping is not obviously safe either. - Surface the analytics server's own message on a sub-500 mapping failure (e.g. a 409 naming the workflow that already holds a job id) instead of the generic "try again", which is wrong advice for a collision no retry can fix. Co-Authored-By: Claude Opus 5 --- .../pages/api/clustox/jenkins/mappings.ts | 44 ++- .../src/components/ClustoxJenkinsMapping.tsx | 311 ++++++++++++++---- 2 files changed, 281 insertions(+), 74 deletions(-) diff --git a/web-server/pages/api/clustox/jenkins/mappings.ts b/web-server/pages/api/clustox/jenkins/mappings.ts index 822cab2a..d0fb094a 100644 --- a/web-server/pages/api/clustox/jenkins/mappings.ts +++ b/web-server/pages/api/clustox/jenkins/mappings.ts @@ -1,11 +1,16 @@ -// CLUSTOX: BFF route for the Jenkins job-to-repo mapping. POST creates the -// mapping, DELETE removes it; both proxy straight through to the analytics -// server, which owns the one-active-deployment-source-per-repo invariant. +// CLUSTOX: BFF route for the Jenkins job-to-repo mapping. GET lists what is +// mapped, POST creates a mapping, DELETE removes it; all three proxy straight +// through to the analytics server, which owns the one-active-deployment-source +// -per-repo invariant. import * as yup from 'yup'; -import { Endpoint } from '@/api-helpers/global'; import { internal } from '@/api-helpers/axios'; +import { Endpoint } from '@/api-helpers/global'; +import { forwardInternalError } from '@/api-helpers/internal-error'; +// Only what every method here shares. A method-specific schema in the +// constructor is validated on *all* methods, which made the others a permanent +// 400 -- fixed once already, so the per-method shapes stay below. const pathSchema = yup.object().shape({ org_id: yup.string().uuid().required() }); @@ -23,24 +28,35 @@ const deleteSchema = yup.object().shape({ const endpoint = new Endpoint(pathSchema); +// CLUSTOX: what the mapping table renders as already-mapped. Each row carries +// the repo_workflow_id, which is the only thing that can address a mapping for +// removal -- without this call a mapping can be created but never undone. +endpoint.handle.GET(pathSchema, async (req, res) => { + const result = await internal + .get(`/orgs/${req.payload.org_id}/integrations/jenkins/mappings`) + .catch(forwardInternalError); + res.send({ mappings: result.data }); +}); + endpoint.handle.POST(postSchema, async (req, res) => { // One deployment source per repo: creating this mapping deactivates the - // repo's GitHub Actions workflows so deployments are not counted twice. - const result = await internal.post( - `/orgs/${req.payload.org_id}/integrations/jenkins/mappings`, - { + // repo's GitHub Actions workflows and switches every team tracking the repo + // to workflow-based deployments, so Jenkins runs are what get counted. + const result = await internal + .post(`/orgs/${req.payload.org_id}/integrations/jenkins/mappings`, { org_repo_id: req.payload.org_repo_id, job_full_name: req.payload.job_full_name - } - ); + }) + .catch(forwardInternalError); res.send(result.data); }); endpoint.handle.DELETE(deleteSchema, async (req, res) => { - const result = await internal.delete( - `/orgs/${req.payload.org_id}/integrations/jenkins/mappings`, - { data: { repo_workflow_id: req.payload.repo_workflow_id } } - ); + const result = await internal + .delete(`/orgs/${req.payload.org_id}/integrations/jenkins/mappings`, { + data: { repo_workflow_id: req.payload.repo_workflow_id } + }) + .catch(forwardInternalError); res.send(result.data); }); diff --git a/web-server/src/components/ClustoxJenkinsMapping.tsx b/web-server/src/components/ClustoxJenkinsMapping.tsx index 07b187d8..adce27cd 100644 --- a/web-server/src/components/ClustoxJenkinsMapping.tsx +++ b/web-server/src/components/ClustoxJenkinsMapping.tsx @@ -1,3 +1,4 @@ +import { LinkOff } from '@mui/icons-material'; import { LoadingButton } from '@mui/lab'; import { Alert, @@ -7,6 +8,7 @@ import { DialogActions, DialogContent, DialogTitle, + IconButton, MenuItem, Table, TableBody, @@ -15,6 +17,7 @@ import { TableHead, TableRow, TextField, + Tooltip, useTheme } from '@mui/material'; import { useCallback, useEffect, useMemo, useState } from 'react'; @@ -26,18 +29,70 @@ import { Integration } from '@/constants/integrations'; import { useAuth } from '@/hooks/useAuth'; // CLUSTOX: server-resolved workspace, consistent with ClustoxJenkinsSetup. import { useClustoxUser } from '@/hooks/useClustoxUser'; +import { readApiError } from '@/utils/api-error'; type JenkinsJob = { name: string; full_name: string; url: string }; type WorkspaceRepo = { id: string; name: string; provider: Integration }; -type Pending = { repoId: string; repoName: string; jobFullName: string }; +type JenkinsMapping = { + repo_workflow_id: string; + org_repo_id: string; + job_full_name: string; + repo_name: string; +}; +type Pending = + | { + action: 'map'; + repoId: string; + repoName: string; + jobFullName: string; + deactivatesWorkflows: boolean; + teamCount: number; + } + | { + action: 'unmap'; + repoId: string; + repoName: string; + jobFullName: string; + repoWorkflowId: string; + teamCount: number; + }; + +const codeProviders = [ + Integration.GITHUB, + Integration.GITLAB, + Integration.BITBUCKET +]; + +type TeamReposResponse = { + teamReposMap: Record; +}; + +// CLUSTOX: the change is per repo, not per team, so it lands on every team +// tracking the repo -- including teams this admin is not currently looking at. +const everyTeamSentence = (teamCount: number) => + `This affects every team tracking this repo${ + teamCount > 0 + ? ` (${teamCount === 1 ? '1 team' : `${teamCount} teams`})` + : '' + }, not only the team you are viewing.`; + +// CLUSTOX: the server's own words whenever it sent any, because the cases it +// reports are ones retrying cannot fix -- a 409 means another workflow already +// holds this job id, and "try again" is actively wrong advice there. A 5xx or a +// dropped connection carries nothing to show, so those keep the fallback. +const mutationError = (err: unknown, fallback: string) => { + const { status, message } = readApiError(err); + return status && status < 500 && message ? message : fallback; +}; /** - * CLUSTOX: there is no endpoint to list existing repo-to-job mappings, only - * to create (POST) or remove (DELETE, given a repo_workflow_id this UI never - * receives back) one. So this table's selections reflect what has been - * mapped in the current session, not necessarily what a prior session saved - * -- the mapping itself does persist server-side even though this screen - * cannot re-display it after a reload. + * CLUSTOX: maps each repo to the Jenkins job whose builds count as its + * deployments. + * + * Selections are read back from the server on every load, so a mapped repo + * still reads as mapped after a reload. It used to show "Select a Jenkins job" + * for a repo that was mapped and ingesting builds, which reads as a lost + * configuration and invites an admin to map it a second time. */ export const ClustoxJenkinsMapping = () => { const theme = useTheme(); @@ -47,6 +102,9 @@ export const ClustoxJenkinsMapping = () => { const [repos, setRepos] = useState([]); const [jobs, setJobs] = useState([]); + // CLUSTOX: the live mappings, keyed by repo. The backend allows one active + // deployment source per repo, so a repo has at most one entry here. + const [mappings, setMappings] = useState>({}); // CLUSTOX: repos whose active workflow could be double-counted with a new // Jenkins mapping. Sourced from teams/v2's repo_workflows, the only // existing endpoint that already joins OrgRepo with active workflow rows -- @@ -56,60 +114,94 @@ export const ClustoxJenkinsMapping = () => { const [reposWithActiveWorkflow, setReposWithActiveWorkflow] = useState< Set >(new Set()); - const [selections, setSelections] = useState>({}); + // CLUSTOX: how many teams track each repo. Mapping switches the deployment + // source for all of them, and an admin looking at one team's numbers has no + // other way to know he is changing what the other teams see. + const [teamCounts, setTeamCounts] = useState>({}); const [rowErrors, setRowErrors] = useState>({}); const [savingRepoId, setSavingRepoId] = useState(null); const [pending, setPending] = useState(null); const [loading, setLoading] = useState(true); const [failed, setFailed] = useState(false); + const fetchMappings = useCallback(async () => { + const res = await handleApi<{ mappings: JenkinsMapping[] }>( + '/clustox/jenkins/mappings', + { params: { org_id: orgId } } + ); + return Object.fromEntries( + (res.mappings || []).map((mapping) => [mapping.org_repo_id, mapping]) + ) as Record; + }, [orgId]); + + const fetchTeamState = useCallback(async () => { + const res = await handleApi( + `/resources/orgs/${orgId}/teams/v2`, + { params: { providers: codeProviders } } + ); + + const withWorkflow = new Set(); + const counts: Record = {}; + Object.entries(res.teamReposMap || {}).forEach(([teamId, teamRepos]) => { + teamRepos.forEach((repo) => { + if (repo.repo_workflows?.length) withWorkflow.add(repo.id); + // teams/v2 left-joins TeamRepos, so repos belonging to no team arrive + // under a null key. Counting that bucket would tell an admin one team + // tracks a repo that none does. + if (teamId && teamId !== 'null' && teamId !== 'undefined') + counts[repo.id] = (counts[repo.id] || 0) + 1; + }); + }); + return { withWorkflow, counts }; + }, [orgId]); + const load = useCallback(async () => { if (!orgId) return; setLoading(true); setFailed(false); try { - const codeProviders = [ - Integration.GITHUB, - Integration.GITLAB, - Integration.BITBUCKET - ]; - const [repoRows, jobsRes, teamsRes] = await Promise.all([ + const [repoRows, jobsRes, mappingsByRepo, teamState] = await Promise.all([ handleApi(`/resources/orgs/${orgId}/repos`), handleApi<{ jobs: JenkinsJob[] }>('/clustox/jenkins/jobs', { params: { org_id: orgId } }), - handleApi<{ - teamReposMap: Record< - string, - { id: string; repo_workflows?: unknown[] }[] - >; - }>(`/resources/orgs/${orgId}/teams/v2`, { - params: { providers: codeProviders } - }) + fetchMappings(), + fetchTeamState() ]); setRepos(repoRows || []); setJobs(jobsRes.jobs || []); - - const withWorkflow = new Set(); - Object.values(teamsRes.teamReposMap || {}).forEach((teamRepos) => { - teamRepos.forEach((repo) => { - if (repo.repo_workflows?.length) withWorkflow.add(repo.id); - }); - }); - setReposWithActiveWorkflow(withWorkflow); + setMappings(mappingsByRepo); + setReposWithActiveWorkflow(teamState.withWorkflow); + setTeamCounts(teamState.counts); } catch (e) { console.error('Failed to load Jenkins mapping data', e); setFailed(true); } finally { setLoading(false); } - }, [orgId]); + }, [fetchMappings, fetchTeamState, orgId]); useEffect(() => { load(); }, [load]); + // CLUSTOX: re-read what the server now holds after a mapping changed, rather + // than patching local state to match a guess. Mapping and unmapping both move + // rows this screen only sees indirectly -- displaced workflows, restored + // ones, deployment types -- and the repo_workflow_id needed to undo a fresh + // mapping exists nowhere until it is fetched. The Jenkins job list is left + // alone: it costs a round trip to Jenkins and nothing here changes it. + const refreshMappingState = useCallback(async () => { + const [mappingsByRepo, teamState] = await Promise.all([ + fetchMappings(), + fetchTeamState() + ]); + setMappings(mappingsByRepo); + setReposWithActiveWorkflow(teamState.withWorkflow); + setTeamCounts(teamState.counts); + }, [fetchMappings, fetchTeamState]); + const submitMapping = useCallback( async (repoId: string, jobFullName: string) => { setRowErrors((errs) => ({ ...errs, [repoId]: '' })); @@ -123,44 +215,96 @@ export const ClustoxJenkinsMapping = () => { job_full_name: jobFullName } }); - setSelections((s) => ({ ...s, [repoId]: jobFullName })); - // CLUSTOX: the backend just deactivated this repo's GitHub Actions - // workflows as part of creating the mapping. - setReposWithActiveWorkflow((prev) => { - const next = new Set(prev); - next.delete(repoId); - return next; - }); + await refreshMappingState(); } catch (e) { console.error('Failed to map Jenkins job', e); setRowErrors((errs) => ({ ...errs, - [repoId]: 'Could not save this mapping. Try again.' + [repoId]: mutationError(e, 'Could not save this mapping. Try again.') + })); + } finally { + setSavingRepoId(null); + } + }, + [orgId, refreshMappingState] + ); + + const submitUnmapping = useCallback( + async (repoId: string, repoWorkflowId: string) => { + setRowErrors((errs) => ({ ...errs, [repoId]: '' })); + setSavingRepoId(repoId); + try { + await handleApi('/clustox/jenkins/mappings', { + method: 'delete', + data: { org_id: orgId, repo_workflow_id: repoWorkflowId } + }); + await refreshMappingState(); + } catch (e) { + console.error('Failed to remove Jenkins mapping', e); + setRowErrors((errs) => ({ + ...errs, + [repoId]: mutationError( + e, + 'Could not remove this mapping. Try again.' + ) })); } finally { setSavingRepoId(null); } }, - [orgId] + [orgId, refreshMappingState] + ); + + const askToUnmap = useCallback( + (repo: WorkspaceRepo) => { + const mapping = mappings[repo.id]; + if (!mapping) return; + setPending({ + action: 'unmap', + repoId: repo.id, + repoName: repo.name, + jobFullName: mapping.job_full_name, + repoWorkflowId: mapping.repo_workflow_id, + teamCount: teamCounts[repo.id] || 0 + }); + }, + [mappings, teamCounts] ); const handleSelect = useCallback( (repo: WorkspaceRepo, jobFullName: string) => { - if (!jobFullName) return; - if (reposWithActiveWorkflow.has(repo.id)) { - setPending({ repoId: repo.id, repoName: repo.name, jobFullName }); + // The empty option on a mapped row means "stop using Jenkins here", which + // is the same destructive change as the explicit button beside it. + if (!jobFullName) { + if (mappings[repo.id]) askToUnmap(repo); return; } - submitMapping(repo.id, jobFullName); + if (jobFullName === mappings[repo.id]?.job_full_name) return; + + // CLUSTOX: confirmed for every repo, not only those with a workflow to + // displace. Mapping now also switches the repo to workflow-based + // deployments for every team tracking it, so even a repo with no active + // workflow changes where its deployment numbers come from. + setPending({ + action: 'map', + repoId: repo.id, + repoName: repo.name, + jobFullName, + deactivatesWorkflows: + reposWithActiveWorkflow.has(repo.id) && !mappings[repo.id], + teamCount: teamCounts[repo.id] || 0 + }); }, - [reposWithActiveWorkflow, submitMapping] + [askToUnmap, mappings, reposWithActiveWorkflow, teamCounts] ); const confirmPending = useCallback(async () => { if (!pending) return; - await submitMapping(pending.repoId, pending.jobFullName); + if (pending.action === 'map') + await submitMapping(pending.repoId, pending.jobFullName); + else await submitUnmapping(pending.repoId, pending.repoWorkflowId); setPending(null); - }, [pending, submitMapping]); + }, [pending, submitMapping, submitUnmapping]); const noJobs = useMemo(() => !loading && !jobs.length, [loading, jobs]); @@ -215,7 +359,7 @@ export const ClustoxJenkinsMapping = () => { size="small" sx={{ minWidth: 280 }} disabled={savingRepoId === repo.id || !jobs.length} - value={selections[repo.id] || ''} + value={mappings[repo.id]?.job_full_name || ''} onChange={(e) => handleSelect(repo, e.target.value as string) } @@ -233,13 +377,22 @@ export const ClustoxJenkinsMapping = () => { {savingRepoId === repo.id && ( )} - {selections[repo.id] && - savingRepoId !== repo.id && - !rowErrors[repo.id] && ( + {mappings[repo.id] && savingRepoId !== repo.id && ( + <> Mapped - )} + + askToUnmap(repo)} + > + + + + + )} {rowErrors[repo.id] && ( @@ -262,13 +415,50 @@ export const ClustoxJenkinsMapping = () => { )} setPending(null)}> - Map {pending?.repoName} to Jenkins? + + {pending?.action === 'unmap' + ? `Remove ${pending?.repoName}'s Jenkins mapping?` + : `Map ${pending?.repoName} to Jenkins?`} + - - Mapping a Jenkins job stops counting this repo's GitHub Actions - runs as deployments, so they aren't counted twice. You can undo - this by removing the mapping. - + + {pending?.action === 'unmap' ? ( + <> + + {pending.repoName} will stop counting runs of{' '} + {pending.jobFullName} as deployments, and + go back to the deployment source it had before it was mapped — + its GitHub Actions workflows if this mapping switched any off, + otherwise merged pull requests. + + {everyTeamSentence(pending.teamCount)} + + Builds already ingested from Jenkins are kept. You can map the + job again at any time. + + + ) : ( + <> + + {pending?.repoName} will start counting runs of{' '} + {pending?.jobFullName} as its deployments, + instead of counting merged pull requests. Deployment frequency + and lead time will be measured from Jenkins from now on. + + {pending?.action === 'map' && pending.deactivatesWorkflows && ( + + This repo's active GitHub Actions deployment workflows + will be switched off so the same deployment isn't + counted twice. + + )} + {everyTeamSentence(pending?.teamCount ?? 0)} + + You can undo this by removing the mapping. + + + )} + - Map job + {pending?.action === 'unmap' ? 'Remove mapping' : 'Map job'} From b88b535fea77078ad2f58abfe38a1c4ddbfeacbe Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Tue, 11 Aug 2026 18:44:14 +0500 Subject: [PATCH 087/194] fix(jenkins): report the real reason a Jenkins connection probe fails The setup form reported every failed connection probe as "check your credentials," including a 400 the analytics server now returns when base_url points at an address it refuses to fetch (loopback, link-local, or private -- the SSRF hardening added server-side). An admin who typed an internal hostname was told to recheck a token and username that were never the problem. The jobs BFF route now forwards the analytics server's own status and message instead of swallowing it into a generic axios error. The setup form uses that to show the server's message specifically when the credential save succeeded and the probe itself came back 400 -- any other failure (network error, 5xx, a 400 from the credential save's own schema check) keeps the existing generic connection message, since those don't carry a message about this URL. Co-Authored-By: Claude Opus 5 --- web-server/pages/api/clustox/jenkins/jobs.ts | 13 ++++++++---- .../src/components/ClustoxJenkinsSetup.tsx | 21 ++++++++++++++++++- 2 files changed, 29 insertions(+), 5 deletions(-) diff --git a/web-server/pages/api/clustox/jenkins/jobs.ts b/web-server/pages/api/clustox/jenkins/jobs.ts index ec03c1b0..55d8535a 100644 --- a/web-server/pages/api/clustox/jenkins/jobs.ts +++ b/web-server/pages/api/clustox/jenkins/jobs.ts @@ -1,7 +1,8 @@ import * as yup from 'yup'; -import { Endpoint } from '@/api-helpers/global'; import { internal } from '@/api-helpers/axios'; +import { Endpoint } from '@/api-helpers/global'; +import { forwardInternalError } from '@/api-helpers/internal-error'; // CLUSTOX: proxies the workspace's Jenkins job list. org_id is validated by // Endpoint.serve(), which asserts the caller may act on that workspace. @@ -12,9 +13,13 @@ const getSchema = yup.object().shape({ const endpoint = new Endpoint(getSchema); endpoint.handle.GET(getSchema, async (req, res) => { - const jobs = await internal.get( - `/orgs/${req.payload.org_id}/integrations/jenkins/jobs` - ); + // CLUSTOX: the analytics server distinguishes a base URL it refuses to fetch + // (400, with the address and the reason) from an unreachable Jenkins (502). + // Collapsing both into a generic failure here is what made the setup form + // tell an admin who typed an internal hostname to check his API token. + const jobs = await internal + .get(`/orgs/${req.payload.org_id}/integrations/jenkins/jobs`) + .catch(forwardInternalError); res.send({ jobs: jobs.data }); }); diff --git a/web-server/src/components/ClustoxJenkinsSetup.tsx b/web-server/src/components/ClustoxJenkinsSetup.tsx index 26e68d3d..da00cd95 100644 --- a/web-server/src/components/ClustoxJenkinsSetup.tsx +++ b/web-server/src/components/ClustoxJenkinsSetup.tsx @@ -14,6 +14,7 @@ import { useClustoxUser } from '@/hooks/useClustoxUser'; import { useBoolState, useEasyState } from '@/hooks/useEasyState'; import { fetchCurrentOrg } from '@/slices/auth'; import { useDispatch } from '@/store'; +import { readApiError } from '@/utils/api-error'; import { linkProvider, unlinkProvider } from '@/utils/auth'; import { checkDomainWithRegex } from '@/utils/domainCheck'; import { depFn } from '@/utils/fn'; @@ -81,11 +82,17 @@ export const ClustoxJenkinsSetup: FC<{ onLinked?: () => void }> = ({ if (hasError) return; depFn(isLoading.true); + // CLUSTOX: which of the two calls below failed decides whether the server's + // message is about Jenkins at all. A 400 from the credential save is a + // schema complaint about this form; a 400 from the probe is the analytics + // server refusing the address, and that text is the whole point. + let credentialsSaved = false; try { await linkProvider(apiToken.value, orgId, Integration.JENKINS, { base_url: baseUrl.value, username: username.value }); + credentialsSaved = true; // CLUSTOX: the actual connection probe -- this is the first call that // hits the real Jenkins instance with the credentials just saved. @@ -109,8 +116,20 @@ export const ClustoxJenkinsSetup: FC<{ onLinked?: () => void }> = ({ ); // CLUSTOX: don't leave a workspace "linked" to credentials that don't // work -- the mapping screen would otherwise open to a broken job list. + // True of a refused address too: nothing here can reach that Jenkins. await unlinkProvider(orgId, Integration.JENKINS).catch(() => {}); - depFn(baseUrlError.set, CONNECTION_ERROR); + + // CLUSTOX: a 400 from the probe is the server naming something about this + // URL it will not fetch -- a private, loopback or link-local address, or + // a missing scheme. Reported as CONNECTION_ERROR it sends the admin to + // re-check a username and token that are fine. + const { status, message } = readApiError(e); + depFn( + baseUrlError.set, + credentialsSaved && status === 400 && message + ? message + : CONNECTION_ERROR + ); } finally { depFn(isLoading.false); } From 59db98a4de1367a81dc39673b0155e5aa67f442d Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Tue, 11 Aug 2026 19:02:33 +0500 Subject: [PATCH 088/194] fix(deployments): a repo with no prod_branches must not hide every deployment TeamRepos.prod_branches is nullable, and the BFF flattens it straight into the workflow filter's head_branches, so a null arrives as [None]. The top-level WorkflowFilter did not drop None entries the way RepoWorkflowFilter does four lines above it, and `head_branch ~ NULL` is NULL in Postgres -- never true. Every deployment was filtered out. The dashboard then read "No Deployments" with no error anywhere, which is indistinguishable from a misconfigured integration. Found by querying a repo with 14 synced Jenkins runs two ways: 13 deployments without a filter, 0 through the dashboard. Not Jenkins-specific despite being found there -- it would hide GitHub Actions deployments on the same repo just as completely. --- .../mhq/store/models/code/workflows/filter.py | 15 +++++-- .../test_workflow_filter_null_branches.py | 42 +++++++++++++++++++ 2 files changed, 53 insertions(+), 4 deletions(-) create mode 100644 backend/analytics_server/tests/store/models/code/workflows/test_workflow_filter_null_branches.py diff --git a/backend/analytics_server/mhq/store/models/code/workflows/filter.py b/backend/analytics_server/mhq/store/models/code/workflows/filter.py index baf220ef..d3bca705 100644 --- a/backend/analytics_server/mhq/store/models/code/workflows/filter.py +++ b/backend/analytics_server/mhq/store/models/code/workflows/filter.py @@ -55,10 +55,17 @@ def _head_branches_query(): if not self.head_branches: return None - return or_( - RepoWorkflowRuns.head_branch.op("~")(term) - for term in self.head_branches - ) + # CLUSTOX: drop None entries, matching RepoWorkflowFilter above. + # A repo whose prod_branches is null arrives here as [None], and + # `head_branch ~ NULL` is NULL in Postgres -- never true -- so every + # deployment was filtered out and the dashboard read "No + # Deployments" with no error. Absent branch config must mean "do not + # filter", not "match nothing". + terms = [term for term in self.head_branches if term is not None] + if not terms: + return None + + return or_(RepoWorkflowRuns.head_branch.op("~")(term) for term in terms) def _repo_filters_query(): if not self.repo_filters: diff --git a/backend/analytics_server/tests/store/models/code/workflows/test_workflow_filter_null_branches.py b/backend/analytics_server/tests/store/models/code/workflows/test_workflow_filter_null_branches.py new file mode 100644 index 00000000..b6c94387 --- /dev/null +++ b/backend/analytics_server/tests/store/models/code/workflows/test_workflow_filter_null_branches.py @@ -0,0 +1,42 @@ +""" +A repo with no prod_branches configured must not silently hide every +deployment. + +TeamRepos.prod_branches is nullable. The BFF flattens it into the workflow +filter's head_branches, so a null arrives here as [None]. `head_branch ~ NULL` +is NULL in Postgres -- never true -- so an unguarded filter dropped every row +and the dashboard reported "No Deployments" with no error anywhere. + +Observed live: a repo with 14 synced Jenkins runs returned 13 deployments when +queried without a filter and 0 through the dashboard. +""" + +from mhq.store.models.code.workflows.filter import WorkflowFilter + + +def test_a_single_none_branch_produces_no_condition(): + # The live failure: prod_branches was null, so head_branches was [None]. + assert WorkflowFilter(head_branches=[None]).filter_query == [] + + +def test_none_entries_are_dropped_but_real_ones_survive(): + conditions = WorkflowFilter(head_branches=[None, "^main$"]).filter_query + + assert len(conditions) == 1 + assert "head_branch" in str(conditions[0]) + + +def test_all_none_entries_produce_no_condition(): + assert WorkflowFilter(head_branches=[None, None]).filter_query == [] + + +def test_a_real_branch_still_filters(): + # Regression guard: the common case must be unchanged. + conditions = WorkflowFilter(head_branches=["^main$"]).filter_query + + assert len(conditions) == 1 + assert "head_branch" in str(conditions[0]) + + +def test_empty_list_is_still_ignored(): + assert WorkflowFilter(head_branches=[]).filter_query == [] From b84da5c9187c54b8bd1916fe8f36b0896122cd37 Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Tue, 11 Aug 2026 19:15:58 +0500 Subject: [PATCH 089/194] docs: move the contributor filter plan off this branch It belongs with the contributor filter work, which lives on its own branch and its own PR. It was committed here before that branch existed. --- docs/CONTRIBUTOR_FILTER_PLAN.md | 753 -------------------------------- 1 file changed, 753 deletions(-) delete mode 100644 docs/CONTRIBUTOR_FILTER_PLAN.md diff --git a/docs/CONTRIBUTOR_FILTER_PLAN.md b/docs/CONTRIBUTOR_FILTER_PLAN.md deleted file mode 100644 index 2dc797ed..00000000 --- a/docs/CONTRIBUTOR_FILTER_PLAN.md +++ /dev/null @@ -1,753 +0,0 @@ -# Contributor Filter Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Filter the DORA metrics page by contributor — Lead Time by pull-request author, Deployment Frequency by deploy actor. - -**Architecture:** The dropdown lists distinct git usernames found in the selected team's synced pull requests, bots excluded. Lead Time reuses `PRFilter.authors`, a field upstream declared and parsed but never applied. Deployment Frequency needs a genuinely new `event_actors` filter on `RepoWorkflowRuns`. No new tables, no identity resolution. - -**Tech Stack:** Python 3.9, Flask, SQLAlchemy 2, pytest (backend); Next.js 15 pages router, TypeScript, MUI 5 (frontend). - -## Decisions already made — do not revisit - -| Decision | Choice | -|---|---| -| Identity | Raw git usernames as synced. No mapping table, no merging of handles. A person using two handles appears twice; that is accepted. | -| Metric scope | Lead Time and Deployment Frequency only. **CFR and MTTR stay team-level** — per-person versions wait for the future Jira integration, because GitHub-issue assignees are too weak a foundation. | -| Involvement | **Authored only.** Reviewers are not included. | -| List source | Distinct authors from the selected team's PRs within the active date range, so the dropdown can never offer someone with no data on screen. | -| Bots | Excluded by default, no UI toggle. | - -## Global Constraints - -- **No database migration, no new tables.** -- Backend style: `black` formatted, `flake8` clean. Run from `backend/analytics_server` using `./venv/bin/python`. -- Frontend must be `tsc` clean. Local Node is 20 and the project needs 22 — verify inside the `middleware-dev` container, which has Node 22.16. -- Every Clustox-authored change carries a `# CLUSTOX:` or `// CLUSTOX:` comment explaining **why**, per `docs/FORK_STRATEGY.md`. -- Backend suite baseline is **214 passing**. Do not break any. -- Bot matching is on the exact `[bot]` suffix plus an explicit list — never a substring match on `bot`, which would exclude a real user called `robotnik`. - -## The trap both filter classes share — read before Task 1 or 2 - -`PRFilter.filter_query` and `WorkflowFilter.filter_query` both end with: - -```python -return [ - conditions[x] - for x in self.__dict__.keys() - if getattr(self, x) is not None and conditions[x] is not None -] -``` - -It iterates **every** dataclass field and indexes `conditions` by name. A field with no matching `conditions` key raises `KeyError` the moment that field is set. - -This is not hypothetical. `PRFilter.authors` exists today with no `conditions` entry, so sending `authors` in a filter payload raises `KeyError: 'authors'` — verified: - -``` -PRFilter(authors=['hamad']).filter_query → KeyError: 'authors' -``` - -**Consequence for this plan:** adding a field without adding its `conditions` entry in the same commit reproduces the bug. Both tasks below add both halves together. - ---- - -## File Structure - -**Backend — modify:** -- `mhq/store/models/code/filter.py` — apply `authors` in `PRFilter.filter_query`. -- `mhq/store/models/code/workflows/filter.py` — add `event_actors` to `WorkflowFilter`. -- `mhq/service/workflows/workflow_filter.py` — parse `event_actors` from the request. -- `mhq/store/repos/code.py` — contributor query. -- `mhq/api/pull_requests.py` — contributors route. - -**Backend — create:** -- `tests/store/models/code/test_pr_filter_authors.py` -- `tests/store/models/code/workflows/test_workflow_filter_actors.py` -- `tests/service/code/test_contributors.py` - -**Frontend — create:** -- `web-server/pages/api/internal/team/[team_id]/contributors.ts` -- `web-server/src/components/ContributorFilter.tsx` - -**Frontend — modify:** -- `web-server/src/api-helpers/team.ts` — thread `authors` into `pr_filter`. -- `web-server/pages/api/internal/team/[team_id]/dora_metrics.ts` — thread `event_actors`. -- `web-server/pages/dora-metrics/index.tsx` — render the filter. - ---- - -### Task 1: Apply the author filter (and fix the KeyError) - -**Files:** -- Modify: `backend/analytics_server/mhq/store/models/code/filter.py` -- Test: `backend/analytics_server/tests/store/models/code/test_pr_filter_authors.py` (create) - -**Interfaces:** -- Consumes: nothing. -- Produces: `PRFilter(authors=[...])` yields a `PullRequest.author.in_(authors)` condition. `ParsePRFilterProcessor` already populates the field — no change needed there. - -- [ ] **Step 1: Write the failing tests** - -Create `backend/analytics_server/tests/store/models/code/test_pr_filter_authors.py`: - -```python -from mhq.store.models.code.filter import PRFilter - - -def test_setting_authors_no_longer_raises(): - # Before this change the conditions dict had no "authors" key, so the - # filter_query comprehension raised KeyError as soon as the field was set. - conditions = PRFilter(authors=["hamad-clustox"]).filter_query - assert len(conditions) == 1 - - -def test_authors_produces_an_author_condition(): - conditions = PRFilter(authors=["hamad-clustox"]).filter_query - # Compiling to string is the cheapest way to assert the column and operator - # without a database. - rendered = str(conditions[0]) - assert "author" in rendered - assert "IN" in rendered.upper() - - -def test_no_authors_produces_no_author_condition(): - conditions = PRFilter(base_branches=["^main$"]).filter_query - assert all("author" not in str(c) for c in conditions) - - -def test_empty_author_list_is_ignored(): - # An empty list means "no filter selected", not "match nothing". - conditions = PRFilter(authors=[]).filter_query - assert conditions == [] - - -def test_existing_filters_are_unaffected(): - # Regression guard: the dashboards everyone already uses must not change. - conditions = PRFilter(base_branches=["^main$"], excluded_pr_ids=["a"]).filter_query - assert len(conditions) == 2 -``` - -- [ ] **Step 2: Run tests to verify they fail** - -Run: `cd backend/analytics_server && ./venv/bin/python -m pytest tests/store/models/code/test_pr_filter_authors.py -v` -Expected: FAIL — the first two with `KeyError: 'authors'` - -- [ ] **Step 3: Add the condition** - -In `mhq/store/models/code/filter.py`, inside `filter_query`, add this alongside the other inner functions: - -```python - # CLUSTOX: authors was declared on PRFilter and parsed from the request - # payload upstream, but never added to the conditions dict below -- so - # sending it raised KeyError rather than filtering. This applies it. - def _authors_query(): - if not self.authors: - return None - - return PullRequest.author.in_(self.authors) -``` - -Then add the entry to the `conditions` dict, as the first key: - -```python - conditions = { - "authors": _authors_query(), - "base_branches": _base_branch_query(), - "repo_filters": _repo_filters_query(), - "excluded_pr_ids": _excluded_pr_ids_query(), - "max_cycle_time": _include_prs_below_max_cycle_time(), - "incident_pr_filters": _incident_pr_filters_query(), - } -``` - -- [ ] **Step 4: Run tests to verify they pass** - -Run: `cd backend/analytics_server && ./venv/bin/python -m pytest tests/store/models/code/test_pr_filter_authors.py -v` -Expected: PASS (5 tests) - -- [ ] **Step 5: Run the full suite** - -Run: `cd backend/analytics_server && ./venv/bin/python -m pytest tests -q` -Expected: 219 passed (214 baseline + 5) - -- [ ] **Step 6: Format, lint, commit** - -```bash -cd backend/analytics_server -./venv/bin/python -m black mhq/store/models/code/filter.py tests/store/models/code/test_pr_filter_authors.py -./venv/bin/python -m flake8 mhq/store/models/code/filter.py -cd ../.. -git add backend/analytics_server/mhq/store/models/code/filter.py \ - backend/analytics_server/tests/store/models/code/test_pr_filter_authors.py -git commit -m "fix(filters): apply the pull request author filter" -``` - ---- - -### Task 2: Add the deploy-actor filter - -**Files:** -- Modify: `backend/analytics_server/mhq/store/models/code/workflows/filter.py` -- Modify: `backend/analytics_server/mhq/service/workflows/workflow_filter.py` -- Test: `backend/analytics_server/tests/store/models/code/workflows/test_workflow_filter_actors.py` (create) - -**Interfaces:** -- Consumes: nothing from Task 1. -- Produces: `WorkflowFilter(event_actors=[...])` yields a `RepoWorkflowRuns.event_actor.in_(...)` condition; `ParseWorkflowFilterProcessor` reads `event_actors` from the request dict. - -- [ ] **Step 1: Write the failing tests** - -Create `backend/analytics_server/tests/store/models/code/workflows/test_workflow_filter_actors.py`: - -```python -from mhq.service.workflows.workflow_filter import ParseWorkflowFilterProcessor -from mhq.store.models.code.workflows.filter import WorkflowFilter - - -def test_event_actors_produces_an_actor_condition(): - conditions = WorkflowFilter(event_actors=["hamad-clustox"]).filter_query - assert len(conditions) == 1 - rendered = str(conditions[0]) - assert "event_actor" in rendered - assert "IN" in rendered.upper() - - -def test_no_event_actors_produces_no_actor_condition(): - conditions = WorkflowFilter(head_branches=["^main$"]).filter_query - assert all("event_actor" not in str(c) for c in conditions) - - -def test_empty_actor_list_is_ignored(): - assert WorkflowFilter(event_actors=[]).filter_query == [] - - -def test_existing_workflow_filters_are_unaffected(): - # Regression guard for the deployment frequency everyone already sees. - conditions = WorkflowFilter(head_branches=["^main$"]).filter_query - assert len(conditions) == 1 - - -def test_parser_reads_event_actors_from_the_request(): - parsed = ParseWorkflowFilterProcessor().apply({"event_actors": ["hamad-clustox"]}) - assert parsed.event_actors == ["hamad-clustox"] - - -def test_parser_defaults_event_actors_to_none(): - parsed = ParseWorkflowFilterProcessor().apply({}) - assert parsed.event_actors is None -``` - -- [ ] **Step 2: Run tests to verify they fail** - -Run: `cd backend/analytics_server && ./venv/bin/python -m pytest tests/store/models/code/workflows/test_workflow_filter_actors.py -v` -Expected: FAIL — `TypeError: __init__() got an unexpected keyword argument 'event_actors'` - -- [ ] **Step 3: Add the field and its condition together** - -In `mhq/store/models/code/workflows/filter.py`, change the `WorkflowFilter` dataclass. Add the field **and** the conditions entry in this same step — adding one without the other reproduces the `KeyError` described at the top of this plan: - -```python -@dataclass -class WorkflowFilter: - head_branches: List[str] = None - repo_filters: Dict[str, Dict] = None - # CLUSTOX: who triggered the deployment. Lead Time filters on the pull - # request author; deployments have no author, so the nearest equivalent is - # the actor who ran the build. - event_actors: List[str] = None - - @property - def filter_query(self) -> List: - def _head_branches_query(): - if not self.head_branches: - return None - - return or_( - RepoWorkflowRuns.head_branch.op("~")(term) - for term in self.head_branches - ) - - def _repo_filters_query(): - if not self.repo_filters: - return None - - return or_( - RepoWorkflowFilter(repo_id, repo_filters).filter_query - for repo_id, repo_filters in self.repo_filters.items() - if repo_filters - ) - - # CLUSTOX: runs with a null event_actor -- Jenkins builds without the - # git plugin, for instance -- are excluded by IN, so per-contributor - # deployment counts will not sum to the team total. That is intended: - # an unattributable deploy belongs to nobody. - def _event_actors_query(): - if not self.event_actors: - return None - - return RepoWorkflowRuns.event_actor.in_(self.event_actors) - - conditions = { - "head_branches": _head_branches_query(), - "repo_filters": _repo_filters_query(), - "event_actors": _event_actors_query(), - } - return [ - conditions[x] - for x in self.__dict__.keys() - if getattr(self, x) is not None and conditions[x] is not None - ] -``` - -- [ ] **Step 4: Parse it from the request** - -In `mhq/service/workflows/workflow_filter.py`, extend `ParseWorkflowFilterProcessor.apply`: - -```python - def apply(self, workflow_filter: Dict = None) -> WorkflowFilter: - workflow_filter = workflow_filter or {} - head_branches: List[str] = self._parse_head_branches(workflow_filter) - repo_filters: Dict[str, Dict] = self._parse_repo_filters(workflow_filter) - # CLUSTOX: contributor filter for deployment frequency. - event_actors: List[str] = workflow_filter.get("event_actors") - - return WorkflowFilter( - head_branches=head_branches, - repo_filters=repo_filters, - event_actors=event_actors, - ) -``` - -Note the added `workflow_filter = workflow_filter or {}` guard — the signature defaults to `None` and the existing body would raise `AttributeError` on a bare `.get`. - -- [ ] **Step 5: Run tests to verify they pass** - -Run: `cd backend/analytics_server && ./venv/bin/python -m pytest tests/store/models/code/workflows/test_workflow_filter_actors.py -v` -Expected: PASS (6 tests) - -- [ ] **Step 6: Run the full suite, format, lint, commit** - -```bash -cd backend/analytics_server -./venv/bin/python -m pytest tests -q # expect 225 passed -./venv/bin/python -m black mhq/store/models/code/workflows/filter.py mhq/service/workflows/workflow_filter.py tests/store/models/code/workflows/test_workflow_filter_actors.py -./venv/bin/python -m flake8 mhq/store/models/code/workflows/filter.py mhq/service/workflows/workflow_filter.py -cd ../.. -git add backend/analytics_server/mhq/store/models/code/workflows/filter.py \ - backend/analytics_server/mhq/service/workflows/workflow_filter.py \ - backend/analytics_server/tests/store/models/code/workflows/test_workflow_filter_actors.py -git commit -m "feat(filters): filter deployments by the actor who triggered them" -``` - ---- - -### Task 3: Contributor list query and route - -**Files:** -- Modify: `backend/analytics_server/mhq/store/repos/code.py` -- Modify: `backend/analytics_server/mhq/api/pull_requests.py` -- Test: `backend/analytics_server/tests/service/code/test_contributors.py` (create) - -**Interfaces:** -- Consumes: nothing from Tasks 1-2. -- Produces: - - `is_bot_author(username: str) -> bool` in `mhq/utils/string.py` - - `CodeRepoService.get_contributors_for_repos(repo_ids: List[str], from_time, to_time) -> List[Tuple[str, int]]` returning `(author, pr_count)` ordered by count descending - - `GET /teams//contributors?from_time=&to_time=` returning `[{"username": str, "pr_count": int}]` - -- [ ] **Step 1: Write the failing bot-exclusion tests** - -Create `backend/analytics_server/tests/service/code/test_contributors.py`: - -```python -from mhq.utils.string import is_bot_author - - -def test_bracket_bot_suffix_is_a_bot(): - assert is_bot_author("dependabot[bot]") is True - assert is_bot_author("renovate[bot]") is True - - -def test_known_bot_names_are_bots(): - assert is_bot_author("dependabot") is True - assert is_bot_author("github-actions") is True - - -def test_a_human_whose_name_contains_bot_is_not_a_bot(): - # Substring matching on "bot" would wrongly exclude these people. - assert is_bot_author("robotnik") is False - assert is_bot_author("botond") is False - assert is_bot_author("abbott") is False - - -def test_none_and_empty_are_not_bots(): - assert is_bot_author(None) is False - assert is_bot_author("") is False - - -def test_matching_is_case_insensitive(): - assert is_bot_author("Dependabot[Bot]") is True -``` - -- [ ] **Step 2: Run tests to verify they fail** - -Run: `cd backend/analytics_server && ./venv/bin/python -m pytest tests/service/code/test_contributors.py -v` -Expected: FAIL — `ImportError: cannot import name 'is_bot_author'` - -- [ ] **Step 3: Write the bot predicate** - -Append to `backend/analytics_server/mhq/utils/string.py`: - -```python -# CLUSTOX: bot accounts author real pull requests, so they appear in the -# contributor list unless excluded. Matching is on the exact "[bot]" suffix -# plus an explicit list -- a substring match on "bot" would exclude real -# people called robotnik or abbott. -KNOWN_BOT_AUTHORS = frozenset( - { - "dependabot", - "renovate", - "github-actions", - "snyk-bot", - "greenkeeper", - "codecov", - "imgbot", - } -) - - -def is_bot_author(username: Optional[str]) -> bool: - if not username: - return False - - lowered = username.lower() - return lowered.endswith("[bot]") or lowered in KNOWN_BOT_AUTHORS -``` - -Add `from typing import Optional` to that file's imports if it is not already present. - -- [ ] **Step 4: Run tests to verify they pass** - -Run: `cd backend/analytics_server && ./venv/bin/python -m pytest tests/service/code/test_contributors.py -v` -Expected: PASS (5 tests) - -- [ ] **Step 5: Add the repository query** - -Append to `CodeRepoService` in `backend/analytics_server/mhq/store/repos/code.py`: - -```python - # CLUSTOX: distinct authors for the contributor filter, scoped to the - # repos and window currently on screen so the dropdown can never offer - # someone with no data in view. - def get_contributors_for_repos( - self, repo_ids: List[str], from_time: datetime, to_time: datetime - ) -> List[Tuple[str, int]]: - if not repo_ids: - return [] - - rows = ( - self._db.session.query( - PullRequest.author, func.count(PullRequest.id).label("pr_count") - ) - .filter( - PullRequest.repo_id.in_(repo_ids), - PullRequest.author.isnot(None), - PullRequest.state_changed_at.between(from_time, to_time), - ) - .group_by(PullRequest.author) - .order_by(func.count(PullRequest.id).desc()) - .all() - ) - - return [(author, count) for author, count in rows if not is_bot_author(author)] -``` - -Add whatever of `datetime`, `Tuple`, `func`, and `is_bot_author` that file does not already import. Check its existing imports first rather than adding duplicates. - -- [ ] **Step 6: Add the Flask route** - -Append to `backend/analytics_server/mhq/api/pull_requests.py`. The decorator stack below matches `get_lead_time_prs` in the same file (line 60) — `@queryschema` with a voluptuous `Schema`, coercing the ISO timestamps: - -```python -# CLUSTOX: contributor list backing the DORA metrics filter. Scoped to the -# team's repos and the window on screen, so the dropdown cannot offer someone -# with no data in view. -@app.route("/teams//contributors", methods={"GET"}) -@queryschema( - Schema( - { - Required("from_time"): All(str, Coerce(datetime.fromisoformat)), - Required("to_time"): All(str, Coerce(datetime.fromisoformat)), - } - ), -) -def get_team_contributors(team_id: str, from_time: datetime, to_time: datetime): - query_validator = get_query_validator() - team: Team = query_validator.team_validator(team_id) - - repos = CodeRepoService().get_team_repos(team) - repo_ids = [str(repo.id) for repo in repos] - - contributors = CodeRepoService().get_contributors_for_repos( - repo_ids, from_time, to_time - ) - return [ - {"username": username, "pr_count": pr_count} - for username, pr_count in contributors - ] -``` - -`Schema`, `Required`, `All`, `Coerce`, `queryschema`, `get_query_validator`, `Team` and `CodeRepoService` are all already imported by that file — confirm rather than re-import. - -- [ ] **Step 7: Run the full suite, format, lint, commit** - -```bash -cd backend/analytics_server -./venv/bin/python -m pytest tests -q # expect 230 passed -./venv/bin/python -m black mhq/utils/string.py mhq/store/repos/code.py mhq/api/pull_requests.py tests/service/code/test_contributors.py -./venv/bin/python -m flake8 mhq/utils/string.py mhq/store/repos/code.py mhq/api/pull_requests.py -cd ../.. -git add backend/analytics_server/mhq/utils/string.py \ - backend/analytics_server/mhq/store/repos/code.py \ - backend/analytics_server/mhq/api/pull_requests.py \ - backend/analytics_server/tests/service/code/test_contributors.py -git commit -m "feat(contributors): list a team's pull request authors, bots excluded" -``` - ---- - -### Task 4: Thread the filter through the BFF - -**Files:** -- Create: `web-server/pages/api/internal/team/[team_id]/contributors.ts` -- Modify: `web-server/src/api-helpers/team.ts` -- Modify: `web-server/pages/api/internal/team/[team_id]/dora_metrics.ts` - -**Interfaces:** -- Consumes: the Flask route from Task 3. -- Produces: - - `GET /api/internal/team/{team_id}/contributors?from_date=&to_date=` → `[{username, pr_count}]` - - `updatePrFilterParams(teamId, params, filters)` accepts `filters.authors?: string[]` and emits it inside `pr_filter` - - `dora_metrics.ts` forwards `event_actors` into the workflow filter payload - -- [ ] **Step 1: Create the contributors BFF route** - -Create `web-server/pages/api/internal/team/[team_id]/contributors.ts`: - -```ts -import * as yup from 'yup'; - -import { Endpoint } from '@/api-helpers/global'; -import { internal } from '@/api-helpers/axios'; - -// CLUSTOX: backs the contributor filter on the DORA metrics page. team_id is -// validated by Endpoint.serve(), which asserts the caller may read that team. -const pathSchema = yup.object().shape({ - team_id: yup.string().uuid().required() -}); - -const getSchema = yup.object().shape({ - team_id: yup.string().uuid().required(), - from_date: yup.date().required(), - to_date: yup.date().required() -}); - -const endpoint = new Endpoint(pathSchema); - -endpoint.handle.GET(getSchema, async (req, res) => { - const { team_id, from_date, to_date } = req.payload; - - const contributors = await internal.get(`/teams/${team_id}/contributors`, { - params: { from_time: from_date, to_time: to_date } - }); - - res.send(contributors.data); -}); - -export default endpoint.serve(); -``` - -Note `pathSchema` contains only `team_id` — the same construction bug fixed during the Jenkins work, where passing a method-specific schema as the path schema made every other method a permanent 400. - -- [ ] **Step 2: Accept authors in the PR filter builder** - -In `web-server/src/api-helpers/team.ts`, change `updatePrFilterParams`: - -```ts -export const updatePrFilterParams = async ( - _teamId: ID, - params: T, - filters?: Partial<{ - branches: string; - repo_filters: RepoFilterConfig; - // CLUSTOX: contributor filter -- git usernames, as synced. - authors: string[]; - }> -) => { - const updatedParams = { - base_branches: filters?.branches?.split(','), - repo_filters: filters?.repo_filters, - authors: filters?.authors?.length ? filters.authors : undefined - }; - const reducedParams = reject(isNil, updatedParams); - const pr_filter = equals({}, reducedParams) ? null : reducedParams; - - return reject(isNil, { - ...params, - pr_filter - }) as T & { pr_filter?: Partial }; -}; -``` - -The `?.length ? ... : undefined` matters: an empty array must become `undefined` so `reject(isNil, ...)` drops it. Sending `authors: []` would filter to nobody. - -- [ ] **Step 3: Accept event actors in the workflow filter builder** - -In `web-server/src/utils/filterUtils.ts`, extend `getWorkFlowFiltersAsPayloadForSingleTeam` (line 84). The new parameter is optional, so the other caller — `pages/api/internal/team/[team_id]/deployment_analytics.ts` — is unaffected: - -```ts -export const getWorkFlowFiltersAsPayloadForSingleTeam = async (params: { - orgId: ID; - teamId: ID; - // CLUSTOX: contributor filter. Deployments have no author, so the nearest - // equivalent is the actor who triggered the run. - eventActors?: string[]; -}) => { - const { orgId, teamId, eventActors } = params; - const teamProdBranchesMap = - await getAllTeamsReposProdBranchesForOrgAsMap(orgId); - const filter = Object.fromEntries( - Object.entries(workFlowFiltersFromTeamProdBranches(teamProdBranchesMap)) - )[teamId]; - - if (!eventActors?.length) return filter; - return { ...filter, event_actors: eventActors }; -}; -``` - -Add a case to the existing suite at `web-server/src/utils/__tests__/filterUtils.test.ts` (its `getWorkFlowFiltersAsPayloadForSingleTeam` describe block starts at line 264) asserting that passing `eventActors` adds `event_actors` to the payload, and that omitting it leaves the payload unchanged. - -- [ ] **Step 4: Thread the selection through the DORA route** - -In `web-server/pages/api/internal/team/[team_id]/dora_metrics.ts`, add `authors: yup.array().of(yup.string()).optional()` to the request schema, destructure it alongside the other params, then change the `Promise.all` block at line 66: - -```ts - const [prFilters, workflowFilters] = await Promise.all([ - updatePrFilterParams(teamId, {}, { ...branchAndRepoFilters, authors }).then( - ({ pr_filter }) => ({ - pr_filter - }) - ), - getWorkFlowFiltersAsPayloadForSingleTeam({ - orgId: org_id, - teamId: teamId, - eventActors: authors - }) - ]); -``` - -The same `authors` value feeds both builders — that is what makes one control drive both metrics. - -- [ ] **Step 5: Run the frontend unit tests** - -Run: `docker exec middleware-dev sh -c 'cd /app/web-server && yarn test --testPathPattern=filterUtils'` -Expected: PASS, including the two cases added in Step 3 - -- [ ] **Step 6: Verify types compile** - -```bash -docker cp web-server/pages/api/internal/team/\[team_id\]/contributors.ts middleware-dev:/app/web-server/pages/api/internal/team/\[team_id\]/contributors.ts -docker cp web-server/src/api-helpers/team.ts middleware-dev:/app/web-server/src/api-helpers/team.ts -docker cp web-server/src/utils/filterUtils.ts middleware-dev:/app/web-server/src/utils/filterUtils.ts -docker cp web-server/pages/api/internal/team/\[team_id\]/dora_metrics.ts middleware-dev:/app/web-server/pages/api/internal/team/\[team_id\]/dora_metrics.ts -docker exec middleware-dev sh -c 'cd /app/web-server && ./node_modules/.bin/tsc --noEmit 2>&1 | grep -c "error TS"' -``` -Expected: `0` - -The container now runs this branch, so these files overwrite their own current versions — no restore step is needed, but re-copy from git if you abandon a change. - -- [ ] **Step 7: Commit** - -```bash -git add web-server/pages/api/internal/team/\[team_id\]/contributors.ts \ - web-server/src/api-helpers/team.ts \ - web-server/src/utils/filterUtils.ts \ - web-server/src/utils/__tests__/filterUtils.test.ts \ - web-server/pages/api/internal/team/\[team_id\]/dora_metrics.ts -git commit -m "feat(contributors): expose the contributor list and thread the filter" -``` - ---- - -### Task 5: Contributor dropdown - -**Files:** -- Create: `web-server/src/components/ContributorFilter.tsx` -- Modify: `web-server/pages/dora-metrics/index.tsx` - -**Interfaces:** -- Consumes: `GET /api/internal/team/{team_id}/contributors` from Task 4. -- Produces: nothing consumed by later tasks. - -- [ ] **Step 1: Build the dropdown** - -Create `web-server/src/components/ContributorFilter.tsx`: an MUI `Autocomplete` (multiple, `size="small"`) fetching from `/api/internal/team/${teamId}/contributors` whenever `teamId`, `fromDate` or `toDate` change. Each option shows the username with its PR count as secondary text. Follow the styling conventions in `web-server/src/components/ClustoxWorkspaceMetrics.tsx`. - -Required behaviours: - -- Selection is held in the URL query string, so the filter survives a refresh and can be shared. -- When the fetched list no longer contains a selected username — because the team or date range changed — drop that selection and show a brief inline notice. Do not silently clear it; the numbers would move with no visible cause. -- While loading, disable the control rather than showing an empty list, which reads as "no contributors". -- On fetch failure, disable the control and show the error inline. The dashboard must still render unfiltered — the filter is an enhancement, and losing it should not cost the page. -- When the list is empty, disable the control with the text "No contributors in this range". - -- [ ] **Step 2: Render it and label the metrics** - -In `web-server/pages/dora-metrics/index.tsx`, render `ContributorFilter` alongside the existing selectors and pass the selection into the metrics requests. - -When a contributor is selected, each affected card gains a qualifier: - -- Lead Time: `authored by ` -- Deployment Frequency: `deployed by ` - -These wordings are deliberate and load-bearing. One control drives two different relationships — the PR author for Lead Time, the deploy actor for Deployment Frequency — and those are often different people. Without the qualifiers, a user seeing the two numbers disagree will reasonably conclude the page is broken. - -Change Failure Rate and MTTR are **not** filtered. When a contributor is selected, show on both: `team-wide — per-contributor arrives with Jira`. - -- [ ] **Step 3: Verify types and lint** - -```bash -docker cp web-server/src/components/ContributorFilter.tsx middleware-dev:/app/web-server/src/components/ContributorFilter.tsx -docker cp web-server/pages/dora-metrics/index.tsx middleware-dev:/app/web-server/pages/dora-metrics/index.tsx -docker exec middleware-dev sh -c 'cd /app/web-server && ./node_modules/.bin/tsc --noEmit 2>&1 | grep -c "error TS"' -``` -Expected: `0` - -- [ ] **Step 4: Verify by hand** - -The container runs this branch, so the running app at `http://localhost:3333/dora-metrics` picks these up. Confirm: - -1. The dropdown lists contributors with PR counts, no `[bot]` entries. -2. Selecting one changes Lead Time and Deployment Frequency; CFR and MTTR do not move. -3. Both filtered cards show their qualifier. -4. Refreshing preserves the selection. -5. Narrowing the date range until the selected contributor has no PRs clears the selection with a notice. - -- [ ] **Step 5: Commit** - -```bash -git add web-server/src/components/ContributorFilter.tsx \ - web-server/pages/dora-metrics/index.tsx -git commit -m "feat(contributors): add the contributor filter to dora metrics" -``` - ---- - -## Final verification - -- [ ] `cd backend/analytics_server && ./venv/bin/python -m pytest tests -q` — expect 230 passed -- [ ] `./venv/bin/python -m black --check mhq && ./venv/bin/python -m flake8 mhq` — clean -- [ ] `docker exec middleware-dev sh -c 'cd /app/web-server && ./node_modules/.bin/tsc --noEmit'` — 0 errors -- [ ] Unfiltered dashboards are byte-identical to before this work — the regression tests in Tasks 1 and 2 cover the query layer, but confirm in the running app too - -**The test that matters most** is the one asserting the numbers *change*. The defect being fixed here is a filter that was accepted and never applied; a test that only checks the request succeeds would have passed against the broken behaviour. From 0301f8901a2707edaa1f0a5bbd7a2e81ffef6df5 Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Tue, 11 Aug 2026 19:55:55 +0500 Subject: [PATCH 090/194] ci: warm the app's routes after deploy The container serves the app through `next dev`, which compiles each route the first time it is requested -- inside the user's request, not at boot. Measured on a cold container: /login 7.5s, /workspaces 5.7s, /users 21.8s, against 0.02s once warm. Requesting the pages here moves that cost into the deploy. The warm-up has to be authenticated, which is the non-obvious part. middleware.ts redirects unauthenticated page requests to /login before Next renders anything, so an anonymous request returns 307 in ~37ms and compiles nothing. Verified directly: after an anonymous request to /system-logs there is still no .next/server/pages/system-logs.js. A loop without a session warms /login and reports success for eight routes it never touched. Needs a Jenkins credential 'middleware-warmup-account' holding an account that can load the dashboard. Without it the stage says so loudly rather than passing quietly, because a warm-up that silently does nothing is worse than no warm-up. This is a workaround. The compile cache is in memory, so it dies on restart and can be evicted under pressure -- observed once during testing, a warmed route went back to 6.6s on its own. The real fix is building the production image. --- Jenkinsfile | 65 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 65 insertions(+) diff --git a/Jenkinsfile b/Jenkinsfile index 20f0e5ef..fdc34d75 100644 --- a/Jenkinsfile +++ b/Jenkinsfile @@ -36,6 +36,71 @@ pipeline { ''' } } + + // CLUSTOX: the container serves the app through `next dev`, which + // compiles each route the first time it is requested -- inside the + // user's request, not at boot. Measured on a cold container: /login + // 7.5s, /workspaces 5.7s, /users 21.8s, against 0.02s once warm. + // + // Warming has to be authenticated. middleware.ts redirects + // unauthenticated page requests to /login BEFORE Next renders, so an + // anonymous request returns 307 in ~37ms and compiles nothing -- + // verified by checking .next/server/pages afterwards. A loop without a + // session warms /login and nothing else. + // + // This is a workaround, not a fix. The compile cache lives in memory, + // so it is lost on restart and can be evicted under memory pressure. + // The real fix is building the production image, which runs + // `yarn build` and serves a compiled bundle -- tracked separately. + stage('Warm Routes') { + steps { + withCredentials([usernamePassword( + credentialsId: 'middleware-warmup-account', + usernameVariable: 'WARM_EMAIL', + passwordVariable: 'WARM_PASSWORD' + )]) { + sh ''' + BASE="http://127.0.0.1:${PORT:-3333}" + JAR=$(mktemp) + trap 'rm -f "$JAR"' EXIT + + CSRF=$(curl -s -c "$JAR" "${BASE}/api/auth/csrf" \ + | sed -n 's/.*"csrfToken":"\\([^"]*\\)".*/\\1/p') + + if [ -z "$CSRF" ]; then + echo "Could not obtain a CSRF token; skipping warm-up." + exit 0 + fi + + curl -s -b "$JAR" -c "$JAR" -o /dev/null \ + -X POST "${BASE}/api/auth/callback/credentials" \ + --data-urlencode "csrfToken=${CSRF}" \ + --data-urlencode "email=${WARM_EMAIL}" \ + --data-urlencode "password=${WARM_PASSWORD}" \ + --data-urlencode "json=true" + + # A 200 here means the session took and the page really + # compiled. A 307 means we are still anonymous, so the + # warm-up is not doing anything and should be noisy + # about it rather than reporting false success. + WARMED=0 + for ROUTE in /dora-metrics /teams /integrations \ + /settings /users /workspaces /collaborate; do + CODE=$(curl -s -b "$JAR" -o /dev/null \ + -w '%{http_code}' --max-time 180 "${BASE}${ROUTE}") + echo "warmed ${ROUTE} -> ${CODE}" + [ "$CODE" = "200" ] && WARMED=$((WARMED + 1)) + done + + if [ "$WARMED" -eq 0 ]; then + echo "WARNING: no route compiled -- check the warm-up credentials." + else + echo "Route warm-up complete (${WARMED} compiled)." + fi + ''' + } + } + } } post { From b285251cde2649be7ee0259b3339990f83d079c3 Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Tue, 11 Aug 2026 19:58:29 +0500 Subject: [PATCH 091/194] ci: deploy the production image instead of the dev server docker-compose.yml is upstream's local development file. It builds Dockerfile.dev, which never runs `yarn build`, so the container serves the app through `next dev` and compiles each route inside the first request that asks for it. Measured on a cold container: /login 7.5s, /workspaces 5.7s, /users 21.8s, against 0.02s once warm. The server has been running that way since the pipeline was added. docker-compose.prod.yml builds the production Dockerfile, which runs `yarn build` and serves a compiled bundle. It keeps the volume names byte for byte so the existing database is reused rather than replaced, and pins ENVIRONMENT under `environment:` because the server's .env still says dev and env_file would otherwise win -- the prod image would build correctly and then start `next dev` anyway. Build and deploy are now separate stages. `docker compose build && docker compose up -d` in one step is a trap: when the build fails, `up -d` recreates the container from the previous image, so a broken release presents as a healthy app on stale code and the pipeline reports success. That happened during development when a dependency pin was wrong for the image's Python version, and it would be invisible to anyone without shell access to the box. Verify Environment fails before anything is torn down, since .env is gitignored and cannot carry a newly required variable. The health check curled / after sleeping 10s. A signed-out request there answers 307 and curl counts a redirect as success, so it passed against an app whose API was entirely broken; 10s was also short for a cold start that runs migrations. It now polls for readiness and asserts /api/clustox/me returns 401, which is only reachable when Next.js, the database and auth are all working. Replaces the route warm-up added a moment ago. Warming was a workaround for the dev server -- it needed a credential, and the compile cache it built lived in memory, so it died on restart and could be evicted under pressure. --- Jenkinsfile | 209 +++++++++++++++++++++++++++------------- docker-compose.prod.yml | 76 +++++++++++++++ 2 files changed, 218 insertions(+), 67 deletions(-) create mode 100644 docker-compose.prod.yml diff --git a/Jenkinsfile b/Jenkinsfile index fdc34d75..d4ad9efa 100644 --- a/Jenkinsfile +++ b/Jenkinsfile @@ -1,6 +1,13 @@ pipeline { agent any + environment { + APP_DIR = '/opt/app/middleware' + // CLUSTOX: the production compose file, not the default one. See the + // Build & Deploy stage for why that distinction is the whole point. + COMPOSE_FILE = 'docker-compose.prod.yml' + } + stages { stage('Checkout') { steps { @@ -11,94 +18,162 @@ pipeline { stage('Update Deployment Copy') { steps { sh ''' - cd /opt/app/middleware + cd ${APP_DIR} git fetch origin main git reset --hard origin/main ''' } } - stage('Build & Deploy') { + // CLUSTOX: .env is gitignored, so a release that needs a new variable + // cannot deliver it. Without this stage the app builds, starts, and + // then fails at runtime in ways the health check cannot see -- an unset + // INTERNAL_API_TOKEN makes the Flask servers reject every request, and + // an unset SUPERADMIN_PASSWORD means no account is ever created. + // + // Failing here, before anything is torn down, leaves the running + // deployment alive through a misconfigured release. + stage('Verify Environment') { + steps { + sh ''' + cd ${APP_DIR} + + if [ ! -f .env ]; then + echo "FATAL: ${APP_DIR}/.env does not exist." + exit 1 + fi + + MISSING="" + for VAR in NEXTAUTH_URL NEXTAUTH_SECRET INTERNAL_API_TOKEN \ + SUPERADMIN_EMAIL SUPERADMIN_PASSWORD; do + # Present AND non-empty. `grep -q "^VAR="` alone would + # pass on a variable set to nothing. + if ! grep -qE "^${VAR}=.+" .env; then + MISSING="${MISSING} ${VAR}" + fi + done + + if [ -n "${MISSING}" ]; then + echo "FATAL: required variables missing or empty in ${APP_DIR}/.env:" + for VAR in ${MISSING}; do echo " - ${VAR}"; done + echo "" + echo "See env.example. Nothing has been changed." + exit 1 + fi + + echo "All required environment variables are present." + ''' + } + } + + stage('Build') { steps { + // CLUSTOX: builds docker-compose.prod.yml, which uses the + // production Dockerfile and runs `yarn build`. The default + // docker-compose.yml is upstream's *local development* file: it + // builds Dockerfile.dev, which never builds the frontend, so the + // container serves the app through `next dev` and compiles every + // route inside the first request that asks for it. Measured on a + // cold container: /login 7.5s, /workspaces 5.7s, /users 21.8s, + // against 0.02s once warm. + // + // The build is deliberately its own stage. `docker compose build + // && docker compose up -d` in one step is a trap: when the build + // fails, `up -d` happily recreates the container from the + // PREVIOUS image, so a broken release presents as a healthy app + // serving stale code and the pipeline reports success. That is + // not hypothetical -- it happened during development when a + // dependency pin was wrong for the image's Python version. sh ''' - cd /opt/app/middleware - docker compose build - docker compose up -d + cd ${APP_DIR} + docker compose -f ${COMPOSE_FILE} build + ''' + } + } + + stage('Deploy') { + steps { + sh ''' + cd ${APP_DIR} + docker compose -f ${COMPOSE_FILE} up -d ''' } } stage('Health Check') { steps { + // CLUSTOX: the previous check was `sleep 10` then a curl of `/`. + // Both halves were wrong. `/` answers 307 to /login for a + // signed-out request and curl treats a redirect as success, so + // it passed against an app whose API was entirely broken. And + // 10s is short for a cold start that also runs migrations, so it + // produced false failures too. + // + // Instead: poll until the app answers, then assert the API + // returns 401. Reaching 401 proves Next.js is serving, the + // database is reachable, and auth is wired up. A misconfigured + // instance returns 500 or refuses the connection. sh ''' - sleep 10 - curl -sf http://127.0.0.1:${PORT:-3333} || (echo "Health check failed" && exit 1) + PORT_NUM=${PORT:-3333} + BASE="http://127.0.0.1:${PORT_NUM}" + + echo "Waiting for the app to accept connections..." + READY=0 + for i in $(seq 1 60); do + if curl -sf -o /dev/null "${BASE}/login"; then + READY=1 + echo "App responded after $((i * 5))s." + break + fi + sleep 5 + done + + if [ "${READY}" != "1" ]; then + echo "FATAL: app did not respond within 300s." + cd ${APP_DIR} && docker compose -f ${COMPOSE_FILE} logs --tail=80 + exit 1 + fi + + echo "Checking that the API is up and enforcing auth..." + CODE=$(curl -s -o /dev/null -w '%{http_code}' "${BASE}/api/clustox/me") + if [ "${CODE}" != "401" ]; then + echo "FATAL: /api/clustox/me returned ${CODE}, expected 401." + echo "The app is serving pages but the API is not healthy." + cd ${APP_DIR} && docker compose -f ${COMPOSE_FILE} logs --tail=80 + exit 1 + fi + + echo "Health check passed." ''' } } - // CLUSTOX: the container serves the app through `next dev`, which - // compiles each route the first time it is requested -- inside the - // user's request, not at boot. Measured on a cold container: /login - // 7.5s, /workspaces 5.7s, /users 21.8s, against 0.02s once warm. - // - // Warming has to be authenticated. middleware.ts redirects - // unauthenticated page requests to /login BEFORE Next renders, so an - // anonymous request returns 307 in ~37ms and compiles nothing -- - // verified by checking .next/server/pages afterwards. A loop without a - // session warms /login and nothing else. + // CLUSTOX: proves the deploy is actually serving a compiled bundle + // rather than silently falling back to the dev server. A production + // build answers a cold page request in well under a second; `next dev` + // takes seconds to tens of seconds because it compiles on demand. // - // This is a workaround, not a fix. The compile cache lives in memory, - // so it is lost on restart and can be evicted under memory pressure. - // The real fix is building the production image, which runs - // `yarn build` and serves a compiled bundle -- tracked separately. - stage('Warm Routes') { + // This is a warning rather than a failure: a slow disk or a loaded host + // could trip it without the deploy being wrong, and failing a healthy + // release over a timing measurement would be worse than the problem. + stage('Verify Production Build') { steps { - withCredentials([usernamePassword( - credentialsId: 'middleware-warmup-account', - usernameVariable: 'WARM_EMAIL', - passwordVariable: 'WARM_PASSWORD' - )]) { - sh ''' - BASE="http://127.0.0.1:${PORT:-3333}" - JAR=$(mktemp) - trap 'rm -f "$JAR"' EXIT - - CSRF=$(curl -s -c "$JAR" "${BASE}/api/auth/csrf" \ - | sed -n 's/.*"csrfToken":"\\([^"]*\\)".*/\\1/p') - - if [ -z "$CSRF" ]; then - echo "Could not obtain a CSRF token; skipping warm-up." - exit 0 - fi + sh ''' + BASE="http://127.0.0.1:${PORT:-3333}" - curl -s -b "$JAR" -c "$JAR" -o /dev/null \ - -X POST "${BASE}/api/auth/callback/credentials" \ - --data-urlencode "csrfToken=${CSRF}" \ - --data-urlencode "email=${WARM_EMAIL}" \ - --data-urlencode "password=${WARM_PASSWORD}" \ - --data-urlencode "json=true" - - # A 200 here means the session took and the page really - # compiled. A 307 means we are still anonymous, so the - # warm-up is not doing anything and should be noisy - # about it rather than reporting false success. - WARMED=0 - for ROUTE in /dora-metrics /teams /integrations \ - /settings /users /workspaces /collaborate; do - CODE=$(curl -s -b "$JAR" -o /dev/null \ - -w '%{http_code}' --max-time 180 "${BASE}${ROUTE}") - echo "warmed ${ROUTE} -> ${CODE}" - [ "$CODE" = "200" ] && WARMED=$((WARMED + 1)) - done - - if [ "$WARMED" -eq 0 ]; then - echo "WARNING: no route compiled -- check the warm-up credentials." - else - echo "Route warm-up complete (${WARMED} compiled)." - fi - ''' - } + ELAPSED=$(curl -s -o /dev/null -w '%{time_total}' "${BASE}/login") + echo "Cold /login response: ${ELAPSED}s" + + # Integer compare; `sh` has no float arithmetic. + WHOLE=$(echo "${ELAPSED}" | cut -d. -f1) + if [ "${WHOLE}" -ge 3 ]; then + echo "WARNING: ${ELAPSED}s is slow for a compiled build." + echo "The container may be running 'next dev' -- check that" + echo "ENVIRONMENT=prod reached it and that ${COMPOSE_FILE} was used." + else + echo "Response time is consistent with a production build." + fi + ''' } } } @@ -125,4 +200,4 @@ pipeline { } } } -} \ No newline at end of file +} diff --git a/docker-compose.prod.yml b/docker-compose.prod.yml new file mode 100644 index 00000000..41491803 --- /dev/null +++ b/docker-compose.prod.yml @@ -0,0 +1,76 @@ +# CLUSTOX: production deployment. +# +# docker-compose.yml is upstream's *local development* file: it builds +# Dockerfile.dev, which never runs `yarn build`, so the container serves the +# app with `next dev` and compiles every route on first request. That is +# correct for a laptop with hot reload and wrong for a server, where it costs +# 7-22s on each page's first visit and holds the webpack compiler in memory. +# +# This file builds the production Dockerfile instead, which runs `yarn build` +# and serves a compiled bundle. +# +# Deploy with: +# docker compose -f docker-compose.prod.yml up -d --build +# +# Two things here are deliberate and load-bearing: +# +# 1. The volume names match docker-compose.yml exactly. Compose namespaces +# volumes by project (directory) name, so `dev_postgres_data` resolves to +# the same `middleware_dev_postgres_data` that the running deployment +# already uses. Renaming it would silently start against an empty database +# and lose every synced metric and integration. +# +# 2. ENVIRONMENT is pinned under `environment:` rather than left to `.env`. +# `environment:` takes precedence over `env_file:`, and the server's .env +# still says ENVIRONMENT=dev. Without this pin the prod image would build +# correctly and then start `next dev` anyway, which is the exact bug this +# file exists to fix. + +volumes: + dev_postgres_data: + driver: local + dev_keys: + driver: local + +services: + # Service and container names match docker-compose.yml so that `up -d` + # replaces the existing container rather than starting a second one that + # fights it for port 3333. + middleware-dev: + container_name: middleware-dev + build: + context: ./ + dockerfile: Dockerfile + args: + ENVIRONMENT: prod + POSTGRES_DB_ENABLED: ${POSTGRES_DB_ENABLED:-true} + DB_INIT_ENABLED: ${DB_INIT_ENABLED:-true} + REDIS_ENABLED: ${REDIS_ENABLED:-true} + BACKEND_ENABLED: ${BACKEND_ENABLED:-true} + FRONTEND_ENABLED: ${FRONTEND_ENABLED:-true} + CRON_ENABLED: ${CRON_ENABLED:-true} + DB_PORT: ${DB_PORT:-5434} + + env_file: + - .env + + environment: + # See note 2 above -- these must win over .env. + ENVIRONMENT: prod + NEXT_PUBLIC_APP_ENVIRONMENT: prod + + restart: unless-stopped + + ports: + - "127.0.0.1:${ANALYTICS_SERVER_PORT:-9696}:${ANALYTICS_SERVER_PORT:-9696}" + - "127.0.0.1:${SYNC_SERVER_PORT:-9697}:${SYNC_SERVER_PORT:-9697}" + - "127.0.0.1:${PORT:-3333}:${PORT:-3333}" + - "127.0.0.1:${DB_PORT:-5434}:${DB_PORT:-5434}" + - "127.0.0.1:${REDIS_PORT:-6385}:${REDIS_PORT:-6385}" + + extra_hosts: + - "host.docker.internal:host-gateway" + + volumes: + - dev_postgres_data:/var/lib/postgresql/15/main + - dev_keys:/app/backend/analytics_server/mhq/config From b36c563c3d39dde72dd012680614ff3938bb1aa7 Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Tue, 11 Aug 2026 21:23:03 +0500 Subject: [PATCH 092/194] docs: design for per-team DORA benchmarks Each team sets its own target for the four metrics; a team without one inherits a baseline the superadmin sets. Each card shows the target as a line on the trend it already draws, with a caption giving the gap and which benchmark applied. Six decisions are recorded with their consequences. Two are worth surfacing here. Resolution happens in the backend and the response reports which benchmark applied, so an admin who thinks they set a target can see that they did not, and no consumer reimplements the fallback. And the feature is additive: the industry Elite/High/Medium/Low pills stay untouched as an external reference, because a team can be High by industry standards and still missing its own goal. No migration. Settings.entity_type is character varying and the schema declares no native enum types, so the new EntityType.GLOBAL is a code change. data is JSONB, so the lines-of-code target arrives later without one either. Two traps are called out because they would be silent. Absent is not zero: a form sending 0 for empty fields would turn inherit into target-zero on every save and show every team failing everything. And direction is per metric, so under target is good on three cards and bad on deployment frequency. Lines of code is deliberately a separate spec. Noting for whoever takes it: the data is already synced -- all 429 pull requests carry meta.code_stats and PullRequest already exposes additions, deletions and changed_files. It is aggregation and a card, not an integration. --- docs/BENCHMARKS.md | 253 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 253 insertions(+) create mode 100644 docs/BENCHMARKS.md diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md new file mode 100644 index 00000000..7e2d019b --- /dev/null +++ b/docs/BENCHMARKS.md @@ -0,0 +1,253 @@ +# Per-team DORA benchmarks — design + +**Status:** approved, not implemented +**Date:** 2026-08-11 +**Estimate:** 3–4 days + +Each team can set its own target for the four DORA metrics. A team without a +target inherits a baseline the superadmin sets. Each metric card shows its +target as a line on the trend it already draws, with a caption saying how far +off it is and which benchmark applied. + +--- + +## Decisions + +Settled before design. Changing one of these changes the design, not the +implementation. + +| Decision | Choice | Consequence | +|---|---|---| +| Scope unit | **Team** | Teams are already what the dashboard filters by, so a benchmark per team matches what is on screen. `Settings` already stores per-team rows, so no new pattern. | +| Benchmark shape | **One target value per metric** | Four numbers, not twelve band thresholds. Simple to enter, simple to explain, and the graph becomes obvious: actual against a target line. | +| Relationship to the industry pills | **Additive** | Elite/High/Medium/Low stay exactly as they are, as an external reference. The benchmark is your own goal. A team can be "High" by industry standards and still missing its own target — usually the more useful conversation. | +| Graph | **Trend with a target line** | The Lead Time and Deployment Frequency cards already render a trend behind the number. Adding a line makes an existing decorative graphic informative. | +| Storage | **Reuse `Settings`** | No migration. `data` is JSONB, so the LOC target arrives later for free. | +| Resolution | **Backend** | The API returns the resolved target and its source. The browser draws a line; it never implements inheritance. | + +## Scope + +This spec covers benchmarks and the graphs. **Lines of code is a separate +feature** with its own spec — it is a new metric, not part of this one, and +depends on nothing here. Worth noting for whoever picks it up: the data is +already synced. All 429 pull requests in the current database have +`meta.code_stats` populated, and `PullRequest` already exposes `additions`, +`deletions` and `changed_files` as properties. LOC is aggregation and a card, +not an integration. + +--- + +## Data model + +Four numbers stored as JSONB under a new `SettingType.BENCHMARK_SETTING`: + +```json +{ + "lead_time": 86400, + "deployment_frequency": 5, + "change_failure_rate": 15, + "mean_time_to_recovery": 3600 +} +``` + +Units match what the metrics API already returns — **seconds** for lead time +and MTTR, **deployments per week** for frequency, **percent** for CFR. No +conversion layer: the API compares like with like and the UI formats targets +exactly as it already formats actual values. + +Every field is optional. A team can set a lead-time target and inherit the +rest — **fallback is per metric, not all-or-nothing**. Teams have opinions +about one or two metrics, rarely all four. + +### Where rows live + +| Scope | Row | Who sets it | +|---|---|---| +| Team | `(team_id, TEAM, BENCHMARK_SETTING)` | Workspace admin, own teams only | +| Global baseline | `(sentinel-uuid, GLOBAL, BENCHMARK_SETTING)` | Superadmin only | + +`EntityType.GLOBAL` is new. `Settings.entity_type` is `character varying` and +the schema declares no native enum types, so this is a code change — +**no migration anywhere in this feature**. + +`entity_id` is `NOT NULL`, so the global row uses a fixed sentinel UUID. That +is a documented wart, not a clever trick. + +### Resolution + +Per metric, in the backend: + +``` +team value set? → use it, source: "team" +otherwise → global baseline, source: "global" +neither → no target, source: null +``` + +The metrics response carries the result: + +```json +"benchmarks": { + "lead_time": { "target": 86400, "source": "team" }, + "deployment_frequency": { "target": 5, "source": "global" }, + "change_failure_rate": { "target": null, "source": null } +} +``` + +Three reasons this is server-side. Every consumer would otherwise reimplement +the fallback and eventually disagree. The API can report *which* benchmark +applied, so an admin who thinks they set one can see that they did not. And +when LOC arrives it is a fifth key here, not a fifth reimplementation. + +--- + +## API surface + +**Team benchmark** reuses the existing settings routes. No new Flask code: + +``` +GET /teams//settings?setting_type=BENCHMARK_SETTING +PUT /teams//settings +``` + +Workspace scoping comes free: `Endpoint.serve()` already asserts team access +on any payload carrying a `team_id`. + +**Global baseline** is the one new route, because settings routes today are +team- and org-scoped and this is neither: + +``` +GET /settings/global?setting_type=BENCHMARK_SETTING +PUT /settings/global +``` + +Superadmin only, enforced with `assertRole(session, 'SUPERADMIN')` at the BFF. +The Flask layer has no notion of who is calling — it trusts the internal +token — so the check cannot live there. + +**Resolved benchmarks ride on the existing metrics response** rather than +getting their own endpoint. The dashboard already makes one call for all four +metrics; a second call for targets would let the numbers and the lines drawn +against them disagree, and would add a round trip to a page that is already +the slowest in the product. + +### Already exists — not being built + +`SettingsService.get_settings`, `save_settings`, `get_settings_map`; `GET`/`PUT` +on `/teams//settings`; and `get_default_setting_data`, which is where +the shipped global baseline lives before anyone edits it. + +--- + +## The graph + +Each card gains a dashed target line on its existing trend, plus a caption: + +> **6h 49m under target** · your team's benchmark + +Three things in a few words: the gap, the direction, and which benchmark +applied. `your team's benchmark` versus `the default benchmark` is how an +admin discovers their setting did not save. + +**Direction is per metric.** Lower is better for lead time, CFR and MTTR; +higher is better for deployment frequency. So "under target" is good on three +cards and bad on the fourth. The caption states the fact — `above target` / +`below target` — and colour carries the judgement: green on the good side, +amber on the other. **Never red.** A missed internal goal is not an error, and +colouring it like one makes the dashboard punitive. + +**No benchmark at any level** means no line, no caption, and the card renders +exactly as it does today. That is the state every card is in before anyone +configures anything, so it must look deliberate rather than broken. + +### Configuration screen + +One form per scope: four labelled number inputs with units shown, each with a +placeholder showing the inherited value — an admin sees `24h (default)` greyed +into the lead-time field and understands what clearing it does. + +Team benchmarks live in the existing team settings area. The global baseline +gets a section on the superadmin's Workspaces page, already the superadmin-only +surface. + +Clearing a field removes that key and falls back. That is the only way to +*un*-set a target, so "empty means inherit" is stated in the UI, not just +implied by the code. + +--- + +## Validation + +Server-side on `PUT`. The browser is not a trust boundary and these numbers +feed a shared dashboard. + +| Rule | Why | +|---|---| +| Positive numbers only | A negative or zero lead-time target is unachievable; a zero deployment target is meaningless | +| CFR is 0–100 | It is a percentage | +| Unknown keys rejected | A typo like `leadtime` would store silently and inherit forever | +| Absent is not zero | Omitting a key means inherit; `0` is deliberate and must be preserved | + +The last rule is the trap. A form sending `0` for empty fields would turn +"inherit" into "target zero" on every save, and the dashboard would show every +team failing everything. The UI omits empty fields rather than sending zero, +and the API distinguishes the two. + +## Edge cases + +**A team spanning very different repos** gets one benchmark, because the +dashboard aggregates to team level. A real limitation, stated rather than +solved — per-repo targets would need a rule for combining them into the single +number on the card. + +**A deleted team's benchmark** is orphaned in `Settings`. Harmless and +invisible; cleaning it up means touching team deletion for no user-facing gain. + +**A superadmin viewing another workspace** sees that workspace's team +benchmarks. Resolution keys on the team being viewed, not on who is looking. + +**A benchmark changed while the dashboard is open** does not appear until +reload — the same as every other filter on that page. + +**Zero deployments against a frequency target** still shows the line and +`5/week below target`, which is more useful than hiding the line because the +actual value happens to be zero. + +--- + +## Testing + +Resolution is where the bugs will be, so that is where the tests concentrate: + +- Team value set → used, `source: "team"` +- Team value absent → global used, `source: "global"` +- Neither set → `target: null`, card renders without a line +- **Per-metric mixing** — team sets only lead time; the other three inherit and + report `source: "global"` in the same response +- **`0` is preserved, not treated as absent** + +**Validation:** negatives rejected, CFR above 100 rejected, unknown keys +rejected, omitted keys left untouched rather than nulled. + +**Isolation, request-level e2e:** an admin cannot read or write another +workspace's team benchmark → 403. A non-superadmin cannot `PUT` the global +baseline → 403. + +**Regression, and the most important test here:** with no benchmarks +configured anywhere, every existing metric response is byte-identical to +today. This feature is additive, and the four cards people already rely on +must not shift because a `benchmarks` key appeared. + +**Frontend:** direction handled per metric, so "under target" reads as good on +lead time and bad on deployment frequency; the caption names the source; no +line at all when `target` is null. + +--- + +## Deliberately excluded + +- **No per-workspace tier.** Two levels were specified; a third needs its own + precedence rules for no stated benefit. +- **No custom band thresholds.** The industry pills stay as they are. +- **No history of benchmark changes.** Who changed a target and when is a + reasonable thing to want, and not this feature. +- **No LOC.** Separate spec, separate PR. From dcceaa12a7954b9509736ae2c6c24ccec5d6cc00 Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Tue, 11 Aug 2026 21:35:15 +0500 Subject: [PATCH 093/194] docs: implementation plan for per-team DORA benchmarks Seven tasks. The backend ones carry the code; Task 6, the configuration forms, specifies behaviour rather than MUI bodies and says so in the task. Leads with the thing most likely to go wrong: adding a SettingType to this codebase means touching ten separate extension points, each marked with an upstream comment. Missing one is a silent failure rather than a crash -- the setting saves and reads back as None, or validates and never persists. Task 1 hits all ten and the plan tabulates them. Two constraints repeated at every level because both fail quietly. Absent is not zero: a form sending 0 for empty fields turns inherit into target-zero on every save and shows every team failing everything. And direction is per metric, so under target is good on three cards and bad on deployment frequency -- get it backwards and the dashboard congratulates a team for shipping less. The highest-value check is the regression one: with no benchmarks configured anywhere, the four existing cards must be byte-identical. This feature is additive and the dashboards people already rely on must not shift because a benchmarks key appeared. --- docs/BENCHMARKS_PLAN.md | 837 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 837 insertions(+) create mode 100644 docs/BENCHMARKS_PLAN.md diff --git a/docs/BENCHMARKS_PLAN.md b/docs/BENCHMARKS_PLAN.md new file mode 100644 index 00000000..888ed5f2 --- /dev/null +++ b/docs/BENCHMARKS_PLAN.md @@ -0,0 +1,837 @@ +# Per-team DORA Benchmarks Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Each team can set its own target for the four DORA metrics, inheriting a superadmin baseline when unset, with the target drawn on each metric card. + +**Architecture:** Benchmarks are a new `SettingType` stored in the existing `Settings` table — team rows keyed `(team_id, TEAM)`, the global baseline keyed `(sentinel-uuid, GLOBAL)`. The backend resolves team-then-global **per metric** and returns the resolved target plus its source on the existing metrics response. The frontend draws a line and a caption; it never implements inheritance. + +**Tech Stack:** Python 3.9, Flask, SQLAlchemy 2, pytest (backend); Next.js 15 pages router, TypeScript, MUI 5 (frontend). + +**Design spec:** `docs/BENCHMARKS.md`. Read it before starting. + +## Global Constraints + +- **No database migration, no new tables.** `Settings.entity_type` is `character varying` and the schema declares no native enum types, so `EntityType.GLOBAL` is a code change. `Settings.data` is JSONB. +- **Units match the metrics API exactly** — seconds for `lead_time` and `mean_time_to_recovery`, deployments per week for `deployment_frequency`, percent for `change_failure_rate`. No conversion layer. +- **Absent is not zero.** Omitting a key means inherit; `0` is a deliberate value and must be preserved. A form sending `0` for empty fields would turn inherit into target-zero on every save and show every team failing everything. +- **Direction is per metric.** Lower is better for lead time, CFR and MTTR; higher is better for deployment frequency. +- **Additive only.** With no benchmarks configured anywhere, every existing metric response must be byte-identical to today and the four cards must not shift. +- Backend style: `black` formatted, `flake8` clean. Run from `backend/analytics_server` using `./venv/bin/python`. Baseline: **269 passing**. +- Frontend must be `tsc` clean. Local Node is 20 and the project needs 22 — verify inside the `middleware-dev` container. +- Every Clustox-authored change carries a `# CLUSTOX:` or `// CLUSTOX:` comment explaining **why**, per `docs/FORK_STRATEGY.md`. + +## The ten extension points — read before Task 1 + +Adding a `SettingType` to this codebase means touching ten places. Upstream +marks each with a comment. **Missing one is a silent failure, not a crash** — +the setting saves but reads back wrong, or validates but never persists. + +| # | File | Marker | +|---|---|---| +| 1 | `mhq/store/models/settings/configuration_settings.py:22` | `ADD NEW SETTING TYPE ENUM HERE` | +| 2 | `mhq/service/settings/models.py:60` | `ADD NEW SETTING CLASS HERE` | +| 3 | `mhq/service/settings/default_settings_data.py:38` | `ADD NEW DEFAULT SETTING HERE` | +| 4 | `mhq/service/settings/configuration_settings.py:76` | `ADD NEW DICT TO DATACLASS ADAPTERS HERE` | +| 5 | `mhq/service/settings/configuration_settings.py:101` | `ADD NEW HANDLE FROM DB SETTINGS HERE` | +| 6 | `mhq/service/settings/configuration_settings.py:201` | `ADD NEW DICT TO API ADAPTERS HERE` | +| 7 | `mhq/service/settings/configuration_settings.py:226` | `ADD NEW HANDLE FROM JSON DATA HERE` | +| 8 | `mhq/service/settings/configuration_settings.py:272` | `ADD NEW DATACLASS TO JSON DATA ADAPTERS HERE` | +| 9 | `mhq/service/settings/configuration_settings.py:308` | `ADD NEW HANDLE TO DB SETTINGS HERE` | +| 10 | `mhq/service/settings/setting_type_validator.py:25` | `ADD NEW VALIDATOR HERE` | + +Line numbers drift — search for the marker text, not the number. + +--- + +## File Structure + +**Backend — modify:** +- `mhq/store/models/settings/configuration_settings.py` — `SettingType.BENCHMARK_SETTING` +- `mhq/store/models/settings/enums.py` — `EntityType.GLOBAL` +- `mhq/service/settings/models.py` — `BenchmarkSetting` dataclass +- `mhq/service/settings/default_settings_data.py` — shipped baseline +- `mhq/service/settings/configuration_settings.py` — six adapter/dispatch points +- `mhq/service/settings/setting_type_validator.py` — accept the new type +- `mhq/api/settings.py` — global baseline routes +- `mhq/api/deployment_analytics.py` — attach resolved benchmarks (see Task 3 note) + +**Backend — create:** +- `mhq/service/settings/benchmarks.py` — resolution and validation +- `tests/service/settings/test_benchmarks.py` +- `tests/service/settings/test_benchmark_setting_type.py` + +**Frontend — create:** +- `web-server/pages/api/clustox/benchmarks/global.ts` +- `web-server/src/components/BenchmarkTargetLine.tsx` +- `web-server/src/components/BenchmarkSettingsForm.tsx` +- `web-server/src/utils/benchmarks.ts` + +**Frontend — modify:** +- the four cards under `web-server/src/content/DoraMetrics/DoraCards/` +- `web-server/pages/workspaces.tsx` — global baseline section + +--- + +### Task 1: Register BENCHMARK_SETTING across all ten extension points + +**Files:** +- Modify: `backend/analytics_server/mhq/store/models/settings/configuration_settings.py` +- Modify: `backend/analytics_server/mhq/store/models/settings/enums.py` +- Modify: `backend/analytics_server/mhq/service/settings/models.py` +- Modify: `backend/analytics_server/mhq/service/settings/default_settings_data.py` +- Modify: `backend/analytics_server/mhq/service/settings/configuration_settings.py` +- Modify: `backend/analytics_server/mhq/service/settings/setting_type_validator.py` +- Test: `backend/analytics_server/tests/service/settings/test_benchmark_setting_type.py` (create) + +**Interfaces:** +- Consumes: nothing. +- Produces: + - `SettingType.BENCHMARK_SETTING` (value `"BENCHMARK_SETTING"`) + - `EntityType.GLOBAL` (value `"GLOBAL"`) + - `BenchmarkSetting(lead_time, deployment_frequency, change_failure_rate, mean_time_to_recovery)` — every field `Optional[float]`, defaulting to `None` + - `settings_type_validator("BENCHMARK_SETTING")` returns the enum member + +- [ ] **Step 1: Write the failing tests** + +Create `backend/analytics_server/tests/service/settings/test_benchmark_setting_type.py`: + +```python +from mhq.service.settings.default_settings_data import get_default_setting_data +from mhq.service.settings.models import BenchmarkSetting +from mhq.service.settings.setting_type_validator import settings_type_validator +from mhq.store.models.settings import SettingType +from mhq.store.models.settings.enums import EntityType + + +def test_benchmark_setting_type_exists(): + assert SettingType.BENCHMARK_SETTING.value == "BENCHMARK_SETTING" + + +def test_global_entity_type_exists(): + # The superadmin baseline belongs to no team and no workspace. + assert EntityType.GLOBAL.value == "GLOBAL" + + +def test_validator_accepts_the_new_type(): + assert settings_type_validator("BENCHMARK_SETTING") == SettingType.BENCHMARK_SETTING + + +def test_default_baseline_has_all_four_metrics(): + data = get_default_setting_data(SettingType.BENCHMARK_SETTING) + + assert set(data.keys()) == { + "lead_time", + "deployment_frequency", + "change_failure_rate", + "mean_time_to_recovery", + } + + +def test_benchmark_setting_fields_default_to_none(): + # Every field is optional: a team may set one target and inherit the rest. + setting = BenchmarkSetting() + + assert setting.lead_time is None + assert setting.deployment_frequency is None + assert setting.change_failure_rate is None + assert setting.mean_time_to_recovery is None +``` + +- [ ] **Step 2: Run tests to verify they fail** + +Run: `cd backend/analytics_server && ./venv/bin/python -m pytest tests/service/settings/test_benchmark_setting_type.py -v` +Expected: FAIL with `AttributeError: BENCHMARK_SETTING` + +- [ ] **Step 3: Add the enums** + +In `mhq/store/models/settings/configuration_settings.py`, at the `ADD NEW SETTING TYPE ENUM HERE` marker: + +```python + # CLUSTOX: per-team DORA targets. Stored here rather than in a new table + # because Settings already has the entity scoping, service and API this + # needs, and its JSONB data column means adding a fifth metric later costs + # no migration. + BENCHMARK_SETTING = "BENCHMARK_SETTING" +``` + +In `mhq/store/models/settings/enums.py`: + +```python +class EntityType(Enum): + USER = "USER" + TEAM = "TEAM" + ORG = "ORG" + # CLUSTOX: the superadmin's baseline benchmark belongs to no team and no + # workspace. entity_type is character varying with no native enum, so this + # needs no migration. + GLOBAL = "GLOBAL" +``` + +- [ ] **Step 4: Add the dataclass** + +In `mhq/service/settings/models.py`, at the `ADD NEW SETTING CLASS HERE` marker: + +```python +# CLUSTOX: every field is optional so fallback is per metric, not +# all-or-nothing -- a team may set a lead-time target and inherit the rest. +@dataclass +class BenchmarkSetting(BaseSetting): + lead_time: Optional[float] = None + deployment_frequency: Optional[float] = None + change_failure_rate: Optional[float] = None + mean_time_to_recovery: Optional[float] = None +``` + +Check that file's existing imports for `dataclass` and `Optional` before adding them. + +- [ ] **Step 5: Add the shipped baseline** + +In `mhq/service/settings/default_settings_data.py`, at the `ADD NEW DEFAULT SETTING HERE` marker: + +```python + # CLUSTOX: the baseline a workspace sees before a superadmin edits it. + # Values are the DORA "High" band, in the same units the metrics API + # returns: seconds, deployments per week, percent. + if setting_type == SettingType.BENCHMARK_SETTING: + return { + "lead_time": 86400, + "deployment_frequency": 5, + "change_failure_rate": 15, + "mean_time_to_recovery": 3600, + } +``` + +- [ ] **Step 6: Add the six adapters and dispatch branches** + +In `mhq/service/settings/configuration_settings.py`, add an adapter at the `ADD NEW DICT TO DATACLASS ADAPTERS HERE` marker: + +```python + # CLUSTOX: `data.get(key)` rather than `data.get(key, 0)` -- absent means + # inherit, and 0 is a deliberate target. Collapsing them would make every + # unset metric read as a target of zero. + def _adapt_benchmark_setting_from_setting_data(self, data: Dict[str, any]): + return BenchmarkSetting( + lead_time=data.get("lead_time"), + deployment_frequency=data.get("deployment_frequency"), + change_failure_rate=data.get("change_failure_rate"), + mean_time_to_recovery=data.get("mean_time_to_recovery"), + ) +``` + +At `ADD NEW HANDLE FROM DB SETTINGS HERE`: + +```python + if setting_type == SettingType.BENCHMARK_SETTING: + return self._adapt_benchmark_setting_from_setting_data(setting_data) +``` + +At `ADD NEW DICT TO API ADAPTERS HERE`, and at `ADD NEW DATACLASS TO JSON DATA ADAPTERS HERE`, add adapters mirroring the two `default_sync_days` adapters immediately above each marker — read those first and follow their shape exactly, substituting the four benchmark keys. At `ADD NEW HANDLE FROM JSON DATA HERE` and `ADD NEW HANDLE TO DB SETTINGS HERE`, add the matching `if setting_type == SettingType.BENCHMARK_SETTING:` branches. + +All four keys must round-trip through every adapter. A key missing from one of them saves and then reads back as `None`. + +- [ ] **Step 7: Add the validator branch** + +In `mhq/service/settings/setting_type_validator.py`, at the `ADD NEW VALIDATOR HERE` marker: + +```python + if setting_type == SettingType.BENCHMARK_SETTING.value: + return SettingType.BENCHMARK_SETTING +``` + +- [ ] **Step 8: Run tests to verify they pass** + +Run: `cd backend/analytics_server && ./venv/bin/python -m pytest tests/service/settings/test_benchmark_setting_type.py -v` +Expected: PASS (5 tests) + +- [ ] **Step 9: Run the full suite, format, lint, commit** + +```bash +cd backend/analytics_server +./venv/bin/python -m pytest tests -q # expect 274 passed +./venv/bin/python -m black mhq/store/models/settings/ mhq/service/settings/ tests/service/settings/ +./venv/bin/python -m flake8 mhq/store/models/settings/ mhq/service/settings/ +cd ../.. +git add backend/analytics_server/mhq backend/analytics_server/tests +git commit -m "feat(benchmarks): register the benchmark setting type" +``` + +--- + +### Task 2: Resolution and validation + +**Files:** +- Create: `backend/analytics_server/mhq/service/settings/benchmarks.py` +- Test: `backend/analytics_server/tests/service/settings/test_benchmarks.py` (create) + +**Interfaces:** +- Consumes: `BenchmarkSetting`, `SettingType.BENCHMARK_SETTING`, `EntityType.GLOBAL` (Task 1). +- Produces: + - `GLOBAL_BENCHMARK_ENTITY_ID: str` — the sentinel UUID for the global row + - `BENCHMARK_METRICS: List[str]` — the four metric keys + - `resolve_benchmarks(team_setting: Optional[BenchmarkSetting], global_setting: Optional[BenchmarkSetting]) -> Dict[str, Dict]` returning `{metric: {"target": float|None, "source": "team"|"global"|None}}` + - `validate_benchmark_payload(data: Dict) -> Dict` — returns the cleaned dict, raises `BadRequest` on invalid input + +- [ ] **Step 1: Write the failing tests** + +Create `backend/analytics_server/tests/service/settings/test_benchmarks.py`: + +```python +import pytest +from werkzeug.exceptions import BadRequest + +from mhq.service.settings.benchmarks import ( + BENCHMARK_METRICS, + resolve_benchmarks, + validate_benchmark_payload, +) +from mhq.service.settings.models import BenchmarkSetting + + +def test_team_value_wins_and_reports_its_source(): + team = BenchmarkSetting(lead_time=3600) + glob = BenchmarkSetting(lead_time=86400) + + resolved = resolve_benchmarks(team, glob) + + assert resolved["lead_time"] == {"target": 3600, "source": "team"} + + +def test_absent_team_value_falls_back_to_global(): + team = BenchmarkSetting(lead_time=3600) + glob = BenchmarkSetting(lead_time=86400, deployment_frequency=5) + + resolved = resolve_benchmarks(team, glob) + + assert resolved["deployment_frequency"] == {"target": 5, "source": "global"} + + +def test_neither_set_yields_no_target(): + resolved = resolve_benchmarks(BenchmarkSetting(), BenchmarkSetting()) + + assert resolved["change_failure_rate"] == {"target": None, "source": None} + + +def test_metrics_mix_sources_within_one_response(): + # The case that makes per-metric fallback worth having. + team = BenchmarkSetting(lead_time=3600) + glob = BenchmarkSetting(lead_time=86400, deployment_frequency=5) + + resolved = resolve_benchmarks(team, glob) + + assert resolved["lead_time"]["source"] == "team" + assert resolved["deployment_frequency"]["source"] == "global" + assert resolved["mean_time_to_recovery"]["source"] is None + + +def test_zero_is_a_real_target_not_an_absent_one(): + # A team targeting zero failures must not silently inherit 15%. + team = BenchmarkSetting(change_failure_rate=0) + glob = BenchmarkSetting(change_failure_rate=15) + + resolved = resolve_benchmarks(team, glob) + + assert resolved["change_failure_rate"] == {"target": 0, "source": "team"} + + +def test_missing_settings_are_treated_as_empty(): + resolved = resolve_benchmarks(None, None) + + assert set(resolved.keys()) == set(BENCHMARK_METRICS) + assert all(v["target"] is None for v in resolved.values()) + + +def test_validation_rejects_negatives(): + with pytest.raises(BadRequest): + validate_benchmark_payload({"lead_time": -1}) + + +def test_validation_rejects_change_failure_rate_above_100(): + with pytest.raises(BadRequest): + validate_benchmark_payload({"change_failure_rate": 101}) + + +def test_validation_rejects_unknown_keys(): + # A typo would otherwise store silently and inherit forever. + with pytest.raises(BadRequest): + validate_benchmark_payload({"leadtime": 3600}) + + +def test_validation_keeps_zero_and_omits_nothing_else(): + cleaned = validate_benchmark_payload({"change_failure_rate": 0}) + + assert cleaned == {"change_failure_rate": 0} + + +def test_validation_accepts_a_partial_payload(): + cleaned = validate_benchmark_payload({"lead_time": 3600}) + + assert cleaned == {"lead_time": 3600} +``` + +- [ ] **Step 2: Run tests to verify they fail** + +Run: `cd backend/analytics_server && ./venv/bin/python -m pytest tests/service/settings/test_benchmarks.py -v` +Expected: FAIL with `ModuleNotFoundError: No module named 'mhq.service.settings.benchmarks'` + +- [ ] **Step 3: Write the module** + +Create `backend/analytics_server/mhq/service/settings/benchmarks.py`: + +```python +# CLUSTOX: benchmark resolution and validation. +# +# Resolution lives in the backend rather than the browser for three reasons: +# every consumer would otherwise reimplement the fallback and eventually +# disagree; the response can report WHICH benchmark applied, so an admin who +# thinks they set a target can see that they did not; and a fifth metric later +# is one more key here rather than one more reimplementation. +from typing import Dict, List, Optional + +from werkzeug.exceptions import BadRequest + +from mhq.service.settings.models import BenchmarkSetting + +# The global baseline belongs to no team and no workspace, but Settings. +# entity_id is NOT NULL. A fixed sentinel is the documented cost of reusing +# that table instead of adding one. +GLOBAL_BENCHMARK_ENTITY_ID = "00000000-0000-4000-8000-000000000001" + +BENCHMARK_METRICS: List[str] = [ + "lead_time", + "deployment_frequency", + "change_failure_rate", + "mean_time_to_recovery", +] + +# Percent, so it has an upper bound the others do not. +_BOUNDED_METRICS = {"change_failure_rate": 100} + + +def resolve_benchmarks( + team_setting: Optional[BenchmarkSetting], + global_setting: Optional[BenchmarkSetting], +) -> Dict[str, Dict]: + resolved = {} + + for metric in BENCHMARK_METRICS: + team_value = getattr(team_setting, metric, None) if team_setting else None + global_value = getattr(global_setting, metric, None) if global_setting else None + + # `is not None` rather than truthiness: 0 is a deliberate target. + if team_value is not None: + resolved[metric] = {"target": team_value, "source": "team"} + elif global_value is not None: + resolved[metric] = {"target": global_value, "source": "global"} + else: + resolved[metric] = {"target": None, "source": None} + + return resolved + + +def validate_benchmark_payload(data: Dict) -> Dict: + unknown = set(data.keys()) - set(BENCHMARK_METRICS) + if unknown: + raise BadRequest(f"Unknown benchmark metrics: {', '.join(sorted(unknown))}") + + cleaned = {} + for metric, value in data.items(): + if value is None: + continue + + if not isinstance(value, (int, float)) or isinstance(value, bool): + raise BadRequest(f"{metric} must be a number") + + if value < 0: + raise BadRequest(f"{metric} must not be negative") + + upper = _BOUNDED_METRICS.get(metric) + if upper is not None and value > upper: + raise BadRequest(f"{metric} must be between 0 and {upper}") + + cleaned[metric] = value + + return cleaned +``` + +- [ ] **Step 4: Run tests to verify they pass** + +Run: `cd backend/analytics_server && ./venv/bin/python -m pytest tests/service/settings/test_benchmarks.py -v` +Expected: PASS (11 tests) + +- [ ] **Step 5: Run the full suite, format, lint, commit** + +```bash +cd backend/analytics_server +./venv/bin/python -m pytest tests -q # expect 285 passed +./venv/bin/python -m black mhq/service/settings/benchmarks.py tests/service/settings/test_benchmarks.py +./venv/bin/python -m flake8 mhq/service/settings/benchmarks.py +cd ../.. +git add backend/analytics_server/mhq/service/settings/benchmarks.py \ + backend/analytics_server/tests/service/settings/test_benchmarks.py +git commit -m "feat(benchmarks): resolve team targets against the global baseline" +``` + +--- + +### Task 3: Attach resolved benchmarks to the metrics response + +**Files:** +- Modify: `backend/analytics_server/mhq/api/deployment_analytics.py` +- Test: extend `backend/analytics_server/tests/service/settings/test_benchmarks.py` + +**Interfaces:** +- Consumes: `resolve_benchmarks`, `GLOBAL_BENCHMARK_ENTITY_ID` (Task 2); `SettingsService.get_settings` (existing). +- Produces: `get_resolved_benchmarks_for_team(team_id: str) -> Dict[str, Dict]`, and a `"benchmarks"` key on the team DORA metrics response. + +- [ ] **Step 1: Write the failing test** + +Append to `backend/analytics_server/tests/service/settings/test_benchmarks.py`: + +```python +def test_resolution_asks_for_the_team_row_and_the_global_row(): + from mhq.service.settings.benchmarks import GLOBAL_BENCHMARK_ENTITY_ID + from mhq.store.models.settings import SettingType + from mhq.store.models.settings.enums import EntityType + + asked = [] + + class FakeSettingsService: + def get_settings(self, setting_type, entity_type, entity_id): + asked.append((setting_type, entity_type, entity_id)) + return None + + from mhq.api.deployment_analytics import get_resolved_benchmarks_for_team + + get_resolved_benchmarks_for_team("team-1", settings_service=FakeSettingsService()) + + assert ( + SettingType.BENCHMARK_SETTING, + EntityType.TEAM, + "team-1", + ) in asked + assert ( + SettingType.BENCHMARK_SETTING, + EntityType.GLOBAL, + GLOBAL_BENCHMARK_ENTITY_ID, + ) in asked +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `cd backend/analytics_server && ./venv/bin/python -m pytest tests/service/settings/test_benchmarks.py::test_resolution_asks_for_the_team_row_and_the_global_row -v` +Expected: FAIL with `ImportError: cannot import name 'get_resolved_benchmarks_for_team'` + +- [ ] **Step 3: Add the helper** + +Add to `backend/analytics_server/mhq/api/deployment_analytics.py`: + +```python +# CLUSTOX: resolve a team's benchmarks for the metrics response. Injectable +# settings_service so the resolution order can be tested without a database. +def get_resolved_benchmarks_for_team(team_id: str, settings_service=None): + from mhq.service.settings.benchmarks import ( + GLOBAL_BENCHMARK_ENTITY_ID, + resolve_benchmarks, + ) + from mhq.service.settings.configuration_settings import get_settings_service + from mhq.store.models.settings import SettingType + from mhq.store.models.settings.enums import EntityType + + settings_service = settings_service or get_settings_service() + + team_setting = settings_service.get_settings( + SettingType.BENCHMARK_SETTING, EntityType.TEAM, team_id + ) + global_setting = settings_service.get_settings( + SettingType.BENCHMARK_SETTING, EntityType.GLOBAL, GLOBAL_BENCHMARK_ENTITY_ID + ) + + return resolve_benchmarks( + getattr(team_setting, "specific_settings", None), + getattr(global_setting, "specific_settings", None), + ) +``` + +- [ ] **Step 4: Attach it to the response** + +Find the route in `mhq/api/deployment_analytics.py` that serves the team's DORA metrics (`/teams//deployment_analytics`). Add `"benchmarks": get_resolved_benchmarks_for_team(team_id)` to the dict it returns. + +Read the existing return statement before editing. Do not reorder or rename any existing key — the regression guard in Step 5 exists because the four cards already rely on this response. + +- [ ] **Step 5: Run the full suite** + +Run: `cd backend/analytics_server && ./venv/bin/python -m pytest tests -q` +Expected: 286 passed, with no pre-existing test failing. If an existing test breaks, you changed the response shape rather than adding to it. + +- [ ] **Step 6: Format, lint, commit** + +```bash +cd backend/analytics_server +./venv/bin/python -m black mhq/api/deployment_analytics.py tests/service/settings/test_benchmarks.py +./venv/bin/python -m flake8 mhq/api/deployment_analytics.py +cd ../.. +git add backend/analytics_server/mhq/api/deployment_analytics.py \ + backend/analytics_server/tests/service/settings/test_benchmarks.py +git commit -m "feat(benchmarks): return resolved targets with team metrics" +``` + +--- + +### Task 4: Global baseline routes + +**Files:** +- Modify: `backend/analytics_server/mhq/api/settings.py` +- Create: `web-server/pages/api/clustox/benchmarks/global.ts` + +**Interfaces:** +- Consumes: `GLOBAL_BENCHMARK_ENTITY_ID`, `validate_benchmark_payload` (Task 2). +- Produces: + - Flask: `GET /settings/global?setting_type=BENCHMARK_SETTING`, `PUT /settings/global` + - BFF: `GET`/`PUT` on `/api/clustox/benchmarks/global`, superadmin only + +- [ ] **Step 1: Add the Flask routes** + +In `backend/analytics_server/mhq/api/settings.py`, read `get_team_settings` and `put_team_settings` first and mirror their decorator stack, schema style and response adapter exactly. Add: + +```python +# CLUSTOX: the superadmin's baseline belongs to no team and no workspace, so +# it cannot use the team- or org-scoped routes above. Authorisation is enforced +# at the BFF -- this layer only knows the internal token, not who is calling. +@app.route("/settings/global", methods={"GET"}) +``` + +and the matching `PUT`, both operating on `entity_type=EntityType.GLOBAL` and `entity_id=GLOBAL_BENCHMARK_ENTITY_ID`. The `PUT` must pass its payload through `validate_benchmark_payload` before saving. + +- [ ] **Step 2: Add the BFF route** + +Create `web-server/pages/api/clustox/benchmarks/global.ts`: + +```ts +import * as yup from 'yup'; + +import { Endpoint, nullSchema } from '@/api-helpers/global'; +import { internal } from '@/api-helpers/axios'; +import { assertRole } from '@/auth/guard'; + +// CLUSTOX: the global baseline is superadmin-only. The Flask layer trusts the +// internal token and has no notion of who is calling, so the role check has to +// live here. +const putSchema = yup.object().shape({ + lead_time: yup.number().min(0).nullable().optional(), + deployment_frequency: yup.number().min(0).nullable().optional(), + change_failure_rate: yup.number().min(0).max(100).nullable().optional(), + mean_time_to_recovery: yup.number().min(0).nullable().optional() +}); + +const endpoint = new Endpoint(nullSchema); + +endpoint.handle.GET(nullSchema, async (req, res) => { + assertRole((req as any).session, 'SUPERADMIN'); + + const result = await internal.get('/settings/global', { + params: { setting_type: 'BENCHMARK_SETTING' } + }); + res.send(result.data); +}); + +endpoint.handle.PUT(putSchema, async (req, res) => { + assertRole((req as any).session, 'SUPERADMIN'); + + const result = await internal.put('/settings/global', { + setting_type: 'BENCHMARK_SETTING', + setting_data: req.payload + }); + res.send(result.data); +}); + +export default endpoint.serve(); +``` + +Note the constructor receives `nullSchema`, not `putSchema`. `Endpoint.serve()` validates the constructor's schema against every method before dispatch, so a method-specific schema there makes `GET` a permanent 400. That exact bug has been found in this codebase twice. + +- [ ] **Step 3: Verify** + +```bash +cd backend/analytics_server && ./venv/bin/python -m pytest tests -q +docker cp web-server/pages/api/clustox/benchmarks/global.ts middleware-dev:/app/web-server/pages/api/clustox/benchmarks/global.ts +docker exec middleware-dev sh -c 'cd /app/web-server && ./node_modules/.bin/tsc --noEmit 2>&1 | grep -c "error TS"' +``` +Expected: suite unchanged at 286; tsc reports `0`. + +- [ ] **Step 4: Commit** + +```bash +git add backend/analytics_server/mhq/api/settings.py \ + web-server/pages/api/clustox/benchmarks/global.ts +git commit -m "feat(benchmarks): expose the superadmin baseline" +``` + +--- + +### Task 5: Target line and caption on the metric cards + +**Files:** +- Create: `web-server/src/utils/benchmarks.ts` +- Create: `web-server/src/components/BenchmarkTargetLine.tsx` +- Modify: the four cards in `web-server/src/content/DoraMetrics/DoraCards/` + +**Interfaces:** +- Consumes: the `benchmarks` key on the metrics response (Task 3). +- Produces: + - `benchmarkCaption(metric, actual, target, source) -> { text: string; tone: 'good' | 'warn' }` + - `` + +- [ ] **Step 1: Write the failing test** + +Create `web-server/src/utils/__tests__/benchmarks.test.ts`: + +```ts +import { benchmarkCaption } from '../benchmarks'; + +describe('benchmarkCaption', () => { + it('treats being under target as good for lead time', () => { + const r = benchmarkCaption('lead_time', 3600, 7200, 'team'); + expect(r.tone).toBe('good'); + expect(r.text).toContain('under target'); + }); + + it('treats being under target as bad for deployment frequency', () => { + // Higher is better here -- the same direction means the opposite thing. + const r = benchmarkCaption('deployment_frequency', 2, 5, 'team'); + expect(r.tone).toBe('warn'); + expect(r.text).toContain('below target'); + }); + + it('names the source so a team can see its setting did not save', () => { + expect(benchmarkCaption('lead_time', 3600, 7200, 'global').text).toContain( + 'default' + ); + expect(benchmarkCaption('lead_time', 3600, 7200, 'team').text).toContain( + 'team' + ); + }); + + it('returns null when there is no target', () => { + expect(benchmarkCaption('lead_time', 3600, null, null)).toBeNull(); + }); +}); +``` + +- [ ] **Step 2: Run it to verify it fails** + +Run: `docker exec middleware-dev sh -c 'cd /app/web-server && yarn test --testPathPattern=benchmarks'` +Expected: FAIL — module not found + +- [ ] **Step 3: Write the helper** + +Create `web-server/src/utils/benchmarks.ts` exporting `benchmarkCaption`. Encode direction per metric — `lead_time`, `change_failure_rate` and `mean_time_to_recovery` are better when lower; `deployment_frequency` is better when higher. Return `tone: 'good'` on the favourable side and `'warn'` on the other. **Never red**: a missed internal goal is not an error, and colouring it like one makes the dashboard punitive. Return `null` when `target` is null so cards render exactly as they do today. + +- [ ] **Step 4: Add the line and caption to the cards** + +Create `BenchmarkTargetLine.tsx` — a dashed horizontal rule positioned on the card's existing trend chart — and render it plus the caption in each of `ChangeTimeCard`, `WeeklyDeliveryVolumeCard`, `ChangeFailureRateCard` and `MeanTimeToRestoreCard`. Read one card fully before editing; follow its existing layout rather than introducing a new one. + +When `benchmarks[metric].target` is null, render neither. That is the state every card is in before anyone configures anything, so it must look deliberate. + +- [ ] **Step 5: Verify and commit** + +```bash +docker exec middleware-dev sh -c 'cd /app/web-server && yarn test --testPathPattern=benchmarks' +docker exec middleware-dev sh -c 'cd /app/web-server && ./node_modules/.bin/tsc --noEmit 2>&1 | grep -c "error TS"' +git add web-server/src/utils/benchmarks.ts web-server/src/utils/__tests__/benchmarks.test.ts \ + web-server/src/components/BenchmarkTargetLine.tsx \ + web-server/src/content/DoraMetrics/DoraCards/ +git commit -m "feat(benchmarks): draw the target on each metric card" +``` + +--- + +### Task 6: Configuration forms + +**Files:** +- Create: `web-server/src/components/BenchmarkSettingsForm.tsx` +- Modify: `web-server/pages/workspaces.tsx` + +**Interfaces:** +- Consumes: `/api/clustox/benchmarks/global` (Task 4) and the existing team settings routes. +- Produces: nothing consumed by later tasks. + +> **Resolution note.** Tasks 1–4 give exact code. This task specifies behaviour +> and contracts rather than full MUI bodies, because the markup is long and +> mechanical and is better written against the live components. Treat the +> acceptance criteria as the requirement. + +- [ ] **Step 1: Build the form** + +`BenchmarkSettingsForm.tsx` — four labelled number inputs with units shown (`hours`, `per week`, `%`, `hours`), taking a `scope` prop of `'team'` or `'global'` and posting to the corresponding endpoint. + +Required behaviours: + +- **An empty input means inherit, and must be omitted from the payload — never sent as `0`.** Sending `0` would turn inherit into target-zero on every save and show every team failing everything. +- Each input's placeholder shows the inherited value, e.g. `24h (default)`, so an admin can see what clearing it does. +- Hours are entered as hours and converted to seconds on submit; the API stores seconds. +- Validation errors from the server are shown inline against the field named in the message, not as a toast. + +- [ ] **Step 2: Mount both scopes** + +Team benchmarks go in the existing team settings area. The global baseline gets a section on `pages/workspaces.tsx`, already the superadmin-only surface, rendered only when `isSuperadmin`. + +- [ ] **Step 3: Verify** + +```bash +docker exec middleware-dev sh -c 'cd /app/web-server && ./node_modules/.bin/tsc --noEmit 2>&1 | grep -c "error TS"' +docker exec middleware-dev sh -c 'cd /app/web-server && yarn lint' +``` + +Then by hand: set a team lead-time target, confirm the card shows a line and `your team's benchmark`; clear it, confirm it reverts to `the default benchmark`; set a target of `0` for change failure rate and confirm it persists as 0 rather than inheriting. + +- [ ] **Step 4: Commit** + +```bash +git add web-server/src/components/BenchmarkSettingsForm.tsx web-server/pages/workspaces.tsx +git commit -m "feat(benchmarks): add the benchmark configuration forms" +``` + +--- + +### Task 7: Cross-workspace isolation tests + +**Files:** +- Create: `web-server/e2e/benchmarks.spec.ts` + +**Interfaces:** +- Consumes: the endpoints from Tasks 4 and 6. +- Produces: nothing. + +- [ ] **Step 1: Write the isolation tests** + +Create `web-server/e2e/benchmarks.spec.ts` following the structure of the existing `web-server/e2e/multitenancy.spec.ts` — read it first and match its sign-in helper, admin provisioning and `afterAll` cleanup exactly. Cover: + +- An admin cannot read another workspace's team benchmark → **403** +- An admin cannot write another workspace's team benchmark → **403** +- A non-superadmin `PUT` to `/api/clustox/benchmarks/global` → **403** +- An unauthenticated request to either → **401** +- A superadmin can read and write the global baseline → **200** + +Use RFC-4122 valid UUIDs for any placeholder ids. `yup.string().uuid()` enforces the version and variant nibbles, so `00000000-0000-0000-0000-000000000001` is rejected at schema validation and the test would assert 400-as-403 — a bug already found once in this codebase. + +Use `00000000-0000-4000-8000-0000000000ff` for placeholders. Do **not** reuse `00000000-0000-4000-8000-000000000001`: that is `GLOBAL_BENCHMARK_ENTITY_ID`, the real global baseline row, and a test using it as a "nonexistent team" would be reasoning about live data. + +- [ ] **Step 2: Run and commit** + +```bash +cd web-server && SUPERADMIN_PASSWORD='' yarn playwright test e2e/benchmarks.spec.ts +git add web-server/e2e/benchmarks.spec.ts +git commit -m "test(benchmarks): cross-workspace isolation" +``` + +--- + +## Final verification + +- [ ] `cd backend/analytics_server && ./venv/bin/python -m pytest tests -q` — expect 286 passed +- [ ] `./venv/bin/python -m black --check mhq && ./venv/bin/python -m flake8 mhq` on changed files — clean +- [ ] `docker exec middleware-dev sh -c 'cd /app/web-server && ./node_modules/.bin/tsc --noEmit'` — 0 errors +- [ ] `cd web-server && yarn playwright test e2e/` — all suites pass +- [ ] **With no benchmarks configured anywhere, the four cards render exactly as they do today.** This is the highest-value check in the plan: the feature is additive, and the dashboards people already rely on must not shift because a `benchmarks` key appeared. +- [ ] Manual: set a team target, confirm the line and `your team's benchmark`; clear it, confirm `the default benchmark`; set `0` and confirm it persists. From f64a0a4e147d6c84c6a24d1fc287bd14e4adf5b2 Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Tue, 11 Aug 2026 21:40:44 +0500 Subject: [PATCH 094/194] feat(benchmarks): register the benchmark setting type --- .../settings/configuration_settings.py | 44 +++++++++++++++++++ .../service/settings/default_settings_data.py | 12 ++++- .../mhq/service/settings/models.py | 12 ++++- .../settings/setting_type_validator.py | 3 ++ .../models/settings/configuration_settings.py | 6 +++ .../mhq/store/models/settings/enums.py | 4 ++ .../tests/service/settings/__init__.py | 0 .../settings/test_benchmark_setting_type.py | 39 ++++++++++++++++ 8 files changed, 118 insertions(+), 2 deletions(-) create mode 100644 backend/analytics_server/tests/service/settings/__init__.py create mode 100644 backend/analytics_server/tests/service/settings/test_benchmark_setting_type.py diff --git a/backend/analytics_server/mhq/service/settings/configuration_settings.py b/backend/analytics_server/mhq/service/settings/configuration_settings.py index 14d04661..2408fb49 100644 --- a/backend/analytics_server/mhq/service/settings/configuration_settings.py +++ b/backend/analytics_server/mhq/service/settings/configuration_settings.py @@ -3,6 +3,7 @@ from mhq.service.settings.default_settings_data import get_default_setting_data from mhq.service.settings.models import ( + BenchmarkSetting, ConfigurationSettings, DefaultSyncDaysSetting, ExcludedPRsSetting, @@ -73,6 +74,17 @@ def _adapt_incident_prs_setting_from_setting_data(self, data: Dict[str, any]): filters=data.get("filters", []), ) + # CLUSTOX: `data.get(key)` rather than `data.get(key, 0)` -- absent means + # inherit, and 0 is a deliberate target. Collapsing them would make every + # unset metric read as a target of zero. + def _adapt_benchmark_setting_from_setting_data(self, data: Dict[str, any]): + return BenchmarkSetting( + lead_time=data.get("lead_time"), + deployment_frequency=data.get("deployment_frequency"), + change_failure_rate=data.get("change_failure_rate"), + mean_time_to_recovery=data.get("mean_time_to_recovery"), + ) + # ADD NEW DICT TO DATACLASS ADAPTERS HERE def _handle_config_setting_from_db_setting( @@ -98,6 +110,9 @@ def _handle_config_setting_from_db_setting( if setting_type == SettingType.INCIDENT_PRS_SETTING: return self._adapt_incident_prs_setting_from_setting_data(setting_data) + if setting_type == SettingType.BENCHMARK_SETTING: + return self._adapt_benchmark_setting_from_setting_data(setting_data) + # ADD NEW HANDLE FROM DB SETTINGS HERE raise Exception(f"Invalid Setting Type: {setting_type}") @@ -198,6 +213,17 @@ def _adapt_incident_prs_setting_from_json(self, data: Dict[str, any]): filters=data.get("filters", []), ) + # CLUSTOX: `data.get(key)` rather than `data.get(key, 0)` -- absent means + # inherit, and 0 is a deliberate target. Collapsing them would make every + # unset metric read as a target of zero. + def _adapt_benchmark_setting_from_json(self, data: Dict[str, any]): + return BenchmarkSetting( + lead_time=data.get("lead_time"), + deployment_frequency=data.get("deployment_frequency"), + change_failure_rate=data.get("change_failure_rate"), + mean_time_to_recovery=data.get("mean_time_to_recovery"), + ) + # ADD NEW DICT TO API ADAPTERS HERE def _handle_config_setting_from_json_data( @@ -223,6 +249,9 @@ def _handle_config_setting_from_json_data( if setting_type == SettingType.INCIDENT_PRS_SETTING: return self._adapt_incident_prs_setting_from_json(setting_data) + if setting_type == SettingType.BENCHMARK_SETTING: + return self._adapt_benchmark_setting_from_json(setting_data) + # ADD NEW HANDLE FROM JSON DATA HERE raise Exception(f"Invalid Setting Type: {setting_type}") @@ -269,6 +298,16 @@ def _adapt_incident_prs_setting_json_data( "filters": specific_setting.filters, } + def _adapt_benchmark_setting_json_data( + self, specific_setting: BenchmarkSetting + ) -> Dict: + return { + "lead_time": specific_setting.lead_time, + "deployment_frequency": specific_setting.deployment_frequency, + "change_failure_rate": specific_setting.change_failure_rate, + "mean_time_to_recovery": specific_setting.mean_time_to_recovery, + } + # ADD NEW DATACLASS TO JSON DATA ADAPTERS HERE def _handle_config_setting_to_db_setting( @@ -305,6 +344,11 @@ def _handle_config_setting_to_db_setting( ): return self._adapt_incident_prs_setting_json_data(specific_setting) + if setting_type == SettingType.BENCHMARK_SETTING and isinstance( + specific_setting, BenchmarkSetting + ): + return self._adapt_benchmark_setting_json_data(specific_setting) + # ADD NEW HANDLE TO DB SETTINGS HERE raise Exception(f"Invalid Setting Type: {setting_type}") diff --git a/backend/analytics_server/mhq/service/settings/default_settings_data.py b/backend/analytics_server/mhq/service/settings/default_settings_data.py index 0e151a70..bc65ca6e 100644 --- a/backend/analytics_server/mhq/service/settings/default_settings_data.py +++ b/backend/analytics_server/mhq/service/settings/default_settings_data.py @@ -1,7 +1,6 @@ from mhq.store.models.incidents import IncidentSource, IncidentType from mhq.store.models.settings import SettingType - MIN_CYCLE_TIME_THRESHOLD = 3600 @@ -35,6 +34,17 @@ def get_default_setting_data(setting_type: SettingType): "filters": [], } + # CLUSTOX: the baseline a workspace sees before a superadmin edits it. + # Values are the DORA "High" band, in the same units the metrics API + # returns: seconds, deployments per week, percent. + if setting_type == SettingType.BENCHMARK_SETTING: + return { + "lead_time": 86400, + "deployment_frequency": 5, + "change_failure_rate": 15, + "mean_time_to_recovery": 3600, + } + # ADD NEW DEFAULT SETTING HERE raise Exception(f"Invalid Setting Type: {setting_type}") diff --git a/backend/analytics_server/mhq/service/settings/models.py b/backend/analytics_server/mhq/service/settings/models.py index d568cbea..b4514535 100644 --- a/backend/analytics_server/mhq/service/settings/models.py +++ b/backend/analytics_server/mhq/service/settings/models.py @@ -1,6 +1,6 @@ from dataclasses import dataclass from datetime import datetime -from typing import List, TypedDict, Literal +from typing import List, TypedDict, Literal, Optional from mhq.store.models import EntityType from mhq.store.models.incidents.enums import IncidentSource, IncidentType @@ -57,6 +57,16 @@ class IncidentPRsSetting(BaseSetting): filters: List[IncidentPRFilter] +# CLUSTOX: every field is optional so fallback is per metric, not +# all-or-nothing -- a team may set a lead-time target and inherit the rest. +@dataclass +class BenchmarkSetting(BaseSetting): + lead_time: Optional[float] = None + deployment_frequency: Optional[float] = None + change_failure_rate: Optional[float] = None + mean_time_to_recovery: Optional[float] = None + + # ADD NEW SETTING CLASS HERE # Sample Future Settings diff --git a/backend/analytics_server/mhq/service/settings/setting_type_validator.py b/backend/analytics_server/mhq/service/settings/setting_type_validator.py index 7d71ba7e..31bba609 100644 --- a/backend/analytics_server/mhq/service/settings/setting_type_validator.py +++ b/backend/analytics_server/mhq/service/settings/setting_type_validator.py @@ -22,6 +22,9 @@ def settings_type_validator(setting_type: str): if setting_type == SettingType.INCIDENT_PRS_SETTING.value: return SettingType.INCIDENT_PRS_SETTING + if setting_type == SettingType.BENCHMARK_SETTING.value: + return SettingType.BENCHMARK_SETTING + # ADD NEW VALIDATOR HERE raise BadRequest(f"Invalid Setting Type: {setting_type}") diff --git a/backend/analytics_server/mhq/store/models/settings/configuration_settings.py b/backend/analytics_server/mhq/store/models/settings/configuration_settings.py index e33c1463..10f81123 100644 --- a/backend/analytics_server/mhq/store/models/settings/configuration_settings.py +++ b/backend/analytics_server/mhq/store/models/settings/configuration_settings.py @@ -19,6 +19,12 @@ class SettingType(Enum): DEFAULT_SYNC_DAYS_SETTING = "DEFAULT_SYNC_DAYS_SETTING" INCIDENT_PRS_SETTING = "INCIDENT_PRS_SETTING" + # CLUSTOX: per-team DORA targets. Stored here rather than in a new table + # because Settings already has the entity scoping, service and API this + # needs, and its JSONB data column means adding a fifth metric later costs + # no migration. + BENCHMARK_SETTING = "BENCHMARK_SETTING" + # ADD NEW SETTING TYPE ENUM HERE diff --git a/backend/analytics_server/mhq/store/models/settings/enums.py b/backend/analytics_server/mhq/store/models/settings/enums.py index caa351eb..197f66e4 100644 --- a/backend/analytics_server/mhq/store/models/settings/enums.py +++ b/backend/analytics_server/mhq/store/models/settings/enums.py @@ -5,3 +5,7 @@ class EntityType(Enum): USER = "USER" TEAM = "TEAM" ORG = "ORG" + # CLUSTOX: the superadmin's baseline benchmark belongs to no team and no + # workspace. entity_type is character varying with no native enum, so this + # needs no migration. + GLOBAL = "GLOBAL" diff --git a/backend/analytics_server/tests/service/settings/__init__.py b/backend/analytics_server/tests/service/settings/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/backend/analytics_server/tests/service/settings/test_benchmark_setting_type.py b/backend/analytics_server/tests/service/settings/test_benchmark_setting_type.py new file mode 100644 index 00000000..2414b1e7 --- /dev/null +++ b/backend/analytics_server/tests/service/settings/test_benchmark_setting_type.py @@ -0,0 +1,39 @@ +from mhq.service.settings.default_settings_data import get_default_setting_data +from mhq.service.settings.models import BenchmarkSetting +from mhq.service.settings.setting_type_validator import settings_type_validator +from mhq.store.models.settings import SettingType +from mhq.store.models.settings.enums import EntityType + + +def test_benchmark_setting_type_exists(): + assert SettingType.BENCHMARK_SETTING.value == "BENCHMARK_SETTING" + + +def test_global_entity_type_exists(): + # The superadmin baseline belongs to no team and no workspace. + assert EntityType.GLOBAL.value == "GLOBAL" + + +def test_validator_accepts_the_new_type(): + assert settings_type_validator("BENCHMARK_SETTING") == SettingType.BENCHMARK_SETTING + + +def test_default_baseline_has_all_four_metrics(): + data = get_default_setting_data(SettingType.BENCHMARK_SETTING) + + assert set(data.keys()) == { + "lead_time", + "deployment_frequency", + "change_failure_rate", + "mean_time_to_recovery", + } + + +def test_benchmark_setting_fields_default_to_none(): + # Every field is optional: a team may set one target and inherit the rest. + setting = BenchmarkSetting() + + assert setting.lead_time is None + assert setting.deployment_frequency is None + assert setting.change_failure_rate is None + assert setting.mean_time_to_recovery is None From a2e11272ea09f07b87c356d121d765142d1e8669 Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Tue, 11 Aug 2026 21:45:25 +0500 Subject: [PATCH 095/194] feat(benchmarks): resolve team targets against the global baseline --- .../mhq/service/settings/benchmarks.py | 73 +++++++++++++++ .../tests/service/settings/test_benchmarks.py | 90 +++++++++++++++++++ 2 files changed, 163 insertions(+) create mode 100644 backend/analytics_server/mhq/service/settings/benchmarks.py create mode 100644 backend/analytics_server/tests/service/settings/test_benchmarks.py diff --git a/backend/analytics_server/mhq/service/settings/benchmarks.py b/backend/analytics_server/mhq/service/settings/benchmarks.py new file mode 100644 index 00000000..62742e1a --- /dev/null +++ b/backend/analytics_server/mhq/service/settings/benchmarks.py @@ -0,0 +1,73 @@ +# CLUSTOX: benchmark resolution and validation. +# +# Resolution lives in the backend rather than the browser for three reasons: +# every consumer would otherwise reimplement the fallback and eventually +# disagree; the response can report WHICH benchmark applied, so an admin who +# thinks they set a target can see that they did not; and a fifth metric later +# is one more key here rather than one more reimplementation. +from typing import Dict, List, Optional + +from werkzeug.exceptions import BadRequest + +from mhq.service.settings.models import BenchmarkSetting + +# The global baseline belongs to no team and no workspace, but Settings. +# entity_id is NOT NULL. A fixed sentinel is the documented cost of reusing +# that table instead of adding one. +GLOBAL_BENCHMARK_ENTITY_ID = "00000000-0000-4000-8000-000000000001" + +BENCHMARK_METRICS: List[str] = [ + "lead_time", + "deployment_frequency", + "change_failure_rate", + "mean_time_to_recovery", +] + +# Percent, so it has an upper bound the others do not. +_BOUNDED_METRICS = {"change_failure_rate": 100} + + +def resolve_benchmarks( + team_setting: Optional[BenchmarkSetting], + global_setting: Optional[BenchmarkSetting], +) -> Dict[str, Dict]: + resolved = {} + + for metric in BENCHMARK_METRICS: + team_value = getattr(team_setting, metric, None) if team_setting else None + global_value = getattr(global_setting, metric, None) if global_setting else None + + # `is not None` rather than truthiness: 0 is a deliberate target. + if team_value is not None: + resolved[metric] = {"target": team_value, "source": "team"} + elif global_value is not None: + resolved[metric] = {"target": global_value, "source": "global"} + else: + resolved[metric] = {"target": None, "source": None} + + return resolved + + +def validate_benchmark_payload(data: Dict) -> Dict: + unknown = set(data.keys()) - set(BENCHMARK_METRICS) + if unknown: + raise BadRequest(f"Unknown benchmark metrics: {', '.join(sorted(unknown))}") + + cleaned = {} + for metric, value in data.items(): + if value is None: + continue + + if not isinstance(value, (int, float)) or isinstance(value, bool): + raise BadRequest(f"{metric} must be a number") + + if value < 0: + raise BadRequest(f"{metric} must not be negative") + + upper = _BOUNDED_METRICS.get(metric) + if upper is not None and value > upper: + raise BadRequest(f"{metric} must be between 0 and {upper}") + + cleaned[metric] = value + + return cleaned diff --git a/backend/analytics_server/tests/service/settings/test_benchmarks.py b/backend/analytics_server/tests/service/settings/test_benchmarks.py new file mode 100644 index 00000000..2e4e37d5 --- /dev/null +++ b/backend/analytics_server/tests/service/settings/test_benchmarks.py @@ -0,0 +1,90 @@ +import pytest +from werkzeug.exceptions import BadRequest + +from mhq.service.settings.benchmarks import ( + BENCHMARK_METRICS, + resolve_benchmarks, + validate_benchmark_payload, +) +from mhq.service.settings.models import BenchmarkSetting + + +def test_team_value_wins_and_reports_its_source(): + team = BenchmarkSetting(lead_time=3600) + glob = BenchmarkSetting(lead_time=86400) + + resolved = resolve_benchmarks(team, glob) + + assert resolved["lead_time"] == {"target": 3600, "source": "team"} + + +def test_absent_team_value_falls_back_to_global(): + team = BenchmarkSetting(lead_time=3600) + glob = BenchmarkSetting(lead_time=86400, deployment_frequency=5) + + resolved = resolve_benchmarks(team, glob) + + assert resolved["deployment_frequency"] == {"target": 5, "source": "global"} + + +def test_neither_set_yields_no_target(): + resolved = resolve_benchmarks(BenchmarkSetting(), BenchmarkSetting()) + + assert resolved["change_failure_rate"] == {"target": None, "source": None} + + +def test_metrics_mix_sources_within_one_response(): + # The case that makes per-metric fallback worth having. + team = BenchmarkSetting(lead_time=3600) + glob = BenchmarkSetting(lead_time=86400, deployment_frequency=5) + + resolved = resolve_benchmarks(team, glob) + + assert resolved["lead_time"]["source"] == "team" + assert resolved["deployment_frequency"]["source"] == "global" + assert resolved["mean_time_to_recovery"]["source"] is None + + +def test_zero_is_a_real_target_not_an_absent_one(): + # A team targeting zero failures must not silently inherit 15%. + team = BenchmarkSetting(change_failure_rate=0) + glob = BenchmarkSetting(change_failure_rate=15) + + resolved = resolve_benchmarks(team, glob) + + assert resolved["change_failure_rate"] == {"target": 0, "source": "team"} + + +def test_missing_settings_are_treated_as_empty(): + resolved = resolve_benchmarks(None, None) + + assert set(resolved.keys()) == set(BENCHMARK_METRICS) + assert all(v["target"] is None for v in resolved.values()) + + +def test_validation_rejects_negatives(): + with pytest.raises(BadRequest): + validate_benchmark_payload({"lead_time": -1}) + + +def test_validation_rejects_change_failure_rate_above_100(): + with pytest.raises(BadRequest): + validate_benchmark_payload({"change_failure_rate": 101}) + + +def test_validation_rejects_unknown_keys(): + # A typo would otherwise store silently and inherit forever. + with pytest.raises(BadRequest): + validate_benchmark_payload({"leadtime": 3600}) + + +def test_validation_keeps_zero_and_omits_nothing_else(): + cleaned = validate_benchmark_payload({"change_failure_rate": 0}) + + assert cleaned == {"change_failure_rate": 0} + + +def test_validation_accepts_a_partial_payload(): + cleaned = validate_benchmark_payload({"lead_time": 3600}) + + assert cleaned == {"lead_time": 3600} From 301908cb9dd1c98db71c33b963eac0fa55fa5997 Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Tue, 11 Aug 2026 21:51:07 +0500 Subject: [PATCH 096/194] feat(benchmarks): return resolved targets with team metrics --- .../mhq/api/deployment_analytics.py | 27 ++++++++++++++++++ .../tests/service/settings/test_benchmarks.py | 28 +++++++++++++++++++ 2 files changed, 55 insertions(+) diff --git a/backend/analytics_server/mhq/api/deployment_analytics.py b/backend/analytics_server/mhq/api/deployment_analytics.py index 0d7d9114..e5545a80 100644 --- a/backend/analytics_server/mhq/api/deployment_analytics.py +++ b/backend/analytics_server/mhq/api/deployment_analytics.py @@ -35,6 +35,32 @@ app = Blueprint("deployment_analytics", __name__) +# CLUSTOX: resolve a team's benchmarks for the metrics response. Injectable +# settings_service so the resolution order can be tested without a database. +def get_resolved_benchmarks_for_team(team_id: str, settings_service=None): + from mhq.service.settings.benchmarks import ( + GLOBAL_BENCHMARK_ENTITY_ID, + resolve_benchmarks, + ) + from mhq.service.settings.configuration_settings import get_settings_service + from mhq.store.models.settings import SettingType + from mhq.store.models.settings.enums import EntityType + + settings_service = settings_service or get_settings_service() + + team_setting = settings_service.get_settings( + SettingType.BENCHMARK_SETTING, EntityType.TEAM, team_id + ) + global_setting = settings_service.get_settings( + SettingType.BENCHMARK_SETTING, EntityType.GLOBAL, GLOBAL_BENCHMARK_ENTITY_ID + ) + + return resolve_benchmarks( + getattr(team_setting, "specific_settings", None), + getattr(global_setting, "specific_settings", None), + ) + + @app.route("/teams//deployment_analytics", methods={"GET"}) @queryschema( Schema( @@ -97,6 +123,7 @@ def get_team_deployment_analytics( } for repo in org_repos }, + "benchmarks": get_resolved_benchmarks_for_team(team_id), } diff --git a/backend/analytics_server/tests/service/settings/test_benchmarks.py b/backend/analytics_server/tests/service/settings/test_benchmarks.py index 2e4e37d5..708774af 100644 --- a/backend/analytics_server/tests/service/settings/test_benchmarks.py +++ b/backend/analytics_server/tests/service/settings/test_benchmarks.py @@ -88,3 +88,31 @@ def test_validation_accepts_a_partial_payload(): cleaned = validate_benchmark_payload({"lead_time": 3600}) assert cleaned == {"lead_time": 3600} + + +def test_resolution_asks_for_the_team_row_and_the_global_row(): + from mhq.service.settings.benchmarks import GLOBAL_BENCHMARK_ENTITY_ID + from mhq.store.models.settings import SettingType + from mhq.store.models.settings.enums import EntityType + + asked = [] + + class FakeSettingsService: + def get_settings(self, setting_type, entity_type, entity_id): + asked.append((setting_type, entity_type, entity_id)) + return None + + from mhq.api.deployment_analytics import get_resolved_benchmarks_for_team + + get_resolved_benchmarks_for_team("team-1", settings_service=FakeSettingsService()) + + assert ( + SettingType.BENCHMARK_SETTING, + EntityType.TEAM, + "team-1", + ) in asked + assert ( + SettingType.BENCHMARK_SETTING, + EntityType.GLOBAL, + GLOBAL_BENCHMARK_ENTITY_ID, + ) in asked From 9e8055e185fea352a3d58b2434ee13ae10403a3d Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Tue, 11 Aug 2026 21:52:30 +0500 Subject: [PATCH 097/194] fix: point invite and share links at the public domain Both links were built from NEXTAUTH_URL, which the production .env still sets to env.example's default of http://localhost:3333. Invite emails therefore went out with a link only the server itself could open. Nothing caught it because every other reader of NEXTAUTH_URL is same-origin -- the app works fine with a wrong value, and only links that leave the box break. Hardcoded for now. Setting NEXTAUTH_URL to the public domain on the server is still the real fix (it also governs NextAuth's own callback and redirect URLs, which this does not touch); once that is done these constants can go back to reading it. Note that locally generated invite links now also point at production. The token stays valid, so swap the host by hand when testing on localhost. Co-Authored-By: Claude Opus 5 --- web-server/pages/api/clustox/invites/index.ts | 12 +++++++++++- web-server/pages/api/resources/share.ts | 9 ++++++++- 2 files changed, 19 insertions(+), 2 deletions(-) diff --git a/web-server/pages/api/clustox/invites/index.ts b/web-server/pages/api/clustox/invites/index.ts index f0108f3f..01e2fd2b 100644 --- a/web-server/pages/api/clustox/invites/index.ts +++ b/web-server/pages/api/clustox/invites/index.ts @@ -14,6 +14,16 @@ import { Errors, ResponseError } from '@/constants/error'; import { db } from '@/utils/db'; import { sendInviteEmail } from '@/utils/mailer'; +// CLUSTOX: hardcoded on purpose. This was built from NEXTAUTH_URL, which the +// server's .env still sets to http://localhost:3333 -- so every invite email +// carried a link only the server itself could open. Nothing else caught it, +// because every other reader of that variable is same-origin. +// +// Set NEXTAUTH_URL to the public domain on the server and this can go back to +// reading it. Note that local invites now also point at production; the token +// is still valid, so swap the host by hand when testing on localhost. +const APP_URL = 'https://middleware.theclustox.com'; + const postSchema = yup.object().shape({ name: yup.string().required(), email: yup.string().email().required(), @@ -45,7 +55,7 @@ endpoint.handle.POST(postSchema, async (req, res) => { createdBy: session.userId }); - const inviteUrl = `${process.env.NEXTAUTH_URL}/accept-invite?token=${token}`; + const inviteUrl = `${APP_URL}/accept-invite?token=${token}`; // Adopting an existing workspace is the only case where a name is known // up front -- a brand-new one is created (and named) on acceptance, so diff --git a/web-server/pages/api/resources/share.ts b/web-server/pages/api/resources/share.ts index 8bb882bd..c97ea7f8 100644 --- a/web-server/pages/api/resources/share.ts +++ b/web-server/pages/api/resources/share.ts @@ -5,6 +5,13 @@ import { Endpoint, nullSchema } from '@/api-helpers/global'; import { Columns, Table } from '@/constants/db'; import { db } from '@/utils/db'; +// CLUSTOX: hardcoded for the same reason as the invite link -- this was built +// from NEXTAUTH_URL, which the server's .env still sets to localhost, so a +// share link generated server-side pointed at a host the recipient cannot +// reach. Only used when the client does not send its own base_url. +// See pages/api/clustox/invites/index.ts. +const APP_URL = 'https://middleware.theclustox.com'; + const getSchema = yup.object().shape({ share_link_id: yup.string().required() }); @@ -55,7 +62,7 @@ endpoint.handle.POST(postSchema, async (req, res) => { response.url = shareLink; res.send(response); } - shareLink = `${process.env.NEXTAUTH_URL}?share=${id}`; + shareLink = `${APP_URL}?share=${id}`; response.url = shareLink; res.send(response); } catch (e) { From c59e79ec7f186906bc586d6d8dbecdc22e359ea5 Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Tue, 11 Aug 2026 21:56:26 +0500 Subject: [PATCH 098/194] feat(benchmarks): expose the superadmin baseline --- backend/analytics_server/mhq/api/settings.py | 57 +++++++++++++++++++ .../pages/api/clustox/benchmarks/global.ts | 38 +++++++++++++ 2 files changed, 95 insertions(+) create mode 100644 web-server/pages/api/clustox/benchmarks/global.ts diff --git a/backend/analytics_server/mhq/api/settings.py b/backend/analytics_server/mhq/api/settings.py index b0015f78..9afe64ce 100644 --- a/backend/analytics_server/mhq/api/settings.py +++ b/backend/analytics_server/mhq/api/settings.py @@ -8,6 +8,10 @@ from mhq.api.resources.settings_resource import adapt_configuration_settings_response from mhq.service.query_validator import get_query_validator from mhq.service.settings import get_settings_service, settings_type_validator +from mhq.service.settings.benchmarks import ( + GLOBAL_BENCHMARK_ENTITY_ID, + validate_benchmark_payload, +) from mhq.store.models import SettingType, EntityType app = Blueprint("settings", __name__) @@ -170,3 +174,56 @@ def put_org_settings( setting_data=setting_data, ) return adapt_configuration_settings_response(settings) + + +# CLUSTOX: the superadmin's baseline belongs to no team and no workspace, so +# it cannot use the team- or org-scoped routes above. Authorisation is enforced +# at the BFF -- this layer only knows the internal token, not who is calling. +@app.route("/settings/global", methods={"GET"}) +@queryschema( + Schema( + { + Required("setting_type"): All(str, Coerce(settings_type_validator)), + } + ), +) +def get_global_settings(setting_type: SettingType): + + settings_service = get_settings_service() + settings = settings_service.get_settings( + setting_type=setting_type, + entity_type=EntityType.GLOBAL, + entity_id=GLOBAL_BENCHMARK_ENTITY_ID, + ) + + if not settings: + settings = settings_service.save_settings( + setting_type=setting_type, + entity_type=EntityType.GLOBAL, + entity_id=GLOBAL_BENCHMARK_ENTITY_ID, + ) + + return adapt_configuration_settings_response(settings) + + +@app.route("/settings/global", methods={"PUT"}) +@dataschema( + Schema( + { + Required("setting_type"): All(str, Coerce(settings_type_validator)), + Required("setting_data"): dict, + } + ), +) +def put_global_settings(setting_type: SettingType, setting_data: Dict = None): + + setting_data = validate_benchmark_payload(setting_data) + + settings_service = get_settings_service() + settings = settings_service.save_settings( + setting_type=setting_type, + entity_type=EntityType.GLOBAL, + entity_id=GLOBAL_BENCHMARK_ENTITY_ID, + setting_data=setting_data, + ) + return adapt_configuration_settings_response(settings) diff --git a/web-server/pages/api/clustox/benchmarks/global.ts b/web-server/pages/api/clustox/benchmarks/global.ts new file mode 100644 index 00000000..d1b047f3 --- /dev/null +++ b/web-server/pages/api/clustox/benchmarks/global.ts @@ -0,0 +1,38 @@ +import * as yup from 'yup'; + +import { Endpoint, nullSchema } from '@/api-helpers/global'; +import { internal } from '@/api-helpers/axios'; +import { assertRole } from '@/auth/guard'; + +// CLUSTOX: the global baseline is superadmin-only. The Flask layer trusts the +// internal token and has no notion of who is calling, so the role check has to +// live here. +const putSchema = yup.object().shape({ + lead_time: yup.number().min(0).nullable().optional(), + deployment_frequency: yup.number().min(0).nullable().optional(), + change_failure_rate: yup.number().min(0).max(100).nullable().optional(), + mean_time_to_recovery: yup.number().min(0).nullable().optional() +}); + +const endpoint = new Endpoint(nullSchema); + +endpoint.handle.GET(nullSchema, async (req, res) => { + assertRole((req as any).session, 'SUPERADMIN'); + + const result = await internal.get('/settings/global', { + params: { setting_type: 'BENCHMARK_SETTING' } + }); + res.send(result.data); +}); + +endpoint.handle.PUT(putSchema, async (req, res) => { + assertRole((req as any).session, 'SUPERADMIN'); + + const result = await internal.put('/settings/global', { + setting_type: 'BENCHMARK_SETTING', + setting_data: req.payload + }); + res.send(result.data); +}); + +export default endpoint.serve(); From f996a76805d13ccd19308dfd1211a8113656c581 Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Tue, 11 Aug 2026 21:59:37 +0500 Subject: [PATCH 099/194] docs: the settings extension has eleven points, not ten mhq/api/resources/settings_resource.py carries its own ADD NEW API ADAPTER HERE marker. The search that found the other ten covered only mhq/service/settings/ and mhq/store/models/settings/, so it never reached it. Without that branch, adapt_configuration_settings_response never populates response['setting'] for a new setting type, so a GET returns only created_at and updated_at. Nothing errors -- it is invisible until something tries to read a saved value, which for this feature is the configuration form two tasks later. Found by the Task 4 implementer, who flagged it as out of scope for the file list it was given. The file list was wrong. --- docs/BENCHMARKS_PLAN.md | 18 +++++++++++++++--- 1 file changed, 15 insertions(+), 3 deletions(-) diff --git a/docs/BENCHMARKS_PLAN.md b/docs/BENCHMARKS_PLAN.md index 888ed5f2..79657f59 100644 --- a/docs/BENCHMARKS_PLAN.md +++ b/docs/BENCHMARKS_PLAN.md @@ -21,9 +21,9 @@ - Frontend must be `tsc` clean. Local Node is 20 and the project needs 22 — verify inside the `middleware-dev` container. - Every Clustox-authored change carries a `# CLUSTOX:` or `// CLUSTOX:` comment explaining **why**, per `docs/FORK_STRATEGY.md`. -## The ten extension points — read before Task 1 +## The eleven extension points — read before Task 1 -Adding a `SettingType` to this codebase means touching ten places. Upstream +Adding a `SettingType` to this codebase means touching eleven places. Upstream marks each with a comment. **Missing one is a silent failure, not a crash** — the setting saves but reads back wrong, or validates but never persists. @@ -39,9 +39,21 @@ the setting saves but reads back wrong, or validates but never persists. | 8 | `mhq/service/settings/configuration_settings.py:272` | `ADD NEW DATACLASS TO JSON DATA ADAPTERS HERE` | | 9 | `mhq/service/settings/configuration_settings.py:308` | `ADD NEW HANDLE TO DB SETTINGS HERE` | | 10 | `mhq/service/settings/setting_type_validator.py:25` | `ADD NEW VALIDATOR HERE` | +| 11 | `mhq/api/resources/settings_resource.py` | `ADD NEW API ADAPTER HERE` | Line numbers drift — search for the marker text, not the number. +**Point 11 was missed when this plan was first written**, because the search +that found the others covered only `mhq/service/settings/` and +`mhq/store/models/settings/`. Without it, `adapt_configuration_settings_response` +never populates `response["setting"]` for a benchmark, so `GET` returns only +timestamps — invisible until something tries to read a saved value. Search the +whole of `mhq/` for the marker text, not a subtree: + +``` +grep -rn "ADD NEW" mhq/ +``` + --- ## File Structure @@ -73,7 +85,7 @@ Line numbers drift — search for the marker text, not the number. --- -### Task 1: Register BENCHMARK_SETTING across all ten extension points +### Task 1: Register BENCHMARK_SETTING across the extension points **Files:** - Modify: `backend/analytics_server/mhq/store/models/settings/configuration_settings.py` From 84716f033fd935f1bf6ec5817aa08664da87fe42 Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Tue, 11 Aug 2026 22:00:30 +0500 Subject: [PATCH 100/194] fix(benchmarks): populate the setting body and scope marker for benchmark responses --- .../mhq/api/resources/settings_resource.py | 20 ++++++++ .../tests/service/settings/test_benchmarks.py | 49 ++++++++++++++++++- 2 files changed, 68 insertions(+), 1 deletion(-) diff --git a/backend/analytics_server/mhq/api/resources/settings_resource.py b/backend/analytics_server/mhq/api/resources/settings_resource.py index 9e95e693..a1da4aad 100644 --- a/backend/analytics_server/mhq/api/resources/settings_resource.py +++ b/backend/analytics_server/mhq/api/resources/settings_resource.py @@ -6,6 +6,7 @@ IncidentTypesSetting, IncidentSourcesSetting, IncidentPRsSetting, + BenchmarkSetting, ) from mhq.store.models import EntityType @@ -22,6 +23,13 @@ def _add_entity(config_settings: ConfigurationSettings, response): if config_settings.entity_type == EntityType.ORG: response["org_id"] = str(config_settings.entity_id) + # CLUSTOX: the global row's entity_id is a fixed sentinel, not a real + # entity -- echoing it back as an id invites a caller to treat it as + # one (e.g. look it up as a team). A "scope" marker says what the row + # is without implying it resolves to anything. + if config_settings.entity_type == EntityType.GLOBAL: + response["scope"] = "global" + return response def _add_setting_data(config_settings: ConfigurationSettings, response): @@ -62,6 +70,18 @@ def _add_setting_data(config_settings: ConfigurationSettings, response): "filters": config_settings.specific_settings.filters, } + if isinstance(config_settings.specific_settings, BenchmarkSetting): + # CLUSTOX: None means "not set at this scope, inherit" and must + # round-trip as null, not be dropped or coerced to 0 -- the + # config form distinguishes "no target", "target of 0" and + # "target inherited from the global baseline". + response["setting"] = { + "lead_time": config_settings.specific_settings.lead_time, + "deployment_frequency": config_settings.specific_settings.deployment_frequency, + "change_failure_rate": config_settings.specific_settings.change_failure_rate, + "mean_time_to_recovery": config_settings.specific_settings.mean_time_to_recovery, + } + # ADD NEW API ADAPTER HERE return response diff --git a/backend/analytics_server/tests/service/settings/test_benchmarks.py b/backend/analytics_server/tests/service/settings/test_benchmarks.py index 708774af..f2ca107b 100644 --- a/backend/analytics_server/tests/service/settings/test_benchmarks.py +++ b/backend/analytics_server/tests/service/settings/test_benchmarks.py @@ -1,12 +1,16 @@ +from datetime import datetime + import pytest from werkzeug.exceptions import BadRequest +from mhq.api.resources.settings_resource import adapt_configuration_settings_response from mhq.service.settings.benchmarks import ( BENCHMARK_METRICS, resolve_benchmarks, validate_benchmark_payload, ) -from mhq.service.settings.models import BenchmarkSetting +from mhq.service.settings.models import BenchmarkSetting, ConfigurationSettings +from mhq.store.models.settings.enums import EntityType def test_team_value_wins_and_reports_its_source(): @@ -116,3 +120,46 @@ def get_settings(self, setting_type, entity_type, entity_id): EntityType.GLOBAL, GLOBAL_BENCHMARK_ENTITY_ID, ) in asked + + +def test_benchmark_setting_response_always_has_all_four_keys(): + # Set, zero, and unset must all round-trip distinctly: a dropped or + # zero-coerced key would make the config form unable to tell "inherit" + # apart from "target of 0". + config_settings = ConfigurationSettings( + entity_id="team-1", + entity_type=EntityType.TEAM, + specific_settings=BenchmarkSetting(lead_time=3600, change_failure_rate=0), + updated_by=None, + created_at=datetime(2026, 1, 1), + updated_at=datetime(2026, 1, 1), + ) + + response = adapt_configuration_settings_response(config_settings) + + assert response["setting"] == { + "lead_time": 3600, + "deployment_frequency": None, + "change_failure_rate": 0, + "mean_time_to_recovery": None, + } + + +def test_global_benchmark_row_reports_scope_not_the_sentinel_id(): + from mhq.service.settings.benchmarks import GLOBAL_BENCHMARK_ENTITY_ID + + config_settings = ConfigurationSettings( + entity_id=GLOBAL_BENCHMARK_ENTITY_ID, + entity_type=EntityType.GLOBAL, + specific_settings=BenchmarkSetting(lead_time=3600), + updated_by=None, + created_at=datetime(2026, 1, 1), + updated_at=datetime(2026, 1, 1), + ) + + response = adapt_configuration_settings_response(config_settings) + + assert response["scope"] == "global" + assert "team_id" not in response + assert "org_id" not in response + assert "user_id" not in response From 2c7e65fe299db6c990cafb6d66d227c7a938b245 Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Wed, 12 Aug 2026 00:04:17 +0500 Subject: [PATCH 101/194] feat(benchmarks): draw the target on each metric card Adds benchmarkCaption() and BenchmarkTargetLine to compute and render a per-metric target line + caption on all four DORA cards (lead time, deployment frequency, change failure rate, MTTR). Direction is per metric: lower is better for lead time/CFR/MTTR, higher is better for deployment frequency. Tone is good/warn only, never an error color -- a missed internal target isn't a system failure. Renders nothing when a metric has no configured target, so existing cards are unaffected until benchmarks are set. --- .../src/components/BenchmarkTargetLine.tsx | 42 +++++++ .../DoraCards/ChangeFailureRateCard.tsx | 49 ++++++++ .../DoraMetrics/DoraCards/ChangeTimeCard.tsx | 42 +++++++ .../DoraCards/MeanTimeToRestoreCard.tsx | 42 +++++++ .../DoraCards/WeeklyDeliveryVolumeCard.tsx | 59 +++++++++ web-server/src/types/resources.ts | 6 + .../src/utils/__tests__/benchmarks.test.ts | 29 +++++ web-server/src/utils/benchmarks.ts | 119 ++++++++++++++++++ 8 files changed, 388 insertions(+) create mode 100644 web-server/src/components/BenchmarkTargetLine.tsx create mode 100644 web-server/src/utils/__tests__/benchmarks.test.ts create mode 100644 web-server/src/utils/benchmarks.ts diff --git a/web-server/src/components/BenchmarkTargetLine.tsx b/web-server/src/components/BenchmarkTargetLine.tsx new file mode 100644 index 00000000..e2a096f4 --- /dev/null +++ b/web-server/src/components/BenchmarkTargetLine.tsx @@ -0,0 +1,42 @@ +import { Box } from '@mui/material'; +import { FC } from 'react'; + +type BenchmarkTargetLineProps = { + /** The resolved benchmark target, in the same unit as `values`. */ + target: number; + // CLUSTOX: the same array the card's own trend chart plots. Chart2's grid + // charts always `beginAtZero` and add `grace: '10%'` headroom on top of + // the max value (see InternalChart2.tsx) -- mirroring that here is what + // makes the dashed line land at the same height on the card as it would + // inside the chart's own y-scale, instead of floating at an arbitrary + // fixed position that doesn't track the data. + values?: number[]; +}; + +// CLUSTOX: dashed horizontal rule marking the benchmark target on a card's +// trend chart. Deliberately just a positioned line, not a chart.js +// annotation -- the four cards already hide their y-axis (`display: false`), +// so there's no on-chart scale for a plugin-drawn line to key off either; +// the exact number lives in the caption text next to it. +export const BenchmarkTargetLine: FC = ({ + target, + values = [] +}) => { + const max = Math.max(target, 0, ...values) * 1.1 || 1; + const clampedTarget = Math.min(Math.max(target, 0), max); + const fromBottomPct = (clampedTarget / max) * 100; + + return ( + + ); +}; diff --git a/web-server/src/content/DoraMetrics/DoraCards/ChangeFailureRateCard.tsx b/web-server/src/content/DoraMetrics/DoraCards/ChangeFailureRateCard.tsx index 113bf3ec..0aea8190 100644 --- a/web-server/src/content/DoraMetrics/DoraCards/ChangeFailureRateCard.tsx +++ b/web-server/src/content/DoraMetrics/DoraCards/ChangeFailureRateCard.tsx @@ -3,6 +3,7 @@ import pluralize from 'pluralize'; import { head } from 'ramda'; import { useMemo } from 'react'; +import { BenchmarkTargetLine } from '@/components/BenchmarkTargetLine'; import { Chart2, ChartOptions } from '@/components/Chart2'; import { useSelectedContributors } from '@/components/ContributorFilter'; import { FlexBox } from '@/components/FlexBox'; @@ -22,6 +23,7 @@ import { } from '@/hooks/useStateTeamConfig'; import { useSelector } from '@/store'; import { IntegrationGroup } from '@/types/resources'; +import { benchmarkCaption } from '@/utils/benchmarks'; import { NoIncidentsLabel } from './NoIncidentsLabel'; import { useChangeFailureRateProps } from './sharedHooks'; @@ -85,6 +87,10 @@ export const ChangeFailureRateCard = () => { const changeFailureRateCount = useCountUp(changeFailureRateProps.count || 0); + const changeFailureRateBenchmark = useSelector( + (s) => s.doraMetrics.metrics_summary?.benchmarks?.change_failure_rate + ); + const series = useMemo( () => [ { @@ -109,6 +115,27 @@ export const ChangeFailureRateCard = () => { (changeFailureRateProps.count || prevChangeFailureRate) ); + // CLUSTOX: gated on the same canShowIncidentsData/isCfrDataAvailable pair + // the card already uses to decide whether real data (vs NoIncidentsLabel + // or a missing-provider link) is on screen. + const changeFailureRateBenchmarkCaption = useMemo( + () => + canShowIncidentsData && isCfrDataAvailable && changeFailureRateBenchmark + ? benchmarkCaption( + 'change_failure_rate', + changeFailureRateProps.count, + changeFailureRateBenchmark.target, + changeFailureRateBenchmark.source + ) + : null, + [ + canShowIncidentsData, + isCfrDataAvailable, + changeFailureRateBenchmark, + changeFailureRateProps.count + ] + ); + const { addPage } = useOverlayPage(); return ( { team-wide — per-contributor arrives with Jira )} + {changeFailureRateBenchmarkCaption && ( + + {changeFailureRateBenchmarkCaption.text} + + )} {canShowIncidentsData ? ( @@ -194,6 +235,14 @@ export const ChangeFailureRateCard = () => { )} + {canShowIncidentsData && + isCfrDataAvailable && + changeFailureRateBenchmark?.target != null && ( + + )} {canShowIncidentsData ? ( diff --git a/web-server/src/content/DoraMetrics/DoraCards/ChangeTimeCard.tsx b/web-server/src/content/DoraMetrics/DoraCards/ChangeTimeCard.tsx index 7308d7b1..5474fe3b 100644 --- a/web-server/src/content/DoraMetrics/DoraCards/ChangeTimeCard.tsx +++ b/web-server/src/content/DoraMetrics/DoraCards/ChangeTimeCard.tsx @@ -6,6 +6,7 @@ import Link from 'next/link'; import pluralize from 'pluralize'; import { useMemo } from 'react'; +import { BenchmarkTargetLine } from '@/components/BenchmarkTargetLine'; import { Chart2, ChartOptions } from '@/components/Chart2'; import { useSelectedContributors } from '@/components/ContributorFilter'; import { FlexBox } from '@/components/FlexBox'; @@ -23,6 +24,7 @@ import { useAuth } from '@/hooks/useAuth'; import { useCountUp } from '@/hooks/useCountUp'; import { useSelector } from '@/store'; import { ChangeTimeModes } from '@/types/resources'; +import { benchmarkCaption } from '@/utils/benchmarks'; import { merge } from '@/utils/datatype'; import { getDurationString, getSortedDatesAsArrayFromMap } from '@/utils/date'; @@ -77,6 +79,10 @@ export const ChangeTimeCard = () => { ] ); + const leadTimeBenchmark = useSelector( + (s) => s.doraMetrics.metrics_summary?.benchmarks?.lead_time + ); + const isCodeProviderIntegrationEnabled = true; const showClassificationBadge = @@ -105,6 +111,23 @@ export const ChangeTimeCard = () => { const leadTimeDuration = useCountUp(activeModeProps.count || 0); + // CLUSTOX: gated on isSufficientDataAvailable -- same guard the card + // already uses to decide between the chart and "Insufficient data". A + // target line/caption over a placeholder state would compare a real + // benchmark against a meaningless zero. + const leadTimeBenchmarkCaption = useMemo( + () => + isSufficientDataAvailable && leadTimeBenchmark + ? benchmarkCaption( + 'lead_time', + activeModeProps.count, + leadTimeBenchmark.target, + leadTimeBenchmark.source + ) + : null, + [isSufficientDataAvailable, leadTimeBenchmark, activeModeProps.count] + ); + return ( { @@ -281,6 +304,18 @@ export const ChangeTimeCard = () => { authored by {selectedContributors.join(', ')} )} + {leadTimeBenchmarkCaption && ( + + {leadTimeBenchmarkCaption.text} + + )} {isSufficientDataAvailable ? ( @@ -295,6 +330,13 @@ export const ChangeTimeCard = () => { )} + {isSufficientDataAvailable && leadTimeBenchmark?.target != null && ( + + )} + {isSufficientDataAvailable ? ( diff --git a/web-server/src/content/DoraMetrics/DoraCards/MeanTimeToRestoreCard.tsx b/web-server/src/content/DoraMetrics/DoraCards/MeanTimeToRestoreCard.tsx index d723c309..5e73e4b9 100644 --- a/web-server/src/content/DoraMetrics/DoraCards/MeanTimeToRestoreCard.tsx +++ b/web-server/src/content/DoraMetrics/DoraCards/MeanTimeToRestoreCard.tsx @@ -3,6 +3,7 @@ import { Chip } from '@mui/material'; import { head } from 'ramda'; import { useMemo } from 'react'; +import { BenchmarkTargetLine } from '@/components/BenchmarkTargetLine'; import { Chart2, ChartOptions } from '@/components/Chart2'; import { useSelectedContributors } from '@/components/ContributorFilter'; import { FlexBox } from '@/components/FlexBox'; @@ -15,6 +16,8 @@ import { } from '@/content/DoraMetrics/DoraCards/sharedComponents'; import { useCountUp } from '@/hooks/useCountUp'; import { useDoraMetricsGraph } from '@/hooks/useDoraMetricsGraph'; +import { useSelector } from '@/store'; +import { benchmarkCaption } from '@/utils/benchmarks'; import { getDurationString } from '@/utils/date'; import { NoIncidentsLabel } from './NoIncidentsLabel'; @@ -81,6 +84,25 @@ export const MeanTimeToRestoreCard = () => { const meanTimeToRestoreCount = useCountUp(meanTimeToRestoreProps.count || 0); + const meanTimeToRecoveryBenchmark = useSelector( + (s) => s.doraMetrics.metrics_summary?.benchmarks?.mean_time_to_recovery + ); + + // CLUSTOX: gated on canShowMTRData -- the same guard the card uses to + // decide between real data and the "No incidents reported" state. + const meanTimeToRecoveryBenchmarkCaption = useMemo( + () => + canShowMTRData && meanTimeToRecoveryBenchmark + ? benchmarkCaption( + 'mean_time_to_recovery', + meanTimeToRestoreProps.count, + meanTimeToRecoveryBenchmark.target, + meanTimeToRecoveryBenchmark.source + ) + : null, + [canShowMTRData, meanTimeToRecoveryBenchmark, meanTimeToRestoreProps.count] + ); + const { addPage } = useOverlayPage(); return ( @@ -153,6 +175,20 @@ export const MeanTimeToRestoreCard = () => { team-wide — per-contributor arrives with Jira )} + {meanTimeToRecoveryBenchmarkCaption && ( + + {meanTimeToRecoveryBenchmarkCaption.text} + + )} {canShowMTRData ? ( @@ -166,6 +202,12 @@ export const MeanTimeToRestoreCard = () => { )} + {canShowMTRData && meanTimeToRecoveryBenchmark?.target != null && ( + + )} {canShowMTRData ? ( diff --git a/web-server/src/content/DoraMetrics/DoraCards/WeeklyDeliveryVolumeCard.tsx b/web-server/src/content/DoraMetrics/DoraCards/WeeklyDeliveryVolumeCard.tsx index 2e94b5aa..4ae241ed 100644 --- a/web-server/src/content/DoraMetrics/DoraCards/WeeklyDeliveryVolumeCard.tsx +++ b/web-server/src/content/DoraMetrics/DoraCards/WeeklyDeliveryVolumeCard.tsx @@ -2,6 +2,7 @@ import { alpha, Chip } from '@mui/material'; import pluralize from 'pluralize'; import { useMemo } from 'react'; +import { BenchmarkTargetLine } from '@/components/BenchmarkTargetLine'; import { Chart2, ChartOptions } from '@/components/Chart2'; import { useSelectedContributors } from '@/components/ContributorFilter'; import { FlexBox } from '@/components/FlexBox'; @@ -19,6 +20,7 @@ import { } from '@/hooks/useStateTeamConfig'; import { useSelector } from '@/store'; import { IntegrationGroup } from '@/types/resources'; +import { benchmarkCaption } from '@/utils/benchmarks'; import { getSortedDatesAsArrayFromMap } from '@/utils/date'; import { useAvgIntervalBasedDeploymentFrequency } from './sharedHooks'; @@ -84,6 +86,21 @@ export const WeeklyDeliveryVolumeCard = () => { .total_deployments || 0 ); + const deploymentFrequencyBenchmark = useSelector( + (s) => s.doraMetrics.metrics_summary?.benchmarks?.deployment_frequency + ); + + // CLUSTOX: the benchmark target is always "deployments/week" (see + // task-5-brief), but the card's headline count adapts to the selected + // interval (day/week/month) via useAvgIntervalBasedDeploymentFrequency. + // Comparing against that would silently mix units, so the weekly-specific + // stat is used here instead, regardless of which interval is displayed. + const avgWeeklyDeploymentFrequency = useSelector( + (s) => + s.doraMetrics.metrics_summary?.deployment_frequency_stats.current + .avg_weekly_deployment_frequency + ); + const series = useMemo( () => [ { @@ -100,6 +117,27 @@ export const WeeklyDeliveryVolumeCard = () => { [deploymentFrequencyProps?.backgroundColor, weekDeliveryVolumeData] ); + // CLUSTOX: gated on isCodeProviderIntegrationEnabled -- the same guard the + // card uses to decide between the chart and NoDataImg. Zero deployments is + // still a meaningful comparison against a weekly target (it's the "below + // target" case), so no additional zero-count guard here. + const deploymentFrequencyBenchmarkCaption = useMemo( + () => + isCodeProviderIntegrationEnabled && deploymentFrequencyBenchmark + ? benchmarkCaption( + 'deployment_frequency', + avgWeeklyDeploymentFrequency || 0, + deploymentFrequencyBenchmark.target, + deploymentFrequencyBenchmark.source + ) + : null, + [ + isCodeProviderIntegrationEnabled, + deploymentFrequencyBenchmark, + avgWeeklyDeploymentFrequency + ] + ); + const { weeksCovered, daysCovered } = useStateDateConfig(); const dateRangeLabelString = `${ @@ -181,6 +219,20 @@ export const WeeklyDeliveryVolumeCard = () => { deployed by {selectedContributors.join(', ')} )} + {deploymentFrequencyBenchmarkCaption && ( + + {deploymentFrequencyBenchmarkCaption.text} + + )} {isCodeProviderIntegrationEnabled ? ( @@ -194,6 +246,13 @@ export const WeeklyDeliveryVolumeCard = () => { )} + {isCodeProviderIntegrationEnabled && + deploymentFrequencyBenchmark?.target != null && ( + + )} {isCodeProviderIntegrationEnabled ? ( = { [P in keyof T]?: T[P] extends object ? DeepPartial : T[P]; @@ -592,6 +593,11 @@ export type TeamDoraMetricsApiResponseType = { lead_time_prs: PR[]; assigned_repos: (Row<'TeamRepos'> & Row<'OrgRepo'>)[]; unsynced_repos: ID[]; + // CLUSTOX: optional -- older responses (and this BFF route today, until + // it's wired to the resolved-benchmarks helper) simply omit the key. The + // four cards must treat a missing `benchmarks` exactly like every target + // being null: no line, no caption, unchanged from before this feature. + benchmarks?: Benchmarks; }; export enum ActiveBranchMode { diff --git a/web-server/src/utils/__tests__/benchmarks.test.ts b/web-server/src/utils/__tests__/benchmarks.test.ts new file mode 100644 index 00000000..05e0a796 --- /dev/null +++ b/web-server/src/utils/__tests__/benchmarks.test.ts @@ -0,0 +1,29 @@ +import { benchmarkCaption } from '../benchmarks'; + +describe('benchmarkCaption', () => { + it('treats being under target as good for lead time', () => { + const r = benchmarkCaption('lead_time', 3600, 7200, 'team'); + expect(r.tone).toBe('good'); + expect(r.text).toContain('under target'); + }); + + it('treats being under target as bad for deployment frequency', () => { + // Higher is better here -- the same direction means the opposite thing. + const r = benchmarkCaption('deployment_frequency', 2, 5, 'team'); + expect(r.tone).toBe('warn'); + expect(r.text).toContain('below target'); + }); + + it('names the source so a team can see its setting did not save', () => { + expect(benchmarkCaption('lead_time', 3600, 7200, 'global').text).toContain( + 'default' + ); + expect(benchmarkCaption('lead_time', 3600, 7200, 'team').text).toContain( + 'team' + ); + }); + + it('returns null when there is no target', () => { + expect(benchmarkCaption('lead_time', 3600, null, null)).toBeNull(); + }); +}); diff --git a/web-server/src/utils/benchmarks.ts b/web-server/src/utils/benchmarks.ts new file mode 100644 index 00000000..e05ae534 --- /dev/null +++ b/web-server/src/utils/benchmarks.ts @@ -0,0 +1,119 @@ +// CLUSTOX: per-team DORA benchmarks -- the caption shown under a metric card +// when a target (team-set or org-wide default) exists for it. +// +// Deliberately dependency-light (no imports at all) so `benchmarkCaption` is +// unit testable in isolation. `@/utils/date` -- the obvious place to borrow a +// duration formatter from -- chains into `@/utils/mock` -> `@faker-js/faker`, +// which ships ESM-only output and blows up ts-jest with "Cannot use import +// statement outside a module" the instant it's required (verified directly: +// a probe test importing `getDurationString` fails with that exact error). +// `contributorFilters.ts` hit the same wall for the same reason; this file +// stays equally self-contained. +export type BenchmarkMetric = + | 'lead_time' + | 'deployment_frequency' + | 'change_failure_rate' + | 'mean_time_to_recovery'; + +export type BenchmarkSource = 'team' | 'global' | null; + +export type ResolvedBenchmark = { + target: number | null; + source: BenchmarkSource; +}; + +export type Benchmarks = Record; + +export type BenchmarkTone = 'good' | 'warn'; + +export type BenchmarkCaption = { + text: string; + tone: BenchmarkTone; +}; + +// CLUSTOX: direction is per metric, not global. Lead time, change failure +// rate and MTTR are all "smaller is better" -- deployment frequency is the +// one metric in the set where "smaller is better" is exactly backwards. +// Getting this list wrong silently inverts the whole feature's meaning, so +// it's the one thing under direct test coverage (see benchmarks.test.ts). +const LOWER_IS_BETTER: ReadonlySet = new Set([ + 'lead_time', + 'change_failure_rate', + 'mean_time_to_recovery' +]); + +const formatDuration = (seconds: number): string => { + const abs = Math.abs(seconds); + if (abs < 60) return `${Math.round(seconds)}s`; + if (abs < 3600) return `${Math.round(seconds / 60)}m`; + if (abs < 86400) return `${Math.round((seconds / 3600) * 10) / 10}h`; + return `${Math.round((seconds / 86400) * 10) / 10}d`; +}; + +const formatBenchmarkValue = ( + metric: BenchmarkMetric, + value: number +): string => { + switch (metric) { + case 'lead_time': + case 'mean_time_to_recovery': + return formatDuration(value); + case 'change_failure_rate': + return `${value}%`; + case 'deployment_frequency': + return `${value} ${value === 1 ? 'deployment' : 'deployments'}/week`; + } +}; + +/** + * Compares `actual` against `target` for `metric` and returns a caption + * stating the fact (over/under/above/below target) plus which side of the + * target is favourable for this particular metric. + * + * Returns `null` when there is no target to compare against -- that is the + * state every card is in before anyone configures a benchmark, and it must + * render nothing rather than a caption about a target that doesn't exist. + */ +export const benchmarkCaption = ( + metric: BenchmarkMetric, + actual: number, + target: number | null, + source: BenchmarkSource +): BenchmarkCaption | null => { + if (target === null) return null; + + const lowerIsBetter = LOWER_IS_BETTER.has(metric); + // CLUSTOX: `<=`/`>=` so hitting the target exactly reads as "good", not as + // a coin flip between the two tones. + const favourable = lowerIsBetter ? actual <= target : actual >= target; + + let direction: string; + if (actual === target) { + direction = 'at'; + } else if (lowerIsBetter) { + direction = actual < target ? 'under' : 'over'; + } else { + direction = actual > target ? 'above' : 'below'; + } + + // CLUSTOX: names which benchmark applied so a team that set a target but + // sees the org-wide default knows their setting didn't save, rather than + // silently comparing against the wrong number. + const sourceClause = + source === 'team' ? "your team's benchmark" : 'the default benchmark'; + + const text = `${formatBenchmarkValue( + metric, + actual + )} is ${direction} target (${formatBenchmarkValue( + metric, + target + )}) — ${sourceClause}`; + + return { + text, + // CLUSTOX: never 'error'/red -- a missed internal goal isn't a system + // failure, and colouring it like one makes the dashboard punitive. + tone: favourable ? 'good' : 'warn' + }; +}; From a5d6e8553960b3f88706b4cda748f639c4629b17 Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Wed, 12 Aug 2026 00:21:09 +0500 Subject: [PATCH 102/194] feat(benchmarks): add the benchmark configuration forms --- web-server/pages/workspaces.tsx | 116 ++--- .../src/components/BenchmarkSettingsForm.tsx | 411 ++++++++++++++++++ .../DoraMetricsConfigurationSettings.tsx | 25 ++ 3 files changed, 497 insertions(+), 55 deletions(-) create mode 100644 web-server/src/components/BenchmarkSettingsForm.tsx diff --git a/web-server/pages/workspaces.tsx b/web-server/pages/workspaces.tsx index f82bec84..142332a8 100644 --- a/web-server/pages/workspaces.tsx +++ b/web-server/pages/workspaces.tsx @@ -29,6 +29,7 @@ import { useSnackbar } from 'notistack'; import { useCallback, useEffect, useState } from 'react'; import ExtendedSidebarLayout from 'src/layouts/ExtendedSidebarLayout'; +import { BenchmarkSettingsForm } from '@/components/BenchmarkSettingsForm'; import { ClustoxWorkspaceMetrics } from '@/components/ClustoxWorkspaceMetrics'; import { FlexBox } from '@/components/FlexBox'; import { Line } from '@/components/Text'; @@ -82,7 +83,7 @@ function WorkspacesPage() { const [workspaces, setWorkspaces] = useState([]); const [loading, setLoading] = useState(true); const [syncing, setSyncing] = useState(false); - const [tab, setTab] = useState<'health' | 'metrics'>('health'); + const [tab, setTab] = useState<'health' | 'metrics' | 'benchmarks'>('health'); const load = useCallback(async () => { const res = await fetch('/api/clustox/workspace-status'); @@ -231,69 +232,74 @@ function WorkspacesPage() { > + )} {isSuperadmin && tab === 'metrics' && } + {isSuperadmin && tab === 'benchmarks' && ( + + )} + {(!isSuperadmin || tab === 'health') && ( - - - - - Workspace - Last sync - When - - Repos - - - Teams - - - - - {workspaces.map((w) => ( - - - - - - {w.name} - - {w.ownerEmail ?? 'no owner'} - - - - - {statusChip(w)} - - - {w.hasIntegration - ? relative(w.lastSync?.finished_at ?? null) - : '—'} - - - - {w.repoCount} + +
+ + + Workspace + Last sync + When + + Repos - - {w.teamCount} + + Teams - ))} - -
-
+ + + {workspaces.map((w) => ( + + + + + + {w.name} + + {w.ownerEmail ?? 'no owner'} + + + + + {statusChip(w)} + + + {w.hasIntegration + ? relative(w.lastSync?.finished_at ?? null) + : '—'} + + + + {w.repoCount} + + + {w.teamCount} + + + ))} + + +
)} {(!isSuperadmin || tab === 'health') && ( diff --git a/web-server/src/components/BenchmarkSettingsForm.tsx b/web-server/src/components/BenchmarkSettingsForm.tsx new file mode 100644 index 00000000..a338716a --- /dev/null +++ b/web-server/src/components/BenchmarkSettingsForm.tsx @@ -0,0 +1,411 @@ +import { LoadingButton } from '@mui/lab'; +import { + Alert, + Button, + CircularProgress, + Divider, + TextField +} from '@mui/material'; +import { useSnackbar } from 'notistack'; +import { FC, useCallback, useEffect, useMemo, useState } from 'react'; + +import { handleApi } from '@/api-helpers/axios-api-instance'; +import { useSingleTeamConfig } from '@/hooks/useStateTeamConfig'; +import { FetchTeamSettingsAPIResponse } from '@/types/resources'; +import { BenchmarkMetric } from '@/utils/benchmarks'; + +import { FlexBox } from './FlexBox'; +import { Line } from './Text'; + +// CLUSTOX: the API stores seconds for these two, but nobody thinks in +// seconds when setting a target. Hours are entered/displayed; converted at +// the read/write boundary only, so the rest of the component (validation, +// dirty-checking, payload building) never has to know which unit a metric is +// really in. +const SECONDS_PER_HOUR = 3600; +const DURATION_METRICS: ReadonlySet = new Set([ + 'lead_time', + 'mean_time_to_recovery' +]); + +const METRICS: BenchmarkMetric[] = [ + 'lead_time', + 'deployment_frequency', + 'change_failure_rate', + 'mean_time_to_recovery' +]; + +const METRIC_LABEL: Record = { + lead_time: 'Lead time', + deployment_frequency: 'Deployment frequency', + change_failure_rate: 'Change failure rate', + mean_time_to_recovery: 'Mean time to recovery' +}; + +const UNIT_LABEL: Record = { + lead_time: 'hours', + deployment_frequency: 'per week', + change_failure_rate: '%', + mean_time_to_recovery: 'hours' +}; + +type BenchmarkSettingValues = Record; +type FormValues = Record; + +const EMPTY_FORM_VALUES: FormValues = { + lead_time: '', + deployment_frequency: '', + change_failure_rate: '', + mean_time_to_recovery: '' +}; + +/** seconds/percent/deployments-per-week -> the string shown in the input. */ +const toDisplayValue = (metric: BenchmarkMetric, apiValue: number | null) => { + if (apiValue === null || apiValue === undefined) return ''; + const display = DURATION_METRICS.has(metric) + ? apiValue / SECONDS_PER_HOUR + : apiValue; + return String(display); +}; + +/** + * The string in the input -> the payload value, or `undefined` to omit the + * key entirely. + * + * CLUSTOX: an empty string must become "key absent", never "key: 0" -- the + * backend treats an absent key as "inherit" and an explicit 0 as a real + * target of zero. Collapsing empty to 0 here would silently turn every + * cleared field into "fail this metric at every value above zero" the next + * time the form is saved. + */ +const toApiValue = ( + metric: BenchmarkMetric, + raw: string +): number | undefined => { + if (raw.trim() === '') return undefined; + const n = Number(raw); + if (Number.isNaN(n)) return undefined; + return DURATION_METRICS.has(metric) ? n * SECONDS_PER_HOUR : n; +}; + +const roundForDisplay = (n: number) => Math.round(n * 100) / 100; + +/** `24h (default)` / `3.5/week (default)` / `12% (default)`. */ +const formatInheritedPlaceholder = ( + metric: BenchmarkMetric, + apiValue: number +): string => { + const display = DURATION_METRICS.has(metric) + ? apiValue / SECONDS_PER_HOUR + : apiValue; + const n = roundForDisplay(display); + switch (metric) { + case 'lead_time': + case 'mean_time_to_recovery': + return `${n}h (default)`; + case 'deployment_frequency': + return `${n}/week (default)`; + case 'change_failure_rate': + return `${n}% (default)`; + } +}; + +/** Mirrors `validate_benchmark_payload` on the backend, for instant feedback + * without a round trip. The server is still the source of truth -- see the + * catch block in `handleSave`, which surfaces whatever it rejects even when + * this check passed. */ +const clientErrorFor = ( + metric: BenchmarkMetric, + raw: string +): string | null => { + if (raw.trim() === '') return null; + const n = Number(raw); + if (Number.isNaN(n)) return 'Must be a number'; + if (n < 0) return 'Must not be negative'; + if (metric === 'change_failure_rate' && n > 100) + return 'Must be between 0 and 100'; + return null; +}; + +const GLOBAL_URL = '/clustox/benchmarks/global'; +const teamUrl = (teamId: string) => `/internal/team/${teamId}/settings`; + +/** + * A validation message coming back from `validate_benchmark_payload` + * (`mhq/service/settings/benchmarks.py`) always contains the offending + * metric's key verbatim, e.g. "change_failure_rate must be between 0 and + * 100". Flask's default error page wraps that text in HTML, so tags are + * stripped before searching -- this way the field is found whether the BFF + * forwarded a clean JSON message or the raw werkzeug error page. + */ +const fieldFromServerError = ( + raw: unknown +): { metric: BenchmarkMetric | null; message: string } => { + const text = String(raw ?? '') + .replace(/<[^>]+>/g, ' ') + .replace(/\s+/g, ' ') + .trim(); + const metric = METRICS.find((m) => text.includes(m)) ?? null; + return { metric, message: text || 'Could not save benchmarks.' }; +}; + +export const BenchmarkSettingsForm: FC<{ + scope: 'team' | 'global'; + /** Called after a successful save. Team-scope callers use this to close + * the modal that hosts the form; the global section on the Workspaces + * page renders inline and has none to close. */ + onClose?: () => void; +}> = ({ scope, onClose }) => { + const { enqueueSnackbar } = useSnackbar(); + const { singleTeamId } = useSingleTeamConfig(); + const teamId = scope === 'team' ? singleTeamId : null; + + const [values, setValues] = useState(null); + const [initialValues, setInitialValues] = useState(null); + const [inherited, setInherited] = useState( + null + ); + const [loading, setLoading] = useState(true); + const [loadFailed, setLoadFailed] = useState(false); + const [saving, setSaving] = useState(false); + const [serverErrors, setServerErrors] = useState< + Partial> + >({}); + const [formError, setFormError] = useState(null); + + useEffect(() => { + if (scope === 'team' && !teamId) { + setLoading(false); + return; + } + + let cancelled = false; + setLoading(true); + setLoadFailed(false); + + const request = + scope === 'global' + ? handleApi<{ setting: BenchmarkSettingValues }>(GLOBAL_URL) + : handleApi>( + teamUrl(teamId), + { params: { setting_type: 'BENCHMARK_SETTING' } } + ); + + request + .then((data) => { + if (cancelled) return; + const next = METRICS.reduce((acc, metric) => { + acc[metric] = toDisplayValue(metric, data.setting?.[metric] ?? null); + return acc; + }, {} as FormValues); + setValues(next); + setInitialValues(next); + }) + .catch(() => { + if (!cancelled) setLoadFailed(true); + }) + .finally(() => { + if (!cancelled) setLoading(false); + }); + + return () => { + cancelled = true; + }; + }, [scope, teamId]); + + // CLUSTOX: the placeholder wants to show what clearing a field reverts to. + // For a team, that's the global baseline -- but `/clustox/benchmarks/global` + // is superadmin-only, and the person editing a team's benchmarks is + // ordinarily a workspace admin/EM, not the platform superadmin. This fetch + // is best-effort: on any failure (403, most of the time, in practice) the + // placeholder just falls back to a generic hint instead of a figure, + // rather than the form breaking or an error surfacing for something that + // isn't the user's fault. + useEffect(() => { + if (scope !== 'team') return; + let cancelled = false; + handleApi<{ setting: BenchmarkSettingValues }>(GLOBAL_URL) + .then((data) => { + if (!cancelled) setInherited(data.setting); + }) + .catch(() => { + if (!cancelled) setInherited(null); + }); + return () => { + cancelled = true; + }; + }, [scope]); + + const clientErrors = useMemo(() => { + if (!values) return {}; + return METRICS.reduce( + (acc, metric) => { + const err = clientErrorFor(metric, values[metric]); + if (err) acc[metric] = err; + return acc; + }, + {} as Partial> + ); + }, [values]); + + const isDirty = useMemo(() => { + if (!values || !initialValues) return false; + return METRICS.some((metric) => values[metric] !== initialValues[metric]); + }, [values, initialValues]); + + const handleChange = useCallback((metric: BenchmarkMetric, raw: string) => { + setValues((v) => ({ ...(v ?? EMPTY_FORM_VALUES), [metric]: raw })); + setServerErrors((errs) => { + if (!errs[metric]) return errs; + const next = { ...errs }; + delete next[metric]; + return next; + }); + }, []); + + const handleDiscard = useCallback(() => { + if (!initialValues) return; + setValues(initialValues); + setServerErrors({}); + setFormError(null); + }, [initialValues]); + + const handleSave = useCallback(async () => { + if (!values || (scope === 'team' && !teamId)) return; + + const payload = METRICS.reduce((acc, metric) => { + const apiValue = toApiValue(metric, values[metric]); + if (apiValue !== undefined) acc[metric] = apiValue; + return acc; + }, {} as Partial); + + setSaving(true); + setServerErrors({}); + setFormError(null); + + try { + if (scope === 'global') { + await handleApi(GLOBAL_URL, { method: 'PUT', data: payload }); + } else { + await handleApi(teamUrl(teamId), { + method: 'PUT', + data: { setting_type: 'BENCHMARK_SETTING', setting_data: payload } + }); + } + + setInitialValues(values); + enqueueSnackbar('Benchmarks updated', { + variant: 'success', + autoHideDuration: 3000 + }); + // CLUSTOX: only close on success. The existing team-settings modals + // (TeamProductionBranchSelector, TeamIncidentPRsFilter) close + // unconditionally and rely on a toast to report failure -- that can't + // work here, since a rejected value has to stay on screen, next to the + // field it was rejected for, for the admin to fix. + onClose?.(); + } catch (err: any) { + // CLUSTOX: server errors are shown against the field named in the + // message, never as a toast -- see the comment above `handleSave`. + const raw = err?.data?.message ?? err?.data ?? err?.message; + const { metric, message } = fieldFromServerError(raw); + if (metric) setServerErrors({ [metric]: message }); + else setFormError(message); + } finally { + setSaving(false); + } + }, [values, scope, teamId, enqueueSnackbar, onClose]); + + if (scope === 'team' && !teamId) { + return ( + Select a team to configure its benchmarks. + ); + } + + if (loading) { + return ( + + + + ); + } + + if (loadFailed || !values) { + return ( + + Could not load benchmark settings. Try again shortly. + + ); + } + + return ( + + {scope === 'global' && ( + <> + + Global DORA Benchmarks + + + The default target every team is measured against unless it sets its + own. + + + + )} + + {formError && {formError}} + + + {METRICS.map((metric) => { + const errorText = clientErrors[metric] || serverErrors[metric]; + const placeholder = + scope === 'team' && inherited?.[metric] != null + ? formatInheritedPlaceholder(metric, inherited[metric] as number) + : 'not set'; + + return ( + handleChange(metric, e.target.value)} + /> + ); + })} + + + + + 0} + sx={{ width: '140px' }} + onClick={handleSave} + > + Save + + + + ); +}; diff --git a/web-server/src/components/DoraMetricsConfigurationSettings.tsx b/web-server/src/components/DoraMetricsConfigurationSettings.tsx index 068b66a8..13f2e3c6 100644 --- a/web-server/src/components/DoraMetricsConfigurationSettings.tsx +++ b/web-server/src/components/DoraMetricsConfigurationSettings.tsx @@ -2,6 +2,7 @@ import SettingsIcon from '@mui/icons-material/Settings'; import { Button, Menu, MenuItem } from '@mui/material'; import { useCallback, useRef, useEffect } from 'react'; +import { BenchmarkSettingsForm } from '@/components/BenchmarkSettingsForm'; import { FlexBox } from '@/components/FlexBox'; import { TeamIncidentPRsFilter } from '@/components/TeamIncidentPRsFilter'; import { TeamProductionBranchSelector } from '@/components/TeamProductionBranchSelector'; @@ -45,6 +46,22 @@ export const DoraMetricsConfigurationSettings = () => { }); }, [addModal, closeModal]); + // CLUSTOX: per-team DORA benchmark targets. Same modal shape as the other + // team settings above -- a menu item opens a dedicated form, which closes + // itself on save. + const openBenchmarkSettingsModal = useCallback(async () => { + const modal = addModal({ + title: `Configure DORA Benchmarks`, + body: ( + closeModal(modal.key)} + /> + ), + showCloseIcon: true + }); + }, [addModal, closeModal]); + const anchorEl = useRef(null); const open = useBoolState(false); @@ -95,6 +112,14 @@ export const DoraMetricsConfigurationSettings = () => { > Configure Filters for Incident PRs + { + open.false(); + openBenchmarkSettingsModal(); + }} + > + Configure DORA Benchmarks + ); From 3a006c4f0c1a7fe9a4d4f23e274a8d2a746ac7de Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Wed, 12 Aug 2026 00:39:58 +0500 Subject: [PATCH 103/194] fix(benchmarks): let any admin read the global baseline, not just superadmins GET /clustox/benchmarks/global was superadmin-gated, so the team benchmark form's inherited-default placeholder 403'd for the workspace admins who actually use it and fell back to a generic hint instead of the real number. PUT stays superadmin-only -- writing the baseline is the operation that actually needs the extra restriction. Co-Authored-By: Claude Opus 5 --- docs/BENCHMARKS.md | 15 ++++++++--- .../pages/api/clustox/benchmarks/global.ts | 26 ++++++++++++++----- .../src/components/BenchmarkSettingsForm.tsx | 15 ++++++----- 3 files changed, 39 insertions(+), 17 deletions(-) diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md index 7e2d019b..c066e827 100644 --- a/docs/BENCHMARKS.md +++ b/docs/BENCHMARKS.md @@ -120,9 +120,18 @@ GET /settings/global?setting_type=BENCHMARK_SETTING PUT /settings/global ``` -Superadmin only, enforced with `assertRole(session, 'SUPERADMIN')` at the BFF. -The Flask layer has no notion of who is calling — it trusts the internal -token — so the check cannot live there. +**`PUT` is superadmin only**, enforced with `assertRole(session, 'SUPERADMIN')` +at the BFF. The Flask layer has no notion of who is calling — it trusts the +internal token — so the check cannot live there. + +**`GET` requires only an authenticated session.** Gating the read on superadmin +was the original design and it was wrong: the team configuration form shows the +inherited value as a placeholder, and a workspace admin loading that form would +get a 403 and see a generic "not set" instead. The baseline is four numbers that +already appear on that admin's own dashboard as target lines, resolved +server-side — hiding them from the form that explains where a target came from +protects nothing and makes the inheritance invisible. Writing stays locked, +because one superadmin's numbers become every unset team's targets. **Resolved benchmarks ride on the existing metrics response** rather than getting their own endpoint. The dashboard already makes one call for all four diff --git a/web-server/pages/api/clustox/benchmarks/global.ts b/web-server/pages/api/clustox/benchmarks/global.ts index d1b047f3..1e58179e 100644 --- a/web-server/pages/api/clustox/benchmarks/global.ts +++ b/web-server/pages/api/clustox/benchmarks/global.ts @@ -1,12 +1,13 @@ import * as yup from 'yup'; -import { Endpoint, nullSchema } from '@/api-helpers/global'; import { internal } from '@/api-helpers/axios'; +import { Endpoint, nullSchema } from '@/api-helpers/global'; import { assertRole } from '@/auth/guard'; -// CLUSTOX: the global baseline is superadmin-only. The Flask layer trusts the -// internal token and has no notion of who is calling, so the role check has to -// live here. +// CLUSTOX: writing the global baseline is superadmin-only -- one +// superadmin's numbers become every unset team's targets, so that stays +// locked down. The Flask layer trusts the internal token and has no notion +// of who is calling, so that role check has to live here. const putSchema = yup.object().shape({ lead_time: yup.number().min(0).nullable().optional(), deployment_frequency: yup.number().min(0).nullable().optional(), @@ -16,9 +17,20 @@ const putSchema = yup.object().shape({ const endpoint = new Endpoint(nullSchema); -endpoint.handle.GET(nullSchema, async (req, res) => { - assertRole((req as any).session, 'SUPERADMIN'); - +// CLUSTOX: reading the global baseline is deliberately NOT superadmin-gated. +// These four numbers already reach every admin's own dashboard as resolved +// target lines on the metric cards (see resolve_benchmarks / the +// `benchmarks` field on team DORA metrics) -- they are not a secret, they're +// the other half of "why did my team's card say this target." Gating the +// read behind SUPERADMIN hid the number from the one form whose entire job +// is to explain where an inherited target came from, with no +// confidentiality benefit, since it's already visible elsewhere to the same +// audience. `Endpoint.serve()` still requires an authenticated session +// (`this.authenticated` defaults to true and is not opted out here), so this +// is "any signed-in admin may read" -- not "anyone may read." Do not add +// `assertRole` back here without re-checking that reasoning; PUT below is +// the one that actually needs it. +endpoint.handle.GET(nullSchema, async (_req, res) => { const result = await internal.get('/settings/global', { params: { setting_type: 'BENCHMARK_SETTING' } }); diff --git a/web-server/src/components/BenchmarkSettingsForm.tsx b/web-server/src/components/BenchmarkSettingsForm.tsx index a338716a..bace09c7 100644 --- a/web-server/src/components/BenchmarkSettingsForm.tsx +++ b/web-server/src/components/BenchmarkSettingsForm.tsx @@ -214,13 +214,14 @@ export const BenchmarkSettingsForm: FC<{ }, [scope, teamId]); // CLUSTOX: the placeholder wants to show what clearing a field reverts to. - // For a team, that's the global baseline -- but `/clustox/benchmarks/global` - // is superadmin-only, and the person editing a team's benchmarks is - // ordinarily a workspace admin/EM, not the platform superadmin. This fetch - // is best-effort: on any failure (403, most of the time, in practice) the - // placeholder just falls back to a generic hint instead of a figure, - // rather than the form breaking or an error surfacing for something that - // isn't the user's fault. + // For a team, that's the global baseline. `GET /clustox/benchmarks/global` + // is readable by any authenticated admin (see the CLUSTOX comment on that + // route) specifically so this call succeeds for the workspace admin/EM who + // ordinarily opens this form, not just for the platform superadmin. Still + // wrapped as best-effort -- on any failure (a logged-out session, a + // network error) the placeholder falls back to a generic hint instead of a + // figure, rather than the form breaking over something that isn't the + // user's fault. useEffect(() => { if (scope !== 'team') return; let cancelled = false; From 7bfb7d63c8c0205553cb917a6eb21fa340209394 Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Wed, 12 Aug 2026 00:51:44 +0500 Subject: [PATCH 104/194] test(benchmarks): cross-workspace isolation --- web-server/e2e/benchmarks.spec.ts | 245 ++++++++++++++++++++++++++++++ 1 file changed, 245 insertions(+) create mode 100644 web-server/e2e/benchmarks.spec.ts diff --git a/web-server/e2e/benchmarks.spec.ts b/web-server/e2e/benchmarks.spec.ts new file mode 100644 index 00000000..ce911293 --- /dev/null +++ b/web-server/e2e/benchmarks.spec.ts @@ -0,0 +1,245 @@ +/** + * Cross-workspace isolation for the per-team DORA benchmark endpoints. + * + * Two endpoints are under test: + * - the settings routes, reused for benchmarks via setting_type=BENCHMARK_SETTING: + * GET/PUT /api/internal/team/{team_id}/settings + * workspace-scoped by Endpoint.serve() asserting team access whenever the + * payload carries a team_id (here, taken from the URL path). + * - the global baseline: + * GET/PUT /api/clustox/benchmarks/global + * GET is open to any authenticated admin (the four numbers already appear + * on every team's own dashboard as target lines); PUT is SUPERADMIN-only, + * because one superadmin's numbers become every unset team's targets. + * + * Running: + * docker compose up -d + * cd web-server && SUPERADMIN_PASSWORD='' yarn playwright test e2e/benchmarks.spec.ts + * + * The suite provisions its own admins through the API and cleans them up, so + * it does not depend on whatever happens to be in the database. + */ +import { APIRequestContext, expect, request, test } from '@playwright/test'; + +const APP = 'http://localhost:3333'; + +const SUPERADMIN = { + email: process.env.SUPERADMIN_EMAIL || 'admin@clustox.com', + password: process.env.SUPERADMIN_PASSWORD || '' +}; + +/** A signed-in API context. next-auth needs the CSRF token from a prior GET. */ +const signIn = async (email: string, password: string) => { + const ctx = await request.newContext({ baseURL: APP }); + + const csrf = await (await ctx.get('/api/auth/csrf')).json(); + const res = await ctx.post('/api/auth/callback/credentials', { + form: { csrfToken: csrf.csrfToken, email, password, json: 'true' }, + failOnStatusCode: false + }); + expect(res.status(), `sign-in failed for ${email}`).toBe(200); + + return ctx; +}; + +const unique = (prefix: string) => + `${prefix}.${Date.now()}${Math.floor(Math.random() * 1000)}@clustox.com`; + +/** Provision an admin with their own fresh workspace. */ +const createAdmin = async (su: APIRequestContext, name: string) => { + const email = unique(name.toLowerCase().replace(/\s+/g, '.')); + const password = 'E2eBenchmarksPass123'; + + const res = await su.post('/api/clustox/users', { + data: { name, email, password, role: 'ADMIN', team_ids: [] } + }); + expect(res.status()).toBe(200); + const { user_id, org_id } = await res.json(); + + return { userId: user_id, orgId: org_id as string, email, password }; +}; + +/** + * Remove accounts the suite created, and their workspaces with them. + * + * Without this every run left behind an admin and a workspace, which + * accumulated until the superadmin's cross-workspace view was unreadable. + */ +const deleteUsers = async (su: APIRequestContext, userIds: string[]) => { + for (const id of userIds) { + await su.fetch(`/api/clustox/users/${id}`, { + method: 'DELETE', + failOnStatusCode: false + }); + } +}; + +// CLUSTOX: yup.string().uuid() enforces the RFC-4122 version/variant nibbles, +// so an all-zero placeholder like ...-000000000001 is rejected at schema +// validation -- a test using it would get a 400 while asserting 403, passing +// for the wrong reason. This value is v4/variant-8 shaped but not a real row. +// It must NOT be GLOBAL_BENCHMARK_ENTITY_ID (...-000000000001), which is the +// real global baseline row -- using that as a "nonexistent team" would be +// reasoning about live data. +const FOREIGN_TEAM_ID = '00000000-0000-4000-8000-0000000000ff'; + +const BENCHMARK_PAYLOAD = { + lead_time: 24, + deployment_frequency: 3, + change_failure_rate: 10, + mean_time_to_recovery: 12 +}; + +test.describe('benchmarks workspace isolation', () => { + test.skip(!SUPERADMIN.password, 'SUPERADMIN_PASSWORD not set'); + + let su: APIRequestContext; + let alpha: Awaited>; + let beta: Awaited>; + let alphaCtx: APIRequestContext; + let betaCtx: APIRequestContext; + let betaTeamId: string; + + test.beforeAll(async () => { + su = await signIn(SUPERADMIN.email, SUPERADMIN.password); + alpha = await createAdmin(su, 'E2e Bench Alpha'); + beta = await createAdmin(su, 'E2e Bench Beta'); + alphaCtx = await signIn(alpha.email, alpha.password); + betaCtx = await signIn(beta.email, beta.password); + + const created = await betaCtx.post( + `/api/resources/orgs/${beta.orgId}/teams/v2`, + { data: { name: 'Bench Beta Team', org_repos: {} } } + ); + expect(created.status()).toBe(200); + betaTeamId = (await created.json()).team.id; + }); + + test.afterAll(async () => { + await deleteUsers(su, [alpha.userId, beta.userId]); + }); + + test('an admin cannot read another workspace team benchmark', async () => { + const res = await alphaCtx.get( + `/api/internal/team/${betaTeamId}/settings`, + { + params: { setting_type: 'BENCHMARK_SETTING' }, + failOnStatusCode: false + } + ); + expect(res.status()).toBe(403); + }); + + test('an admin cannot write another workspace team benchmark', async () => { + const res = await alphaCtx.fetch( + `/api/internal/team/${betaTeamId}/settings`, + { + method: 'PUT', + data: { + setting_type: 'BENCHMARK_SETTING', + setting_data: BENCHMARK_PAYLOAD + }, + failOnStatusCode: false + } + ); + expect(res.status()).toBe(403); + }); + + test('an admin cannot read a nonexistent/foreign team benchmark either', async () => { + // CLUSTOX: same 403, via a placeholder id rather than a real foreign + // team -- covers the case where the id does not resolve to any team in + // the caller's own workspace, not just a team that visibly belongs to + // someone else. + const res = await alphaCtx.get( + `/api/internal/team/${FOREIGN_TEAM_ID}/settings`, + { + params: { setting_type: 'BENCHMARK_SETTING' }, + failOnStatusCode: false + } + ); + expect(res.status()).toBe(403); + }); + + test('a non-superadmin PUT to the global baseline is forbidden', async () => { + const res = await alphaCtx.fetch('/api/clustox/benchmarks/global', { + method: 'PUT', + data: BENCHMARK_PAYLOAD, + failOnStatusCode: false + }); + expect(res.status()).toBe(403); + }); + + test('a non-superadmin GET of the global baseline is allowed', async () => { + // CLUSTOX: deliberately not a 403. The four global numbers already + // appear on every admin's own dashboard as resolved target lines, so + // reading them is open to any authenticated admin; only writing is + // superadmin-only. Asserting 403 here would assert the opposite of the + // intended design. + const res = await alphaCtx.get('/api/clustox/benchmarks/global', { + failOnStatusCode: false + }); + expect(res.status()).toBe(200); + }); + + test('unauthenticated requests to either endpoint are rejected', async ({ + request: req + }) => { + const teamRes = await req.get( + `${APP}/api/internal/team/${betaTeamId}/settings`, + { + params: { setting_type: 'BENCHMARK_SETTING' }, + failOnStatusCode: false, + maxRedirects: 0 + } + ); + expect(teamRes.status()).toBe(401); + + const globalRes = await req.get(`${APP}/api/clustox/benchmarks/global`, { + failOnStatusCode: false, + maxRedirects: 0 + }); + expect(globalRes.status()).toBe(401); + }); + + test('a superadmin can read and write the global baseline', async () => { + // CLUSTOX: this endpoint has no per-run row to delete afterwards -- it's + // the one real, shared global baseline (GLOBAL_BENCHMARK_ENTITY_ID), not + // an ephemeral fixture. Capture whatever is there before writing so the + // test can put it back, instead of leaving the shared row permanently + // changed to test values for every other admin's dashboard. + const before = await su.get('/api/clustox/benchmarks/global', { + failOnStatusCode: false + }); + expect(before.status()).toBe(200); + const original = (await before.json()).setting; + + try { + const putRes = await su.fetch('/api/clustox/benchmarks/global', { + method: 'PUT', + data: BENCHMARK_PAYLOAD, + failOnStatusCode: false + }); + expect(putRes.status()).toBe(200); + + const getRes = await su.get('/api/clustox/benchmarks/global', { + failOnStatusCode: false + }); + expect(getRes.status()).toBe(200); + const body = await getRes.json(); + expect(body.setting).toMatchObject({ + lead_time: BENCHMARK_PAYLOAD.lead_time + }); + } finally { + await su.fetch('/api/clustox/benchmarks/global', { + method: 'PUT', + data: original ?? { + lead_time: null, + deployment_frequency: null, + change_failure_rate: null, + mean_time_to_recovery: null + }, + failOnStatusCode: false + }); + } + }); +}); From 586a9626d7e4d454da79222e6fea54ef440e8501 Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Wed, 12 Aug 2026 01:15:03 +0500 Subject: [PATCH 105/194] fix(benchmarks): serve resolved targets, and stop inventing ones nobody set MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Whole-branch review findings C1 (backend half), C2, C3, I1, I2, M1, M4, M5. They land together because they interlock: the new route returns all-nulls for an unconfigured install precisely because of the I2 decision, and the validation fix is what makes an all-null write reachable at all. C1/M1 — the resolved benchmarks were attached to /deployment_analytics, whose response lands in the team_deployments redux slice. All four cards read metrics_summary.benchmarks, written only by the dora_metrics BFF route, so every target was undefined on every render and the entire user-visible half of the feature was dead. Adds GET /teams//benchmarks for the dora_metrics BFF to fold into the response the cards consume, and drops the key -- and its two per-request Settings queries -- from deployment_analytics. I2/C2 — nothing may materialise a benchmark row. get_team_settings auto-created one on first read, so opening the config form for a team that had never set a benchmark wrote all four defaults at TEAM scope: the team could never inherit again and every card claimed "your team's benchmark" for targets nobody chose. get_global_settings did the same for the whole installation, and the team form fetches it unconditionally. Resolved in favour of the spec's zero-config guarantee: the shipped default is now all-None, and both GETs return an unsaved all-None setting instead of writing one. The auto-create is gated on BENCHMARK_SETTING only -- INCIDENT_SETTING and the rest still rely on it. C3 — the form omits empty fields, so clearing all four posted {}, which save_settings treats as "no data supplied" and replaces with the defaults. "Clear everything to go back to inheriting" saved targets instead. validate_benchmark_payload now normalises to all four keys, None for cleared: truthy, so save_settings stores it, and resolve_benchmarks falls back per metric. I1 — the team PUT, the route every workspace admin's form actually writes to, performed no benchmark validation at all. A change_failure_rate of 5000 stored fine and a typo'd `leadtime` vanished behind a success toast. M4/M5 — GLOBAL scope is benchmarks-only; other setting types are rejected rather than being run through the benchmark validator or given a GLOBAL row under the sentinel id. Tests: 288 -> 303. The new ones go through Flask's test client, because every defect here lived in the seam between a correct function and its caller -- the existing unit tests on resolve_benchmarks passed with all three Criticals live. Co-Authored-By: Claude Opus 5 --- .../mhq/api/deployment_analytics.py | 27 -- backend/analytics_server/mhq/api/settings.py | 86 +++++- .../mhq/service/settings/benchmarks.py | 63 ++++- .../service/settings/default_settings_data.py | 20 +- .../tests/api/test_benchmark_routes.py | 255 ++++++++++++++++++ .../tests/service/settings/test_benchmarks.py | 63 ++++- 6 files changed, 464 insertions(+), 50 deletions(-) create mode 100644 backend/analytics_server/tests/api/test_benchmark_routes.py diff --git a/backend/analytics_server/mhq/api/deployment_analytics.py b/backend/analytics_server/mhq/api/deployment_analytics.py index e5545a80..0d7d9114 100644 --- a/backend/analytics_server/mhq/api/deployment_analytics.py +++ b/backend/analytics_server/mhq/api/deployment_analytics.py @@ -35,32 +35,6 @@ app = Blueprint("deployment_analytics", __name__) -# CLUSTOX: resolve a team's benchmarks for the metrics response. Injectable -# settings_service so the resolution order can be tested without a database. -def get_resolved_benchmarks_for_team(team_id: str, settings_service=None): - from mhq.service.settings.benchmarks import ( - GLOBAL_BENCHMARK_ENTITY_ID, - resolve_benchmarks, - ) - from mhq.service.settings.configuration_settings import get_settings_service - from mhq.store.models.settings import SettingType - from mhq.store.models.settings.enums import EntityType - - settings_service = settings_service or get_settings_service() - - team_setting = settings_service.get_settings( - SettingType.BENCHMARK_SETTING, EntityType.TEAM, team_id - ) - global_setting = settings_service.get_settings( - SettingType.BENCHMARK_SETTING, EntityType.GLOBAL, GLOBAL_BENCHMARK_ENTITY_ID - ) - - return resolve_benchmarks( - getattr(team_setting, "specific_settings", None), - getattr(global_setting, "specific_settings", None), - ) - - @app.route("/teams//deployment_analytics", methods={"GET"}) @queryschema( Schema( @@ -123,7 +97,6 @@ def get_team_deployment_analytics( } for repo in org_repos }, - "benchmarks": get_resolved_benchmarks_for_team(team_id), } diff --git a/backend/analytics_server/mhq/api/settings.py b/backend/analytics_server/mhq/api/settings.py index 9afe64ce..c945bd60 100644 --- a/backend/analytics_server/mhq/api/settings.py +++ b/backend/analytics_server/mhq/api/settings.py @@ -10,6 +10,8 @@ from mhq.service.settings import get_settings_service, settings_type_validator from mhq.service.settings.benchmarks import ( GLOBAL_BENCHMARK_ENTITY_ID, + empty_benchmark_settings, + get_resolved_benchmarks_for_team, validate_benchmark_payload, ) from mhq.store.models import SettingType, EntityType @@ -48,16 +50,47 @@ def get_team_settings(team_id: str, setting_type: SettingType, setter_id: str = ) if not settings: - settings = settings_service.save_settings( - setting_type=setting_type, - entity_type=EntityType.TEAM, - entity_id=team_id, - setter=setter, - ) + # CLUSTOX: benchmarks opt out of the auto-create below. Every other + # setting type has a meaningful shipped default and relies on this + # row being written on first read; a benchmark has no default that is + # safe to invent, because writing one at TEAM scope silently ends + # that team's per-metric inheritance from the global baseline and + # makes the card claim "your team's benchmark" for a target the team + # never set. Gated on the setting type rather than removed, so + # INCIDENT_SETTING, EXCLUDED_PRS_SETTING and the rest are untouched. + if setting_type == SettingType.BENCHMARK_SETTING: + settings = empty_benchmark_settings(EntityType.TEAM, team_id) + else: + settings = settings_service.save_settings( + setting_type=setting_type, + entity_type=EntityType.TEAM, + entity_id=team_id, + setter=setter, + ) return adapt_configuration_settings_response(settings) +# CLUSTOX: the resolved benchmarks for a team -- the team row falling back to +# the global baseline, per metric, with the source of each. +# +# This exists because the four DORA cards read the targets out of +# `metrics_summary.benchmarks`, which is written only by the dora_metrics BFF +# route. The resolved values used to be attached to +# /teams//deployment_analytics, whose response lands in a different redux +# slice that no card reads, so the entire user-visible half of the feature +# rendered nothing. Serving them here lets the dora_metrics BFF fold them into +# the response the cards actually consume, inside the Promise.all it already +# runs, without adding a round trip to the page. +@app.route("/teams//benchmarks", methods={"GET"}) +def get_team_benchmarks(team_id: str): + + query_validator = get_query_validator() + query_validator.team_validator(team_id) + + return get_resolved_benchmarks_for_team(team_id) + + @app.route("/teams//settings", methods={"PUT"}) @dataschema( Schema( @@ -87,6 +120,17 @@ def put_team_settings( if setter and str(setter.org_id) != str(team.org_id): raise BadRequest(f"User {setter_id} does not belong to team {team_id}") + # CLUSTOX: this is the route every workspace admin's benchmark form + # writes to, and it used to store setting_data verbatim -- a lead_time of + # -1, a change_failure_rate of 5000, or a typo'd `leadtime` key that + # _adapt_benchmark_setting_from_json drops on the floor while the admin + # gets a success toast. The client-side mirror in BenchmarkSettingsForm is + # not a trust boundary. Validation also normalises the payload to all four + # keys, which is what makes clearing a field mean "inherit" rather than + # "write the defaults" (see validate_benchmark_payload). + if setting_type == SettingType.BENCHMARK_SETTING: + setting_data = validate_benchmark_payload(setting_data) + settings_service = get_settings_service() settings = settings_service.save_settings( setting_type=setting_type, @@ -176,6 +220,21 @@ def put_org_settings( return adapt_configuration_settings_response(settings) +# CLUSTOX: GLOBAL scope exists for exactly one setting type. These routes are +# reachable by anything holding the internal token, and every other setting +# type is meaningless without an entity to scope it to -- an unguarded +# setting_type would let a caller create a GLOBAL row under the benchmark +# sentinel id for, say, EXCLUDED_PRS_SETTING, and would run the benchmark +# validator's unknown-keys check against a payload it knows nothing about, +# rejecting it with a confusing message. Reject it plainly instead. +def _assert_global_scope_is_benchmarks(setting_type: SettingType): + if setting_type != SettingType.BENCHMARK_SETTING: + raise BadRequest( + f"Global scope is only supported for " + f"{SettingType.BENCHMARK_SETTING.value}, got {setting_type.value}" + ) + + # CLUSTOX: the superadmin's baseline belongs to no team and no workspace, so # it cannot use the team- or org-scoped routes above. Authorisation is enforced # at the BFF -- this layer only knows the internal token, not who is calling. @@ -189,6 +248,8 @@ def put_org_settings( ) def get_global_settings(setting_type: SettingType): + _assert_global_scope_is_benchmarks(setting_type) + settings_service = get_settings_service() settings = settings_service.get_settings( setting_type=setting_type, @@ -196,11 +257,14 @@ def get_global_settings(setting_type: SettingType): entity_id=GLOBAL_BENCHMARK_ENTITY_ID, ) + # CLUSTOX: read-only. Creating the row here meant a single admin opening a + # single team form (which fetches this to build its placeholders) + # materialised the installation-wide baseline, so "no benchmark anywhere" + # -- the state the spec requires every card to render unchanged in -- was + # unreachable in practice. A superadmin creates the row by saving one. if not settings: - settings = settings_service.save_settings( - setting_type=setting_type, - entity_type=EntityType.GLOBAL, - entity_id=GLOBAL_BENCHMARK_ENTITY_ID, + settings = empty_benchmark_settings( + EntityType.GLOBAL, GLOBAL_BENCHMARK_ENTITY_ID ) return adapt_configuration_settings_response(settings) @@ -217,6 +281,8 @@ def get_global_settings(setting_type: SettingType): ) def put_global_settings(setting_type: SettingType, setting_data: Dict = None): + _assert_global_scope_is_benchmarks(setting_type) + setting_data = validate_benchmark_payload(setting_data) settings_service = get_settings_service() diff --git a/backend/analytics_server/mhq/service/settings/benchmarks.py b/backend/analytics_server/mhq/service/settings/benchmarks.py index 62742e1a..635eba8a 100644 --- a/backend/analytics_server/mhq/service/settings/benchmarks.py +++ b/backend/analytics_server/mhq/service/settings/benchmarks.py @@ -5,11 +5,14 @@ # disagree; the response can report WHICH benchmark applied, so an admin who # thinks they set a target can see that they did not; and a fifth metric later # is one more key here rather than one more reimplementation. +from datetime import datetime from typing import Dict, List, Optional from werkzeug.exceptions import BadRequest -from mhq.service.settings.models import BenchmarkSetting +from mhq.service.settings.models import BenchmarkSetting, ConfigurationSettings +from mhq.store.models.settings.enums import EntityType +from mhq.utils.time import time_now # The global baseline belongs to no team and no workspace, but Settings. # entity_id is NOT NULL. A fixed sentinel is the documented cost of reusing @@ -70,4 +73,60 @@ def validate_benchmark_payload(data: Dict) -> Dict: cleaned[metric] = value - return cleaned + # CLUSTOX: always all four keys, None for the ones the form cleared. + # The form omits empty fields, so clearing every field posts `{}` -- and + # `save_settings` treats a falsy setting_data as "no data supplied" and + # substitutes get_default_setting_data(). "Clear everything to go back to + # inheriting" therefore used to *write* whatever the defaults were. A + # dict of four Nones is truthy, round-trips through + # _adapt_benchmark_setting_from_json unchanged, and resolve_benchmarks' + # `is not None` check then falls back per metric, which is the point. + return {metric: cleaned.get(metric) for metric in BENCHMARK_METRICS} + + +def empty_benchmark_settings( + entity_type: EntityType, entity_id: str +) -> ConfigurationSettings: + """An all-`None` benchmark setting that is NOT persisted. + + CLUSTOX: a GET must never materialise a row. Reading the config form for + a team that has never set a benchmark used to write a TEAM-scoped row, + which permanently killed that team's per-metric inheritance -- and the + same on the global route meant one admin opening one form created the + installation-wide baseline. Returning this instead keeps "nothing + configured anywhere" reachable, which is the state the spec requires + every card to render unchanged in. + """ + now: datetime = time_now() + return ConfigurationSettings( + entity_id=entity_id, + entity_type=entity_type, + specific_settings=BenchmarkSetting(), + updated_by=None, + created_at=now, + updated_at=now, + ) + + +def get_resolved_benchmarks_for_team(team_id: str, settings_service=None) -> Dict: + """Resolve a team's benchmarks against the global baseline. + + CLUSTOX: `settings_service` is injectable so the resolution order can be + tested without a database. + """ + from mhq.service.settings.configuration_settings import get_settings_service + from mhq.store.models.settings import SettingType + + settings_service = settings_service or get_settings_service() + + team_setting = settings_service.get_settings( + SettingType.BENCHMARK_SETTING, EntityType.TEAM, team_id + ) + global_setting = settings_service.get_settings( + SettingType.BENCHMARK_SETTING, EntityType.GLOBAL, GLOBAL_BENCHMARK_ENTITY_ID + ) + + return resolve_benchmarks( + getattr(team_setting, "specific_settings", None), + getattr(global_setting, "specific_settings", None), + ) diff --git a/backend/analytics_server/mhq/service/settings/default_settings_data.py b/backend/analytics_server/mhq/service/settings/default_settings_data.py index bc65ca6e..ca1e00a9 100644 --- a/backend/analytics_server/mhq/service/settings/default_settings_data.py +++ b/backend/analytics_server/mhq/service/settings/default_settings_data.py @@ -34,15 +34,21 @@ def get_default_setting_data(setting_type: SettingType): "filters": [], } - # CLUSTOX: the baseline a workspace sees before a superadmin edits it. - # Values are the DORA "High" band, in the same units the metrics API - # returns: seconds, deployments per week, percent. + # CLUSTOX: deliberately empty, not the DORA "High" band. The spec's + # zero-config guarantee -- "no benchmark at any level means no line, no + # caption, and the card renders exactly as it does today" -- is only + # reachable if nothing materialises a target nobody asked for. Shipping + # numbers here made every card in every workspace sprout a target line + # and a "the default benchmark" caption the first time any admin opened + # any settings form, and made a team that had never set a benchmark + # report source: "team" for all four metrics. A superadmin types the + # baseline in; the code does not guess it. if setting_type == SettingType.BENCHMARK_SETTING: return { - "lead_time": 86400, - "deployment_frequency": 5, - "change_failure_rate": 15, - "mean_time_to_recovery": 3600, + "lead_time": None, + "deployment_frequency": None, + "change_failure_rate": None, + "mean_time_to_recovery": None, } # ADD NEW DEFAULT SETTING HERE diff --git a/backend/analytics_server/tests/api/test_benchmark_routes.py b/backend/analytics_server/tests/api/test_benchmark_routes.py new file mode 100644 index 00000000..816e777a --- /dev/null +++ b/backend/analytics_server/tests/api/test_benchmark_routes.py @@ -0,0 +1,255 @@ +# CLUSTOX: route-level tests for the benchmark settings surface. +# +# These deliberately go through Flask's test client rather than calling the +# service functions, because every benchmark defect this branch shipped lived +# in the seam between a correct function and its caller: resolved targets +# attached to a response no card read, a GET that wrote a row, a cleared form +# that saved defaults. Unit tests on resolve_benchmarks and +# validate_benchmark_payload cannot see any of that -- they were passing the +# whole time. +import pytest +from flask import Flask + +import mhq.api.settings as settings_module +from mhq.service.settings.benchmarks import ( + BENCHMARK_METRICS, + GLOBAL_BENCHMARK_ENTITY_ID, +) +from mhq.service.settings.models import BenchmarkSetting, ConfigurationSettings +from mhq.store.models.settings import SettingType +from mhq.store.models.settings.enums import EntityType +from mhq.utils.time import time_now + +TEAM_ID = "b8b069ff-048d-4635-b859-9a6609e34cf4" +ORG_ID = "e2a1d0ea-9f0e-4d3e-9b23-4a7c1e5d0f11" + + +class FakeTeam: + id = TEAM_ID + org_id = ORG_ID + + +class FakeQueryValidator: + def team_validator(self, team_id): + return FakeTeam() + + def user_validator(self, user_id): + raise AssertionError("no setter in these tests") + + +class FakeSettingsService: + """Records writes so a test can assert a GET did not perform one.""" + + def __init__(self, rows=None): + self.rows = rows or {} + self.saved = [] + + def get_settings(self, setting_type, entity_type, entity_id): + return self.rows.get((setting_type, entity_type, str(entity_id))) + + def save_settings( + self, setting_type, entity_type, entity_id, setter=None, setting_data=None + ): + self.saved.append((setting_type, entity_type, str(entity_id), setting_data)) + saved = _settings_row(entity_type, entity_id, setting_data or {}) + self.rows[(setting_type, entity_type, str(entity_id))] = saved + return saved + + +def _settings_row(entity_type, entity_id, data): + now = time_now() + return ConfigurationSettings( + entity_id=entity_id, + entity_type=entity_type, + specific_settings=BenchmarkSetting( + lead_time=data.get("lead_time"), + deployment_frequency=data.get("deployment_frequency"), + change_failure_rate=data.get("change_failure_rate"), + mean_time_to_recovery=data.get("mean_time_to_recovery"), + ), + updated_by=None, + created_at=now, + updated_at=now, + ) + + +@pytest.fixture +def routes(monkeypatch): + service = FakeSettingsService() + + monkeypatch.setattr( + settings_module, "get_query_validator", lambda: FakeQueryValidator() + ) + monkeypatch.setattr(settings_module, "get_settings_service", lambda: service) + # get_resolved_benchmarks_for_team resolves its service lazily, from the + # module it lives next to. + monkeypatch.setattr( + "mhq.service.settings.configuration_settings.get_settings_service", + lambda: service, + ) + + app = Flask(__name__) + app.register_blueprint(settings_module.app) + return app.test_client(), service + + +def _team_row(service, **values): + service.rows[(SettingType.BENCHMARK_SETTING, EntityType.TEAM, TEAM_ID)] = ( + _settings_row(EntityType.TEAM, TEAM_ID, values) + ) + + +def _global_row(service, **values): + service.rows[ + (SettingType.BENCHMARK_SETTING, EntityType.GLOBAL, GLOBAL_BENCHMARK_ENTITY_ID) + ] = _settings_row(EntityType.GLOBAL, GLOBAL_BENCHMARK_ENTITY_ID, values) + + +def test_the_benchmarks_route_serves_resolved_targets_over_http(routes): + # The shape the four DORA cards read out of metrics_summary.benchmarks: + # `.target` and `.source`. If this body changes, every + # card silently renders nothing. + client, service = routes + _team_row(service, lead_time=3600) + _global_row(service, lead_time=86400, deployment_frequency=5) + + response = client.get(f"/teams/{TEAM_ID}/benchmarks") + + assert response.status_code == 200 + assert response.json == { + "lead_time": {"target": 3600, "source": "team"}, + "deployment_frequency": {"target": 5, "source": "global"}, + "change_failure_rate": {"target": None, "source": None}, + "mean_time_to_recovery": {"target": None, "source": None}, + } + + +def test_the_benchmarks_route_is_all_null_when_nothing_is_configured(routes): + # The zero-config state, over the wire. Every card must be able to see + # "no target" and render exactly as it did before this feature. + client, service = routes + + response = client.get(f"/teams/{TEAM_ID}/benchmarks") + + assert response.status_code == 200 + assert response.json == { + metric: {"target": None, "source": None} for metric in BENCHMARK_METRICS + } + assert service.saved == [], "resolving benchmarks must not write anything" + + +def test_reading_a_team_benchmark_does_not_create_a_team_row(routes): + client, service = routes + + response = client.get( + f"/teams/{TEAM_ID}/settings", query_string={"setting_type": "BENCHMARK_SETTING"} + ) + + assert response.status_code == 200 + assert response.json["setting"] == {metric: None for metric in BENCHMARK_METRICS} + # The bug: an auto-created row of defaults ended that team's per-metric + # inheritance forever, and made the card claim "your team's benchmark". + assert service.saved == [] + + +def test_reading_the_global_baseline_does_not_create_the_global_row(routes): + client, service = routes + + response = client.get( + "/settings/global", query_string={"setting_type": "BENCHMARK_SETTING"} + ) + + assert response.status_code == 200 + assert response.json["setting"] == {metric: None for metric in BENCHMARK_METRICS} + assert response.json["scope"] == "global" + assert service.saved == [] + + +def test_a_non_benchmark_read_still_auto_creates_its_row(routes): + # The auto-create is pre-existing behaviour other setting types rely on; + # only benchmarks opt out of it. + client, service = routes + + client.get( + f"/teams/{TEAM_ID}/settings", query_string={"setting_type": "INCIDENT_SETTING"} + ) + + assert service.saved == [ + (SettingType.INCIDENT_SETTING, EntityType.TEAM, TEAM_ID, None) + ] + + +def test_clearing_every_field_on_a_team_writes_nulls_not_defaults(routes): + client, service = routes + _team_row(service, lead_time=3600, change_failure_rate=10) + + response = client.put( + f"/teams/{TEAM_ID}/settings", + json={"setting_type": "BENCHMARK_SETTING", "setting_data": {}}, + ) + + assert response.status_code == 200 + assert response.json["setting"] == {metric: None for metric in BENCHMARK_METRICS} + _, _, _, written = service.saved[-1] + assert written == {metric: None for metric in BENCHMARK_METRICS} + + +def test_a_team_put_rejects_an_out_of_range_value(routes): + client, service = routes + + response = client.put( + f"/teams/{TEAM_ID}/settings", + json={ + "setting_type": "BENCHMARK_SETTING", + "setting_data": {"change_failure_rate": 5000}, + }, + ) + + assert response.status_code == 400 + assert service.saved == [] + + +def test_a_team_put_rejects_a_typoed_metric_key(routes): + # Silently dropping this key stored nothing and reported success. + client, service = routes + + response = client.put( + f"/teams/{TEAM_ID}/settings", + json={"setting_type": "BENCHMARK_SETTING", "setting_data": {"leadtime": 3600}}, + ) + + assert response.status_code == 400 + assert service.saved == [] + + +def test_a_team_put_of_a_non_benchmark_setting_skips_benchmark_validation(routes): + client, service = routes + + response = client.put( + f"/teams/{TEAM_ID}/settings", + json={ + "setting_type": "INCIDENT_SETTING", + "setting_data": {"title_includes": ["outage"]}, + }, + ) + + assert response.status_code == 200 + assert service.saved[-1][0] == SettingType.INCIDENT_SETTING + + +@pytest.mark.parametrize("method", ["get", "put"]) +def test_the_global_scope_refuses_non_benchmark_setting_types(routes, method): + client, service = routes + + if method == "get": + response = client.get( + "/settings/global", query_string={"setting_type": "INCIDENT_SETTING"} + ) + else: + response = client.put( + "/settings/global", + json={"setting_type": "INCIDENT_SETTING", "setting_data": {}}, + ) + + assert response.status_code == 400 + assert service.saved == [] diff --git a/backend/analytics_server/tests/service/settings/test_benchmarks.py b/backend/analytics_server/tests/service/settings/test_benchmarks.py index f2ca107b..170cae7f 100644 --- a/backend/analytics_server/tests/service/settings/test_benchmarks.py +++ b/backend/analytics_server/tests/service/settings/test_benchmarks.py @@ -82,16 +82,71 @@ def test_validation_rejects_unknown_keys(): validate_benchmark_payload({"leadtime": 3600}) -def test_validation_keeps_zero_and_omits_nothing_else(): +def test_validation_rejects_booleans(): + # `False is not None`, so a boolean surviving validation would reach + # resolve_benchmarks and be emitted as {"target": false} -- a target the + # cards would then try to draw a line at. + with pytest.raises(BadRequest): + validate_benchmark_payload({"deployment_frequency": True}) + + with pytest.raises(BadRequest): + validate_benchmark_payload({"deployment_frequency": False}) + + +def test_validation_keeps_zero_and_nulls_the_rest(): cleaned = validate_benchmark_payload({"change_failure_rate": 0}) - assert cleaned == {"change_failure_rate": 0} + # 0 survives as a real target; the untouched metrics come back as + # explicit Nones rather than absent keys. + assert cleaned == { + "lead_time": None, + "deployment_frequency": None, + "change_failure_rate": 0, + "mean_time_to_recovery": None, + } def test_validation_accepts_a_partial_payload(): cleaned = validate_benchmark_payload({"lead_time": 3600}) - assert cleaned == {"lead_time": 3600} + assert cleaned["lead_time"] == 3600 + assert set(cleaned.keys()) == set(BENCHMARK_METRICS) + + +def test_clearing_every_field_is_a_truthy_all_none_payload(): + # The form omits empty fields, so clearing all four posts `{}`. + # save_settings substitutes get_default_setting_data() for a *falsy* + # setting_data, so an empty dict here would write defaults instead of + # clearing -- "clear everything to go back to inheriting" has to survive + # this function as something truthy that means "no target anywhere". + cleaned = validate_benchmark_payload({}) + + assert cleaned == {metric: None for metric in BENCHMARK_METRICS} + assert cleaned, "an all-None payload must stay truthy for save_settings" + + +def test_the_shipped_default_sets_no_target_at_any_scope(): + # Zero-config: nothing may invent a benchmark. A default with numbers in + # it gave every card a target line and a "the default benchmark" caption + # the first time any settings form was opened. + from mhq.service.settings.default_settings_data import get_default_setting_data + from mhq.store.models.settings import SettingType + + default = get_default_setting_data(SettingType.BENCHMARK_SETTING) + + assert default == {metric: None for metric in BENCHMARK_METRICS} + + +def test_an_unsaved_benchmark_setting_is_all_none_and_not_persisted(): + from mhq.service.settings.benchmarks import empty_benchmark_settings + from mhq.store.models.settings.enums import EntityType + + settings = empty_benchmark_settings(EntityType.TEAM, "team-1") + + response = adapt_configuration_settings_response(settings) + + assert response["setting"] == {metric: None for metric in BENCHMARK_METRICS} + assert response["team_id"] == "team-1" def test_resolution_asks_for_the_team_row_and_the_global_row(): @@ -106,7 +161,7 @@ def get_settings(self, setting_type, entity_type, entity_id): asked.append((setting_type, entity_type, entity_id)) return None - from mhq.api.deployment_analytics import get_resolved_benchmarks_for_team + from mhq.service.settings.benchmarks import get_resolved_benchmarks_for_team get_resolved_benchmarks_for_team("team-1", settings_service=FakeSettingsService()) From a781aeda7b20d9531d55c82c5cd97ab99542b1ab Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Wed, 12 Aug 2026 01:19:31 +0500 Subject: [PATCH 106/194] fix(benchmarks): send the resolved targets on the route the cards read Whole-branch review finding C1, consumer half. All four DORA cards read `metrics_summary.benchmarks.`, and that slice key is written only by `fetchTeamDoraMetrics` -- i.e. only by what this BFF route returns. The resolved benchmarks were being produced on /teams//deployment_analytics instead, whose response the reducer files under `state.team_deployments`, read by the deployments table and nothing else. The value was computed correctly and then discarded one layer later, so `benchmark?.target != null` was false on every card, in every workspace, in every state: no line, no caption, ever. `fetchTeamBenchmarks` joins the Promise.all this route already runs, so the targets arrive with the numbers they are drawn against and cost no extra round trip. Verified against the real HTTP response, not a test double -- this is the third time on this project a value has been produced correctly and dropped at the next layer, and on two of those the test asserted the broken shape. GET /teams//benchmarks with a team lead-time target and a global baseline returns, over the wire: {"lead_time":{"source":"team","target":3600}, "deployment_frequency":{"source":"global","target":5}, "change_failure_rate":{"source":"global","target":15}, "mean_time_to_recovery":{"source":null,"target":null}} which is `.target` / `.source` -- the exact shape the cards destructure, and `benchmarks` is not in the reducer's omit list. Co-Authored-By: Claude Opus 5 --- .../internal/team/[team_id]/dora_metrics.ts | 18 ++++++++++++++---- web-server/src/types/resources.ts | 9 +++++---- web-server/src/utils/cockpitMetricUtils.ts | 15 +++++++++++++++ 3 files changed, 34 insertions(+), 8 deletions(-) diff --git a/web-server/pages/api/internal/team/[team_id]/dora_metrics.ts b/web-server/pages/api/internal/team/[team_id]/dora_metrics.ts index 5a20f53e..31487894 100644 --- a/web-server/pages/api/internal/team/[team_id]/dora_metrics.ts +++ b/web-server/pages/api/internal/team/[team_id]/dora_metrics.ts @@ -14,7 +14,9 @@ import { fetchLeadTimeStats, fetchChangeFailureRateStats, fetchMeanTimeToRestoreStats, - fetchDeploymentFrequencyStats + fetchDeploymentFrequencyStats, + // CLUSTOX: per-team DORA benchmarks. + fetchTeamBenchmarks } from '@/utils/cockpitMetricUtils'; // CLUSTOX: contributor filter. import { stripContributorFilters } from '@/utils/contributorFilters'; @@ -61,14 +63,19 @@ endpoint.handle.GET(getSchema, async (req, res) => { const from_date = isoDateString(startOfDay(new Date(rawFromDate))); const to_date = isoDateString(endOfDay(new Date(rawToDate))); - const [branchAndRepoFilters, unsyncedRepos] = await Promise.all([ + const [branchAndRepoFilters, unsyncedRepos, benchmarks] = await Promise.all([ getBranchesAndRepoFilter({ orgId: org_id, teamId, branchMode: branch_mode as ActiveBranchMode, branches }), - getUnsyncedRepos(teamId) + getUnsyncedRepos(teamId), + // CLUSTOX: per-team DORA benchmarks. Deliberately fetched here and sent + // below, not on any other route: `metrics_summary` -- the only slice the + // four cards read `benchmarks` from -- is written solely by whatever this + // handler returns. + fetchTeamBenchmarks(teamId) ]); const [prFilters, workflowFilters] = await Promise.all([ // CLUSTOX: contributor filter -- `authors` narrows Lead Time by PR author, @@ -205,7 +212,10 @@ endpoint.handle.GET(getSchema, async (req, res) => { deploymentFrequencyResponse.deployment_frequency_trends, lead_time_prs: leadtimePrs, assigned_repos: teamRepos, - unsynced_repos: unsyncedRepos + unsynced_repos: unsyncedRepos, + // CLUSTOX: read by all four DORA cards as + // `metrics_summary.benchmarks..target` / `.source`. + benchmarks } as TeamDoraMetricsApiResponseType); }); diff --git a/web-server/src/types/resources.ts b/web-server/src/types/resources.ts index 89ddb186..f82db5bc 100644 --- a/web-server/src/types/resources.ts +++ b/web-server/src/types/resources.ts @@ -593,10 +593,11 @@ export type TeamDoraMetricsApiResponseType = { lead_time_prs: PR[]; assigned_repos: (Row<'TeamRepos'> & Row<'OrgRepo'>)[]; unsynced_repos: ID[]; - // CLUSTOX: optional -- older responses (and this BFF route today, until - // it's wired to the resolved-benchmarks helper) simply omit the key. The - // four cards must treat a missing `benchmarks` exactly like every target - // being null: no line, no caption, unchanged from before this feature. + // CLUSTOX: populated by the dora_metrics BFF route from + // `GET /teams//benchmarks`. Optional because the mock response and any + // cached older response omit the key -- the four cards must treat a missing + // `benchmarks` exactly like every target being null: no line, no caption, + // unchanged from before this feature. benchmarks?: Benchmarks; }; diff --git a/web-server/src/utils/cockpitMetricUtils.ts b/web-server/src/utils/cockpitMetricUtils.ts index ccd500e2..7864b10f 100644 --- a/web-server/src/utils/cockpitMetricUtils.ts +++ b/web-server/src/utils/cockpitMetricUtils.ts @@ -13,9 +13,24 @@ import { ChangeFailureRateTrendsApiResponse, UpdatedDeploymentFrequencyAnalyticsResponseV2 } from '@/types/resources'; +import { Benchmarks } from '@/utils/benchmarks'; import { adaptDeploymentFrequencyAndGetBadge } from './adapt_deployment_frequency'; +// CLUSTOX: the resolved benchmark targets for a team -- the team's own row +// falling back to the global baseline, per metric, resolved server-side. +// +// This has to be fetched on the dora_metrics path specifically. The four DORA +// cards read `metrics_summary.benchmarks`, and that slice key is written only +// by `fetchTeamDoraMetrics`, i.e. only by whatever this BFF route sends. A +// benchmark produced anywhere else -- it previously rode on +// /deployment_analytics -- lands in a different slice and no card ever sees +// it. It joins the Promise.all the route already runs, so it costs no extra +// round trip and the targets cannot disagree with the numbers drawn against +// them. +export const fetchTeamBenchmarks = (teamId: ID) => + handleRequest(`/teams/${teamId}/benchmarks`); + export const getFilters = (filtersArray: any[], teamIds: ID[]) => { return teamIds.reduce((objectSoFar, currentTeamId, currentIndex) => { return { From 2cf1444e322863153e2a34e05796e17cf2bb6026 Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Wed, 12 Aug 2026 01:19:31 +0500 Subject: [PATCH 107/194] fix(benchmarks): harden the caption and target line against bad inputs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Whole-branch review findings M2, M3, M6, plus the two asserts the parked-minor triage asked for. M2 — `benchmarkCaption` guarded `target === null` but not `undefined`, which would have built "3h is over target (NaN)". `== null` covers both. M3 — `BenchmarkTargetLine` computed `Math.max(target, 0, ...values)`, and a single non-finite entry makes that `NaN`; `NaN || 1` then silently becomes `1`, clamping the target to 1 and pinning the line at 100% of the card. The CFR series comes from an optional-chained `head(...)?.data.map(...)`, so a `NaN` entry is reachable. Non-finite values are dropped before the max. M6 — CFR and deployment frequency were interpolated raw, so a stored target of 15.000000000000002 rendered verbatim beside an actual that useChangeFailureRateProps had already rounded to two decimals. Both now use the same rounding. Tests — CFR and MTTR direction. LOWER_IS_BETTER is a four-line literal and deployment frequency is the one metric where the direction inverts; a member wrongly added or dropped would reverse a card's meaning with nothing else to catch it. These pass against the unfixed code by design: the set was already correct, and these are regression guards, not bug reproductions. Co-Authored-By: Claude Opus 5 --- .../src/components/BenchmarkTargetLine.tsx | 9 +++++++- .../src/utils/__tests__/benchmarks.test.ts | 13 ++++++++++++ web-server/src/utils/benchmarks.ts | 21 +++++++++++++++---- 3 files changed, 38 insertions(+), 5 deletions(-) diff --git a/web-server/src/components/BenchmarkTargetLine.tsx b/web-server/src/components/BenchmarkTargetLine.tsx index e2a096f4..27b31f2b 100644 --- a/web-server/src/components/BenchmarkTargetLine.tsx +++ b/web-server/src/components/BenchmarkTargetLine.tsx @@ -22,7 +22,14 @@ export const BenchmarkTargetLine: FC = ({ target, values = [] }) => { - const max = Math.max(target, 0, ...values) * 1.1 || 1; + // CLUSTOX: filter to finite numbers first. The Change Failure Rate series + // is built from an optional-chained `head(...)?.data.map(...)`, so a + // `undefined`/`NaN` entry is reachable -- and one of those makes `max` + // `NaN`, which `NaN || 1` quietly turns into `1`, clamping any target to 1 + // and pinning the line at 100% of the card. A wrong line is worse than no + // line, so the bad values are dropped rather than propagated. + const finiteValues = values.filter((value) => Number.isFinite(value)); + const max = Math.max(target, 0, ...finiteValues) * 1.1 || 1; const clampedTarget = Math.min(Math.max(target, 0), max); const fromBottomPct = (clampedTarget / max) * 100; diff --git a/web-server/src/utils/__tests__/benchmarks.test.ts b/web-server/src/utils/__tests__/benchmarks.test.ts index 05e0a796..2d67ad7b 100644 --- a/web-server/src/utils/__tests__/benchmarks.test.ts +++ b/web-server/src/utils/__tests__/benchmarks.test.ts @@ -14,6 +14,19 @@ describe('benchmarkCaption', () => { expect(r.text).toContain('below target'); }); + it('treats being under target as good for CFR and MTTR too', () => { + // The other two members of LOWER_IS_BETTER. Deployment frequency is the + // single metric in the set where the direction inverts, so an entry + // wrongly added to or dropped from that set would silently reverse the + // meaning of a card with nothing else to catch it. + expect(benchmarkCaption('change_failure_rate', 5, 15, 'team').tone).toBe( + 'good' + ); + expect( + benchmarkCaption('mean_time_to_recovery', 900, 3600, 'team').tone + ).toBe('good'); + }); + it('names the source so a team can see its setting did not save', () => { expect(benchmarkCaption('lead_time', 3600, 7200, 'global').text).toContain( 'default' diff --git a/web-server/src/utils/benchmarks.ts b/web-server/src/utils/benchmarks.ts index e05ae534..eb3b1813 100644 --- a/web-server/src/utils/benchmarks.ts +++ b/web-server/src/utils/benchmarks.ts @@ -50,6 +50,13 @@ const formatDuration = (seconds: number): string => { return `${Math.round((seconds / 86400) * 10) / 10}d`; }; +// CLUSTOX: two decimals, matching what useChangeFailureRateProps already does +// to the *actual* value. Targets go in through a number input and come back +// as stored, so a baseline entered as 15 can arrive as 15.000000000000002 and +// would otherwise render verbatim next to a tidily rounded actual. +const roundForDisplay = (value: number): number => + Math.round(value * 100) / 100; + const formatBenchmarkValue = ( metric: BenchmarkMetric, value: number @@ -59,9 +66,11 @@ const formatBenchmarkValue = ( case 'mean_time_to_recovery': return formatDuration(value); case 'change_failure_rate': - return `${value}%`; - case 'deployment_frequency': - return `${value} ${value === 1 ? 'deployment' : 'deployments'}/week`; + return `${roundForDisplay(value)}%`; + case 'deployment_frequency': { + const rounded = roundForDisplay(value); + return `${rounded} ${rounded === 1 ? 'deployment' : 'deployments'}/week`; + } } }; @@ -80,7 +89,11 @@ export const benchmarkCaption = ( target: number | null, source: BenchmarkSource ): BenchmarkCaption | null => { - if (target === null) return null; + // CLUSTOX: `== null`, so `undefined` is treated as "no target" too. Every + // call site gates on the parent object today, but a `target: undefined` + // slipping through would otherwise build a caption comparing against + // `undefined` -- "3h is over target (NaN)". + if (target == null) return null; const lowerIsBetter = LOWER_IS_BETTER.has(metric); // CLUSTOX: `<=`/`>=` so hitting the target exactly reads as "good", not as From d36fd8889e1066278be3a6ea6c24970186855150 Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Wed, 12 Aug 2026 01:20:25 +0500 Subject: [PATCH 108/194] docs(benchmarks): record the shipped design, not the proposed one MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Whole-branch review findings M7, M8, and the I2 decision. M7 — status was still "approved, not implemented". M8 — the validation table said "positive numbers only… a zero deployment target is meaningless" one row above "0 is deliberate and must be preserved". The code allows `0` and that is the right call: a team targeting zero failed deployments has said something. The table was wrong, not the code. I2 — the spec claimed both that `get_default_setting_data` is "where the shipped global baseline lives before anyone edits it" and that "no benchmark at any level means no line, no caption, and the card renders exactly as it does today". Both cannot hold: the team form fetches the global baseline unconditionally to build its placeholders, so one admin opening one form materialised DORA-High targets for the whole installation and the zero-config state became unreachable. Resolved in favour of the zero-config guarantee, which is the stated user-visible contract; the doc now says there is no shipped baseline. Also documents the new GET /teams//benchmarks route and why it must be on the dora_metrics path specifically, that a benchmark PUT is a full-state write, and that the routes are tested through Flask's test client because the seam is where every defect here actually lived. Co-Authored-By: Claude Opus 5 --- docs/BENCHMARKS.md | 55 +++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 50 insertions(+), 5 deletions(-) diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md index c066e827..ced584e4 100644 --- a/docs/BENCHMARKS.md +++ b/docs/BENCHMARKS.md @@ -1,6 +1,6 @@ # Per-team DORA benchmarks — design -**Status:** approved, not implemented +**Status:** implemented **Date:** 2026-08-11 **Estimate:** 3–4 days @@ -139,11 +139,32 @@ metrics; a second call for targets would let the numbers and the lines drawn against them disagree, and would add a round trip to a page that is already the slowest in the product. +In Flask this is a second route: + +``` +GET /teams//benchmarks +``` + +but the BFF folds it into the existing `Promise.all` behind +`/api/internal/team//dora_metrics`, so the browser still makes one +call and the targets still arrive with the numbers they are drawn against. +The route that serves them has to be on this path and no other: the cards read +`metrics_summary.benchmarks`, and that redux slice is written **only** by the +`dora_metrics` response. Benchmarks attached to any other endpoint land in a +slice no card reads and render nothing. + ### Already exists — not being built `SettingsService.get_settings`, `save_settings`, `get_settings_map`; `GET`/`PUT` -on `/teams//settings`; and `get_default_setting_data`, which is where -the shipped global baseline lives before anyone edits it. +on `/teams//settings`; and `get_default_setting_data`. + +**There is no shipped global baseline.** `get_default_setting_data` returns all +four benchmark keys as `None`, and no `GET` at any scope creates a row. This is +what makes the zero-config state below reachable: a default with numbers in it +gave every card in every workspace a target line and a "the default benchmark" +caption the moment any admin opened any settings form, and gave a team that had +never set a benchmark `source: "team"` on all four metrics. A superadmin types +the baseline in; the code does not guess it. --- @@ -189,9 +210,13 @@ implied by the code. Server-side on `PUT`. The browser is not a trust boundary and these numbers feed a shared dashboard. +Server-side on **both** the team and global `PUT`. The team route is the one +every workspace admin's form actually writes to, so validating only the global +route validates the route nobody uses. + | Rule | Why | |---|---| -| Positive numbers only | A negative or zero lead-time target is unachievable; a zero deployment target is meaningless | +| Non-negative numbers only | A negative target is unachievable | | CFR is 0–100 | It is a percentage | | Unknown keys rejected | A typo like `leadtime` would store silently and inherit forever | | Absent is not zero | Omitting a key means inherit; `0` is deliberate and must be preserved | @@ -201,6 +226,18 @@ The last rule is the trap. A form sending `0` for empty fields would turn team failing everything. The UI omits empty fields rather than sending zero, and the API distinguishes the two. +`0` is therefore **accepted**, not rejected — an earlier draft of the first +rule said "positive numbers only… a zero deployment target is meaningless", +which contradicted the last rule. A team targeting zero failed deployments has +said something meaningful and must not be silently switched back to inheriting. + +**A benchmark `PUT` is a full-state write.** Validation normalises the payload +to all four keys, `None` for the ones the form left empty, because the form +omits empty fields and clearing every field would otherwise post `{}` — +which `save_settings` treats as "no data supplied" and replaces with the +defaults. Without the normalisation, "clear everything to go back to +inheriting" saves targets. + ## Edge cases **A team spanning very different repos** gets one benchmark, because the @@ -235,7 +272,15 @@ Resolution is where the bugs will be, so that is where the tests concentrate: - **`0` is preserved, not treated as absent** **Validation:** negatives rejected, CFR above 100 rejected, unknown keys -rejected, omitted keys left untouched rather than nulled. +rejected, booleans rejected, `0` preserved, and an empty payload normalised to +four explicit `None`s rather than falling through to the defaults. + +**The seam, not just the function.** Resolution and validation are pure +functions with thorough unit tests, and every defect this feature shipped was +still in the seam between one of them and its caller — targets attached to the +wrong response, a `GET` that wrote a row, a cleared form that saved defaults. +So the benchmark routes are tested through Flask's test client, asserting the +body a caller actually receives. **Isolation, request-level e2e:** an admin cannot read or write another workspace's team benchmark → 403. A non-superadmin cannot `PUT` the global From 7ad2a4f904cd7e3367a6571c73bc989cb0740d2d Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Wed, 12 Aug 2026 01:30:34 +0500 Subject: [PATCH 109/194] fix(benchmarks): degrade to an unbenchmarked dashboard when targets fail to load A rejection inside the dora_metrics Promise.all takes down the whole response and blanks all four cards. Targets are an optional decoration and every card already treats undefined as 'no target', so failing soft is the correct behaviour. Caught in whole-branch re-review. Co-Authored-By: Claude Opus 5 --- .../pages/api/internal/team/[team_id]/dora_metrics.ts | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/web-server/pages/api/internal/team/[team_id]/dora_metrics.ts b/web-server/pages/api/internal/team/[team_id]/dora_metrics.ts index 31487894..3b72a127 100644 --- a/web-server/pages/api/internal/team/[team_id]/dora_metrics.ts +++ b/web-server/pages/api/internal/team/[team_id]/dora_metrics.ts @@ -75,7 +75,15 @@ endpoint.handle.GET(getSchema, async (req, res) => { // below, not on any other route: `metrics_summary` -- the only slice the // four cards read `benchmarks` from -- is written solely by whatever this // handler returns. - fetchTeamBenchmarks(teamId) + // + // The `catch` is load-bearing. Inside a Promise.all, a rejection here + // would take down the entire DORA response and blank all four cards over + // an optional decoration. That is not hypothetical: an unrelated 400 on a + // sibling call in this same Promise.all rendered "something went wrong" + // across the whole dashboard during the contributor-filter work. Targets + // are already optional, and every card treats `undefined` as "no target", + // so degrading to an unbenchmarked dashboard is the correct failure. + fetchTeamBenchmarks(teamId).catch(() => undefined) ]); const [prFilters, workflowFilters] = await Promise.all([ // CLUSTOX: contributor filter -- `authors` narrows Lead Time by PR author, From d9103942fcb3b36a6b3d82ba3327fd1b1215cdae Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Wed, 12 Aug 2026 01:50:02 +0500 Subject: [PATCH 110/194] test(benchmarks): let the isolation suite target a non-default instance E2E_BASE_URL overrides the hardcoded localhost:3333 so the suite can run against a second server -- which is how it was first executed green. Co-Authored-By: Claude Opus 5 --- web-server/e2e/benchmarks.spec.ts | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/web-server/e2e/benchmarks.spec.ts b/web-server/e2e/benchmarks.spec.ts index ce911293..8f8b624e 100644 --- a/web-server/e2e/benchmarks.spec.ts +++ b/web-server/e2e/benchmarks.spec.ts @@ -21,7 +21,10 @@ */ import { APIRequestContext, expect, request, test } from '@playwright/test'; -const APP = 'http://localhost:3333'; +// CLUSTOX: overridable so the suite can run against an instance other than the +// default dev one -- e.g. a second server started from a branch build while the +// usual port is still serving something else. +const APP = process.env.E2E_BASE_URL || 'http://localhost:3333'; const SUPERADMIN = { email: process.env.SUPERADMIN_EMAIL || 'admin@clustox.com', From 2f8974dff558d8aae51c738e371fee8ae8365468 Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Wed, 12 Aug 2026 02:23:30 +0500 Subject: [PATCH 111/194] fix(benchmarks): refresh the dashboard when a team target is saved The targets ride on the dora_metrics response, so saving from the modal left the cards drawing the previous target line until a full reload -- the admin saves, the modal closes, and nothing on screen changes. Both sibling modals in the same menu already refetch on save; this matches them. Team scope only: the global form is on the Workspaces page, where no DORA state is mounted. The spec had recorded the old behaviour as acceptable ('the same as every other filter on that page'). That was wrong -- the form is on that page. Corrected. Co-Authored-By: Claude Opus 5 --- docs/BENCHMARKS.md | 10 +++- .../src/components/BenchmarkSettingsForm.tsx | 47 +++++++++++++++++-- 2 files changed, 52 insertions(+), 5 deletions(-) diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md index ced584e4..a8c6e566 100644 --- a/docs/BENCHMARKS.md +++ b/docs/BENCHMARKS.md @@ -251,8 +251,14 @@ invisible; cleaning it up means touching team deletion for no user-facing gain. **A superadmin viewing another workspace** sees that workspace's team benchmarks. Resolution keys on the team being viewed, not on who is looking. -**A benchmark changed while the dashboard is open** does not appear until -reload — the same as every other filter on that page. +**Saving a team benchmark refetches the dashboard**, so the new target line +appears immediately. This was originally specified as "does not appear until +reload — the same as every other filter on that page", which was wrong on both +counts: the form is *on* that page, so the admin saves and watches nothing +happen, and the two sibling modals in the same menu already refetch on save. + +**A benchmark changed elsewhere** — another admin, another tab, or the global +baseline — still does not appear until reload. **Zero deployments against a frequency target** still shows the line and `5/week below target`, which is more useful than hiding the line because the diff --git a/web-server/src/components/BenchmarkSettingsForm.tsx b/web-server/src/components/BenchmarkSettingsForm.tsx index bace09c7..ff197d8a 100644 --- a/web-server/src/components/BenchmarkSettingsForm.tsx +++ b/web-server/src/components/BenchmarkSettingsForm.tsx @@ -10,8 +10,17 @@ import { useSnackbar } from 'notistack'; import { FC, useCallback, useEffect, useMemo, useState } from 'react'; import { handleApi } from '@/api-helpers/axios-api-instance'; -import { useSingleTeamConfig } from '@/hooks/useStateTeamConfig'; -import { FetchTeamSettingsAPIResponse } from '@/types/resources'; +import { useDoraMetricsFetchArgs } from '@/hooks/useDoraMetricsFetchArgs'; +import { + useSingleTeamConfig, + useStateBranchConfig +} from '@/hooks/useStateTeamConfig'; +import { fetchTeamDoraMetrics } from '@/slices/dora_metrics'; +import { useDispatch, useSelector } from '@/store'; +import { + ActiveBranchMode, + FetchTeamSettingsAPIResponse +} from '@/types/resources'; import { BenchmarkMetric } from '@/utils/benchmarks'; import { FlexBox } from './FlexBox'; @@ -159,6 +168,10 @@ export const BenchmarkSettingsForm: FC<{ const { enqueueSnackbar } = useSnackbar(); const { singleTeamId } = useSingleTeamConfig(); const teamId = scope === 'team' ? singleTeamId : null; + const dispatch = useDispatch(); + const doraMetricsFetchArgs = useDoraMetricsFetchArgs(); + const branches = useStateBranchConfig(); + const activeBranchMode = useSelector((s) => s.app.branchMode); const [values, setValues] = useState(null); const [initialValues, setInitialValues] = useState(null); @@ -295,6 +308,24 @@ export const BenchmarkSettingsForm: FC<{ } setInitialValues(values); + + // CLUSTOX: the targets ride on the dora_metrics response, so without + // this the cards keep drawing the OLD target line until a full page + // reload -- the admin saves, the modal closes, and nothing on screen + // changes. Both sibling modals in the same menu + // (TeamProductionBranchSelector, TeamIncidentPRsFilter) already refetch + // on save; this matches them. Team scope only: the global form lives on + // the Workspaces page, where no DORA state is mounted to refresh. + if (scope === 'team') { + await dispatch( + fetchTeamDoraMetrics({ + ...doraMetricsFetchArgs, + branches: + activeBranchMode === ActiveBranchMode.PROD ? null : branches + }) + ); + } + enqueueSnackbar('Benchmarks updated', { variant: 'success', autoHideDuration: 3000 @@ -315,7 +346,17 @@ export const BenchmarkSettingsForm: FC<{ } finally { setSaving(false); } - }, [values, scope, teamId, enqueueSnackbar, onClose]); + }, [ + values, + scope, + teamId, + enqueueSnackbar, + onClose, + dispatch, + doraMetricsFetchArgs, + activeBranchMode, + branches + ]); if (scope === 'team' && !teamId) { return ( From 75adbf8f3dbcf427687197b20c039de9d8920172 Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Wed, 12 Aug 2026 02:36:47 +0500 Subject: [PATCH 112/194] docs(benchmarks): spec and plan for target bands and the LOC metric Co-Authored-By: Claude Opus 5 --- docs/BENCHMARK_GRAPHS_AND_LOC.md | 216 +++++++++++ docs/BENCHMARK_GRAPHS_AND_LOC_PLAN.md | 499 ++++++++++++++++++++++++++ 2 files changed, 715 insertions(+) create mode 100644 docs/BENCHMARK_GRAPHS_AND_LOC.md create mode 100644 docs/BENCHMARK_GRAPHS_AND_LOC_PLAN.md diff --git a/docs/BENCHMARK_GRAPHS_AND_LOC.md b/docs/BENCHMARK_GRAPHS_AND_LOC.md new file mode 100644 index 00000000..8e269595 --- /dev/null +++ b/docs/BENCHMARK_GRAPHS_AND_LOC.md @@ -0,0 +1,216 @@ +# Benchmark graphs and the LOC metric — design + +**Status:** approved, not implemented +**Date:** 2026-08-12 +**Estimate:** 2–3 days +**Follows:** `docs/BENCHMARKS.md` (merged as `feat/benchmarks`) + +Two changes to the same four cards, done in one pass so the layout is only +reworked once: + +1. Replace the dashed target rule with a **shaded target band** on each card's + trend — the region on the good side of the target is tinted, so the whole + graph is coloured by whether the target is being met. +2. Add **lines of code** as a fifth metric, turning the 2×2 card grid into 2×3. + +--- + +## Why the current visual is being replaced + +The shipped version draws a 1px dashed `
` positioned by percentage over +the card's trend. Two problems, both visible on a real dashboard: + +- **It pins to an edge.** The line's height is `target / max(target, ...values)`. + A deployment-frequency target of 1/week against a series topping 35 sits at + 3% — indistinguishable from the card's bottom border. A lead-time target + larger than every plotted value pins to the top. Correct arithmetic, + unreadable result. +- **Two of four cards render nothing.** Change Failure Rate and MTTR gate both + the caption and the line on `canShowIncidentsData` / `canShowMTRData`. With + no incidents — the common case on a healthy team — half the dashboard shows + no benchmark at all, despite a target being set. + +## Decisions + +| Decision | Choice | Consequence | +|---|---|---| +| Visual | **Shaded target band on the existing trend** | The graph itself carries the verdict; no extra vertical space on a dense page. | +| Drawn by | **`chartjs-plugin-annotation`** | Already a dependency and already registered in `InternalChart2.tsx`, unused. Annotations are positioned in *data* space, so the band tracks the y-scale instead of being hand-positioned in percent. Deletes `BenchmarkTargetLine.tsx` and its clamping arithmetic. | +| LOC placement | **2×3 grid** | LOC is a peer metric, not an addendum. Costs a reflow of all four existing cards and leaves one empty slot. | +| LOC source | **Merged PRs only** | Matches the contributor dropdown, which already counts merged PRs only. An abandoned 5,000-line PR is not delivered work. | +| LOC benchmark | **Average lines changed per merged PR** | See below — this is the one genuinely contentious choice. | + +### The LOC benchmark targets PR size, not weekly volume + +The obvious reading of "a LOC benchmark" is a target for lines changed per +week. It is the wrong thing to benchmark, because **it has no good direction.** +Higher weekly LOC might mean a productive week or a bloated one; lower might +mean focus or a stall. A caption saying `above target` would be neither praise +nor warning, and the band would tint a region that carries no meaning. + +Average lines per merged PR does have a direction: **lower is better.** Small +PRs review faster and merge sooner, which is why it is worth putting next to +lead time — the two move together, and a team missing its lead-time target can +look one card across and see PR size as a likely cause. + +So the card shows **both**: weekly volume as the headline (that is what people +mean by "lines of code"), and average PR size as the benchmarked figure with +the band drawn against it. + +If this is wrong for your team, it is one line — `LOWER_IS_BETTER` and which +series the band keys off. Nothing else changes. + +--- + +## The band + +Each card's trend gains two annotations: + +- a **line** at the target, solid, low contrast +- a **box** from the target to the good-side edge of the scale, tinted + +Colour follows the shipped rule: `success` when the current value is on the +good side, `warning` when not. **Never red** — a missed internal goal is not an +error. + +Direction per metric, unchanged from `docs/BENCHMARKS.md`: + +| Metric | Lower is better | Band covers | +|---|---|---| +| Lead time | yes | target → 0 | +| Change failure rate | yes | target → 0 | +| MTTR | yes | target → 0 | +| Deployment frequency | no | target → top | +| **Average PR size** | **yes** | target → 0 | + +**The y-scale must include the target.** Chart2's grids `beginAtZero` with +`grace: '10%'` on the max. If the target exceeds the plotted max the band would +be clipped off-canvas, so the scale's `suggestedMax` becomes +`max(dataMax, target)`. This is the fix for the pinning problem: the target is +always on-canvas, in proportion. + +### The two empty-incident cards + +This is the known cost of choosing a band over a bar, and it is handled per +metric rather than left blank: + +**Change Failure Rate with no incidents** is not missing data — it is a +**genuine 0%**, which beats any target. The card currently suppresses the +caption entirely. It will instead render the band and +`0% is under target (15%) — the default benchmark`. This is a strict +improvement: the best possible result stops being displayed as nothing. + +**MTTR with no incidents** has genuinely no value — there is no recovery time +to measure. No band, no comparison. The card keeps *"No incidents reported"* +and adds one grey line: `target 1h — the default benchmark`, so an admin can +still see the target exists without being shown a comparison against nothing. + +The distinction matters: conflating "zero failures" with "no data" would +either hide a perfect score or invent a fake one. + +--- + +## LOC data + +**Nothing needs syncing.** Verified in the live database: 431 of 431 pull +requests have `meta.code_stats` populated, and `PullRequest` already exposes +`additions`, `deletions` and `changed_files` as properties. This is +aggregation and a card. + +```json +{"commits": 1, "comments": 1, "additions": 23, "deletions": 2, "changed_files": 4} +``` + +### Backend + +New service `mhq/service/code/loc.py`, and two routes on +`mhq/api/pull_requests.py` mirroring the lead-time pair exactly: + +``` +GET /teams//loc -> current + previous aggregate +GET /teams//loc/trends -> weekly series +``` + +Both take the existing `PRFilter`, so **the contributor filter, branch mode and +the excluded-PRs setting all apply with no extra work** — LOC per person comes +along for free. + +Response shape, matching the other stats blocks: + +```json +"loc_stats": { + "current": {"additions": 12043, "deletions": 4110, "avg_pr_size": 214}, + "previous": {"additions": 9800, "deletions": 3050, "avg_pr_size": 187} +} +``` + +`avg_pr_size` is `(additions + deletions) / merged PR count`, `0` when there +are no merged PRs — never a division by zero, and never `null`, so the card +does not need a third empty state. + +### Frontend + +`fetchLocStats` in `cockpitMetricUtils.ts`, added to the `Promise.all` in the +`dora_metrics` BFF route and sent in the same response — the same path the +benchmarks now take, and for the same reason: one call, one consistent set of +numbers. + +**It must be `.catch(() => undefined)`** like `fetchTeamBenchmarks`. A new +route failing inside that `Promise.all` would blank all five cards. + +--- + +## Benchmarks for LOC + +`lines_of_code` becomes a fifth key. The storage decision in +`docs/BENCHMARKS.md` was made for exactly this: + +- `BENCHMARK_METRICS` gains one entry — **no migration**, `data` is JSONB +- `validate_benchmark_payload` picks it up from that list automatically +- both forms gain one field, in **lines** (no unit conversion) +- `LOWER_IS_BETTER` gains it + +Unit is average lines per PR, matching what the band is drawn against. + +--- + +## Layout + +The four cards move from a 2×2 to a 2×3 grid, LOC in slot five, slot six +empty. The empty slot is deliberate and will not be filled with a placeholder +— an empty cell reads as "room for more", a "coming soon" tile reads as +unfinished. + +At `md` and below the grid is already single-column, so LOC simply appends and +nothing reflows. + +--- + +## Testing + +**The band, per metric:** target above the plotted max is on-canvas and not +clipped; target below the min likewise; colour flips with the direction table +above; deployment frequency's band covers the opposite side from the other +four. + +**The two empty-incident paths:** CFR with zero incidents renders a band and a +`0% is under target` caption; MTTR with zero incidents renders neither, plus +the grey target line. + +**LOC aggregation:** unmerged PRs excluded; `avg_pr_size` is 0 rather than a +crash when no PRs merged; the contributor filter narrows it; branch mode +narrows it. + +**Regression, and the most important test again:** with no benchmarks +configured anywhere, all five cards render exactly as they do today — no band, +no caption. And with LOC absent from the response entirely (an old backend +behind a new frontend), the other four cards must still render. + +--- + +## Deliberately excluded + +- **No per-repo LOC breakdown.** The dashboard aggregates to team level. +- **No language or file-type split.** `code_stats` does not carry it. +- **No LOC in the DORA score.** The score is the four DORA metrics; adding a + fifth input would change every historical number on every dashboard. diff --git a/docs/BENCHMARK_GRAPHS_AND_LOC_PLAN.md b/docs/BENCHMARK_GRAPHS_AND_LOC_PLAN.md new file mode 100644 index 00000000..0ed07f13 --- /dev/null +++ b/docs/BENCHMARK_GRAPHS_AND_LOC_PLAN.md @@ -0,0 +1,499 @@ +# Benchmark Graphs and LOC Metric Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Replace the dashed benchmark rule with a shaded target band drawn inside each card's trend, and add lines of code as a fifth metric in a 2×3 grid. + +**Architecture:** LOC is aggregated in Flask from `PullRequest.meta.code_stats` (already synced) through the existing `PRFilter`, served on two routes mirroring the lead-time pair, and delivered on the existing `dora_metrics` BFF response. The band is drawn by `chartjs-plugin-annotation`, already a dependency and already registered but unused, so it positions in data space instead of by CSS percentage. + +**Tech Stack:** Flask 3, SQLAlchemy 2, Postgres 15, Next.js 15 (pages router), MUI 5, chart.js 3.9 + chartjs-plugin-annotation 2.0. + +**Spec:** `docs/BENCHMARK_GRAPHS_AND_LOC.md`. **Predecessor:** `docs/BENCHMARKS.md`. + +## Global Constraints + +- **No database migration and no new tables.** `Settings.data` is JSONB and every enum column is `character varying`. +- Backend tests must pass. Baseline is **303**. Run `cd backend/analytics_server && ./venv/bin/python -m pytest` — that venv is the only working one; the repo-root venv lacks Flask. +- Frontend must typecheck: `npx tsc --noEmit`. Host `node_modules` is not installed — run inside the `middleware-dev` container. +- `black` and `flake8` clean on changed Python files. +- `# CLUSTOX:` / `// CLUSTOX:` comments explain **why**, never what. +- **Never run any `docker compose` command.** Copying files into the container and reloading a process is fine; rebuilding the image is not authorised. +- LOC counts **merged PRs only**. +- Units: lead time and MTTR **seconds**, deployment frequency **per week**, CFR **percent**, LOC benchmark **lines per PR**. +- **`0` is a deliberate target, never "absent".** Every check must be `is not None` / `!= null`, never truthiness. +- Commit in logical chunks. Do not amend or force-push. + +--- + +### Task 1: LOC aggregation service and Flask routes + +**Files:** +- Create: `backend/analytics_server/mhq/service/code/loc.py` +- Modify: `backend/analytics_server/mhq/api/pull_requests.py` (add routes after the `lead_time/trends` route ending line 175) +- Test: `backend/analytics_server/tests/service/code/test_loc.py` + +**Interfaces:** +- Consumes: `PRFilter` (`mhq/service/code/pr_filter.py`), `PullRequest.additions` / `.deletions` / `.changed_files` (`mhq/store/models/code/pull_requests.py:59-68`), `get_query_validator()`. +- Produces: `LOCMetrics` dataclass and `get_team_loc_metrics(team_id, interval, pr_filter) -> LOCMetrics`, plus `get_team_loc_trends(...) -> Dict[datetime, LOCMetrics]`. Task 2 consumes the two routes. + +Model the service and both routes on the lead-time pair in the same files — `/teams//lead_time` (line 98) and `/teams//lead_time/trends` (line 135). Read them first and match their filter parsing, `interval` handling and decorators exactly. + +- [ ] **Step 1: Write the failing test** + +```python +# tests/service/code/test_loc.py +from mhq.service.code.loc import LOCMetrics, aggregate_loc + + +def _pr(additions, deletions, state="MERGED"): + class FakePR: + pass + pr = FakePR() + pr.additions = additions + pr.deletions = deletions + pr.state = state + return pr + + +def test_aggregate_sums_additions_and_deletions(): + result = aggregate_loc([_pr(10, 5), _pr(20, 1)]) + assert result.additions == 30 + assert result.deletions == 6 + + +def test_avg_pr_size_is_gross_lines_over_pr_count(): + # (10+5 + 20+1) / 2 == 18 + result = aggregate_loc([_pr(10, 5), _pr(20, 1)]) + assert result.avg_pr_size == 18 + + +def test_avg_pr_size_is_zero_not_a_crash_when_no_prs(): + # A team with no merged PRs in range must not divide by zero, and must not + # report None -- the card would then need a third empty state. + result = aggregate_loc([]) + assert result == LOCMetrics(additions=0, deletions=0, avg_pr_size=0) +``` + +- [ ] **Step 2: Run it and watch it fail** + +Run: `cd backend/analytics_server && ./venv/bin/python -m pytest tests/service/code/test_loc.py -v` +Expected: FAIL, `ModuleNotFoundError: No module named 'mhq.service.code.loc'` + +- [ ] **Step 3: Write the service** + +```python +# mhq/service/code/loc.py +from dataclasses import dataclass +from datetime import datetime +from typing import Dict, List, Optional + +from mhq.service.code.pr_filter import PRFilter +from mhq.store.models.code.pull_requests import PullRequest +from mhq.store.repos.code import CodeRepoService +from mhq.utils.time import Interval + + +@dataclass +class LOCMetrics: + additions: int = 0 + deletions: int = 0 + # CLUSTOX: gross lines per merged PR, not net. A 2,000-line refactor that + # nets to zero is still 2,000 lines to review, and PR size is benchmarked + # precisely because it predicts review latency. + avg_pr_size: int = 0 + + +def aggregate_loc(prs: List[PullRequest]) -> LOCMetrics: + if not prs: + # Zero, never None: the card would otherwise need a third empty state + # on top of "no data" and "no target". + return LOCMetrics() + + additions = sum(pr.additions for pr in prs) + deletions = sum(pr.deletions for pr in prs) + return LOCMetrics( + additions=additions, + deletions=deletions, + avg_pr_size=round((additions + deletions) / len(prs)), + ) +``` + +Then add `get_team_loc_metrics` and `get_team_loc_trends` alongside, fetching merged PRs through `CodeRepoService` with the supplied `PRFilter` — mirror how `mhq/service/code/lead_time.py` obtains its PRs rather than writing a new query. + +- [ ] **Step 4: Run the tests** + +Run: `cd backend/analytics_server && ./venv/bin/python -m pytest tests/service/code/test_loc.py -v` +Expected: PASS, 3 tests. + +- [ ] **Step 5: Add the two routes** + +In `mhq/api/pull_requests.py`, after the trends route: + +```python +@app.route("/teams//loc", methods={"GET"}) +@queryschema( + Schema( + { + Required("from_time"): All(str, Coerce(datetime.fromisoformat)), + Required("to_time"): All(str, Coerce(datetime.fromisoformat)), + Optional("pr_filter"): All(str, Coerce(json.loads)), + } + ), +) +def get_team_loc(team_id: str, from_time: datetime, to_time: datetime, pr_filter: Dict = None): + ... +``` + +Match the surrounding routes' validator and filter-parsing calls exactly; do not invent a new pattern. + +- [ ] **Step 6: Verify over real HTTP, not only in tests** + +``` +docker exec middleware-dev sh -lc 'curl -s -H "X-Internal-Token: $INTERNAL_API_TOKEN" "http://localhost:9696/teams/754e3764-cb0a-4bc8-9e63-ff887fd33ebd/loc?from_time=2026-05-20T00:00:00&to_time=2026-08-12T00:00:00"' +``` + +Copy the changed `.py` files in with `docker cp` and reload gunicorn first (`kill -HUP` the master on :9696). Paste the actual output into your report. The database has 431 PRs with `code_stats`, so a zero result means something is wrong. + +- [ ] **Step 7: Commit** + +```bash +git add backend/analytics_server/mhq/service/code/loc.py backend/analytics_server/mhq/api/pull_requests.py backend/analytics_server/tests/service/code/test_loc.py +git commit -m "feat(loc): aggregate lines of code from merged pull requests" +``` + +--- + +### Task 2: Deliver LOC on the metrics response + +**Files:** +- Modify: `web-server/src/utils/cockpitMetricUtils.ts` (add beside `fetchLeadTimeStats`, line 43) +- Modify: `web-server/pages/api/internal/team/[team_id]/dora_metrics.ts` (the `Promise.all` at line ~65 and the `res.send` at ~217) +- Modify: `web-server/src/types/resources.ts` +- Test: `web-server/pages/api/internal/team/[team_id]/__tests__/dora_metrics.test.ts` (extend the existing shape test) + +**Interfaces:** +- Consumes: Task 1's two routes. +- Produces: `loc_stats` on the `dora_metrics` response, reaching `state.doraMetrics.metrics_summary.loc_stats`. Task 6 renders it. + +- [ ] **Step 1: Add the fetcher** + +```ts +export const fetchLocStats = async (params: { + teamId: ID; + from_date: DateString; + to_date: DateString; + pr_filter?: PRFilter; +}) => { /* mirror fetchLeadTimeStats exactly, including its previous-period call */ }; +``` + +- [ ] **Step 2: Wire it into the existing Promise.all** + +```ts +const [branchAndRepoFilters, unsyncedRepos, benchmarks, locStats] = await Promise.all([ + // ...existing three... + // CLUSTOX: same soft-failure contract as fetchTeamBenchmarks above -- a + // rejection here would blank all five cards over one optional metric. + fetchLocStats({ teamId, from_date, to_date }).catch(() => undefined) +]); +``` + +and add `loc_stats: locStats` to the `res.send({...})` object. + +- [ ] **Step 3: Pin the contract in a test** + +Assert the response carries `loc_stats` **and** that the four existing `*_stats` keys are still present and unchanged. A test that only checks the new key would not catch the regression that matters. + +- [ ] **Step 4: Typecheck and commit** + +```bash +git add web-server/src/utils/cockpitMetricUtils.ts "web-server/pages/api/internal/team/[team_id]/dora_metrics.ts" web-server/src/types/resources.ts +git commit -m "feat(loc): send loc_stats on the dora metrics response" +``` + +--- + +### Task 3: Register `lines_of_code` as a fifth benchmark metric + +**Files — every site that names a metric explicitly. All seven backend, all six frontend:** + +Backend: +- `mhq/service/settings/models.py:67` — add `lines_of_code: Optional[float] = None` +- `mhq/service/settings/benchmarks.py:26` — add to `BENCHMARK_METRICS` +- `mhq/service/settings/configuration_settings.py:85` and `:224` — both `data.get(...)` adapters +- `mhq/service/settings/configuration_settings.py:308` — the to-dict adapter +- `mhq/api/resources/settings_resource.py:82` — the API response adapter +- `mhq/service/settings/default_settings_data.py:51` — add `"lines_of_code": None` + +Frontend: +- `src/utils/benchmarks.ts:16` — the `BenchmarkMetric` union +- `src/utils/benchmarks.ts:42` — `LOWER_IS_BETTER` (LOC **is** lower-is-better) +- `src/utils/benchmarks.ts:66` — `formatBenchmarkValue` (`${value} lines`) +- `src/components/BenchmarkSettingsForm.tsx:37,44,51,58,68,113` — `METRICS`, `DURATION_METRICS` (LOC is **not** one), label `Average PR size`, unit `lines`, empty value, validation switch +- `pages/api/clustox/benchmarks/global.ts:15` — `lines_of_code: yup.number().min(0).nullable().optional()` + +**Interfaces:** Produces `benchmarks.lines_of_code` with the same `{target, source}` shape as the other four. Task 5 draws its band. + +- [ ] **Step 1: Write the failing test** + +```python +def test_lines_of_code_resolves_like_every_other_metric(): + resolved = resolve_benchmarks( + BenchmarkSetting(lines_of_code=200), + BenchmarkSetting(lines_of_code=400, lead_time=86400), + ) + assert resolved["lines_of_code"] == {"target": 200, "source": "team"} + assert resolved["lead_time"] == {"target": 86400, "source": "global"} + + +def test_zero_lines_of_code_is_a_real_target(): + resolved = resolve_benchmarks(BenchmarkSetting(lines_of_code=0), None) + assert resolved["lines_of_code"] == {"target": 0, "source": "team"} +``` + +- [ ] **Step 2: Run it, watch it fail** + +Expected: `KeyError: 'lines_of_code'`. + +- [ ] **Step 3: Make the change at all thirteen sites, then run the whole suite** + +Run: `cd backend/analytics_server && ./venv/bin/python -m pytest` +Expected: 303 + your new tests, zero failures. A failure elsewhere means you missed an adapter. + +- [ ] **Step 4: Verify the round trip over HTTP** + +`PUT` a team benchmark containing only `lines_of_code`, then `GET /teams//benchmarks` and confirm it comes back `source: "team"` while the other four report `global` or `null`. Paste the output into your report. + +- [ ] **Step 5: Commit** + +```bash +git commit -am "feat(benchmarks): add lines_of_code as a fifth benchmark metric" +``` + +--- + +### Task 4: The target band + +**Files:** +- Create: `web-server/src/utils/benchmarkBand.ts` +- Delete: `web-server/src/components/BenchmarkTargetLine.tsx` +- Test: `web-server/src/utils/__tests__/benchmarkBand.test.ts` + +**Interfaces:** +- Consumes: `LOWER_IS_BETTER` from `src/utils/benchmarks.ts`. +- Produces: `benchmarkBandOptions({metric, target, actual, values, theme})` returning a partial chart.js options object — the `annotation` plugin block plus the `y.suggestedMax` override. Task 5 merges it into each card's `chartOptions`. + +`chartjs-plugin-annotation` is imported and registered at `src/components/Chart2/InternalChart2.tsx:9,33` and used nowhere. Cards hide both axes (`display: false`), which does not affect annotations — they position in data space regardless. + +- [ ] **Step 1: Write the failing test** + +```ts +describe('benchmarkBandOptions', () => { + it('extends the y scale so a target above the data is not clipped', () => { + const opts = benchmarkBandOptions({ + metric: 'lead_time', target: 100, actual: 20, values: [10, 20, 30] + }); + // Without this the band is drawn off-canvas and the card looks unchanged. + expect(opts.scales.y.suggestedMax).toBeGreaterThanOrEqual(100); + }); + + it('covers target down to zero when lower is better', () => { + const { yMin, yMax } = boxOf(benchmarkBandOptions({ + metric: 'lead_time', target: 100, actual: 20, values: [10, 20, 30] + })); + expect(yMin).toBe(0); + expect(yMax).toBe(100); + }); + + it('covers target upward for deployment frequency', () => { + const { yMin } = boxOf(benchmarkBandOptions({ + metric: 'deployment_frequency', target: 5, actual: 9, values: [4, 9] + })); + expect(yMin).toBe(5); + }); + + it('tints success when the actual value is on the good side', () => { + const opts = benchmarkBandOptions({ + metric: 'lead_time', target: 100, actual: 20, values: [20] + }); + expect(colorOf(opts)).toMatch(/success/); + }); + + it('never returns red', () => { + const opts = benchmarkBandOptions({ + metric: 'lead_time', target: 10, actual: 900, values: [900] + }); + expect(colorOf(opts)).not.toMatch(/error|red/); + }); +}); +``` + +- [ ] **Step 2: Run it, watch it fail** + +Run: `docker exec middleware-dev sh -lc 'cd /app/web-server && npx jest src/utils/__tests__/benchmarkBand.test.ts'` + +- [ ] **Step 3: Implement** + +```ts +// CLUSTOX: replaces BenchmarkTargetLine, a CSS-positioned div whose height was +// target / max(target, ...values). A deployment-frequency target of 1 against +// a series topping 35 landed at 3% -- visually identical to the card's bottom +// border. An annotation is positioned in data space by chart.js itself, so it +// cannot disagree with the series it is drawn over. +export const benchmarkBandOptions = ({ metric, target, actual, values, theme }) => { + const lowerIsBetter = LOWER_IS_BETTER.has(metric); + const dataMax = Math.max(0, ...values.filter(Number.isFinite)); + // The target must be inside the scale or the band is clipped off-canvas. + const suggestedMax = Math.max(dataMax, target) * 1.1; + const onGoodSide = lowerIsBetter ? actual <= target : actual >= target; + // Never red: a missed internal goal is not an error (spec, "The graph"). + const tone = onGoodSide ? theme.colors.success : theme.colors.warning; + return { + scales: { y: { suggestedMax } }, + plugins: { + annotation: { + annotations: { + band: { + type: 'box', + yMin: lowerIsBetter ? 0 : target, + yMax: lowerIsBetter ? target : suggestedMax, + backgroundColor: alpha(tone, 0.12), + borderWidth: 0 + }, + targetLine: { + type: 'line', + yMin: target, + yMax: target, + borderColor: alpha(tone, 0.6), + borderWidth: 1 + } + } + } + } + }; +}; +``` + +- [ ] **Step 4: Run the tests, then delete the old component** + +Confirm no imports of `BenchmarkTargetLine` remain: `grep -rn BenchmarkTargetLine web-server/src` must return nothing. + +- [ ] **Step 5: Commit** + +```bash +git commit -am "feat(benchmarks): draw the target as a shaded band in data space" +``` + +--- + +### Task 5: Apply the band to the four existing cards + +**Files:** +- Modify: `src/content/DoraMetrics/DoraCards/ChangeTimeCard.tsx` (chartOptions line 36, chart line 322) +- Modify: `src/content/DoraMetrics/DoraCards/WeeklyDeliveryVolumeCard.tsx` +- Modify: `src/content/DoraMetrics/DoraCards/ChangeFailureRateCard.tsx` (caption gate line 124, line render line 238) +- Modify: `src/content/DoraMetrics/DoraCards/MeanTimeToRestoreCard.tsx` (caption gate line 96, line render line 206) + +Each card currently builds a module-level constant `chartOptions`. The band depends on props, so it must move into a `useMemo` merged with the constant via ramda's `mergeDeepRight` (already used in `InternalChart2.tsx`). + +**The two empty-incident paths are the point of this task** — do not treat them as edge cases. + +- [ ] **Step 1: Change Failure Rate — zero incidents is a real 0%, not missing data** + +The caption is gated on `canShowIncidentsData && isCfrDataAvailable`. With no incidents the team has a **0% failure rate**, which beats any target, and today that renders as nothing at all. Gate the band and caption on the **target** being set rather than on incidents existing, and pass `actual = 0`: + +```tsx +// CLUSTOX: zero incidents is a genuine 0% change failure rate -- the best +// possible result -- not absent data. Gating this on canShowIncidentsData +// meant a perfect score displayed as an empty card. +const cfrActual = isCfrDataAvailable ? changeFailureRateProps.count : 0; +``` + +- [ ] **Step 2: MTTR — zero incidents genuinely has no value** + +There is no recovery time to measure, so no band and no comparison. Render only a grey target line beneath *"No incidents reported"*: + +```tsx +{!canShowMTRData && meanTimeToRecoveryBenchmark?.target != null && ( + + target {formatBenchmarkValue('mean_time_to_recovery', meanTimeToRecoveryBenchmark.target)} + {' — '}{sourceLabel(meanTimeToRecoveryBenchmark.source)} + +)} +``` + +- [ ] **Step 3: Lead Time and Deployment Frequency** + +Merge `benchmarkBandOptions` into each card's options. Deployment frequency is the one metric where the band covers *upward* — confirm visually that its band is on the opposite side from the other three. + +- [ ] **Step 4: Typecheck, run the frontend tests, commit** + +```bash +git commit -am "feat(benchmarks): render the target band on all four DORA cards" +``` + +--- + +### Task 6: The LOC card and the 2×3 grid + +**Files:** +- Create: `src/content/DoraMetrics/DoraCards/LinesOfCodeCard.tsx` +- Modify: `src/content/DoraMetrics/DoraMetricsBody.tsx:143-157` (the ``) + +- [ ] **Step 1: Build the card** + +Copy the structure of `ChangeTimeCard.tsx` — the same `FlexBox` shell, the same `Chart2` trend behind the number, the same caption slot. Headline is **weekly lines changed**, with `+additions / −deletions` beneath it and the benchmarked **average PR size** as the figure the band is drawn against. + +- [ ] **Step 2: Add the sixth grid slot** + +```tsx + + + +``` + +Leave slot six empty — no placeholder tile. An empty cell reads as room for more; a "coming soon" tile reads as unfinished. + +- [ ] **Step 3: Handle the old-backend case** + +With `loc_stats` absent from the response the other four cards must still render. Verify by deleting the key from the response in the browser devtools, or by pointing the frontend at a backend without Task 1. + +- [ ] **Step 4: Commit** + +```bash +git commit -am "feat(loc): add the lines of code card in a 2x3 grid" +``` + +--- + +### Task 7: Regression and end-to-end verification + +**Files:** +- Modify: `web-server/e2e/benchmarks.spec.ts` +- Test: `backend/analytics_server/tests/service/code/test_loc.py` (filter coverage) + +- [ ] **Step 1: LOC respects every existing filter** + +Assert in the backend tests that `get_team_loc_metrics` narrows with a contributor `authors` filter and with `branch_mode = PROD`, and that unmerged PRs are excluded. LOC inherits these through `PRFilter`; a test proves it stayed inherited. + +- [ ] **Step 2: Extend the isolation suite** + +Add `lines_of_code` to the payloads in `e2e/benchmarks.spec.ts` and assert it round-trips. The suite honours `E2E_BASE_URL`, so it can run against a second server: + +``` +docker exec -e SUPERADMIN_PASSWORD='' -e E2E_BASE_URL='http://localhost:3334' \ + middleware-dev sh -lc 'cd /app/web-server && npx playwright test e2e/benchmarks.spec.ts --reporter=list' +``` + +Playwright's browsers are **not** installed in the image — `APIRequestContext` tests run fine, page-level ones do not. Do not try to install them. + +- [ ] **Step 3: The regression guarantee, restated** + +With no benchmark rows at all: all five cards render with no band and no caption, and every existing `*_stats` value is unchanged. This is the single most important test in the plan — the feature is additive and the four cards people already rely on must not move. + +- [ ] **Step 4: Full suites, then commit** + +Backend `pytest`, frontend `tsc` and `jest`, and the isolation suite. Report the actual numbers. + +```bash +git commit -am "test(loc): cover filter inheritance and the five-metric round trip" +``` From 86abba8eec14dc76f45fcc7f1150325eebf3c0ec Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Wed, 12 Aug 2026 02:48:53 +0500 Subject: [PATCH 113/194] feat(loc): aggregate lines of code from merged pull requests Sums additions and deletions across a team's merged PRs and reports average gross PR size, weekly-bucketed for trends. PRs come from the shared filtered query so the contributor filter, branch mode and the excluded-PRs setting apply without a LOC-specific code path. Co-Authored-By: Claude Opus 5 --- .../analytics_server/mhq/service/code/loc.py | 99 +++++++++++++++ .../tests/service/code/test_loc.py | 118 ++++++++++++++++++ 2 files changed, 217 insertions(+) create mode 100644 backend/analytics_server/mhq/service/code/loc.py create mode 100644 backend/analytics_server/tests/service/code/test_loc.py diff --git a/backend/analytics_server/mhq/service/code/loc.py b/backend/analytics_server/mhq/service/code/loc.py new file mode 100644 index 00000000..94a7baa0 --- /dev/null +++ b/backend/analytics_server/mhq/service/code/loc.py @@ -0,0 +1,99 @@ +from dataclasses import dataclass +from datetime import datetime +from typing import Dict, List + +from mhq.store.models.code import PRFilter, PullRequest +from mhq.store.models.code.repository import TeamRepos +from mhq.store.repos.code import CodeRepoService +from mhq.utils.time import ( + Interval, + fill_missing_week_buckets, + generate_expanded_buckets, +) + + +@dataclass +class LOCMetrics: + additions: int = 0 + deletions: int = 0 + # CLUSTOX: gross lines per merged PR, not net. A 2,000-line refactor that + # nets to zero is still 2,000 lines to review, and PR size is benchmarked + # precisely because it predicts review latency. + avg_pr_size: int = 0 + + +def aggregate_loc(prs: List[PullRequest]) -> LOCMetrics: + if not prs: + # CLUSTOX: zero, never None -- the card would otherwise need a third + # empty state on top of "no data" and "no target". + return LOCMetrics() + + additions = sum(pr.additions for pr in prs) + deletions = sum(pr.deletions for pr in prs) + return LOCMetrics( + additions=additions, + deletions=deletions, + avg_pr_size=round((additions + deletions) / len(prs)), + ) + + +class LOCService: + def __init__(self, code_repo_service: CodeRepoService): + self._code_repo_service = code_repo_service + + def get_team_loc_metrics( + self, + team_id: str, + interval: Interval, + pr_filter: PRFilter = None, + ) -> LOCMetrics: + return aggregate_loc(self._get_team_merged_prs(team_id, interval, pr_filter)) + + def get_team_loc_trends( + self, + team_id: str, + interval: Interval, + pr_filter: PRFilter = None, + ) -> Dict[datetime, LOCMetrics]: + + prs: List[PullRequest] = self._get_team_merged_prs(team_id, interval, pr_filter) + + weekly_prs_map: Dict[datetime, List[PullRequest]] = generate_expanded_buckets( + prs, interval, "state_changed_at", "weekly" + ) + + weekly_loc_metrics_map: Dict[datetime, LOCMetrics] = { + week: aggregate_loc(week_prs) for week, week_prs in weekly_prs_map.items() + } + + # CLUSTOX: a week with no merges has to come back as a zero bucket, not + # be missing -- the trend line is plotted per week and a hole in the map + # would shift every later point left. + return fill_missing_week_buckets(weekly_loc_metrics_map, interval, LOCMetrics) + + def _get_team_merged_prs( + self, + team_id: str, + interval: Interval, + pr_filter: PRFilter = None, + ) -> List[PullRequest]: + + team_repos: List[TeamRepos] = ( + self._code_repo_service.get_active_team_repos_by_team_id(team_id) + ) + + repo_ids = [str(team_repo.org_repo_id) for team_repo in team_repos] + if not repo_ids: + return [] + + # CLUSTOX: go through the shared filtered query rather than a LOC-specific + # one. It is what makes the contributor filter, branch mode and the + # excluded-PRs setting apply here for free, and it already restricts to + # MERGED PRs -- an abandoned 5,000-line PR is not delivered work. + return self._code_repo_service.get_prs_merged_in_interval( + repo_ids, interval, pr_filter + ) + + +def get_loc_service() -> LOCService: + return LOCService(CodeRepoService()) diff --git a/backend/analytics_server/tests/service/code/test_loc.py b/backend/analytics_server/tests/service/code/test_loc.py new file mode 100644 index 00000000..e87f9ca1 --- /dev/null +++ b/backend/analytics_server/tests/service/code/test_loc.py @@ -0,0 +1,118 @@ +from datetime import datetime, timezone + +from mhq.service.code.loc import LOCMetrics, LOCService, aggregate_loc +from mhq.utils.time import Interval + + +def _pr(additions, deletions, state="MERGED", state_changed_at=None): + class FakePR: + pass + + pr = FakePR() + pr.additions = additions + pr.deletions = deletions + pr.state = state + pr.state_changed_at = state_changed_at + return pr + + +def test_aggregate_sums_additions_and_deletions(): + result = aggregate_loc([_pr(10, 5), _pr(20, 1)]) + assert result.additions == 30 + assert result.deletions == 6 + + +def test_avg_pr_size_is_gross_lines_over_pr_count(): + # (10+5 + 20+1) / 2 == 18 + result = aggregate_loc([_pr(10, 5), _pr(20, 1)]) + assert result.avg_pr_size == 18 + + +def test_avg_pr_size_is_zero_not_a_crash_when_no_prs(): + # A team with no merged PRs in range must not divide by zero, and must not + # report None -- the card would then need a third empty state. + result = aggregate_loc([]) + assert result == LOCMetrics(additions=0, deletions=0, avg_pr_size=0) + + +class FakeTeamRepo: + def __init__(self, org_repo_id): + self.org_repo_id = org_repo_id + + +class FakeCodeRepoService: + def __init__(self, prs, team_repos=None): + self._prs = prs + self._team_repos = ( + team_repos if team_repos is not None else [FakeTeamRepo("repo-1")] + ) + self.merged_in_interval_calls = [] + + def get_active_team_repos_by_team_id(self, team_id): + return self._team_repos + + def get_prs_merged_in_interval(self, repo_ids, interval, pr_filter=None): + self.merged_in_interval_calls.append((repo_ids, interval, pr_filter)) + return self._prs + + +def test_team_loc_metrics_are_fetched_through_the_shared_pr_filter(): + # The contributor filter, branch mode and the excluded-PRs setting all live + # on PRFilter -- a hand-rolled query would silently drop all three. + prs = [_pr(10, 5), _pr(20, 1)] + code_repo_service = FakeCodeRepoService(prs) + interval = Interval(datetime(2024, 1, 1), datetime(2024, 1, 15)) + pr_filter = object() + + result = LOCService(code_repo_service).get_team_loc_metrics( + "team-1", interval, pr_filter + ) + + assert result == LOCMetrics(additions=30, deletions=6, avg_pr_size=18) + assert code_repo_service.merged_in_interval_calls == [ + (["repo-1"], interval, pr_filter) + ] + + +def test_team_loc_trends_bucket_prs_into_the_week_they_merged(): + # 2024-01-01 and 2024-01-08 are Mondays. + prs = [ + _pr(10, 5, state_changed_at=datetime(2024, 1, 3)), + _pr(20, 1, state_changed_at=datetime(2024, 1, 9)), + ] + interval = Interval(datetime(2024, 1, 1), datetime(2024, 1, 14)) + + trends = LOCService(FakeCodeRepoService(prs)).get_team_loc_trends( + "team-1", interval, None + ) + + assert trends[datetime(2024, 1, 1, tzinfo=timezone.utc)] == LOCMetrics( + additions=10, deletions=5, avg_pr_size=15 + ) + assert trends[datetime(2024, 1, 8, tzinfo=timezone.utc)] == LOCMetrics( + additions=20, deletions=1, avg_pr_size=21 + ) + + +def test_team_loc_trends_report_zero_for_a_week_with_no_merges(): + # Every week in the window must be present, so the chart has no gaps. + prs = [_pr(10, 5, state_changed_at=datetime(2024, 1, 3))] + interval = Interval(datetime(2024, 1, 1), datetime(2024, 1, 14)) + + trends = LOCService(FakeCodeRepoService(prs)).get_team_loc_trends( + "team-1", interval, None + ) + + assert trends[datetime(2024, 1, 8, tzinfo=timezone.utc)] == LOCMetrics() + + +def test_a_team_with_no_active_repos_reports_zero_rather_than_querying(): + code_repo_service = FakeCodeRepoService([], team_repos=[]) + interval = Interval(datetime(2024, 1, 1), datetime(2024, 1, 15)) + + result = LOCService(code_repo_service).get_team_loc_metrics( + "team-1", interval, None + ) + + assert result == LOCMetrics() + assert code_repo_service.merged_in_interval_calls == [] From 85e64e0b8e069ee4c76749731055ff8356e268e3 Mon Sep 17 00:00:00 2001 From: Hamad Rana Date: Wed, 12 Aug 2026 02:48:59 +0500 Subject: [PATCH 114/194] feat(loc): expose team lines-of-code totals and trends over HTTP Adds GET /teams//loc and /teams//loc/trends, mirroring the lead_time pair's schema, filter parsing and interval handling. Co-Authored-By: Claude Opus 5 --- .../analytics_server/mhq/api/pull_requests.py | 74 +++++++++++++++++++ .../mhq/api/resources/code_resouces.py | 10 +++ 2 files changed, 84 insertions(+) diff --git a/backend/analytics_server/mhq/api/pull_requests.py b/backend/analytics_server/mhq/api/pull_requests.py index 64d7eecd..f35f3ed9 100644 --- a/backend/analytics_server/mhq/api/pull_requests.py +++ b/backend/analytics_server/mhq/api/pull_requests.py @@ -7,6 +7,7 @@ from voluptuous import Required, Schema, Coerce, All, Optional from mhq.service.code.models.lead_time import LeadTimeMetrics from mhq.service.code.lead_time import get_lead_time_service +from mhq.service.code.loc import LOCMetrics, get_loc_service from mhq.service.code.pr_filter import apply_pr_filter from mhq.store.models.code import PRFilter @@ -16,6 +17,7 @@ from mhq.api.request_utils import queryschema from mhq.api.resources.code_resouces import ( adapt_lead_time_metrics, + adapt_loc_metrics, adapt_pull_request, get_non_paginated_pr_response, ) @@ -170,6 +172,78 @@ def get_team_lead_time_trends( } +# CLUSTOX: lines of code is a fifth metric on the same dashboard as the four +# DORA metrics, so it goes through the same filter path -- the contributor +# filter, branch mode and the excluded-PRs setting then apply to it for free. +@app.route("/teams//loc", methods={"GET"}) +@queryschema( + Schema( + { + Required("from_time"): All(str, Coerce(datetime.fromisoformat)), + Required("to_time"): All(str, Coerce(datetime.fromisoformat)), + Optional("pr_filter"): All(str, Coerce(json.loads)), + } + ), +) +def get_team_loc( + team_id: str, + from_time: datetime, + to_time: datetime, + pr_filter: Dict = None, +): + + query_validator = get_query_validator() + + interval: Interval = query_validator.interval_validator(from_time, to_time) + team: Team = query_validator.team_validator(team_id) + + pr_filter: PRFilter = apply_pr_filter( + pr_filter, EntityType.TEAM, team_id, [SettingType.EXCLUDED_PRS_SETTING] + ) + + loc_metrics: LOCMetrics = get_loc_service().get_team_loc_metrics( + str(team.id), interval, pr_filter + ) + + return adapt_loc_metrics(loc_metrics) + + +@app.route("/teams//loc/trends", methods={"GET"}) +@queryschema( + Schema( + { + Required("from_time"): All(str, Coerce(datetime.fromisoformat)), + Required("to_time"): All(str, Coerce(datetime.fromisoformat)), + Optional("pr_filter"): All(str, Coerce(json.loads)), + } + ), +) +def get_team_loc_trends( + team_id: str, + from_time: datetime, + to_time: datetime, + pr_filter: Dict = None, +): + + query_validator = get_query_validator() + + interval: Interval = query_validator.interval_validator(from_time, to_time) + team: Team = query_validator.team_validator(team_id) + + pr_filter: PRFilter = apply_pr_filter( + pr_filter, EntityType.TEAM, team_id, [SettingType.EXCLUDED_PRS_SETTING] + ) + + weekly_loc_metrics_map: Dict[datetime, LOCMetrics] = ( + get_loc_service().get_team_loc_trends(str(team.id), interval, pr_filter) + ) + + return { + week.isoformat(): adapt_loc_metrics(loc_metrics) + for week, loc_metrics in weekly_loc_metrics_map.items() + } + + # CLUSTOX: contributor list backing the DORA metrics filter. Scoped to the # team's repos and the window on screen, so the dropdown cannot offer someone # with no data in view. diff --git a/backend/analytics_server/mhq/api/resources/code_resouces.py b/backend/analytics_server/mhq/api/resources/code_resouces.py index c709b511..3a2e3b79 100644 --- a/backend/analytics_server/mhq/api/resources/code_resouces.py +++ b/backend/analytics_server/mhq/api/resources/code_resouces.py @@ -1,4 +1,5 @@ from typing import Dict, List +from mhq.service.code.loc import LOCMetrics from mhq.service.code.models.lead_time import LeadTimeMetrics from mhq.api.resources.core_resources import adapt_user_info from mhq.store.models.code import PullRequest, OrgRepo, TeamRepos @@ -121,6 +122,15 @@ def adapt_lead_time_metrics(lead_time_metric: LeadTimeMetrics) -> Dict[str, any] } +def adapt_loc_metrics(loc_metrics: LOCMetrics) -> Dict[str, any]: + return { + "additions": loc_metrics.additions, + "deletions": loc_metrics.deletions, + "total": loc_metrics.additions + loc_metrics.deletions, + "avg_pr_size": loc_metrics.avg_pr_size, + } + + def adapt_org_repo(org_repo: OrgRepo) -> Dict[str, any]: return { "id": str(org_repo.id), From 8c69138650e08dfda7e49383750aa2cd33a4a5cd Mon Sep 17 00:00:00 2001 From: Batool Fatima Date: Tue, 11 Aug 2026 01:57:35 +0500 Subject: [PATCH 115/194] feat: Jira integration Phase 1 (link only) + unit and E2E test coverage Feature (see docs/JIRA_INTEGRATION_PROPOSAL.md): - Link/unlink Jira via a Configure Jira modal (site URL + email + API token) -- Jira Cloud auth is Basic (email+token), not a single PAT like GitHub/GitLab. - pages/api/integrations/jira/validate.ts -- server-side credential check. Jira Cloud's API doesn't send CORS headers permissive enough for a browser to call it directly the way checkGitHubValidity/ checkGitLabValidity do, so this proxies through our own backend instead. Also guards two real bugs found while building it: a 2xx response that isn't actually a Jira /myself payload (id.atlassian.com answers this same path with HTTP 202 + HTML) is rejected rather than accepted, and a 404 (typo'd/nonexistent site) reports as "unreachable" rather than a generic error, since Atlassian's edge answers a nonexistent *.atlassian.net subdomain with a JSON 404, not a DNS failure. - Reuses the existing generic Integration table/save route unchanged -- no schema or backend save-path changes were needed. - backend/.../enums.py: added JIRA to UserIdentityProvider so CoreRepoService can resolve a Jira token the same way it already does for GITHUB/GITLAB. Sync/fetch logic is a separate, later phase. - Fixed a real regression found during implementation: pages/integrations.tsx's hasCodeProviderLinked was `integrationList.length > 0` -- true for *any* linked integration, not specifically a code provider. Linking Jira alone would have wrongly surfaced the DORA CTA, force-sync button, and "create a team" prompt. Extracted CODE_PROVIDERS to its own module (src/constants/codeProviders.ts) so this has a direct, cheap-to-import regression guard. Unit tests (Jest): - pages/api/integrations/jira/__tests__/validate.test.ts -- the classification logic above, plus auth enforcement. - src/utils/__tests__/auth-jira.test.ts -- checkJiraValidity, normalizeJiraSiteUrl. - src/content/Dashboards/__tests__/ConfigureJiraModalBody.test.tsx, JiraIntegrationCard.test.tsx -- component-level validation/link/unlink behavior, mocked network. - src/constants/__tests__/codeProviders.test.ts -- the regression guard above. - Fixed three more "no .tsx-adjacent test had ever exercised this import path before" infra gaps: tsconfig's baseUrl-style bare imports (modulePaths), .svg imports Jest can't parse (moduleNameMapper stub, ordered before the @/ alias rule so it isn't shadowed), and rendering outside the app's real ThemeProvider (custom theme.colors fields throw when a component reaches for them) -- added src/utils/testUtils.tsx's renderWithTheme for this and future component tests to reuse. E2E (Playwright): - e2e/jira-integration.spec.ts -- verifies the Jira card's rendered state against the backend's own ground truth, and that declining the unlink confirmation leaves a real linked integration untouched. Deliberately does not exercise a full successful link (needs a real Jira Cloud site + token, and this repo has no precedent for that kind of dependency in E2E for GitHub/GitLab either) -- that logic is covered in isolation by the ConfigureJiraModalBody unit tests instead. Regression check: full Jest suite (203 passed, same 2 pre-existing unrelated failures as before this branch), full Python suite (159 passed), tsc --noEmit (0 errors), and the existing e2e specs re-run clean, all before and after this commit. (cherry picked from commit 41afb7afda5e53e7d7cbe0aed8ca55968793868d) --- .../mhq/store/models/integrations/enums.py | 7 + docs/JIRA_INTEGRATION_PROPOSAL.md | 176 +++++++++ web-server/e2e/jira-integration.spec.ts | 129 +++++++ web-server/jest.config.js | 33 +- web-server/jest.setup.after-env.js | 4 + web-server/libdefs/ambient.d.ts | 6 +- web-server/libdefs/jest-dom.d.ts | 6 + web-server/package.json | 3 + .../jira/__tests__/validate.test.ts | 214 +++++++++++ .../pages/api/integrations/jira/validate.ts | 79 ++++ web-server/pages/integrations.tsx | 18 +- .../constants/__tests__/codeProviders.test.ts | 21 ++ web-server/src/constants/codeProviders.ts | 11 + .../Dashboards/ConfigureJiraModalBody.tsx | 170 +++++++++ .../Dashboards/JiraIntegrationCard.tsx | 253 +++++++++++++ .../__tests__/ConfigureJiraModalBody.test.tsx | 188 ++++++++++ .../__tests__/JiraIntegrationCard.test.tsx | 114 ++++++ .../content/Dashboards/githubIntegration.tsx | 14 + .../Dashboards/useIntegrationHandlers.tsx | 10 +- .../src/contexts/ThirdPartyAuthContext.tsx | 8 +- web-server/src/mocks/__mocks__/svg.js | 7 + .../src/utils/__tests__/auth-jira.test.ts | 99 +++++ web-server/src/utils/auth.ts | 45 +++ web-server/src/utils/testUtils.tsx | 17 + web-server/yarn.lock | 342 +++++++++++++++++- 25 files changed, 1962 insertions(+), 12 deletions(-) create mode 100644 docs/JIRA_INTEGRATION_PROPOSAL.md create mode 100644 web-server/e2e/jira-integration.spec.ts create mode 100644 web-server/jest.setup.after-env.js create mode 100644 web-server/libdefs/jest-dom.d.ts create mode 100644 web-server/pages/api/integrations/jira/__tests__/validate.test.ts create mode 100644 web-server/pages/api/integrations/jira/validate.ts create mode 100644 web-server/src/constants/__tests__/codeProviders.test.ts create mode 100644 web-server/src/constants/codeProviders.ts create mode 100644 web-server/src/content/Dashboards/ConfigureJiraModalBody.tsx create mode 100644 web-server/src/content/Dashboards/JiraIntegrationCard.tsx create mode 100644 web-server/src/content/Dashboards/__tests__/ConfigureJiraModalBody.test.tsx create mode 100644 web-server/src/content/Dashboards/__tests__/JiraIntegrationCard.test.tsx create mode 100644 web-server/src/mocks/__mocks__/svg.js create mode 100644 web-server/src/utils/__tests__/auth-jira.test.ts create mode 100644 web-server/src/utils/testUtils.tsx diff --git a/backend/analytics_server/mhq/store/models/integrations/enums.py b/backend/analytics_server/mhq/store/models/integrations/enums.py index c5dbfc1b..0c9d29f0 100644 --- a/backend/analytics_server/mhq/store/models/integrations/enums.py +++ b/backend/analytics_server/mhq/store/models/integrations/enums.py @@ -6,6 +6,13 @@ class UserIdentityProvider(Enum): GITLAB = "gitlab" # CLUSTOX: Jenkins credentials live in Integration like any other provider. JENKINS = "jenkins" + # CLUSTOX: Jira is a project-tracker integration, not a code provider -- + # added here (Phase 1: link + store the token) so CoreRepoService's + # generic get_access_token(org_id, provider) can resolve a Jira token + # the same way it already does for GITHUB/GITLAB. Sync/fetch logic is a + # separate, later phase (see docs/JIRA_INTEGRATION_PROPOSAL.md) -- this + # value on its own does not add Jira to any code-sync path. + JIRA = "jira" @classmethod def get_enum(self, provider: str): diff --git a/docs/JIRA_INTEGRATION_PROPOSAL.md b/docs/JIRA_INTEGRATION_PROPOSAL.md new file mode 100644 index 00000000..e8c44b22 --- /dev/null +++ b/docs/JIRA_INTEGRATION_PROPOSAL.md @@ -0,0 +1,176 @@ +# Jira Integration — Ticket-Linked DORA Metrics + +**For review and approval** +Prepared 2026-08-10 · Clustox/middleware + +--- + +## 1. What this is + +Today, DORA metrics (Lead Time for Changes, Cycle Time, etc.) start the clock at the **first +commit**. This adds Jira as a linked data source so the clock can start earlier — at the moment a +ticket was created — and lets teams see how their Jira workflow (sprints, ticket status) relates to +what actually shipped. + +In one sentence: **link Jira tickets to the PRs that closed them, so "idea to production" is a real, +measurable number instead of just "commit to production."** + +--- + +## 2. The good news — this isn't starting from zero + +This fork is a strip-down of an upstream product that **already had Jira support**, later removed. +The schema, error handling, and naming are still there, just disconnected: + +| Already exists, unused | What it's for | +|---|---| +| `Integration.JIRA` enum | Marks Jira as a linkable provider, same list as GitHub/GitLab | +| `Integration` table (generic, works today) | Stores an encrypted token per org per provider — no changes needed to store a Jira token | +| `IntegrationGroup.PROJECT` | The category Jira belongs to, alongside `CODE` (GitHub/GitLab) | +| `OrgProject` / `TeamProjects` tables | Exactly the shape needed for "which Jira project belongs to which team" — mirrors how repos map to teams today | +| `Tickets` / `TicketState` tables | Landing tables for issues and their status history — already has fields for sprint, story points, status | +| Jira error codes + banner UI | "Your Jira token expired" messaging is already wired up, just never triggered | + +**What's actually missing:** the Python backend doesn't yet recognize Jira as a provider, there's no +job that pulls data *from* Jira, and nothing yet matches a PR to the ticket it closes. Those three +gaps are the real work — everything else is reusable as-is. + +--- + +## 3. What we're required to have + +To connect an organization's Jira, we need three things from whoever sets it up (a Jira admin or +any user with access to the relevant projects): + +| Field | What it is | Where to get it | +|---|---|---| +| **Jira Site URL** | e.g. `yourcompany.atlassian.net` | Already known — it's the URL they use to open Jira | +| **Email** | The Atlassian account email | Their own login email | +| **API Token** | A personal access token for that account | Generated at `id.atlassian.com/manage-profile/security/api-tokens` — takes about 30 seconds | + +No Jira admin approval or app installation is required — this uses the same kind of personal API +token GitHub/GitLab linking already asks for, just with an email attached (Jira Cloud's API uses +email + token instead of a single token). + +--- + +## 4. What we'll fetch + +Once linked, we pull two kinds of data, both **read-only** — nothing is written back to Jira in this +phase: + +1. **Projects** — the list of Jira projects the linked account can see, so a team can pick which + one(s) it works out of (a team might map to one Jira project, or several). +2. **Issues, for the projects a team has selected** — ticket key, summary, status, assignee, + reporter, story points, sprint, and **status history** (when it moved from "To Do" → "In + Progress" → "Done", etc.). This is what lets us measure time-in-status, not just current status. + +Fetching happens on a schedule (same mechanism as the existing GitHub/GitLab sync), pulling only +what's changed since the last sync — not the whole project history every time. + +--- + +## 5. How it will work — the flow + +``` + 1. Link Jira → Site URL + email + API token, verified with a test call + (mirrors the existing "Configure GitHub" flow) + + 2. Pick projects → For each team: which Jira project(s) does it work out of? + (mirrors the existing repo-picker in team settings) + + 3. Sync runs on schedule → Pull issues + status history for selected projects + (new step alongside the existing repo/incident sync jobs) + + 4. Match tickets to PRs → Scan each PR's title and branch name for a ticket key + (e.g. a PR titled "PAY-482: fix refund rounding" or a + branch named "PAY-482-fix-rounding" both match "PAY-482") + + 5. Metrics + UI update → Once a PR is matched to a ticket, its lead time can include + the ticket's own timeline, and ticket-only metrics (sprint + velocity, cycle time) become available independent of code +``` + +Step 4 is the one genuinely new piece of logic in this whole integration — everything else follows +an existing pattern already used for GitHub/GitLab. + +--- + +## 6. What we'll display + +**A. Integrations page** — a third card next to GitHub/GitLab, same look: shows "Linked" once +connected, "Link" to connect, standard unlink flow. + +**B. Team settings** — a new "Projects" tab next to the existing "Repos" tab, where a team checks +off which Jira project(s) it's tracking against. + +**C. DORA Metrics page — two additions:** + +- **Extended Lead Time breakdown.** The existing phase chart (`first commit → PR opened → review → + merge → deploy`) gets an optional leading phase, `ticket created → first commit`, for any PR that + matched a ticket. This is the actual payoff of the whole integration — a true "idea to production" + number. +- **Ticket Cycle Time widget.** Independent of code: how long tickets spend in each status, sourced + from `TicketState`. Useful even for teams that want workflow visibility without caring about the + code-metrics correlation. + +**D. Sprint rollup chart** — tickets planned vs. shipped per sprint, and velocity trend, using the +sprint data Jira already tracks. + +**E. Data-hygiene callout** — a small, honest counter: *"14 PRs merged this week with no linked +ticket."* This surfaces process gaps (people not following a ticket-naming convention) rather than +silently having those PRs excluded from ticket-aware metrics with no explanation. + +--- + +## 7. Decisions needed + +**Q1. What's the actual ticket-key format your team uses in PR titles/branches?** +The matching in step 4 needs a real pattern to match against — Jira's default is +`[PROJECT-KEY]-[NUMBER]` (e.g. `PAY-482`), but teams sometimes prefix branches differently (e.g. +`feature/PAY-482-...` vs `PAY-482/...`). *Recommend: confirm the convention(s) actually in use before +building the matcher, rather than guessing and re-tuning later.* + +**Q2. How often should Jira sync run?** +GitHub/GitLab sync already runs on a schedule; Jira can either share that cadence or run +independently. *Recommend: same cadence as the existing repo sync* — no reason for it to be +different, and it's one less schedule to reason about. + +**Q3. Should a PR with no matching ticket block anything, or just be a visibility callout?** +*Recommend: visibility only (§6E).* Blocking merges on ticket-linking is a process decision separate +from this integration and would need its own discussion. + +--- + +## 8. Honest costs + +- **Phase 4 (ticket-PR matching) is the one piece with real design risk.** Everything before it is + "wire up an existing pattern to a new provider"; this step is genuinely new logic, and its accuracy + depends entirely on how consistently the team already follows a ticket-naming convention. +- **This is read-only from Jira's side in phase 1.** No comments, transitions, or writes back to + Jira — keeps the initial scope safe and reversible. Bi-directional sync (e.g. posting deploy status + as a Jira comment) would be a deliberate later phase, not part of this proposal. +- **Sync cost scales with linked projects**, same as GitHub/GitLab — each org's Jira token has its + own API rate limit, so this doesn't compound against other orgs' syncs. + +--- + +## 9. Suggested sequence + +Each phase below ships independently and is demoable on its own — there's no need to build all four +before anything is usable. + +1. **Link Jira** (§3) — a working "Linked" badge and stored credentials. No visible metrics change + yet, but it's the foundation everything else needs. +2. **Project selection + sync** (§4, §6B) — Jira issues start landing in the database; the Ticket + Cycle Time widget (§6C) becomes possible on its own, independent of code data. +3. **Ticket-PR matching** (§5 step 4) — the risky, judgment-heavy piece; needs Q1 answered first. +4. **Metrics + UI** (§6) — Extended Lead Time breakdown, sprint rollup, and the data-hygiene + callout, once matching is in place. + +--- + +## What I need + +Answers to the three questions in §7 (or "go with the recommendations"), and a go/no-go on starting +with Phase 1. diff --git a/web-server/e2e/jira-integration.spec.ts b/web-server/e2e/jira-integration.spec.ts new file mode 100644 index 00000000..6fea179a --- /dev/null +++ b/web-server/e2e/jira-integration.spec.ts @@ -0,0 +1,129 @@ +/** + * Jira linking (Phase 1 -- see docs/JIRA_INTEGRATION_PROPOSAL.md). + * + * Deliberately does not exercise a full successful link: that needs a real + * Jira Cloud site + API token, and this repo has no precedent for that kind + * of dependency in E2E either -- there's no "link GitHub successfully" or + * "link GitLab successfully" E2E test here, for the exact same reason (a + * real PAT would be needed). ConfigureJiraModalBody's validation/success/ + * error logic is covered in isolation instead, by + * src/content/Dashboards/__tests__/ConfigureJiraModalBody.test.tsx (Jest, + * mocked network). + * + * This dev environment's org has a real, live-linked Jira integration + * (checked directly against the DB while writing this) -- these tests are + * deliberately non-destructive: nothing here unlinks it, since redoing that + * link requires real credentials this test run doesn't have and can't + * restore. + * + * Running (see e2e/auth.spec.ts for why this needs a real browser, and + * therefore the host or CI rather than the middleware-dev container): + * docker compose up -d + * cd web-server && yarn playwright test e2e/jira-integration.spec.ts + */ +import { expect, test } from '@playwright/test'; + +const APP = 'http://localhost:3333'; + +const SUPERADMIN = { + email: process.env.SUPERADMIN_EMAIL || 'admin@clustox.com', + password: process.env.SUPERADMIN_PASSWORD || '' +}; + +const signIn = async (page: import('@playwright/test').Page) => { + await page.goto(`${APP}/login`); + await page.fill('input[type="email"]', SUPERADMIN.email); + await page.fill('input[type="password"]', SUPERADMIN.password); + await Promise.all([ + page.waitForNavigation({ waitUntil: 'load', timeout: 30_000 }), + page.click('button[type="submit"]') + ]); +}; + +const gotoIntegrations = async (page: import('@playwright/test').Page) => { + // The sidebar is closed by default (SidebarContext's sidebarToggle starts + // false) and the post-login landing page shows its own "Getting app + // data" loading state first -- see integration-link-status.spec.ts for + // both. + await expect(page.getByText('Getting app data')).toHaveCount(0, { + timeout: 90_000 + }); + await page.getByRole('navigation').click(); + await page + .getByRole('link', { name: 'Manage Integrations', exact: true }) + .click(); + await page.waitForURL(/\/integrations$/, { timeout: 15_000 }); +}; + +test('the Jira card reflects the real link status on first paint after login, without a refresh', async ({ + page +}) => { + test.skip(!SUPERADMIN.password, 'SUPERADMIN_PASSWORD not set'); + test.setTimeout(180_000); + + await signIn(page); + // gotoIntegrations waits out the post-login "Getting app data" loading + // state first -- firing page.request calls before the page has settled + // was observed to starve the (single-process, dev mode) Next server of + // the request budget the page's own hydration needs, which can leave + // /api/auth/session itself answering with no org yet. + await gotoIntegrations(page); + + const session = await (await page.request.get(`${APP}/api/auth/session`)).json(); + const orgId = session?.org?.id; + expect(orgId, 'signed-in session should resolve an org').toBeTruthy(); + + const map = await ( + await page.request.get( + `${APP}/api/integrations/integrations-map?org_id=${orgId}` + ) + ).json(); + const trulyLinked = Boolean(map.jira); + + const jiraCard = page.getByTestId('jira-integration-card'); + if (trulyLinked) { + await expect(jiraCard.getByText('Unlink', { exact: true })).toBeVisible({ + timeout: 5_000 + }); + } else { + await expect(jiraCard.getByText('Link', { exact: true })).toBeVisible({ + timeout: 5_000 + }); + } +}); + +test('declining the unlink confirmation leaves a linked Jira integration untouched', async ({ + page +}) => { + test.skip(!SUPERADMIN.password, 'SUPERADMIN_PASSWORD not set'); + test.setTimeout(180_000); + + await signIn(page); + await gotoIntegrations(page); + + const session = await (await page.request.get(`${APP}/api/auth/session`)).json(); + const orgId = session?.org?.id; + const before = await ( + await page.request.get( + `${APP}/api/integrations/integrations-map?org_id=${orgId}` + ) + ).json(); + test.skip(!before.jira, 'this environment has no Jira integration linked to protect'); + + // Scoped to the Jira card specifically -- GitHub is also linked in this + // environment, so an unscoped getByText('Unlink') would be ambiguous. + const jiraCard = page.getByTestId('jira-integration-card'); + page.once('dialog', (dialog) => dialog.dismiss()); + await jiraCard.getByText('Unlink', { exact: true }).click(); + + // Give the (declined) confirm a moment to resolve, then confirm nothing + // changed -- no DELETE was ever sent. + await page.waitForTimeout(500); + const after = await ( + await page.request.get( + `${APP}/api/integrations/integrations-map?org_id=${orgId}` + ) + ).json(); + expect(after.jira).toBeTruthy(); + await expect(jiraCard.getByText('Unlink', { exact: true })).toBeVisible(); +}); diff --git a/web-server/jest.config.js b/web-server/jest.config.js index d8a8cca0..e999036b 100644 --- a/web-server/jest.config.js +++ b/web-server/jest.config.js @@ -2,6 +2,13 @@ module.exports = { preset: 'ts-jest/presets/js-with-babel', // Use the TypeScript preset with Babel testEnvironment: 'jsdom', // Use jsdom as the test environment (for browser-like behavior) setupFiles: ['/jest.setup.js'], + setupFilesAfterEnv: ['/jest.setup.after-env.js'], + // tsconfig.json's baseUrl:"." is what makes a bare `from 'src/components/X'` + // import (no @/ alias, no relative path) resolve in the real app -- Jest + // doesn't read tsconfig's baseUrl on its own, so without this it can't + // find any module imported that way. First hit via + // pages/integrations.tsx importing 'src/components/Authenticated'. + modulePaths: [''], moduleFileExtensions: ['ts', 'tsx', 'js', 'jsx', 'json', 'node'], testMatch: [ '**/__tests__/**/*.test.(ts|tsx|js|jsx)', @@ -9,14 +16,36 @@ module.exports = { '**/*.test.tsx' ], transform: { - '^.+\\.(ts|tsx)$': 'ts-jest' // Transform TypeScript files using ts-jest + // tsconfig.json sets jsx:"preserve" so Next's own SWC build handles + // JSX -- ts-jest respects that setting too, which means it never + // lowers JSX to React.createElement()/jsx() calls, leaving raw JSX in + // its output for Node to choke on. Override just for the test + // transform, without touching the real tsconfig. + '^.+\\.(ts|tsx)$': ['ts-jest', { tsconfig: { jsx: 'react-jsx' } }] }, testPathIgnorePatterns: ['/node_modules/', 'auth.spec.ts'], moduleNameMapper: { + // Checked in order -- first match wins. This must come before the + // '^@/(.*)$' alias rule below: @/mocks/icons/gitlab.svg matches both, + // and the alias rule matching first would just resolve it to another + // raw, still-unparseable .svg file, never reaching this one. Next's + // build treats .svg imports as React components via an SVGR-style + // loader; Jest has no such transform and chokes trying to parse raw + // SVG/XML as JS. First hit via JiraIntegrationCard -> + // githubIntegration.tsx -> gitlab.svg / jira-icon.svg. + '\\.svg$': '/src/mocks/__mocks__/svg.js', '^@/public/(.*)$': '/public/$1', '^@/api/(.*)$': '/pages/api/$1', '^@/(.*)$': '/src/$1', - '^uuid$': require.resolve('uuid') + '^uuid$': require.resolve('uuid'), + // Same reason as the uuid mapping above: jsdom's testEnvironment makes + // Jest's resolver prefer this package's ESM build (import-only syntax), + // which then fails to parse since node_modules isn't transformed. + // Forcing the CJS entry (resolved by plain Node here, in the config + // file itself, so it correctly follows the "require" export condition) + // sidesteps that. First hit via FlexBox -> Shared.tsx -> date.ts -> + // mock.ts -> @faker-js/faker, none of which any test imported before. + '^@faker-js/faker$': require.resolve('@faker-js/faker') }, moduleDirectories: ['node_modules', 'src'] }; diff --git a/web-server/jest.setup.after-env.js b/web-server/jest.setup.after-env.js new file mode 100644 index 00000000..252eb247 --- /dev/null +++ b/web-server/jest.setup.after-env.js @@ -0,0 +1,4 @@ +// Runs after the test framework (expect, jest globals) is installed -- +// unlike jest.setup.js's setupFiles hook, jest-dom's matchers need `expect` +// to already exist to extend it. +require('@testing-library/jest-dom'); diff --git a/web-server/libdefs/ambient.d.ts b/web-server/libdefs/ambient.d.ts index 4ff6c84b..08f2fb53 100644 --- a/web-server/libdefs/ambient.d.ts +++ b/web-server/libdefs/ambient.d.ts @@ -46,8 +46,10 @@ declare type IntegrationsMap = Partial< Record< // CLUSTOX: 'jenkins' added so the workspace's linked state is readable // via useAuth().integrations.jenkins. It is a deployment provider, not a - // code provider -- see CODE_PROVIDER_INTEGRATIONS_MAP. - 'github' | 'gitlab' | 'bitbucket' | 'jenkins', + // code provider -- see CODE_PROVIDER_INTEGRATIONS_MAP. 'jira' is a + // project-tracker integration, not a code provider either -- see + // docs/JIRA_INTEGRATION_PROPOSAL.md. + 'github' | 'gitlab' | 'bitbucket' | 'jenkins' | 'jira', { integrated: Boolean; linked_at: DateString | null; diff --git a/web-server/libdefs/jest-dom.d.ts b/web-server/libdefs/jest-dom.d.ts new file mode 100644 index 00000000..24d506c1 --- /dev/null +++ b/web-server/libdefs/jest-dom.d.ts @@ -0,0 +1,6 @@ +// Side-effect-only import so TS picks up @testing-library/jest-dom's global +// `expect(...).toBeInTheDocument()`-style matcher types everywhere in the +// program. jest.setup.after-env.js requires the same package at runtime; +// that require alone doesn't reach ts-jest's type-checking since it's a +// plain .js file, not part of the TS module graph. +import '@testing-library/jest-dom'; diff --git a/web-server/package.json b/web-server/package.json index 54c0f1ee..771cf155 100644 --- a/web-server/package.json +++ b/web-server/package.json @@ -104,6 +104,9 @@ "@next/bundle-analyzer": "^12.3.0", "@octokit/types": "^6.34.0", "@playwright/test": "^1.28.0", + "@testing-library/jest-dom": "^6.4.2", + "@testing-library/react": "^14.2.1", + "@testing-library/user-event": "^14.5.2", "@types/bcryptjs": "^3.0.0", "@types/js-cookie": "^3.0.2", "@types/node": "^16.17.0", diff --git a/web-server/pages/api/integrations/jira/__tests__/validate.test.ts b/web-server/pages/api/integrations/jira/__tests__/validate.test.ts new file mode 100644 index 00000000..c379c731 --- /dev/null +++ b/web-server/pages/api/integrations/jira/__tests__/validate.test.ts @@ -0,0 +1,214 @@ +jest.mock('@/auth/session', () => ({ getAuthSession: jest.fn() })); +jest.mock('axios'); + +import axios from 'axios'; + +import { getAuthSession } from '@/auth/session'; + +import validateHandler from '../validate'; + +const mockRes = () => { + const res: any = {}; + res.status = jest.fn(() => res); + res.send = jest.fn(() => res); + return res; +}; + +const mockReq = (body: Record) => + ({ method: 'POST', query: {}, body, headers: {} }) as any; + +const asAuthed = () => + (getAuthSession as jest.Mock).mockResolvedValue({ + userId: 'u1', + email: 'admin@clustox.com', + name: 'Admin', + role: 'SUPERADMIN' + }); + +describe('POST /api/integrations/jira/validate', () => { + beforeEach(() => { + jest.clearAllMocks(); + asAuthed(); + }); + + it('requires an authenticated session, same as every other Endpoint route', async () => { + (getAuthSession as jest.Mock).mockResolvedValue(null); + const res = mockRes(); + + await validateHandler( + mockReq({ + site_url: 'x.atlassian.net', + email: 'a@b.com', + api_token: 't' + }), + res + ); + + expect(res.status).toHaveBeenCalledWith(401); + expect(axios.get).not.toHaveBeenCalled(); + }); + + it('reports valid, with the display name, when Jira answers with a real /myself payload', async () => { + (axios.get as jest.Mock).mockResolvedValue({ + data: { accountId: 'acc-1', displayName: 'Jordan Diaz' } + }); + const res = mockRes(); + + await validateHandler( + mockReq({ + site_url: 'https://mycompany.atlassian.net/', + email: 'jordan@mycompany.com', + api_token: 'tok' + }), + res + ); + + expect(res.send).toHaveBeenCalledWith({ + valid: true, + display_name: 'Jordan Diaz' + }); + // Normalized: protocol and trailing slash stripped before the request. + expect(axios.get).toHaveBeenCalledWith( + 'https://mycompany.atlassian.net/rest/api/3/myself', + expect.objectContaining({ + auth: { username: 'jordan@mycompany.com', password: 'tok' } + }) + ); + }); + + it('falls back to the submitted email as display name when Jira omits displayName', async () => { + (axios.get as jest.Mock).mockResolvedValue({ + data: { accountId: 'acc-1' } + }); + const res = mockRes(); + + await validateHandler( + mockReq({ + site_url: 'mycompany.atlassian.net', + email: 'jordan@mycompany.com', + api_token: 'tok' + }), + res + ); + + expect(res.send).toHaveBeenCalledWith({ + valid: true, + display_name: 'jordan@mycompany.com' + }); + }); + + // CLUSTOX: regression test for a real bug found while building this -- + // id.atlassian.com (a genuine Atlassian domain, just not a Jira Cloud + // *site*) answers this exact path with HTTP 202 and an HTML body, which + // a naive "did the request throw" check accepts as valid. Every real + // Jira /myself response is JSON with an accountId; this must be + // rejected despite the "successful" HTTP status. + it('rejects a 2xx response that is not actually a Jira /myself payload', async () => { + (axios.get as jest.Mock).mockResolvedValue({ + data: 'not jira' + }); + const res = mockRes(); + + await validateHandler( + mockReq({ + site_url: 'id.atlassian.com', + email: 'a@b.com', + api_token: 'tok' + }), + res + ); + + expect(res.send).toHaveBeenCalledWith({ valid: false, reason: 'unknown' }); + }); + + it.each([401, 403])( + 'reports "unauthorized" when Jira responds %d', + async (status) => { + (axios.get as jest.Mock).mockRejectedValue({ response: { status } }); + const res = mockRes(); + + await validateHandler( + mockReq({ + site_url: 'mycompany.atlassian.net', + email: 'a@b.com', + api_token: 'wrong' + }), + res + ); + + expect(res.send).toHaveBeenCalledWith({ + valid: false, + reason: 'unauthorized' + }); + } + ); + + // CLUSTOX: regression test for a real bug found while building this -- + // a nonexistent *.atlassian.net subdomain resolves fine at the DNS/edge + // level and answers with a JSON 404, it does not fail to connect. A + // check that only looked for "no response at all" would misclassify a + // typo'd site as reason:"unknown" instead of the more useful + // reason:"unreachable" ("check the URL"). + it('reports "unreachable" for a 404 (a typo\'d or nonexistent site)', async () => { + (axios.get as jest.Mock).mockRejectedValue({ response: { status: 404 } }); + const res = mockRes(); + + await validateHandler( + mockReq({ + site_url: 'this-does-not-exist.atlassian.net', + email: 'a@b.com', + api_token: 'tok' + }), + res + ); + + expect(res.send).toHaveBeenCalledWith({ + valid: false, + reason: 'unreachable' + }); + }); + + it('reports "unreachable" when there is no response at all (DNS/network failure)', async () => { + (axios.get as jest.Mock).mockRejectedValue({ code: 'ENOTFOUND' }); + const res = mockRes(); + + await validateHandler( + mockReq({ + site_url: 'unreachable.example', + email: 'a@b.com', + api_token: 'tok' + }), + res + ); + + expect(res.send).toHaveBeenCalledWith({ + valid: false, + reason: 'unreachable' + }); + }); + + it('reports "unknown" for any other error status', async () => { + (axios.get as jest.Mock).mockRejectedValue({ response: { status: 500 } }); + const res = mockRes(); + + await validateHandler( + mockReq({ + site_url: 'mycompany.atlassian.net', + email: 'a@b.com', + api_token: 'tok' + }), + res + ); + + expect(res.send).toHaveBeenCalledWith({ valid: false, reason: 'unknown' }); + }); + + it('rejects a request missing required fields before ever calling Jira', async () => { + const res = mockRes(); + + await validateHandler(mockReq({ site_url: 'x.atlassian.net' }), res); + + expect(axios.get).not.toHaveBeenCalled(); + expect(res.status).toHaveBeenCalledWith(400); + }); +}); diff --git a/web-server/pages/api/integrations/jira/validate.ts b/web-server/pages/api/integrations/jira/validate.ts new file mode 100644 index 00000000..ec120e5e --- /dev/null +++ b/web-server/pages/api/integrations/jira/validate.ts @@ -0,0 +1,79 @@ +import axios from 'axios'; +import * as yup from 'yup'; + +import { Endpoint, nullSchema } from '@/api-helpers/global'; + +const postSchema = yup.object().shape({ + site_url: yup.string().required(), + email: yup.string().email().required(), + api_token: yup.string().required() +}); + +// Strip any protocol/trailing slash so both "mycompany.atlassian.net" and +// "https://mycompany.atlassian.net/" work as input. +const normalizeSiteUrl = (input: string) => + input + .trim() + .replace(/^https?:\/\//i, '') + .replace(/\/+$/, ''); + +const endpoint = new Endpoint(nullSchema); + +// CLUSTOX: Jira's Cloud REST API does not send permissive CORS headers for +// arbitrary third-party origins doing Basic-auth token requests (unlike +// GitHub/GitLab's APIs, which checkGitHubValidity/checkGitLabValidity call +// directly from the browser). A direct browser fetch to +// https://{site}/rest/api/3/myself fails with a CORS error before the +// request even reaches Jira, regardless of whether the credentials are +// valid. Proxying through our own backend (server-to-server, no CORS) +// sidesteps that -- this route exists only for that reason, and does not +// persist anything; linking still goes through the existing generic +// /api/resources/orgs/[org_id]/integration route unchanged. +endpoint.handle.POST(postSchema, async (req, res) => { + const { site_url, email, api_token } = req.payload; + const domain = normalizeSiteUrl(site_url); + + try { + const response = await axios.get( + `https://${domain}/rest/api/3/myself`, + { + auth: { username: email, password: api_token }, + timeout: 8000 + } + ); + + // CLUSTOX: a 2xx status alone doesn't mean this was actually Jira's + // /myself endpoint -- e.g. id.atlassian.com (a real Atlassian domain, + // just not a Jira Cloud *site*) answers this same path with HTTP 202 + // and an HTML body, which axios treats as success. Every genuine + // Jira /myself response is JSON with an accountId; require that + // instead of trusting the status code alone, or a wrong-but-plausible + // domain reports as "linked" without ever having verified anything. + if (!response.data || typeof response.data !== 'object' || !response.data.accountId) { + return res.send({ valid: false, reason: 'unknown' }); + } + + return res.send({ + valid: true, + display_name: response.data.displayName ?? email + }); + } catch (error: any) { + const status = error?.response?.status; + return res.send({ + valid: false, + // Distinguishes "reached Jira, credentials were wrong" from "that + // site doesn't exist" from "couldn't reach it at all" -- Atlassian's + // edge answers a nonexistent *.atlassian.net subdomain with a JSON + // 404 (not a DNS failure), so a typo'd site shows up as a 404 here, + // not as the network-level "unreachable" case. + reason: + status === 401 || status === 403 + ? 'unauthorized' + : status === 404 || !error?.response + ? 'unreachable' + : 'unknown' + }); + } +}); + +export default endpoint.serve(); diff --git a/web-server/pages/integrations.tsx b/web-server/pages/integrations.tsx index cde7060a..9e59a4fe 100644 --- a/web-server/pages/integrations.tsx +++ b/web-server/pages/integrations.tsx @@ -14,10 +14,12 @@ import { ClustoxJenkinsSetup } from '@/components/ClustoxJenkinsSetup'; // END CLUSTOX import { FlexBox } from '@/components/FlexBox'; import { Line } from '@/components/Text'; +import { CODE_PROVIDERS } from '@/constants/codeProviders'; import { ROUTES } from '@/constants/routes'; import { FetchState } from '@/constants/ui-states'; import { GithubIntegrationCard } from '@/content/Dashboards/GithubIntegrationCard'; import { GitlabIntegrationCard } from '@/content/Dashboards/GitlabIntegrationCard'; +import { JiraIntegrationCard } from '@/content/Dashboards/JiraIntegrationCard'; import { PageWrapper } from '@/content/PullRequests/PageWrapper'; import { useModal } from '@/contexts/ModalContext'; import { useAuth } from '@/hooks/useAuth'; @@ -65,7 +67,14 @@ export default Integrations; const Content = () => { const { orgId, integrations, integrationList } = useAuth(); - const hasCodeProviderLinked = integrationList.length > 0; + const codeProviderList = useMemo( + () => + integrationList.filter((item) => + (CODE_PROVIDERS as readonly string[]).includes(item) + ), + [integrationList] + ); + const hasCodeProviderLinked = codeProviderList.length > 0; const teamCount = useSelector((s) => s.team.teams?.length); const dispatch = useDispatch(); const loadedTeams = useBoolState(false); @@ -75,7 +84,7 @@ const Content = () => { hasCodeProviderLinked && !teamCount && loadedTeams.value; const lastSyncMap = useMemo(() => { - return integrationList + return codeProviderList .map((item) => { const linkedAt = integrations[item as 'github' | 'gitlab' | 'bitbucket'].linked_at; @@ -89,7 +98,7 @@ const Content = () => { return diff; }) .filter(Boolean); - }, [integrationList, integrations]); + }, [codeProviderList, integrations]); const showForceSyncBtn = useMemo(() => { if (!hasCodeProviderLinked) return false; @@ -166,9 +175,10 @@ const Content = () => { - + + {/* CLUSTOX: Jenkins as a third deployment-detection provider. */} {/* END CLUSTOX */} diff --git a/web-server/src/constants/__tests__/codeProviders.test.ts b/web-server/src/constants/__tests__/codeProviders.test.ts new file mode 100644 index 00000000..e1722458 --- /dev/null +++ b/web-server/src/constants/__tests__/codeProviders.test.ts @@ -0,0 +1,21 @@ +import { CODE_PROVIDERS } from '@/constants/codeProviders'; + +// CLUSTOX: regression guard for a real bug found while building Jira +// linking. pages/integrations.tsx's hasCodeProviderLinked used to be +// `integrationList.length > 0` -- true for *any* linked integration, not +// specifically a code provider. Once Jira became linkable, linking Jira +// alone would have incorrectly flagged "code provider linked," wrongly +// surfacing the DORA CTA, the force-sync button, and the "create a team" +// prompt for a workspace with no actual code repos. Filtering +// integrationList through this exact list is the fix; this test guards it +// directly so a future non-code integration (Slack, PagerDuty, ...) doesn't +// get added here by mistake the same way. +describe('CODE_PROVIDERS', () => { + it('is exactly the code-hosting providers', () => { + expect(CODE_PROVIDERS).toEqual(['github', 'gitlab', 'bitbucket']); + }); + + it('does not include jira', () => { + expect(CODE_PROVIDERS).not.toContain('jira'); + }); +}); diff --git a/web-server/src/constants/codeProviders.ts b/web-server/src/constants/codeProviders.ts new file mode 100644 index 00000000..af201c8f --- /dev/null +++ b/web-server/src/constants/codeProviders.ts @@ -0,0 +1,11 @@ +// CLUSTOX: extracted so this stays importable on its own, without pulling +// in pages/integrations.tsx's full dependency tree (PageWrapper -> +// ErrorBoundaryFallback -> ... -> a .png import) just for one array -- +// that tree has nothing to do with what this constant is actually for. +// +// integrationList (from useAuth()) is every linked integration, including +// non-code ones like jira. Team creation, the DORA CTA, and force-sync are +// specifically about code providers -- filtering integrationList through +// this list, once, is what keeps that distinction from leaking into every +// computation that follows. See docs/JIRA_INTEGRATION_PROPOSAL.md. +export const CODE_PROVIDERS = ['github', 'gitlab', 'bitbucket'] as const; diff --git a/web-server/src/content/Dashboards/ConfigureJiraModalBody.tsx b/web-server/src/content/Dashboards/ConfigureJiraModalBody.tsx new file mode 100644 index 00000000..b3daaa2c --- /dev/null +++ b/web-server/src/content/Dashboards/ConfigureJiraModalBody.tsx @@ -0,0 +1,170 @@ +import { Link, TextField } from '@mui/material'; +import { LoadingButton } from '@mui/lab'; +import { useSnackbar } from 'notistack'; +import { FC, useCallback } from 'react'; + +import { FlexBox } from '@/components/FlexBox'; +import { Line } from '@/components/Text'; +import { Integration } from '@/constants/integrations'; +import { useAuth } from '@/hooks/useAuth'; +// CLUSTOX: server-resolved workspace, immune to stale persisted redux state. +import { useClustoxUser } from '@/hooks/useClustoxUser'; +import { useBoolState, useEasyState } from '@/hooks/useEasyState'; +import { fetchCurrentOrg } from '@/slices/auth'; +import { useDispatch } from '@/store'; +import { + checkJiraValidity, + linkProvider, + normalizeJiraSiteUrl +} from '@/utils/auth'; +import { depFn } from '@/utils/fn'; + +export const ConfigureJiraModalBody: FC<{ + onClose: () => void; +}> = ({ onClose }) => { + const siteUrl = useEasyState(''); + const email = useEasyState(''); + const token = useEasyState(''); + const { orgId: contextOrgId } = useAuth(); + // CLUSTOX: prefer the server-resolved workspace. The redux copy is persisted + // by redux-persist and can rehydrate empty from a previous session, which + // previously produced a POST to /orgs/undefined/integration. + const { orgId: sessionOrgId } = useClustoxUser(); + const orgId = sessionOrgId ?? contextOrgId; + const { enqueueSnackbar } = useSnackbar(); + const dispatch = useDispatch(); + const isLoading = useBoolState(); + + const showError = useEasyState(''); + + const setError = useCallback( + (error: string) => { + depFn(showError.set, error); + }, + [showError.set] + ); + + const handleSubmission = useCallback(async () => { + if (!siteUrl.value || !email.value || !token.value) { + setError('Please fill in all three fields'); + return; + } + if (!orgId) { + setError('No workspace selected. Reload the page and try again.'); + return; + } + + isLoading.true(); + try { + const result = await checkJiraValidity( + siteUrl.value, + email.value, + token.value + ); + + if (!result.valid) { + setError( + result.reason === 'unauthorized' + ? 'That email/API token combination was rejected by Jira.' + : result.reason === 'unreachable' + ? "Couldn't reach that Jira site — check the URL." + : 'Could not verify this Jira account. Please try again.' + ); + return; + } + + await linkProvider(token.value, orgId, Integration.JIRA, { + site_url: normalizeJiraSiteUrl(siteUrl.value), + email: email.value + }); + + // Await, don't fire-and-forget: closing the modal before this + // resolves would let the Jira card render on the pre-link ("not + // linked") state for a moment -- same race already fixed for + // GitHub/GitLab's own link flow. + await dispatch(fetchCurrentOrg()); + + enqueueSnackbar( + result.displayName + ? `Jira linked — signed in as ${result.displayName}` + : 'Jira linked successfully', + { variant: 'success', autoHideDuration: 3000 } + ); + onClose(); + } catch (e: any) { + setError(e.message || 'Unknown error'); + console.error(e); + } finally { + isLoading.false(); + } + }, [ + siteUrl.value, + email.value, + token.value, + dispatch, + enqueueSnackbar, + isLoading.false, + isLoading.true, + onClose, + orgId, + setError + ]); + + return ( + + Enter your Jira workspace details below. + + siteUrl.set(e.target.value)} + InputLabelProps={{ shrink: true }} + /> + email.set(e.target.value)} + InputLabelProps={{ shrink: true }} + /> + + token.set(e.target.value)} + // Browsers apply their password-autofill heuristics to any + // type="password" field, token or not. Keeping the label + // permanently shrunk removes any race between that and React + // learning the value. + InputLabelProps={{ shrink: true }} + /> + + Generate one at{' '} + + id.atlassian.com/manage-profile/security + + + + + + {showError.value} + + + + + Verify & link + + + + ); +}; diff --git a/web-server/src/content/Dashboards/JiraIntegrationCard.tsx b/web-server/src/content/Dashboards/JiraIntegrationCard.tsx new file mode 100644 index 00000000..96f84d8f --- /dev/null +++ b/web-server/src/content/Dashboards/JiraIntegrationCard.tsx @@ -0,0 +1,253 @@ +import { + ArrowForwardIosRounded, + ChevronRightRounded, + SettingsRounded +} from '@mui/icons-material'; +import { Button, useTheme } from '@mui/material'; +import CircularProgress from '@mui/material/CircularProgress'; +import { useSnackbar } from 'notistack'; +import { FC, ReactNode, useEffect } from 'react'; + +import { FlexBox } from '@/components/FlexBox'; +import { Line } from '@/components/Text'; +import { track } from '@/constants/events'; +import { FetchState } from '@/constants/ui-states'; +import { jiraIntegrationDisplay } from '@/content/Dashboards/githubIntegration'; +import { useIntegrationHandlers } from '@/content/Dashboards/useIntegrationHandlers'; +import { useAuth } from '@/hooks/useAuth'; +import { useBoolState } from '@/hooks/useEasyState'; +import { fetchCurrentOrg } from '@/slices/auth'; +import { useDispatch, useSelector } from '@/store'; + +const cardRadius = 10.5; +const cardBorder = 1.5; +const getRadiusWithPadding = (radius: number, padding: number) => + `${radius + padding}px`; + +// CLUSTOX: mirrors GithubIntegrationCard/GitlabIntegrationCard exactly -- +// same sliceLoading source, same layout, same unlink confirmation flow. +// Jira Phase 1 (see docs/JIRA_INTEGRATION_PROPOSAL.md) is link-only: no +// sync, no project picker yet, so there's nothing else for this card to do +// beyond reflect integrations.jira. +export const JiraIntegrationCard = () => { + const theme = useTheme(); + const { integrations } = useAuth(); + const isJiraIntegrated = integrations.jira; + const sliceLoading = useSelector( + (s) => s.auth.requests.org === FetchState.REQUEST + ); + const { link, unlink } = useIntegrationHandlers(); + + const localLoading = useBoolState(false); + + const isLoading = sliceLoading || localLoading.value; + + const dispatch = useDispatch(); + + const { enqueueSnackbar } = useSnackbar(); + + return ( + + {isJiraIntegrated && ( + + + + )} + + + + + + {jiraIntegrationDisplay.icon} + + + {jiraIntegrationDisplay.name} + + + + { + track( + isJiraIntegrated + ? 'INTEGRATION_UNLINK_TRIGGERED' + : 'INTEGRATION_LINK_TRIGGERED', + { integration_name: jiraIntegrationDisplay.name } + ); + if (!isJiraIntegrated) { + link.jira(); + return; + } + const shouldExecute = window.confirm( + 'Are you sure you want to unlink?' + ); + if (shouldExecute) { + localLoading.true(); + await unlink + .jira() + .then(() => { + enqueueSnackbar('Jira unlinked successfully', { + variant: 'success' + }); + }) + .then(async () => dispatch(fetchCurrentOrg())) + .catch((e) => { + console.error('Failed to unlink Jira', e); + enqueueSnackbar('Failed to unlink Jira', { + variant: 'error' + }); + }) + .finally(localLoading.false); + } + }} + label={!isJiraIntegrated ? 'Link' : 'Unlink'} + bgOpacity={!isJiraIntegrated ? 0.45 : 0.25} + endIcon={ + isLoading ? ( + + ) : ( + + ) + } + minWidth="72px" + /> + + + + + ); +}; + +const IntegrationActionsButton: FC<{ + onClick: AnyFunction; + label: ReactNode; + bgOpacity?: number; + startIcon?: ReactNode; + endIcon?: ReactNode; + minWidth?: string; +}> = ({ + label, + onClick, + bgOpacity = 0.45, + endIcon = ( + + ), + startIcon = , + minWidth = '80px' +}) => { + const theme = useTheme(); + + return ( + + ); +}; + +const LinkedIcon = () => { + const isVisible = useBoolState(false); + useEffect(() => { + setTimeout(isVisible.true, 200); + }, [isVisible.true]); + return ( + + + + + + + + + + + ); +}; diff --git a/web-server/src/content/Dashboards/__tests__/ConfigureJiraModalBody.test.tsx b/web-server/src/content/Dashboards/__tests__/ConfigureJiraModalBody.test.tsx new file mode 100644 index 00000000..f3ec7ce1 --- /dev/null +++ b/web-server/src/content/Dashboards/__tests__/ConfigureJiraModalBody.test.tsx @@ -0,0 +1,188 @@ +jest.mock('@/hooks/useAuth', () => ({ useAuth: jest.fn() })); +jest.mock('@/hooks/useClustoxUser', () => ({ useClustoxUser: jest.fn() })); +jest.mock('@/store', () => ({ useDispatch: jest.fn() })); +jest.mock('notistack', () => ({ useSnackbar: jest.fn() })); +jest.mock('@/slices/auth', () => ({ fetchCurrentOrg: jest.fn(() => ({ type: 'mock' })) })); +jest.mock('@/utils/auth', () => ({ + checkJiraValidity: jest.fn(), + linkProvider: jest.fn(), + normalizeJiraSiteUrl: jest.requireActual('@/utils/auth').normalizeJiraSiteUrl +})); + +import { render, screen, waitFor } from '@testing-library/react'; +import userEvent from '@testing-library/user-event'; +import { useSnackbar } from 'notistack'; + +import { ConfigureJiraModalBody } from '@/content/Dashboards/ConfigureJiraModalBody'; +import { useAuth } from '@/hooks/useAuth'; +import { useClustoxUser } from '@/hooks/useClustoxUser'; +import { useDispatch } from '@/store'; +import { checkJiraValidity, linkProvider } from '@/utils/auth'; + +const fillForm = async ( + siteUrl = 'mycompany.atlassian.net', + email = 'jordan@mycompany.com', + token = 'a-token' +) => { + const user = userEvent.setup(); + if (siteUrl) await user.type(screen.getByLabelText('Jira Site URL'), siteUrl); + if (email) await user.type(screen.getByLabelText('Email'), email); + if (token) await user.type(screen.getByLabelText('API Token'), token); + await user.click(screen.getByRole('button', { name: /verify.*link/i })); +}; + +describe('ConfigureJiraModalBody', () => { + const onClose = jest.fn(); + const dispatch = jest.fn(() => Promise.resolve()); + const enqueueSnackbar = jest.fn(); + + beforeEach(() => { + jest.clearAllMocks(); + (useAuth as jest.Mock).mockReturnValue({ orgId: 'org-1' }); + (useClustoxUser as jest.Mock).mockReturnValue({ orgId: null }); + (useDispatch as jest.Mock).mockReturnValue(dispatch); + (useSnackbar as jest.Mock).mockReturnValue({ enqueueSnackbar }); + dispatch.mockReturnValue(Promise.resolve()); + }); + + it('requires all three fields before attempting to verify anything', async () => { + render(); + await fillForm('', '', ''); + + expect( + screen.getByText('Please fill in all three fields') + ).toBeInTheDocument(); + expect(checkJiraValidity).not.toHaveBeenCalled(); + }); + + it('refuses to submit with no workspace resolved, rather than link an undefined org', async () => { + (useAuth as jest.Mock).mockReturnValue({ orgId: null }); + (useClustoxUser as jest.Mock).mockReturnValue({ orgId: null }); + render(); + await fillForm(); + + expect( + screen.getByText('No workspace selected. Reload the page and try again.') + ).toBeInTheDocument(); + expect(checkJiraValidity).not.toHaveBeenCalled(); + }); + + it("prefers the server-resolved workspace (useClustoxUser) over the redux-persisted one", async () => { + (useAuth as jest.Mock).mockReturnValue({ orgId: 'stale-redux-org' }); + (useClustoxUser as jest.Mock).mockReturnValue({ orgId: 'fresh-server-org' }); + (checkJiraValidity as jest.Mock).mockResolvedValue({ valid: true }); + (linkProvider as jest.Mock).mockResolvedValue(undefined); + render(); + + await fillForm(); + + await waitFor(() => + expect(linkProvider).toHaveBeenCalledWith( + 'a-token', + 'fresh-server-org', + 'jira', + expect.anything() + ) + ); + }); + + it.each([ + ['unauthorized', 'That email/API token combination was rejected by Jira.'], + ['unreachable', "Couldn't reach that Jira site — check the URL."], + ['unknown', 'Could not verify this Jira account. Please try again.'] + ])('shows the right message for reason=%s, and never links', async (reason, message) => { + (checkJiraValidity as jest.Mock).mockResolvedValue({ valid: false, reason }); + render(); + + await fillForm(); + + expect(await screen.findByText(message)).toBeInTheDocument(); + expect(linkProvider).not.toHaveBeenCalled(); + expect(onClose).not.toHaveBeenCalled(); + }); + + it('normalizes the site URL before storing it, even though it validates the raw input', async () => { + (checkJiraValidity as jest.Mock).mockResolvedValue({ valid: true }); + (linkProvider as jest.Mock).mockResolvedValue(undefined); + render(); + + await fillForm('https://mycompany.atlassian.net/', 'jordan@mycompany.com', 'tok'); + + await waitFor(() => + expect(checkJiraValidity).toHaveBeenCalledWith( + 'https://mycompany.atlassian.net/', // raw input, unmodified + 'jordan@mycompany.com', + 'tok' + ) + ); + await waitFor(() => + expect(linkProvider).toHaveBeenCalledWith( + 'tok', + 'org-1', + 'jira', + { site_url: 'mycompany.atlassian.net', email: 'jordan@mycompany.com' } // normalized + ) + ); + }); + + it('awaits fetchCurrentOrg before closing -- the card behind this modal reads Redux directly', async () => { + const order: string[] = []; + (checkJiraValidity as jest.Mock).mockResolvedValue({ valid: true }); + (linkProvider as jest.Mock).mockResolvedValue(undefined); + dispatch.mockImplementation(() => { + order.push('fetchCurrentOrg'); + return Promise.resolve(); + }); + onClose.mockImplementation(() => order.push('onClose')); + render(); + + await fillForm(); + + await waitFor(() => expect(onClose).toHaveBeenCalled()); + expect(order).toEqual(['fetchCurrentOrg', 'onClose']); + }); + + it('shows the signed-in-as name on success when Jira returned one', async () => { + (checkJiraValidity as jest.Mock).mockResolvedValue({ + valid: true, + displayName: 'Jordan Diaz' + }); + (linkProvider as jest.Mock).mockResolvedValue(undefined); + render(); + + await fillForm(); + + await waitFor(() => + expect(enqueueSnackbar).toHaveBeenCalledWith( + 'Jira linked — signed in as Jordan Diaz', + expect.objectContaining({ variant: 'success' }) + ) + ); + }); + + it('falls back to a generic success message when Jira did not return a display name', async () => { + (checkJiraValidity as jest.Mock).mockResolvedValue({ valid: true }); + (linkProvider as jest.Mock).mockResolvedValue(undefined); + render(); + + await fillForm(); + + await waitFor(() => + expect(enqueueSnackbar).toHaveBeenCalledWith( + 'Jira linked successfully', + expect.objectContaining({ variant: 'success' }) + ) + ); + }); + + it('shows the error message and does not close when linkProvider itself throws', async () => { + (checkJiraValidity as jest.Mock).mockResolvedValue({ valid: true }); + (linkProvider as jest.Mock).mockRejectedValue(new Error('org already has jira')); + render(); + + await fillForm(); + + expect(await screen.findByText('org already has jira')).toBeInTheDocument(); + expect(onClose).not.toHaveBeenCalled(); + }); +}); diff --git a/web-server/src/content/Dashboards/__tests__/JiraIntegrationCard.test.tsx b/web-server/src/content/Dashboards/__tests__/JiraIntegrationCard.test.tsx new file mode 100644 index 00000000..67aa061f --- /dev/null +++ b/web-server/src/content/Dashboards/__tests__/JiraIntegrationCard.test.tsx @@ -0,0 +1,114 @@ +jest.mock('@/hooks/useAuth', () => ({ useAuth: jest.fn() })); +jest.mock('@/content/Dashboards/useIntegrationHandlers', () => ({ + useIntegrationHandlers: jest.fn() +})); +jest.mock('@/store', () => ({ + useDispatch: jest.fn(), + useSelector: jest.fn() +})); +jest.mock('notistack', () => ({ useSnackbar: jest.fn() })); +jest.mock('@/slices/auth', () => ({ fetchCurrentOrg: jest.fn(() => ({ type: 'mock' })) })); + +import { screen } from '@testing-library/react'; +import userEvent from '@testing-library/user-event'; +import { useSnackbar } from 'notistack'; + +import { JiraIntegrationCard } from '@/content/Dashboards/JiraIntegrationCard'; +import { useIntegrationHandlers } from '@/content/Dashboards/useIntegrationHandlers'; +import { useAuth } from '@/hooks/useAuth'; +import { useDispatch, useSelector } from '@/store'; +import { renderWithTheme as render } from '@/utils/testUtils'; + +const link = { jira: jest.fn() }; +const unlink = { jira: jest.fn().mockResolvedValue(undefined) }; + +describe('JiraIntegrationCard', () => { + beforeEach(() => { + jest.clearAllMocks(); + (useIntegrationHandlers as jest.Mock).mockReturnValue({ link, unlink }); + (useSelector as jest.Mock).mockReturnValue(false); // requests.org !== REQUEST + (useDispatch as jest.Mock).mockReturnValue(jest.fn()); + (useSnackbar as jest.Mock).mockReturnValue({ enqueueSnackbar: jest.fn() }); + }); + + it('shows "Link" and no Linked badge when integrations.jira is falsy', () => { + (useAuth as jest.Mock).mockReturnValue({ integrations: {} }); + render(); + + expect(screen.getByText('Link')).toBeInTheDocument(); + expect(screen.queryByTestId('jira-linked-badge')).not.toBeInTheDocument(); + }); + + it('shows "Unlink" and the Linked badge when integrations.jira is truthy', () => { + (useAuth as jest.Mock).mockReturnValue({ + integrations: { jira: { integrated: true } } + }); + render(); + + expect(screen.getByText('Unlink')).toBeInTheDocument(); + expect(screen.getByTestId('jira-linked-badge')).toBeInTheDocument(); + }); + + it('opens the Configure Jira modal via link.jira() when not yet linked', async () => { + (useAuth as jest.Mock).mockReturnValue({ integrations: {} }); + render(); + + await userEvent.click(screen.getByText('Link')); + + expect(link.jira).toHaveBeenCalledTimes(1); + expect(unlink.jira).not.toHaveBeenCalled(); + }); + + it('asks for confirmation before unlinking, and does nothing if declined', async () => { + (useAuth as jest.Mock).mockReturnValue({ + integrations: { jira: { integrated: true } } + }); + const confirmSpy = jest.spyOn(window, 'confirm').mockReturnValue(false); + render(); + + await userEvent.click(screen.getByText('Unlink')); + + expect(confirmSpy).toHaveBeenCalled(); + expect(unlink.jira).not.toHaveBeenCalled(); + confirmSpy.mockRestore(); + }); + + it('unlinks and shows a success snackbar when confirmed', async () => { + (useAuth as jest.Mock).mockReturnValue({ + integrations: { jira: { integrated: true } } + }); + const enqueueSnackbar = jest.fn(); + (useSnackbar as jest.Mock).mockReturnValue({ enqueueSnackbar }); + jest.spyOn(window, 'confirm').mockReturnValue(true); + render(); + + await userEvent.click(screen.getByText('Unlink')); + + expect(unlink.jira).toHaveBeenCalledTimes(1); + expect(enqueueSnackbar).toHaveBeenCalledWith( + 'Jira unlinked successfully', + expect.objectContaining({ variant: 'success' }) + ); + }); + + it('shows a failure snackbar, not a thrown error, when unlink rejects', async () => { + (useAuth as jest.Mock).mockReturnValue({ + integrations: { jira: { integrated: true } } + }); + unlink.jira.mockRejectedValueOnce(new Error('network down')); + const enqueueSnackbar = jest.fn(); + (useSnackbar as jest.Mock).mockReturnValue({ enqueueSnackbar }); + jest.spyOn(window, 'confirm').mockReturnValue(true); + render(); + + await userEvent.click(screen.getByText('Unlink')); + + expect( + await screen.findByText((t) => t === 'Unlink') + ).toBeInTheDocument(); + expect(enqueueSnackbar).toHaveBeenCalledWith( + 'Failed to unlink Jira', + expect.objectContaining({ variant: 'error' }) + ); + }); +}); diff --git a/web-server/src/content/Dashboards/githubIntegration.tsx b/web-server/src/content/Dashboards/githubIntegration.tsx index 4a6cad42..4e5a7e00 100644 --- a/web-server/src/content/Dashboards/githubIntegration.tsx +++ b/web-server/src/content/Dashboards/githubIntegration.tsx @@ -2,6 +2,7 @@ import faker from '@faker-js/faker'; import { GitHub } from '@mui/icons-material'; import GitlabIcon from '@/mocks/icons/gitlab.svg'; +import JiraIcon from '@/mocks/icons/jira-icon.svg'; export const githubIntegrationsDisplay = { id: faker.datatype.uuid(), @@ -23,4 +24,17 @@ export const gitLabIntegrationDisplay = { icon: } as IntegrationItem; +// CLUSTOX: Jira -- a project-tracker integration, not a code provider (see +// docs/JIRA_INTEGRATION_PROPOSAL.md). Phase 1 only links and stores the +// token; ticket sync and PR correlation are later phases. +export const jiraIntegrationDisplay = { + id: '8f0a3f2e-2f1d-4b1a-9b6b-6a2b8f6e9c4d', + type: 'jira', + name: 'Jira', + description: 'Ticket-linked cycle time', + color: '#2684FF', + bg: 'linear-gradient(135deg, hsla(212, 96%, 55%, 0.6) 0%, hsla(217, 100%, 40%, 0.6) 100%)', + icon: +} as IntegrationItem; + export type IntegrationItem = typeof githubIntegrationsDisplay; diff --git a/web-server/src/content/Dashboards/useIntegrationHandlers.tsx b/web-server/src/content/Dashboards/useIntegrationHandlers.tsx index 0ded2063..9f2b4b4d 100644 --- a/web-server/src/content/Dashboards/useIntegrationHandlers.tsx +++ b/web-server/src/content/Dashboards/useIntegrationHandlers.tsx @@ -2,6 +2,7 @@ import { useMemo } from 'react'; import { Integration } from '@/constants/integrations'; import { ConfigureGitlabModalBody } from '@/content/Dashboards/ConfigureGitlabModalBody'; +import { ConfigureJiraModalBody } from '@/content/Dashboards/ConfigureJiraModalBody'; import { useModal } from '@/contexts/ModalContext'; import { useAuth } from '@/hooks/useAuth'; import { unlinkProvider } from '@/utils/auth'; @@ -27,11 +28,18 @@ export const useIntegrationHandlers = () => { title: 'Configure Gitlab', body: , showCloseIcon: true + }), + jira: () => + addModal({ + title: 'Configure Jira', + body: , + showCloseIcon: true }) }, unlink: { github: () => unlinkProvider(orgId, Integration.GITHUB), - gitlab: () => unlinkProvider(orgId, Integration.GITLAB) + gitlab: () => unlinkProvider(orgId, Integration.GITLAB), + jira: () => unlinkProvider(orgId, Integration.JIRA) } }; diff --git a/web-server/src/contexts/ThirdPartyAuthContext.tsx b/web-server/src/contexts/ThirdPartyAuthContext.tsx index 687fde5e..f5717e4b 100644 --- a/web-server/src/contexts/ThirdPartyAuthContext.tsx +++ b/web-server/src/contexts/ThirdPartyAuthContext.tsx @@ -125,9 +125,15 @@ export const AuthProvider: FC = (props) => { (integrations.github || integrations.gitlab) && IntegrationGroup.CODE ) + // CLUSTOX: Jira is a project-tracker integration, not a code + // provider -- see docs/JIRA_INTEGRATION_PROPOSAL.md. Kept as its + // own concat term rather than folded into the CODE check above, + // same shape as that check, so the two groups stay independently + // readable as the set of integration kinds grows. + .concat(integrations.jira && IntegrationGroup.PROJECT) .filter(Boolean) ), - [integrations.github, integrations.gitlab] + [integrations.github, integrations.gitlab, integrations.jira] ); // CLUSTOX: consumers of integrationList/activeCodeProvider (DORA metrics diff --git a/web-server/src/mocks/__mocks__/svg.js b/web-server/src/mocks/__mocks__/svg.js new file mode 100644 index 00000000..21c14758 --- /dev/null +++ b/web-server/src/mocks/__mocks__/svg.js @@ -0,0 +1,7 @@ +// Next's build treats .svg imports as React components via an SVGR-style +// loader; Jest has no such transform and chokes trying to parse raw SVG/XML +// as JS. Stubbed to a trivial component -- nothing under test asserts on +// icon markup itself. +const React = require('react'); +module.exports = (props) => React.createElement('svg', props); +module.exports.default = module.exports; diff --git a/web-server/src/utils/__tests__/auth-jira.test.ts b/web-server/src/utils/__tests__/auth-jira.test.ts new file mode 100644 index 00000000..398cccd0 --- /dev/null +++ b/web-server/src/utils/__tests__/auth-jira.test.ts @@ -0,0 +1,99 @@ +jest.mock('axios'); + +import axios from 'axios'; + +import { checkJiraValidity, normalizeJiraSiteUrl } from '@/utils/auth'; + +describe('normalizeJiraSiteUrl', () => { + it('strips a leading https:// and trailing slash', () => { + expect(normalizeJiraSiteUrl('https://mycompany.atlassian.net/')).toBe( + 'mycompany.atlassian.net' + ); + }); + + it('strips a leading http:// too', () => { + expect(normalizeJiraSiteUrl('http://mycompany.atlassian.net')).toBe( + 'mycompany.atlassian.net' + ); + }); + + it('leaves a bare domain unchanged', () => { + expect(normalizeJiraSiteUrl('mycompany.atlassian.net')).toBe( + 'mycompany.atlassian.net' + ); + }); + + it('trims surrounding whitespace', () => { + expect(normalizeJiraSiteUrl(' mycompany.atlassian.net ')).toBe( + 'mycompany.atlassian.net' + ); + }); + + it('strips multiple trailing slashes', () => { + expect(normalizeJiraSiteUrl('https://mycompany.atlassian.net///')).toBe( + 'mycompany.atlassian.net' + ); + }); +}); + +describe('checkJiraValidity', () => { + beforeEach(() => jest.clearAllMocks()); + + it('posts the three fields to the server-side validate endpoint', async () => { + (axios.post as jest.Mock).mockResolvedValue({ + data: { valid: true, display_name: 'Jordan Diaz' } + }); + + await checkJiraValidity('mycompany.atlassian.net', 'jordan@mycompany.com', 'tok'); + + expect(axios.post).toHaveBeenCalledWith( + '/api/integrations/jira/validate', + { + site_url: 'mycompany.atlassian.net', + email: 'jordan@mycompany.com', + api_token: 'tok' + } + ); + }); + + it('maps a valid response through, including the display name', async () => { + (axios.post as jest.Mock).mockResolvedValue({ + data: { valid: true, display_name: 'Jordan Diaz' } + }); + + await expect( + checkJiraValidity('mycompany.atlassian.net', 'a@b.com', 'tok') + ).resolves.toEqual({ + valid: true, + displayName: 'Jordan Diaz', + reason: undefined + }); + }); + + it('maps an invalid response through, including the reason', async () => { + (axios.post as jest.Mock).mockResolvedValue({ + data: { valid: false, reason: 'unauthorized' } + }); + + await expect( + checkJiraValidity('mycompany.atlassian.net', 'a@b.com', 'wrong') + ).resolves.toEqual({ + valid: false, + displayName: undefined, + reason: 'unauthorized' + }); + }); + + // CLUSTOX: unlike checkGitHubValidity/checkGitLabValidity (which hit the + // provider's API directly from the browser), this call goes through our + // own backend -- so a thrown error here means *our* server was + // unreachable, not Jira. Must still resolve, not throw, so the modal's + // calling code doesn't need a second try/catch layer on top of its own. + it('resolves to an unknown-reason failure, rather than throwing, if our own backend request fails', async () => { + (axios.post as jest.Mock).mockRejectedValue(new Error('network down')); + + await expect( + checkJiraValidity('mycompany.atlassian.net', 'a@b.com', 'tok') + ).resolves.toEqual({ valid: false, reason: 'unknown' }); + }); +}); diff --git a/web-server/src/utils/auth.ts b/web-server/src/utils/auth.ts index 22852b37..01b2e548 100644 --- a/web-server/src/utils/auth.ts +++ b/web-server/src/utils/auth.ts @@ -99,3 +99,48 @@ export const getMissingGitLabScopes = (scopes: string[]): string[] => { ); return missingScopes; }; + +// Jira functions + +// Strip any protocol/trailing slash so "mycompany.atlassian.net" and +// "https://mycompany.atlassian.net/" are stored the same way. Mirrors the +// normalizer in pages/api/integrations/jira/validate.ts -- kept as two +// small copies rather than a shared import across the client/server +// boundary, same as this file's other provider helpers. +export const normalizeJiraSiteUrl = (input: string) => + input + .trim() + .replace(/^https?:\/\//i, '') + .replace(/\/+$/, ''); + +export type JiraValidityResult = { + valid: boolean; + displayName?: string; + reason?: 'unauthorized' | 'unreachable' | 'unknown'; +}; + +// CLUSTOX: unlike checkGitHubValidity/checkGitLabValidity, this cannot call +// Jira's API directly from the browser -- Jira Cloud doesn't send CORS +// headers permissive enough for a third-party origin's Basic-auth request, +// so it's proxied through our own backend instead (see +// pages/api/integrations/jira/validate.ts for why). +export const checkJiraValidity = async ( + siteUrl: string, + email: string, + apiToken: string +): Promise => { + try { + const { data } = await axios.post('/api/integrations/jira/validate', { + site_url: siteUrl, + email, + api_token: apiToken + }); + return { valid: data.valid, displayName: data.display_name, reason: data.reason }; + } catch (error) { + return { valid: false, reason: 'unknown' }; + } +}; + +// Jira API tokens inherit the full permissions of the account they belong +// to -- there's no OAuth-style scope grant to inspect the way there is for +// a GitHub/GitLab PAT, so there's no getMissingJiraScopes equivalent. diff --git a/web-server/src/utils/testUtils.tsx b/web-server/src/utils/testUtils.tsx new file mode 100644 index 00000000..10159854 --- /dev/null +++ b/web-server/src/utils/testUtils.tsx @@ -0,0 +1,17 @@ +import { render, RenderOptions } from '@testing-library/react'; +import { ReactElement } from 'react'; + +import ThemeProviderWrapper from '@/theme/ThemeProvider'; + +// CLUSTOX: the app always renders inside this ThemeProvider (see +// pages/_app.tsx) -- it's what supplies theme.colors, theme.header, etc. on +// top of bare MUI, which several components reach for even indirectly (a +// FlexBox's `title` prop renders a MUI Tooltip via Shared.tsx's +// TooltipWrapper, which reads theme.colors.alpha). Without this wrapper, +// useTheme() falls back to MUI's bare default theme and anything touching +// those custom fields throws. First hit by JiraIntegrationCard's Linked +// badge. +export const renderWithTheme = ( + ui: ReactElement, + options?: Omit +) => render(ui, { wrapper: ThemeProviderWrapper, ...options }); diff --git a/web-server/yarn.lock b/web-server/yarn.lock index 07d31dd2..0e906942 100644 --- a/web-server/yarn.lock +++ b/web-server/yarn.lock @@ -7,6 +7,11 @@ resolved "https://registry.yarnpkg.com/@aashutoshrathi/word-wrap/-/word-wrap-1.2.6.tgz#bd9154aec9983f77b3a034ecaa015c2e4201f6cf" integrity sha512-1Yjs2SvM8TflER/OD3cOjhWWOZb58A2t7wpE2S9XfBYTiIl+XFhQG2bjy4Pu1I+EAlCNUzRDYDdFwFYUKvXcIA== +"@adobe/css-tools@^4.4.0": + version "4.5.0" + resolved "https://registry.yarnpkg.com/@adobe/css-tools/-/css-tools-4.5.0.tgz#b5b71a25a4d16afa2482592ddfa62fccc60bc7d1" + integrity sha512-6OzddxPio9UiWTCemp4N8cYLV2ZN1ncRnV1cVGtve7dhPOtRkleRyx32GQCYSwDYgaHU3USMm84tNsvKzRCa1Q== + "@ampproject/remapping@^2.2.0": version "2.2.1" resolved "https://registry.yarnpkg.com/@ampproject/remapping/-/remapping-2.2.1.tgz#99e8e11851128b8702cd57c33684f1d0f260b630" @@ -23,6 +28,15 @@ "@babel/highlight" "^7.22.13" chalk "^2.4.2" +"@babel/code-frame@^7.10.4": + version "7.29.7" + resolved "https://registry.yarnpkg.com/@babel/code-frame/-/code-frame-7.29.7.tgz#f2fbbfea87c44a21590ec515b778b2c26d8866e7" + integrity sha512-Aup7aUOfpbAUg2ROOJN6Iw5f9DMBlzu0mIkm/malLQFN/YQgO48wCj0Kxa3sEHJvPVFg7siR+qRInwXd2qhQKw== + dependencies: + "@babel/helper-validator-identifier" "^7.29.7" + js-tokens "^4.0.0" + picocolors "^1.1.1" + "@babel/code-frame@^7.25.7": version "7.25.7" resolved "https://registry.yarnpkg.com/@babel/code-frame/-/code-frame-7.25.7.tgz#438f2c524071531d643c6f0188e1e28f130cebc7" @@ -263,6 +277,11 @@ resolved "https://registry.yarnpkg.com/@babel/helper-validator-identifier/-/helper-validator-identifier-7.25.7.tgz#77b7f60c40b15c97df735b38a66ba1d7c3e93da5" integrity sha512-AM6TzwYqGChO45oiuPqwL2t20/HdMC1rTPAesnBCgPCSF1x3oN9MVUwQV2iyz4xqWrctwK5RNC8LV22kaQCNYg== +"@babel/helper-validator-identifier@^7.29.7": + version "7.29.7" + resolved "https://registry.yarnpkg.com/@babel/helper-validator-identifier/-/helper-validator-identifier-7.29.7.tgz#bd87084ced0c796ec46bda492de6e83d29e89fc2" + integrity sha512-qehxGkRj55h/ff8EMaJ+cYhyaKlHIxqYDn682wQD7RNp9UujOQsHog2uS0r2vzr4pW+sXf90NeeayjcNaX3fFg== + "@babel/helper-validator-option@^7.16.7", "@babel/helper-validator-option@^7.22.15": version "7.22.15" resolved "https://registry.yarnpkg.com/@babel/helper-validator-option/-/helper-validator-option-7.22.15.tgz#694c30dfa1d09a6534cdfcafbe56789d36aba040" @@ -2468,6 +2487,46 @@ dependencies: tslib "^2.4.0" +"@testing-library/dom@^9.0.0": + version "9.3.4" + resolved "https://registry.yarnpkg.com/@testing-library/dom/-/dom-9.3.4.tgz#50696ec28376926fec0a1bf87d9dbac5e27f60ce" + integrity sha512-FlS4ZWlp97iiNWig0Muq8p+3rVDjRiYE+YKGbAqXOu9nwJFFOdL00kFpz42M+4huzYi86vAK1sOOfyOG45muIQ== + dependencies: + "@babel/code-frame" "^7.10.4" + "@babel/runtime" "^7.12.5" + "@types/aria-query" "^5.0.1" + aria-query "5.1.3" + chalk "^4.1.0" + dom-accessibility-api "^0.5.9" + lz-string "^1.5.0" + pretty-format "^27.0.2" + +"@testing-library/jest-dom@^6.4.2": + version "6.10.0" + resolved "https://registry.yarnpkg.com/@testing-library/jest-dom/-/jest-dom-6.10.0.tgz#8a76841e94b72d55d09d2a34b9db9d75da9cbc08" + integrity sha512-HQwu0KaB2zyT0iLzBL+8CLyZDL3KlZlZJ+2iyc9uCUnlJVskJU/UlPuVCyIPhtukjPQdT2QNoR5nCP5FqTmmDQ== + dependencies: + "@adobe/css-tools" "^4.4.0" + aria-query "^5.0.0" + css.escape "^1.5.1" + dom-accessibility-api "^0.6.3" + picocolors "^1.1.1" + redent "^3.0.0" + +"@testing-library/react@^14.2.1": + version "14.3.1" + resolved "https://registry.yarnpkg.com/@testing-library/react/-/react-14.3.1.tgz#29513fc3770d6fb75245c4e1245c470e4ffdd830" + integrity sha512-H99XjUhWQw0lTgyMN05W3xQG1Nh4lq574D8keFf1dDoNTJgp66VbJozRaczoF+wsiaPJNt/TcnfpLGufGxSrZQ== + dependencies: + "@babel/runtime" "^7.12.5" + "@testing-library/dom" "^9.0.0" + "@types/react-dom" "^18.0.0" + +"@testing-library/user-event@^14.5.2": + version "14.6.3" + resolved "https://registry.yarnpkg.com/@testing-library/user-event/-/user-event-14.6.3.tgz#360bd2e2f496ba738dfd59f114368dc479199a7f" + integrity sha512-6dBq67jT8lE+JTE8Exm02Kt6ze43hz1jdiSpSJwtTZiT1xQQ6b7nZYTTQ9njdArdU8XklOwaDp/AbT/eYSKF4g== + "@tootallnate/once@1": version "1.1.2" resolved "https://registry.yarnpkg.com/@tootallnate/once/-/once-1.1.2.tgz#ccb91445360179a04e7fe6aff78c00ffc1eeaf82" @@ -2483,6 +2542,11 @@ resolved "https://registry.yarnpkg.com/@trysound/sax/-/sax-0.2.0.tgz#cccaab758af56761eb7bf37af6f03f326dd798ad" integrity sha512-L7z9BgrNEcYyUYtF+HaEfiS5ebkh9jXqbszz7pC0hRBPaatV0XjSD3+eHrpqFemQfgwiFF0QPIarnIihIDn7OA== +"@types/aria-query@^5.0.1": + version "5.0.4" + resolved "https://registry.yarnpkg.com/@types/aria-query/-/aria-query-5.0.4.tgz#1a31c3d378850d2778dabb6374d036dcba4ba708" + integrity sha512-rfT93uj5s0PRL7EzccGMs3brplhcrghnDoV26NqKhCAS1hVo+WdNsPvE/yb6ilfr5hi2MEk6d5EWJTKdxg8jVw== + "@types/babel__core@^7.1.14": version "7.20.1" resolved "https://registry.yarnpkg.com/@types/babel__core/-/babel__core-7.20.1.tgz#916ecea274b0c776fec721e333e55762d3a9614b" @@ -2690,6 +2754,11 @@ dependencies: "@types/react" "^17" +"@types/react-dom@^18.0.0": + version "18.3.7" + resolved "https://registry.yarnpkg.com/@types/react-dom/-/react-dom-18.3.7.tgz#b89ddf2cd83b4feafcc4e2ea41afdfb95a0d194f" + integrity sha512-MEe3UeoENYVFXzoXEWsvcpg6ZvlrFNlOQ7EOsvhI3CfAXwzPfO8Qwuxd40nepsYKqyyVQnTdEfv68q91yLcKrQ== + "@types/react-redux@7.1.20": version "7.1.20" resolved "https://registry.yarnpkg.com/@types/react-redux/-/react-redux-7.1.20.tgz#42f0e61ababb621e12c66c96dda94c58423bd7df" @@ -3084,7 +3153,14 @@ argparse@^2.0.1: resolved "https://registry.yarnpkg.com/argparse/-/argparse-2.0.1.tgz#246f50f3ca78a3240f6c997e8a9bd1eac49e4b38" integrity sha512-8+9WqebbFzpX9OR+Wa6O29asIogeRMzcGtAINdpMHHyAg10f05aSFVBbcEqGf/PXw1EjAZ+q2/bEBg3DvurK3Q== -aria-query@^5.3.2: +aria-query@5.1.3: + version "5.1.3" + resolved "https://registry.yarnpkg.com/aria-query/-/aria-query-5.1.3.tgz#19db27cd101152773631396f7a95a3b58c22c35e" + integrity sha512-R5iJ5lkuHybztUfuOAznmboyjWq8O6sqNqtK7CLOqdydi54VNbORp49mb14KbWgG1QD3JFO9hJdZ+y4KutfdOQ== + dependencies: + deep-equal "^2.0.5" + +aria-query@^5.0.0, aria-query@^5.3.2: version "5.3.2" resolved "https://registry.yarnpkg.com/aria-query/-/aria-query-5.3.2.tgz#93f81a43480e33a338f19163a3d10a50c01dcd59" integrity sha512-COROpnaoap1E2F000S62r6A60uHZnmlvomhfyT2DlTcrY1OrBKn2UhH7qn5wTC9zMvD0AY7csdPSNwKP+7WiQw== @@ -3530,6 +3606,14 @@ bytes@3.1.2: resolved "https://registry.yarnpkg.com/bytes/-/bytes-3.1.2.tgz#8b0beeb98605adf1b128fa4386403c009e0221a5" integrity sha512-/Nf7TyzTx6S3yRJObOAV7956r8cr2+Oj8AC5dt8wSP3BQAoeX58NoHyCU8P8zGkNXStjTSi6fzO6F0pBdcYbEg== +call-bind-apply-helpers@^1.0.1, call-bind-apply-helpers@^1.0.2: + version "1.0.2" + resolved "https://registry.yarnpkg.com/call-bind-apply-helpers/-/call-bind-apply-helpers-1.0.2.tgz#4b5428c222be985d79c3d82657479dbe0b59b2d6" + integrity sha512-Sp1ablJ0ivDkSzjcaJdxEunN5/XvksFJ2sMBFfq6x0ryhQV/2b/KwFe21cMpmHtPOSij8K99/wSfoEuTObmuMQ== + dependencies: + es-errors "^1.3.0" + function-bind "^1.1.2" + call-bind@^1.0.0, call-bind@^1.0.2, call-bind@^1.0.4, call-bind@^1.0.5: version "1.0.5" resolved "https://registry.yarnpkg.com/call-bind/-/call-bind-1.0.5.tgz#6fa2b7845ce0ea49bf4d8b9ef64727a2c2e2e513" @@ -3550,6 +3634,14 @@ call-bind@^1.0.6, call-bind@^1.0.7: get-intrinsic "^1.2.4" set-function-length "^1.2.1" +call-bound@^1.0.2: + version "1.0.4" + resolved "https://registry.yarnpkg.com/call-bound/-/call-bound-1.0.4.tgz#238de935d2a2a692928c538c7ccfa91067fd062a" + integrity sha512-+ys997U96po4Kx/ABpBCqhA9EuxJaQWDQg7295H4hBphv3IZg0boBKuwYpt4YXp6MZ5AmZQnU/tyMTlRpaSejg== + dependencies: + call-bind-apply-helpers "^1.0.2" + get-intrinsic "^1.3.0" + callsites@^3.0.0: version "3.1.0" resolved "https://registry.yarnpkg.com/callsites/-/callsites-3.1.0.tgz#b3630abd8943432f54b3f0519238e33cd7df2f73" @@ -3941,6 +4033,11 @@ css-what@^6.0.1: resolved "https://registry.yarnpkg.com/css-what/-/css-what-6.1.0.tgz#fb5effcf76f1ddea2c81bdfaa4de44e79bac70f4" integrity sha512-HTUrgRJ7r4dsZKU6GjmpfRK1O76h97Z8MfS1G0FozR+oF2kG6Vfe8JE6zwrkbxigziPHinCJ+gCPjA9EaBDtRw== +css.escape@^1.5.1: + version "1.5.1" + resolved "https://registry.yarnpkg.com/css.escape/-/css.escape-1.5.1.tgz#42e27d4fa04ae32f931a4b4d4191fa9cddee97cb" + integrity sha512-YUifsXXuknHlUsmlgyY0PKzgPOr7/FjCePfHNt0jxm83wHZi44VDMQ7/fGNkjY3/jV1MC+1CmZbaHzugyeRtpg== + csso@^4.2.0: version "4.2.0" resolved "https://registry.yarnpkg.com/csso/-/csso-4.2.0.tgz#ea3a561346e8dc9f546d6febedd50187cf389529" @@ -4155,6 +4252,30 @@ dedent@^1.0.0: resolved "https://registry.yarnpkg.com/dedent/-/dedent-1.5.1.tgz#4f3fc94c8b711e9bb2800d185cd6ad20f2a90aff" integrity sha512-+LxW+KLWxu3HW3M2w2ympwtqPrqYRzU8fqi6Fhd18fBALe15blJPI/I4+UHveMVG6lJqB4JNd4UG0S5cnVHwIg== +deep-equal@^2.0.5: + version "2.2.3" + resolved "https://registry.yarnpkg.com/deep-equal/-/deep-equal-2.2.3.tgz#af89dafb23a396c7da3e862abc0be27cf51d56e1" + integrity sha512-ZIwpnevOurS8bpT4192sqAowWM76JDKSHYzMLty3BZGSswgq6pBaH3DhCSW5xVAZICZyKdOBPjwww5wfgT/6PA== + dependencies: + array-buffer-byte-length "^1.0.0" + call-bind "^1.0.5" + es-get-iterator "^1.1.3" + get-intrinsic "^1.2.2" + is-arguments "^1.1.1" + is-array-buffer "^3.0.2" + is-date-object "^1.0.5" + is-regex "^1.1.4" + is-shared-array-buffer "^1.0.2" + isarray "^2.0.5" + object-is "^1.1.5" + object-keys "^1.1.1" + object.assign "^4.1.4" + regexp.prototype.flags "^1.5.1" + side-channel "^1.0.4" + which-boxed-primitive "^1.0.2" + which-collection "^1.0.1" + which-typed-array "^1.1.13" + deep-is@^0.1.3, deep-is@~0.1.3: version "0.1.4" resolved "https://registry.yarnpkg.com/deep-is/-/deep-is-0.1.4.tgz#a6f2dce612fadd2ef1f519b73551f17e85199831" @@ -4275,6 +4396,16 @@ doctrine@^3.0.0: dependencies: esutils "^2.0.2" +dom-accessibility-api@^0.5.9: + version "0.5.16" + resolved "https://registry.yarnpkg.com/dom-accessibility-api/-/dom-accessibility-api-0.5.16.tgz#5a7429e6066eb3664d911e33fb0e45de8eb08453" + integrity sha512-X7BJ2yElsnOJ30pZF4uIIDfBEVgF4XEBxL9Bxhy6dnrm5hkzqmsWHGTiHqRiITNhMyFLyAiWndIJP7Z1NTteDg== + +dom-accessibility-api@^0.6.3: + version "0.6.3" + resolved "https://registry.yarnpkg.com/dom-accessibility-api/-/dom-accessibility-api-0.6.3.tgz#993e925cc1d73f2c662e7d75dd5a5445259a8fd8" + integrity sha512-7ZgogeTnjuHbo+ct10G9Ffp0mif17idi0IyWNVA/wcwcm7NPOD/WEHVP3n7n3MhXqxoIYm8d6MuZohYWIZ4T3w== + dom-css@^2.0.0: version "2.1.0" resolved "https://registry.yarnpkg.com/dom-css/-/dom-css-2.1.0.tgz#fdbc2d5a015d0a3e1872e11472bbd0e7b9e6a202" @@ -4334,6 +4465,15 @@ dotenv@^16.0.3: resolved "https://registry.yarnpkg.com/dotenv/-/dotenv-16.0.3.tgz#115aec42bac5053db3c456db30cc243a5a836a07" integrity sha512-7GO6HghkA5fYG9TYnNxi14/7K9f5occMlp3zXAuSxn7CKCxt9xbNWG7yF8hTCSUchlfWSe3uLmlPfigevRItzQ== +dunder-proto@^1.0.1: + version "1.0.1" + resolved "https://registry.yarnpkg.com/dunder-proto/-/dunder-proto-1.0.1.tgz#d7ae667e1dc83482f8b70fd0f6eefc50da30f58a" + integrity sha512-KIN/nDJBQRcXw0MLVhZE9iQHmG68qAVIBg9CqmUYjmQIhgij9U5MFvrqkUL5FbtyyzZuOeOt0zdeRe4UY7ct+A== + dependencies: + call-bind-apply-helpers "^1.0.1" + es-errors "^1.3.0" + gopd "^1.2.0" + duplexer2@^0.1.2: version "0.1.4" resolved "https://registry.yarnpkg.com/duplexer2/-/duplexer2-0.1.4.tgz#8b12dab878c0d69e3e7891051662a32fc6bddcc1" @@ -4519,11 +4659,31 @@ es-define-property@^1.0.0: dependencies: get-intrinsic "^1.2.4" +es-define-property@^1.0.1: + version "1.0.1" + resolved "https://registry.yarnpkg.com/es-define-property/-/es-define-property-1.0.1.tgz#983eb2f9a6724e9303f61addf011c72e09e0b0fa" + integrity sha512-e3nRfgfUZ4rNGL232gUgX06QNyyez04KdjFrF+LTRoOXmrOgFKDg4BCdsjW8EnT69eqdYGmRpJwiPVYNrCaW3g== + es-errors@^1.2.1, es-errors@^1.3.0: version "1.3.0" resolved "https://registry.yarnpkg.com/es-errors/-/es-errors-1.3.0.tgz#05f75a25dab98e4fb1dcd5e1472c0546d5057c8f" integrity sha512-Zf5H2Kxt2xjTvbJvP2ZWLEICxA6j+hAmMzIlypy4xcBg1vKVnx89Wy0GbS+kf5cwCVFFzdCFh2XSCFNULS6csw== +es-get-iterator@^1.1.3: + version "1.1.3" + resolved "https://registry.yarnpkg.com/es-get-iterator/-/es-get-iterator-1.1.3.tgz#3ef87523c5d464d41084b2c3c9c214f1199763d6" + integrity sha512-sPZmqHBe6JIiTfN5q2pEi//TwxmAFHwj/XEuYjTuse78i8KxaqMTTzxPoFKuzRpDpTJ+0NAbpfenkmH2rePtuw== + dependencies: + call-bind "^1.0.2" + get-intrinsic "^1.1.3" + has-symbols "^1.0.3" + is-arguments "^1.1.1" + is-map "^2.0.2" + is-set "^2.0.2" + is-string "^1.0.7" + isarray "^2.0.5" + stop-iteration-iterator "^1.0.0" + es-iterator-helpers@^1.1.0: version "1.1.0" resolved "https://registry.yarnpkg.com/es-iterator-helpers/-/es-iterator-helpers-1.1.0.tgz#f6d745d342aea214fe09497e7152170dc333a7a6" @@ -4551,6 +4711,13 @@ es-object-atoms@^1.0.0: dependencies: es-errors "^1.3.0" +es-object-atoms@^1.1.1: + version "1.1.2" + resolved "https://registry.yarnpkg.com/es-object-atoms/-/es-object-atoms-1.1.2.tgz#a2d0b373205724dfa525d23b0c3e1b1ca582c99b" + integrity sha512-HWcBoN6NileqtSydK2FqHbS/LoDd2pqrnQHLyJzBj4kOp/ky2MWMN694xOfkK8/SnUsW2DH7EfyVlydKCsm1Zw== + dependencies: + es-errors "^1.3.0" + es-set-tostringtag@^2.0.1: version "2.0.1" resolved "https://registry.yarnpkg.com/es-set-tostringtag/-/es-set-tostringtag-2.0.1.tgz#338d502f6f674301d710b80c8592de8a15f09cd8" @@ -5238,11 +5405,35 @@ get-intrinsic@^1.2.3, get-intrinsic@^1.2.4: has-symbols "^1.0.3" hasown "^2.0.0" +get-intrinsic@^1.2.5, get-intrinsic@^1.3.0: + version "1.3.0" + resolved "https://registry.yarnpkg.com/get-intrinsic/-/get-intrinsic-1.3.0.tgz#743f0e3b6964a93a5491ed1bffaae054d7f98d01" + integrity sha512-9fSjSaos/fRIVIp+xSJlE6lfwhES7LNtKaCBIamHsjr2na1BiABJPo0mOjjz8GJDURarmCPGqaiVg5mfjb98CQ== + dependencies: + call-bind-apply-helpers "^1.0.2" + es-define-property "^1.0.1" + es-errors "^1.3.0" + es-object-atoms "^1.1.1" + function-bind "^1.1.2" + get-proto "^1.0.1" + gopd "^1.2.0" + has-symbols "^1.1.0" + hasown "^2.0.2" + math-intrinsics "^1.1.0" + get-package-type@^0.1.0: version "0.1.0" resolved "https://registry.yarnpkg.com/get-package-type/-/get-package-type-0.1.0.tgz#8de2d803cff44df3bc6c456e6668b36c3926e11a" integrity sha512-pjzuKtY64GYfWizNAJ0fr9VqttZkNiK2iS430LtIHzjBEr6bX8Am2zm4sW4Ro5wjWW5cAlRL1qAMTcXbjNAO2Q== +get-proto@^1.0.1: + version "1.0.1" + resolved "https://registry.yarnpkg.com/get-proto/-/get-proto-1.0.1.tgz#150b3f2743869ef3e851ec0c49d15b1d14d00ee1" + integrity sha512-sTSfBjoXBp89JvIKIefqw7U2CCebsc74kiY6awiGogKtoSGbgjYE/G/+l9sF3MWFPNc9IcoOC4ODfKHfxFmp0g== + dependencies: + dunder-proto "^1.0.1" + es-object-atoms "^1.0.0" + get-stream@^6.0.0: version "6.0.1" resolved "https://registry.yarnpkg.com/get-stream/-/get-stream-6.0.1.tgz#a262d8eef67aced57c2852ad6167526a43cbf7b7" @@ -5395,6 +5586,11 @@ gopd@^1.0.1: dependencies: get-intrinsic "^1.1.3" +gopd@^1.2.0: + version "1.2.0" + resolved "https://registry.yarnpkg.com/gopd/-/gopd-1.2.0.tgz#89f56b8217bdbc8802bd299df6d7f1081d7e51a1" + integrity sha512-ZUKRh6/kUFoAiTAtTYPZJ3hw9wNxx+BIBOijnlG9PnrJsCcSjs1wyyD6vJpaYtgnzDrKYRSqf3OO6Rfa93xsRg== + graceful-fs@^4.1.6, graceful-fs@^4.2.0, graceful-fs@^4.2.4, graceful-fs@^4.2.9: version "4.2.11" resolved "https://registry.yarnpkg.com/graceful-fs/-/graceful-fs-4.2.11.tgz#4183e4e8bf08bb6e05bbb2f7d2e0c8f712ca40e3" @@ -5461,6 +5657,11 @@ has-symbols@^1.0.2, has-symbols@^1.0.3: resolved "https://registry.yarnpkg.com/has-symbols/-/has-symbols-1.0.3.tgz#bb7b2c4349251dce87b125f7bdf874aa7c8b39f8" integrity sha512-l3LCuF6MgDNwTDKkdYGEihYjt5pRPbEg46rtlmnSPlUbgmB8LOIrKJbYYFBSbnPaJexMKtiPO8hmeRjRz2Td+A== +has-symbols@^1.1.0: + version "1.1.0" + resolved "https://registry.yarnpkg.com/has-symbols/-/has-symbols-1.1.0.tgz#fc9c6a783a084951d0b971fe1018de813707a338" + integrity sha512-1cDNdwJ2Jaohmb3sg4OmKaMBwuC48sYni5HUw2DvsC8LjGTLK9h+eb1X6RyuOHe4hT0ULCW68iomhjUoKUqlPQ== + has-tostringtag@^1.0.0: version "1.0.0" resolved "https://registry.yarnpkg.com/has-tostringtag/-/has-tostringtag-1.0.0.tgz#7e133818a7d394734f941e73c3d3f9291e658b25" @@ -5661,6 +5862,11 @@ imurmurhash@^0.1.4: resolved "https://registry.yarnpkg.com/imurmurhash/-/imurmurhash-0.1.4.tgz#9218b9b2b928a238b13dc4fb6b6d576f231453ea" integrity sha1-khi5srkoojixPcT7a21XbyMUU+o= +indent-string@^4.0.0: + version "4.0.0" + resolved "https://registry.yarnpkg.com/indent-string/-/indent-string-4.0.0.tgz#624f8f4497d619b2d9768531d58f4122854d7251" + integrity sha512-EdDDZu4A2OyIK7Lr/2zG+w5jmbuk1DVBnEwREQvBzspBJkCEbRa8GxU1lghYcaGJCnRWibjDXlq779X1/y5xwg== + inflight@^1.0.4: version "1.0.6" resolved "https://registry.yarnpkg.com/inflight/-/inflight-1.0.6.tgz#49bd6331d7d02d0c09bc910a1075ba8165b56df9" @@ -5702,6 +5908,15 @@ internal-slot@^1.0.7: hasown "^2.0.0" side-channel "^1.0.4" +internal-slot@^1.1.0: + version "1.1.0" + resolved "https://registry.yarnpkg.com/internal-slot/-/internal-slot-1.1.0.tgz#1eac91762947d2f7056bc838d93e13b2e9604961" + integrity sha512-4gd7VpWNQNB4UKKCFFVcp1AVv+FMOgs9NKzjHKusc8jTMhd5eL1NqQqOpE0KzMds804/yHlglp3uxgluOqAPLw== + dependencies: + es-errors "^1.3.0" + hasown "^2.0.2" + side-channel "^1.1.0" + internmap@^1.0.0: version "1.0.1" resolved "https://registry.yarnpkg.com/internmap/-/internmap-1.0.1.tgz#0017cc8a3b99605f0302f2b198d272e015e5df95" @@ -5735,6 +5950,14 @@ is-alphanumerical@^2.0.0: is-alphabetical "^2.0.0" is-decimal "^2.0.0" +is-arguments@^1.1.1: + version "1.2.0" + resolved "https://registry.yarnpkg.com/is-arguments/-/is-arguments-1.2.0.tgz#ad58c6aecf563b78ef2bf04df540da8f5d7d8e1b" + integrity sha512-7bVbi0huj/wrIAOzb8U1aszg9kdi3KN/CyU19CTI7tAoZYEZoL9yCDXpbXN+uPsuWnP02cyug1gleqq+TU+YCA== + dependencies: + call-bound "^1.0.2" + has-tostringtag "^1.0.2" + is-array-buffer@^3.0.1, is-array-buffer@^3.0.2: version "3.0.2" resolved "https://registry.yarnpkg.com/is-array-buffer/-/is-array-buffer-3.0.2.tgz#f2653ced8412081638ecb0ebbd0c41c6e0aecbbe" @@ -5885,6 +6108,11 @@ is-map@^2.0.1: resolved "https://registry.yarnpkg.com/is-map/-/is-map-2.0.2.tgz#00922db8c9bf73e81b7a335827bc2a43f2b91127" integrity sha512-cOZFQQozTha1f4MxLFzlgKYPTyj26picdZTx82hbc/Xf4K/tZOOXSCkMvU4pKioRXGDLJRn0GM7Upe7kR721yg== +is-map@^2.0.2: + version "2.0.3" + resolved "https://registry.yarnpkg.com/is-map/-/is-map-2.0.3.tgz#ede96b7fe1e270b3c4465e3a465658764926d62e" + integrity sha512-1Qed0/Hr2m+YqxnM09CjA2d/i6YZNfF6R2oRAOj36eUdS6qIV/huPJNSEpKbupewFs+ZsJlxsjjPbc0/afW6Lw== + is-negative-zero@^2.0.2: version "2.0.2" resolved "https://registry.yarnpkg.com/is-negative-zero/-/is-negative-zero-2.0.2.tgz#7bf6f03a28003b8b3965de3ac26f664d765f3150" @@ -5945,6 +6173,11 @@ is-set@^2.0.1: resolved "https://registry.yarnpkg.com/is-set/-/is-set-2.0.2.tgz#90755fa4c2562dc1c5d4024760d6119b94ca18ec" integrity sha512-+2cnTEZeY5z/iXGbLhPrOAaK/Mau5k5eXq9j14CpRTftq0pAJu2MwVRSZhyZWBzx3o6X795Lz6Bpb6R0GKf37g== +is-set@^2.0.2: + version "2.0.3" + resolved "https://registry.yarnpkg.com/is-set/-/is-set-2.0.3.tgz#8ab209ea424608141372ded6e0cb200ef1d9d01d" + integrity sha512-iPAjerrse27/ygGLxw+EBR9agv9Y6uLeYVJMu+QNCoouJ1/1ri0mGrcWpfCqFZuzzx3WjtwxG098X+n4OuRkPg== + is-shared-array-buffer@^1.0.2: version "1.0.2" resolved "https://registry.yarnpkg.com/is-shared-array-buffer/-/is-shared-array-buffer-1.0.2.tgz#8f259c573b60b6a32d4058a1a07430c0a7344c79" @@ -6834,6 +7067,11 @@ lru-cache@^6.0.0: dependencies: yallist "^4.0.0" +lz-string@^1.5.0: + version "1.5.0" + resolved "https://registry.yarnpkg.com/lz-string/-/lz-string-1.5.0.tgz#c1ab50f77887b712621201ba9fd4e3a6ed099941" + integrity sha512-h5bgJWpxJNswbU7qCrV0tIKQCaS3blPDrqKWx+QxzuzL1zGUzij9XCWLrSLsJPu5t+eWA/ycetzYAO5IOMcWAQ== + make-dir@^4.0.0: version "4.0.0" resolved "https://registry.yarnpkg.com/make-dir/-/make-dir-4.0.0.tgz#c3c2307a771277cd9638305f915c29ae741b614e" @@ -6858,6 +7096,11 @@ markdown-table@^3.0.0: resolved "https://registry.yarnpkg.com/markdown-table/-/markdown-table-3.0.3.tgz#e6331d30e493127e031dd385488b5bd326e4a6bd" integrity sha512-Z1NL3Tb1M9wH4XESsCDEksWoKTdlUafKc4pt0GRwjUyXaCFZ+dc3g2erqB6zm3szA2IUSi7VnPI+o/9jnxh9hw== +math-intrinsics@^1.1.0: + version "1.1.0" + resolved "https://registry.yarnpkg.com/math-intrinsics/-/math-intrinsics-1.1.0.tgz#a0dd74be81e2aa5c2f27e65ce283605ee4e2b7f9" + integrity sha512-/IXtbwEk5HTPyEwyKX6hGkYXxM9nbj64B+ilVJnC/R6B0pH5G4V3b0pVbL7DBj4tkhBAppbQUlf6F6Xl9LHu1g== + mdast-util-find-and-replace@^3.0.0: version "3.0.1" resolved "https://registry.yarnpkg.com/mdast-util-find-and-replace/-/mdast-util-find-and-replace-3.0.1.tgz#a6fc7b62f0994e973490e45262e4bc07607b04e0" @@ -7351,6 +7594,11 @@ mimic-fn@^2.1.0: resolved "https://registry.yarnpkg.com/mimic-fn/-/mimic-fn-2.1.0.tgz#7ed2c2ccccaf84d3ffcb7a69b57711fc2083401b" integrity sha512-OqbOk5oEQeAZ8WXWydlu9HJjz9WVdEIvamMCcXmuqUYjTknH/sqsWvhQ3vgwKFRR1HpjvNBKQ37nbJgYzGqGcg== +min-indent@^1.0.0: + version "1.0.1" + resolved "https://registry.yarnpkg.com/min-indent/-/min-indent-1.0.1.tgz#a63f681673b30571fbe8bc25686ae746eefa9869" + integrity sha512-I9jwMn07Sy/IwOj3zVkVik2JTvgpaykDZEigL6Rx6N9LbMywwUSMtxET+7lVoDLLd3O3IXwJwvuuns8UB/HeAg== + minimatch@^3.0.4, minimatch@^3.0.5, minimatch@^3.1.1, minimatch@^3.1.2: version "3.1.2" resolved "https://registry.yarnpkg.com/minimatch/-/minimatch-3.1.2.tgz#19cd194bfd3e428f049a70817c038d89ab4be35b" @@ -7575,6 +7823,19 @@ object-inspect@^1.13.1, object-inspect@^1.9.0: resolved "https://registry.yarnpkg.com/object-inspect/-/object-inspect-1.13.1.tgz#b96c6109324ccfef6b12216a956ca4dc2ff94bc2" integrity sha512-5qoj1RUiKOMsCCNLV1CBiPYE10sziTsnmNxkAI/rZhiD63CF7IqdFGC/XzjWjpSgLf0LxXX3bDFIh0E18f6UhQ== +object-inspect@^1.13.3, object-inspect@^1.13.4: + version "1.13.4" + resolved "https://registry.yarnpkg.com/object-inspect/-/object-inspect-1.13.4.tgz#8375265e21bc20d0fa582c22e1b13485d6e00213" + integrity sha512-W67iLl4J2EXEGTbfeHCffrjDfitvLANg0UlX3wFUUSTx92KXRFegMHUVgSqE+wvhAbi4WqjGg9czysTV2Epbew== + +object-is@^1.1.5: + version "1.1.6" + resolved "https://registry.yarnpkg.com/object-is/-/object-is-1.1.6.tgz#1a6a53aed2dd8f7e6775ff870bea58545956ab07" + integrity sha512-F8cZ+KfGlSGi09lJT7/Nd6KJZ9ygtvYC0/UYYLI9nmQKLMnydpB9yvbv9K1uSkEu7FU9vYPmVwLg328tX+ot3Q== + dependencies: + call-bind "^1.0.7" + define-properties "^1.2.1" + object-keys@^1.1.1: version "1.1.1" resolved "https://registry.yarnpkg.com/object-keys/-/object-keys-1.1.1.tgz#1c47f272df277f3b1daf061677d9c82e2322c60e" @@ -7933,6 +8194,11 @@ picocolors@^1.0.0: resolved "https://registry.yarnpkg.com/picocolors/-/picocolors-1.0.0.tgz#cb5bdc74ff3f51892236eaf79d68bc44564ab81c" integrity sha512-1fygroTLlHu66zi26VoTDv8yRgm0Fccecssto+MhsZ0D/DGW2sm8E8AjW7NU5VVTRt5GxbeZ5qBuJr+HyLYkjQ== +picocolors@^1.1.1: + version "1.1.1" + resolved "https://registry.yarnpkg.com/picocolors/-/picocolors-1.1.1.tgz#3d321af3eab939b083c8f929a1d12cda81c26b6b" + integrity sha512-xceH2snhtb5M9liqDsmEw56le376mTZkEX/jEb/RxNFyegNul7eNslCXP9FDj/Lcu0X8KEyMceP2ntpaHrDEVA== + picomatch@^2.0.4, picomatch@^2.2.1, picomatch@^2.2.3, picomatch@^2.3.1: version "2.3.1" resolved "https://registry.yarnpkg.com/picomatch/-/picomatch-2.3.1.tgz#3ba3833733646d9d3e4995946c1365a67fb07a42" @@ -8139,6 +8405,15 @@ prettier@^3.0.3: resolved "https://registry.yarnpkg.com/prettier/-/prettier-3.0.3.tgz#432a51f7ba422d1469096c0fdc28e235db8f9643" integrity sha512-L/4pUDMxcNa8R/EthV08Zt42WBO4h1rarVtK0K+QJG0X187OLo7l699jWw0GKuwzkPQ//jMFA/8Xm6Fh3J/DAg== +pretty-format@^27.0.2: + version "27.5.1" + resolved "https://registry.yarnpkg.com/pretty-format/-/pretty-format-27.5.1.tgz#2181879fdea51a7a5851fb39d920faa63f01d88e" + integrity sha512-Qb1gy5OrP5+zDf2Bvnzdl3jsTf1qXVMazbvCoKhtKqVs4/YK4ozX4gKQJJVyNe+cajNPn0KoC0MC3FUmaHWEmQ== + dependencies: + ansi-regex "^5.0.1" + ansi-styles "^5.0.0" + react-is "^17.0.1" + pretty-format@^29.0.0, pretty-format@^29.6.3, pretty-format@^29.7.0: version "29.7.0" resolved "https://registry.yarnpkg.com/pretty-format/-/pretty-format-29.7.0.tgz#ca42c758310f365bfa71a0bda0a807160b776812" @@ -8312,7 +8587,7 @@ react-is@^16.13.1, react-is@^16.7.0: resolved "https://registry.yarnpkg.com/react-is/-/react-is-16.13.1.tgz#789729a4dc36de2999dc156dd6c1d9c18cea56a4" integrity sha512-24e6ynE2H+OKt4kqsOvNd8kBpV65zoxbA4BVsEOB3ARVWQki/DHzaUoC5KuON/BiccDaCCTZBuOcfZs70kR8bQ== -react-is@^17.0.2: +react-is@^17.0.1, react-is@^17.0.2: version "17.0.2" resolved "https://registry.yarnpkg.com/react-is/-/react-is-17.0.2.tgz#e691d4a8e9c789365655539ab372762b0efb54f0" integrity sha512-w2GsyukL62IJnlaff/nRegPQR94C/XXamvMWmSHRJ4y7Ts/4ocGRmTHvOs8PSE6pB3dWOrD/nueuU5sduBsQ4w== @@ -8426,6 +8701,14 @@ rechoir@^0.8.0: dependencies: resolve "^1.20.0" +redent@^3.0.0: + version "3.0.0" + resolved "https://registry.yarnpkg.com/redent/-/redent-3.0.0.tgz#e557b7998316bb53c9f1f56fa626352c6963059f" + integrity sha512-6tDA8g98We0zd0GvVeMT9arEOnTw9qM03L9cJXaCjrip1OO764RDBLBfrB4cwzNGDj5OA5ioymC9GkizgWJDUg== + dependencies: + indent-string "^4.0.0" + strip-indent "^3.0.0" + redux-persist@^6.0.0: version "6.0.0" resolved "https://registry.yarnpkg.com/redux-persist/-/redux-persist-6.0.0.tgz#b4d2972f9859597c130d40d4b146fecdab51b3a8" @@ -8874,6 +9157,35 @@ shimmer@^1.1.0, shimmer@^1.2.0: resolved "https://registry.yarnpkg.com/shimmer/-/shimmer-1.2.1.tgz#610859f7de327b587efebf501fb43117f9aff337" integrity sha512-sQTKC1Re/rM6XyFM6fIAGHRPVGvyXfgzIDvzoq608vM+jeyVD0Tu1E6Np0Kc2zAIFWIj963V2800iF/9LPieQw== +side-channel-list@^1.0.1: + version "1.0.1" + resolved "https://registry.yarnpkg.com/side-channel-list/-/side-channel-list-1.0.1.tgz#c2e0b5a14a540aebee3bbc6c3f8666cc9b509127" + integrity sha512-mjn/0bi/oUURjc5Xl7IaWi/OJJJumuoJFQJfDDyO46+hBWsfaVM65TBHq2eoZBhzl9EchxOijpkbRC8SVBQU0w== + dependencies: + es-errors "^1.3.0" + object-inspect "^1.13.4" + +side-channel-map@^1.0.1: + version "1.0.1" + resolved "https://registry.yarnpkg.com/side-channel-map/-/side-channel-map-1.0.1.tgz#d6bb6b37902c6fef5174e5f533fab4c732a26f42" + integrity sha512-VCjCNfgMsby3tTdo02nbjtM/ewra6jPHmpThenkTYh8pG9ucZ/1P8So4u4FGBek/BjpOVsDCMoLA/iuBKIFXRA== + dependencies: + call-bound "^1.0.2" + es-errors "^1.3.0" + get-intrinsic "^1.2.5" + object-inspect "^1.13.3" + +side-channel-weakmap@^1.0.2: + version "1.0.2" + resolved "https://registry.yarnpkg.com/side-channel-weakmap/-/side-channel-weakmap-1.0.2.tgz#11dda19d5368e40ce9ec2bdc1fb0ecbc0790ecea" + integrity sha512-WPS/HvHQTYnHisLo9McqBHOJk2FkHO/tlpvldyrnem4aeQp4hai3gythswg6p01oSoTl58rcpiFAjF2br2Ak2A== + dependencies: + call-bound "^1.0.2" + es-errors "^1.3.0" + get-intrinsic "^1.2.5" + object-inspect "^1.13.3" + side-channel-map "^1.0.1" + side-channel@^1.0.4: version "1.0.4" resolved "https://registry.yarnpkg.com/side-channel/-/side-channel-1.0.4.tgz#efce5c8fdc104ee751b25c58d4290011fa5ea2cf" @@ -8893,6 +9205,17 @@ side-channel@^1.0.6: get-intrinsic "^1.2.4" object-inspect "^1.13.1" +side-channel@^1.1.0: + version "1.1.1" + resolved "https://registry.yarnpkg.com/side-channel/-/side-channel-1.1.1.tgz#ea02c62e05dc4bea67d4442f0fb71ee192f8e0ab" + integrity sha512-6x6dK6zJdpTzF4sQeNYxwtvBzf6Eg4GtlesS94HOvTudUeyK2WXAaIfmDgsyslYrRBeFIlsi54AYsFGUuhmvrQ== + dependencies: + es-errors "^1.3.0" + object-inspect "^1.13.4" + side-channel-list "^1.0.1" + side-channel-map "^1.0.1" + side-channel-weakmap "^1.0.2" + signal-exit@^3.0.3, signal-exit@^3.0.7: version "3.0.7" resolved "https://registry.yarnpkg.com/signal-exit/-/signal-exit-3.0.7.tgz#a9a1767f8af84155114eaabd73f99273c8f59ad9" @@ -9024,6 +9347,14 @@ statuses@2.0.1: resolved "https://registry.yarnpkg.com/statuses/-/statuses-2.0.1.tgz#55cb000ccf1d48728bd23c685a063998cf1a1b63" integrity sha512-RwNA9Z/7PrK06rYLIzFMlaF+l73iwpzsqRIFgbMLbTcLD6cOao82TaWefPXQvB2fOC4AjuYSEndS7N/mTCbkdQ== +stop-iteration-iterator@^1.0.0: + version "1.1.0" + resolved "https://registry.yarnpkg.com/stop-iteration-iterator/-/stop-iteration-iterator-1.1.0.tgz#f481ff70a548f6124d0312c3aa14cbfa7aa542ad" + integrity sha512-eLoXW/DHyl62zxY4SCaIgnRhuMr6ri4juEYARS8E6sCEqzKpOiE521Ucofdx+KnDZl5xmvGYaaKCk5FEOxJCoQ== + dependencies: + es-errors "^1.3.0" + internal-slot "^1.1.0" + streamsearch@^1.1.0: version "1.1.0" resolved "https://registry.yarnpkg.com/streamsearch/-/streamsearch-1.1.0.tgz#404dd1e2247ca94af554e841a8ef0eaa238da764" @@ -9178,6 +9509,13 @@ strip-final-newline@^2.0.0: resolved "https://registry.yarnpkg.com/strip-final-newline/-/strip-final-newline-2.0.0.tgz#89b852fb2fcbe936f6f4b3187afb0a12c1ab58ad" integrity sha512-BrpvfNAE3dcvq7ll3xVumzjKjZQ5tI1sEUIKr3Uoks0XUl45St3FlatVqef9prk4jRDzhW6WZg+3bk93y6pLjA== +strip-indent@^3.0.0: + version "3.0.0" + resolved "https://registry.yarnpkg.com/strip-indent/-/strip-indent-3.0.0.tgz#c32e1cee940b6b3432c771bc2c54bcce73cd3001" + integrity sha512-laJTa3Jb+VQpaC6DseHhF7dXVqHTfJPCRDaEbid/drOhgitgYku/letMUqOXFoWV0zIIUbjpdH2t+tYj4bQMRQ== + dependencies: + min-indent "^1.0.0" + strip-json-comments@^3.1.1: version "3.1.1" resolved "https://registry.yarnpkg.com/strip-json-comments/-/strip-json-comments-3.1.1.tgz#31f1281b3832630434831c310c01cccda8cbe006" From 82c1a581f589f143e0832bcf79884abf98e91cf4 Mon Sep 17 00:00:00 2001 From: Batool Fatima Date: Tue, 11 Aug 2026 03:27:45 +0500 Subject: [PATCH 116/194] fix(jira): centralize code-provider detection, fix URL normalization Code-review pass on the Jira linking work (feat/jira_link), addressing best-practices findings: - integrationList.length was used in 6 places as a proxy for "a code provider (GitHub/GitLab/Bitbucket) is linked." That became wrong the moment Jira -- a non-code integration -- became linkable: a Jira-only org would silently unlock Teams, DORA Metrics, Settings, the sidebar nav, and team-selector data fetching, none of which have anything to show without an actual code provider. Only pages/integrations.tsx had been fixed for this; useFilteredSidebarItems.tsx, DoraMetricsBody.tsx, teams/index.tsx, settings.tsx, and useTeamSelectorSetup.tsx were still broken. Fixed by adding one shared computation to ThirdPartyAuthContext (codeProviderIntegrationList / hasCodeProviderLinked, filtered against the existing CODE_PROVIDERS constant) instead of patching each call site with its own copy of the filter logic, and updating all six places to consume it. - activeCodeProvider searched all integration keys, including 'jira', so it could resolve to Jira as the "active code provider." Constrained the search to CODE_PROVIDERS. - normalizeSiteUrl / normalizeJiraSiteUrl only stripped a leading protocol and trailing slash, so a full URL with a path (e.g. copied from the browser while looking at a board) broke the very next request. Rewritten to extract just the hostname via the URL constructor, which also correctly drops ports and query strings. Fixed in both the client and server copies; added regression tests to both auth-jira.test.ts and validate.test.ts. Regression check: tsc --noEmit clean. Full Jest suite: 206 passed, same 2 pre-existing unrelated failures as before this branch (array.test.ts date-arithmetic, git_org_repos.test.ts jose/ESM parse issue). Python suite: 159 passed, unaffected. E2E: both jira-integration.spec.ts tests pass against the rebuilt container, confirming the ThirdPartyAuthContext refactor didn't change the "no refresh needed" link-status behavior it was built to guard. Co-Authored-By: Claude Sonnet 5 (cherry picked from commit 9e1c38b85b0281d52084e05a042f94a8c1580ce1) --- .../jira/__tests__/validate.test.ts | 28 +++++++++++++++ .../pages/api/integrations/jira/validate.ts | 28 +++++++++++---- web-server/pages/integrations.tsx | 21 +++++------ web-server/pages/settings.tsx | 8 +++-- web-server/pages/teams/index.tsx | 7 ++-- .../TeamSelector/useTeamSelectorSetup.tsx | 9 +++-- .../src/contexts/ThirdPartyAuthContext.tsx | 35 ++++++++++++++++--- .../SidebarMenu/useFilteredSidebarItems.tsx | 10 ++++-- .../src/utils/__tests__/auth-jira.test.ts | 20 +++++++++++ web-server/src/utils/auth.ts | 28 +++++++++++---- 10 files changed, 154 insertions(+), 40 deletions(-) diff --git a/web-server/pages/api/integrations/jira/__tests__/validate.test.ts b/web-server/pages/api/integrations/jira/__tests__/validate.test.ts index c379c731..4db15781 100644 --- a/web-server/pages/api/integrations/jira/__tests__/validate.test.ts +++ b/web-server/pages/api/integrations/jira/__tests__/validate.test.ts @@ -76,6 +76,34 @@ describe('POST /api/integrations/jira/validate', () => { ); }); + // CLUSTOX: regression test for a real bug found in code review. A regex + // that only stripped a leading protocol and trailing slash left any + // path/query intact -- pasting a full address copied from the browser + // while looking at a board would have built a request against the + // wrong path (a plausible thing for a user to paste, since /rest/api/3 + // is not something they'd normally see or think to strip themselves). + it('normalizes a full URL with a path down to just the hostname before requesting', async () => { + (axios.get as jest.Mock).mockResolvedValue({ + data: { accountId: 'acc-1' } + }); + const res = mockRes(); + + await validateHandler( + mockReq({ + site_url: + 'https://mycompany.atlassian.net/jira/software/projects/ABC/boards/1', + email: 'a@b.com', + api_token: 'tok' + }), + res + ); + + expect(axios.get).toHaveBeenCalledWith( + 'https://mycompany.atlassian.net/rest/api/3/myself', + expect.anything() + ); + }); + it('falls back to the submitted email as display name when Jira omits displayName', async () => { (axios.get as jest.Mock).mockResolvedValue({ data: { accountId: 'acc-1' } diff --git a/web-server/pages/api/integrations/jira/validate.ts b/web-server/pages/api/integrations/jira/validate.ts index ec120e5e..e634cd06 100644 --- a/web-server/pages/api/integrations/jira/validate.ts +++ b/web-server/pages/api/integrations/jira/validate.ts @@ -9,13 +9,27 @@ const postSchema = yup.object().shape({ api_token: yup.string().required() }); -// Strip any protocol/trailing slash so both "mycompany.atlassian.net" and -// "https://mycompany.atlassian.net/" work as input. -const normalizeSiteUrl = (input: string) => - input - .trim() - .replace(/^https?:\/\//i, '') - .replace(/\/+$/, ''); +// Reduces any of "mycompany.atlassian.net", "https://mycompany.atlassian.net/", +// or a full URL copied from the browser while looking at a board +// ("https://mycompany.atlassian.net/jira/software/projects/ABC/boards/1") +// to just the hostname. A regex that only stripped a leading protocol and +// trailing slash left any path/query intact, which broke the very next +// request below (404s even with a valid site + credentials) for anyone +// who pasted a full address rather than typing the bare domain. Mirrors +// normalizeJiraSiteUrl in src/utils/auth.ts -- kept as two small copies +// rather than a shared import across the client/server boundary, same as +// that file's other provider helpers. +const normalizeSiteUrl = (input: string) => { + const trimmed = input.trim(); + try { + const withProtocol = /^https?:\/\//i.test(trimmed) + ? trimmed + : `https://${trimmed}`; + return new URL(withProtocol).hostname; + } catch { + return trimmed; + } +}; const endpoint = new Endpoint(nullSchema); diff --git a/web-server/pages/integrations.tsx b/web-server/pages/integrations.tsx index 9e59a4fe..902a19d2 100644 --- a/web-server/pages/integrations.tsx +++ b/web-server/pages/integrations.tsx @@ -14,7 +14,6 @@ import { ClustoxJenkinsSetup } from '@/components/ClustoxJenkinsSetup'; // END CLUSTOX import { FlexBox } from '@/components/FlexBox'; import { Line } from '@/components/Text'; -import { CODE_PROVIDERS } from '@/constants/codeProviders'; import { ROUTES } from '@/constants/routes'; import { FetchState } from '@/constants/ui-states'; import { GithubIntegrationCard } from '@/content/Dashboards/GithubIntegrationCard'; @@ -66,15 +65,13 @@ Integrations.getLayout = (page: PageLayout) => ( export default Integrations; const Content = () => { - const { orgId, integrations, integrationList } = useAuth(); - const codeProviderList = useMemo( - () => - integrationList.filter((item) => - (CODE_PROVIDERS as readonly string[]).includes(item) - ), - [integrationList] - ); - const hasCodeProviderLinked = codeProviderList.length > 0; + const { + orgId, + integrations, + integrationList, + codeProviderIntegrationList, + hasCodeProviderLinked + } = useAuth(); const teamCount = useSelector((s) => s.team.teams?.length); const dispatch = useDispatch(); const loadedTeams = useBoolState(false); @@ -84,7 +81,7 @@ const Content = () => { hasCodeProviderLinked && !teamCount && loadedTeams.value; const lastSyncMap = useMemo(() => { - return codeProviderList + return codeProviderIntegrationList .map((item) => { const linkedAt = integrations[item as 'github' | 'gitlab' | 'bitbucket'].linked_at; @@ -98,7 +95,7 @@ const Content = () => { return diff; }) .filter(Boolean); - }, [codeProviderList, integrations]); + }, [codeProviderIntegrationList, integrations]); const showForceSyncBtn = useMemo(() => { if (!hasCodeProviderLinked) return false; diff --git a/web-server/pages/settings.tsx b/web-server/pages/settings.tsx index 71f649bf..30d040f9 100644 --- a/web-server/pages/settings.tsx +++ b/web-server/pages/settings.tsx @@ -13,7 +13,11 @@ import { PageLayout } from '@/types/resources'; function Settings() { useRedirectWithSession(); - const { integrationList } = useAuth(); + // CLUSTOX: hasCodeProviderLinked, not integrationList.length -- Content + // renders SyncDaysSetting, a code-repo sync-cadence control that has + // nothing to configure for a Jira-only org. See + // docs/JIRA_INTEGRATION_PROPOSAL.md. + const { hasCodeProviderLinked } = useAuth(); const isLoading = useSelector( (s) => s.org.requests?.defaultSyncDays === FetchState.REQUEST ); @@ -31,7 +35,7 @@ function Settings() { showEvenIfNoTeamSelected={true} isLoading={isLoading} > - {integrationList.length > 0 && } + {hasCodeProviderLinked && } ); diff --git a/web-server/pages/teams/index.tsx b/web-server/pages/teams/index.tsx index 4fb1d020..83c85d7c 100644 --- a/web-server/pages/teams/index.tsx +++ b/web-server/pages/teams/index.tsx @@ -16,7 +16,10 @@ import { depFn } from '@/utils/fn'; function Page() { useRedirectWithSession(); const dispatch = useDispatch(); - const { orgId, integrationList } = useAuth(); + // CLUSTOX: hasCodeProviderLinked, not integrationList.length -- a + // Jira-only org has no repos to build a team around. See + // docs/JIRA_INTEGRATION_PROPOSAL.md. + const { orgId, hasCodeProviderLinked } = useAuth(); const teamsList = useSelector((state) => state.team.teams); const loading = useBoolState(!Boolean(teamsList.length)); @@ -46,7 +49,7 @@ function Page() { showEvenIfNoTeamSelected hideAllSelectors > - {integrationList.length && !loading.value ? ( + {hasCodeProviderLinked && !loading.value ? ( {teamsList.length ? : } diff --git a/web-server/src/components/TeamSelector/useTeamSelectorSetup.tsx b/web-server/src/components/TeamSelector/useTeamSelectorSetup.tsx index c9178d5c..d050005f 100644 --- a/web-server/src/components/TeamSelector/useTeamSelectorSetup.tsx +++ b/web-server/src/components/TeamSelector/useTeamSelectorSetup.tsx @@ -21,7 +21,10 @@ export const useTeamSelectorSetup = ({ mode }: UseTeamSelectorSetupArgs) => { const dispatch = useDispatch(); const { dateRange, singleTeam, setRange, partiallyUnselected } = useStateTeamConfig(); - const { orgId, integrationList } = useAuth(); + // CLUSTOX: hasCodeProviderLinked, not integrationList.length -- a + // Jira-only org has no teams/repos for this fetch to be worth firing. + // See docs/JIRA_INTEGRATION_PROPOSAL.md. + const { orgId, hasCodeProviderLinked } = useAuth(); const [showAllTeams, setShowAllTeams] = useState(true); const activeBranchMode = useSelector((state) => state.app.branchMode); const isAllBranchMode = activeBranchMode === ActiveBranchMode.ALL; @@ -65,8 +68,8 @@ export const useTeamSelectorSetup = ({ mode }: UseTeamSelectorSetupArgs) => { useEffect(() => { if (!orgId) return; - if (integrationList.length) fetchAllTeams(); - }, [fetchAllTeams, integrationList.length, orgId]); + if (hasCodeProviderLinked) fetchAllTeams(); + }, [fetchAllTeams, hasCodeProviderLinked, orgId]); const dateRangeLabel = !partiallyUnselected ? `${format(dateRange[0], 'do MMM')} to ${format(dateRange[1], 'do MMM')}` diff --git a/web-server/src/contexts/ThirdPartyAuthContext.tsx b/web-server/src/contexts/ThirdPartyAuthContext.tsx index f5717e4b..b07846e1 100644 --- a/web-server/src/contexts/ThirdPartyAuthContext.tsx +++ b/web-server/src/contexts/ThirdPartyAuthContext.tsx @@ -4,6 +4,7 @@ import { FC, createContext, useEffect, useCallback } from 'react'; import { useMemo } from 'react'; import { CODE_PROVIDER_INTEGRATIONS_MAP } from '@/constants/api'; +import { CODE_PROVIDERS } from '@/constants/codeProviders'; import { Integration } from '@/constants/integrations'; import { useBoolState, useEasyState } from '@/hooks/useEasyState'; import { useRefMounted } from '@/hooks/useRefMounted'; @@ -28,6 +29,13 @@ export interface AuthContextValue extends AuthState { integrations: Org['integrations']; onboardingState: OnboardingStep[]; integrationList: Integration[]; + // CLUSTOX: integrationList filtered to code-hosting providers only -- + // see docs/JIRA_INTEGRATION_PROPOSAL.md. This is the ONE place that + // filter is applied; every consumer that means "is a code provider + // linked" (not "is anything linked") should read hasCodeProviderLinked + // or this list, never integrationList.length directly. + codeProviderIntegrationList: Integration[]; + hasCodeProviderLinked: boolean; integrationSet: Set; activeCodeProvider: CodeProviderIntegrations | null; } @@ -40,6 +48,8 @@ export const AuthContext = createContext({ integrations: {}, onboardingState: [], integrationList: [], + codeProviderIntegrationList: [], + hasCodeProviderLinked: false, integrationSet: new Set(), activeCodeProvider: null }); @@ -160,15 +170,30 @@ export const AuthProvider: FC = (props) => { [integrations, isCodeProvider] ); + // CLUSTOX: the one place integrationList is filtered down to code + // providers -- every other file that used to read integrationList.length + // (or iterate integrationList directly) to mean "is a code provider + // linked" now reads this instead, so a future non-code integration + // doesn't silently break the same way Jira did. See + // docs/JIRA_INTEGRATION_PROPOSAL.md and pages/integrations.tsx's history. + const codeProviderIntegrationList = useMemo( + () => + integrationList.filter((item) => + (CODE_PROVIDERS as readonly string[]).includes(item) + ), + [integrationList] + ); + const hasCodeProviderLinked = codeProviderIntegrationList.length > 0; + const activeCodeProvider = useMemo( () => - Object.keys(state?.org?.integrations || {}).find( + (Object.keys(state?.org?.integrations || {}).find( (integrationName) => - isCodeProvider(integrationName) && + (CODE_PROVIDERS as readonly string[]).includes(integrationName) && state?.org?.integrations[integrationName as keyof IntegrationsMap] ?.integrated - ) as CodeProviderIntegrations | null, - [isCodeProvider, state?.org?.integrations] + ) ?? null) as CodeProviderIntegrations | null, + [state?.org?.integrations] ); return ( @@ -179,6 +204,8 @@ export const AuthProvider: FC = (props) => { orgId: state.org?.id, role, integrations, + codeProviderIntegrationList, + hasCodeProviderLinked, integrationSet, integrationList, activeCodeProvider, diff --git a/web-server/src/layouts/ExtendedSidebarLayout/Sidebar/SidebarMenu/useFilteredSidebarItems.tsx b/web-server/src/layouts/ExtendedSidebarLayout/Sidebar/SidebarMenu/useFilteredSidebarItems.tsx index 4290f01a..763be32f 100644 --- a/web-server/src/layouts/ExtendedSidebarLayout/Sidebar/SidebarMenu/useFilteredSidebarItems.tsx +++ b/web-server/src/layouts/ExtendedSidebarLayout/Sidebar/SidebarMenu/useFilteredSidebarItems.tsx @@ -18,7 +18,7 @@ const checkTag = (tag: string | number[] | number, check: string | number) => { }; export const useFilteredSidebarItems = () => { - const { integrationList } = useAuth(); + const { hasCodeProviderLinked } = useAuth(); // CLUSTOX: the real role, since useAuth().role is hardcoded upstream. const { isSuperadmin } = useClustoxUser(); @@ -36,9 +36,13 @@ export const useFilteredSidebarItems = () => { // viewing may legitimately have no integration yet. Collapsing their nav // to just "Manage Integrations" would strand them, since user management // and the workspace switcher are how they administer the instance. + // Gated on a code provider specifically, not "any integration" -- + // linking only Jira shouldn't unlock Teams/DORA/Settings, which have + // nothing to show without repo data behind them. See + // docs/JIRA_INTEGRATION_PROPOSAL.md. if ( !isSuperadmin && - !integrationList.length && + !hasCodeProviderLinked && item.name !== SideBarItems.MANAGE_INTEGRATIONS ) return false; @@ -58,7 +62,7 @@ export const useFilteredSidebarItems = () => { items: itemsFilter(section.items) })) .filter((section) => section.items?.length); - }, [flagFilteredMenuItems, integrationList, isSuperadmin]); + }, [flagFilteredMenuItems, hasCodeProviderLinked, isSuperadmin]); return sidebarItems; }; diff --git a/web-server/src/utils/__tests__/auth-jira.test.ts b/web-server/src/utils/__tests__/auth-jira.test.ts index 398cccd0..f13ada49 100644 --- a/web-server/src/utils/__tests__/auth-jira.test.ts +++ b/web-server/src/utils/__tests__/auth-jira.test.ts @@ -34,6 +34,26 @@ describe('normalizeJiraSiteUrl', () => { 'mycompany.atlassian.net' ); }); + + // CLUSTOX: regression test for a real bug found in code review. A regex + // that only stripped a leading protocol and trailing slash left any + // path/query intact -- pasting a full address copied from the browser + // while looking at a board (a very plausible thing for a user to do, + // more plausible than typing the bare domain) broke the very next + // request even with valid credentials. + it('reduces a full URL with a path to just the hostname', () => { + expect( + normalizeJiraSiteUrl( + 'https://mycompany.atlassian.net/jira/software/projects/ABC/boards/1' + ) + ).toBe('mycompany.atlassian.net'); + }); + + it('drops a port and query string too', () => { + expect( + normalizeJiraSiteUrl('https://mycompany.atlassian.net:443/?foo=bar') + ).toBe('mycompany.atlassian.net'); + }); }); describe('checkJiraValidity', () => { diff --git a/web-server/src/utils/auth.ts b/web-server/src/utils/auth.ts index 01b2e548..a0b224c5 100644 --- a/web-server/src/utils/auth.ts +++ b/web-server/src/utils/auth.ts @@ -102,16 +102,30 @@ export const getMissingGitLabScopes = (scopes: string[]): string[] => { // Jira functions -// Strip any protocol/trailing slash so "mycompany.atlassian.net" and -// "https://mycompany.atlassian.net/" are stored the same way. Mirrors the +// Reduces any of "mycompany.atlassian.net", "https://mycompany.atlassian.net/", +// or a full URL copied from the browser while looking at a board +// ("https://mycompany.atlassian.net/jira/software/projects/ABC/boards/1") +// to just the hostname. A regex that only stripped a leading protocol and +// trailing slash left any path/query intact, which broke the very next +// request (GET https://{that}/rest/api/3/myself 404s) for anyone who +// pasted a full address rather than typing the bare domain. Mirrors the // normalizer in pages/api/integrations/jira/validate.ts -- kept as two // small copies rather than a shared import across the client/server // boundary, same as this file's other provider helpers. -export const normalizeJiraSiteUrl = (input: string) => - input - .trim() - .replace(/^https?:\/\//i, '') - .replace(/\/+$/, ''); +export const normalizeJiraSiteUrl = (input: string) => { + const trimmed = input.trim(); + try { + const withProtocol = /^https?:\/\//i.test(trimmed) + ? trimmed + : `https://${trimmed}`; + return new URL(withProtocol).hostname; + } catch { + // Not parseable as a URL at all (e.g. empty string) -- fall through to + // the caller's own "fill in all three fields"/validity check rather + // than throw here. + return trimmed; + } +}; export type JiraValidityResult = { valid: boolean; From a458e1df9f1d281179a519404fd06b00e5bd4942 Mon Sep 17 00:00:00 2001 From: Batool Fatima Date: Wed, 12 Aug 2026 04:06:37 +0500 Subject: [PATCH 117/194] docs: drop JIRA_INTEGRATION_PROPOSAL.md from PR scope CLUSTOX: internal design-doc scaffolding, not meant to ship in the per-ticket PRs. --- docs/JIRA_INTEGRATION_PROPOSAL.md | 176 ------------------------------ 1 file changed, 176 deletions(-) delete mode 100644 docs/JIRA_INTEGRATION_PROPOSAL.md diff --git a/docs/JIRA_INTEGRATION_PROPOSAL.md b/docs/JIRA_INTEGRATION_PROPOSAL.md deleted file mode 100644 index e8c44b22..00000000 --- a/docs/JIRA_INTEGRATION_PROPOSAL.md +++ /dev/null @@ -1,176 +0,0 @@ -# Jira Integration — Ticket-Linked DORA Metrics - -**For review and approval** -Prepared 2026-08-10 · Clustox/middleware - ---- - -## 1. What this is - -Today, DORA metrics (Lead Time for Changes, Cycle Time, etc.) start the clock at the **first -commit**. This adds Jira as a linked data source so the clock can start earlier — at the moment a -ticket was created — and lets teams see how their Jira workflow (sprints, ticket status) relates to -what actually shipped. - -In one sentence: **link Jira tickets to the PRs that closed them, so "idea to production" is a real, -measurable number instead of just "commit to production."** - ---- - -## 2. The good news — this isn't starting from zero - -This fork is a strip-down of an upstream product that **already had Jira support**, later removed. -The schema, error handling, and naming are still there, just disconnected: - -| Already exists, unused | What it's for | -|---|---| -| `Integration.JIRA` enum | Marks Jira as a linkable provider, same list as GitHub/GitLab | -| `Integration` table (generic, works today) | Stores an encrypted token per org per provider — no changes needed to store a Jira token | -| `IntegrationGroup.PROJECT` | The category Jira belongs to, alongside `CODE` (GitHub/GitLab) | -| `OrgProject` / `TeamProjects` tables | Exactly the shape needed for "which Jira project belongs to which team" — mirrors how repos map to teams today | -| `Tickets` / `TicketState` tables | Landing tables for issues and their status history — already has fields for sprint, story points, status | -| Jira error codes + banner UI | "Your Jira token expired" messaging is already wired up, just never triggered | - -**What's actually missing:** the Python backend doesn't yet recognize Jira as a provider, there's no -job that pulls data *from* Jira, and nothing yet matches a PR to the ticket it closes. Those three -gaps are the real work — everything else is reusable as-is. - ---- - -## 3. What we're required to have - -To connect an organization's Jira, we need three things from whoever sets it up (a Jira admin or -any user with access to the relevant projects): - -| Field | What it is | Where to get it | -|---|---|---| -| **Jira Site URL** | e.g. `yourcompany.atlassian.net` | Already known — it's the URL they use to open Jira | -| **Email** | The Atlassian account email | Their own login email | -| **API Token** | A personal access token for that account | Generated at `id.atlassian.com/manage-profile/security/api-tokens` — takes about 30 seconds | - -No Jira admin approval or app installation is required — this uses the same kind of personal API -token GitHub/GitLab linking already asks for, just with an email attached (Jira Cloud's API uses -email + token instead of a single token). - ---- - -## 4. What we'll fetch - -Once linked, we pull two kinds of data, both **read-only** — nothing is written back to Jira in this -phase: - -1. **Projects** — the list of Jira projects the linked account can see, so a team can pick which - one(s) it works out of (a team might map to one Jira project, or several). -2. **Issues, for the projects a team has selected** — ticket key, summary, status, assignee, - reporter, story points, sprint, and **status history** (when it moved from "To Do" → "In - Progress" → "Done", etc.). This is what lets us measure time-in-status, not just current status. - -Fetching happens on a schedule (same mechanism as the existing GitHub/GitLab sync), pulling only -what's changed since the last sync — not the whole project history every time. - ---- - -## 5. How it will work — the flow - -``` - 1. Link Jira → Site URL + email + API token, verified with a test call - (mirrors the existing "Configure GitHub" flow) - - 2. Pick projects → For each team: which Jira project(s) does it work out of? - (mirrors the existing repo-picker in team settings) - - 3. Sync runs on schedule → Pull issues + status history for selected projects - (new step alongside the existing repo/incident sync jobs) - - 4. Match tickets to PRs → Scan each PR's title and branch name for a ticket key - (e.g. a PR titled "PAY-482: fix refund rounding" or a - branch named "PAY-482-fix-rounding" both match "PAY-482") - - 5. Metrics + UI update → Once a PR is matched to a ticket, its lead time can include - the ticket's own timeline, and ticket-only metrics (sprint - velocity, cycle time) become available independent of code -``` - -Step 4 is the one genuinely new piece of logic in this whole integration — everything else follows -an existing pattern already used for GitHub/GitLab. - ---- - -## 6. What we'll display - -**A. Integrations page** — a third card next to GitHub/GitLab, same look: shows "Linked" once -connected, "Link" to connect, standard unlink flow. - -**B. Team settings** — a new "Projects" tab next to the existing "Repos" tab, where a team checks -off which Jira project(s) it's tracking against. - -**C. DORA Metrics page — two additions:** - -- **Extended Lead Time breakdown.** The existing phase chart (`first commit → PR opened → review → - merge → deploy`) gets an optional leading phase, `ticket created → first commit`, for any PR that - matched a ticket. This is the actual payoff of the whole integration — a true "idea to production" - number. -- **Ticket Cycle Time widget.** Independent of code: how long tickets spend in each status, sourced - from `TicketState`. Useful even for teams that want workflow visibility without caring about the - code-metrics correlation. - -**D. Sprint rollup chart** — tickets planned vs. shipped per sprint, and velocity trend, using the -sprint data Jira already tracks. - -**E. Data-hygiene callout** — a small, honest counter: *"14 PRs merged this week with no linked -ticket."* This surfaces process gaps (people not following a ticket-naming convention) rather than -silently having those PRs excluded from ticket-aware metrics with no explanation. - ---- - -## 7. Decisions needed - -**Q1. What's the actual ticket-key format your team uses in PR titles/branches?** -The matching in step 4 needs a real pattern to match against — Jira's default is -`[PROJECT-KEY]-[NUMBER]` (e.g. `PAY-482`), but teams sometimes prefix branches differently (e.g. -`feature/PAY-482-...` vs `PAY-482/...`). *Recommend: confirm the convention(s) actually in use before -building the matcher, rather than guessing and re-tuning later.* - -**Q2. How often should Jira sync run?** -GitHub/GitLab sync already runs on a schedule; Jira can either share that cadence or run -independently. *Recommend: same cadence as the existing repo sync* — no reason for it to be -different, and it's one less schedule to reason about. - -**Q3. Should a PR with no matching ticket block anything, or just be a visibility callout?** -*Recommend: visibility only (§6E).* Blocking merges on ticket-linking is a process decision separate -from this integration and would need its own discussion. - ---- - -## 8. Honest costs - -- **Phase 4 (ticket-PR matching) is the one piece with real design risk.** Everything before it is - "wire up an existing pattern to a new provider"; this step is genuinely new logic, and its accuracy - depends entirely on how consistently the team already follows a ticket-naming convention. -- **This is read-only from Jira's side in phase 1.** No comments, transitions, or writes back to - Jira — keeps the initial scope safe and reversible. Bi-directional sync (e.g. posting deploy status - as a Jira comment) would be a deliberate later phase, not part of this proposal. -- **Sync cost scales with linked projects**, same as GitHub/GitLab — each org's Jira token has its - own API rate limit, so this doesn't compound against other orgs' syncs. - ---- - -## 9. Suggested sequence - -Each phase below ships independently and is demoable on its own — there's no need to build all four -before anything is usable. - -1. **Link Jira** (§3) — a working "Linked" badge and stored credentials. No visible metrics change - yet, but it's the foundation everything else needs. -2. **Project selection + sync** (§4, §6B) — Jira issues start landing in the database; the Ticket - Cycle Time widget (§6C) becomes possible on its own, independent of code data. -3. **Ticket-PR matching** (§5 step 4) — the risky, judgment-heavy piece; needs Q1 answered first. -4. **Metrics + UI** (§6) — Extended Lead Time breakdown, sprint rollup, and the data-hygiene - callout, once matching is in place. - ---- - -## What I need - -Answers to the three questions in §7 (or "go with the recommendations"), and a go/no-go on starting -with Phase 1. From 0afb8fad132ccc13fcc8374a1be71d74c9f5f9bc Mon Sep 17 00:00:00 2001 From: Batool Fatima Date: Tue, 11 Aug 2026 04:30:16 +0500 Subject: [PATCH 118/194] feat(jira): Phase 2 -- team-level Jira project selection MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds the "which Jira project(s) does this team track" half of Phase 2 (docs/JIRA_INTEGRATION_PROPOSAL.md §9.2) on top of Phase 1's link/unlink. Deliberately scoped to selection only -- no issue/ticket sync yet, that's a separate, larger follow-up. Backend (analytics_server): - New "OrgProject"/"TeamProjects" tables (migration 20260811120000_clustox_jira_projects.sql), mirroring OrgRepo/TeamRepos minus the code-specific columns. idempotency_key is scoped as "jira::", not the bare Jira id -- unlike GitLab's single shared instance, each org's Jira site is independent, and Jira Cloud project ids are small site-local integers that two orgs' sites can otherwise collide on. - mhq/store/models/projects, mhq/store/repos/projects.py (ProjectRepoService), mhq/service/project/repository_service.py (ProjectService) -- mirror OrgRepo/CodeRepoService/RepositoryService's shape, batch-loading by idempotency_key up front rather than querying per item. - GET/PUT /teams//projects on mhq/api/teams.py, same full-replace contract as the existing /teams//repos. Web-server (BFF + UI): - pages/api/internal/[org_id]/jira_project_search.ts -- live project search proxied to Jira's own /rest/api/3/project/search. This fork's actual repo-picker (pages/api/internal/[org_id]/utils.ts) already searches GitHub/GitLab live rather than a pre-synced catalog, so this mirrors that rather than standing up a sync job just to make the box searchable. - pages/api/resources/team_projects.ts -- thin GET/PUT proxy to the Python endpoint above. - src/components/Teams/TeamJiraProjects.tsx + useTeamJiraProjectsConfig.tsx -- a "Jira Projects" section in the team-edit form, deliberately its own component/hook/save action rather than folded into useTeamsConfig.tsx's repo-selection flow: Jira projects have none of a repo's deployment-type/workflow concerns, and keeping it separate means this can't regress the already-working repo-selection flow. - src/constants/db.ts: added Columns entries for "OrgProject"/ "TeamProjects" -- both names were already sitting unused in the upstream-derived TableT enum (real leftover, see the doc correction below), just never given real columns. Corrected docs/JIRA_INTEGRATION_PROPOSAL.md: its §2 table claimed IntegrationGroup, OrgProject/TeamProjects, and Tickets/TicketState already existed in this fork, disconnected but reusable. They didn't -- only the JIRA enum value and the frontend error-code/banner wiring were real. Left the original table for history, added a correction above it so it stops reading as an accurate account of the codebase. Regression check: tsc --noEmit clean. Jest: 222 passed, same 2 pre-existing unrelated failures as before this branch (array.test.ts date-arithmetic, git_org_repos.test.ts jose/ESM parsing). Python: 164 passed (159 baseline + 5 new ProjectService tests). Verified against a from-scratch container rebuild, including the migration actually applying (dbmate up, confirmed via \d "OrgProject"). Known gaps, deliberately out of scope for this slice: no store-layer DB-backed test (this repo has no DB test fixture to hook into -- see the ProjectService tests' own comment), no direct hook-level test for useTeamJiraProjectsConfig (matches this repo's existing convention of testing at the component/route boundary, not raw hooks), no new Playwright E2E spec for this feature. Co-Authored-By: Claude Sonnet 5 (cherry picked from commit 3c11ff388abcd2b5295e7f9f8f71a43167669ce1) --- .../analytics_server/mhq/api/request_utils.py | 15 ++ .../mhq/api/resources/project_resources.py | 21 ++ backend/analytics_server/mhq/api/teams.py | 46 ++++- .../mhq/service/project/__init__.py | 1 + .../mhq/service/project/models/org_project.py | 10 + .../mhq/service/project/repository_service.py | 157 +++++++++++++++ .../mhq/store/models/projects/__init__.py | 4 + .../mhq/store/models/projects/repository.py | 50 +++++ .../mhq/store/repos/projects.py | 126 ++++++++++++ .../tests/service/project/__init__.py | 0 .../project/test_repository_service.py | 185 ++++++++++++++++++ .../20260811120000_clustox_jira_projects.sql | 54 +++++ .../__tests__/jira_project_search.test.ts | 166 ++++++++++++++++ .../internal/[org_id]/jira_project_search.ts | 97 +++++++++ .../resources/__tests__/team_projects.test.ts | 120 ++++++++++++ .../pages/api/resources/team_projects.ts | 63 ++++++ .../src/components/Teams/CreateTeams.tsx | 9 +- .../src/components/Teams/TeamJiraProjects.tsx | 184 +++++++++++++++++ .../Teams/__tests__/TeamJiraProjects.test.tsx | 127 ++++++++++++ .../Teams/useTeamJiraProjectsConfig.tsx | 169 ++++++++++++++++ web-server/src/constants/db.ts | 29 +++ 21 files changed, 1631 insertions(+), 2 deletions(-) create mode 100644 backend/analytics_server/mhq/api/resources/project_resources.py create mode 100644 backend/analytics_server/mhq/service/project/__init__.py create mode 100644 backend/analytics_server/mhq/service/project/models/org_project.py create mode 100644 backend/analytics_server/mhq/service/project/repository_service.py create mode 100644 backend/analytics_server/mhq/store/models/projects/__init__.py create mode 100644 backend/analytics_server/mhq/store/models/projects/repository.py create mode 100644 backend/analytics_server/mhq/store/repos/projects.py create mode 100644 backend/analytics_server/tests/service/project/__init__.py create mode 100644 backend/analytics_server/tests/service/project/test_repository_service.py create mode 100644 database-docker/db/migrations/20260811120000_clustox_jira_projects.sql create mode 100644 web-server/pages/api/internal/[org_id]/__tests__/jira_project_search.test.ts create mode 100644 web-server/pages/api/internal/[org_id]/jira_project_search.ts create mode 100644 web-server/pages/api/resources/__tests__/team_projects.test.ts create mode 100644 web-server/pages/api/resources/team_projects.ts create mode 100644 web-server/src/components/Teams/TeamJiraProjects.tsx create mode 100644 web-server/src/components/Teams/__tests__/TeamJiraProjects.test.tsx create mode 100644 web-server/src/components/Teams/useTeamJiraProjectsConfig.tsx diff --git a/backend/analytics_server/mhq/api/request_utils.py b/backend/analytics_server/mhq/api/request_utils.py index 0ffc819a..e1fabecf 100644 --- a/backend/analytics_server/mhq/api/request_utils.py +++ b/backend/analytics_server/mhq/api/request_utils.py @@ -9,6 +9,7 @@ from werkzeug.exceptions import BadRequest from mhq.store.models.code.repository import TeamRepos from mhq.service.code.models.org_repo import RawTeamOrgRepo +from mhq.service.project.models.org_project import RawTeamOrgProject from mhq.store.models.code import WorkflowFilter, CodeProvider from mhq.service.workflows.workflow_filter import get_workflow_filter_processor @@ -116,3 +117,17 @@ def coerce_team_repo(team_repo: Dict[str, str]) -> TeamRepos: def coerce_team_repos(repos: List[Dict[str, str]]) -> List[TeamRepos]: return [coerce_team_repo(repo) for repo in repos] + + +def coerce_org_project(project: Dict[str, str]) -> RawTeamOrgProject: + return RawTeamOrgProject( + team_id=project.get("team_id"), + provider=project.get("provider"), + key=project.get("key"), + name=project.get("name"), + idempotency_key=project.get("idempotency_key"), + ) + + +def coerce_org_projects(projects: List[Dict[str, str]]) -> List[RawTeamOrgProject]: + return [coerce_org_project(project) for project in projects] diff --git a/backend/analytics_server/mhq/api/resources/project_resources.py b/backend/analytics_server/mhq/api/resources/project_resources.py new file mode 100644 index 00000000..d09ac807 --- /dev/null +++ b/backend/analytics_server/mhq/api/resources/project_resources.py @@ -0,0 +1,21 @@ +from typing import Dict, List + +from mhq.store.models.projects import OrgProject + + +def adapt_org_project(org_project: OrgProject) -> Dict[str, any]: + return { + "id": str(org_project.id), + "org_id": str(org_project.org_id), + "key": org_project.key, + "name": org_project.name, + "provider": org_project.provider, + "is_active": org_project.is_active, + "idempotency_key": org_project.idempotency_key, + "created_at": org_project.created_at.isoformat(), + "updated_at": org_project.updated_at.isoformat(), + } + + +def adapt_org_projects(org_projects: List[OrgProject]) -> List[Dict[str, any]]: + return [adapt_org_project(project) for project in org_projects] diff --git a/backend/analytics_server/mhq/api/teams.py b/backend/analytics_server/mhq/api/teams.py index 08b0f32d..a06f7ac0 100644 --- a/backend/analytics_server/mhq/api/teams.py +++ b/backend/analytics_server/mhq/api/teams.py @@ -4,16 +4,25 @@ from werkzeug.exceptions import BadRequest from mhq.store.models.code.repository import OrgRepo, TeamRepos from mhq.service.code.models.org_repo import RawTeamOrgRepo +from mhq.service.project.models.org_project import RawTeamOrgProject from mhq.api.resources.code_resouces import ( adapt_team_repos, adapt_team_repo_and_org_repo, ) +from mhq.api.resources.project_resources import adapt_org_projects from mhq.service.code.repository_service import get_repository_service +from mhq.service.project.repository_service import get_project_service from mhq.api.resources.core_resources import adapt_team from mhq.store.models.core.teams import Team +from mhq.store.models.projects import OrgProject from mhq.service.core.teams import get_team_service -from mhq.api.request_utils import coerce_org_repos, coerce_team_repos, dataschema +from mhq.api.request_utils import ( + coerce_org_repos, + coerce_org_projects, + coerce_team_repos, + dataschema, +) from mhq.service.query_validator import get_query_validator app = Blueprint("teams", __name__) @@ -156,3 +165,38 @@ def patch_team_repos_mapping(team_id: str, team_repos_data: List[TeamRepos]): team_repos_service = get_repository_service() team_repos = team_repos_service.patch_team_repos_mapping(team, team_repos_data) return adapt_team_repos(team_repos) + + +# CLUSTOX: Jira integration, Phase 2 (project selection) -- a team's chosen +# Jira project(s), same "GET the current set / PUT the full replacement set" +# shape as /teams//repos above. See +# docs/JIRA_INTEGRATION_PROPOSAL.md. +@app.route("/teams//projects", methods={"GET"}) +def fetch_team_projects(team_id: str): + + query_validator = get_query_validator() + team: Team = query_validator.team_validator(team_id) + + project_service = get_project_service() + team_org_projects: List[OrgProject] = project_service.get_team_projects(team) + + return adapt_org_projects(team_org_projects) + + +@app.route("/teams//projects", methods={"PUT"}) +@dataschema( + Schema( + { + Required("projects"): All(list, Coerce(coerce_org_projects)), + } + ), +) +def update_team_projects(team_id: str, projects: List[RawTeamOrgProject]): + + query_validator = get_query_validator() + team: Team = query_validator.team_validator(team_id) + + project_service = get_project_service() + updated_org_projects = project_service.update_team_projects(team, projects) + + return adapt_org_projects(updated_org_projects) diff --git a/backend/analytics_server/mhq/service/project/__init__.py b/backend/analytics_server/mhq/service/project/__init__.py new file mode 100644 index 00000000..073ab862 --- /dev/null +++ b/backend/analytics_server/mhq/service/project/__init__.py @@ -0,0 +1 @@ +from .repository_service import get_project_service, ProjectService diff --git a/backend/analytics_server/mhq/service/project/models/org_project.py b/backend/analytics_server/mhq/service/project/models/org_project.py new file mode 100644 index 00000000..8f88d492 --- /dev/null +++ b/backend/analytics_server/mhq/service/project/models/org_project.py @@ -0,0 +1,10 @@ +from dataclasses import dataclass + + +@dataclass +class RawTeamOrgProject: + team_id: str + provider: str + key: str + name: str + idempotency_key: str diff --git a/backend/analytics_server/mhq/service/project/repository_service.py b/backend/analytics_server/mhq/service/project/repository_service.py new file mode 100644 index 00000000..5398fccc --- /dev/null +++ b/backend/analytics_server/mhq/service/project/repository_service.py @@ -0,0 +1,157 @@ +from typing import Dict, List + +from mhq.service.project.models.org_project import RawTeamOrgProject +from mhq.store.models.core import Team +from mhq.store.models.projects import OrgProject +from mhq.store.repos.projects import ProjectRepoService, TeamProjects +from mhq.utils.string import uuid4_str + + +class ProjectService: + """ + Business layer for a team's selected project-tracking-tool projects + (Jira, to start). Mirrors RepositoryService's repo/team methods, minus + the incident-service coupling that's specific to code repos -- see + docs/JIRA_INTEGRATION_PROPOSAL.md for the phase this belongs to. + """ + + def __init__(self, project_repo_service: ProjectRepoService): + self._project_repo_service = project_repo_service + + def get_team_projects(self, team: Team) -> List[OrgProject]: + return self._project_repo_service.get_team_projects(team_id=str(team.id)) + + def get_project_id_team_projects_map( + self, team: Team, org_projects: List[OrgProject] + ) -> Dict[str, TeamProjects]: + project_ids = [str(project.id) for project in org_projects] + team_projects: List[TeamProjects] = ( + self._project_repo_service.get_team_projects_by_project_id_for_team( + team.id, project_ids + ) + ) + return {str(tp.org_project_id): tp for tp in team_projects} + + def update_team_projects( + self, team: Team, raw_org_projects: List[RawTeamOrgProject] + ) -> List[OrgProject]: + updated_projects = self._update_org_projects(team.org_id, raw_org_projects) + self._update_team_projects(team, updated_projects, raw_org_projects) + self._set_unused_projects_as_inactive(team.org_id) + + return updated_projects + + def _update_org_projects( + self, org_id: str, raw_org_projects: List[RawTeamOrgProject] + ) -> List[OrgProject]: + idempotency_keys = [project.idempotency_key for project in raw_org_projects] + + # One batch lookup for all incoming projects, not one query per + # project -- avoids the per-item DB round trip this would otherwise + # cost (see docs/JIRA_INTEGRATION_PROPOSAL.md's review notes). + existing_org_projects = ( + self._project_repo_service.get_projects_by_idempotency_keys( + idempotency_keys + ) + ) + idempotency_key_to_project_map = { + project.idempotency_key: project for project in existing_org_projects + } + + updated_org_projects = [] + for raw_project in raw_org_projects: + existing_project = idempotency_key_to_project_map.get( + raw_project.idempotency_key + ) + if existing_project: + if str(existing_project.org_id) != str(org_id): + raise Exception( + "Data integrity error, matching idempotency key across " + f"orgs. Team OrgId: {str(org_id)}. Existing Project " + f"OrgID: {str(existing_project.org_id)}. " + f"idempotency_key: {raw_project.idempotency_key}" + ) + + existing_project.is_active = True + existing_project.key = raw_project.key + existing_project.name = raw_project.name + updated_org_projects.append(existing_project) + else: + updated_org_projects.append( + OrgProject( + id=uuid4_str(), + org_id=org_id, + key=raw_project.key, + name=raw_project.name, + provider=raw_project.provider, + idempotency_key=raw_project.idempotency_key, + # Explicit rather than relying on the column's + # SQLAlchemy-level default -- that default only + # materializes on this instance once it's actually + # flushed through a real DB session, which makes + # the object momentarily wrong (is_active=None) to + # anything inspecting it beforehand. + is_active=True, + ) + ) + + return self._project_repo_service.update_org_projects(updated_org_projects) + + def _update_team_projects( + self, + team: Team, + updated_org_projects: List[OrgProject], + raw_projects_data: List[RawTeamOrgProject], + ): + existing_team_projects = self._project_repo_service.get_existing_team_projects( + team + ) + for team_project in existing_team_projects: + team_project.is_active = False + + project_id_to_team_project_map = { + str(tp.org_project_id): tp for tp in existing_team_projects + } + idempotency_key_raw_project_map = { + raw.idempotency_key: raw for raw in raw_projects_data + } + + updated_team_projects = [] + for project in updated_org_projects: + team_project = project_id_to_team_project_map.get(str(project.id)) + # Every project in updated_org_projects came from raw_projects_data + # (see _update_org_projects), so this lookup should never miss -- + # asserting rather than silently defaulting surfaces a broken + # idempotency_key match instead of writing a mystery row. + assert idempotency_key_raw_project_map.get(project.idempotency_key) + + if team_project: + team_project.is_active = True + else: + team_project = TeamProjects( + team_id=team.id, + org_project_id=str(project.id), + ) + + updated_team_projects.append(team_project) + + self._project_repo_service.update_team_projects(updated_team_projects) + + def _set_unused_projects_as_inactive(self, org_id: str) -> List[OrgProject]: + active_projects = self._project_repo_service.get_active_org_projects(org_id) + active_projects_used_across_teams = ( + self._project_repo_service.get_org_projects_used_across_teams(org_id) + ) + active_used_ids = { + str(project.id) for project in active_projects_used_across_teams + } + + for project in active_projects: + if str(project.id) not in active_used_ids: + project.is_active = False + + return self._project_repo_service.update_org_projects(active_projects) + + +def get_project_service() -> ProjectService: + return ProjectService(ProjectRepoService()) diff --git a/backend/analytics_server/mhq/store/models/projects/__init__.py b/backend/analytics_server/mhq/store/models/projects/__init__.py new file mode 100644 index 00000000..05a69a87 --- /dev/null +++ b/backend/analytics_server/mhq/store/models/projects/__init__.py @@ -0,0 +1,4 @@ +from .repository import ( + OrgProject, + TeamProjects, +) diff --git a/backend/analytics_server/mhq/store/models/projects/repository.py b/backend/analytics_server/mhq/store/models/projects/repository.py new file mode 100644 index 00000000..18267bbc --- /dev/null +++ b/backend/analytics_server/mhq/store/models/projects/repository.py @@ -0,0 +1,50 @@ +import uuid + +from sqlalchemy import func +from sqlalchemy.dialects.postgresql import UUID + +from mhq.store import db + + +class OrgProject(db.Model): + """ + Org-level catalog of a project-tracking tool's projects (Jira, to start). + + Mirrors OrgRepo -- same "org-wide catalog, team join table" shape -- + minus the code-specific columns (default_branch, language, contributors) + that have no equivalent for a project-tracking tool. See + docs/JIRA_INTEGRATION_PROPOSAL.md for the phase this belongs to. + """ + + __tablename__ = "OrgProject" + + id = db.Column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4) + org_id = db.Column(UUID(as_uuid=True), db.ForeignKey("Organization.id")) + key = db.Column(db.String) + name = db.Column(db.String) + provider = db.Column(db.String) + idempotency_key = db.Column(db.String) + is_active = db.Column(db.Boolean, default=True) + created_at = db.Column(db.DateTime(timezone=True), server_default=func.now()) + updated_at = db.Column( + db.DateTime(timezone=True), server_default=func.now(), onupdate=func.now() + ) + + def __hash__(self): + return hash(self.id) + + +class TeamProjects(db.Model): + """Join table: which OrgProject(s) a team tracks. Mirrors TeamRepos.""" + + __tablename__ = "TeamProjects" + + team_id = db.Column(UUID(as_uuid=True), db.ForeignKey("Team.id"), primary_key=True) + org_project_id = db.Column( + UUID(as_uuid=True), db.ForeignKey("OrgProject.id"), primary_key=True + ) + is_active = db.Column(db.Boolean, default=True) + created_at = db.Column(db.DateTime(timezone=True), server_default=func.now()) + updated_at = db.Column( + db.DateTime(timezone=True), server_default=func.now(), onupdate=func.now() + ) diff --git a/backend/analytics_server/mhq/store/repos/projects.py b/backend/analytics_server/mhq/store/repos/projects.py new file mode 100644 index 00000000..1933df3a --- /dev/null +++ b/backend/analytics_server/mhq/store/repos/projects.py @@ -0,0 +1,126 @@ +from typing import List + +from sqlalchemy import and_ + +from mhq.store import db, rollback_on_exc +from mhq.store.models.core import Team +from mhq.store.models.projects import OrgProject, TeamProjects + + +class ProjectRepoService: + """ + Store layer for Jira (and, later, other project-tracking tools') org + project catalog + team selection. Mirrors CodeRepoService's repo/team + methods -- see docs/JIRA_INTEGRATION_PROPOSAL.md. + """ + + def __init__(self): + self._db = db + + @rollback_on_exc + def get_active_org_projects(self, org_id: str) -> List[OrgProject]: + return ( + self._db.session.query(OrgProject) + .filter(OrgProject.org_id == org_id, OrgProject.is_active.is_(True)) + .all() + ) + + @rollback_on_exc + def get_projects_by_idempotency_keys( + self, idempotency_keys: List[str] + ) -> List[OrgProject]: + if not idempotency_keys: + return [] + + return ( + self._db.session.query(OrgProject) + .filter(OrgProject.idempotency_key.in_(idempotency_keys)) + .all() + ) + + @rollback_on_exc + def get_projects_by_ids(self, ids: List[str]) -> List[OrgProject]: + if not ids: + return [] + + return self._db.session.query(OrgProject).filter(OrgProject.id.in_(ids)).all() + + @rollback_on_exc + def update_org_projects(self, org_projects: List[OrgProject]) -> List[OrgProject]: + [self._db.session.merge(org_project) for org_project in org_projects] + self._db.session.commit() + return self.get_projects_by_ids([str(project.id) for project in org_projects]) + + @rollback_on_exc + def get_existing_team_projects(self, team: Team) -> List[TeamProjects]: + return ( + self._db.session.query(TeamProjects) + .filter(TeamProjects.team_id == team.id) + .all() + ) + + @rollback_on_exc + def get_team_projects_by_project_id_for_team( + self, team_id: str, project_ids: List[str] + ) -> List[TeamProjects]: + if not project_ids: + return [] + + return ( + self._db.session.query(TeamProjects) + .filter( + TeamProjects.team_id == team_id, + TeamProjects.org_project_id.in_(project_ids), + ) + .all() + ) + + @rollback_on_exc + def update_team_projects(self, updated_team_projects: List[TeamProjects]): + for team_project in updated_team_projects: + self._db.session.merge(team_project) + + self._db.session.commit() + + @rollback_on_exc + def get_team_projects_by_team_id(self, team_id: str) -> List[TeamProjects]: + return ( + self._db.session.query(TeamProjects) + .filter( + and_( + TeamProjects.team_id == team_id, + TeamProjects.is_active == True, # noqa E712 + ) + ) + .all() + ) + + @rollback_on_exc + def get_team_projects(self, team_id: str) -> List[OrgProject]: + team_projects = self.get_team_projects_by_team_id(team_id) + if not team_projects: + return [] + + project_ids = [tp.org_project_id for tp in team_projects] + return self.get_projects_by_ids(project_ids) + + @rollback_on_exc + def get_org_projects_used_across_teams(self, org_id: str) -> List[OrgProject]: + """ + Active org projects that are also actively selected by at least one + non-deleted team -- mirrors + CodeRepoService.get_org_repos_used_across_teams, used to retire + catalog rows nothing points at anymore. + """ + return ( + self._db.session.query(OrgProject) + .join(TeamProjects, TeamProjects.org_project_id == OrgProject.id) + .join(Team, TeamProjects.team_id == Team.id) + .filter( + OrgProject.org_id == org_id, + OrgProject.is_active.is_(True), + TeamProjects.is_active.is_(True), + Team.is_deleted.is_(False), + ) + .all() + ) diff --git a/backend/analytics_server/tests/service/project/__init__.py b/backend/analytics_server/tests/service/project/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/backend/analytics_server/tests/service/project/test_repository_service.py b/backend/analytics_server/tests/service/project/test_repository_service.py new file mode 100644 index 00000000..623268ca --- /dev/null +++ b/backend/analytics_server/tests/service/project/test_repository_service.py @@ -0,0 +1,185 @@ +from unittest.mock import MagicMock + +import pytest + +from mhq.service.project.models.org_project import RawTeamOrgProject +from mhq.service.project.repository_service import ProjectService +from mhq.store.models.projects import OrgProject, TeamProjects + +# CLUSTOX: Jira integration, Phase 2 (project selection). Mirrors this +# repo's existing service-layer test style (fake/mock collaborators, no +# real DB -- see tests/service/code/test_lead_time_service.py) since this +# codebase has no DB-backed test fixture to hook a store-layer integration +# test into. See docs/JIRA_INTEGRATION_PROPOSAL.md. + + +class FakeTeam: + def __init__(self, id="team-1", org_id="org-1"): + self.id = id + self.org_id = org_id + + +def _raw( + team_id="team-1", + key="PAY", + name="Payments", + idempotency_key="jira:org-1:10001", + provider="jira", +) -> RawTeamOrgProject: + return RawTeamOrgProject( + team_id=team_id, + provider=provider, + key=key, + name=name, + idempotency_key=idempotency_key, + ) + + +def _service(project_repo_service=None) -> ProjectService: + return ProjectService(project_repo_service or MagicMock()) + + +def test_creates_a_new_org_project_for_an_unseen_idempotency_key(): + repo = MagicMock() + repo.get_projects_by_idempotency_keys.return_value = [] + repo.update_org_projects.side_effect = lambda projects: projects + repo.get_existing_team_projects.return_value = [] + repo.get_active_org_projects.return_value = [] + repo.get_org_projects_used_across_teams.return_value = [] + + team = FakeTeam() + raw = _raw() + + result = _service(repo).update_team_projects(team, [raw]) + + assert len(result) == 1 + created = result[0] + assert str(created.org_id) == team.org_id + assert created.key == raw.key + assert created.name == raw.name + assert created.idempotency_key == raw.idempotency_key + assert created.is_active is True + + +def test_reuses_and_reactivates_an_existing_org_project_matched_by_idempotency_key(): + # Previously unselected by every team (is_active=False) and, per the + # incoming payload, renamed/rekeyed in Jira since we last saw it. + existing = OrgProject( + id="proj-1", + org_id="org-1", + key="OLD-KEY", + name="Old Name", + provider="jira", + idempotency_key="jira:org-1:10001", + is_active=False, + ) + repo = MagicMock() + repo.get_projects_by_idempotency_keys.return_value = [existing] + repo.update_org_projects.side_effect = lambda projects: projects + repo.get_existing_team_projects.return_value = [] + repo.get_active_org_projects.return_value = [existing] + repo.get_org_projects_used_across_teams.return_value = [existing] + + team = FakeTeam() + raw = _raw(key="PAY", name="Payments") + + result = _service(repo).update_team_projects(team, [raw]) + + # Reused, not duplicated into a second OrgProject row. + assert result == [existing] + assert existing.key == "PAY" + assert existing.name == "Payments" + assert existing.is_active is True + + +def test_raises_if_the_same_idempotency_key_already_belongs_to_a_different_org(): + # Same construction as OrgRepo's cross-org idempotency-key guard -- + # this should never happen in practice (idempotency_key is scoped by + # org_id, see the migration comment), but a match across orgs means + # something is badly wrong and must fail loudly, not silently adopt + # another org's project row. + existing = OrgProject( + id="proj-1", + org_id="some-other-org", + key="PAY", + name="Payments", + provider="jira", + idempotency_key="jira:org-1:10001", + is_active=True, + ) + repo = MagicMock() + repo.get_projects_by_idempotency_keys.return_value = [existing] + + team = FakeTeam(org_id="org-1") + + with pytest.raises(Exception): + _service(repo).update_team_projects(team, [_raw()]) + + +def test_deselecting_every_project_deactivates_the_teams_link_and_the_org_catalog_row(): + org_project = OrgProject( + id="proj-1", + org_id="org-1", + key="PAY", + name="Payments", + provider="jira", + idempotency_key="jira:org-1:10001", + is_active=True, + ) + existing_team_project = TeamProjects( + team_id="team-1", org_project_id="proj-1", is_active=True + ) + repo = MagicMock() + repo.get_projects_by_idempotency_keys.return_value = [] + repo.update_org_projects.side_effect = lambda projects: projects + repo.get_existing_team_projects.return_value = [existing_team_project] + repo.get_active_org_projects.return_value = [org_project] + # Not used by any team anymore -- this was the only one. + repo.get_org_projects_used_across_teams.return_value = [] + + team = FakeTeam() + + result = _service(repo).update_team_projects(team, []) + + assert result == [] + # The deactivation itself is what matters -- *not* which exact list + # object gets passed to update_team_projects. existing_team_project + # came back from get_existing_team_projects already attached to the + # store's session (same as CodeRepoService.get_existing_team_repos), + # so mutating it here is enough for the next real commit to persist + # it regardless of whether it's also in this particular call's list -- + # mirrors RepositoryService._update_team_repos exactly. + assert existing_team_project.is_active is False + repo.update_team_projects.assert_called_once() + # Nothing points at it anymore, so the org-level catalog row itself is + # retired too, not just this team's link to it -- mirrors + # CodeRepoService.set_unused_repos_as_inactive. + assert org_project.is_active is False + + +def test_a_second_team_selecting_an_already_active_project_does_not_touch_the_first_teams_link(): + org_project = OrgProject( + id="proj-1", + org_id="org-1", + key="PAY", + name="Payments", + provider="jira", + idempotency_key="jira:org-1:10001", + is_active=True, + ) + repo = MagicMock() + repo.get_projects_by_idempotency_keys.return_value = [org_project] + repo.update_org_projects.side_effect = lambda projects: projects + # This team (team-2) has no existing rows of its own yet. + repo.get_existing_team_projects.return_value = [] + repo.get_active_org_projects.return_value = [org_project] + repo.get_org_projects_used_across_teams.return_value = [org_project] + + team_two = FakeTeam(id="team-2", org_id="org-1") + + _service(repo).update_team_projects(team_two, [_raw()]) + + created_team_projects = repo.update_team_projects.call_args[0][0] + assert len(created_team_projects) == 1 + assert str(created_team_projects[0].team_id) == "team-2" + assert str(created_team_projects[0].org_project_id) == "proj-1" diff --git a/database-docker/db/migrations/20260811120000_clustox_jira_projects.sql b/database-docker/db/migrations/20260811120000_clustox_jira_projects.sql new file mode 100644 index 00000000..f562ca42 --- /dev/null +++ b/database-docker/db/migrations/20260811120000_clustox_jira_projects.sql @@ -0,0 +1,54 @@ +-- migrate:up +-- CLUSTOX: Jira integration, Phase 2 (project selection) -- see +-- docs/JIRA_INTEGRATION_PROPOSAL.md. Mirrors "OrgRepo"/"TeamRepos" exactly +-- (org-level catalog + team join table), minus the code-specific columns +-- (default_branch, language, contributors, deployment_type, prod_branches) +-- that don't apply to a project-tracking tool. +-- +-- idempotency_key is NOT the bare Jira project id the way OrgRepo's is the +-- bare GitLab project id -- GitLab project ids are unique across the one +-- shared gitlab.com instance, but every org here links its own independent +-- Jira site, and Jira Cloud project ids are small site-local integers +-- (starting at 10000) that two different orgs' sites can easily collide +-- on. Scoping it as "::" keeps it globally +-- unique the same way OrgRepo's happens to be, rather than assuming a +-- property Jira doesn't actually have. +CREATE TABLE "OrgProject" ( + id uuid PRIMARY KEY DEFAULT extensions.uuid_generate_v4(), + org_id uuid REFERENCES "Organization"(id), + key varchar NOT NULL, + name varchar NOT NULL, + provider varchar NOT NULL, + idempotency_key varchar NOT NULL, + is_active boolean NOT NULL DEFAULT true, + created_at timestamptz NOT NULL DEFAULT now(), + updated_at timestamptz NOT NULL DEFAULT now() +); + +CREATE UNIQUE INDEX orgproject_unique_idempotency_key + ON "OrgProject"(idempotency_key); + +CREATE INDEX org_project_fetch_active_index + ON "OrgProject" USING btree (org_id, is_active); + +CREATE INDEX org_project_search_index + ON "OrgProject" USING btree (org_id, name); + +CREATE TABLE "TeamProjects" ( + team_id uuid NOT NULL REFERENCES "Team"(id), + org_project_id uuid NOT NULL REFERENCES "OrgProject"(id), + is_active boolean NOT NULL DEFAULT true, + created_at timestamptz NOT NULL DEFAULT now(), + updated_at timestamptz NOT NULL DEFAULT now(), + PRIMARY KEY (team_id, org_project_id) +); + +CREATE INDEX team_projects_fetch_index + ON "TeamProjects" USING btree (team_id, is_active); + +CREATE INDEX team_projects_org_project_id_index + ON "TeamProjects" USING btree (org_project_id); + +-- migrate:down +DROP TABLE "TeamProjects"; +DROP TABLE "OrgProject"; diff --git a/web-server/pages/api/internal/[org_id]/__tests__/jira_project_search.test.ts b/web-server/pages/api/internal/[org_id]/__tests__/jira_project_search.test.ts new file mode 100644 index 00000000..e0022856 --- /dev/null +++ b/web-server/pages/api/internal/[org_id]/__tests__/jira_project_search.test.ts @@ -0,0 +1,166 @@ +jest.mock('@/auth/session', () => ({ getAuthSession: jest.fn() })); +jest.mock('axios'); +jest.mock('@/utils/db', () => ({ db: jest.fn() })); +jest.mock('@/utils/auth-supplementary', () => ({ dec: jest.fn() })); + +import axios from 'axios'; + +import { getAuthSession } from '@/auth/session'; +import { dec } from '@/utils/auth-supplementary'; +import { db } from '@/utils/db'; + +import searchHandler from '../jira_project_search'; + +const ORG_ID = '11111111-1111-4111-8111-111111111111'; + +const mockRes = () => { + const res: any = {}; + res.status = jest.fn(() => res); + res.send = jest.fn(() => res); + return res; +}; + +const mockReq = (query: Record = {}) => + ({ method: 'GET', query: { org_id: ORG_ID, ...query }, body: {}, headers: {} }) as any; + +const asAuthed = () => + (getAuthSession as jest.Mock).mockResolvedValue({ + userId: 'u1', + email: 'admin@clustox.com', + name: 'Admin', + role: 'SUPERADMIN' + }); + +// CLUSTOX: this route reads an Integration row via a real knex chain +// (db('Integration').select(...).where(...).first()), not a one-shot +// call -- stub the whole chain rather than just `db` itself. Mirrors the +// same auth/no-DB test setup already used by +// pages/api/integrations/jira/__tests__/validate.test.ts for the rest. +const mockDbChain = (row: any) => { + const chain: any = {}; + chain.select = jest.fn(() => chain); + chain.where = jest.fn(() => chain); + chain.first = jest.fn().mockResolvedValue(row); + (db as unknown as jest.Mock).mockReturnValue(chain); + return chain; +}; + +describe('GET /api/internal/[org_id]/jira_project_search', () => { + beforeEach(() => { + jest.clearAllMocks(); + asAuthed(); + }); + + it('requires an authenticated session', async () => { + (getAuthSession as jest.Mock).mockResolvedValue(null); + mockDbChain(undefined); + const res = mockRes(); + + await searchHandler(mockReq(), res); + + expect(res.status).toHaveBeenCalledWith(401); + expect(axios.get).not.toHaveBeenCalled(); + }); + + it('reports not-linked, and never calls Jira, when the org has no Jira integration row', async () => { + mockDbChain(undefined); + const res = mockRes(); + + await searchHandler(mockReq(), res); + + expect(res.status).toHaveBeenCalledWith(404); + expect(axios.get).not.toHaveBeenCalled(); + }); + + it('searches Jira and maps results, scoping idempotency_key by org_id', async () => { + mockDbChain({ + provider_meta: { site_url: 'mycompany.atlassian.net', email: 'a@b.com' }, + access_token_enc_chunks: ['enc1'] + }); + (dec as jest.Mock).mockReturnValue('decrypted-token'); + (axios.get as jest.Mock).mockResolvedValue({ + data: { + values: [ + { id: '10001', key: 'PAY', name: 'Payments' }, + { id: '10002', key: 'ENG', name: 'Engineering' } + ] + } + }); + const res = mockRes(); + + await searchHandler(mockReq({ search_text: 'pay' }), res); + + expect(axios.get).toHaveBeenCalledWith( + 'https://mycompany.atlassian.net/rest/api/3/project/search', + expect.objectContaining({ + auth: { username: 'a@b.com', password: 'decrypted-token' }, + params: { maxResults: 50, query: 'pay' } + }) + ); + expect(res.status).toHaveBeenCalledWith(200); + expect(res.send).toHaveBeenCalledWith([ + { + id: '10001', + key: 'PAY', + name: 'Payments', + idempotency_key: `jira:${ORG_ID}:10001`, + provider: 'jira' + }, + { + id: '10002', + key: 'ENG', + name: 'Engineering', + idempotency_key: `jira:${ORG_ID}:10002`, + provider: 'jira' + } + ]); + }); + + it('omits the query param when no search_text was given', async () => { + mockDbChain({ + provider_meta: { site_url: 'mycompany.atlassian.net', email: 'a@b.com' }, + access_token_enc_chunks: ['enc1'] + }); + (dec as jest.Mock).mockReturnValue('decrypted-token'); + (axios.get as jest.Mock).mockResolvedValue({ data: { values: [] } }); + const res = mockRes(); + + await searchHandler(mockReq(), res); + + expect(axios.get).toHaveBeenCalledWith( + expect.any(String), + expect.objectContaining({ params: { maxResults: 50 } }) + ); + }); + + it.each([401, 403])( + 'reports 401 when Jira rejects the stored credentials (%d)', + async (status) => { + mockDbChain({ + provider_meta: { site_url: 'mycompany.atlassian.net', email: 'a@b.com' }, + access_token_enc_chunks: ['enc1'] + }); + (dec as jest.Mock).mockReturnValue('decrypted-token'); + (axios.get as jest.Mock).mockRejectedValue({ response: { status } }); + const res = mockRes(); + + await searchHandler(mockReq(), res); + + expect(res.status).toHaveBeenCalledWith(401); + } + ); + + it('reports 502 when Jira is unreachable', async () => { + mockDbChain({ + provider_meta: { site_url: 'mycompany.atlassian.net', email: 'a@b.com' }, + access_token_enc_chunks: ['enc1'] + }); + (dec as jest.Mock).mockReturnValue('decrypted-token'); + (axios.get as jest.Mock).mockRejectedValue(new Error('network down')); + const res = mockRes(); + + await searchHandler(mockReq(), res); + + expect(res.status).toHaveBeenCalledWith(502); + }); +}); diff --git a/web-server/pages/api/internal/[org_id]/jira_project_search.ts b/web-server/pages/api/internal/[org_id]/jira_project_search.ts new file mode 100644 index 00000000..2585e69a --- /dev/null +++ b/web-server/pages/api/internal/[org_id]/jira_project_search.ts @@ -0,0 +1,97 @@ +import axios from 'axios'; +import * as yup from 'yup'; + +import { Endpoint } from '@/api-helpers/global'; +import { Integration } from '@/constants/integrations'; +import { dec } from '@/utils/auth-supplementary'; +import { db } from '@/utils/db'; + +const pathSchema = yup.object().shape({ + org_id: yup.string().uuid().required() +}); + +const getSchema = yup.object().shape({ + search_text: yup.string().optional().nullable() +}); + +export type JiraProjectSearchResult = { + id: string; + key: string; + name: string; + idempotency_key: string; + provider: Integration.JIRA; +}; + +const endpoint = new Endpoint(pathSchema); + +// CLUSTOX: Jira integration, Phase 2 (project selection) -- live project +// search for the team-creation picker. Unlike the GitHub/GitLab repo +// search this mirrors (searchGithubRepos/gitlabSearch in +// pages/api/internal/[org_id]/utils.ts), there's no pre-synced OrgProject +// catalog to search yet in this phase -- Jira's own /project/search +// endpoint already supports server-side name/key filtering, so querying +// it directly here is simpler than standing up a sync job just to make +// this box searchable. See docs/JIRA_INTEGRATION_PROPOSAL.md. +endpoint.handle.GET(getSchema, async (req, res) => { + const { org_id, search_text } = req.payload; + + const integration = await getJiraIntegration(org_id); + const siteUrl = integration?.provider_meta?.site_url; + const email = integration?.provider_meta?.email; + const apiToken = integration?.access_token_enc_chunks + ? dec(integration.access_token_enc_chunks) + : null; + + if (!siteUrl || !email || !apiToken) { + return res + .status(404) + .send({ error: 'Jira is not linked for this workspace' }); + } + + try { + const response = await axios.get( + `https://${siteUrl}/rest/api/3/project/search`, + { + auth: { username: email, password: apiToken }, + params: { + maxResults: 50, + ...(search_text ? { query: search_text } : {}) + }, + timeout: 8000 + } + ); + + const projects: JiraProjectSearchResult[] = ( + response.data?.values || [] + ).map((project: { id: string; key: string; name: string }) => ({ + id: String(project.id), + key: project.key, + name: project.name, + // Scoped by org_id, not the bare Jira id -- two different orgs' + // independent Jira sites can otherwise land on the same small, + // site-local project id. Mirrors the same reasoning as the + // OrgProject migration's idempotency_key comment. + idempotency_key: `jira:${org_id}:${project.id}`, + provider: Integration.JIRA + })); + + return res.status(200).send(projects); + } catch (error: any) { + // Surfaced as a plain failure rather than a 500 -- an expired/revoked + // Jira token, or Jira itself being briefly unreachable, isn't our own + // server breaking. + const status = error?.response?.status; + return res.status(status === 401 || status === 403 ? 401 : 502).send({ + error: 'Could not reach Jira to search projects.' + }); + } +}); + +export default endpoint.serve(); + +const getJiraIntegration = async (org_id: string) => { + return db('Integration') + .select('provider_meta', 'access_token_enc_chunks') + .where({ org_id, name: Integration.JIRA }) + .first(); +}; diff --git a/web-server/pages/api/resources/__tests__/team_projects.test.ts b/web-server/pages/api/resources/__tests__/team_projects.test.ts new file mode 100644 index 00000000..65db71aa --- /dev/null +++ b/web-server/pages/api/resources/__tests__/team_projects.test.ts @@ -0,0 +1,120 @@ +jest.mock('@/auth/session', () => ({ getAuthSession: jest.fn() })); +jest.mock('@/api-helpers/axios', () => ({ handleRequest: jest.fn() })); + +import { handleRequest } from '@/api-helpers/axios'; +import { getAuthSession } from '@/auth/session'; + +import projectsHandler from '../team_projects'; + +const TEAM_ID = '22222222-2222-4222-8222-222222222222'; + +const mockRes = () => { + const res: any = {}; + res.status = jest.fn(() => res); + res.send = jest.fn(() => res); + return res; +}; + +const mockReq = (method: string, query: Record, body: Record = {}) => + ({ method, query, body, headers: {} }) as any; + +const asAuthed = () => + (getAuthSession as jest.Mock).mockResolvedValue({ + userId: 'u1', + email: 'admin@clustox.com', + name: 'Admin', + role: 'SUPERADMIN' + }); + +// CLUSTOX: this route is a thin proxy to the Python backend's +// /teams//projects -- mirrors team_repos.ts, which this file's +// GET/PUT handlers are modeled on. See +// docs/JIRA_INTEGRATION_PROPOSAL.md. +describe('/api/resources/team_projects', () => { + beforeEach(() => { + jest.clearAllMocks(); + asAuthed(); + }); + + describe('GET', () => { + it('fetches the current team projects and stamps team_id onto every row', async () => { + (handleRequest as jest.Mock).mockResolvedValue([ + { org_project_id: 'proj-1', key: 'PAY', name: 'Payments' } + ]); + const res = mockRes(); + + await projectsHandler(mockReq('GET', { team_id: TEAM_ID }), res); + + expect(handleRequest).toHaveBeenCalledWith(`/teams/${TEAM_ID}/projects`); + expect(res.send).toHaveBeenCalledWith([ + { org_project_id: 'proj-1', key: 'PAY', name: 'Payments', team_id: TEAM_ID } + ]); + }); + }); + + describe('PUT', () => { + it('forwards the full replacement project set to the backend, with team_id on each item', async () => { + (handleRequest as jest.Mock).mockResolvedValue([ + { + org_project_id: 'proj-1', + key: 'PAY', + name: 'Payments', + provider: 'jira', + idempotency_key: 'jira:org-1:10001' + } + ]); + const res = mockRes(); + + await projectsHandler( + mockReq('PUT', { team_id: TEAM_ID }, { + projects: [ + { + key: 'PAY', + name: 'Payments', + provider: 'jira', + idempotency_key: 'jira:org-1:10001' + } + ] + }), + res + ); + + expect(handleRequest).toHaveBeenCalledWith( + `/teams/${TEAM_ID}/projects`, + { + method: 'PUT', + data: { + projects: [ + { + key: 'PAY', + name: 'Payments', + provider: 'jira', + idempotency_key: 'jira:org-1:10001', + team_id: TEAM_ID + } + ] + } + } + ); + expect(res.send).toHaveBeenCalledWith([ + { + org_project_id: 'proj-1', + key: 'PAY', + name: 'Payments', + provider: 'jira', + idempotency_key: 'jira:org-1:10001', + team_id: TEAM_ID + } + ]); + }); + + it('rejects an empty body missing the required projects array', async () => { + const res = mockRes(); + + await projectsHandler(mockReq('PUT', { team_id: TEAM_ID }, {}), res); + + expect(handleRequest).not.toHaveBeenCalled(); + expect(res.status).toHaveBeenCalledWith(400); + }); + }); +}); diff --git a/web-server/pages/api/resources/team_projects.ts b/web-server/pages/api/resources/team_projects.ts new file mode 100644 index 00000000..7619b0be --- /dev/null +++ b/web-server/pages/api/resources/team_projects.ts @@ -0,0 +1,63 @@ +import * as yup from 'yup'; + +import { handleRequest } from '@/api-helpers/axios'; +import { Endpoint, nullSchema } from '@/api-helpers/global'; +import { Row } from '@/constants/db'; + +// CLUSTOX: Jira integration, Phase 2 (project selection) -- a team's +// selected Jira project(s). Deliberately its own standalone resource +// (mirrors team_repos.ts's shape, not wired into teams/v2.ts's combined +// create/update-team request) rather than folded into the existing +// team-repos save flow: project selection has none of repos' deployment +// type/workflow concerns, and keeping it separate means this can't +// regress the already-working repo-selection flow. See +// docs/JIRA_INTEGRATION_PROPOSAL.md. +export type TeamProject = Row<'TeamProjects'> & Row<'OrgProject'>; + +const getSchema = yup.object().shape({ + team_id: yup.string().uuid().required() +}); + +const putSchema = yup.object().shape({ + team_id: yup.string().uuid().required(), + projects: yup + .array() + .of( + yup.object().shape({ + key: yup.string().required(), + name: yup.string().required(), + provider: yup.string().required(), + idempotency_key: yup.string().required() + }) + ) + .required() +}); + +const endpoint = new Endpoint(nullSchema); + +endpoint.handle.GET(getSchema, async (req, res) => { + res.send(await getTeamProjects(req.payload.team_id)); +}); + +export const getTeamProjects = (team_id: ID) => + handleRequest(`/teams/${team_id}/projects`).then((projects) => + projects.map((p) => ({ ...p, team_id })) + ); + +endpoint.handle.PUT(putSchema, async (req, res) => { + const { team_id, projects } = req.payload; + + const updated = await handleRequest( + `/teams/${team_id}/projects`, + { + method: 'PUT', + data: { + projects: projects.map((project) => ({ ...project, team_id })) + } + } + ); + + res.send(updated.map((p) => ({ ...p, team_id }))); +}); + +export default endpoint.serve(); diff --git a/web-server/src/components/Teams/CreateTeams.tsx b/web-server/src/components/Teams/CreateTeams.tsx index 7f57d8a8..6913e704 100644 --- a/web-server/src/components/Teams/CreateTeams.tsx +++ b/web-server/src/components/Teams/CreateTeams.tsx @@ -39,6 +39,7 @@ import { BaseRepo, DeploymentSources } from '@/types/resources'; import { trimWithEllipsis } from '@/utils/stringFormatting'; import { BatchImportModal } from './BatchImportModal'; +import { TeamJiraProjects } from './TeamJiraProjects'; import AnimatedInputWrapper from '../AnimatedInputWrapper/AnimatedInputWrapper'; import { FlexBox } from '../FlexBox'; @@ -65,7 +66,7 @@ export const CreateEditTeams: FC = ({ ); }; -const TeamsCRUD: FC = ({ onSave, onDiscard }) => { +const TeamsCRUD: FC = ({ onSave, onDiscard, teamId }) => { const { isPageLoading, editingTeam, isEditing } = useTeamCRUD(); return ( <> @@ -88,6 +89,12 @@ const TeamsCRUD: FC = ({ onSave, onDiscard }) => { + {/* CLUSTOX: Jira project selection saves independently of the + team's name/repos above (its own Save button) and needs + an existing team_id, so it only appears once a team has + already been created. See + docs/JIRA_INTEGRATION_PROPOSAL.md. */} + {isEditing && } )} diff --git a/web-server/src/components/Teams/TeamJiraProjects.tsx b/web-server/src/components/Teams/TeamJiraProjects.tsx new file mode 100644 index 00000000..a3d77776 --- /dev/null +++ b/web-server/src/components/Teams/TeamJiraProjects.tsx @@ -0,0 +1,184 @@ +import { Close } from '@mui/icons-material'; +import { LoadingButton } from '@mui/lab'; +import { + Autocomplete, + Card, + CircularProgress, + IconButton, + Table, + TableBody, + TableCell, + TableContainer, + TableHead, + TableRow, + TextField, + useTheme +} from '@mui/material'; +import { FC } from 'react'; + +import { FlexBox } from '@/components/FlexBox'; +import { Line } from '@/components/Text'; +import { useAuth } from '@/hooks/useAuth'; + +import { + SelectedJiraProject, + useTeamJiraProjectsConfig +} from './useTeamJiraProjectsConfig'; + +// CLUSTOX: Jira integration, Phase 2 (project selection) -- see +// docs/JIRA_INTEGRATION_PROPOSAL.md. Only rendered for an existing team +// (Jira projects link to a team_id that must already exist) in an org +// that has actually linked Jira; saves independently of the team's +// name/repos via its own button, deliberately decoupled from +// useTeamsConfig.tsx's repo-save flow. +export const TeamJiraProjects: FC<{ teamId: ID }> = ({ teamId }) => { + const { integrations } = useAuth(); + const isJiraLinked = Boolean(integrations?.jira?.integrated); + + if (!teamId || !isJiraLinked) return null; + + return ; +}; + +const TeamJiraProjectsBody: FC<{ teamId: ID }> = ({ teamId }) => { + const { + selectedProjects, + projectOptions, + handleSelectionChange, + unselectProject, + onSearchChange, + isSearching, + isLoading, + isSaving, + onSave + } = useTeamJiraProjectsConfig(teamId); + const theme = useTheme(); + + if (isLoading) { + return ( + + + Loading Jira projects... + + ); + } + + return ( + + + + Jira Projects + + Select the Jira project(s) this team works out of + + + + `${option.key} — ${option.name}`} + isOptionEqualToValue={(option, value) => + option.idempotency_key === value.idempotency_key + } + renderInput={(params) => ( + + {isSearching ? ( + + ) : null} + {params.InputProps.endAdornment} + + ) + }} + /> + )} + renderOption={(props, option, { selected }) => ( +
  • + + + {option.key} — {option.name} + + {selected ? : null} + +
  • + )} + renderTags={() => null} + /> + + Save + +
    + + +
    + ); +}; + +const SelectedProjectsTable: FC<{ + projects: SelectedJiraProject[]; + onRemove: (idempotencyKey: string) => void; + borderColor: string; +}> = ({ projects, onRemove, borderColor }) => { + if (!projects.length) return null; + + return ( + + + + + Key + Name + Action + + + + {projects.map((project) => ( + + {project.key} + {project.name} + + onRemove(project.idempotency_key)} + size="small" + > + + + + + ))} + +
    +
    + ); +}; diff --git a/web-server/src/components/Teams/__tests__/TeamJiraProjects.test.tsx b/web-server/src/components/Teams/__tests__/TeamJiraProjects.test.tsx new file mode 100644 index 00000000..1c66d38d --- /dev/null +++ b/web-server/src/components/Teams/__tests__/TeamJiraProjects.test.tsx @@ -0,0 +1,127 @@ +jest.mock('@/hooks/useAuth', () => ({ useAuth: jest.fn() })); +jest.mock('../useTeamJiraProjectsConfig', () => ({ + useTeamJiraProjectsConfig: jest.fn() +})); + +import { screen } from '@testing-library/react'; +import userEvent from '@testing-library/user-event'; + +import { useAuth } from '@/hooks/useAuth'; +import { renderWithTheme as render } from '@/utils/testUtils'; + +import { TeamJiraProjects } from '../TeamJiraProjects'; +import { + SelectedJiraProject, + useTeamJiraProjectsConfig +} from '../useTeamJiraProjectsConfig'; + +const TEAM_ID = 'team-1'; + +const baseConfig = { + selectedProjects: [] as SelectedJiraProject[], + projectOptions: [] as SelectedJiraProject[], + handleSelectionChange: jest.fn(), + unselectProject: jest.fn(), + onSearchChange: jest.fn(), + isSearching: false, + isLoading: false, + isSaving: false, + onSave: jest.fn() +}; + +describe('TeamJiraProjects', () => { + beforeEach(() => { + jest.clearAllMocks(); + (useTeamJiraProjectsConfig as jest.Mock).mockReturnValue(baseConfig); + }); + + it('renders nothing when there is no team_id yet', () => { + (useAuth as jest.Mock).mockReturnValue({ + integrations: { jira: { integrated: true } } + }); + const { container } = render(); + + expect(container).toBeEmptyDOMElement(); + }); + + it('renders nothing when Jira is not linked for this org', () => { + (useAuth as jest.Mock).mockReturnValue({ integrations: {} }); + const { container } = render(); + + expect(container).toBeEmptyDOMElement(); + }); + + it('shows the project search once Jira is linked and a team exists', () => { + (useAuth as jest.Mock).mockReturnValue({ + integrations: { jira: { integrated: true } } + }); + render(); + + // MUI's outlined TextField renders the label text twice (once as the + // real
    @@ -103,6 +119,30 @@ const TeamsCRUD: FC = ({ onSave, onDiscard, teamId }) => { ); }; +// CLUSTOX: a small eyebrow (icon + uppercase label) above each data-source +// section, so "Code repositories" and "Jira integration" read as two +// distinct, independently-saved sources sharing one page -- not one +// shared form. Deliberately lighter-weight than the h6-style headings +// each section already renders internally ("Team Name", "Jira +// Projects"): this labels the section, those label the field. +const DataSourceSection: FC<{ + icon: ReactNode; + title: string; + children: ReactNode; +}> = ({ icon, title, children }) => ( + + + {icon} + + {title} + + + + {children} + + +); + export const Loader: FC<{ label?: string }> = ({ label = 'Loading...' }) => { return ( @@ -392,7 +432,11 @@ const ActionTray: FC = ({ } }} > - {isSaveLoading ? : 'Save'} + {isSaveLoading ? ( + + ) : ( + 'Save repositories' + )} ); }; + +type UnlinkedPr = { + id: string; + title: string; + url: string; + head_branch: string; + author: string; + merged_at: string; +}; + +// CLUSTOX: Jira integration, Phase 4 (§6E) -- the Data Hygiene +// drill-down. Its own lazy fetch, only issued when the modal actually +// opens -- the count above already answers "how many", this answers +// "which ones", so it isn't worth loading on every DORA Metrics page +// view. See docs/JIRA_INTEGRATION_PROPOSAL.md. +export const UnlinkedPrsModalBody: FC<{ + teamId: ID; + dates: { start: Date; end: Date }; +}> = ({ teamId, dates }) => { + const prs = useEasyState(null); + const isLoading = useBoolState(true); + const hasError = useBoolState(false); + + useEffect(() => { + axios(`/api/internal/team/${teamId}/unlinked_prs`, { + params: { + from_date: dates.start.toISOString(), + to_date: dates.end.toISOString() + } + }) + .then((res) => depFn(prs.set, res.data)) + .catch((error) => { + console.error(error); + depFn(hasError.true); + }) + .finally(isLoading.false); + // eslint-disable-next-line react-hooks/exhaustive-deps + }, [teamId, dates.start, dates.end]); + + if (isLoading.value) + return ( + + + Loading... + + ); + + if (hasError.value) + return Could not load PRs. Please try again.; + + return ( + + + Check each branch/title against your ticket-key convention (e.g. + PROJ-123) -- a PR here either never referenced a ticket, or + referenced one this regex didn't recognize. + + + {prs.value.map((pr) => ( + + + + {pr.title} + + + + + {pr.head_branch} · {pr.author} · merged{' '} + {format(new Date(pr.merged_at), 'MMM d, yyyy')} + + + ))} + + + ); +}; diff --git a/web-server/src/content/DoraMetrics/DoraCards/__tests__/TicketCycleTimeCard.test.tsx b/web-server/src/content/DoraMetrics/DoraCards/__tests__/TicketCycleTimeCard.test.tsx index 7a771c6e..c258f6d0 100644 --- a/web-server/src/content/DoraMetrics/DoraCards/__tests__/TicketCycleTimeCard.test.tsx +++ b/web-server/src/content/DoraMetrics/DoraCards/__tests__/TicketCycleTimeCard.test.tsx @@ -2,16 +2,23 @@ jest.mock('@/hooks/useAuth', () => ({ useAuth: jest.fn() })); jest.mock('@/hooks/useStateTeamConfig', () => ({ useSingleTeamConfig: jest.fn() })); +jest.mock('@/contexts/ModalContext', () => ({ useModal: jest.fn() })); jest.mock('axios'); import { screen, waitFor } from '@testing-library/react'; +import userEvent from '@testing-library/user-event'; import axios from 'axios'; +import { useModal } from '@/contexts/ModalContext'; import { useAuth } from '@/hooks/useAuth'; import { useSingleTeamConfig } from '@/hooks/useStateTeamConfig'; import { renderWithTheme as render } from '@/utils/testUtils'; -import { segmentWidths, TicketCycleTimeCard } from '../TicketCycleTimeCard'; +import { + segmentWidths, + TicketCycleTimeCard, + UnlinkedPrsModalBody +} from '../TicketCycleTimeCard'; const TEAM_ID = 'team-1'; @@ -40,6 +47,7 @@ describe('TicketCycleTimeCard', () => { singleTeamId: TEAM_ID, dates: { start: new Date('2026-05-01'), end: new Date('2026-08-01') } }); + (useModal as jest.Mock).mockReturnValue({ addModal: jest.fn() }); }); it('renders nothing when Jira is not linked for this org', () => { @@ -133,6 +141,28 @@ describe('TicketCycleTimeCard', () => { ).toBeInTheDocument(); }); + it('opens the unlinked-PRs drill-down modal, scoped to the current team and date range, on "View PRs"', async () => { + (useAuth as jest.Mock).mockReturnValue({ + integrations: { jira: { integrated: true } } + }); + (axios as unknown as jest.Mock).mockResolvedValue({ + data: { cycle_time_by_project: [project()], prs_without_ticket_count: 4 } + }); + const addModal = jest.fn(); + (useModal as jest.Mock).mockReturnValue({ addModal }); + render(); + + await userEvent.click(await screen.findByRole('button', { name: /view prs/i })); + + expect(addModal).toHaveBeenCalledTimes(1); + const [modalArgs] = addModal.mock.calls[0]; + expect(modalArgs.title).toBe('PRs with no linked Jira ticket'); + expect(modalArgs.body.props).toMatchObject({ + teamId: TEAM_ID, + dates: { start: new Date('2026-05-01'), end: new Date('2026-08-01') } + }); + }); + it('omits the Data Hygiene card entirely when every merged PR is linked', async () => { (useAuth as jest.Mock).mockReturnValue({ integrations: { jira: { integrated: true } } @@ -232,3 +262,73 @@ describe('segmentWidths', () => { expect(widths.find((w) => w.category === 'Done').seconds).toBe(2227); }); }); + +const unlinkedPr = (overrides = {}) => ({ + id: 'pr-1', + title: 'feat(payments): add refund flow', + url: 'https://github.com/org/repo/pull/42', + head_branch: 'feat/refund-flow', + author: 'jordan', + merged_at: '2026-07-15T10:00:00+00:00', + ...overrides +}); + +// CLUSTOX: Jira integration, Phase 4 (§6E) -- the Data Hygiene +// drill-down's own lazy fetch. See docs/JIRA_INTEGRATION_PROPOSAL.md. +describe('UnlinkedPrsModalBody', () => { + const DATES = { start: new Date('2026-05-01'), end: new Date('2026-08-01') }; + + beforeEach(() => jest.clearAllMocks()); + + it('fetches this team\'s unlinked PRs for the given date range', async () => { + (axios as unknown as jest.Mock).mockResolvedValue({ data: [unlinkedPr()] }); + + render(); + + await screen.findByText(/feat\(payments\): add refund flow/); + expect(axios).toHaveBeenCalledWith( + `/api/internal/team/${TEAM_ID}/unlinked_prs`, + { + params: { + from_date: DATES.start.toISOString(), + to_date: DATES.end.toISOString() + } + } + ); + }); + + it('renders branch, author, and merge date for each PR', async () => { + (axios as unknown as jest.Mock).mockResolvedValue({ data: [unlinkedPr()] }); + + render(); + + expect( + await screen.findByText(/feat\/refund-flow/) + ).toBeInTheDocument(); + expect(screen.getByText(/jordan/)).toBeInTheDocument(); + expect(screen.getByText(/Jul 15, 2026/)).toBeInTheDocument(); + }); + + it('renders one entry per PR, in the order the backend returned them', async () => { + (axios as unknown as jest.Mock).mockResolvedValue({ + data: [ + unlinkedPr({ id: 'pr-new', title: 'newer PR' }), + unlinkedPr({ id: 'pr-old', title: 'older PR' }) + ] + }); + + render(); + + expect(await screen.findByText('newer PR')).toBeInTheDocument(); + expect(screen.getByText('older PR')).toBeInTheDocument(); + }); + + it('shows an error message instead of crashing when the fetch fails', async () => { + (axios as unknown as jest.Mock).mockRejectedValue(new Error('network error')); + jest.spyOn(console, 'error').mockImplementation(() => {}); + + render(); + + expect(await screen.findByText(/Could not load PRs/)).toBeInTheDocument(); + }); +}); From 6b4590676ecec11884516d36791b16e1ffc98c4f Mon Sep 17 00:00:00 2001 From: Batool Fatima Date: Tue, 11 Aug 2026 23:11:32 +0500 Subject: [PATCH 128/194] fix(dora): match Data Hygiene's layout and copy to the design reference User feedback: pulled the exact "Metrics" tab markup from the design mockup artifact and compared it against the live page. Two real mismatches, not just a vague "doesn't look right": 1. Layout -- the mockup has Data Hygiene as its own full-width card below Ticket Cycle Time, with a description line ("Surfaces process gaps rather than silently excluding unmatched PRs from ticket-aware metrics.") above the tile. Ours had it squeezed side-by-side with Ticket Cycle Time, filling the slot left empty by the deferred Sprint rollup chart. Changed TicketCycleTimeCard's layout from a row (flexWrap) to a column, so Data Hygiene now stacks full-width below, matching the reference. 2. Copy -- the mockup's tile reads "...no linked ticket -- check branch naming against the PAY-123 convention." Ours only said "no linked Jira ticket", with no hint about what to actually check. Added the same convention-check line, but with the team's real tracked project key (e.g. "PZDA-123"), not a hardcoded mockup placeholder -- a hint is only useful if it names the actual convention. Also clarified, separately from the UI: the ~70 unmatched PRs the Data Hygiene count reflects are not a matching bug to fix in code -- spot- checked several real ones (e.g. "chore: salvage-orphaned-worktrees") and none reference a ticket anywhere in their title or branch. That's a real process gap the drill-down (View PRs, added in c7ed236) exists to surface, not something a better regex could close. Updated the one test whose assertion depended on the old wording; added assertions for the new convention-check line using the real project key. tsc clean; jest 250 passed (same 2 pre-existing, unrelated baseline failures -- array.test.ts, git_org_repos.test.ts). See docs/JIRA_INTEGRATION_PROPOSAL.md. Co-Authored-By: Claude Sonnet 5 (cherry picked from commit 6acaa25a2564a8d282336eb1f091a3befd4ca824) --- .../DoraCards/TicketCycleTimeCard.tsx | 42 ++++++++++++------- .../__tests__/TicketCycleTimeCard.test.tsx | 7 +++- 2 files changed, 33 insertions(+), 16 deletions(-) diff --git a/web-server/src/content/DoraMetrics/DoraCards/TicketCycleTimeCard.tsx b/web-server/src/content/DoraMetrics/DoraCards/TicketCycleTimeCard.tsx index a918a539..b214cbcf 100644 --- a/web-server/src/content/DoraMetrics/DoraCards/TicketCycleTimeCard.tsx +++ b/web-server/src/content/DoraMetrics/DoraCards/TicketCycleTimeCard.tsx @@ -140,12 +140,13 @@ export const TicketCycleTimeCard: FC = () => { if (!insights?.cycle_time_by_project.length) return null; return ( - + p.project_key)} /> ); @@ -154,7 +155,7 @@ export const TicketCycleTimeCard: FC = () => { const ProjectCycleTimeCard: FC<{ projects: ProjectCycleTime[] }> = ({ projects }) => ( - + Ticket Cycle Time @@ -220,7 +221,8 @@ const DataHygieneCard: FC<{ count: number; teamId: ID; dates: { start: Date; end: Date }; -}> = ({ count, teamId, dates }) => { + projectKeys: string[]; +}> = ({ count, teamId, dates, projectKeys }) => { const { addModal } = useModal(); if (!count) return null; @@ -233,24 +235,34 @@ const DataHygieneCard: FC<{ }); }; + // Matches the design reference's own example ("...against the PAY-123 + // convention") but with this team's real project key, not a + // placeholder -- a hint is only useful if it's the actual convention + // to check against. + const exampleKey = `${projectKeys[0] ?? 'PROJ'}-123`; + return ( - - - Data Hygiene - - + + + + Data Hygiene + + + Surfaces process gaps rather than silently excluding unmatched PRs + from ticket-aware metrics. + + + {count} {' '} - PR{count === 1 ? '' : 's'} merged this period with no linked Jira - ticket + PR{count === 1 ? '' : 's'} merged this period with{' '} + + no linked ticket + {' '} + — check branch naming against the {exampleKey} convention.