[rhit-mathewjl]

#711 and #712 were merged into main as d1ccafe and 644c797. This PR now targets main as the final PR in the stack.

Security hardening: isolate experiment runners and infrastructure credentials

Closes #709. Carries over the findings from #476.

User-submitted experiment code now runs behind a clear boundary. A runner can compute and report its own results, and nothing else. The Python coverage scripts now propagate pytest failures to CI.

Identities and permissions

• Backend, frontend and runner each get their own ServiceAccount (kubernetes_init/security/). Nothing is bound to default any more; the old job-creator ClusterRole is removed.
• Backend: create/delete Jobs in default, create TokenReviews. Frontend: list Pods, read Pod logs, patch the two Deployments and the worker node (admin redeploy). Runner: no permissions, no API token mounted.
• Both Tilt and kubernetes_init/init.py apply the manifests. init.py --update migrates an existing cluster.

Runner containment

• Every runner Job is built from apps/backend/job-runner.yaml by modules/runner_job.py: non-root user 10001, all capabilities dropped, no privilege escalation, read-only root filesystem with four scratch volumes, default seccomp profile.
• CPU, memory, ephemeral storage and Job lifetime limits, configurable through backend environment variables.
• Admin/privileged users are resolved from the current database record when dispatching. Their package-installation profile runs as root with a writable filesystem and retains default container capabilities.
• The runner receives no database login and no mail credentials. The backend sends the completion email itself.
• Uploaded zips are rejected on path traversal or excessive size (apps/runner/modules/archive.py).

Backend authentication

• Every endpoint except the health checks requires a token.
• The frontend authenticates with a projected ServiceAccount token (audience glados-backend) that the backend checks with the TokenReview API.
• Each runner gets a random per-Job token, stored only as a hash, that works for its own experiment only, can change only the progress fields, and expires when the runner finishes, the experiment is cancelled, or the Job deadline passes.
• The frontend preserves Enforce server-side ownership and session authorization (#708) #711's Web and CLI owner checks. Start and shared cancellation calls carry the frontend service credential. The backend resolves the creator's current database role, supports legacy BSON identities, and rejects ambiguous identities.

Network and MongoDB

• kubernetes_init/security/network-policies.yaml: runners accept nothing inbound and may reach only DNS, the backend and public addresses; the backend accepts only frontend and runner Pods. Flannel does not enforce NetworkPolicy, so until the cluster has an enforcing plugin the token checks are the boundary that holds. Documented as a known gap.
• The MongoDB password is removed from the production chart values. deploy.sh creates the glados-mongodb-auth Secret instead. The previously published password should be rotated (steps in the docs).

Runtime versions

• The backend image uses Python 3.8 and the runner image uses Python 3.9. Runtime and dependency upgrades require separate work.

Tests and checks

• coverage.sh in both apps now returns pytest's exit status. It previously ended with an echo, so CI could never fail; six stale runner fixture failures are now resolved by adding the current required fields and updating the expected defaults in the tests.
• New unit tests: manifests (permissions per identity, nothing bound to default, secrets per Deployment, non-root runner), Job construction, endpoint authentication, mail, archive extraction.
• development_scripts/security/verify_isolation.sh checks a development cluster: what each identity may do, and whether a runner-shaped Pod can reach any backend endpoint without credentials, MongoDB, or the API server.
• Docs: new docs/docs/infrastructure/security.md describing the boundary, limits, rollout and known gaps.

Rollout

Allow existing runner Jobs to finish before rolling out the authenticated backend. Existing Jobs have no runner token and cannot report to the new endpoints.

Restarting the Deployments is not enough the first time. From kubernetes_init/ with cluster access, run python3 init.py --update, then rotate the MongoDB password and create glados-mongodb-auth before helm upgrade.

Known gaps left open

• NetworkPolicy is not enforced under Flannel.
• User code and the runner share a container, so user code can tamper with its own experiment's results.
• Frontend and backend still use the MongoDB root user; MongoDB is still exposed on node ports 30000 and 30001.
• The backend container runs as root; the MongoDB chart uses an archived image.

Cancellation changes here connect #712's authorized cancellation to the runner-token lifecycle. Tokens are revoked after accepted deletion, failed deletion preserves runner access, and missing-Job retries remain supported.

Follow-up validation

• Local Node 20 full suite, 647 passed, including the inherited Enforce server-side ownership and session authorization (#708) #711/Add authenticated cancel experiment API for the CLI #712 suites and service-credential integration regressions.
• Type check passed. ESLint passed with 12 existing warnings. Backend pylint passed at 9.63/10.
• Backend suite, 91 passed on Python 3.10.
• Fresh independent PR and integration reviews completed. The archive/runner race identified during review was fixed in Add authenticated cancel experiment API for the CLI #712 and included here; independent re-review found no remaining actionable issues.
• Both coverage scripts preserve exit codes 0 and 7 in controlled probes.
• Hosted Linux runner verification on ebc13ba passed all 43 tests. Local Windows execution passed 40 tests with three previously documented CSV newline failures.
• Live Kubernetes deletion, network enforcement, and rollout verification remain pending.

Hosted CI on the latest commit

All 9 applicable checks passed on 249e7be, including frontend tests/type checks/lint (647 tests), backend tests (91), Google authentication checks, all container builds, documentation build, and runner lint/tests (43 tests; pylint 9.53/10). Frontend CI, backend CI, runner CI.

Collaborator review integration

Includes #711 at d1faa36 and #712 at 99dc835. The collaborator's 54d2183 cancellation UI and structured refusal behavior are preserved. Late runner writes cannot overwrite archived terminal experiments, including archived cancellations. Runner service credentials and token revocation remain integrated with the common cancellation service.

Runner fixture follow-up

Only apps/runner/tests/test_experiment.py and apps/runner/tests/test_parameters.py changed in ebc13ba. Experiment fixtures now include creatorRole, creatorEmail, trialResultLineNumber, and sendEmail. Expected defaults include experimentExecutable and status, and Boolean/String fixtures include useDefault. Existing assertions continue to cover valid inputs, missing defaults, and invalid inputs.

An independent reviewer found no actionable issues and independently reproduced 25 passing targeted tests on Python 3.9. The full Linux runner CI passed all 43 tests. CI log https://github.com/AutomatingSciencePipeline/Monorepo/actions/runs/43299522155/job/131466624451 .