Skip to content

Ship a per-site MySQL workload dashboard - #307

Merged
colinmollenhour merged 5 commits into
mainfrom
t3/implement-issue-304
Nov 18, 2026
Merged

colinmollenhour merged 5 commits into
mainfrom
t3/implement-issue-304

Conversation

@colinmollenhour

@colinmollenhour colinmollenhour commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

AI Megamind · T3 Code with Codex GPT-6.2 Sol, Claude and Grok

Summary

Add a MySQL Workload dashboard to the Bloodraven Grafana distribution. Operators can compare connection utilization, InnoDB row lock waits, and client queries per second with replication lag, configured site role, and observed writable state. The dashboard links to the HA Overview and keeps its time range and filters.

Why

A healthy replication topology can coexist with lock contention or connection exhaustion. The workload dashboard gives on-call operators a per-site view of those symptoms during normal traffic and primary handoffs.

Closes #304.

Changes

  • Ship the opt-in exporter deployment and per-site Prometheus scrape examples. Reuse the monitor account through Kubernetes Secret references and collect global status and variables.
  • Add setup, verified TLS, network policy, and missing-series troubleshooting instructions. A measured zero appears as a number, and missing workload samples appear as gaps or No exporter data.
  • Plot the operator’s replication lag in seconds for read-only sites. Unknown lag and unreachable sites leave gaps. Lag remains available while the workload exporter is unavailable.
  • Exercise the example scrape config and dashboard queries in a Docker-backed CI job with the supported MySQL image.

Screenshots

Grafana with sample data. The lag series switches to iad when pdx becomes writable.

Workload dashboard with replication lag through a handoff

The exporter stops reporting workload for iad. Bloodraven still reports its replication lag.

Replication lag remains visible when iad workload metrics are missing

Test Plan

  • make generate && make manifests produced a clean tree. make vet, make lint, and make test passed. The first make test encountered a PITR property-test failure. The failing subtest passed three individual reruns and the full gate passed on retry.
  • make test-envtest passed with Kubernetes 1.37.0 and 1.31.0 assets. The CI path classifier passed all 69 cases.
  • BLOODRAVEN_MYSQL_NATIVE_TEST=1 go test ./internal/controller -run '^TestMySQLWorkloadNative$' -count=1 -v -timeout 7m passed with the monitor role, the example scrape configuration, real lock contention, and a MySQL handoff.
  • In a dedicated kind cluster with MySQL 9.7.4, Prometheus 3.10, and Grafana 12.3.6, a Bloodraven planned failover moved writes from iad to pdx. Both sites retained their scrape identities. Restarting iad and stopping the exporter exercised the two availability states and the distinction between missing and zero. Grafana links preserved time and filters.
  • A MySQL 9.7.4 server requiring secure transport accepted the documented CA and server name settings and rejected an incorrect name. The kind fixture did not exercise NetworkPolicy enforcement.
  • npm test, npm run build, and npm run verify:llms passed in site/.
  • Prometheus 3.10 evaluated the shipped lag query against nine sample-data cases covering a measured zero, unknown lag, unreachable and writable sites, handoff, exporter loss, and dashboard filters. Grafana 12.3.6 rendered both updated screenshots.

Risk and Rollout

The exporter is a separate Deployment. Groups that enable workload monitoring share one exporter Service across their sites. The guide covers spreading exporter replicas across failure domains. A full MySQL connection pool can refuse the monitor login, which appears as a gap and a corresponding connection error.

Educational Brief

Journey

The design review compared a standalone exporter, operator-managed per-site Deployments, and injection into the MySQL pods. The standalone configuration met the issue's scope with existing internal Services and the monitor account. Real Grafana rendering caught confusing timeline labels and a throughput stat that stayed blank during warmup. The handoff and missing-data screenshots exercise the revised panels.

Architecture

Prometheus sends one probe per internal site Service to the group exporter. The exporter reads global status and variables from MySQL. Prometheus also scrapes the Bloodraven operator, so Grafana can plot the configured role, writable state, and follower lag when the exporter is unavailable. Both series families use the same namespace, group, and site dimensions.

Lesson and Evidence

Scrape reachability, MySQL reachability, and collector success each identify a different failure. Verify all three before interpreting a zero-rate or zero-wait sample. The exporter, scrape job, and dashboard queries exercised by the native test are the files shipped to users.

Described at: 7799c31

@colinmollenhour

Copy link
Copy Markdown
Collaborator Author

AI Commit Note · Commit: 7799c31 · By: T3 Code with Codex GPT-6.2 Sol

Added replication lag in seconds for read-only sites. The panel uses the Bloodraven operator metric, so it remains available while workload measurements are missing. Validated zero, unknown lag, handoff, and exporter loss in Prometheus and refreshed the Grafana screenshots.

@colinmollenhour
colinmollenhour merged commit a6cb572 into main Nov 18, 2026
28 checks passed
@colinmollenhour
colinmollenhour deleted the t3/implement-issue-304 branch November 18, 2026 22:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Ship a per-site MySQL workload dashboard alongside Bloodraven HA monitoring

1 participant