Repository navigation
Ship a per-site MySQL workload dashboard - #307
Merged
Merged
Conversation
Collaborator
Author
Added replication lag in seconds for read-only sites. The panel uses the Bloodraven operator metric, so it remains available while workload measurements are missing. Validated zero, unknown lag, handoff, and exporter loss in Prometheus and refreshed the Grafana screenshots. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add a MySQL Workload dashboard to the Bloodraven Grafana distribution. Operators can compare connection utilization, InnoDB row lock waits, and client queries per second with replication lag, configured site role, and observed writable state. The dashboard links to the HA Overview and keeps its time range and filters.
Why
A healthy replication topology can coexist with lock contention or connection exhaustion. The workload dashboard gives on-call operators a per-site view of those symptoms during normal traffic and primary handoffs.
Closes #304.
Changes
Screenshots
Grafana with sample data. The lag series switches to iad when pdx becomes writable.
The exporter stops reporting workload for iad. Bloodraven still reports its replication lag.
Test Plan
make generate && make manifestsproduced a clean tree.make vet,make lint, andmake testpassed. The firstmake testencountered a PITR property-test failure. The failing subtest passed three individual reruns and the full gate passed on retry.make test-envtestpassed with Kubernetes 1.37.0 and 1.31.0 assets. The CI path classifier passed all 69 cases.BLOODRAVEN_MYSQL_NATIVE_TEST=1 go test ./internal/controller -run '^TestMySQLWorkloadNative$' -count=1 -v -timeout 7mpassed with the monitor role, the example scrape configuration, real lock contention, and a MySQL handoff.npm test,npm run build, andnpm run verify:llmspassed insite/.Risk and Rollout
The exporter is a separate Deployment. Groups that enable workload monitoring share one exporter Service across their sites. The guide covers spreading exporter replicas across failure domains. A full MySQL connection pool can refuse the monitor login, which appears as a gap and a corresponding connection error.
Educational Brief
Journey
The design review compared a standalone exporter, operator-managed per-site Deployments, and injection into the MySQL pods. The standalone configuration met the issue's scope with existing internal Services and the monitor account. Real Grafana rendering caught confusing timeline labels and a throughput stat that stayed blank during warmup. The handoff and missing-data screenshots exercise the revised panels.
Architecture
Prometheus sends one probe per internal site Service to the group exporter. The exporter reads global status and variables from MySQL. Prometheus also scrapes the Bloodraven operator, so Grafana can plot the configured role, writable state, and follower lag when the exporter is unavailable. Both series families use the same namespace, group, and site dimensions.
Lesson and Evidence
Scrape reachability, MySQL reachability, and collector success each identify a different failure. Verify all three before interpreting a zero-rate or zero-wait sample. The exporter, scrape job, and dashboard queries exercised by the native test are the files shipped to users.
Described at: 7799c31