Repository navigation
Move heartbeats to ClickHouse - #1745
Draft
skyfallwastaken wants to merge 28 commits into
Draft
skyfallwastaken wants to merge 28 commits into
skyfallwastaken wants to merge 28 commits into
Conversation
Heartbeats move from Postgres to ClickHouse. Dashboard rollups become a per-user ClickHouse table rebuilt from Rails, the Postgres rollup table and cache jobs are removed, and Postgres heartbeats becomes read-only.
Adds an opt-in ClickHouse service to docker-compose, a ClickHouse container in CI, the production Coolify compose file and server config, and the nightly backup script.
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub.
|
Empty rollups report zero totals, the rollup activity graph stops at today, today's range includes the day's last second and the Sailors' Log leaderboard no longer embeds a Postgres subquery in a ClickHouse query. Also corrects the rounding description: halves round to even, as in Postgres.
bin/brakeman requires the latest release, so scan_ruby failed on 8.0.6 before analysing anything. 8.1.0 flags the ClickHouse SQL built from Integer() casts, quoted values, validated timezones and Active Record relations; each is reviewed and ignored with a note.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
There was a problem hiding this comment.
Brakeman found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.
Drops the opt-in compose profile: ClickHouse now always starts with the stack and web waits for it, as it does for Postgres.
created_at, updated_at and deleted_at drop from microseconds to whole seconds. On a full copy of production the two populated columns shrink from 1.0 GiB to 236 MiB each, taking the table from 7.0 GiB to 5.5 GiB. time keeps its fractional seconds.
Replaces the hand-rolled SQL loader with the adapter's multi-database support: migrations in db/clickhouse_migrate, a SQL structure dump in db/clickhouse_structure.sql and Rails' own db:create, db:prepare, db:schema:load and per-worker test databases.
On rollups built for every user from a full production copy, the project column shrinks from 28 MiB to 16 MiB and dashboard reads get about 1 ms faster. Rebuild time is unchanged.
A request that read HeartbeatRollupState just before a rebuild published queried a generation the rebuild had already deleted and rendered an empty dashboard. Keep the generation being replaced; account deletion still removes everything at once. Co-authored-by: Amp <amp@ampcode.com>
Site-wide reads filter on time alone, but ORDER BY starts with user_id, so they read the newest granules of every user. The index cuts rows read by 3 to 12 times for the currently hacking, footer and weekly summary queries. Co-authored-by: Amp <amp@ampcode.com>
ClickHouse costs a few milliseconds per query however small, so light users paid for every round trip. Live dashboards now take one aggregate query instead of nine and one filter-options query instead of five, last_7_days one instead of seven, and the stats API one instead of three.
For the heaviest users' all-time stats, one combined query needed over three times the memory of the separate ones and was slower.
Summing every user's heartbeats in one window query exceeds ClickHouse's 8 GiB per-query limit.
Admin lookups by IP address or machine and the repo sync job's created_at filter scanned all 400M rows. The indexes take under 6 MiB and cut these from 110-490 ms to 28-45 ms.
The rebuild that publishes a generation writes its dashboard snapshot to the cache, so the dashboard, profile and projects pages read it instead of querying ClickHouse. Whether a user has heartbeats also comes from the snapshot.
Total time, file count and the language, editor, OS, category, file and branch breakdowns came from eight queries.
Requests no longer pay 80-300 ms to recompute currently hacking, the footer counts and the homepage totals when their cache expires.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary of the problem
Heartbeats are the bulk of our Postgres: 755 GB for 426M rows, 592 GB of which is 32 indexes. Keeping that hot needs a lot of RAM, the dashboard rollups that make it fast take about 17 worker-hours a day to refresh and global analytics (admin scans, leaderboards, Sailors' Log) take seconds to minutes.
Describe your changes
Heartbeats now live in ClickHouse. Everything else stays in Postgres.
MergeTreeordered by(user_id, time, id). All of production fits in 5.5 GiB.timekeeps fractional seconds;created_at,updated_atanddeleted_atare stored to the second.fields_hashis gone: exact duplicates are rejected at ingest (an identity check under a per-user advisory lock) and the 23.4M existing duplicates are removed during the copy. Ids still come from Postgres'sheartbeats_id_seq, so they stay unique and below 2^53.Heartbeatableand the dashboard queries are rewritten for ClickHouse with the same gap rules as before. Halves round to even, as Postgres did.heartbeat_rollupstable in ClickHouse (one row per local hour and dimension), rebuilt byDashboardRollupRefreshJobwith the same debounce. Each rebuild writes a new generation and publishes it in Postgres (heartbeat_rollup_states), so readers never see a half-built rollup.dashboard_rollupstable, the rollup staleness logic and theCache::*jobs (now a 1 minuteRails.cacheon the owning models).heartbeats: becomes read-only via a trigger and keeps its data for verification until we drop it. Its foreign keys are dropped so deleting users or JA4 fingerprints still works.clickhouseservice in docker-compose alongside Postgres, a ClickHouse container in CI and the production Coolify compose file with nightly S3 backups. Brakeman is bumped to 8.1.0, whichscan_rubyneeded; its new warnings are on ClickHouse SQL built from casts, quoted values and validated timezones, and each is ignored with a note.script/clickhouse/cutover.shcopies everything ahead of time, pauses writes with the read-only trigger, copies what changed and verifies every row afterwards. Editor clients queue heartbeats rejected during the pause and resend them. It has been rehearsed end to end against a copy of production data.Screenshots / Media
No visual changes.