Skip to content

perf: index metricarchive.metricid used by the metrics archive job - #2918

Open
constantine2nd wants to merge 1 commit into
OpenBankProject:developfrom
constantine2nd:perf/metricarchive-metricid-index
Open

constantine2nd wants to merge 1 commit into
OpenBankProject:developfrom
constantine2nd:perf/metricarchive-metricid-index

Conversation

@constantine2nd

Copy link
Copy Markdown
Contributor

Problem

MetricsArchiveScheduler.conditionalDeleteMetricsRow moves old metric rows to metricarchive one row at a time, and for every row it looks the archive row up by metricId twice:

  1. MappedMetrics.saveMetricsArchive - MetricArchive.find(By(MetricArchive.metricId, primaryKey)) (dedup before insert)
  2. conditionalDeleteMetricsRow - MetricArchive.find(By(MetricArchive.metricId, i.getMetricId())) (verify before deleting the source row)

MetricArchive.dbIndexes has no index on metricId, so on PostgreSQL both lookups are sequential scans of the whole metricarchive table. The cost of archiving one row therefore grows linearly with the size of the archive.

Evidence

  • A local OBP database (138k archive rows) shows metricarchive with 1,542 sequential scans for a run that moved 766 rows: two full scans per archived row. That run took 19.5 s (about 25 ms per row).
  • Reproduced on a copy scaled to 15M archive rows (PostgreSQL 14), running the same per-row statements the scheduler issues (lookup by metricid, insert, lookup by metricid, delete):
per archived row without index with Index(metricId)
138k archive rows (one lookup measured) ~74 ms per lookup, seq scan index scan
15M archive rows ~1.6 s ~0.8 ms

With the default retain_metrics_move_limit of 10,000 rows per run, a 15M-row archive would need several hours per run instead of fitting the ~10 minute scheduler interval; the JobScheduler lock then skips the following runs and the live metric table keeps growing.

Change

Add Index(metricId) to MetricArchive.dbIndexes (one line, with a comment explaining why).

Deployment note

Schemifier creates missing indexes at boot, so existing deployments get metricarchive_metricid automatically on the next start. It is a plain CREATE INDEX, which blocks writes to metricarchive while it builds. Only the archive job writes to that table, and the build is fast (seconds per few million rows on commodity hardware), so no separate migration is added.

Testing

Not compiled locally (the build requires JDK 25, which is not installed on this machine); relying on CI. The change follows the existing dbIndexes pattern; MappedLong fields are already used in index declarations elsewhere (for example UniqueIndex(mCustomerId, ...) in MappedTaxResidence).

MetricsArchiveScheduler looks each archived row up by metricId twice:
once in saveMetricsArchive (dedup) and once before deleting the source
row (verify). MetricArchive declared no index on metricId, so both
lookups were full scans of metricarchive and the cost of archiving a
single row grew linearly with the size of the archive.

Adding Index(metricId) to MetricArchive.dbIndexes makes both lookups
index scans. Schemifier creates the missing index on existing
deployments at boot.
@sonarqubecloud

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant