Skip to content

Port performance optimizations onto rmt-server 3.2.0 (fiberless) - #1

Draft
Pihaar wants to merge 320 commits into
rmt_3from
performance-optimization
Draft

Pihaar wants to merge 320 commits into
rmt_3from
performance-optimization

Conversation

@Pihaar

@Pihaar Pihaar commented Oct 13, 2026

Copy link
Copy Markdown
Owner

Ports the configurable-concurrency / exponential-backoff / 429-rate-limit / deferred-retry performance features onto the fiberless rmt-server 3.2.0 downloader (upstream removed fibers in 3.2.0 via bsc#1276939). Draft to run CI.

jcomello and others added 30 commits June 1, 2026 08:51
…gurable

remove use of ActiveSupport::Configurable
Fix ci/README.md with correct container image
…y and functionality

- Updated phony targets in Makefile to include ansible-related tasks.
- Changed default repository name in Ansible role to rmt_3.
- Enhanced Ansible tasks for installing dependencies and managing services.
- Improved comments and organization in Ansible role files for better maintainability.
To allow Rancher Manager to register against an RMT, the architecture
needs to match what Rancher currently sends. At the time of writing this
is 'unknown'.
The SCC API returns arch: null for 'unknown' architecture. To allow the
registration to succeed, the arch for rancher products is now hard set
to 'unknown'.
…ity and testing

- Added ansible-core and python3-PyMySQL to Dockerfile dependencies
- Updated ansible-test target in Makefile to run tests in SLE 16.x container
- Modified ansible.cfg to clarify inventory usage and changed stdout callback
- Introduced new SSL and plain text Nginx configuration files
- Added upstream configuration for Nginx
- Updated Ansible role metadata to specify minimum Ansible version
- Changed task to copy upstream configuration instead of using a template
- Enhanced test playbook to clarify Ansible version requirements
- Updated rmt-server.spec to require ansible-core and python3-PyMySQL
Set architecture to unknown for rancher based products (RMT3)
when purging a large amount of systems, the command crashes,
these changes aim to remove the bottle-necks and to provide
more information on the process and a recovery path

add quiet option

show exception class in the error message

update manual
- Simplified example configuration in all.yml.example
- Removed deprecated repository settings from main.yml
- Added database setup tasks in db_setup.yml
- Introduced SSL setup tasks in ssl_setup.yml
- Updated main playbook to check for existing database and provision as needed
- Adjusted variable definitions in vars/main.yml for clarity
- Corrected package requirements in rmt-server.spec
jesusbv and others added 30 commits September 28, 2026 15:56
fix simplecov warnings and deprecations
…er); fix stale concurrency=/RETRY_DELAY_SECONDS tests
… Integer(exception:), redundant freeze, symbol proc
…ig + completion-logging coverage, robustify 429/compute_delay tests
…oring_type validation + cache-clear specs for coverage
…llback delay, budget-exhausted raise/ignore) for downloader coverage
The new RMT::Config accessors (download_concurrency, head_concurrency,
retry_count, retry_delay, exponential_backoff?) and helpers
(full_revalidation_day_today?, revalidation pinning) had no tests: config.rb
carries an upstream-wide simplecov:disable, so the 100% gate never flagged them.
Add table-driven tests covering defaults, valid values, below-min, above-max,
non-numeric input, and the warn-on-invalid path; day matching by name/integer/
numeric-string/array and the YAML Config::Options (respond_to?(:values)) branch;
and the pin/unpin/pinned midnight-consistency behaviour. retry_delay is unstubbed
locally via and_call_original since rails_helper stubs it globally to 0.
handle_failure and handle_rate_limit both branched on @failed_downloads.nil?
to either abort+raise (ignore_errors==false) or record+enqueue_next. Extract the
shared terminal path into give_up_on_download, keeping the per-file warn as an
optional block that runs only on the record path, so no behaviour changes: the
'x failed' log is still suppressed before an abort/raise.
- download_concurrency: drop 1.5 from invalid inputs; Integer(1.5) truncates
  to 1 (valid), it does not raise, so no fallback/warn occurs.
- exponential_backoff?: 'maybe' coerces to true (ActiveModel Boolean treats any
  non-FALSE_VALUE as true); only an empty string casts to nil and triggers the
  warn/default-false path. Split into two correct contexts.
- downloader.rb: remove non-ASCII multiplication sign from the give_up_on_download
  comment (Style/AsciiComments); the x in the warn string literal is unaffected.
…ectations)

config_spec.rb is not excluded from RSpec/MultipleExpectations, so each example
must assert exactly once. Separate the return-value check from the log-assertion
(stubbing the logger with allow on the value-path examples), and split the
multi-step pinning sequences into one-expectation examples. Also drop explanatory
comments per project convention.
A per-iteration before block closes over the loop variable and all share one
scope, so every example in the valid-values tables saw the last input (got 8 /
got 20). Moving the Settings assignment into each example binds the value
correctly. Only the valid-values tables (distinct expected per input) were
affected; same-result loops were masking the bug.
On hosts where an earlier perf build already created repositories.mirroring_type,
the plain add_column aborted the whole db:migrate with 'Duplicate column name'.
Guard with column_exists? and use explicit up/down so it is a no-op where the
column is present, runs normally on fresh DBs, and stays cleanly reversible.
Keeps the mirroring_type caching feature fully intact.
…ervers

A server under load (e.g. public.dhe.ibm.com during a full sync) throttles via
connection-close / silent drop -> operation_timedout rather than HTTP 429.
Hammering it every retry_delay seconds does not help. compute_delay now uses the
flat retry_delay for the first FAST_RETRIES(2) attempts (transient blips) and only
then switches to exponential backoff with jitter, giving a throttling host room to
recover. Backoff-disabled behaviour (flat throughout) is unchanged.
repomd.xml / signature downloads from an occasionally unresponsive upstream
(e.g. public.dhe.ibm.com) exhausted the default 4 retries and aborted the whole
mirror run with operation_timedout -- sporadically (3 of 4 runs were fine).
download_multi now takes a retries: override; download_cached! and check_signature
pass METADATA_RETRIES(8). compute_delay keys its attempt counter off the active run
budget (@run_retries) instead of @max_retries so the staged fast->backoff curve
stays correct for both package (4) and metadata (8) budgets. Worst-case wait for a
dead metadata host is ~8min (one repo), acceptable for a nightly mirror.
download_cached! and check_signature now pass retries: METADATA_RETRIES; the
repomd mirror-flow stubs matched download_multi without it and failed. SumaProductTree
and debian_spec are unaffected (direct download_multi / mirror-level stubs).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

9 participants