You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Add scale down to HistoryPubnetParallelCatchupV2 - #433
Currently, the CatchupV2 mission does not scale down workers when the amount of remaining/in-progress jobs becomes less than the amount of parallel workers.
Implementation
Before, each parallel worker was a replica of the same statefulset, which prevented picking out an idle worker to scale down.
Workers are now bare Pods (hostname/subdomain for headless-service DNS) instead of statefulset replicas: one pod per worker, no controller.
The marking decision lives in job_monitor.py (phase 2b). It marks surplus idle workers into the Redis set retiring, where surplus is measured against outstanding = queued + in_progress, keeping a reserve of MIN_UNMARKED_WORKERS (default 8) counted against all unmarked workers.
/status publishes retirable: workers marked on an earlier pass and still idle.
The F# driver only reads retirable, collects logs, deletes those pods, and tracks livePods. No Redis access from the driver; readyPods and the redis-cli exec are gone.
worker.sh checks SISMEMBER retiring $POD_NAME before claiming a job.
The orphan sweep discovers pc-v2 releases via worker pods and reaps ownerless pods.
Up to 64 workers can have their logs collected and scaled down in one pass; keeping the log collection synchronous and blocking the loop. This keeps the code cleaner for not much affect on the loop time and avoids managing threads. A log collection failure doesn't blocks retiring the batch -- any error skips that pod's logs.
Worker pods are not recreated if lost (accepted).
retiring is never pruned, so names stay in retirable after deletion; the driver filters them via livePods.
A dead pod that still owns a job occupies a reserve slot until the end-of-run requeue, hence the floor of 8.
The job monitor image must be rebuilt and pushed before merge, since the driver reads retirable from /status.
A run holds its full fleet for its whole duration, including the tail
where almost nothing is left: 37.7% of prod worker-hours are idle, and at
the extreme 1020 workers wait 4.5 hours on 4 remaining jobs.
Each worker becomes its own single-replica StatefulSet, so any idle
worker can be removed without regard to position; scaling one shared
StatefulSet only ever deletes a suffix of its ordinals, which a single
long range on a high ordinal can pin for hours.
Each poll marks up to `ready - outstanding` idle workers into a Redis set
that worker.sh checks before claiming, and deletes the ones marked on an
earlier pass that are still idle. The one-pass gap is the safety
property: a marked worker stops claiming within its 10s loop, so by the
next poll it provably takes no new work. Busy workers come from reading
job_owners directly rather than from the status snapshot, which the
monitor only republishes after serially pinging every owner and so can be
older than the mark.
Capacity is Running pods, not the replica count: a Pending pod owns no
job and so reads as idle capacity it cannot supply.
Logs are collected before each deletion, since /data is emptyDir, and a
failed collection skips the removal rather than losing them.
A pod being deleted still reports phase Running, so readyPods counted
pods on their way out. That inflated capacity, marked more workers than
the queue could spare, and re-selected the pods just deleted -- whose
StatefulSets were already gone, so the delete threw NotFound and aborted
the whole pass. A run lost a pass after every successful wave.
The ready set also has to drop the workers removed earlier in the same
pass, or the marking step reads a fleet size that is one wave stale and
re-marks what it just deleted.
Both reads ran on every poll of a multi-hour run even while the queue
outran the fleet and nothing was marked, and the pod list at 1024 workers
is not cheap. They are now skipped unless something is marked or the
queue has dropped below the fleet, which also stops the ramp logging a
skipped pass every poll: with no pod Running yet, taking the head of an
empty ready set threw and the handler reported it as a real failure.
The guard reads livePods rather than the ready set, since consulting the
ready set is the cost being avoided. livePods is always the larger of the
two, so the guard errs toward doing the work rather than skipping it.
redisIn now takes the ready set and picks a host itself, returning
nothing when there is none, so callers stop reaching for Seq.head.
Log collection is serial and runs inside the poll loop, so an unbounded
batch blocks it for as long as the batch takes. A 400-worker run retired
213 in one pass and spent 103 seconds collecting, and that was with tiny
archives: each worker had run only 2 ranges. A prod worker churns dozens
over hours, so per-pod cost is seconds rather than half a second, and an
uncapped pass late in a 1024-worker run would block for tens of minutes
with no timeout on any single exec.
Capped at 32 per pass, so the worst case is bounded by the cap rather
than by the fleet size. Later passes drain the rest.
helm was left to use the kubeconfig's current namespace while every other
call in the mission honours context.namespaceProperty. A run launched
with --namespace ssc-config-test therefore installed its chart into
stellar-supercluster: 1024 workers and a monitor landed in the
production namespace, while the driver polled and scaled an empty one,
so scale-down silently did nothing for the whole run.
Both uninstall paths get it too. They were previously consistent with
the install only by accident -- both defaulted to the same current
namespace -- which would have broken the moment the install was fixed
alone.
Once every worker is marked retiring, a job the monitor puts back on the
queue can never be claimed: worker.sh refuses to claim while marked, and
there is no way to unmark. A single recoverable orphan then hangs the run.
Observed on a 1024-worker run: all 4009 ranges were accounted for, one job
lost its owner upstream, the monitor correctly requeued it, and nothing was
left willing to take it. The run had to be aborted.
Two changes, both in the mark step. The reserve is now counted against
unmarked workers rather than `ready`, which includes marked pods that have
not been deleted yet and so let the reserve erode away over successive
passes. And the reserve never drops below minUnmarkedWorkers, so a run whose
queue empties still has somewhere to put returned work.
Three rather than one so the reserve survives a worker restarting or its
node being disrupted, and so several returned jobs drain in parallel. The
cost is three idle workers, about two nodes, against a tail that otherwise
idles the whole fleet.
RunRemoteCommandAndCaptureOutput returned void and wrote the k8s status
channel to the console, so neither caller could tell a failed exec from a
command that produced no output. Two fail-open paths followed.
`redisIn` reads job_owners to decide which workers are idle. A failed exec
left it with zero lines, which reads as "no worker is busy" and makes every
worker look retirable -- the exact check that is supposed to stop a busy
worker being retired. Two such passes in a row would mark a worker and then
delete it mid-range.
`collectLogsFromPods` treats an exception as the only failure, but tar
exiting non-zero raises nothing, so a failed archive counted as collected
and the worker was deleted with its logs.
The exit code was already available: channel 3 carries a V1Status whose
causes hold it, and Kubernetes.GetExitCodeOrThrow already parses exactly
that for RunRemoteCommand in the same file. Renamed the locals and fixed
the comments too -- a variable called `stderr` holding the status channel
is what made this invisible.
Both callers now fail closed: redisIn raises into the poll loop's handler,
which skips the pass with no state mutated, and a failed tar puts the pod
on the failed list so retirement is skipped for it.
The reserve was subtracted from the idle pool alone, but `outstanding`
counts in-progress jobs too, and those already have a worker each. With
1000 workers, 700 busy and 700 jobs outstanding, the surplus came out as
300 - 700 and nothing was marked at all: 300 idle workers were held to
cover work that was already being served.
Counting the surplus against every unmarked worker gives 1000 - 700 = 300,
and those 300 idle workers are marked. Only idle workers are ever marked,
and the reserve still keeps `max outstanding minUnmarkedWorkers` workers
unmarked, so a requeued job still has somewhere to go -- an unmarked busy
worker becomes claimable as soon as it finishes.
Run stellar#192 never sat in that state: in-progress fell from 1022 to 80 in three
minutes, so the largest instantaneous gap was about 80 workers and
retirement was capped at 32 a pass throughout. A run whose ranges vary
enough for in-progress to decay gradually would sit there and mark nothing.
This updated signature takes explicit pod names, but the function documentation still says it automatically derives them from pubnetParallelCatchupNumWorkers. Update the comment so callers are not given an obsolete contract.
// Returns the pods whose collection raised; an empty archive is success.
The new StatefulSet name makes every pod end in -0 (...-stellar-core-<worker-index>-0), but worker.sh derives core_id from the final hyphen-separated segment and stores it in each Redis metric. Consequently, every worker is now reported as core 0. Pass the worker index to the container or update the script to extract the penultimate segment.
A worker that never claimed a job has an empty /data, so the tar glob
matches nothing and tar exits 2. Checking that exit code put those workers
on the failed list, which blocks retirement for the whole batch and never
clears, because the batch is rebuilt the same way every pass.
It lands on precisely the workers scale-down exists to remove: a surplus
worker that never got work. Seen on ssc-test with 12 workers and 8 ranges,
where the 4 that never claimed blocked retirement indefinitely.
redisIn keeps its exit-code check. There a failed exec yields an empty busy
set, which reads as "no worker is busy" and makes every worker look
retirable, so failing closed there is worth having.
Running is not equivalent to a ready pod: for example, a pod in CrashLoopBackOff normally retains phase Running. Because redisIn always selects the first member of this set, one such lower-sorted pod can make every Redis exec fail and disable scale-down indefinitely even when other workers are healthy. Filter on the pod's Ready=True condition as well.
This new StatefulSet name makes each pod end in -<worker-index>-0. The worker still derives core_id from the final segment (worker.sh:88), so every worker now reports ID 0 in the raw catchup metrics instead of its actual worker index. Update that extraction (or pass the worker-index label through the downward API) alongside this naming change.
Per-worker StatefulSets renamed pods to <release>-stellar-core-<i>-0, so the
last hyphen-separated segment is the StatefulSet's own -0 suffix rather than
the worker index. Every metrics record in run stellar#192 reported core_id 0.
Nothing consumes the field -- the monitor discards it and keeps only the two
durations -- but a value that is always 0 reads as real data, so pass the
index the chart already has.
If a later chunk fails, earlier SADD calls have already committed in Redis, but marked is updated only after every chunk succeeds. Those workers stop claiming while the driver still treats them as unmarked; if the outstanding count then rises, they may never be selected again for tracking/removal. Record each chunk in marked immediately after its successful SADD.
for chunk in List.chunkBySize 30 toMark do
let names = chunk |> List.map (sprintf "'%s'") |> String.concat " "
redisIn context ready (sprintf "SADD \"%s-retiring\" %s" helmReleaseName names)
The function no longer derives pod names from pubnetParallelCatchupNumWorkers, so the preceding numbered contract is stale and can mislead callers. Document that the caller supplies the pod list.
// Returns the pods whose collection raised; an empty archive is success.
let collectLogsFromPods (context: MissionContext) (podNames: string list) : string list =
The sweep helm-uninstalls a PCv2 release before deleting resources
individually, so the release secret does not end up dangling at deleted
workloads. It found the release by matching a StatefulSet name ending in
-stellar-core, which no longer happens: per-worker StatefulSets end in
-stellar-core-<index>. Abandoned runs therefore had their workloads deleted
one by one with the release left behind.
Matching on the -stellar-core segment instead, and collapsing the names to a
set, so a 1024-worker release is uninstalled once rather than 1024 times.
Running does not imply that the stellar-core container is Ready; CrashLoopBackOff pods commonly retain phase Running. Because redisIn always chooses Seq.tryHead, one low-index crash-looping pod can be selected every pass and prevent scale-down even while other workers are healthy. Restrict this set to pods whose worker container reports Ready (and ideally retry another member if an exec races with termination).
The retirement check is separate from both LMOVE and the ownership HSET, leaving a deletion race: a worker can observe “not retiring,” be paused, then get marked; on the next poll the driver sees no owner and deletes it just as it claims a range. Make the membership check, claim, and ownership registration one atomic Redis operation (for example, a Lua script) so every claimed range is visible before a marked worker can be considered removable.
# Stop claiming once the driver marks us, so it can remove us without interrupting a range.
if [ "$(redis-cli -h "$REDIS_HOST" -p "$REDIS_PORT" SISMEMBER "$RELEASE_NAME-retiring" "$POD_NAME")" = "1" ]; then
echo "$(date) $POD_NAME is retiring; not claiming."
sleep $SLEEP_INTERVAL
continue
fi
This loop can issue up to 64 pod deletions back-to-back, bypassing the mission's configured Kubernetes API limiter. Other Kubernetes operations consistently call ApiRateLimit.sleepUntilNextRateLimitedApiCallTime before each request (for example, StellarNamespaceContent.fs:29-39 and StellarOrphanSweep.fs:42-52); without it, throttling can abort the remaining retirement batch, and DELETE requests are intentionally not retried by ThrottleRetryHandler. Throttle each deletion with context.apiRateLimit.
A non-404 failure deleting one pod exits the whole batch through the outer catch before livePods is updated. The next pass retries already-deleted pods and stops at the same failing pod, so later retired workers can be starved indefinitely. Handle failures per pod and remove successful/absent pods from livePods immediately.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
jayz22
left a comment
Purpose
Currently, the CatchupV2 mission does not scale down workers when the amount of remaining/in-progress jobs becomes less than the amount of parallel workers.
Implementation
Before, each parallel worker was a replica of the same statefulset, which prevented picking out an idle worker to scale down.
job_monitor.py(phase 2b). It marks surplus idle workers into the Redis setretiring, where surplus is measured againstoutstanding = queued + in_progress, keeping a reserve ofMIN_UNMARKED_WORKERS(default 8) counted against all unmarked workers./statuspublishesretirable: workers marked on an earlier pass and still idle.retirable, collects logs, deletes those pods, and trackslivePods. No Redis access from the driver;readyPodsand the redis-cli exec are gone.worker.shchecksSISMEMBER retiring $POD_NAMEbefore claiming a job.Testing
retirablecapped at 32 = 40 - 8), 48/48 ranges completed.Caveats
retiringis never pruned, so names stay inretirableafter deletion; the driver filters them vialivePods.retirablefrom/status.🤖 Generated with Claude Code