Skip to content

Pack parallel catchup v2 workers 3 per node - #447

Merged
jayz22 merged 1 commit into
stellar:mainfrom
Jonathan-Eid:jonathan/catchup-maxskew-3
Oct 19, 2026
Merged

jayz22 merged 1 commit into
stellar:mainfrom
Jonathan-Eid:jonathan/catchup-maxskew-3

Conversation

@Jonathan-Eid

Copy link
Copy Markdown
Contributor

Bumps the worker spread maxSkew from 2 to 3 so Karpenter packs 3 workers per node instead of 2.

  • With a hard hostname spread, Karpenter plans each new node with maxSkew workers. At 2, 1024 workers need 512 nodes.
  • Prod only got 3 per node (384 nodes) because the catchup nodepool's 3072 cpu limit capped it. stellar/kube#5513 added fallback pools with their own limits, so build 1963 went to 512 nodes: $239/hr and $732 for catchup vs ~$635 on 384 nodes.
  • m8a.2xlarge fits 3 workers by memory (3 × 8Gi of ~28.7Gi allocatable), so 3 fills the node.

Testing (ssc-test, prod's catchup pools mirrored, 60 pods with the worker's requests and spread rule):

  • maxSkew: 2: 30 nodes, 2 per node
  • maxSkew: 3: 20 nodes, 3 per node, 84% memory requested
  • helm template renders maxSkew: 3 on every worker pod

🤖 Generated with Claude Code

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jayz22
jayz22 merged commit f7476ba into stellar:main Oct 19, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants