Repository navigation
fix(jobs): expire overlap locks, and back off between retries - #173
Merged
Merged
Conversation
WithoutOverlapping already scopes its lock by the job's class (laravel-queue-overlap:<class>:<key>), and no job shares a key with another, so prefixes like "server.send-power-command#" repeated what the class name already says. Key each lock by the server, backup or ISO ID alone.
Locks: WithoutOverlapping's lock never expires by default and is only released when the job finishes, so a job the worker killed for running past its timeout left it held forever, and every later job of that class for that server failed at once. SendPowerCommandJob (15s) and UpdatePasswordJob (10s) could hit that with a single slow Proxmox call, which is itself allowed 15s. A new LocksPerSubject trait gives each lock an expiry of the job's timeout + 30s and a 5s release when held. Jobs without a timeout of their own get one, since the expiry is based on it: 300s for SyncBuildJob and PurgeBackupsJob, 60s for the rest. Retries: jobs whose Proxmox call throws were retried immediately until their tries or retryUntil ran out -- 7,388 attempts in two seconds for a MonitorStateJob on a missing VM. The polling jobs now back off 3s, 10s, then 30s; the three-try jobs 5s then 15s; the node sync jobs 10s then 30s. WaitUntilVmIsDeletedJob is left to the delete/disconnect branch, which already changes it. Tested: a unit test pins every lock's key, expiry and release; a power cycle (start, wait running, kill, wait stopped) through Horizon on a real Proxmox node, which left no locks behind.
This was referenced Oct 16, 2026
Contributor License AgreementAll identified human contributors have signed the current CLA. |
# Conflicts: # CHANGELOG.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
WithoutOverlappinglocks never expired. A job killed for running past its timeout left its lock held forever, and every later job of that class for that server then failed instantly. For example, a single slow force-stop broke reinstall and delete for that server until the lock was cleared from Redis by hand.LocksPerSubjecttrait expires each lock 30s after the job's timeout. A job that finds the lock held waits 5s and tries again.server.send-power-command#were redundant.Testing
Notes
WaitUntilVmIsDeletedJobisn't changed here; feat(servers): split Delete and Disconnect, and make Delete reliable #172 already changes it.vmExists()from feat(servers): split Delete and Disconnect, and make Delete reliable #172. It's a follow-up once that merges.