backup: keep the scheduler running when a schedule's async job is gone

[calvix]

Description

Every backup poll first checks the schedules that carry an async job id, then submits the scheduled
backups that are due:

poll()
  checkStatusOfCurrentlyExecutingBackups()   // every schedule with asyncJobId != null
  scheduleBackups()                          // submits the due ones

checkStatusOfCurrentlyExecutingBackups() dereferenced asyncJobManager.getAsyncJob(id).getStatus()
without handling a missing job. A job can go missing while a schedule still points at it: the async
job GC (AsyncJobManagerImpl.getGCTask()) expunges expired jobs, unfinished ones included, and does
not clear the backup_schedule.async_job_id values that reference them. Once that happens:

• the NullPointerException leaves poll() before scheduleBackups() runs;
• the timer task catches and logs it, and the next tick hits the same record again;
• so every schedule handled by that poll stops firing - not just the one whose job is gone - and
  nothing repairs it while the management server runs. Only a restart clears it, because start()
  resets every schedule's job id through scheduleNextBackupJob().

Nothing outside the log says anything is wrong: scheduled backups simply stop being taken.

This change:

• treats a schedule whose job no longer exists as finished and moves it on to its next run, with a
  WARN naming the schedule, the Instance and the missing job id;
• checks each schedule inside its own try/catch, so one record that fails for any reason can no
  longer stop the others from being checked or from being scheduled;
• logs the failure to clean up an Instance after a failed create-Instance-from-backup at WARN instead
  of DEBUG. That Instance remains, and its volumes may hold none of the backup's data, so it should
  not be visible only at debug level.

checkStatusOfCurrentlyExecutingBackups() and scheduleNextBackupJob() become protected so the test
can exercise the loop without a database. Their behaviour is otherwise unchanged.

Types of changes

• Breaking change (fix or feature that would cause existing functionality to change)
• New feature (non-breaking change which adds functionality)
• Bug fix (non-breaking change which fixes an issue)
• Enhancement (improves an existing feature and functionality)
• Cleanup (Code refactoring and cleanup, that may add test cases)
• Build/CI
• Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

• Major
• Minor

Bug Severity

• BLOCKER
• Critical
• Major
• Minor
• Trivial

Screenshots (if appropriate):

n/a

How Has This Been Tested?

Unit test BackupManagerTest.checkStatusOfCurrentlyExecutingBackupsRepairsAScheduleWhoseJobIsGone,
with three schedules in one poll:

schedule   async job                         expected
---------  --------------------------------  ----------------------------------
1          expunged (getAsyncJob -> null)    rescheduled (job id cleared)
2          lookup throws                     skipped, loop continues
3          SUCCEEDED                         rescheduled

Before the change the test fails with exactly the production error:

java.lang.NullPointerException: Cannot invoke "AsyncJobVO.getStatus()" because "asyncJob" is null

After it, BackupManagerTest runs 87 tests, 0 failures, with checkstyle enabled
(mvn -pl server -Dtest=BackupManagerTest test, JDK 17). The server module compiles with the
UserVmManagerImpl change.

The failure itself was found by reading the source, not by observing it on a cluster: it needs a
schedule's job to outlive job.expire.minutes without the schedule being reconciled, which a long
backup job or a poll that cannot complete for that long produces.

How did you try to break this feature and the system with this change?

• A lookup that throws for one schedule: the remaining schedules are still checked and rescheduled
  (covered by the test).
• A missing job is not treated as "still running": that would keep async_job_id set for ever, and
  getSchedulesToExecute() only selects schedules whose async_job_id is null, so that schedule
  would never fire again.
• scheduleNextBackupJob() computes the next run from the schedule itself and is already what
  start() does for every schedule, so the repair takes the same path a restart does.