🐞 Harvest: voter import stays Pending when its task can't be sent to RabbitMQ (release/10.1)

[Findeton]

Parent issue: https://github.com/sequentech/meta/issues/14271

A voter import stayed Pending after RabbitMQ was drained during node consolidation: import_users_f ignored the send_task error and returned HTTP 200 with the task still In Progress, so nothing reported the failure.

Change

packages/harvest/src/routes/users.rs, import_users_f. When send_task fails:

• It logs the error, with its details ({err:?}).
• It marks the task Failed with that error, through the existing update_export_fail. That update applies only while the task is still IN_PROGRESS, so a worker's final status is never overwritten.
• It still returns the task. The admin portal's import widget follows the task by id, so it shows Failed and the log message.

The Portal also shows the existing import-error notification if the action rejects (including failure to persist the enqueue error). A widget without a task id now follows its local failure status; once a task id is assigned, persisted worker status remains authoritative.

Successful sends behave as before.

What a failed send means

• In lapin 2.5.5, a send on a closed channel fails before anything is written (InvalidChannelState). Harvest didn't record the error it swallowed in the incident; what is known is the broker shutdown and the missing send and worker-receipt markers. The isolated broker check below reproduced InvalidChannelState after the same kind of shutdown, which supports, but doesn't prove, that this was the error. Other send errors, such as I/O failures while writing, could happen after the broker received the task.
• rusty-celery (Findeton/rusty-celery@b4145925) doesn't wait for publisher confirms, so a successful send doesn't confirm the task was queued either.
• So the task log says it is not known whether the task reached the queue, and tells the admin to check the voters before importing the file again.
• This change adds no automatic retry. If the task did reach the queue and a worker later completes it, the worker can write its final status after Harvest's. The widget keeps polling a failed task (at the slower interval), so it shows that later status while it stays open.

Recovery after the broker comes back

This PR doesn't change recovery. Harvest's live and ready probes already call broker.reconnect (windmill/src/services/probe.rs), which rebuilds the connection and channels once the broker is back. This existing recovery is consistent with the later import succeeding without a Harvest restart.

Testing

• Follow-up Portal regression check (2026-10-18): isolated React/jsdom harness using actual Widget components on all three branches reproduced the baseline Pending state and verified local FAILED, preservation of persisted SUCCESS, and later worker SUCCESS. Executed import handlers with a synthetic rejected mutation and no election event: the new error notification is shown. No shared services were used. Both fresh independent reviewers re-reviewed the correction with no remaining actionable findings.

• cargo check -p harvest --locked passes on this branch.

• Synthetic broker check (pinned rusty-celery and lapin, against a throwaway RabbitMQ 3.12.11 container, not the shared one):

  • a graceful docker stop closed the connection with connection_forced ... 'shutdown', as in the incident;
  • the next send_task returned Err(BrokerError(AMQPError(InvalidChannelState(Error))));
  • after the broker restarted, sends kept failing until broker.reconnect (what the probes call) ran, and then succeeded;
  • the queue held exactly the 2 tasks sent while connected.

• Hasura behaviour (checked against a throwaway Hasura v2.33.1): a 5xx action response reaches the client as a generic internal error, so the failure is reported through the task the portal already tracks, not through an HTTP error. A 5xx is still returned if the task can't be marked Failed; the portal's catch then marks its widget failed.

• Reviews: two independent reviews, one of task status and API/UI propagation, one of AMQP recovery, delivery ambiguity and the backports. Both passed after their findings were fixed.

• Not done: no end-to-end import against a real Harvest and RabbitMQ.

Not in this PR

Keeping RabbitMQ from being evicted by node consolidation during elections (a PodDisruptionBudget or karpenter.sh/do-not-disrupt) belongs in the infrastructure repo.

🤖 Generated with Claude Code

https://claude.ai/code/session_0129Cs8ygVE7UH6N1XzXk8WP

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

🗂️ Base branches to auto review: 2

• release/10.0
• feat/meta-12767/.*

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

• Configuration used: Repository: sequentech/step/.coderabbit.yaml
• Review profile: CHILL
• Plan: Essentials
• Run ID: d69e7585-cd0f-4387-92bb-3be370120a67

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}