Skip to content

Bump amqp091-go to 1.15.0 and fix projector shutdown race - #118

Open
klemmster wants to merge 3 commits into
masterfrom
fix/projection-notification-queue-shutdown-race
Open

klemmster wants to merge 3 commits into
masterfrom
fix/projection-notification-queue-shutdown-race

Conversation

@klemmster

Copy link
Copy Markdown
Contributor

Summary

  • Bumps github.com/rabbitmq/amqp091-go from 1.13.0 to 1.15.0 (carries over dependabot branch, now merged with current master)
  • Fixes the intermittent TestAggregateProjectorSuite/AdvisoryLock/TestRun integration test failure that broke CI for the dependabot branch

Root cause

When the background notification processor shuts down, it closes the done channel and then the queue channel. A processor goroutine waiting in NotificationQueue.Next could then randomly pick the closed queue channel in its select, receive the channel's zero value (nil), and mistake it for a legitimate "trigger out-of-sync projections" notification. Each phantom handler then failed to queue its notifications ("processor was stopped"), producing the alternating failed to queue notification / the ProcessHandler produced an error log pairs the suite flags as errors — while the projections themselves still converged, so the test only failed on its no-error-logs assertion, and only on some runners/Go versions.

Fix

Next now verifies after every receive that the queue was not closed, and always reports stopped so no handler runs after shutdown begins. A unit test covers both the shutdown path (which fails without the fix) and normal nil delivery.

Verification

  • New regression test fails on the old code, passes with the fix
  • make test-unit, make test-examples green
  • Full integration suite green with -race (projector suites additionally run 10x)

Supersedes #115.

💘 Generated with Crush

dependabot Bot and others added 3 commits September 18, 2026 10:33
Bumps [github.com/rabbitmq/amqp091-go](https://github.com/rabbitmq/amqp091-go) from 1.13.0 to 1.15.0.
- [Release notes](https://github.com/rabbitmq/amqp091-go/releases)
- [Changelog](https://github.com/rabbitmq/amqp091-go/blob/main/CHANGELOG.md)
- [Commits](rabbitmq/amqp091-go@v1.13.0...v1.15.0)

---
updated-dependencies:
- dependency-name: github.com/rabbitmq/amqp091-go
  dependency-version: 1.15.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
…fication-queue-shutdown-race

# Conflicts:
#	go.mod
…down

When the notification processor is stopped, the done channel is closed
followed by the queue channel. A processor waiting in Next could then
randomly receive the zero value of the closed queue channel and mistake
it for a real nil notification, running the ProcessHandler during
shutdown. Every out-of-sync notification the handler tried to queue then
failed with "processor was stopped", which surfaced as intermittent
"failed to queue notification" errors in the aggregate projector
integration tests.

Next now checks whether the queue was closed after each receive and
always reports the processor as stopped, so no handler runs after
shutdown begins. A regression test covers both the shutdown and the
normal nil-delivery paths.

Assisted-by: Crush:zai-org/GLM-5.3

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant