Skip to content

Stop reconnect storms when the web socket is rate limited - #226

Merged
natekspencer merged 8 commits into
mainfrom
fix/ws-rate-limit-backoff
Oct 14, 2026
Merged

natekspencer merged 8 commits into
mainfrom
fix/ws-rate-limit-backoff

Conversation

@natekspencer

@natekspencer natekspencer commented Oct 14, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Fixes the library side of bretterer/home-assistant-rivian#305. After a restart, setup kept failing because the web socket was closed with 1013 "Rate limited". Each failed retry left a client reconnecting, and each new client restarted its backoff, so the limit never cleared until a fresh login.

Changes

  • Stop reconnecting with nothing to deliver. A monitor with no subscriptions stops after a disconnect instead of reconnecting forever. A failed subscribe closes the connection when nothing else is subscribed, and so does removing the last subscription.
  • Share the backoff per user session. A new client waits out another client's backoff instead of starting over. The backoff only holds off opening new connections, and the monitor re-checks it after waiting.
  • Never lose a backoff. It's recorded as soon as the cause is known (a 1013 close or a missing connection_ack), so close() can't drop it.
  • Fail subscribes fast, and say why. A subscribe stops waiting when the connection closes and logs the cause: rate limited, not acknowledged, or closed (with the server's reason, e.g. Unauthenticated). Failures caused by a backoff are warnings; others are errors.
  • New public properties on Rivian:
    • ws_backoff_remaining: seconds until the web socket may reconnect;
    • ws_rate_limited: whether the web socket is backing off after a rate limit, for any client on the session;
    • ws_unauthenticated: whether the server closed the socket as unauthenticated.
  • Keep connections up through callback errors. A subscription callback that raises is logged, and the connection keeps going.

Behavior changes for callers

  • Unsubscribing every subscription closes the web socket. The next subscribe reconnects, and that counts toward the server's rate limit.
  • Failed subscribes return sooner. While backing off, subscribe returns None without connecting.
  • To trigger reauth, use ws_unauthenticated and ws_rate_limited instead of reading _ws_monitor.
  • Integrations still need to close the client when setup fails.

Tests

tests/test_ws_monitor.py covers:

  • stopping when nothing is subscribed;
  • the shared backoff, including a backoff pushed back by another client and an open connection that isn't blocked;
  • fast, explained subscribe failures (rate limited, not acknowledged, unauthenticated) and their log levels;
  • a backoff surviving close();
  • closing on the last unsubscribe;
  • callback errors;
  • the per-client rate-limit flag.

The backoff fixture uses a fake clock, so the tests don't depend on the random part of the delay.

🤖 Generated with Claude Code

natekspencer and others added 8 commits October 14, 2026 12:37
- A monitor with nothing subscribed stops instead of reconnecting
  forever, and a failed subscribe closes it, so a client left behind
  (e.g. by a failed Home Assistant setup) doesn't keep the rate limit
  active
- Share backoff state per user session, so a new client waits out
  another's backoff instead of starting over at the first attempt
- Log why a subscription failed instead of a blank error

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Record the backoff as soon as its reason is known, so closing the
  monitor after a failed subscribe can't lose it
- End the subscribe wait as soon as the connection closes, and report
  why (e.g. "the web socket connection was rate limited")
- Add Rivian.ws_backoff_remaining and Rivian.ws_rate_limited
- Close the connection when its last subscription is removed

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The monitor sleeps for the remaining backoff, which can fall just under
30 seconds when the jitter is small, so the fake sleep missed it and the
test timed out. Treat any sleep longer than the 1 second poll as a
backoff, and allow for elapsed time in the delay assertions.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…koffs

- Log a subscription callback's exception instead of letting it end the
  receiver, and only clear the ack when the socket has closed, so a
  failing callback no longer makes a healthy connection back off
- Only hold off new connections during a backoff, so another client's
  rate limit on the same session doesn't block subscribing on an open
  connection
- Re-check the shared backoff after waiting, in case it was pushed back

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Keep the server's close reason, report it when a subscribe fails, and
  add Rivian.ws_unauthenticated so integrations can trigger a reauth
- Track whether the latest backoff was a rate limit per monitor, so
  another client's healthy connection on the same session can't clear it
- Log subscribe failures caused by a backoff as warnings

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Share whether the current backoff is for a rate limit, and report it
only while that backoff lasts, so a new client created for a retry sees
it and another client's healthy connection can't clear it early.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Only move the shared retry time later, so a missing ack on one client
can't shorten another's rate-limit backoff or clear its rate-limit flag.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@natekspencer
natekspencer merged commit 7d60282 into main Oct 14, 2026
6 checks passed
@natekspencer
natekspencer deleted the fix/ws-rate-limit-backoff branch October 14, 2026 20:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant