Skip to content

fix(proxy): follow a replaced daemon instead of dying with it - #11

Merged
kolkov merged 4 commits into
mainfrom
fix/proxy-reconnect
Oct 19, 2026
Merged

kolkov merged 4 commits into
mainfrom
fix/proxy-reconnect

Conversation

@kolkov

@kolkov kolkov commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Problem

Proxy read the daemon's port once at start. When the daemon was replaced while a client was connected (a restart during a deploy, a crash, an idle shutdown):

  • the next request failed with connection refused, Proxy returned the error, and the host process exited, so every MCP client had to reconnect by hand;
  • the lease was not re-attached, so the new daemon saw zero agents although clients were still open, and its idle timer could shut it down under them.

Fix

The proxy now follows a replaced daemon:

What happened What the proxy does
Its lease dropped Polls the PID file for up to Config.Timeout and re-attaches. It never starts a daemon here, so Stop, Hold and the idle timer are not undone by an idle client
A request's dial failed, or a daemon answered 401 (another daemon took the old port) The request never ran: reconnect (EnsureRunning when Config.Binary is set, otherwise the PID file) and send it once more
A request failed after it was sent It may have run (a tool call may have changed files): the client gets a JSON-RPC error for that id (-32603, "daemon unavailable: …") and the request is never sent again. Notifications get no answer
The daemon is held Requests get the JSON-RPC error; the proxy stays up and works again after Release

Proxy now returns only when its context ends, stdin ends or fails, or stdout fails.

proxyAttach returns a lease (cancel + done channel) instead of a bare cancel func, so the watcher can see the connection end. The public API is unchanged.

Tests

  • Integration tests (helper-process daemon, real sockets):
    • request after a restart;
    • lease re-attached after a restart under a 2 s idle timeout, with an idle client;
    • a stopped daemon is not restarted by an idle proxy, only by the next request;
    • a daemon killed mid-request: JSON-RPC error for that id, and the request executed exactly once;
    • Hold, then Release.
  • Unit tests:
    • which errors count as not delivered (dial vs read/EOF/cancel);
    • the error reply (numeric and string ids, notification, null id, invalid input);
    • 401 is retried once, 500 and 404 are not.
  • All new tests fail on v0.4.0.
  • Five mutations each fail a test: no re-attach, no retry, retry everything, the watcher starts the daemon, no 401 retry.
  • Local results:
    • go test ./...: ok;
    • go vet for windows/linux/darwin: ok;
    • golangci-lint run ./... (whole repo) for windows/linux/darwin: 0 issues;
    • full suite run three times in a row: green.

Found while running GODE: every deploy forced /mcp reconnect in every session.

Also in this PR

  • test: the serve tests require Serve to return nil after cancel, and the attached-clients test requires Proxy to return nil once stdin closes. golangci-lint (latest, as CI runs it) flagged the discarded receives.
  • chore: prepare release v0.4.1: CHANGELOG and ROADMAP.
  • fix(proxy): look for the next daemon without probing the PID lock: windows-latest CI showed that polling pidlock.IsHeld holds the PID file exclusively for a moment on Windows, so a starting daemon failed with "already running" and Status read a stale file as running. The watcher now only reads the PID file and lets the lease prove the daemon alive. A regression test checks that a tight TryLock loop never finds the lock taken while a proxy is watching (before the fix it hit 2–3 times in ~170k tries).

Proxy read the daemon's port once. After a restart (a deploy, a crash, an
idle shutdown) the next request failed and Proxy returned, so every MCP
client had to reconnect by hand; the lease was not re-attached either, so
the new daemon saw zero agents while clients were still open.

- A dropped lease makes the proxy poll the PID file for up to
  Config.Timeout and re-attach. It never starts a daemon there: Stop, Hold
  and the idle timer must not be undone by an idle client.
- A request whose dial failed, or that a daemon answered 401 (another
  daemon on the old port), never ran: it is sent once more after
  reconnecting (EnsureRunning when Config.Binary is set, else the PID file).
- A request that failed after it was sent may have run: the client gets a
  JSON-RPC error for its id (-32603) and it is never sent again.
  Notifications get no answer.
- Proxy returns only on context end, stdin end/error or stdout error.

Tests: integration (helper-process daemon) for restart, lease re-attach
under a 2 s idle timeout, on-demand start after Stop, mid-flight kill
(executed exactly once), Hold/Release; unit tests for the dial/401
classification and the error reply. All new tests fail on v0.4.0; five
mutations (no re-attach, no retry, retry everything, watcher starts the
daemon, no 401 retry) each fail a test.
golangci-lint (latest, as CI runs it) flags receives from error channels
whose value is discarded. The serve tests now require Serve to return nil
after cancel, and the attached-clients test requires Proxy to return nil
once stdin is closed and its ctx cancelled (the daemon being gone no
longer ends it). In Group.Run the remaining actors' errors are drained on
purpose: the first error decides the result; nolint with that reason.
windows-latest CI failed two of the new proxy tests. The lease watcher
polled pidlock.IsHeld every 200 ms, and IsHeld opens the PID file for
writing: on Windows that holds it exclusively for a moment. A daemon
starting at that moment failed with "already running: pid file locked",
and Status read the stale file of a stopped daemon as running. With many
proxies watching during a deploy, a restart could fail the same way.

The watcher and reconnect now only read the PID file (a shared read) and
let the lease prove the daemon alive: a stale file names a dead port
(attach refused), a foreign daemon on that port rejects the token (401).

Regression test: while a proxy watches and no daemon runs, a tight
TryLock loop never finds the lock taken (red before: 2 and 3 hits in
~170k tries), and Start succeeds. Mutation (IsHeld back in the watcher)
fails it.
@kolkov
kolkov merged commit 79557f0 into main Oct 19, 2026
8 checks passed
@kolkov
kolkov deleted the fix/proxy-reconnect branch October 19, 2026 12:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant