Repository navigation
Restart an engine that does not serve the viewer - #15
Merged
Merged
Conversation
An engine stopped by SIGTERM hung in its cleanup for days: async file reads never finished, so it kept port 4747 and the writer lock, and /health (answered from memory) still said ok while every viewer file hung. /quantext view then opened a page that never loaded. The stop now has a 10 s time limit. Past it the engine says so on stderr, removes its engine.json and exits 1, so the next start finds the port free. The root cause of the stuck reads is not known yet.
/quantext view trusted /health alone. An engine stuck in its stop still answered /health from memory while its file reads never finished, so the mod opened a page that loaded for ever. The mod now also fetches / (5 s). If the page does not come, it stops that pid with kill, then kill -KILL (3 s each), and starts the bundled engine as when none runs. The pid is the one /health answered with, so the mod only signals its own engine. If the engine does not go, the toast names the pid to kill. A /health that hangs is still treated as no engine, as before; a new start then fails on the writer lock. The engine's new stop time limit makes that case rare.
process.exit waits for the libuv thread pool. A pool thread blocked in a system call (reproduced with open() on a FIFO) never returns, so the stop limit fired and the process stayed: still alive 40 s later. SIGKILL ends it at once; the OS releases the writer lock, and engine.json is removed first. The stopTimeoutMs and exit hooks had no test left, so they are gone.
logFiles took any entry named agent-<id>.meta.json. If it is a FIFO, open() waits for a writer and holds a thread-pool thread for ever; the ingest pass never ends, and neither does the engine's stop. This is how the stuck stop was reproduced. Such a meta file is now ignored, as a missing one is. It is not shown that this caused the stall on 14 Nov: there, no pool thread was busy, so the cause of that stall is still not known.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
An engine stuck in its stop kept port 4747 and the writer lock for five days.
/health(answered from memory) saidok; every viewer file hung./quantext viewtrusted/healthand opened a page that never loaded.process.exitis not used there: it waits for the thread pool, and a pool thread blocked in a system call never returns. Also, ingest now ignores a subagent*.meta.jsonthat is not a regular file (a FIFO blockedopen()for ever).The
killcommands are added to the no-egress allowlist; ARCHITECTURE "Starting the engine" says the new steps.Evidence
stopping):GET /health→ 200 in 1 ms;GET /→ no answer after 3 s. 13FSReqPromisepending, thread pool idle. The browser loaded for ever.After: mod tests (
claude plugin test packages/mod, 51 pass):deadline.test.ts, work that never settles callsonLate; work that ends or fails in time never does. Ingest test: a FIFO meta file no longer blocksingestAll(red: "blocked" after 2 s).Before: "did not stop within 10 s; exiting", then the process was still alive (blocked in
__open).After: killed by SIGKILL at 10.5 s.
pnpm checkgreen; Bun ingest tests green.Merge Danger
Door: two-way
Revert the four commits; no stored data or file format changes.
Blast Radius: viewer-start
The mod now signals a pid, but only the one
/healthanswered with, so only its own engine. An engine still catching up that takes over 5 s to serve/would be restarted (its ingest resumes from the store). A stop that legitimately takes over 10 s is now ended with SIGKILL; the next start goes on from the store (SQLite WAL survives a kill). Not covered: a/healththat hangs is still treated as no engine. The root cause of the 14 Nov stall is not known: the stop hung inwatcher.stop()on one file read, but no pool thread was busy, unlike the FIFO case. Possibly a Node 24.14 / libuv fault on macOS (low confidence; not reproduced).