Skip to content

Restart an engine that does not serve the viewer - #15

Merged
maxsoldev merged 4 commits into
mainfrom
fix-hung-engine
Nov 19, 2026
Merged

maxsoldev merged 4 commits into
mainfrom
fix-hung-engine

Conversation

@maxsoldev

@maxsoldev maxsoldev commented Nov 17, 2026 •

Copy link
Copy Markdown
Owner

Summary

An engine stuck in its stop kept port 4747 and the writer lock for five days. /health (answered from memory) said ok; every viewer file hung. /quantext view trusted /health and opened a page that never loaded.

 /quantext view
   findEngine            # engine.json + /health, pid must match
+  if same version and GET / does not answer in 5 s
+    toast "the engine (pid N) does not serve the viewer; starting it again…"
+    kill N, wait 3 s; else kill -KILL N, wait 3 s
+    still there → toast "stop it (kill -9 N)", stop
   if none → find runtime, start the bundled engine, poll /health
   open the viewer
 serve() finally
-  await catchUp; watcher.stop; api.close; worker.close; close store; remove engine.json
+  withDeadline(10 s, same cleanup)
+    late → stderr "did not stop within 10 s; exiting", remove engine.json, SIGKILL self

process.exit is not used there: it waits for the thread pool, and a pool thread blocked in a system call never returns. Also, ingest now ignores a subagent *.meta.json that is not a regular file (a FIFO blocked open() for ever).

The kill commands are added to the no-egress allowlist; ARCHITECTURE "Starting the engine" says the new steps.

Evidence

  • Before: the stuck engine (Node 24.14, phase stopping): GET /health → 200 in 1 ms; GET / → no answer after 3 s. 13 FSReqPromise pending, thread pool idle. The browser loaded for ever.
    After: mod tests (claude plugin test packages/mod, 51 pass):
    /health ok, / hangs        → kill 4242, start, open        ✓
    / answers 500, kill ignored → kill 4242, kill -KILL, start   ✓
    engine will not stop       → toast names "kill -9 4242"     ✓
    
    Engine: deadline.test.ts, work that never settles calls onLate; work that ends or fails in time never does. Ingest test: a FIFO meta file no longer blocks ingestAll (red: "blocked" after 2 s).
  • Stuck stop, reproduced (an engine on a test corpus with a FIFO meta file, SIGTERM during catch-up):
    Before: "did not stop within 10 s; exiting", then the process was still alive (blocked in __open).
    After: killed by SIGKILL at 10.5 s.
  • pnpm check green; Bun ingest tests green.

Merge Danger

Door: two-way

Revert the four commits; no stored data or file format changes.

Blast Radius: viewer-start

The mod now signals a pid, but only the one /health answered with, so only its own engine. An engine still catching up that takes over 5 s to serve / would be restarted (its ingest resumes from the store). A stop that legitimately takes over 10 s is now ended with SIGKILL; the next start goes on from the store (SQLite WAL survives a kill). Not covered: a /health that hangs is still treated as no engine. The root cause of the 14 Nov stall is not known: the stop hung in watcher.stop() on one file read, but no pool thread was busy, unlike the FIFO case. Possibly a Node 24.14 / libuv fault on macOS (low confidence; not reproduced).

An engine stopped by SIGTERM hung in its cleanup for days: async file
reads never finished, so it kept port 4747 and the writer lock, and
/health (answered from memory) still said ok while every viewer file
hung. /quantext view then opened a page that never loaded.

The stop now has a 10 s time limit. Past it the engine says so on
stderr, removes its engine.json and exits 1, so the next start finds
the port free. The root cause of the stuck reads is not known yet.
/quantext view trusted /health alone. An engine stuck in its stop still
answered /health from memory while its file reads never finished, so
the mod opened a page that loaded for ever.

The mod now also fetches / (5 s). If the page does not come, it stops
that pid with kill, then kill -KILL (3 s each), and starts the bundled
engine as when none runs. The pid is the one /health answered with, so
the mod only signals its own engine. If the engine does not go, the
toast names the pid to kill.

A /health that hangs is still treated as no engine, as before; a new
start then fails on the writer lock. The engine's new stop time limit
makes that case rare.
process.exit waits for the libuv thread pool. A pool thread blocked
in a system call (reproduced with open() on a FIFO) never returns, so
the stop limit fired and the process stayed: still alive 40 s later.
SIGKILL ends it at once; the OS releases the writer lock, and
engine.json is removed first.

The stopTimeoutMs and exit hooks had no test left, so they are gone.
logFiles took any entry named agent-<id>.meta.json. If it is a FIFO,
open() waits for a writer and holds a thread-pool thread for ever; the
ingest pass never ends, and neither does the engine's stop. This is
how the stuck stop was reproduced. Such a meta file is now ignored,
as a missing one is.

It is not shown that this caused the stall on 14 Nov: there, no pool
thread was busy, so the cause of that stall is still not known.
@maxsoldev
maxsoldev merged commit c8c5471 into main Nov 19, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant