diff --git a/aai_cli/AGENTS.md b/aai_cli/AGENTS.md index 816589cb..951e914f 100644 --- a/aai_cli/AGENTS.md +++ b/aai_cli/AGENTS.md @@ -150,7 +150,7 @@ heavily-reworked commands with long bodies; small commands keep the inline ### Feature subsystems -- **`streaming/`** + `client.stream_audio` — v3 realtime API. Event callbacks run on the SDK reader thread and guard against `BrokenPipeError` (`stdio.silence_stdout()`) so a closed pipe never dumps a thread traceback. +- **`streaming/`** + `client.stream_audio` — v3 realtime API. Event callbacks run on the SDK reader thread and guard against `BrokenPipeError` (`stdio.silence_stdout()`) so a closed pipe never dumps a thread traceback. Under `stream --llm` the `run_chain` request runs outside `StreamSession._callback_lock` (so a second `--system-audio` source keeps streaming), so refreshes can finish out of order: `_publish` orders publication under that lock by the published transcript turn count (`_published_len`), dropping an answer that covers no more turns, and the closing flush compares against that published coverage rather than the claimed `_summarized_len`, so a refresh still in flight at Ctrl-C can't leave the `--save-dir` note short. - **`core/dictation.py`** + **`core/signals.py`** + `commands/dictate/` — `assembly dictate`: headless dictation over the **Dictation API** (; `Environment.dictation_base`, one multipart POST `/v1/transcribe/live` per utterance — the JSON `config` part **must precede** the raw-PCM `audio` part or the server answers 400; up to 120 s). The response carries the verbatim `text` and the send-ready cleanup rewrite `llm_response` (best-effort: on `llm_error` it is null and `DictationResult.final_text` falls back to `text`); dictate prints the rewrite by default, `--verbatim` the transcript, and `--instruction`/`--rewrite-mode` map to `llm_instruction`/`rewrite_mode`. `--keyterm` is `keyterms_prompt` (`--word-boost` is a hidden deprecated alias). It needs no terminal: recording starts immediately and `dictate_exec._record` polls `signals.stop_on_terminate` between ~100 ms mic chunks for a SIGTERM, which finishes the utterance (clean exit 0) — so a hotkey tool like Hammerspoon can launch it as a background task and `kill -TERM`/`task:terminate()` to transcribe. SIGINT (Ctrl-C) still cancels (exit 130). Both boundaries (the stop latch, mic, HTTP) are injectable, so the suite never needs a real signal or microphone (`tests/test_dictate_exec.py` scripts the SIGTERM latch). Contrast `signals.terminate_as_interrupt` (used by `stream`/`agent`/`speak`), which routes SIGTERM into the *cancel* path instead. - **`agent/`** — full-duplex voice agent (mic in, TTS out via `voices.py`). - **`agent_cascade/`** + `commands/agent_cascade/` — `assembly agent-cascade`: the same live terminal conversation as `assembly agent`, but **client-orchestrated** — `engine.run_cascade` wires Streaming STT → the LLM Gateway → streaming TTS itself instead of talking to the Voice Agent endpoint, mirroring what the `agent-cascade` `assembly init` template does server-side. Reuses the agent slice's `DuplexAudio`/`AgentRenderer` and `core.client.stream_audio`/`core.llm.complete`/`tts.session.synthesize`; the three network legs are injected through `engine.CascadeDeps` (the `tts/session.py` seam) so the cascade — greeting, clause-level streaming TTS, barge-in — is unit-tested against fakes with no sockets/mic/speaker. The LLM leg is a deepagents graph **assembled** by `brain.py` (tools, backend, middleware, `build_graph`, plus the shared `SpeechDelta`/`ToolNotice`/`ApprovalPause` event types and tool-affordance vocabulary) and **driven** turn-by-turn by `streamer.py` (`build_streamer`, `graph.stream(stream_mode="messages")`) — the natural build-vs-drive seam, split so each file stays under the 500-line gate (the dependency is one-directional: `streamer` imports `brain`, never the reverse). **Context-window management is the graph's job, not the engine's** — `create_deep_agent` wires deepagents' own `SummarizationMiddleware` into the stack (summarize the oldest turns, offload the evicted history to a file), so the engine feeds the *full* untrimmed running history each turn and lets the graph compact it; the old client-side `text.trim_history`/`config.max_history` sliding window is gone from this path (`max_history` now only drives the hand-rolled `--show-code`/`assembly init` cascade, which doesn't use deepagents). The engine buffers `SpeechDelta`s, flushes complete clauses with `text.pop_clauses` (soft-separator clauses gated by `engine._MIN_CLAUSE_CHARS`), and synthesizes each clause with **streaming TTS** (`tts.session.synthesize(on_audio=…)`) so audio starts on the first frame instead of after the whole reply. The reply runs on a throwaway producer thread feeding a `queue.Queue` the worker drains under a monotonic deadline (the wall-clock backstop that replaced `_complete_within`), and an abandoned-on-timeout graph leg's langchain `ThreadPoolExecutor` worker is detached (`_detach_executor_threads_since`) so it can't wedge interpreter exit. A `ToolNotice` surfaces the "Searching the web…" affordance and drops any unspoken preamble. **The `write_todos` plan** (deepagents auto-installs langchain's `TodoListMiddleware`) is reassembled from the streamed call args by `plan.py`'s `TodoCollector` and surfaced as a `plan.TodoUpdate` (not a `ToolNotice`); `engine._consume` forwards it to `renderer.todos_updated` without gating speech, so the voice TUI shows an in-place `messages.TodoList` checklist (and `AgentRenderer` emits a `plan` NDJSON event / a stderr checklist) — a visible plan to go with the spoken "first I'll…, then I'll…". Under `-v` (`debuglog.active()`) `streamer._stream_graph` logs each accumulated assistant line, tool call, and tool result as it streams. **Front-end:** an interactive mic session in human mode runs a **voice-only Textual TUI** (`agent_cascade/tui.py`, `LiveAgentApp`) by default — there's no text input (you can't type to it), just a transcript + an animated voice bar tracking listening/thinking/speaking. It uses its own `banner` wordmark, `messages` widgets, and `tui_status.voicebar_markup`/`VOICE_FRAMES` — all modules that now live in `agent_cascade/`; the blocking `run_cascade` runs on a worker thread and reaches the UI through a `_TuiRenderer` (the `engine.Renderer` protocol) that hops each call onto the UI thread, and a quit calls `DuplexAudio.close` to end the mic iterator and unblock that worker. `_exec._should_use_tui` gates it: file/sample input, `--json`/`-o text`, and a non-TTY all fall back to the plain `AgentRenderer` line output. **`--files`** (on by default; `--no-files` opts out) swaps the brain's in-memory backend for a real-cwd, sandbox-capable `SandboxedShellBackend` (`aai_cli/agent_cascade/sandbox.py`): file ops behave as before (traversal-blocked `virtual_mode`), and because it implements `SandboxBackendProtocol` deepagents binds a *functional* `execute` that runs commands OS-sandboxed in the real cwd — `sandbox-exec` (SBPL) on macOS, `bwrap` on Linux, refused (never an unconfined fallback) on any other platform or with the sandbox binary missing; the OS sandbox blocks the network, confines writes to cwd (+ the temp dir), and read-denies credential stores (`~/.ssh`/`~/.aws`/…, `.env*`, `.claude/`). The policy renderers are pure and the subprocess/capability boundaries injected, so the suite asserts *what we'd run* with no real sandbox. `write_file`/`edit_file`/`execute` are gated via `interrupt_on` + an `InMemorySaver`. The write gate is **path-scoped (the allow/interrupt permission model), not all-or-nothing**: `write_gate.write_interrupt_on(config.auto_write_paths)` hands `write_file`/`edit_file` one `InterruptOnConfig` whose `when` predicate fires only for writes *outside* every `--auto-write` subtree, so a write inside one runs ungated while others still pause (empty paths gate every write, the prior behavior). The matching is a transparent posix path-segment prefix check — `--auto-write DIR` is a directory subtree, and reads stay ungated, so no glob/bulk-tool machinery is needed (and no reach into deepagents internals: `create_deep_agent(permissions=…)` is unusable here — it raises `NotImplementedError` against an execute-capable backend like our `SandboxedShellBackend`). A non-string or `..`-containing target fails safe to gating. `execute` stays a plain `True` (unconditionally gated) — a command isn't a single file path to scope. `--auto-write DIR` is normalized to a cwd-rooted virtual root in `_exec._normalize_auto_write` (rejecting `..`/`~`/whole-tree aliases) and requires `--files`. `streamer._stream_gated` detects the post-stream interrupt (`graph.get_state(config).interrupts`), asks an injected `Approver`, and resumes with `Command(resume=…)`, bracketing the human wait in `ApprovalPause` events so `engine._consume` suspends its reply deadline (`risk.py` surfaces a shell-risk warning on the prompt). The voice TUI supplies the approver via `agent_cascade.modals.ApprovalScreen` (`y`/`a`/`n`), which can *also* be resolved hands-free by voice: while a write awaits approval, `_consume` arms `_awaiting_approval` and `engine.on_turn` routes the next final transcript to `app.submit_voice_approval` → `ApprovalScreen.try_voice`, which applies `spoken_approval.spoken_decision` (an unambiguous affirmative approves, anything else rejects — fail-safe; destructive `risk.py`-flagged commands ignore the spoken answer and require a keypress). **Project grounding (independent of `--files`):** `_exec.run_agent_cascade` reads the launch directory's `AGENTS.md`/`CLAUDE.md` via `agent_cascade/project_context.load_project_context()` into `CascadeConfig.project_context`, which `brain.build_graph` threads into `prompt.build_system_prompt(..., project_context=…)` (appended as project background after the persona/tool guidance). `AGENTS.md` wins precedence, identical content (a symlinked `CLAUDE.md`) is de-duplicated, and the total is capped at `project_context.MAX_CONTEXT_CHARS`. It's read at the command boundary (not in `build_graph`) so the brain stays hermetic, and the `--show-code` path builds its own config without it. Headless runs auto-deny (`_exec._deny_writes`). `--files` also turns on durable per-project memory via deepagents' `MemoryMiddleware` (`memory=["./.deepagents/AGENTS.md"]`), distinct from the in-session `InMemorySaver`. The gateway-bound, sandbox-backed general-purpose subagent (deepagents' `task` tool) for delegating a focused subtask is **auto-added by deepagents** — we don't declare it. We only override its prose for a voice turn (a spoken-length summary, not the SDK's "complete answer" default) via a harness profile keyed by the gateway model's provider (`subagents.register_gp_subagent_profile`, called from `build_graph` so the deepagents import stays lazy — and kept off `brain.py`, which sits at the 500-line gate). It inherits the gateway-bound model, the sandboxed toolset, *and* the top-level `interrupt_on` (deepagents' `graph.py` merges the top-level config into the auto-added subagent), so a delegated `write_file`/`edit_file`/`execute` surfaces at the *parent* `get_state().interrupts` with no per-subagent restatement (so `_pending_writes` gates it too — verified by a HITL spike, locked in `tests/test_agent_cascade_subagents.py`). Reads (incl. `grep`) stay ungated. diff --git a/aai_cli/streaming/session.py b/aai_cli/streaming/session.py index 7e77ccd2..984722b4 100644 --- a/aai_cli/streaming/session.py +++ b/aai_cli/streaming/session.py @@ -91,6 +91,9 @@ class StreamSession: # -inf so the very first finalized turn always produces an immediate summary. _summarized_len: int = 0 _last_summary_at: float = float("-inf") + # Turn count of the answer last shown/kept. Refreshes run unlocked and can finish + # out of order, so an answer covering no more turns than this one is stale. + _published_len: int = 0 # First CLIError raised by an --llm refresh. The chain runs on the SDK reader # thread, where raising would dump a thread traceback instead of reaching # run_command, so the error is recorded (and warned once) there and re-raised @@ -177,7 +180,7 @@ def _maybe_summarize(self, *, final: bool = False) -> None: return with self._callback_lock: turns = len(self.transcript) - if turns <= self._summarized_len: + if turns <= self._refreshed_through(final=final): return # nothing new since the last refresh now = self.clock() throttled = self.llm_interval > 0 and now - self._last_summary_at < self.llm_interval @@ -204,9 +207,31 @@ def _maybe_summarize(self, *, final: bool = False) -> None: f"[aai.muted]--llm refresh failed: {exc.message}[/aai.muted]" ) return - # Hold the latest answer so --save-dir can write it as the .md note at the end. - self._last_answer = answer - follow(answer, turns) + self._publish(follow, answer, turns) + + def _refreshed_through(self, *, final: bool) -> int: + """The turn count a refresh must exceed to be worth running. + + Mid-stream, a claimed refresh already covers its turns even while it is still in + flight. The closing flush compares against the last *published* answer instead: a + reader thread still inside the chain when Ctrl-C ends the run may never land. + """ + return self._published_len if final else self._summarized_len + + def _publish(self, follow: FollowRenderer, answer: str, turns: int) -> None: + """Show and keep ``answer`` unless a refresh over as many turns already landed. + + Two --system-audio sources (or a request outlasting --llm-interval) can finish + refreshes out of order; checking and publishing under the lock keeps the panel, + the NDJSON stream, and the --save-dir note from regressing to an older answer. + """ + with self._callback_lock: + if turns <= self._published_len: + return + self._published_len = turns + # Held so --save-dir can write the latest answer as the .md note at the end. + self._last_answer = answer + follow(answer, turns) def _audio_target(self, source_label: str | None) -> Path | None: """The WAV path this source tees to, if any. diff --git a/tests/test_stream_llm_ordering.py b/tests/test_stream_llm_ordering.py new file mode 100644 index 00000000..1c92e4cb --- /dev/null +++ b/tests/test_stream_llm_ordering.py @@ -0,0 +1,191 @@ +"""Ordering of overlapping ``stream --llm`` refreshes. + +The chain runs outside the session lock so a second source's turns keep flowing during a +refresh, which means two refreshes can be in flight at once and finish out of order. +These tests script that interleaving with events (no sleeps, sockets, or microphone): +the first ``run_chain`` call parks until released, so a later, longer refresh lands first. +""" + +import io +import threading +import types +from datetime import datetime + +import pytest + +from aai_cli.core.errors import APIError +from aai_cli.streaming import naming, savedir +from aai_cli.streaming.render import StreamRenderer +from aai_cli.streaming.session import StreamSession +from aai_cli.ui.follow import FollowRenderer + +_WAIT = 5.0 + + +class _GatedChain: + """A fake ``llm.run_chain`` whose first call blocks until ``release`` is set.""" + + def __init__(self, *, fail_first=False): + self.calls: list[str] = [] + self.entered = threading.Event() + self.release = threading.Event() + self.fail_first = fail_first + + def __call__(self, api_key, prompts, *, transcript_text, model, max_tokens): + index = len(self.calls) + self.calls.append(transcript_text) + if index == 0: + self.entered.set() + assert self.release.wait(_WAIT) + if self.fail_first: + raise APIError("gateway down") + return f"{transcript_text}#{index + 1}" + + +def _turn(text): + return types.SimpleNamespace(transcript=text, end_of_turn=True, speaker_label=None) + + +def _session(monkeypatch, emitted, chain, *, interval=0.0, clock=lambda: 0.0, **extra): + monkeypatch.setattr("aai_cli.ui.follow.output.emit_ndjson", emitted.append) + monkeypatch.setattr("aai_cli.streaming.session.llm.run_chain", chain) + return StreamSession( + api_key="sk", + base_flags={}, + overrides=None, + config_file=None, + renderer=StreamRenderer(json_mode=True, out=io.StringIO()), + follow=FollowRenderer(json_mode=True), + llm_prompts=["sum"], + model="m", + max_tokens=10, + llm_interval=interval, + clock=clock, + **extra, + ) + + +def _start(target): + errors: list[Exception] = [] + + def run(): + try: + target() + except (AssertionError, APIError) as exc: + errors.append(exc) + + thread = threading.Thread(target=run, daemon=True) + thread.start() + return thread, errors + + +def _finish(thread, errors): + thread.join(_WAIT) + assert not thread.is_alive() + assert errors == [] + + +def test_late_older_refresh_does_not_replace_newer_answer(monkeypatch): + emitted: list[dict] = [] + chain = _GatedChain() + session = _session(monkeypatch, emitted, chain) + slow = _start(lambda: session.on_turn(_turn("a"))) # refresh over turn 1 parks + assert chain.entered.wait(_WAIT) + session.on_turn(_turn("b")) # refresh over turns 1-2 lands first + chain.release.set() + _finish(*slow) + assert emitted == [{"type": "answer", "turns": 2, "output": "a b#2"}] + assert session._last_answer == "a b#2" + session._maybe_summarize(final=True) # both turns already published -> no re-run + assert chain.calls == ["a", "a b"] + assert len(emitted) == 1 + + +def test_refresh_outlasting_the_interval_cannot_regress_the_answer(monkeypatch): + now = {"t": 1000.0} + emitted: list[dict] = [] + chain = _GatedChain() + session = _session(monkeypatch, emitted, chain, interval=30.0, clock=lambda: now["t"]) + slow = _start(lambda: session.on_turn(_turn("a"))) + assert chain.entered.wait(_WAIT) + now["t"] = 1040.0 # the slow request has outlived the 30s window + session.on_turn(_turn("b")) + chain.release.set() + _finish(*slow) + now["t"] = 1050.0 + session.on_turn(_turn("c")) # inside the new window -> throttled + session._maybe_summarize(final=True) # closing flush covers the tail + assert [(e["turns"], e["output"]) for e in emitted] == [(2, "a b#2"), (3, "a b c#3")] + assert session._last_answer == "a b c#3" + + +def test_closing_flush_does_not_wait_on_an_unfinished_refresh(monkeypatch): + # Ctrl-C can end the stream while a reader thread is still inside run_chain; the + # closing flush must produce an answer for the full transcript itself. + emitted: list[dict] = [] + chain = _GatedChain() + session = _session(monkeypatch, emitted, chain) + slow = _start(lambda: session.on_turn(_turn("a"))) + assert chain.entered.wait(_WAIT) + session._maybe_summarize() # mid-stream, the in-flight refresh already claims turn 1 + assert chain.calls == ["a"] + session._maybe_summarize(final=True) + assert session._last_answer == "a#2" + chain.release.set() + _finish(*slow) + # The parked refresh covered the same turns, so it lands as a duplicate and is dropped. + assert emitted == [{"type": "answer", "turns": 1, "output": "a#2"}] + assert session._last_answer == "a#2" + + +def test_failed_older_refresh_keeps_newer_answer_and_still_fails_close(monkeypatch, capsys): + emitted: list[dict] = [] + chain = _GatedChain(fail_first=True) + session = _session(monkeypatch, emitted, chain) + slow = _start(lambda: session.on_turn(_turn("a"))) + assert chain.entered.wait(_WAIT) + session.on_turn(_turn("b")) + chain.release.set() + _finish(*slow) + assert emitted == [{"type": "answer", "turns": 2, "output": "a b#2"}] + assert session._last_answer == "a b#2" + assert "gateway down" in capsys.readouterr().err + session.on_turn(_turn("c")) # the latched error stops further refreshes + assert chain.calls == ["a", "a b"] + with pytest.raises(APIError): + session._maybe_summarize(final=True) + + +def test_parallel_sources_save_note_for_latest_transcript(monkeypatch, tmp_path): + # Drives the real run_parallel -> on_turn -> closing flush -> --save-dir finalize path, + # faking only the two network boundaries (the streaming socket and the LLM chain). + emitted: list[dict] = [] + chain = _GatedChain() + plan = savedir.SaveDirPlan( + save_dir=tmp_path, + now=datetime(2026, 1, 2, 3, 4, 5), + name="sync", + auto_name=False, + write_note=True, + ) + naming.ensure_dir(plan.paths.directory) + + def fake_stream_audio(api_key, audio, *, params, on_begin, on_turn, on_termination): + source = next(iter(audio)) + if source == b"system": + on_turn(_turn("first")) # parks inside the chain + else: + assert chain.entered.wait(_WAIT) + on_turn(_turn("second")) + chain.release.set() + + monkeypatch.setattr("aai_cli.streaming.session.client.stream_audio", fake_stream_audio) + session = _session( + monkeypatch, emitted, chain, save_plan=plan, save_transcript=plan.paths.transcript + ) + session.run_parallel([("system", [b"system"], 16000), ("you", [b"you"], 16000)]) + latest = " ".join(session.transcript) + assert [e["turns"] for e in emitted] == [2] + assert session._last_answer == f"{latest}#2" + assert plan.paths.note.read_text(encoding="utf-8") == f"{latest}#2\n" + assert plan.paths.transcript.read_text(encoding="utf-8").splitlines() == session.transcript