Skip to content

feat(flow): local models, a spend cap in every mode, and a pre-run estimate - #24

Open
SeeRay11 wants to merge 2 commits into
devfrom
claude-local-spend
Open

SeeRay11 wants to merge 2 commits into
devfrom
claude-local-spend

Conversation

@SeeRay11

Copy link
Copy Markdown
Owner

Issue for this PR

None. This follows a design discussion about how users can run Claude, and free models, in OpenFlow.

Type of change

  • Bug fix
  • New feature
  • Refactor / code improvement
  • Documentation

What does this PR do?

A Claude subscription can't drive OpenFlow. Anthropic doesn't allow it, and OpenRouter's BYOK takes API keys only. So this PR makes the paid API path cheaper and more predictable, and adds a free local path.

  • Local models (lib/local.ts, /flow/api/local, provider panel group). The panel finds Ollama and LM Studio, or a custom OpenAI-compatible URL, lists their models, and declares them in the global config. The declaration needs four fields because the engine's defaults are wrong for a local server:
    • a local- id, because the catalog hides a models.dev id like lmstudio until a key is stored;
    • tools: true, because tools default to off;
    • a context limit, because it defaults to {0,0};
    • a zero price row, so spend reads $0 while the unpriced-never-$0 rule still holds.
  • Spend cap in every mode. Pipeline.maxSpend is read through spendCapOf and defaults to $5. Non-gauntlet runs hard-stop in enforceCap and record RunLog.capped. A gauntlet keeps its existing graceful cap. There's a cap: selector in the runbar.
  • Pre-run estimate (graph/estimate.ts). Card turns are counted from the graph and each is priced between a short turn and a heavy one. It shows as a range, with an open end for a gauntlet.
  • Preflight:
    • weak-seat fires only when an orchestrator, critic or synthesizer is cheaper than a card it works over. It suggests a change and never sets a model.
    • unpriced-cap fires when a model with no price makes the cap partial.
  • Spend panel shows what share of the input came from the prompt cache.
  • Docs: README sections "Using Claude", "Local models" and "Spend cap and estimate", plus notes on the Anthropic and OpenRouter rows in the provider panel. FLOW.md gets the rules.

Fork-clean: only packages/flow and FLOW.md change.

How did you verify your code works?

  • bun test passes (1123 tests) and bun run typecheck is clean. New tests cover local.ts, the /flow/api/local routes (against a temp XDG_CONFIG_HOME), the engine cap (pipeline stopped at the cap, default cap on, a run under it records nothing), the estimate, weak-seat/unpriced-cap, and cacheShare.
  • Live check against a real opencode serve whose global config came from declare(), with a stand-in OpenAI-compatible server in place of Ollama:
    • the engine listed local-custom/fake-qwen with no key, priced at 0, with tools on;
    • a session prompted on it reached the server with all 12 card tools and returned its answer.
  • Not verified: a real Ollama or LM Studio install, and the panel's write-and-restart click. That click writes the developer's real global config, so it is covered by the route tests rather than clicked live.

Screenshots / recordings

The provider panel shows a new Local models · free, on this machine group with Ollama, LM Studio and a custom-server row.

Checklist

  • I have tested my changes locally
  • I have not included unrelated changes in this PR

🤖 Generated with Claude Code

SeeRay11 and others added 2 commits October 18, 2026 18:18
…timate

- Local models: Ollama, LM Studio or any OpenAI-compatible server, declared
  in the global config from the provider panel (lib/local.ts,
  /flow/api/local). local- ids, tools on, a context limit, and a zero price
  row, so spend reads $0 and gauntlet caps still hold.
- Spend cap in every mode: Pipeline.maxSpend, $5 by default, read through
  spendCapOf. The engine hard-stops non-gauntlet runs on the repriced step
  and records RunLog.capped; a gauntlet keeps its own graceful cap.
- Pre-run estimate in the runbar (graph/estimate.ts): card turns from the
  graph, priced as a range between a short and a heavy turn.
- Preflight: weak-seat warns when a dispatching, judging or deciding card
  runs on a cheaper model than a card it works over; unpriced-cap warns
  when a model makes the cap partial.
- Spend panel shows the share of input served from the prompt cache.
- Docs: README "Using Claude", "Local models", "Spend cap and estimate";
  provider panel notes on the Anthropic and OpenRouter rows (API key or
  BYOK, not a subscription).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Measured against a running LM Studio: its OpenAI /v1/models listing put
text-embedding-nomic-embed-text-v1.5 beside the chat models, so the panel
offered it to cards, where it fails on the first turn. LM Studio's own
/api/v0/models says each model's type and the context it is loaded with;
discovery now reads it when present, drops non-chat models, and writes
the measured context instead of the 32k floor. Other servers fall back to
dropping ids that name an embedding model.

Verified end to end: omnicoder-9b, declared from the panel into a v1
global config, read a file with its read tool on a real card session.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant