Local-first knowledge-graph tool for structured corpora, with principled clustering, strict ingest-time schema validation, and honest LLM token-economy accounting.
Nuthatch turns a curated corpus (research papers, patents, internal documents, technical reports, notes, Repomix-preprocessed codebases, or any text-based source) into a navigable knowledge graph and serves it to an MCP-aware agent (Claude Code, Codex, OpenCode, Aider, Pi, Hermes, an Obsidian plugin, or your own client). The agent queries the graph through nine read-only MCP tools; the operator drives the build pipeline through a five-stage CLI.
The rendered corpus also opens as an Obsidian-compatible vault:
per-paper cards live under cards/, per-community pages under
communities/, with dashboard.md / index.md / log.md at
the top. Wikilinks between cards drive Obsidian's graph view;
Dataview queries in the dashboard filter by tag / year / community.
The corpus type is a schema profile, not a category constraint. Nuthatch is designed for any structured-corpus problem where graph-augmented retrieval helps.
The graph-augmented retrieval space has working open-source implementations (Microsoft GraphRAG, graphify, LightRAG, HippoRAG, nano-graphrag) plus adjacent tools (Cognee, PaperQA2, Khoj, Verba). We surveyed them and built Nuthatch anyway because we wanted a specific combination of choices none of them makes:
- Community-aware retrieval, not just clustering. Every chunk
hit carries
community_id,community_path(the nested SBM chain), andcommunity_labelso an agent can route directly into the relevant cluster without paying for a card fetch first. Pluscommunity_searchranks communities semantically by query-to-centroid cosine; flat modularity-based clustering (Leiden, Louvain) cannot do this. Seedocs/Design_Decisions.md§ Community-aware retrieval. - Bayesian Stochastic Block Model as the principled clustering ceiling, implemented via graph-tool from Tiago Peixoto's lab (Professor of Complex Systems and Network Science, Goethe University Frankfurt). Leiden is a graceful fallback when graph-tool is unavailable. The SBM tier emits a nested hierarchy that flat methods cannot, and Nuthatch persists every level so agents can zoom from leaf clusters up to coarser super-clusters.
- Honest per-tool token-economy accounting (BM25 baseline for search, card-token-sum baseline for subgraph and community) instead of whole-corpus headline ratios.
- Build pipeline triggered out-of-band by the operator with quality gates between stages, never end-to-end unattended.
- Strict schema quarantine on ingest with reasoned per-file failure sidecars.
- Authoritative metadata fetched from publisher APIs (arxiv, bioRxiv) rather than parsed from PDF body text.
- User-extensible schema profiles supporting papers, patents, internal documents, and arbitrary user-defined types.
- A read-only MCP query surface that the agent cannot mutate.
See docs/HOWTO.md for the full operator guide
covering install, corpus initialisation, the five-stage build
pipeline, MCP registration per agent host, and stop / resume
semantics. The short version:
sfw uv tool install "nuthatch @ git+https://github.com/evoclock/nuthatch.git@main"
nuthatch init ~/my-corpus --register-as my-corpus --set-default
# Drop sources anywhere under the corpus root (inbox/, arxiv/,
# bioarxiv/, notes/, root-level: your choice). Then trigger each
# pipeline stage in sequence and inspect between stages.
scripts/ops/launch-stage.sh ingest my-corpus
scripts/ops/launch-stage.sh embed my-corpus
scripts/ops/launch-stage.sh graph my-corpus
scripts/ops/launch-stage.sh cluster my-corpus
nuthatch render --corpus my-corpus
# Then serve it to an MCP-aware agent:
nuthatch serve --corpus my-corpusA pre-built reference corpus (nuthatch-kb-demo) is available to explore before building your own.
Hosted LLM stages (OpenAI, Anthropic) read their API keys from your
operating-system credential store via Python keyring — never from
environment variables. Store each key once, before the first hosted
stage:
keyring set nuthatch openai-api-key
keyring set nuthatch anthropic-api-keyIf a key is missing, the pipeline fails closed with
openai credential error: absent_store_entry (plus the provisioning
hint above) rather than silently falling back. Other credential
failures map to closed kinds too: store_unavailable (no usable
keyring backend on this host), access_denied (locked or denied
keychain), invalid_credential (blank entry). Non-sensitive settings
such as OLLAMA_HOST and NUTHATCH_ACCELERATOR remain ordinary
environment configuration.
docs/Design_Decisions.md: architectural choices and the reasoning behind each, including the peer-landscape survey and the token-economy methodologydocs/HOWTO.md: operator runbook for the five-stage pipeline plus the MCP query surfacedocs/SPEC.md: the architecture spinedocs/extraction-benchmarks/ocr-comparison.md: evidence for the OCR routing decisions (Chandra vs Docling vs EasyOCR vs Granite vs SmolDocling across three representative scanned papers)- Per-agent skill files under
agent_skills/(skill-claude-code.md,skill-codex.md,skill-aider.md,skill-opencode.md,skill-pi.md,skill-hermes.md) for MCP registration details and recommended multi-step query flows
This section documents the exact tools and models used to build the reference corpus. It is here so others can replicate the setup or substitute equivalents at each stage.
Every source PDF goes through a cheap pre-flight pass with pdftotext
(poppler) before any extraction cost is paid. If the text layer is present
and dense enough the document is classified as born-digital; otherwise it
is routed to one of the OCR paths below.
| Input type | Condition | Extractor |
|---|---|---|
| Born-digital PDF | always | Docling (no OCR) |
| Scanned PDF | GPU available, max quality | Chandra OCR 2 |
| Scanned PDF | GPU available, not max quality | Granite-Docling 258M VLM |
| Scanned PDF | CPU-only | Docling + EasyOCR |
| Math span retry | per document, broken spans only | Chandra OCR 2 |
Math span retry. Post-extraction validation records every broken inline math span per document (position, broken-ratio, sample). For each document that has broken spans, all of its broken inline math spans are consolidated into a single page; Chandra OCR 2 resolves them in one pass against that page and the corrected expressions are traced back to their original positions. Only the span-level corrections are applied and the deferred record is closed. Math-span retry is a Chandra-specific capability; documents processed under the other extraction backends surface broken spans without the deferred-retry pass.
SmolDocling is documented as a fallback (small VLM, CPU-capable) but is not yet wired into the routing logic.
Nuthatch supports Chandra OCR as a backend because it produces the best results for research-grade math-aware extraction. Chandra-ocr's model weights are distributed under the OpenRAIL-M licence, which carries use restrictions that users adopt when they install Chandra. We surface this directly rather than treat the dependency as silent.
Chandra is therefore an opt-in extra, not a default dependency. To install Nuthatch with the Chandra backend:
sfw uv add "nuthatch[chandra]"Or, as a standalone tool install:
sfw uv tool install "nuthatch[chandra]"The sfw prefix routes the install through Socket Firewall's threat-
intel scan before any tarball lands on disk. We recommend it for
every package install regardless of source.
The dependency-licence compatibility holds at the package level: chandra-ocr the package is Apache 2.0; only the model weights downloaded at runtime carry OpenRAIL-M obligations, and those attach to whoever downloads them.
| Profile | Model | Notes |
|---|---|---|
scientific_paper |
SPECTER2 (AllenAI) | trained on the scientific citation graph; planned default for this profile |
| All other profiles | BGE-M3 (BAAI) | general-purpose multilingual; current default for all profiles |
Evaluation runs fully locally via Ollama except where noted.
| Role | Model | Source |
|---|---|---|
| RAGAS generator / answerer | gemini-3-flash-preview:cloud |
Ollama (local) |
| RAGAS judge / graph eval / cluster eval | granite3-dense:8b |
Ollama (local) |
| Retrieval embeddings (eval) | BGE-M3 | same checkpoint as ingest, via NuthatchEmbeddings RAGAS adapter |
| Cluster label relabelling | granite3-dense:8b |
Ollama (local), --relabel-llm pass |
All models listed as "Ollama (local)" run on-device with no outbound traffic.
Chandra OCR 2 and BGE-M3 likewise run locally; SPECTER2 can be served locally
via the HuggingFace transformers backend.
The full module graph and architecture diagrams are at
docs/architecture/.
SBM partition with a baseline subset of filters active. Communities coloured via Catppuccin Mocha; floating boxes (Communities, Filters) are draggable and persist position to localStorage. Interactive version lives at docs/graph.html in every nuthatch-published KB; open it in any browser, no backend.
Most relations toggled off and the cites relation off: the visible structure is the SBM community as the clustering inferred it from non-citation edges (authorship, topic / method co-mentions, embedding-driven adjacency).
Same filter set as above, but with cites toggled on. The new edges show how one paper citing another can extend a community beyond what semantic / co-mention signals alone would produce. The contrast between the previous frame and this one is the citation-extension effect made visible.
Per-tool token-economy report from token_econ_report. Each tool is measured against two honest baselines: BM25 (what a flat keyword search over the whole corpus would cost) for search tools, and card-token-sum (the cost of fetching every card) for subgraph and community tools. The ratio shows how much of the corpus a query actually touches. Whole-corpus headline ratios inflate the savings figure by comparing against a ceiling nobody would pay; Nuthatch compares against what a reasonable alternative would actually cost.
Full report after wiring counterfactuals for all community tools. The numbers are worth reading carefully.
What each ratio actually measures - and where to trust it:
The community_hierarchy (138×) and community_core_nodes (139×) figures are the cleanest in the report. Without nuthatch, finding a document's community path means loading and traversing the full communities.json index; getting the high-degree members of a community means loading every member card and ranking by degree. There is no cheaper alternative - these operations are intrinsically whole-index or whole-community reads. The ratios reflect that directly.
corpus_search (4×) is the most methodologically principled number in the report. Its counterfactual is BM25 over the same chunks at the same k - the only variable is whether the retrieval is semantic (dense) or lexical (sparse). It measures exactly the contribution of vector similarity over keyword matching, nothing more. A 4× reduction means the dense retrieval returns materially more relevant chunks per token than BM25 would for the same query budget.
community_get (21×) and community_brief (8×) are likewise honest. The community page bundles information that would otherwise require fetching every member card individually; the brief is a strict subset of the full page and the ratio is exact by construction.
community_search at 22,000× is a known upper bound and should be read as such. The counterfactual used is n_total_chunks × avg_chunk_tokens - the total token cost of scanning every chunk in the corpus to do brute-force cosine grouping by community. That ceiling is real but nobody would actually pay it: no agent working with a 123-paper corpus loads all 123 papers into its context window per query just to find relevant communities. A smarter fallback would be corpus_search(query, k=large), which already returns community_id metadata, at a cost of a few thousand tokens. The honest ratio for community_search is therefore somewhere in the 50-150× range. The 22,000× figure is preserved in the report because it is the correct answer to the specific counterfactual question posed ("what if you had no centroid index at all and had to read every chunk?"), but it is not a number to put in a headline. Some tools in this space make such claims without compunction; this project does not, because the whole point is to keep the numbers honest.
subgraph_extract at 0.57× is correctly negative and stays in the report. Depth-2 BFS on this corpus returns more tokens than the card-sum baseline it is compared against - it fans out too aggressively. The default depth has been changed to 1 and a max_nodes cap added; at depth 1 the ratio becomes positive, and the negative result from the earlier session is kept as a reminder that graph tools can inflate context just as easily as they compress it.
The build pipeline is operational. The five build stages
(ingest, embed, graph, cluster, render) chain into a
queryable corpus, plus three post-pipeline commands serve and
ship it: serve (the MCP server), publish (export a shareable
KB directory), and viz d3 (interactive force-atlas2 + Catppuccin
topology viz). Clustering supports an optional --relabel-llm
pass that replaces the heuristic word-frequency labels with
topical 2-4 word names produced by an LLM (default
granite3-dense:8b via Ollama).
The MCP server exposes nine read-only tools: corpus_search,
subgraph_extract, card_get, community_get, community_brief,
community_search, community_core_nodes, community_hierarchy,
and token_econ_report. The community tools, plus the routing
keys (community_id, community_path, community_label)
carried inside every search hit and every card's frontmatter,
are what make graph-RAG actually work without per-card lookups.
The test suite covers every contract (routing thresholds, model
defaults, chunk coverage, reranker invocation, schema validation,
end-to-end pipeline integration, suffix-rename invariants,
publish-side canonical promotion, --relabel-llm round-trip with
mock LLM). PyPI publication is in progress.
Planned work, not yet landed:
- Air-gapped / corporate environments. A Dockerfile + container entrypoint so the published KB can be lifted into enterprise DevOps environments (AWS, GCP, Azure DevOps, and similar internal container runtimes) and queried via MCP without any outbound traffic. The published KB is already a self-contained bundle (markdown cards + JSON indexes + chroma archive); this task wraps it in a runtime so an internal deployment can reach a full MCP server inside the firewall.
- Routing hardening for heterogeneous inputs. The current ingest pipeline routes by filename profile (arXiv preprint, bioRxiv, internal doc, patent); next is per-content-type detection and per-profile thresholding so mixed corpora (scanned PDFs + born-digital papers + plain-text notes) route cleanly without manual triage.
- Docling as the general-purpose extraction default. Docling
becomes the default extractor for born-digital text, with
SmolDocling and EasyOCR as the documented fallbacks per the
OCR benchmark (
docs/extraction-benchmarks/ocr-comparison.md). - SPECTER2 as the scientific-paper embedding default. The
current default embedding model is
BAAI/bge-m3(general purpose, all profiles). For thescientific_paperprofile, SPECTER2 (AllenAI; trained on the scientific-citation graph and identified during the PhD knowledge-base build as superior to general-purpose embeddings for paper retrieval) becomes the documented default. All other profiles keep BGE-M3.
GNU Affero General Public License v3 only (AGPL-3.0-only) plus a Section 7(b)
author-attribution clause. See LICENSE for the full text.
The plain-English version:
- If you are using Nuthatch in an open-source project: you are asked to credit Julen Gamboa as the original author in your README or equivalent primary documentation, with a link back to this repository. It's a courtesy ask; the project will not chase you to the ends of the earth to enforce it. The AGPL terms still apply (source disclosure on conveyance and network use), and that is the part with real teeth. We support genuine open-source use without friction.
- Commercial use, including forks and substantial modifications, is permitted under the AGPL when all AGPL obligations and the Section 7(b) attribution requirements are followed. This includes offering covered source to network users as required by Section 13. A separate commercial licence is required only when an organisation wants proprietary modifications, alternative attribution terms, or otherwise cannot or does not wish to comply with those obligations. Contact the author for details; pricing is flexible and case-by-case.
- The split exists because we do not have a problem with open-source contributors and we do have a problem with the pattern of enterprises that exploit open-source projects without contributing back. AGPLv3 plus a commercial offering is the standard, OSI-approved pattern (Nextcloud, Plausible, Cal.com, iText) for distinguishing the two populations cleanly.
A commercial licence template will be published in this repository
at COMMERCIAL.md once finalised. Until then, reach out directly.
If you saw this repository's commit log, the licence moved from Apache 2.0 (initial release) to MIT plus a Commercial Attribution Rider (0.0.2) to AGPLv3 plus §7(b) (0.0.3) over two days. Faster than is comfortable, worth explaining once instead of leaving as a thread on the issue tracker.
Each step sharpened the same question: what mechanism actually distinguishes open-source use from commercial use cleanly, without putting friction on either side?
Apache 2.0 was the default and asked nothing of either side, which left no leverage for the commercial-fair-use case. The MIT + Rider asked for attribution but was non-OSI (which scares off the compliant commercial adopters, the ones who would have respected the ask anyway) and was a social-pressure mechanism with weak enforcement teeth against the rest. AGPLv3 + §7(b) + a separate commercial licence is the standard dual-licence pattern used by Nextcloud, Plausible, Cal.com, and iText. AGPLv3 is OSI-approved so genuine open-source adoption sees no friction; Section 13's network-use source-disclosure obligation is the structural lever that gives the licence real teeth; Section 7(b) preserves author attribution as an enforceable term rather than a courtesy ask. The commercial licence is for adopters who cannot or do not want to comply with AGPL, priced case-by-case rather than triggered by a revenue threshold.
The intent is plain. Open-source use should be friction-free. Larger commercial adopters should either contribute back through AGPL source disclosure or pay through the commercial licence. The previous two attempts got most of the way there; this one settles it.
Designed and built by Julen Gamboa, who drove system design, implementation, agent orchestration, and code review. Claude Code and Hermes operated as spec-driven agents, executing implementation tasks under that direction.
Specs and decisions are pinned in docs/SPEC.md and
docs/Design_Decisions.md so every implementation points back
to a spec entry.
The SBM clustering tier is built on graph-tool, developed by Tiago Peixoto (Center for Critical Computational Studies, Goethe University Frankfurt). The nested SBM hierarchy and the principled model-selection criterion it provides are the reason Nuthatch can offer a clustering ceiling that goes beyond what flat modularity methods produce.