distill

Document chunking, estimated token counting, and fact-preserving distillation CLI for LLM processing. Uses cl100k_base for remote-profile preflight estimates and Wormhole provider integrations for embeddings and distillation.

Install

distill uses the pinned remote reliquary chunking engine, so no sibling checkout is required. Build both binaries directly:

# Build both binaries
go build -o distill .
go build -o distill-eval ./cmd/distill-eval
ln -sf "$(pwd)/distill" ~/go/bin/distill
ln -sf "$(pwd)/distill-eval" ~/go/bin/distill-eval

# …or just: build + symlink both
just install

Wormhole, the LLM SDK, is also a pinned remote dependency and needs no local checkout.

Commands

distill ships as two binaries that share the same ~/.config/distill/config.yaml (models, base URL, per-role picks):

distill — the document pipeline:

• count — Count tokens, characters, and lines.
• chunk — Split documents into chunks (all reliquary strategies, inspect, JSONL streaming, glob batches).
• digest — Distill documents into a fact-preserving article; includes the deterministic digest score self-check.
• serve — Serve the chunking HTTP API (POST /chunk, POST /chunk/batch, GET /health).

distill-eval — evaluation & model benchmarking:

• eval — Score extractions: eval judge (LLM precision/recall/F1), eval facts (deterministic golden-fact scoring, no LLM), and eval structured (schema-backed JSON scoring, no LLM).
• grade — Grade digests on merit via the pairwise judge (calibrate, tournament, recognize, panel). (Was distill digest grade.)
• models — Rank models on a task: models rankings (public leaderboards), models code, models trace-go, models comedy, models label.

Old distill eval / distill models / distill digest grade invocations still resolve, but print a one-line pointer to the new distill-eval path.

count

Count tokens, characters, and lines in a file or stdin.

# From file
distill count document.md

# From stdin
cat document.md | distill count

# Plain text output
distill count --format plain document.md

Output (JSON by default):

{"tokens":1234,"chars":5678,"lines":42}

chunk

Split a document into chunks. Three legacy modes keep their original pipelines; any reliquary strategy name also works as a mode and runs the chunk API engine.

# Default: split at markdown headings
distill chunk document.md

# Dense sentence packing
distill chunk --mode cramit --max-tokens 2000 document.md

# Semantic splitting through a Wormhole provider
export OPENROUTER_API_KEY=sk-or-...
distill chunk --mode semantic --embedding-model qwen/qwen3-embedding-8b --threshold 0.3 document.md

# Any reliquary strategy (chunk API engine)
distill chunk --mode markdown_aware document.md
distill chunk --mode recursive --separator-profile cjk_thai document.md
distill chunk --mode token_based --token-encoding o200k_base --max-tokens 512 document.md

# Inspect (no files written): stats + sample chunks
cat document.md | distill chunk --mode sentence_boundary --inspect --inspect-format human -

# Streaming NDJSON to stdout (metadata line + one line per chunk)
cat document.md | distill chunk --mode word_boundary --format jsonl -

# Batch: every glob match becomes its own generation directory
distill chunk --glob "docs/*.md" --out-dir ./chunk-generations

# Custom output directory
distill chunk --out-dir ./chunks document.md

Writes numbered .md files and a manifest.json as one complete generation. An explicit --out-dir must be absent; Distill builds the generation in a sibling staging directory and publishes it atomically, so a failed run does not leave partial chunks or replace an existing path.

Modes

Mode	Strategy	Requires
headings	Splits at markdown headings (H1-H6), recurses into sub-headings when a section exceeds max tokens	Nothing
semantic	Embeds paragraphs via a built-in Wormhole provider, splits at low cosine-similarity boundaries. Embeddings are cached on disk (keyed by model) so repeated runs skip re-embedding.	provider API key, --embedding-model
cramit	Greedy sentence packing to fill chunks to max tokens	Nothing
(strategy)	word_boundary, sentence_boundary, hard_cut, paragraph_aware, token_based, smart_boundary, markdown_aware, heading_aware, recursive run the chunk API engine with chunker's semantics (--max-tokens is the token budget; base strategies derive a character budget at ~4 chars/token, token_based consumes it in tokens)	--token-encoding for token_based

Modes fall back to dense boundary packing when content can't be split further by their native strategy. The chunk API engine's semantic path is strict: a mid-run embedding failure fails the run instead of emitting fallback chunks labeled semantic.

Flags

Flag	Default	Description
--mode	headings	Chunking strategy
--max-tokens	4000	cl100k_base preflight budget per chunk; local Qwen mode uses only the derived character budget
--overlap	200	Tokens of overlap between chunks
--out-dir	temp dir	Output directory; must be absent
--threshold	0.3	Similarity threshold (semantic mode)
--embedding-model	$DISTILL_EMBEDDING_MODEL	Embedding model for semantic mode (default qwen/qwen3-embedding-8b on OpenRouter)
--local	—	Use the on-box Qwen embedding profile (local_* config keys) instead of the OpenRouter default
--min-chunk-chars / --max-chunk-chars	engine default	Semantic chunk size bounds
--smoothing-window / --coherence-window	engine default	Semantic boundary-detection tuning
--format	json	json prints the manifest after writing files; jsonl streams NDJSON chunks to stdout without writing files
--separator-profile	default_text	Recursive-mode separator cascade (cjk_thai for CJK/Thai punctuation)
--token-encoding	cl100k_base	Encoding for token_based mode and --token-ceiling (cl100k_base, o200k_base, p50k_base, r50k_base)
--token-ceiling	0 (off)	Post-strategy cap on tokens per chunk on any strategy; oversized chunks split sentence → word → hard cut
--inspect / --inspect-format / --sample	off / ndjson / 2	Preview chunk stats and samples on stdout; writes no files
--glob	—	Batch-chunk every match into --out-dir/<name>/ (requires --out-dir)
--include-embeddings	off	Attach embedding vectors to every chunk (--format jsonl only)

Manifest

The manifest.json tracks chunk metadata:

{
  "source": "document.md",
  "mode": "headings",
  "total_chunks": 3,
  "total_tokens": 8500,
  "chunks": [
    {"file": "001.md", "tokens": 3200, "start_line": 1, "end_line": 45},
    {"file": "002.md", "tokens": 2800, "start_line": 44, "end_line": 89, "start_char": 4100, "end_char": 8150, "content_hash": "a1b2c3d4e5f60718"},
    {"file": "003.md", "tokens": 2500, "start_line": 88, "end_line": 120}
  ]
}

Lines overlap between chunks when --overlap is set. Chunk-api strategy modes additionally record start_char/end_char rune offsets into the chunked source, content_hash (first 16 hex chars of SHA-256 over the chunk text), path (heading breadcrumb for heading_aware), and metadata strategy annotations (e.g. fenced-code language for markdown_aware). Older manifests without these fields still read.

serve

Serve the chunking HTTP API on 127.0.0.1:8080 (or --port / $PORT):

distill serve --embedding-model text-embedding-3-small   # enables semantic

Endpoints: POST /chunk, POST /chunk/batch (10 MiB request cap), GET /health. Full contract: docs/api with an OpenAPI 3.1 spec. All chunking logic — validation, dispatch, strict semantic, token ceilings, batch envelopes — lives in reliquary/chunkservice; distill adds the HTTP transport, provider wiring, and CLI (HTTP contract tests in internal/chunkapi). This surface replaces the retired standalone chunker service.

Transcript cleaning

digest and chunk auto-detect YouTube/auto-caption transcript input (WebVTT or SubRip) and clean it before processing: the WEBVTT header, cue numbers, timing lines, inline <HH:MM:SS.mmm> timestamps, <c> styling tags, and [Music]/[Applause] bracket labels are stripped; the auto-caption triplication / rolling-overlap duplication is deduplicated; and the ~5-word fragments are reflowed into paragraphs (split on timing gaps, else a fixed line window). Cleaning is deterministic and offline — no LLM, no network.

Detection never false-positives on ordinary markdown. Override with --no-clean (process raw) or --clean (force). count always counts the raw source and is never cleaned.

digest

Distill a long document into a fact-preserving article. Rather than asking one model call to summarize the whole source, digest runs a four-role pipeline — research (extract atomic facts from each chunk), fuse (merge the notes), write (compose a narrative draft in a target style), then edit (polish it) — so no single call ever holds the entire document.

When a directory input contains manifest.json, digest reads only the listed chunk files in manifest order and ignores unlisted Markdown files. An invalid manifest fails the command. Directories without a manifest retain the existing top-level *.md behavior.

export OPENROUTER_API_KEY=sk-or-...   # or OPENAI_API_KEY; omit only with --local against a no-auth endpoint
distill digest document.md \
  --style "concise technical brief" \
  --out document.distilled.md

# Reuse the compiled-facts checkpoint for a free style-only re-run
distill digest document.md --reuse-facts --facts artifacts/facts.compiled.md

# Run against the on-box Qwen profile instead of the OpenRouter default
distill digest document.md --local

# Force the whole run directly against DeepSeek
export DEEPSEEK_API_KEY=sk-...
distill digest document.md --deepseek

Uses Wormhole provider integrations and a text model. An explicit --model or $DISTILL_MODEL pins every text role for the command, including the precision judge and cascade escalation; otherwise per-role *_model config keys select stages (default z-ai/glm-5.2). Non-prefixed remote models use OpenRouter; DeepSeek, Z.AI/GLM, and Gemini model IDs route directly to their built-in Wormhole providers. Remote custom API endpoints are disabled. Pass --local to switch to the on-box Qwen profile (http://127.0.0.1:8000/v1), where --base-url / $DISTILL_BASE_URL may point at a local OpenAI-compatible server, or --deepseek to force the whole run to the direct DeepSeek profile (deepseek_model / deepseek_base_url). Direct DeepSeek calls normalize OpenRouter-style slugs such as deepseek/deepseek-v4.1-flash to deepseek-v4.1-flash, send thinking: disabled by default for normal completions, and include a stable hashed user_id for digest runs so DeepSeek can isolate/cache related requests without exposing local paths. See docs/deepseek-provider.md for the request shape, cache probe, and maintenance checks.

For the current high-quality long-document recipe, see docs/polished-mode.md. It combines cascade extraction, fact merging, cluster-derived outlines, temporary citations, repair, precision checks, and quality gates using existing digest flags. Cascade requires an explicit research_escalation_model because its committed default is blank; an all-role --model / $DISTILL_MODEL pin is the per-command alternative.

Outputs: the final article (--out, default <source>.distilled.md), the facts.compiled.md checkpoint, the fused notes (facts.fused.md, only when --fuse is enabled), the versioned run-ledger.jsonl, an atomic run-summary.json, and provenance-bound per-chunk artifacts (chunks/, responses/, including the writer's draft.md) under the artifacts directory. The durable output and responses/rewrite.md are published only after every mandatory stage succeeds. The research, fuse, write, and editor prompts live in ~/.config/distill/prompts/ and are created on first run — edit them to tune behavior.

digest flags

Flag	Default	Description
--model	$DISTILL_MODEL	Pins every text role and disables cross-provider fallback; otherwise per-role config selects stages (default z-ai/glm-5.2)
--base-url	local profile only	Local OpenAI-compatible endpoint when --local is set; remote custom endpoints are disabled
--local	—	Use the on-box Qwen profile (local_* config keys) instead of the OpenRouter default
--deepseek	—	Force the direct DeepSeek profile (deepseek_* config keys) with $DEEPSEEK_API_KEY
--style	narrative prose	Writer style; a preset name (narrative/brief/faq/reference) or free text
--context	—	Free-text guidance to steer the rewrite's emphasis/framing (injected into outline/section/edit prompts, never extraction)
--context-file	—	Read steering context from a file (mutually exclusive with --context)
--out	<source>.distilled.md	Final article output path
--facts	<artifacts>/facts.compiled.md	Compiled-facts checkpoint path
--artifacts	temp dir	Absent/new, empty, or exact schema-v2 provenance-bound directory only; v1, unbound, or mismatched directories are preserved and refused
--chunk-size	6000	Character budget per chunk (min 1000)
--max-tokens	4000	cl100k_base preflight budget per chunk for remote profiles; local Qwen uses the character budget (0 disables)
--concurrency	4	Max parallel chunk extractions
--no-clean / --clean	auto	Skip / force transcript (VTT/SRT) cleaning before distillation
--timeout	300	Per-LLM-call timeout in seconds
--retries	config, else 3	Maximum primary attempts per logical digest call, including the first; only transient/rate-limit/timeout/empty failures retry
--dry-run	false	Validate and display the exact packed call plan, routing, recovery availability, and paths with zero provider calls and no absent artifact-directory creation
--max-calls	0	Authoritative mandatory-call preflight gate and hard runtime provider-call ceiling (0 unlimited); recovery cannot consume later mandatory reservations. Finite planning refuses dynamically sized merge, target-facts, and precision work.
--reuse-facts	false	Reuse the facts checkpoint (skip research)
--resume	true	Reuse only schema-v2 checkpoints whose response sidecars match source, prompt, upstream input, route, parameters, policy, and stage
--fuse	false	Run the fuse stage that merges per-chunk notes before writing (off by default; can time out on large inputs)
--merge-facts	false	Cluster and merge similar extracted fact bullets before planning; near-duplicates with explicit fact markers merge only when source fact IDs remain represented
--outline-from-clusters	false	Build the outline from merged fact clusters instead of a free-form outline call (requires --merge-facts)
--max-sections	config max_sections	Section cap for cluster-outline mode; stub clusters are merged first and oversized separable clusters may split, so section sizes are balanced but not guaranteed equal
--no-edit	false	Skip the editor stage (output the writer's draft)
--appendix	false	Append the verbatim extracted facts as a lossless appendix (recovers tables/ranked lists the prose stage samples away)

eval judge

Score one or more candidate extractions against a reference, chunk by chunk, using an LLM judge (the source chunk is ground truth; the reference is a coverage checklist). Before any judge call or output write, Distill requires source, reference, and every candidate to have the same readable chunk-*.md set. Useful for comparing how well different models extract facts in the digest pipeline.

# Compare two models' extractions against a trusted reference
distill-eval eval judge \
  --chunks      runA/chunks \
  --reference   runA/responses \
  --candidates  runB/responses,runC/responses \
  --judge-model gpt-4o \
  --out         evals/

Inputs are the artifact directories distill digest writes (chunks/, responses/). Outputs under <out>/evaluations/:

• <candidate>/judgments.jsonl — per-chunk verdicts (SUPPORTED / CONTRADICTED / NOT_IN_SOURCE) plus missed reference facts
• <candidate>/summary.md — per-chunk and aggregate precision/recall/F1 (micro-average)
• INDEX.md — all candidates ranked by F1

The judge model resolves from --judge-model, then $DISTILL_MODEL, then config. DeepSeek, Z.AI/GLM, and Gemini judge model IDs use their direct Wormhole providers; --base-url is local-only.

eval structured

Score one or more JSON extraction candidates against a JSON Schema and gold JSON file, deterministically and without an LLM. Structural compliance is a hard gate: invalid JSON, type mismatches, missing required fields, and disallowed extra fields zero the aggregate scores while still writing field-level outcomes when possible.

distill-eval eval structured \
  --schema     schema.json \
  --gold       gold.json \
  --candidates model-a.json,model-b.json \
  --out        structured-evals/

Supported schema evaluation metrics include exact strings, case-insensitive strings, fuzzy strings, URL-normalized strings, exact/tolerant numbers, exact integers, booleans, and exact array item matching with matched/missed/spurious counts. Outputs include INDEX.md, <candidate>.summary.md, and <candidate>.report.json.

models trace-go

Rank models by exact stdout prediction on deterministic, trusted single-file Go programs. The fixture programs are executed locally to verify or generate gold; model output is never executed. Model calls use greedy decoding (temperature=0) and the scorer compares the last non-empty model output line to byte-exact stdout gold. Console progress prints each task as it completes; HTML reports include per-task predictions and errors.

distill-eval models trace-go \
  --models local \
  --base-url http://127.0.0.1:8000/v1 \
  --only int_div_floor,map_stable \
  --skip map_stable \
  --task-timeout 30 \
  --out docs/go-trace-routing.html

Use only trusted fixtures. The default fixture is testdata/trace-go/tasks.json; override it with --fixtures. Use --only and --skip with comma-separated task IDs to isolate slow or suspicious cases.