Chunker

RETIRED — superseded by distill (v0.12.0+). The chunker service, CLI, inspection modes, and metadata are consolidated into distill, powered by reliquary. chunker serve → distill serve; chunker inspect → distill chunk --inspect; chunker -format jsonl → distill chunk --format jsonl; chunker files "docs/*.md" -out-dir out → distill chunk --glob "docs/*.md" --out-dir out (per-file generation directories with manifests); -strategy X → --mode X. See ../distill/docs/api/API.md for the migration map. This module is frozen; new chunking work lands in reliquary and new front-ends in distill.

A high-performance text chunking service and CLI tool written in Go that intelligently splits text into manageable chunks using boundary-aware sentence detection and token counting.

Features

• Dual Mode Operation: Run as HTTP API server or CLI tool
• Smart Chunking Strategies:
  • smart_boundary (default): delegated to github.com/dotcommander/reliquary (abbreviation-aware, punctuation-based sentence detection)
  • word_boundary: Splits at word boundaries
  • sentence_boundary: Splits at sentence endings
  • hard_cut: Exact character count splits
  • paragraph_aware: Keeps paragraphs together
  • markdown_aware: Preserves markdown headings and fenced code blocks
  • heading_aware: Splits on markdown headings and carries a path breadcrumb per chunk
  • recursive: Cascade splits — paragraphs, then lines, then words, then hard cuts (LangChain-style); separator_profile: cjk_thai adds CJK/Thai punctuation separators for scripts without whitespace word boundaries
  • semantic: Topic-boundary chunking via embedding similarity — needs an embedding provider in the config file (OpenAI or Ollama); falls back to smart_boundary when the embedder fails mid-request, returns a clear 400 when no provider is configured
  • token_based: Token counting for LLM compatibility
• Inspect Mode: Preview chunking stats and sample chunks without full payload output
• Batch Files Mode: Chunk many files from a glob pattern into an output directory
• Token Encoding Support: cl100k_base, o200k_base, p50k_base, r50k_base
• Token ceiling on every strategy: max_tokens (or -max-tokens) splits chunks that exceed the limit (sentence → word → hard cut) while keeping the base strategy semantic; per-chunk token_count and metadata.max_tokens report the applied ceiling
• Richer chunk payloads: content_hash (16-hex SHA-256) on every chunk, heading path breadcrumbs (heading_aware), and strategy metadata such as language/line_count for fenced code blocks (markdown_aware)
• Configurable overlap between chunks for context preservation
• Multiple output formats: JSON, JSON Lines (JSONL)
• Reliquary integration for maintained chunking algorithms and token support

Installation

From source

git clone https://github.com/dotcommander/chunker
cd chunker
just build

# Optional: Install globally
sudo cp bin/chunker /usr/local/bin/
# Or symlink to your Go bin
ln -s $(pwd)/bin/chunker ~/go/bin/chunker

Using go install

go install github.com/dotcommander/chunker/cmd/chunker@latest

Usage

CLI Mode

Chunker automatically detects when text is piped to stdin:

# Basic usage with smart defaults
cat document.txt | chunker

# Token-based chunking for LLM processing
cat article.md | chunker -size 2000 -strategy token_based -encoding cl100k_base

# Output as JSON Lines for streaming
cat book.txt | chunker -format jsonl | while read line; do
  echo "$line" | jq .
done

# Custom settings with pretty output
echo "Your text here..." | chunker -size 500 -overlap 50 -pretty

# Show help
chunker

# Inspect chunking quality quickly (NDJSON by default)
cat document.txt | chunker inspect -sample 3

# Human-readable inspect output
cat document.txt | chunker inspect -inspect-format human -sample 3

Server Mode

Run as an HTTP API server:

# Start server on loopback using the configured port (127.0.0.1:8080 by default)
chunker serve

# Expose on all interfaces with a custom port
chunker serve -bind 0.0.0.0 -port 3000

# Or use environment variable
PORT=3000 chunker serve

Files Mode

Chunk multiple files by glob and write output files:

# Write one output file per input file (JSON by default)
chunker files -out-dir chunks "documents/*.md"

# Write JSONL output files
chunker files -format jsonl -out-dir chunks "documents/*.txt"

CLI Options

-size       Chunk size in characters or tokens (default: 4000)
-strategy   Chunking strategy (default: smart_boundary)
            Options: smart_boundary, word_boundary, sentence_boundary,
                    hard_cut, paragraph_aware, markdown_aware,
                    heading_aware, recursive, token_based, semantic
                    (semantic needs an embedding provider — see below)
-overlap    Overlap between chunks (default: 200)
-encoding   Token encoding for token_based strategy (default: cl100k_base)
            Options: cl100k_base, o200k_base, p50k_base, r50k_base
-max-tokens Cap tokens per chunk on any strategy (default: 0 = no cap);
            oversized chunks split at sentence, then word, then hard cut
-format     Output format: json, jsonl (default: json)
-pretty     Pretty print JSON output
-include-embeddings  Attach an embedding vector to every chunk
            (needs an embedding provider — see below)

Server mode:
chunker serve [-bind address] [-port port]

Inspect mode:
chunker inspect [-sample 2] [-inspect-format ndjson|human]

Files mode:
chunker files [-out-dir chunks] "<glob>"

Semantic chunking is enabled via ~/.config/chunker/config.yaml (see API.md for the request-side options and fallback semantics):

embedding:
  provider: ollama              # or openai; empty disables semantic
  base_url: http://localhost:11434
  model: nomic-embed-text
  # openai example: provider: openai, model: text-embedding-3-small,
  # dimensions: 1536, api_key_env: OPENAI_API_KEY
  timeout_ms: 30000
  max_batch: 64

Task Runner Migration

This project now uses just instead of Task.

# Before
task build
task test
task lint
task server

# Now
just build
just test
just lint
just server

API Reference

See API.md for complete HTTP API documentation — POST /chunk, POST /chunk/batch (many texts, one shared parameter set, per-item results), and GET /health. A machine-readable contract lives in openapi.yaml.

Quick Example

curl -X POST http://localhost:8080/chunk \
  -H "Content-Type: application/json" \
  -d '{
    "text": "Your text here...",
    "chunk_size": 1000,
    "strategy": "smart_boundary",
    "overlap": 100
  }'

Examples

Pipeline with LLM Tools

# Chunk and process with an LLM tool
cat research_paper.pdf | pdftotext -layout - | chunker -size 3000 -strategy smart_boundary | jq -r '.chunks[].text' | llm prompt "Summarize this section"

# Process code files with token limits
find . -name "*.go" -exec cat {} \; | chunker -size 4000 -strategy token_based -encoding cl100k_base -format jsonl

Batch Processing

#!/bin/bash
# Process multiple files
for file in documents/*.txt; do
  cat "$file" | chunker -size 2000 | jq -c '.chunks[]' > "chunks/$(basename "$file" .txt).jsonl"
done

Architecture

The project follows clean architecture principles:

chunker/
├── cmd/chunker/        # CLI and server entry point
├── internal/
│   ├── domain/         # Core types and interfaces
│   ├── service/        # Business logic
│   └── handler/        # HTTP handlers
├── pkg/chunking/       # Chunking strategies
└── API.md             # API documentation

Performance

• Processes large texts efficiently with streaming support
• Memory usage scales linearly with text size
• All strategies run in O(n) time complexity
• Concurrent request handling in server mode

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

License

MIT License - see LICENSE file for details