Semidex

Search your documents, inspect the evidence, and give AI agents a navigable knowledge base.

Semidex is a local-first document retrieval and grounded-answering system. It indexes files into Qdrant, combines semantic and lexical search, and preserves source context so people, applications, and AI agents can check what an answer is based on.

This README covers Full Semidex, run from source: the indexer, Admin UI, read-only MCP server, and HTTP APIs, with local and cloud provider options. For the separate cloud-oriented npm package, see Semidex Lite.

Get started · Capabilities · Architecture · Documentation · Testing

What you can do

• Build a document knowledge base. Index Markdown, plain text, PDFs, and supported Pandoc-converted formats; reuse unchanged files on subsequent runs.
• Keep Markdown structure. Navigate sections and recover original tables, code blocks, and checklists through skeleton-first indexing.
• Find evidence across languages. Combine dense and sparse retrieval with reciprocal rank fusion, source filters, optional tags, and MCP reranking.
• Read beyond a search snippet. Fetch neighboring chunks or bounded section and file context, with source identity and pagination.
• Connect agents and applications. Use MCP for agent-directed retrieval, the Search API for evidence, or Ask for streamed answers with citations.
• Operate locally. Manage collections, indexing jobs, settings, search, and document inspection through the Admin UI. Local ONNX embeddings and Ollama generation are available alongside Qdrant Cloud Inference and Gemini.

Full Semidex is an experimental MVP under active development. Markdown has the strongest structural support; other formats use a legacy chunking path. OCR and image understanding are not implemented. The Admin UI is an operator interface, and Agent API v3 is experimental. See the capability guide for boundaries and prerequisites.

Quick start

You need Git, Node.js 20.16+, npm, and a reachable Qdrant instance. The example below uses local Qdrant and local BGE-M3 ONNX embeddings. Docker is needed only if you use the container command. Ollama is optional for this Markdown indexing path.

1. Install from source

git clone https://github.com/CodeNoob53/semidex.git
cd semidex
npm ci

Copy .env.example to .env (Copy-Item .env.example .env in PowerShell or cp .env.example .env in a POSIX shell). For this local setup, set:

QDRANT_URL=http://127.0.0.1:6333
QDRANT_KEY=
ONNX_EMBED=1

If you do not already have Qdrant running:

docker run -d --name semidex-qdrant -p 127.0.0.1:6333:6333 -v semidex-qdrant-data:/qdrant/storage qdrant/qdrant

The first ONNX embedding operation downloads approximately 2.3 GB of model files and caches them for later runs. See installation for Qdrant Cloud, Ollama, and Windows GPU setup.

2. Index your documents

PowerShell:

$env:COLLECTION = 'my-docs'
npm run index -- .\docs\en

Bash/zsh:

COLLECTION=my-docs npm run index -- ./docs/en

Replace the example folder with your own Markdown documentation. Always set COLLECTION. Keep one compatible embedding profile per collection; provider, model, or schema changes can require reindexing.

3. Open the Admin UI

npm run admin:build
npm run admin

Open http://127.0.0.1:8642 to browse your collection, search, and inspect source evidence. Ask additionally needs a configured generation provider; see configuration.

Use npm run doctor to diagnose environment issues and npm run smoke for offline checks.

Connect an agent or application

Interface	Use it for	Contract
MCP over stdio	Let an external AI agent search and navigate your indexed documents	Read-only tools; launch with npm run mcp
POST /api/v1/search	Retrieve evidence from an application backend	Bearer key with search scope
POST /api/v1/ask	Ask a grounded question with streamed output	Bearer key with generate scope
POST /api/v2/ask	Grounded conversation with caller-supplied history	Bearer key with generate scope; caller owns history
POST /api/v3/ask	Run one experimental model step with caller-defined tools	Bearer key with agent scope; caller executes tools

For MCP clients, configure node as the command and the absolute path to src/mcp/server.js as its argument. The server reads the repository's .env. The MCP guide describes discovery, search, and bounded context retrieval.

For HTTP integration, create a collection-scoped key in the same Full Semidex environment as the Admin server:

npm run key -- add --name my-app --collection my-docs --operation search --operation generate

Keep the returned token in your backend and send it as Authorization: Bearer <token>. Integration routes require configured keys; the Admin server binds to loopback by default, and Admin routes do not use these keys. Key scopes, rate limits, and Ask token budgets are documented in the capability guide. Agent v3 has a separate API contract and does not inherit Ask's retrieval and citation guarantees.

How it works

flowchart LR
    D[Documents] --> I[Parse and chunk]
    I --> E[Embed content]
    I --> N[Markdown structure and navigation]
    E --> Q[(Qdrant)]
    N --> Q
    Q --> R[Hybrid retrieval and bounded evidence]
    R --> M[Read-only MCP tools]
    R --> U[Admin UI and Search API]
    R --> A[Ask v1/v2 and generation]

Loading

The Full composition supplies local and cloud capabilities to shared indexing, storage, retrieval, and Admin modules. Collection embedding profiles keep indexing and query embeddings compatible. Markdown navigation nodes help find the right section; retrieved source content supplies the evidence.

See architecture for runtime boundaries and data flow, and project structure for the source map.

Development and verification

npm run admin:build
npm test
npm run smoke

These are the repository's core CI checks. npm test first type-checks the Lite client contracts, then runs the shared Full/Lite unit suite with bounded concurrency. Offline tests, live integration tests, and retrieval benchmarks serve different purposes; passing one does not establish the others.

The test base covers indexing and structural chunking, retrieval and evidence assembly, MCP, Admin UI and API behavior, provider composition, Ask and agent contracts, and security boundaries. Retrieval evaluation includes internal corpora and external benchmark harnesses. Results apply to their recorded dataset, provider, and settings, rather than proving universal RAG quality.

Read testing for commands and dependencies, and benchmarking for datasets, metrics, and result interpretation. For UI development, run npm run admin and npm run admin:dev in separate terminals.

Documentation

Guide	What it covers
Installation	Source setup, providers, model downloads, and diagnostics
Capabilities	Shipped Full features, optional dependencies, and limitations
Architecture	Runtime composition, indexing, storage, and API boundaries
Retrieval	Hybrid search, profiles, reranking, and evidence
MCP tools	Tool reference and agent workflows
Configuration	Settings, embedding models, generation, and hardware
Operations	Maintenance, synchronization, backfills, and troubleshooting
Testing	Test suites, fixtures, CI, and live checks
Benchmarking	Evaluation datasets, commands, and recorded results

The documentation index links to additional references and design notes. Planned work is tracked separately in the roadmap.

License

Semidex is available under the MIT License.

Developed with AI assistance from Claude and OpenAI Codex. Product direction, architecture, review, testing, and final responsibility remain with the author.