NullRun

Ship AI agents with real-time budget, policy, and human-approval gates.

Zero-refactor cost control, tool policy enforcement, and audit trail for any LLM-powered agent - works with OpenAI, Anthropic, LangGraph, CrewAI, AutoGen, LlamaIndex, and your own stack.

Quickstart · Docs · Examples

---

⚠️ Status: alpha (v0.17.1). The public API may shift between minor versions. Pin your dependency and read the CHANGELOG before upgrading.

---

Why NullRun?

AI agents can overspend, call dangerous tools, and act without audit trails. Existing observability tools tell you after the fact. NullRun enforces before the action.

Without NullRun	With NullRun
Agent calls gpt-4o 10,000 times → surprise $5,000 invoice	Hard budget cap → SDK blocks at 402 before invocation
Agent runs bash rm -rf /	Tool policy → SDK blocks at 403 before execution
Sensitive action with no human in the loop	Approval flow → SDK pauses and waits for WS approval_resolved push
Cost & calls scattered across 4 libraries	Single source of truth: per-org, per-workflow, per-execution
Runaway SDK loop calling /gate without /track	Per-reservation rate cap → 402 budget error (see docs/errors/NR-R001.md)

---

Features

Hard & soft budget gates — atomic Redis-enforced	Tool policy enforcement — block dangerous tools before execution
Human-in-the-loop approvals — pause agent and await approval_resolved via WS push	Immutable audit trail — every decision, every tool call, every cent
Zero-code instrumentation — nullrun.init() patches httpx once for any vendor	LangGraph, CrewAI, AutoGen, LlamaIndex — first-class integrations
Memory-safe streaming — 16 MiB response body; full body for usage extraction	Lightweight — no LLM-key storage, no proxy required
Server-authoritative cost — server-minted execution IDs	MCP support — expose tools to agents via Model Context Protocol

---

Architecture

%%{init: {
'flowchart': {
    'curve': 'basis',
    'htmlLabels': true,
    'nodeSpacing': 80,
    'rankSpacing': 90
}
}}%%

flowchart LR
%% =========================
%% AI RUNTIME
%% =========================
subgraph USER ["👤 AI Runtime"]
direction TB
A["🤖 Agent"]
end

%% =========================
%% NULLRUN LAYER
%% =========================

subgraph LIB ["📦 NullRun Enforcement Layer"]
direction TB
B["NullRun SDK<br/>Interceptor"]
C["🚦 Runtime Gate"]
P["📜 Policy Engine"]
H["👤 Human Approval"]

end

%% =========================
%% PRODUCTION
%% =========================

subgraph PROD ["⚙️ Production Actions"]
direction TB

T["🛠 Tools"]
API["🌐 External APIs"]
DB["🗄 Databases"]
end

STATE["🗂 Audit + Runtime State"]

%% =========================
%% FLOW
%% =========================

A -->|"protected action"| B
B -->|"authorize"| C
C --> P
P -->|"allow"| T
P -->|"allow"| API
P -->|"allow"| DB
C -->|"require approval"| H
H -->|"approved"| T
C --> STATE

%% =========================
%% COLORS
%% =========================
classDef user fill:#dbeafe,stroke:#2563eb,color:#0f172a
classDef sdk fill:#dcfce7,stroke:#16a34a,color:#0f172a
classDef srv fill:#fed7aa,stroke:#ea580c,color:#0f172a
classDef store fill:#f5d0fe,stroke:#a21caf,color:#0f172a
classDef ok fill:#bbf7d0,stroke:#16a34a,color:#0f172a
classDef wait fill:#fef08a,stroke:#ca8a04,color:#0f172a

class A user
class B sdk
class C,P,H srv
class STATE store
class T,API,DB ok
class H wait

style USER fill:#f8fafc,stroke:#64748b,stroke-width:1px
style LIB fill:#f8fafc,stroke:#64748b,stroke-width:1px
style PROD fill:#f8fafc,stroke:#64748b,stroke-width:1px

Loading

The gate is server-authoritative — the SDK never trusts client-supplied cost. Redis is the source of truth for budget and tool-policy state; Postgres holds the immutable audit log.

---

sequenceDiagram

participant Agent
participant SDK
participant Gate
participant Policy
participant Human
participant Tool


Agent->>SDK: execute(tool)
SDK->>Gate: authorize(action)
Gate->>Policy: evaluate rules

alt Allowed
Policy-->>Gate: allow
Gate-->>SDK: continue
SDK->>Tool: execute
else Approval required
Policy-->>Gate: approval_required
Gate-->>SDK: wait
Gate->>Human: request approval
Human-->>Gate: approved
Gate-->>SDK: resume
SDK->>Tool: execute
else Blocked
Policy-->>Gate: deny
Gate-->>SDK: exception
end

Loading

Quickstart

Install:

pip install nullrun
export NULLRUN_API_KEY="nr_..."   # get one at https://nullrun.io/control-center/api-keys

Option — decorator (3 lines)

from nullrun import protect

@protect
def my_agent(prompt: str) -> str:
    return call_llm(prompt)

Framework adapters — auto-detected

NullRun auto-detects installed frameworks and instruments them automatically when init_or_die() runs (or when @protect first fires). You don't need to choose an extra; if a framework is already in your environment, it gets patched in place.

Framework	What gets patched	Trigger
LangGraph (Pregel.invoke / stream / ainvoke / astream)	NullRunCallback injected per call	auto on init_or_die()
LangChain (BaseCallbackManager)	NullRunCallback registered	auto on init_or_die()
OpenAI Agents (Runner.run / run_streamed)	RunHooks / RunStreamedHooks instrumented	auto on init_or_die()
LlamaIndex (get_dispatcher)	LLMChatEndEvent / FunctionCallEvent handlers	auto on init_or_die()
CrewAI (event bus + usage_metrics)	Agent / Task / Crew lifecycle	auto on init_or_die()
AutoGen (Agent.run / a_run)	message-streaming hooks (HTTP path is httpx-based)	auto on init_or_die()

HTTP-level coverage is the foundation — httpx (and requests) are patched once by init_or_die() regardless of vendor. Token counts and model info are extracted from response bodies for OpenAI, Azure, Anthropic, Mistral, Gemini, Cohere, and Bedrock without those vendor SDKs needing to be installed. If you use the raw httpx.Client API directly, you get cost tracking out of the box.

If you call @protect before init_or_die(), the SDK auto-triggers instrumentation lazily on the first decorated call. You can write your agent code with the decorator first and the init second — or skip init entirely if your environment is already configured via NULLRUN_API_KEY.

How NullRun compares

	NullRun	LangChain callbacks	Helicone	Portkey	OpenLLMetry
Enforce before execution	✅	❌	⚠️ async	⚠️ async	❌
Server-authoritative budget	✅	❌	❌	❌	❌
Tool-call policy	✅	❌	❌	⚠️ limited	❌
Human-in-the-loop approvals	✅	❌	❌	❌	❌
Zero-code instrumentation	✅	✅	✅	✅	✅
Immutable audit trail	✅	⚠️	✅	✅	✅
Streaming memory cap (anti-OOM)	✅	❌	⚠️	⚠️	❌
MCP support	✅	⚠️	❌	❌	⚠️

NullRun is the only option that blocks expensive or dangerous calls before they happen, not just observes them.

---

Querying the audit log

Every gate decision, approval resolution, and execution lifecycle event is written to the org's hash-chained audit_events table on the backend. The SDK surfaces a typed read API at runtime.audit.* so backends on ADR-009 (schema_version = 3) return typed dataclasses — not raw dicts.

from nullrun import NullRunRuntime, AuditQuery
from datetime import datetime, timezone, timedelta

runtime = NullRunRuntime(api_key="nr_...")

# 1) Last 50 governance decisions in the last 24h.
since = (datetime.now(timezone.utc) - timedelta(hours=24)).isoformat()
page = runtime.audit.list(
    AuditQuery(event_type="authorization_decision", since=since, limit=50)
)
for entry in page.entries:
    print(entry.timestamp, entry.decision, entry.tool_name, entry.reason_code)

Available surfaces:

Method	Returns	Endpoint
runtime.audit.list(query=...)	AuditLogPage (entries + meta)	GET /api/v1/orgs/{org}/audit-log
runtime.audit.verify(since=...)	AuditVerifyResult (chain head/tail/reason)	GET /api/v1/orgs/{org}/audit-log/verify
runtime.audit.list_exports()	list[AuditExportJob]	GET /api/v1/orgs/{org}/audit-log/export
runtime.audit.create_export()	dict (job_id, status)	POST /api/v1/orgs/{org}/audit-log/export
runtime.audit.export_status(job_id)	AuditExportStatus	GET /api/v1/orgs/{org}/audit-log/export/{job_id}/status

AuditQuery filters on the canonical ADR-009 columns: event_type (authorization_decision / approval_decision / execution_lifecycle), decision, policy_id, execution_id, actor, since, until, limit. Pre-ADR-009 backends return legacy fields only — AuditEntry.is_governance is False for those rows, and the 13 governance columns default to None.

If you call runtime.audit.* before nullrun.init() (no org binding), the proxy raises NullRunAuthenticationError — not a silent 404 — so a misconfigured CI step fails loudly at the audit call site rather than silently dropping the query.

---

Closing orphan grants (v0.18+)

An approval row that lands at status='APPROVED' but never flips to CONSUMED is an "orphan grant" — the operator sees it on the dashboard forever (or until the sweeper runs). Two paths close the orphan:

1. Success path — when the WebSocket approval push resolves outcome=approved, the SDK auto-calls POST /api/v1/approvals/{approval_id}/consume so the row flips to CONSUMED before the function body runs. Best-effort: a network blip is logged at DEBUG and the success path is not blocked.
2. Exception path — @protect's _safe_cancel_active_execution calls cancel_execution and consume_approval (in that order) when an exception fires after /gate succeeded. The reverse-index lookup execution_id → approval_id is populated by the WS push handler, so if the SDK never reached the WS-approval branch the lookup returns None and consume_approval is a no-op.

The new endpoint is structurally distinct from the orchestrator's consume_approved SQL (no execution_id binding per ADR-046, so it does not participate in the cached-replay arm race window) and carries organization_id for C2 closure. Idempotent: replay returns already_consumed; PENDING/DENIED/EXPIRED rows return not_approved, both with HTTP 200. See src/nullrun/runtime.py::consume_approval and src/nullrun/transport.py::consume_approval.

---

Examples

Runnable, copy-pastable examples live in a separate repo so you can adapt without cloning the SDK source:

• LangGraph — multi-node agent with budget + approval
• CrewAI — multi-agent crew with shared budget
• AutoGen — group-chat agent with policy gating
• LlamaIndex — RAG pipeline with cost-per-query enforcement
• Custom tools — register your own tools for policy
• Multi-agent — shared budget across sub-agents

---

Roadmap

Version	Status	Highlights
v0.14.x	✅ alpha	Wire protocol v3.31, server-minted execution IDs, MCP, anti-OOM streaming cap
v0.15.x	✅ alpha	ADR-009 governance audit surface, typed runtime.audit.*, capability probes for /audit-log/verify, fail-OPEN observability closure
v0.16.x	✅ alpha	Phase-1+ action_digest on /gate, /execute tools propagation, transient-5xx retry on gate (NR-006), error-code parity (NR-007, 41→56 entries)
v0.17.x	✅ alpha	Chain-setter Token discipline, _GATE_CACHE staleness closure, lazy-export repair, circuit-breaker lock unification (sync+async), op_id mint-fresh (DEF-OPID-REUSE-HASH-MISMATCH), error-code map closure (DEF-SDKT-004)
v0.18.x (current)	✅ alpha	Close-orphan fix (ADR-047): SDK auto-calls POST /api/v1/approvals/{id}/consume after WS approval resolves to outcome=approved and on the @protect exception path. Closes the structural orphan where mode="inline" tools left approval rows at status=APPROVED past expires_at.
v1.0	🎯 beta target	Stable wire contract, full async support, type-safe decisions

Full roadmap & RFCs →

---

Development setup

git clone https://github.com/nullrunio/nullrun-sdk-python
cd nullrun-sdk-python
python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
pytest -q

We follow Conventional Commits, require tests for new public API, and run ruff + mypy in CI.

---

Security

NullRun does not store or proxy your LLM provider keys — it sits beside your existing clients and observes the calls. The gate is server-authoritative for cost: even a malicious SDK cannot inflate spend by sending a fake cost_cents to /track.

See the security policy for the threat model and disclosure policy.

---

Community & support

• GitHub Issues
• Support

---

Made with care by NullRun and contributors.

⭐ Star us on GitHub · 📖 Read the docs