Splat - Lightweight Error Tracker & APM

Splat is a simple error tracker and lightweight backend APM. It's a fast, reliable alternative to Sentry for applications that need error monitoring and performance insight without the overhead.

Splat supports OIDC but defaults to no authentication. It has an MCP endpoint for your LLM Agents to use. This app was Agentically Engineered, partnering with GLM / Sonnet / Opus.

Initially built as an experiment in using SQLite in a write-heavy service, it's performed well enough for my use case, growing into a capable error tracker, a focused backend APM, and a structured log store — exception grouping plus transaction tracing, span waterfalls, latency percentiles, N+1 detection, and full-text searchable logs that tie back to traces, all queryable by an LLM over MCP.

I've only used Splat with Rails, but there's no reason it shouldn't work with other systems. Happy to accept pull requests for wider compatibility.

If you're looking for other Sentry clones, take a look at Glitchtip, Bugsink & Telebugs.

Run a local Splat instance

Want your own Splat to point your apps at — on your laptop or a box on your network? This gets one running in a couple of minutes. (Hacking on Splat itself is the same steps.)

Prerequisites

Ruby — see .ruby-version (currently 4.0.7)
SQLite3
Docker — bin/dev runs Tuber (the ingestion work queue) in a container, so there's nothing else to install. Prefer a native binary? The tuber: line in Procfile.dev shows how.

1. Start Splat

git clone <repository-url>
cd splat
cp .env.example .env      # sane dev defaults, incl. TUBER_URL
bin/setup                 # bundle install + db:prepare, then boots the stack

Open http://localhost:3030.

bin/setup installs gems, prepares the SQLite databases, and starts everything via bin/dev — the web server (port 3030), the worker (ingest and maintenance), Tailwind's watcher, and Tuber. Boot it again later with just bin/dev.

2. Point an app at it

There's no project out of the box. In the Splat UI, create a project — Splat generates its public key, and the project page has a Copy External DSN button. Paste that DSN into your app's Sentry config (e.g. SENTRY_DSN), and its errors, transactions, and logs start landing in your local Splat.

See Deployment for running it for real, and Authentication to put a login in front of the UI.

Overview

Named after Ruby's splat operator and the satisfying sound of squashing bugs, Splat accepts Sentry-compatible error events and transaction data via a simple API endpoint, processes them asynchronously, and presents them in a clean, fast web interface.

Key Features

Error Tracking

✅ Fingerprint Grouping - Groups events into issues by Sentry fingerprint, or derives one from exception type + location
✅ Issue Lifecycle - Open / resolved / ignored, with resolved issues auto-reopening on new events, or resolved in the next release so only a newer release reopens them
✅ Fast Ingestion - Errors appear in the UI within seconds
✅ One Clock - The project page, the issues list and the Performance page draw everything on one time axis (1h to 30d) with the deploys down it: point at an hour to read it across every row, click it to list what was in it
✅ Users Affected - How many distinct users an issue has reached, counted from the event's user (id, else username, email or IP address) the way Sentry counts them, over the events still kept. Your SDK has to send one: Sentry.set_user(id: current_user.id), or with send_default_pii, the client's IP address. Counts start from the release that added it; earlier events aren't backfilled
✅ Stack Traces & Context - Full event detail: stack trace, request data, breadcrumbs, user context
✅ Burst Alerting - Spike detection with a configurable threshold, plus email, ntfy.sh and webhook notifications (JSON, Slack or Discord) on new and reopened issues
✅ Release Tracking - Stamps issues with first/last seen release, draws deploys down the issue timelines so regressions are visible at a glance

Performance Monitoring

✅ Transaction Tracing - Request timings with db_time / view_time breakdown
✅ Span Waterfall - Per-transaction span tree rendered as a tiered timeline on the transaction detail page
✅ Latency Percentiles - p50 / p95 / p99 via DDSketch histograms (±1% error), not naive array indexing
✅ Endpoint Impact Ranking - Surfaces controllers ranked by total time spent (avg × count) plus p95, so you optimise where it actually pays back
✅ Web Requests vs Background Jobs - The op the SDK sends (http.server, queue.*) splits requests from jobs on the Performance page, each with its own throughput, p95 and share of the time (scaled by sample rate), so slow jobs don't inflate your site's p95
✅ N+1 Query Detection - Mines measurements.query_analysis from the transaction span analyzer, ranks endpoints by N+1 prevalence, exposes a dedicated worklist and an MCP tool
✅ Trends & Comparison - Hourly pre-aggregates power time-series charts and release-over-release endpoint comparison

Logs

✅ Structured Log Ingestion - Accepts both Sentry Logs (envelope item type log) and OpenTelemetry over OTLP/HTTP (POST /v1/logs), normalized into one searchable shape
✅ Full-Text Search - SQLite FTS5 over message body and flattened attributes, with a key:value shorthand (e.g. status:422 method:POST timeout)
✅ Trace Correlation - Logs carry trace_id, so a log links to its transaction and a transaction links back to its logs (sampling-aware — the link only appears when the other side exists)
✅ zstd Dictionary Compression - The full record (including attributes) is compressed into a blob with a trained zstd dictionary; hot query/display fields are promoted to columns
✅ Level & Environment Facets - Filter by severity (Sentry trace→fatal or bucketed OTLP severity numbers) and environment

Platform

✅ Sentry Protocol Compatible - Drop-in replacement for Sentry client SDKs
✅ MCP Integration - Query errors, performance and logs via Claude and other AI assistants. Issue, log and performance queries filter by project and environment (issues and endpoint stats also by release), so an agent can drive a whole load-test → mine → fix → verify cycle headlessly, without a dashboard (details)
✅ Single-Tenant, Multi-Project - Group events under multiple projects/DSNs; no user or team management overhead
✅ Minimal Dependencies - Rails + SQLite + Tuber (a single-binary work queue)
✅ Auto-Cleanup - Configurable data retention (default 90 days)

Why Splat?

When you need error tracking that:

Your code assistant can grab issues and stack traces from
Shows errors within seconds
Can be understood and modified in one sitting
Latest Rails / Ruby on an all-SQLite stack, with Solid Cache + Solid Cable and Tuber for the work queue. The running versions are shown on the Settings page.

Screenshots

_Projects	_{Project overview}	_{Cron monitors}
_Issues	_{Issue detail}	_Event
_Performance	_{Endpoint detail}	_Transaction
_Logs	_{Log entry → trace}

Getting Started

Prerequisites

Ruby (latest — see .ruby-version)
Rails (latest)
SQLite3

Installation

git clone <repository-url>
cd splat
bundle install
bin/rails db:prepare
bin/dev

Configuration

Email Notifications

Configure SMTP settings for email notifications when issues are created or reopened:

# Required settings
SMTP_ADDRESS=smtp.gmail.com
SMTP_PORT=587
SMTP_USER_NAME=your-email@gmail.com
SMTP_PASSWORD=your-app-password

# Optional settings (with defaults shown)
SMTP_DOMAIN=localhost
SMTP_AUTHENTICATION=plain
SMTP_STARTTLS_AUTO=true
SPLAT_HOST=splat.example.com
SPLAT_INTERNAL_HOST=100.x.x.x:3030  # Your Tailscale IP maybe? Used for displaying alternate DSN

# Email recipients
SPLAT_ADMIN_EMAILS=admin@example.com,dev-team@example.com
SPLAT_EMAIL_FROM=noreply@splat.com

Email Notification Control

# Enable email notifications in development
SPLAT_EMAIL_NOTIFICATIONS=true

# In production, emails are sent by default

Deployment

Docker Compose

x-common-variables: &common-variables
  RAILS_ENV: production
  SECRET_KEY_BASE: ${SECRET_KEY_BASE}
  SPLAT_HOST: ${SPLAT_HOST}
  SPLAT_ADMIN_EMAILS: ${SPLAT_ADMIN_EMAILS}
  SPLAT_EMAIL_FROM: ${SPLAT_EMAIL_FROM}

  SMTP_ADDRESS: ${SMTP_ADDRESS}
  SMTP_PORT: ${SMTP_PORT}
  SMTP_USER_NAME: ${SMTP_USER_NAME}
  SMTP_PASSWORD: ${SMTP_PASSWORD}

  # MCP Authentication Token
  MCP_AUTH_TOKEN: ${MCP_AUTH_TOKEN}

  # Splat reports its own errors/traces to a Sentry-compatible DSN (often itself).
  SENTRY_DSN: ${SENTRY_DSN}
  SENTRY_TRACES_SAMPLE_RATE: ${SENTRY_TRACES_SAMPLE_RATE}

  # Send Rails logs to stdout so `docker compose logs <service>` works without
  # fiddling with bind-mount permissions on log/production.log.
  RAILS_LOG_TO_STDOUT: "true"

  # Tuber work queue — ingestion is enqueued here. Container-to-container DNS
  # uses the service name on the container port (11300); the host-side 11330
  # mapping below is irrelevant inside the compose network.
  TUBER_URL: tuber:11300

# Shared config for the non-web Rails containers (the workers). They run
# the same image as web with a different command and no published ports.
x-splat-worker: &splat-worker
  image: ghcr.io/dkam/splat:latest
  pull_policy: always
  restart: unless-stopped
  depends_on:
    - tuber
  volumes:
    - /storage/splat/storage:/rails/storage
  environment:
    <<: *common-variables
  logging:
    driver: "json-file"
    options:
      max-size: "100m"
      max-file: "3"

services:
  # HTTP server: ingest API, MCP endpoint, and the UI.
  web:
    image: ghcr.io/dkam/splat:latest
    pull_policy: always   # :latest moves on each release; always re-pull on `up -d`
    environment:
      <<: *common-variables
      # OIDC is only needed by the web tier (see Authentication below).
      OIDC_CLIENT_ID: ${OIDC_CLIENT_ID}
      OIDC_CLIENT_SECRET: ${OIDC_CLIENT_SECRET}
      OIDC_ISSUER: ${OIDC_ISSUER}
      OIDC_PROVIDER_NAME: ${OIDC_PROVIDER_NAME}
      OIDC_REQUIRE_PKCE: ${OIDC_REQUIRE_PKCE}
      SPLAT_ALLOWED_USERS: ${SPLAT_ALLOWED_USERS}
      # Two Puma processes, so one long query can't stop the UI and ingest
      # answering — see "A long query holds its whole process" below.
      WEB_CONCURRENCY: 2
    volumes:
      - /storage/splat/storage:/rails/storage
      - /storage/splat/logs/web:/rails/log
    ports:
      - "${HOST_IP}:3030:3000"
    depends_on:
      - tuber
    restart: unless-stopped
    logging:
      driver: "json-file"
      options:
        max-size: "100m"
        max-file: "5"

  # Ingestion (the data tubes) and background jobs (splat.maintenance:
  # retention, rollups, dictionary training, storage stats). `all` boots Rails
  # once and forks a process per role — see "Workers" below.
  worker:
    <<: *splat-worker
    command: bin/worker all
    volumes:
      - /storage/splat/storage:/rails/storage
      - /storage/splat/logs/worker:/rails/log
    mem_limit: 1G
    stop_grace_period: 30s

  # No scheduler service: tuber 0.16+ fires the recurring jobs itself. With an
  # older tuber, add one running bin/scheduler — see "Recurring jobs" below.

  tuber:
    image: ghcr.io/tuberq/tuber:latest
    # The image entrypoint is the bare `tuber` binary; it needs a subcommand.
    # --migrate-wal upgrades an older binlog format in place on start.
    command: server --migrate-wal
    environment:
      - TUBER_NAME=splat
      - TUBER_LISTEN=0.0.0.0
      - TUBER_PORT=11300
      - TUBER_BINLOG_DIR=/var/lib/tuber/binlog
      - TUBER_MAX_STORAGE_BYTES=20gb
      - TUBER_MAX_JOBS_SIZE=1gb
      - TUBER_MAX_JOB_SIZE=20mb        # must fit packed batches (~5MB at 100/batch)
      - TUBER_METRICS_PORT=9100
    ports:
      - "${HOST_IP}:11330:11300"        # beanstalkd protocol
      - "${HOST_IP}:9130:9100"          # metrics (Prometheus / Uptime Kuma)
    volumes:
      - /storage/splat/storage/tuber:/var/lib/tuber/binlog
    restart: unless-stopped
    mem_limit: 1500M
    ulimits:
      nofile:
        soft: 65536
        hard: 65536
    logging:
      driver: "json-file"
      options:
        max-size: "10m"
        max-file: "3"

Versioning and releases

Splat reports two numbers, and they answer different questions. Settings → About shows both.

	Lives in	Set by	Answers
version	`config/version.rb` → `Splat::VERSION`	a human, by hand	which release is this?
revision	`config.x.revision`	the build, from `--build-arg GIT_SHA`	is what I built actually running?

The revision is precise, automatic and meaningless to read. The version is SemVer, survives a rebuild of identical code, and is what you say out loud. If the two disagree with what you expect, a deploy didn't land.

Bumping config/version.rb on main is what cuts a release. There is no git tag step for a human to forget:

Edit Splat::VERSION, commit, push to main.
.github/workflows/build.yml fires on that path, builds linux/amd64 + linux/arm64 natively, and publishes ghcr.io/dkam/splat:vX.Y.Z (plus :latest).
The same run creates the vX.Y.Z git tag and a GitHub Release.

A pre-release — any version containing a hyphen, e.g. 1.15.0-dev — publishes :v1.15.0-dev, does not move :latest, and creates no git tag. bin/build applies the same rule for local multi-arch builds, and refuses to build a non-pre-release from anywhere but main.

Authentication

Splat has no login by default — authentication is off until you configure OIDC. The three pick-one options:

None (default): Anyone can access Splat — ensure it's running internal / within a VPN
Basic Auth: Use your webserver to implement Basic Auth, avoiding protecting /api/ and /mcp/ endpoints as they're already authenticated
OIDC: Set the OIDC_CLIENT_ID, OIDC_CLIENT_SECRET, and OIDC_ISSUER values (see OIDC below)

Setting the OIDC values switches Splat into login-required mode. As soon as all three OIDC_* variables are present, the UI is no longer open: every browser request must be authenticated via your OIDC provider, and the signed-in email must match the allowlist (SPLAT_ALLOWED_USERS / SPLAT_ALLOWED_DOMAINS). Note:

All three, or none. Setting some but not all of the OIDC_* variables is treated as a fault, not as "open": the UI returns 503 and names the missing variable, rather than silently serving unauthenticated because of a typo. The login page and the startup log say the same thing. Unset all three to run open deliberately.
An empty allowlist locks everyone out. If OIDC is configured but neither SPLAT_ALLOWED_USERS nor SPLAT_ALLOWED_DOMAINS is set, no email matches, so nobody can log in. Set at least one before enabling OIDC. The login page warns about this before you make the round trip to your provider.
Ingestion and tooling stay open. The /api/, /v1/logs, /mcp, and /_health endpoints skip browser auth (they use token auth or are public), so enabling OIDC doesn't break event ingestion, log ingestion, MCP clients, or health checks — including while OIDC is half-configured, so a bad deploy costs you the UI, not your error events.

Basic Auth

Assuming a Caddy server which forwards traffic to Splat. The following configuration uses basic auth, but allows free access to the /api/ and /mcp/ endpoints.

splat.booko.info {
  encode zstd gzip

  # Handle /api/* and /mcp/* routes without basic auth (both use token auth)
  handle /api/* /mcp* {
    reverse_proxy * {
      to http://<ip address>:3030
    }
  }

  # Handle all other routes with basic auth
  handle {
    basicauth {
      <user> <basic-auth-hash>
    }
    reverse_proxy * {
      to http://<ip address>:3030
    }
  }

  log {
    output file /var/log/caddy/splat.log
  }
}

Generate the basic auth hash with docker compose exec -it caddy caddy hash-password

OIDC

Splat supports OIDC

OIDC_CLIENT_ID=<OIDC CLIENT ID>
OIDC_CLIENT_SECRET=<OIDC CLIENT SECRET>
OIDC_ISSUER=<OIDC ISSUER URL>

SPLAT_ALLOWED_USERS="Comma seperated list of email addresses allowed access Splat"
SPLAT_ALLOWED_DOMAINS="Comma seperated list of email domains allowed access Splat"

# Optional
OIDC_PROVIDER_NAME=<OIDC Providername>
OIDC_REQUIRE_PKCE=<true/false>

Performance

Splat runs in production on one shared instance for Booko and five other apps. These are that instance's numbers, measured in October 2026: load and data on Splat 1.19.4, memory and CPU on 1.21.1.

Production Metrics

Load: ~780,000 transactions a day (~9/s on average) from 6 projects, plus their errors, structured logs and cron check-ins. Ingestion keeps up — queue depth sits at 0.

Data: 244 GB of SQLite — transactions and spans 160 GB, logs 84 GB, issues and events 18 GB.

Memory and CPU, by container (the three services in the compose file, measured over the 15 hours after a deploy):

Container	Resident	Swapped out	`mem_limit`	CPU (average)
web	298 MiB	112 MiB	—	0.06 cores
worker	333 MiB	192 MiB	1024 MiB	0.11 cores
tuber	23 MiB	9 MiB	1500 MiB	0.01 cores
Total	~650 MiB	~310 MiB		~0.18 cores

Resident is each container's own memory in RAM, not counting the kernel's file cache. Swapped out is memory the kernel has paged out because nothing touched it — on that host, mostly what each Rails process loaded at boot and never used again. It costs nothing there, but on a host without swap, budget for both columns: ~1 GB at this load. RAM beyond that goes to the kernel's file cache for the SQLite files.

For scale, a quiet instance — a few apps, ~16,000 transactions a day — runs the same three containers in ~165 MiB resident, ~440 MiB with its swapped-out memory. Most of that is the fixed cost of booting Rails twice (web and worker).

Both are down from 1.19.4, which booted Rails four times — web, ingest, maintenance and a scheduler. Tuber 0.16 now fires the recurring jobs, and ingest and maintenance share one worker container that boots once and forks (see Workers). Counting resident and swapped-out together, the busy instance went from ~1.4 GB to ~1 GB, and the quiet one from ~680 MiB to ~440 MiB.

The web figures are for one Puma process. They were measured before WEB_CONCURRENCY: 2 went into the compose example, and a second worker adds to them — less than a second boot would, since Puma forks it from the booted app.

SQLite Performance

One writer per database is enough. The ingest worker runs one consumer per data tube, and each tube writes to its own database file, so events, transactions and logs never queue behind each other's write lock.
Ingestion keeps up at 160 GB. The transactions database is that size and the queue still sits empty.
A long query holds its whole process. The sqlite3 gem keeps Ruby's global lock for the whole of a statement, so while one thread scans, every other thread in that process waits — more threads don't help. Hence WEB_CONCURRENCY: 2 on the web service: one process can be stuck on a wide search while the other answers the UI and ingest. MCP tool queries are also stopped after 15 seconds (SPLAT_MCP_STATEMENT_TIMEOUT), with an error telling the model to narrow its window or add filters. That cap relies on sqlite3 2.10's statement_timeout; earlier versions stop every query at once (sqlite3-ruby#737).
Lock contention happens, and is expected. The hourly rollup and retention's vacuum write to the same files ingest does. Both treat database is locked as transient: they stop, log it, and pick up the remainder on their next scheduled run.

WAL mode (see config/database.yml) is what lets the UI read while ingest writes.

Storage Architecture: All SQLite

Splat stores everything in SQLite — there is no separate analytics engine to operate. Rather than one big file, the data is split across several SQLite databases by concern, each with its own connection, migration path, and retention window:

Database	File	Holds
primary	`storage/production.sqlite3`	Application data — projects, settings, releases, OIDC sessions
issues_events	`storage/production_issues_events.sqlite3`	Error issues and their raw events
transactions_spans	`storage/production_transactions_spans.sqlite3`	APM transactions, span trees, and the rolled-up performance aggregates
logs	`storage/production_logs.sqlite3`	Structured logs (zstd-compressed bodies, promoted query columns, FTS index)

(Two further Solid-backed databases — production_cache.sqlite3 and production_cable.sqlite3 — hold Solid Cache and Solid Cable data.)

Splitting by concern keeps the write-heavy tables (events, spans, logs) from contending on a single file's write lock, lets each domain migrate independently, and means a domain's retention prune or VACUUM only touches its own file. Within each domain:

Raw rows are the source of truth for ingestion, find-by-id lookups, status changes, and the recent-firehose views. Fast, embedded, no operational overhead.
Aggregates — the time-windowed numbers behind endpoint stats, percentile breakdowns, response-time charts, the "Top Endpoints by Impact" table, and the project dashboard — are rolled up into compact summary tables (e.g. transaction_histograms) by a recurring job, so the dashboards stay fast even as raw rows are pruned.

Raw rows and aggregates have separate retention windows, so you can prune the high-volume raw data aggressively (days) while keeping the tiny aggregate history for much longer (months). Retention windows are configurable on the Settings page.

Ingestion Queue: Tuber

Splat doesn't write incoming events to SQLite on the request path. The ingest endpoints (/api/..., /v1/logs) validate each envelope, enqueue it onto Tuber — a fast beanstalkd-compatible job queue — and return immediately, so a client never waits on parsing or disk. The ingest role drains the data tubes through per-type consumers (events, transactions, logs, forward, and an Active Job bridge) and does the parsing, fingerprinting, compression, and inserts off the hot path. The maintenance role drains splat.maintenance (retention, aggregation, dictionary training, storage stats) in a process of its own, so a heavy background job can't freeze ingestion. Both run in the one worker container — see Workers. Tuber fires the recurring jobs onto their tubes itself (0.16 and later; bin/scheduler does it on an older tuber) — see Recurring jobs.

Tuber is a single Rust binary — a Beanstalkd rewrite — that keeps job metadata in RAM with a write-ahead log for durability, and offloads larger job bodies to disk (a TOAST-style scheme), so memory stays bounded by job count rather than payload size. That gives Splat a durable, RAM-speed ingest buffer with no extra database to tune. See the Tuber project for the server.

When Tuber is down, the web service keeps what it can't enqueue on disk, in storage/spool/, and still answers 200: SDKs drop an envelope after a 5xx rather than sending it again, so answering 503 would lose it. The ingest worker puts the spooled jobs back on their tubes, oldest first, once Tuber answers again. The spool is capped at 1 GB (SPLAT_SPOOL_MAX_MB) so a long outage can't fill the disk the databases share; past the cap, ingest answers 503. While anything is spooled, /_health reports degraded with the count, and get_status shows it. A job can arrive twice if the worker dies mid-replay — harmless for events, which are unique on event_id.

Endpoint Impact Ranking

The performance dashboard ranks endpoints by time spent (avg_duration × count) rather than average duration. A 50ms endpoint hit 10,000×/day costs more total time than a 2s endpoint hit 5×/day — sorting by impact tells you where optimisation actually pays back. The same table also surfaces P95 alongside, so a tail-heavy endpoint isn't hidden by a low average.

Span Storage and SQL Normalization

Each transaction's span tree is stored in SQLite and rendered as a waterfall on the transaction detail page. Spans are 10–100× the volume of transactions, so a single ingest-time normalization pass keeps storage manageable:

SQL normalization at ingest: span descriptions like SELECT * FROM users WHERE id = 42 AND email = 'alice@example.com' are rewritten to SELECT * FROM users WHERE id = ? AND email = ? before being written to disk. The parameterized form collapses the near-infinite variety of literal-bearing queries down to a handful of distinct patterns.
Privacy bonus: because literals never reach disk, user IDs, email addresses in WHERE clauses, names in INSERTs, and tokens in URL paths are automatically redacted. We can show you the pattern of the offending query, but not the literal values that triggered it. This is a deliberate trade-off — and a documented commitment, not an accident.
Cap and retention: spans are capped at 1000 per transaction (excess dropped, transaction flagged) and retained for 30 days by default (configurable separately from transactions, since span volume is far higher).

N+1 Query Detection

Splat flags endpoints that issue the same query in a loop — the classic N+1 — straight from the SQL breadcrumbs a transaction carries, with no extra instrumentation in the monitored app.

How a single transaction is judged (Transaction::SpanAnalyzer.analyze_sql_queries, at ingest):

Pull the sql.active_record breadcrumbs from the payload.
Drop infrastructure queries. Cache, job-queue, cable, schema-bookkeeping, and SQLite-introspection tables (solid_cache_entries, solid_queue_*, solid_cable_messages, schema_migrations, ar_internal_metadata, sqlite_*, dbstat) are framework plumbing, not application work. SolidCache alone fires get + delete + put on every Rails.cache.fetch miss, so a request doing a few cache lookups would otherwise look like an N+1 of identical queries. They're excluded from both the query count and the scan so the signal reflects your DB work.
Normalize each remaining query to a pattern — literals collapse to ?, IN-lists to IN (?), query-log-tag comments (/* ... */) are stripped, but table/column identifiers are preserved so different tables stay distinct (see SQL normalization above).
Group by pattern and count. A pattern that appears more than 3 times in one transaction marks it as a potential N+1 (has_n_plus_one), and the per-transaction query_count is recorded.

How it rolls up. The hourly aggregation job folds each transaction's has_n_plus_one flag and query counts into transaction_hourly_stats per endpoint, so the dashboards stay fast and the signal survives raw-row retention. The N+1 view ranks endpoints by how many of their transactions were flagged, alongside avg/max queries per request and latency.

Where it surfaces:

Endpoints dashboard → the dedicated "N+1" view (/projects/:slug/endpoints/n_plus_one).
Transaction detail page → flagged patterns for a single request.
MCP → the find_n_plus_one_endpoints tool, for asking an AI assistant "which endpoints have N+1 problems?".

Heuristic, not a profiler. This is a count-of-repeated-patterns signal, deliberately simple. Unlike in-process tools such as Prosopite — which group queries by call-site from a live backtrace — Splat only has the stored SQL text, so it can't tell a genuine loop from the same query shape legitimately reached via different code paths. The > 3 threshold and the infrastructure denylist keep false positives down; treat a flag as "worth a look," then confirm against the transaction's span waterfall.

Structured Logs

Splat ingests structured logs alongside errors and transactions, into their own SQLite database (separate retention, separate file). Two wire formats are accepted and normalized into a single shape at ingest:

Sentry Logs — envelope item type log, sent by the same SDK already pointed at Splat.
OpenTelemetry — OTLP/HTTP, JSON-encoded, at POST /v1/logs. Severity numbers are bucketed onto Sentry's trace → debug → info → warn → error → fatal scale so both sources share one level enum.

Compact storage. The full record — message plus arbitrary attributes — is compressed into a blob using a trained zstd dictionary (logs are small and repetitive, so a shared dictionary pays off far more than per-row compression). The hot fields used for listing and filtering (timestamp, level, logger, environment, trace_id, span_id) are promoted to real columns and indexed; everything else is decompressed only when you open a single log.

Full-text search. An FTS5 index covers both the message body and the flattened attributes, kept in sync with the table on insert/delete. The search box accepts free text (each token ANDed) plus a key:value shorthand that scopes a value to its attribute — so status:422 matches a 422 stored under status, not some unrelated field. Pasted UUIDs are collapsed to a single token so a hyphenated id matches the way it's indexed.

Trace correlation. Because logs carry trace_id, the log detail page links to its transaction and the transaction detail page links back to "View N logs for this trace." The two directions are deliberately lopsided: logs are ingested at ~100% while transactions are typically sampled (e.g. 1–10%), so the log→transaction link only appears when that trace happened to be sampled, while transaction→logs almost always resolves. Each link only renders when the other side actually exists, so you never land on an empty page.

Maintenance

Workers

bin/worker runs the consumers that drain tuber, in two roles: ingest (the data tubes and check-ins) and maintenance (splat.maintenance: retention, rollups, dictionary training, storage stats). bin/worker all, which the compose file runs, boots Rails once and forks a process per role:

Why separate processes: the sqlite3 gem holds Ruby's global lock for the whole of a query, so a long maintenance query — the storage scan, a retention delete — would stop every ingest thread sharing its process.
Why forked: the processes share the memory Rails loaded at boot until they write to it. Measured locally, the pair take about half what two separately booted processes do (124 MiB against 252 MiB, idle).
A process that dies is restarted after a pause: 1 second, doubling while it keeps dying, up to 30. Each death is reported to Sentry, one issue per role.
Out of memory: the maintenance process raises its own oom_score_adj, so if the container hits its mem_limit the kernel kills it rather than ingest. It's restarted, and tuber hands the job it was running out again once its reservation times out.
On TERM every process finishes its batch and exits; one still running after 25 seconds is killed. ps and docker top show them as splat-worker ingest and splat-worker maintenance.

bin/worker ingest and bin/worker maintenance each run one role in a single process, if you'd rather run two services, each with its own mem_limit.

Recurring jobs

config/schedule.yml lists the jobs Splat runs on a timetable: retention, the logs search-index merge, the WAL checkpoint, hourly rollups, storage stats, dictionary training, and the minute sweep that raises missed-check-in alerts. If nothing fires them, nothing complains on its own — the disk fills, the logs WAL runs away, and monitors stop alerting — so Splat checks for you.

What fires them

Tuber 0.16 or later fires them itself. When the web container starts, Splat hands tuber the whole schedule (bin/rails splat:schedule:sync, run by bin/docker-entrypoint). Tuber keeps it across restarts with its binlog, and puts each job on its tube at its time. No Splat process needs to be running for that.
If tuber loses it, the ingest worker puts it back. A tuber running without a binlog (-b) comes back from a restart with no schedule; the ingest worker reconnects after every restart, and syncs the schedule if tuber holds none of it. It never replaces a schedule tuber already holds, since during a rolling deploy that may be a newer release's.
On a tuber older than 0.16, bin/scheduler fires them. It's the scheduler service in compose.yml, off unless you start it with --profile old-tuber. Run on 0.16 or later it checks that tuber holds the schedule (syncing it if not) and sits idle; started on an older tuber that's upgraded later, it hands the schedule over to tuber and goes idle.

Either way a run is the same job — same tube, body, priority, TTR and tags — and an entry's idp: key with its 30-second cooldown turns a duplicate fire into a no-op, so tuber and bin/scheduler overlapping during a changeover runs nothing twice.

The schedule is replaced as a whole: an entry removed from the file is removed from tuber at the next sync, and an unchanged one keeps its next fire. Every schedule must be a cron line — five fields, with an optional timezone last — which tuber and bin/scheduler both read; a line tuber refuses fails the sync, and tuber keeps running the schedule it had.

Upgrading to tuber's scheduler

Upgrade tuber to 0.16. Its first start on an older binlog needs --migrate-wal (as in the compose example above); without it tuber refuses to start rather than rewrite the files. Back up its binlog directory first: older tubers can't read the migrated one. Once it has started, the flag can come out.
Upgrade Splat. The web container syncs the schedule as it starts. Check the Scheduled jobs panel on the Settings page: every job should say tuber.
Remove the scheduler service (or leave its profile off). Once tuber is on 0.16 it only sits idle.

Checking on it

Settings → Scheduled jobs lists every entry: what fires it (tuber, bin/scheduler, or nothing), when a worker last ran it, when it fires next, and its status — Late once a whole cycle has passed without a run, or Stuck when tuber's fires are deduplicating against a buried job (kick or delete it). A red banner says so when nothing fires them.
/_health reports "scheduler": {"status": "late"} and an overall degraded when monitor_evaluation, which runs every minute, hasn't run for five minutes. That's what catches a stopped schedule nobody is looking at — see Uptime Kuma Setup.
Tuber's own view: tuber schedules -a <host>:11330 lists each entry with its next and last fire.

Data Retention and Cleanup

A daily job (Maintenance::RetentionJob) deletes data older than its retention window, then hands the freed space back to the filesystem.

Retention windows

Set on the Settings page, each between 1 and 3,650 days. There are no environment variables for these.

Data	Default	What goes
Events	30 days	Raw error events. Their issues stay, with counts recalculated from the events that remain.
Transactions	90 days	Transaction rows, plus any span data still attached to them.
Spans	30 days	Span trees — the waterfall on the transaction page. Shorter than transactions because spans are far bigger.
Logs	14 days	Log records and their search-index entries.
Aggregates	540 days	Latency histograms and hourly endpoint stats. Kept long, so endpoint stats and charts keep working after the raw rows are gone.

Once a weekly deep storage scan has run, the Settings page shows each kind's disk use per day, and a projected size for the window you type in.

When it runs

At 02:00 Australia/Melbourne every day — the data_retention entry in config/schedule.yml, which is where to change the time or zone. A search-index merge for logs follows at 02:40.

The job is built to share the databases with live ingest:

Deletes go in batches of 500, with a short sleep between batches, so ingest gets the write lock in between.
Then incremental_vacuum on each big database (issues/events, transactions/spans, logs) returns freed pages to disk, for up to 15 minutes per database. Whatever is left over carries to the next night. After you shorten a window, expect the files to shrink over several nights, not at once.
If it can't get the lock, it stops and leaves the rest. A busy database ends that night's vacuum; it doesn't fail the job.

Running it now

Queue it for the maintenance worker, the same way the scheduler does. -c 1 shares the concurrency key every maintenance job uses, so it waits for any maintenance job already running:

docker compose exec tuber tuber put -t splat.maintenance -c 1 \
  '{"class":"Maintenance::RetentionJob","args":[]}'

Checking on it

docker compose logs worker | grep RetentionJob

It logs one line per database for the vacuum — free pages before and after, and why it stopped — then a summary of rows deleted per kind. A night on the production instance described under Performance:

[Maintenance::RetentionJob] vacuum IssuesEventsRecord: freelist 4787974 -> 3482974 in 1305000 step(s) — hit 900s budget, resuming next run
[Maintenance::RetentionJob] incremental_vacuum stopped on TransactionsSpansRecord: database locked
[Maintenance::RetentionJob] vacuum LogsRecord: freelist 2164756 -> 1875842 in 287001 step(s) — no further pages reclaimable after 3 checkpoint(s) — WAL still references them, resuming next run
[Maintenance::RetentionJob] done in 14115.96s — events:128823, transactions:642357, spans:0, span_trees:642357, logs:463685, histograms:0, hourly_stats:0, facets:1, issue_facets:22

That run took just under four hours, most of it deleting.

Monitoring

Splat provides a /_health endpoint for monitoring service status and queue depth.

Health Endpoint

GET /_health

Response:

{
  "status": "ok",
  "timestamp": "2025-10-23T12:34:56Z",
  "queue_depth": 0,
  "queue_status": "healthy",
  "scheduler": {"status": "ok", "entry": "monitor_evaluation", "last_run_at": "2025-10-23T12:34:00Z"},
  "spool": {"jobs": 0, "bytes": 0},
  "event_count": 1234,
  "issue_count": 56,
  "transaction_count": 5678,
  "transactions_per_second": 1.23,
  "transactions_per_minute": 73.5
}

Response Fields:

status: Overall system health — degraded when the queue is critical, the recurring jobs have stopped, or ingest is spooling to disk because Tuber can't be reached; ok otherwise
queue_status: Queue health (healthy, warning, or critical)
scheduler.status: late when monitor_evaluation (every minute) hasn't run for five minutes, counted from boot if it never has — nothing is firing the recurring jobs, or the ingest worker that runs it is down. ok otherwise
spool: Jobs ingest kept on disk while Tuber was unreachable, waiting to be queued (see Ingestion Queue)
queue_depth: Number of pending background jobs
timestamp: Current server time (ISO 8601)
event_count: Total error events tracked
issue_count: Number of open issues
transaction_count: Total performance transactions
transactions_per_second: Rate over last minute
transactions_per_minute: Rate over last hour

Environment Variables for Thresholds:

# Optional - defaults shown
QUEUE_WARNING_THRESHOLD=50   # queue_status becomes "warning"
QUEUE_CRITICAL_THRESHOLD=100 # queue_status becomes "critical", status becomes "degraded"

Uptime Kuma Setup

Monitor Configuration:

Monitor Type: HTTP(s) - JSON Query
URL: https://splat.yourdomain.com/_health
Expected Status Code: 200
Check Interval: 60 seconds (or your preference)

Option 1: Monitor Queue Status (Recommended)

JSON Path: $.queue_status
Expected Value: healthy
Alert When: Value is not equal to expected value
Result: Alerts when queue is "warning" or "critical"

Option 2: Monitor Overall Status

JSON Path: $.status
Expected Value: ok
Alert When: Value is not equal to expected value
Result: Alerts when the system is "degraded": critical queue depth, ingest spooling to disk, or the recurring jobs have stopped. The second has no other alert — Splat's own monitor alerts are one of the jobs that stopped — so watch $.status (or $.scheduler.status) even if you also watch the queue.

Notification Settings: Configure Uptime Kuma to send alerts via:

Email
Slack
Discord
Webhook
Or any of the 90+ notification services supported

Monitoring Guidelines:

Normal queue depth: 0-10 jobs (instant processing)
Warning level: 50-99 jobs (queue building up)
Critical level: 100+ jobs (queue backlog)

When queue_status is "warning":

Jobs are processing but slower than ingestion rate
Check the worker (docker compose logs worker) and Tuber
Consider a second ingest process if sustained: a service running bin/worker ingest alongside the worker. SQLite serialises writes per database, so it helps only when per-job CPU (zstd) is the bottleneck, not the write

When queue_status is "critical":

Significant backlog, data delayed
Immediate investigation needed
Check for worker crashes or resource constraints

Backup

Splat splits its data across six separate SQLite database files, and backup tools operate per-file — backing up production.sqlite3 alone captures only projects and settings, not your errors, transactions, or logs. Every database you care about needs its own backup configuration.

Built in — two rake tasks, for a copy on demand or from cron:

# A compacted copy of each database, via VACUUM INTO. Never overwrites one already there.
docker compose exec web bin/rails splat:backup:snapshot                  # storage/backups/<UTC time>/
docker compose exec web bin/rails "splat:backup:snapshot[/backups/splat]"

# Keep one copy of each database current. Where a copy exists and sqlite3_rsync
# is installed, only the changed pages are written; otherwise it's a fresh VACUUM INTO.
docker compose exec web bin/rails splat:backup:sync                      # storage/backups/current/

Both copy the four databases that hold data (not the cache or cable) while Splat runs, and need as much free space as the databases take. The image's SQLite (Debian's 3.46) has no sqlite3_rsync, which arrived in 3.47, so in the container sync makes each copy afresh. The default directories are on the storage volume, the same disk as the databases; mount somewhere else and pass it if the copy should survive that disk.

Or one of these, run outside Splat:

Litestream - Continuous replication to S3-compatible storage

Real-time backup with ~10-30 second lag
Supports AWS S3, Backblaze B2, Cloudflare R2, MinIO
Point-in-time recovery
Requires one dbs: entry per database file — a single entry replicates a single file

sqlite3_rsync - Efficient incremental backups (official SQLite utility)

Creates byte-for-byte clones of live databases
Works while database is in use
Smaller incremental transfers than full copies
Run it once per database file

Do not back up a live database with plain cp or rsync — the databases run in WAL mode, and copying the .sqlite3 file without a consistent snapshot of its -wal file produces a corrupt or stale copy. Use one of the tools above, or sqlite3 <db> ".backup <dest>".

What to Backup

storage/production.sqlite3 - Application data: projects, settings, releases (critical)
storage/production_issues_events.sqlite3 - Error issues and events (critical)
storage/production_transactions_spans.sqlite3 - Transactions, spans, performance aggregates (critical)
storage/production_logs.sqlite3 - Structured logs (critical if you rely on them)
storage/production_cache.sqlite3 / storage/production_cable.sqlite3 - Solid Cache / Cable (optional, regenerated at runtime)
storage/tuber/ - Tuber's write-ahead log — only holds in-flight ingest jobs; not needed for a point-in-time restore

Example Litestream config covering all four critical databases:

# /etc/litestream.yml
dbs:
  - path: /app/storage/production.sqlite3
    replicas: [{ url: s3://your-bucket/splat/production }]
  - path: /app/storage/production_issues_events.sqlite3
    replicas: [{ url: s3://your-bucket/splat/issues_events }]
  - path: /app/storage/production_transactions_spans.sqlite3
    replicas: [{ url: s3://your-bucket/splat/transactions_spans }]
  - path: /app/storage/production_logs.sqlite3
    replicas: [{ url: s3://your-bucket/splat/logs }]

Restoring

Restore all critical databases from (roughly) the same point in time. The issues/events, transactions/spans, and logs databases reference project IDs in the primary database, so wildly mismatched restore points leave rows pointing at projects that don't exist yet. There's no cross-file transaction to preserve, so second-level alignment is fine — just don't mix a week-old primary with today's events.

Each backup is self-contained: the zstd compression dictionaries used for event and log payloads are stored inside the same database file as the rows they compress, so a restored file is always readable on its own.

Model Context Protocol (MCP) Integration

Splat exposes an MCP server that allows Claude and other AI assistants to query error tracking and performance data directly. The MCP endpoint carries its own bearer token and is independent of OIDC — /mcp skips browser auth either way, so this token is the only thing guarding it.

The querying tools are parameterized by project and environment, which is the point: an agent can run a complete load-test → mine → fix → verify loop without a human ever opening a dashboard. Hammer staging, ask find_n_plus_one_endpoints and get_transaction_stats what broke in that environment, read the offending stack trace with get_issue, fix it, deploy, then close the loop with compare_endpoint_performance across the two releases and resolve_issue when the numbers agree. Every step is a tool call, so it works headless — in CI, in a cron job, or in an agent session.

Two arguments are worth knowing about, since they show up on most tools:

project — every listing tool spans all projects by default (MCP credentials are instance-wide) and labels each row with its project. Pass project to narrow to one, given as its slug, display name, or numeric id. An unknown value is an error listing the valid slugs, not an empty result.
environment — supported by list_recent_issues, search_issues, search_logs, get_transaction_stats, search_slow_transactions and the endpoint tools. On the issue tools it means seen in: grouping is deliberately cross-environment, so environment: "staging" matches issues with at least one staging event, and the same issue may also be occurring in production. release works the same way on list_recent_issues and search_issues.

Not every tool takes both — get_issue, get_event and get_status address a single record or the instance itself, and the release dimension only exists where the underlying aggregates carry it (issues, and get_transaction_stats when narrowed to one endpoint). The inputSchema in tools/list is authoritative.

Note the tools are not read-only: resolve_issue, ignore_issue and reopen_issue change issue state. Treat the token as read/write.

There are three kinds of token. Which of the first two applies depends on whether the instance has users; MCP_AUTH_TOKEN works alongside either.

	Per-user token	Instance token	`MCP_AUTH_TOKEN`
Where it comes from	Settings → MCP, one per signed-in user	Settings → MCP, one shared	An environment variable
Active when	OIDC is configured	OIDC is not configured	Always, if set
Regenerate	Button on the settings page	Button on the settings page	Edit the variable, restart
Revoking it	Leave the allowlist, get logged out at the IdP, or stop signing in — see below	Regenerate it	Unset the variable, restart
Good for	People	A Splat with no users	Cron, CI, scripts

Either way, open Settings → MCP: it shows the whole claude mcp add command with host, port and token filled in, plus copy and regenerate buttons. Paste it and you're done — skip to step 3.

The instance and per-user tokens are mutually exclusive. Configure OIDC and the shared token stops being accepted, automatically, in favour of per-user ones — so switching on authentication retires the shared credential instead of leaving a second, unattributed way in. Take OIDC away again and it resumes.

MCP_AUTH_TOKEN is different: it is accepted in every configuration, OIDC or not. It's the headless path — cron, CI, a script — where there's no user to attribute a token to, and an environment variable can't be regenerated from a web page. The trade-off is that it's unattributed and the allowlist can't revoke it; the settings page says so when it's set. Unset it to turn it off.

A per-user token has the same lifespan as web access. Three things end it, matching how you lose the web UI:

Leave the allowlist — checked on every request, so removing an address kills its token on the next call.
Get logged out at your IdP — a backchannel logout revokes the token immediately, the same moment it ends your web session.
Stop signing in — a token expires a configurable number of days after its owner last used the web UI (Settings → MCP, default 7, 0 to disable). Any visit to Splat renews it, so an active user never notices; someone who has quietly left stops working within the window even if their address still matches a domain allowlist. Renewal doesn't change the token value — signing in reactivates the same token, so the client just reconnects. When a token expires this way the MCP error says so and gives the URL to sign in at.

All tokens are stored in the clear, like project DSN public keys — anyone who can read the database can already read everything the tokens reach.

On an instance with no OIDC, the settings page has no sign-in in front of it, so the shared token is visible to anyone who can load the page — but so are the Resolve and Ignore buttons next to it. Guard such deployments at the network or with the basic-auth recipe above.

If you'd rather set the environment variable yourself:

1. Generate an authentication token:

# Using OpenSSL
openssl rand -hex 32

# Or using Ruby
ruby -r securerandom -e 'puts SecureRandom.hex(32)'

2. Add to your environment:

# .env
MCP_AUTH_TOKEN=your-generated-token-here

3. Add it to Claude Code:

Claude Code speaks HTTP directly, so this is a one-liner — no config file, no proxy:

claude mcp add --transport http splat-booko https://splat.booko.info/mcp \
  --header "Authorization: Bearer your-generated-token-here"

Against a local dev server, use the port Procfile.dev serves on:

claude mcp add --transport http splat-dev http://localhost:3030/mcp \
  --header "Authorization: Bearer your-generated-token-here"

The name (splat-booko, splat-dev) is yours to pick and is how you'll refer to the server; add one per Splat instance you want to query. Check it took with claude mcp list, and remove one with claude mcp remove <name>.

Claude Desktop (only if you need it):

Claude Desktop wants stdio, so it needs a shim that pumps stdin to the HTTP endpoint. Prefer the Claude Code route above if you have the choice.

Create a file at ~/splat-mcp-proxy.sh:

#!/bin/bash
# Proxy for Splat MCP over stdio -> HTTP
# Replace TOKEN with your actual MCP_AUTH_TOKEN

while IFS= read -r line; do
  echo "$line" | curl -s -X POST http://localhost:3030/mcp \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer YOUR_TOKEN_HERE" \
    -d @-
done

Make it executable:

chmod +x ~/splat-mcp-proxy.sh

Edit ~/Library/Application Support/Claude/claude_desktop_config.json:

{
  "mcpServers": {
    "splat": {
      "command": "/Users/YOUR_USERNAME/splat-mcp-proxy.sh",
      "transport": {
        "type": "stdio"
      }
    }
  }
}

4. Restart Claude Desktop (Claude Code picks up claude mcp add immediately).

Available MCP Tools

Issue Management:

list_recent_issues - List recent issues by status
search_issues - Search by keyword, exception type, or status
get_issue - Get detailed issue with stack trace
get_issue_events - List event occurrences for an issue
get_event - Get full event details (request ID, breadcrumbs, context)
resolve_issue / ignore_issue / reopen_issue - Lifecycle transitions (resolve_issue takes in_next_release)

Performance Monitoring:

get_transaction_stats - Overall percentiles plus top endpoints ranked by total time spent (avg × count)
get_endpoint_summary - Per-endpoint percentiles, average DB and view time, and fastest/slowest sample requests
get_endpoint_timeseries - Bucketed count + p50/p95/p99 for one endpoint over a time range — built for spotting regressions ("did p95 jump after the 14:00 deploy?")
find_n_plus_one_endpoints - Endpoints ranked by N+1 prevalence (% of transactions affected, avg/max queries per request) so you find the worst offenders quickly
search_slow_transactions - Find slow individual requests, each with its host
get_transactions_by_endpoint - List recent transactions for one endpoint
get_host_breakdown - Requests, avg and max duration per host per time bucket — for spotting one server stalling while the others pick up its load (raw data, so capped at 6h)
compare_endpoint_performance - Before/after percentile comparison around a release or timestamp
get_transaction - Get detailed transaction breakdown

The tools that summarise one endpoint's timings (get_endpoint_summary, get_endpoint_timeseries, compare_endpoint_performance, and get_transaction_stats with an endpoint) need a project whenever the endpoint name has requests in more than one project. Two apps' ProductsController#index are different code, so they won't be pooled. Called without one, the tool replies with the projects to choose from. For the same reason, get_transaction_stats called with neither gives each project its own overall figures rather than one blend, and get_host_breakdown gives two projects on one host a column each.

search_slow_transactions, get_transactions_by_endpoint and get_transaction_stats take a server_name to narrow to one host. It isn't indexed and the hourly rollups don't carry it, so get_transaction_stats reads raw transactions when it's set and caps its window at 6h.

Logs:

search_logs - Search structured logs by level, logger, trace, environment, or message text
get_log - Get a single log record by log_id, including decompressed attributes
get_trace_logs - Get all logs sharing a trace_id, oldest first — ties logs to a transaction/trace

Usage Examples

Once configured, you can ask Claude:

"List recent open issues in Splat"
"Search for NoMethodError issues in production"
"Where is the booko app spending its time?" → impact-ranked top endpoints
"Which endpoints have N+1 query problems?" → ranked worklist
"Show the p95 of UsersController#show over the last 7 days, hourly"
"Compare AlertsController#index performance before and after release v1.42.0"
"Search the logs for status:422 POST requests in the last hour"
"Show me every log line for trace abc123 in order" → reconstructs a request timeline

Full List of Environment Variables

  RAILS_ENV: production
  SECRET_KEY_BASE : generate with `openssl rand -hex 64`
  HOST_IP: ip address to bind to
  PORT: 3000
  SPLAT_DOMAIN: https://splat.example.com # Change this to your domain
  FROM_EMAIL: splat@splat.example.com # Change this to your email
  TUBER_URL: tuber:11300 # host:port of the Tuber work queue
  WEB_CONCURRENCY: 2 # Puma processes for the web service; 2 keeps one free while the other runs a long query

MCP (Model Context Protocol)

MCP_AUTH_TOKEN: Generate with `openssl rand -hex 32`
SPLAT_MCP_STATEMENT_TIMEOUT: Seconds before an MCP tool's query is stopped (default 15; 0 turns it off).

Data Retention

None — retention windows are set on the Settings page. See Data Retention and Cleanup.

SQLite Page Cache

SPLAT_SQLITE_CACHE_KB=2000         # page cache per connection, in KB
SPLAT_SQLITE_CACHE_BOOST_KB=20000  # what an ingest consumer raises its write connection to during a backlog

Every connection gets the small cache. Each ingest consumer raises the cache on the one database it writes while its reserves come back full, then drops it 60 seconds after the last full one. Once a minute each consumer logs its jobs and ms/job at each size, plus process memory, to the container log and to SENTRY_DSN as a structured log. Set SPLAT_SQLITE_CACHE_KB=20000 to go back to the old behaviour: 20 MB on every connection, all the time.

Tuber Work Queue

TUBER_URL: tuber:11300   # host:port of the Tuber (beanstalkd-compatible) server
SPLAT_SPOOL_MAX_MB: 1024 # what ingest keeps on disk while Tuber is down, before answering 503; 0 turns spooling off

Ingestion is enqueued on Tuber. To watch the tubes live — pending/ready/reserved counts, throughput — use tuber-tui:

tuber-tui --url <host>:11330   # the host-mapped Tuber port from compose

OIDC Authentication Setup

Splat supports OpenID Connect (OIDC) authentication, configured from the provider's issuer URL alone. This replaces the basic auth setup with proper user authentication.

Quick Start

Set 3 environment variables:

# Required for OIDC authentication
OIDC_ISSUER=https://your-provider.com
OIDC_CLIENT_ID=your-client-id
OIDC_CLIENT_SECRET=your-client-secret
OIDC_PROVIDER_NAME=Your Provider Name  # Optional: Display name for login button

OIDC_ISSUER is the provider's issuer URL, and everything else is derived from it — discovery is fetched from OIDC_ISSUER + /.well-known/openid-configuration.

Take the value from the "issuer" field of your provider's own discovery document and copy it exactly, trailing slash and path and all. It is what the id_token's iss claim is checked against, and providers differ: Auth0's issuer ends in a slash, Authentik's and Keycloak's carry a path. Splat checks it against the discovery document before sending anyone to log in, as the OIDC Discovery spec requires; get it wrong and the login page says so, with the issuer the provider gave.

Upgrading from OIDC_DISCOVERY_URL? That variable is gone; set OIDC_ISSUER instead. Moving the old value across unchanged works — a /.well-known/openid-configuration suffix is stripped before the discovery URL is built — but the issuer check uses the value verbatim, so a pasted well-known URL fails it. Use the bare issuer.

Important: Configure your OIDC provider with the callback URL: https://your-splat-domain.com/auth/callback

Provider-Specific Examples

Google:

OIDC_ISSUER=https://accounts.google.com
OIDC_CLIENT_ID=your-google-client-id
OIDC_CLIENT_SECRET=your-google-client-secret
OIDC_PROVIDER_NAME=Google

Okta:

OIDC_ISSUER=https://your-domain.okta.com
OIDC_CLIENT_ID=your-okta-client-id
OIDC_CLIENT_SECRET=your-okta-client-secret
OIDC_PROVIDER_NAME=Okta

Auth0:

OIDC_ISSUER=https://your-domain.auth0.com/
OIDC_CLIENT_ID=your-auth0-client-id
OIDC_CLIENT_SECRET=your-auth0-client-secret
OIDC_PROVIDER_NAME=Auth0

Microsoft Azure AD:

OIDC_ISSUER=https://login.microsoftonline.com/your-tenant-id/v2.0
OIDC_CLIENT_ID=your-azure-client-id
OIDC_CLIENT_SECRET=your-azure-client-secret
OIDC_PROVIDER_NAME=Microsoft

Use your tenant's issuer, not /common or /organizations: their discovery documents give the issuer as a {tenantid} placeholder, which no configured value can match.

Providers Without Discovery

There is no manual endpoint configuration. Splat requires a provider that serves a discovery document at OIDC_ISSUER + /.well-known/openid-configuration, and reads the authorization, token, userinfo and JWKS endpoints from it. A provider without one can't be used.

(Earlier versions of this README documented OIDC_AUTH_ENDPOINT, OIDC_TOKEN_ENDPOINT, OIDC_USERINFO_ENDPOINT and OIDC_JWKS_ENDPOINT. No code has ever read them; setting them does nothing.)

How It Works

Discovery: The app fetches OIDC configuration from OIDC_ISSUER + /.well-known/openid-configuration
Authentication: Users are redirected to your OIDC provider for login
Token Storage: JWT tokens are encrypted and stored in secure HTTP-only cookies
Auto-Refresh: Tokens are automatically refreshed when needed (5 minutes before expiry)
Session Migration: Existing sessions are automatically migrated to encrypted cookies

Security Features

Encrypted Cookies: JWT tokens are encrypted using Rails message verifier
HTTP-Only Cookies: Tokens not accessible via JavaScript
SameSite=Strict: Protection against CSRF attacks
JWT Verification: Optional token signature validation
Automatic Cleanup: Tokens cleared on logout or expiry

Email Sending Setup

  https://guides.rubyonrails.org/action_mailer_basics.html#action-mailer-configuration
  https://guides.rubyonrails.org/configuring.html#configuring-action-mailer

  SMTP_ADDRESS - default 'localhost'
  SMTP_PORT - default 587
  SMTP_DOMAIN' - default 'localhost' Some providers require it match a verified domain.
  SMTP_USER_NAME' - default nil
  SMTP_PASSWORD' - default nil
  SMTP_AUTHENTICATION' - default 'plain'
  SMTP_STARTTLS_AUTO' - default 'true'

Development

Services

Tuber: Work queue for ingestion + recurring jobs, drained by bin/worker all (the ingest and maintenance roles, a process each); recurring jobs fired by Tuber itself (0.16+) or by bin/scheduler
Solid Cache: In-memory caching
Solid Cable: Real-time updates (optional)

Email Previews

View email templates at http://localhost:3030/rails/mailers

Name		Name	Last commit message	Last commit date
.claude		.claude
.github		.github
app		app
bin		bin
config		config
db		db
docs		docs
lib		lib
log		log
public		public
script		script
storage		storage
test		test
tmp		tmp
vendor		vendor
zstd_dicts		zstd_dicts
.dockerignore		.dockerignore
.env.example		.env.example
.gitattributes		.gitattributes
.gitignore		.gitignore
.ruby-version		.ruby-version
.standard.yml		.standard.yml
CHANGELOG.md		CHANGELOG.md
CLAUDE.md		CLAUDE.md
Dockerfile		Dockerfile
Gemfile		Gemfile
Gemfile.lock		Gemfile.lock
LICENSE.txt		LICENSE.txt
Procfile.dev		Procfile.dev
README.md		README.md
Rakefile		Rakefile
compose.yml		compose.yml
config.ru		config.ru

Latest commit

History

Folders and files

Repository files navigation

Splat - Lightweight Error Tracker & APM

Run a local Splat instance

1. Start Splat

2. Point an app at it

Overview

Key Features

Why Splat?

Screenshots

Getting Started

Prerequisites

Installation

Configuration

Email Notifications

Email Notification Control

Deployment

Docker Compose

Versioning and releases

Authentication

Basic Auth

OIDC

Performance

Production Metrics

SQLite Performance

Storage Architecture: All SQLite

Ingestion Queue: Tuber

Endpoint Impact Ranking

Span Storage and SQL Normalization

N+1 Query Detection

Structured Logs

Maintenance

Workers

Recurring jobs

What fires them

Upgrading to tuber's scheduler

Checking on it

Data Retention and Cleanup

Retention windows

When it runs

Running it now

Checking on it

Monitoring

Health Endpoint

Uptime Kuma Setup

Backup

What to Backup

Restoring

Model Context Protocol (MCP) Integration

Available MCP Tools

Usage Examples

Full List of Environment Variables

MCP (Model Context Protocol)

Data Retention

SQLite Page Cache

Tuber Work Queue

OIDC Authentication Setup

Quick Start

Provider-Specific Examples

Providers Without Discovery

How It Works

Security Features

Email Sending Setup

Development

Services

Email Previews

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages