feat(self-hosting): report on-prem usage to a Sim instance for valuation

[myxamediyar]

Summary

Self-hosted deployments run without a meter, so Sim has no view of what they consume. This adds an opt-in reporting path, off by default:

usage_log + workflow_execution_logs   (already written on every deployment)
        │  aggregate query, grouped by UTC day — no per-row data leaves Postgres
        ▼
lib/onprem-telemetry/collect.ts       buckets: one per day, counts + sums only
        │  POST, bearer = deployment API key, trailing N days every run
        ▼
POST /api/onprem-telemetry/report     upsert on (deployment_id, period_start)
        ▼
onprem_usage_report                   credits stored; dollars derived at read
        ▼
GET /api/v1/admin/onprem-telemetry/…  rows carry credits + applied rate + usd

Both halves live in this codebase: a self-hosted deployment is the sender, whichever Sim instance owns the admin API is the receiver.

Design notes

The collector reads; it does not instrument. recordUsage already writes usage_log on every deployment regardless of BILLING_ENABLED — its own comment calls the ledger the universal source of truth for cost including self-hosted. Workflow counts, status and duration are likewise already in workflow_execution_logs. Nothing about how a workflow runs changed.

Aggregated in SQL. Both queries GROUP BY day and return counts and sums, so per-execution rows, identifiers, inputs, outputs and tool names never reach the reporting code — the privacy property holds by construction, not by a filter someone must maintain. description is kept only where it names a model.

Separate from lib/core/telemetry.ts on purpose. That pipeline is fire-and-forget by design (trackPlatformEvent swallows errors; /api/telemetry forwards with a 5s abort and no retry). Right for product analytics, wrong for figures with a dollar value. This re-sends a trailing window every run and the receiver upserts, so a day is delivered at least once and converges without an outbox or local state.

Off the execution path, structurally. The cron endpoint is the reporter's only caller. Disabled means getOnPremTelemetryConfig() returns before any query or fetch — the first statement of the run, covered by a test asserting fetch is never called. A failing receiver yields a failed result and a log line.

Credits are the fact; dollars are derived. The receiver stores credits per day plus an append-only, effective-dated rate table. valueUsage joins each day to the rate whose effectiveFrom most recently precedes periodStart. A backdated rate re-values the days it now covers on the next read; no stored credit ever changes; days before the first rate report usd: null and sum into unvaluedCredits rather than being priced silently. Rates live on the receiver, so they are commercial terms the customer cannot edit and changing one needs no redeploy.

Minimal surface. Three tables, one schema-versioned wire contract shared by sender and receiver, one cron route, one ingest route, three admin routes. No new pool profile, no Redis lock (upserts make overlapping runs safe), no outbox (re-sending the window is the retry).

Full rationale and known limitations: apps/sim/lib/onprem-telemetry/README.md. Operator documentation: apps/docs/content/docs/platform/self-hosting/usage-telemetry.mdx.

Known limitations

• Cooperative, not enforced. Nothing proves a deployment reported completely; the customer controls the database and the flag. Enforcement needs attestation or licensing, which does not exist here.
• Copilot chat usage is absent on-prem. POST /api/billing/update-cost skips the cost update when BILLING_ENABLED is unset, so the four Sim-Chat-family sources read as zero. Recording those callbacks unconditionally is the natural follow-up.
• Ledger scan. usage_log has no index led by created_at; the window scan is sequential. The job runs every six hours to keep that cheap. A concurrent index on usage_log (created_at) is the fix if needed — not added here to avoid triggering an index build on Sim Cloud's ledger as a side effect.
• Day granularity, UTC. A day is partial until the first run after UTC midnight; reportedAt records when it was last sent.

Type of Change

• New feature

Testing

bunx vitest run lib/onprem-telemetry app/api/onprem-telemetry app/api/cron/onprem-usage-report — 5 files, 26 tests passing.

Unit coverage: day bucketing and the aggregate shape (collect.test.ts), the disabled-path assertion that fetch is never called plus failure handling (report.test.ts), rate selection including backdating and the pre-first-rate null case (rates.test.ts), and both routes. onprem-telemetry.integration.ts exercises sender → receiver → admin read end to end against a database.

Reviewers may want to focus on rates.ts (the effective-dating rule) and on the claim that the two queries in collect.ts cannot leak per-row data.

Checklist

• Code follows project style guidelines
• Self-reviewed my changes
• Tests added/updated and passing
• No new warnings introduced
• I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

🤖 Generated with Claude Code

[vercel]

@myxamediyar is attempting to deploy a commit to the Sim Team on Vercel.

A member of the Team first needs to authorize it.

[greptile-apps]

Confidence Score: 2/5

[Critical risk] Adds database schema and API endpoints for on-prem usage reporting.

The PR is not ready to merge until first-day valuation, stale-report overwrites, and partial registration are addressed; its test assertions must also satisfy the repository requirement.

Findings

1.  First day goes unvalued ▶
2.  Older reports overwrite newer totals ▶
3.  Failed registration strands deployment ▶
4.  Tests assert mock calls ▶
5.  Usage reads have no bound ▶

Summary

Adds opt-in self-hosted usage reporting: a scheduled collector aggregates daily ledger and execution figures, sends them to an authenticated receiver, and exposes effective-dated dollar valuation through admin APIs.

• Initial-rate timing can leave the first reporting day unvalued.
• Out-of-order deliveries can regress stored totals, and failed registration can leave an unusable deployment.
• The usage read needs a bounded response; new tests also conflict with the repository’s mock-assertion rule.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  A[usage_log and execution logs] --> B[Daily SQL aggregates]
  B --> C[Scheduled sender]
  C -->|Deployment API key| D[Report receiver]
  D --> E[Daily report upsert]
  E --> F[Admin usage read]
  G[Effective-dated rates] --> F

Loading

_{Reviews (1) · Last reviewed commit: "feat(self-hosting): report on-prem usage..."}

[greptile-apps]

First day goes unvalued When a deployment is registered with an initial rate after UTC midnight, that rate takes effect at the current instant, but usage is valued at the start of each day. The first day's usage therefore has no applicable rate and is reported as unvalued, even though the deployment shows a current rate. Make the initial rate cover that day, or require the operator to choose when valuation begins.

[greptile-apps]

Older reports overwrite newer totals If overlapping reports arrive out of order, this upsert replaces every figure without checking when the report was made. An older partial snapshot can overwrite a newer day's totals; after that day leaves the lookback window, the understated figures persist. Reject stale updates or enforce report ordering.

[greptile-apps]

Failed registration strands deployment When an initial rate is requested, the deployment is inserted before the rate. If the rate insert fails, registration returns an error but leaves the deployment stored without ever returning its plaintext API key. Retrying the requested ID then conflicts. Create both records atomically.

[greptile-apps]

Tests assert mock calls This test checks whether fetchMock was called, rather than checking an observable outcome. The repository's testing directive says never to write tests that assert mock calls; the same pattern appears in the receiver and reporter tests. Replace these assertions with boundary-level checks before merging to satisfy that requirement.

Context Used: CLAUDE.md (source)

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

[greptile-apps]

Usage reads have no bound The admin API accepts an unrestricted date range and loads every matching report before valuing and returning it. As a deployment's history grows, one wide-range request can produce a large, slow response. Bound or paginate the returned days while retaining range totals.