Repository navigation
[Feature]: PostgreSQL workflow execution/log retention and supported cleanup guidance (PVC exhaustion) #6942
Description
Is your feature request related to a problem? Please describe.
Our self-hosted Kubernetes deployment exhausted a 40 GiB PostgreSQL data PVC while accumulating workflow execution history. The largest allocations were execution results and execution logs, rather than WAL or alert history alone.
Approximate physical relation sizes observed during the incident:
| Component | Approximate size |
|---|---|
workflowexecution TOAST storage (large execution results) |
24.6 GiB |
workflowexecutionlog heap |
9.9 GiB |
An index on workflowexecutionlog |
1.4 GiB |
pg_wal |
81 MiB |
These are approximate on-disk relation measurements from the incident investigation, not a fresh pg_total_relation_size report or row-count benchmark. The PVC was subsequently expanded to 50 GiB to provide headroom, but this does not bound future growth.
Environment: self-hosted Keep OSS on Kubernetes/Helm, PostgreSQL 16. We currently run Keep v0.55.0 with Helm chart 0.1.97. The disk-full incident occurred before the upgrade to v0.55.0; this report is not claiming a v0.55.0 regression. Workloads include scheduled workflows that persist substantial step outputs in execution history.
Describe the solution you'd like
Could maintainers clarify the supported approach for retaining or purging PostgreSQL workflow execution history?
- Is there a currently supported script, API, CLI command, or configuration option for
workflowexecutionandworkflowexecutionlogon v0.55.0? If there is one, please point us to it. - If not, would a documented, opt-in retention job for completed execution history be appropriate? Useful controls would include a dry-run, retention age, bounded batches/transactions, tenant isolation, and a concurrency guard.
- Which execution-to-alert/incident associations and other dependent records must be handled, and in what order?
- What should be preserved for active/queued/waiting executions and workflows that use previous successful outputs as incremental checkpoints? We want to avoid resetting a polling cursor or breaking a resumable workflow.
- Are the historical batch-delete proposal (chore: batch_delete executions script #4136) or a follow-up intended to cover this use case, and are there plans to document PostgreSQL vacuuming/space-reuse expectations after cleanup?
A supported maintenance script and clear operational guidance would already help; this does not require a frontend change.
Describe alternatives you've considered
- Increasing the PVC provides temporary headroom but does not supply retention.
- The cleanup CronJob referenced by a maintainer in [➕ Feature]: Time-based automatic alert deletion (retention policy) #4802 invokes MySQL and cleans alert-related tables; it does not provide PostgreSQL execution-history cleanup.
- feat: time-based alert retention policy #6616 proposes database-agnostic alert retention. Its current scope does not address the two dominant workflow-history allocations above.
- We have avoided applying an ad-hoc deletion script during this investigation because dependency handling, resumable executions, checkpoints, and schema compatibility need clarification.
Additional context / related issues and PRs
- [➕ Feature]: Add a way delete alerts (database is growing forever) #6524 — broader unbounded database growth report, also mentioning PostgreSQL and workflow execution logs. This report adds a concrete execution/TOAST/log breakdown and asks specifically for supported execution-history cleanup guidance.
- [➕ Feature]: Time-based automatic alert deletion (retention policy) #4802 — alert retention feature request and the maintainer's link to the existing MySQL cleanup CronJob.
- feat: time-based alert retention policy #6616 — proposed time-based alert retention; related but does not clean workflow execution history.
- [internal]: batch delete workflow executions #4135 — historical batch-delete workflow execution task (closed).
- chore: batch_delete executions script #4136 — historical batch-delete execution script proposal (closed without merge). Is any part still recommended for current PostgreSQL deployments?
- [🐛 Bug][Perf]: Workflows list count query materializes a hard-coded 30-day window function over all
workflowexecutionrows — CPU-bound on MySQL at scale #6876 — execution-history growth also affects workflow-list query performance. That report uses MySQL; we are linking it as a related scaling concern, not claiming to have reproduced its query plan on PostgreSQL.
If maintainers prefer extending #6524 instead of tracking this narrower PostgreSQL execution-history question separately, please let us know.
Activity
shahargl commentedon Nov 11, 2026
hey @Danny5487401, ty for the ticket. feel free to implement w/e solution that works for u and i'll review it.
feat(retention): add workflow execution and log cleanup maintenance s…
Danny5487401 commentedon Nov 12, 2026
Following up on this issue and the review of #6959, here is the concrete implementation plan I am working on for PostgreSQL workflow execution/log retention.
Read-only inspection of our self-hosted PostgreSQL 16 deployment found that execution history accounts for approximately 97.6% of database storage:
workflowexecutionis 27.53 GiB, including 26.24 GiB of TOAST, andworkflowexecutionlogis 12.15 GiB. The log table currently has no index onworkflow_execution_id; EXPLAIN for a single execution lookup produces a parallel sequential scan. This needs to be addressed before repeatedly deleting execution batches.Proposed first iteration:
workflowexecutionlog.workflow_execution_id, with an explicit concurrent-build procedure and space/validity checks for existing large installations. Do not silently build a large index from the cleanup command.results. Support separate success/failure retention periods, preserve recent non-test successful executions and the scheduler-relevant last completed record per tenant/workflow, and skip active or ambiguous executions. Keep existing checkpoint lookup semantics unchanged.is_runningis a boolean.For our deployment, success history of 14 days and failure history of 30 days, with protected executions excluded, currently covers about 532k executions and 13.55 GiB of stored results. This is not a promise of immediately reclaimed filesystem space and excludes log volume; retention remains configurable.
I will keep reducing newly persisted intermediate step results as a separate follow-up: Keep 0.55.0 currently saves both action and step outputs, while incremental workflows still need their successful checkpoint action results. That optimization should not be mixed into the first cleanup change or truncate checkpoints.
I will submit the implementation as a linked follow-up PR. No live database cleanup or index creation has been performed.
Implementation submitted: #6961
Local validation: 44 focused tests passed (PostgreSQL 16.2 and SQLite migration compatibility), plus Ruff/Black/isort and documentation navigation checks. A dedicated PostgreSQL 16.14, Python 3.11/3.13 CI workflow is included; remote results are separate from the local test results above. No existing deployment was changed.
feat(retention): add workflow execution and log cleanup maintenance s…
fix(retention): address review feedback on session sync, associations…