[granbellllc-png]

Fixes #6942

Description

This PR addresses #6942 by providing a dedicated maintenance script to purge historical completed workflow executions and execution logs, preventing PVC disk exhaustion on self-hosted PostgreSQL deployments.

Problem

As reported in #6942, workflow execution history and step output TOAST data can exhaust the database storage PVC over time.

Solution

• Added scripts/cleanup_workflow_executions.py:
  • --retention-days: purge terminal workflow runs older than N days (default: 30)
  • --batch-size: chunked deletes (default: 500) to avoid table lock contention and WAL surges
  • --dry-run: non-destructive inspection mode
  • --tenant-id: per-tenant scope filter
  • --keep-failed: retain failure/timeout executions for audit
  • Deletes dependent WorkflowExecutionLog entries before deleting WorkflowExecution
• Added comprehensive unit tests in tests/test_cleanup_workflow_executions.py covering dry-run, actual purge, error preservation, and active execution isolation (3/3 passed).

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ granbellllc-png
❌ Alex Granbell

---

Alex Granbell seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[Danny5487401]

Thanks for implementing a concrete solution for #6942. A standalone maintenance script is a useful starting point. I reviewed commit 8ab1c6c and found four issues to address before using it against an existing database.

Validation scope: these reproductions used the upstream models and get_session implementation in an isolated harness with SQLModel 0.0.22 and SQLite, with foreign keys enabled for the deletion case. These are not PostgreSQL integration or performance test results.

1. The CLI entry point fails before cleanup, including with --dry-run (L120).

   get_session() is a generator dependency, not a context manager. Calling main() raises:

   TypeError: 'generator' object does not support the context manager protocol
   

   Please use a properly managed synchronous session, such as with get_session_sync() as session:, and add a CLI entry-point test.

2. Execution associations are not handled before deleting the parent (L90–L99).

   WorkflowToAlertExecution and WorkflowToIncidentExecution also reference WorkflowExecution. Deleting an associated execution raised:

   IntegrityError: NOT NULL constraint failed:
   workflowtoalertexecution.workflow_execution_id
   

   Please handle both association tables before deleting the execution, preserving the alert/incident records themselves. Verify the full dependency chain in PostgreSQL with actual constraints enabled.

3. Dry-run reports only the first batch (L72–L79).

   With five eligible executions and batch_size=2, it reports two and exits. This understates the proposed deletion scope.

   Please report complete eligible execution/dependency counts, or explicitly label a bounded preview as partial. Prefer aggregate counts over loading every log ID, and test a dataset larger than one batch.

4. Invalid retention values can delete fresh history (L113–L114).

   retention_days=-1 moves the cutoff into the future; the reproduction deleted a newly created successful execution.

   Please validate retention days and batch size before database access. Consider requiring an explicit apply flag and an explicit tenant/all-tenants choice.

Additional suggestions:

• Bound memory and transaction work. The code loads all logs for the selected executions, followed by complete execution objects containing potentially large results JSON. Limiting parent executions does not bound log volume. Prefer ID-based operations, a separate log-row budget, and a runtime limit.
• Add PostgreSQL tests. Cover associations, tenant isolation, multiple batches, invalid arguments, rollback/retry, and terminal statuses. Check providers_not_configured, which is terminal in Keep 0.55.0 but absent from the current lists.
• Document checkpoint behavior accurately. Default 30-day retention does not remove recent successful results used by normally running incremental workflows. Preserving the latest non-test success and scheduler-relevant completed record would still retain recovery evidence. The existing one-day history lookup limit remains a separate limitation.
• Document disk-space expectations. DELETE plus ordinary VACUUM generally makes space reusable inside PostgreSQL rather than immediately reducing filesystem usage. VACUUM FULL requires separate space and downtime planning.

I suggest keeping the first iteration focused on correctness fixes, bounded deletion, and PostgreSQL tests. Separate retention windows and scheduling can follow.

[Danny5487401]

Following up on the review, here is the concrete implementation plan I will work on for #6942, referencing this PR.

Read-only inspection of our self-hosted PostgreSQL 16 deployment found that execution history accounts for approximately 97.6% of database storage: workflowexecution is 27.53 GiB, including 26.24 GiB of TOAST, and workflowexecutionlog is 12.15 GiB. The log table currently has no index on workflow_execution_id; EXPLAIN for a single execution lookup produces a parallel sequential scan. This needs to be addressed before repeatedly deleting execution batches.

Proposed first iteration:

1. Add a PostgreSQL-compatible index migration for workflowexecutionlog.workflow_execution_id, with an explicit concurrent-build procedure and space/validity checks for existing large installations. Do not silently build a large index from the cleanup command.
2. Provide a dry-run-by-default maintenance CLI with an explicit apply mode and explicit tenant/all-tenants selection. Validate retention periods and batch/runtime limits before database access, and fix the synchronous session lifecycle.
3. Select eligible terminal executions without loading results. Support separate success/failure retention periods, preserve recent non-test successful executions and the scheduler-relevant last completed record per tenant/workflow, and skip active or ambiguous executions. Keep existing checkpoint lookup semantics unchanged.
4. Limit log deletions independently from parent execution batches. Remove log and alert/incident execution associations before the parent; preserve alerts and incidents. Commit bounded batches, report actual committed progress, and make interruption/retry safe. Avoid assuming is_running is a boolean.
5. Make dry-run report the full eligible scope, or explicitly identify an incomplete/time-limited report. Add PostgreSQL integration tests for real foreign keys, associations, tenant isolation, multiple batches, checkpoint/scheduler protection, invalid inputs, and rollback/retry, plus CLI tests.
6. Document backup/restore preparation, index prerequisites, staged catch-up, routine VACUUM/autovacuum, and monitoring. Ordinary VACUUM generally makes space reusable rather than immediately reducing filesystem usage; VACUUM FULL remains a separately planned maintenance operation.

For our deployment, success history of 14 days and failure history of 30 days, with protected executions excluded, currently covers about 532k executions and 13.55 GiB of stored results. This is not a promise of immediately reclaimed filesystem space and excludes log volume; retention remains configurable.

I will keep reducing newly persisted intermediate step results as a separate follow-up: Keep 0.55.0 currently saves both action and step outputs, while incremental workflows still need their successful checkpoint action results. That optimization should not be mixed into the first cleanup change or truncate checkpoints.

I will submit the implementation as a linked follow-up PR. No live database cleanup or index creation has been performed.

[granbellllc-png]

Thank you @Danny5487401 for the thorough review and for sharing the production PostgreSQL 16 storage telemetry (27.53 GiB on workflowexecution / 12.15 GiB on workflowexecutionlog)!

I have pushed commit db31adc6 addressing all four correctness issues and the bounded-execution / operational recommendations:

1. Synchronous CLI session management (L120): Switched main() to use with get_session_sync() as session: and added CLI entry-point tests covering both default --dry-run and explicit --apply modes.
2. Association tables & zero-TOAST-load bulk deletes (L90–L99): Before deleting WorkflowExecution rows, the cleanup now deletes child rows in WorkflowExecutionLog (bounded in --log-batch-size chunks, default 5000), WorkflowToAlertExecution, and WorkflowToIncidentExecution while preserving Alert and Incident records. Parent executions are deleted by ID via SQL DELETE without loading WorkflowExecution.results JSON into memory.
3. Complete aggregate counts in --dry-run (L72–L79): --dry-run now executes COUNT(...) aggregate queries over the full eligible subquery (executions_purged, logs_purged, alert_associations_purged, incident_associations_purged) without stopping at the first batch or loading log IDs into memory.
4. Upfront input validation, terminal statuses & safety flags (L113–L114):
   • Validates retention_days >= 1, failed_retention_days >= 1, batch_size >= 1, log_batch_size >= 1, and max_runtime_seconds > 0 before opening a DB session.
   • Added providers_not_configured to TERMINAL_FAILED_STATUSES and avoided assuming is_running is a boolean.
   • Added optional --failed-retention-days, --preserve-latest-per-workflow (retaining both the latest non-test success and the scheduler-relevant last completed execution per (tenant_id, workflow_id)), --max-runtime-seconds, and required explicit --tenant-id <id> or --all-tenants with --dry-run as the CLI default unless --apply is specified.
   • Documented the CREATE INDEX CONCURRENTLY prerequisite on workflowexecutionlog (workflow_execution_id) and VACUUM vs VACUUM FULL space reclamation behavior.