[MSami625]

What Changed?

Imports scale (Linear imports epic) on branch scaling-platform: backend and frontend changes so large CSV/platform imports, diarisation, and evaluations (100k+ rows) stay usable without timing out or melting Postgres/Redis/FlexPrice.

• API structure: Split monolithic call_imports / call_import_evaluations routes into modules; bulk cancel/retry/delete with server-side row filters (not huge UUID lists from the browser).
• Uploads: Presigned CSV staging + manual audio presign (parallel PUTs); legacy in-API upload paths removed or deprecated.
• Platform imports: Hydrate/discovery pipeline fixes (_hand_off), eval allowed when rows are terminal even if discovery is still listing; stale slice reclaim + finalize when hydrate is idle; debounced rollup / batch status.
• Workers & concurrency: Fair import/platform/eval dispatch, shared import retry backoff for telephony/eval slots; Celery revoke fix for thread pool aborts; eager worker import guard.
• Sharding: Scatter/gather for filtered rows, eval list pages, aggregates; keyset-sampled completed rows for TLDR / clusters / user insights.
• Charts / aggregates: Migration 109 cached metric aggregates; on-demand refresh on aggregate GET (no every-25-rows worker rebuild); terminal-only cache rebuild on worker.
• Search: Migration 111 pg_trgm + row_text_search (escaped ILIKE) on conversation id + transcripts across eval rows and import filters.
• Billing: FlexPrice Redis buffer (ack after successful ingest, requeue on failure); batched recording minutes (~50 events per large eval); buffered eval pass deltas.
• UX polling: Active import/eval polling (importPolling, bulk-op progress, any_evaluation_poll_active on import detail/list).
• Data: recording_duration_seconds on rows (migrations 108/110).

(This branch may include other work outside imports; the above is the imports-scale scope for this PR.)

Why?

Imports were hitting limits at tens/hundreds of thousands of rows: full-table scans for search and charts, discovery/import stuck in processing, browser-driven bulk ops, upload timeouts, FlexPrice event storms, and insights/clusters loading every row pair. This PR keeps behavior backward-compatible while moving work to presign + worker paging + cache/sampling + indexes + safer billing.

How to Test?

1. Migrations: Apply 108–111 on catalog + shard DBs; restart API, import-control, imports, eval, usage workers.
2. CSV: Presign stage → map → diarise/eval on a medium batch; confirm no CORS/401 on PUT.
3. Platform: ElevenLabs (or similar) import; rows finish hydrate; eval can run before last discovery page; import leaves processing when rows idle.
4. Scale UX: Large eval — bulk abort/retry with filter; list/search q; charts tab while running (cache stale → refresh on open).
5. FlexPrice (if enabled): large eval completes; recording minutes not ~1 event/row; buffer survives failed ingest then retries.
6. Automated:
   .venv/bin/pytest tests/test_workers/test_call_import_worker_modules.py tests/test_services/test_platform_call_imports.py tests/test_services/test_billing/test_flexprice_buffer.py tests/test_services/test_recording_minutes_batch_size.py tests/test_services/test_metric_aggregate_cache.py tests/test_api/test_call_import_evaluation_rows_search.py tests/test_workers/test_evaluate_call_import_row.py -k rollup_parent -q

Release Label

• major
• minor — backward-compatible imports-scale features and performance
• fix
• No label

Checklist

• I have read the CONTRIBUTING.md guide.
• My code follows the project's style guidelines.
• I have added tests that prove my fix is effective or my feature works.
• I have updated documentation where needed. (S3 CORS example script exists; ops note: migrations 108–111 + worker restarts.)

[greptile-apps]

Confidence Score: 5/5

[High impact] The changes since the previous review appear safe to merge, and the numbered finding is fixed.

Summary

This PR scales call imports and evaluations with direct storage uploads, server-side bulk actions, sampled analysis, cached charts, and buffered billing. It also adds hosted-agent simulations, knowledge bases, and tool connections.

• Large call imports upload, search, and evaluate in manageable batches.
• Large evaluations send FlexPrice usage in buffered batches.
• Teams can run and inspect Harbor trials for agents.
• Agents can use connected tools and search attached knowledge bases.
• Metric and agent workflows gain scenario links, scoped choices, and favorites.

_{Reviews (9) · Last reviewed commit: "feat: enhance SQL masking logic to suppo..." · Reviewed by Greptile}

[MSami625]

@greptile-apps

[MSami625]

@greptile-apps