Skip to content

[local-test-e2e-cover-sentry-error-monitor-auto] test(e2e): cover Sentry error-monitor auto-fix against an offline fake API #363

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
danii1 wants to merge 3 commits into
base: main
Choose a base branch
from
+537 −42
Open

[local-test-e2e-cover-sentry-error-monitor-auto] test(e2e): cover Sentry error-monitor auto-fix against an offline fake API #363

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
dc0f039
feat: implement local-test-e2e-cover-sentry-error-monitor-auto - test…
danii1 Nov 3, 2026
8de64e3
fix: address PR review feedback
danii1 Nov 3, 2026
47a9448
Merge remote-tracking branch 'origin/main' into feature/local-test-e2…
danii1 Nov 4, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 27 additions & 0 deletions packages/e2e/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -337,6 +337,33 @@ A second run with `--skip-comments` proves no additional comment or transition
is posted. The local bare Git remote is the code host; no GitHub token or PR is
required.

## Offline fake Sentry error monitor

The default local suite also runs `tests/local/worker-sentry-monitor.test.ts`,
which starts the real worker daemon with an `[[error_monitors]]` Sentry source
pointed at an in-process fake. The fake binds `127.0.0.1:0`, accepts any Bearer
token (and never reads or logs it), and implements only the project issue-list
and issue-comment endpoints the product calls. Any other route, or an
unexpected organization/project/issue id, returns a diagnostic 404 and fails
the test.

The scenario seeds one unresolved issue and waits for exactly one
`error_monitor` run. The seeded issue title carries a unique marker that the
deterministic agent requires to be present in its own input, so a regression
that drops the Sentry payload from the prompt fails the run instead of silently
editing the fixture. It also asserts the pushed `feature/` branch contains only
the deterministic `src/greeting.ts` edit, the generated `error-fixes/` markdown
carries the untrusted-content block plus the issue title/culprit and no token,
and the fake received one success comment. Two further polls of the same issue
must not create a second run or comment, proving the durable `issue:<id>`
dedupe. No Sentry account, token, project, or network access is used. Run it
with:

```sh
bun run --filter @getdevintern/code build
bun run --filter @devintern/e2e test
```

## Remote tracker fixtures

The Markdown suites above need no task tracker. Separate opt-in suites exercise
Expand Down
29 changes: 15 additions & 14 deletions packages/e2e/docs/coverage-matrix.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ partially exercises, and what is still missing. The `e2e-expander` workspace
automation scouts this file and delegates the highest-value gaps to the worker
as separate ad-hoc tasks. Each delegated task updates this file in its own PR.

Last reviewed: 2026-11-03.
Last reviewed: 2026-11-04.

## Status legend

Expand Down Expand Up @@ -99,17 +99,17 @@ Run: `bun run --filter @devintern/e2e test` (build the CLI first); the sandbox s

### Relay, auth, licensing, and observability services

| Behavior | Status | Evidence | Missing |
| ---------------------------------------------------------------------------------------------------------------------- | --------- | ---------------------------------------------------------------------------- | ----------------------------------------------------------------------------- |
| GitLab relay: delivery, MR-close unwatch, `--disconnect` | `live` | `tests/code/gitlab-relay.test.ts` | GitHub relay delivery and tracker webhooks. |
| `worker connect` targets (`github`, `gitlab`, `linear`, `asana`, `trello`, `azure-devops`, `jira`, `sentry`, `status`) | `gap` | — | Validation, `status`, and non-mutating paths; relay targets stay live/opt-in. |
| Supabase login/logout/whoami and session refresh | `gap` | — | Safe local fake or explicit opt-in account suite. |
| Automation license gate, Worker Pilot trial, offline/expired behavior | `gap` | — | Tests currently disable the gate. |
| Sentry error-monitor polling and auto-fix | `gap` | — | Fake Sentry API or opt-in project; create one fix run and dedupe. |
| Dashboard standalone server/API, run history, retry, run-now | `gap` | — | Fixture queue DB plus HTTP assertions. |
| Worker/dashboard logs and artifacts | `partial` | asserted indirectly by worker tests | Direct readback and redaction checks. |
| Direct webhook server (`webhook serve`) and signature validation | `default` | `tests/local/webhook-serve.test.ts` (`bun run --filter @devintern/e2e test`) | Live provider callbacks and event-specific pipelines; loopback HTTP only. |
| Anonymous telemetry opt-out and update check | `gap` | — | Tests disable both; assert no network when disabled. |
| Behavior | Status | Evidence | Missing |
| ---------------------------------------------------------------------------------------------------------------------- | --------- | ---------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------ |
| GitLab relay: delivery, MR-close unwatch, `--disconnect` | `live` | `tests/code/gitlab-relay.test.ts` | GitHub relay delivery and tracker webhooks. |
| `worker connect` targets (`github`, `gitlab`, `linear`, `asana`, `trello`, `azure-devops`, `jira`, `sentry`, `status`) | `gap` | — | Validation, `status`, and non-mutating paths; relay targets stay live/opt-in. |
| Supabase login/logout/whoami and session refresh | `gap` | — | Safe local fake or explicit opt-in account suite. |
| Automation license gate, Worker Pilot trial, offline/expired behavior | `gap` | — | Tests currently disable the gate. |
| Sentry error-monitor polling and auto-fix | `default` | `tests/local/worker-sentry-monitor.test.ts` | Live Sentry project; issue status resolution after deployment; multiple concurrent issues. |
| Dashboard standalone server/API, run history, retry, run-now | `gap` | — | Fixture queue DB plus HTTP assertions. |
| Worker/dashboard logs and artifacts | `partial` | asserted indirectly by worker tests | Direct readback and redaction checks. |
| Direct webhook server (`webhook serve`) and signature validation | `default` | `tests/local/webhook-serve.test.ts` (`bun run --filter @devintern/e2e test`) | Live provider callbacks and event-specific pipelines; loopback HTTP only. |
| Anonymous telemetry opt-out and update check | `gap` | — | Tests disable both; assert no network when disabled. |

## `devintern` command surface

Expand Down Expand Up @@ -154,7 +154,7 @@ Run: `bun run --filter @devintern/e2e test` (build the CLI first); the sandbox s
| Harness failover | `default` | `tests/local/worker-harness-failover.test.ts` | Failback after the reset window and restart-time restore of the persisted chain. |
| Review polling, CI repair, conflict resolution | `live` | GitLab worker suite | GitHub parity and scheduled/manual modes. |
| Relay instant events | `live` | GitLab relay suite | GitHub relay and tracker webhooks. |
| Sentry error monitors | `gap` | — | Fake Sentry API or opt-in project. |
| Sentry error monitors | `default` | `tests/local/worker-sentry-monitor.test.ts` | Live Sentry project; issue status resolution after deployment. |
| Dashboard embedded in the worker | `gap` | — | API and run-now/retry. |
| Native service install/uninstall | `gap` | — | systemd/launchd path without touching the host service manager. |
| Auto-update | `gap` | — | Disabled in tests; simulate an available update safely. |
Expand Down Expand Up @@ -204,7 +204,8 @@ and each task opens its own PR.
recovery on a deterministic local workspace.
14. ~~Harness failover with stub harnesses~~ (implemented by
`tests/local/worker-harness-failover.test.ts`).
15. Sentry auto-fix with a fake or opt-in project.
15. ~~Sentry auto-fix with a fake or opt-in project~~ (implemented by
`tests/local/worker-sentry-monitor.test.ts`).
16. Auth/license (`login`, `logout`, `whoami`, trial, expired/offline) with a
safe fake or explicit account suite.
17. Linear, Asana, and Azure DevOps read-only checks, then fake lifecycles.
Expand Down
66 changes: 38 additions & 28 deletions packages/e2e/helpers/worker.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,7 @@ const ACTIVE_HARNESS_KEY = "failover:active_harness";
export interface WorkerRunRecord {
id: number;
task_key: string | null;
origin: string;
status: string;
branch: string | null;
repo: string | null;
Expand All @@ -92,6 +93,12 @@ export interface WorkerFixtureOptions {
maxConcurrency?: number;
/** Per-repository agent concurrency; defaults to one. */
maxConcurrencyPerRepo?: number;
/**
* Extra TOML appended to the generated `workspace.toml` (for example an
* `[[error_monitors]]` source). The default tables are kept unchanged, so a
* caller only supplies the additional section.
*/
workspaceTomlExtra?: string;
}

/** Log captured from the worker subprocess, for readiness and failure reports. */
Expand Down Expand Up @@ -178,10 +185,13 @@ export class WorkerFixture {
/**
* Seed the local bare repository and write a credential-free workspace.
*
* The workspace deliberately has no task query, automation, estimation, or
* error monitor. That is valid only while an injected task is pending, so
* enqueue work before {@link start}: the worker then takes the documented
* injection-only exception instead of exiting for lack of configured work.
* The default workspace deliberately has no task query, automation,
* estimation, or error monitor. That is valid only while an injected task is
* pending, so enqueue work before {@link start}: the worker then takes the
* documented injection-only exception instead of exiting for lack of
* configured work. {@link WorkerFixtureOptions.workspaceTomlExtra} can add a
* source such as `[[error_monitors]]`, which counts as configured work on its
* own.
*/
async prepare(): Promise<void> {
const agent = join(this.root, "agent.ts");
Expand All @@ -191,28 +201,28 @@ export class WorkerFixture {
this.env.E2E_AGENT_INVOCATIONS = join(this.root, "agent-invocations.jsonl");

mkdirSync(this.workspaceDir, { recursive: true });
writeFileSync(
join(this.workspaceDir, "workspace.toml"),
[
"[workspace]",
"dashboard = false",
"",
"[workspace.execution]",
'isolation = "best_effort_host"',
`max_concurrency = ${this.options.maxConcurrency ?? 1}`,
`max_concurrency_per_repo = ${this.options.maxConcurrencyPerRepo ?? 1}`,
"",
"[defaults]",
'tracker = "markdown"',
'worker_task_args = "--create-pr --skip-clarity-check --skip-comments"',
"",
"[[repos]]",
`name = "${this.repoName}"`,
`remote = "${this.remote}"`,
`default_branch = "${this.baseBranch}"`,
"",
].join("\n"),
);
const workspaceToml = [
"[workspace]",
"dashboard = false",
"",
"[workspace.execution]",
'isolation = "best_effort_host"',
`max_concurrency = ${this.options.maxConcurrency ?? 1}`,
`max_concurrency_per_repo = ${this.options.maxConcurrencyPerRepo ?? 1}`,
"",
"[defaults]",
'tracker = "markdown"',
'worker_task_args = "--create-pr --skip-clarity-check --skip-comments"',
"",
"[[repos]]",
`name = "${this.repoName}"`,
`remote = "${this.remote}"`,
`default_branch = "${this.baseBranch}"`,
"",
];
const extraToml = this.options.workspaceTomlExtra?.trim();
if (extraToml) workspaceToml.push(extraToml, "");
writeFileSync(join(this.workspaceDir, "workspace.toml"), workspaceToml.join("\n"));

cpSync(fixture, this.seed, { recursive: true });
await checkedCommand(
Expand Down Expand Up @@ -383,10 +393,10 @@ export class WorkerFixture {
runRecords(taskKey?: string): WorkerRunRecord[] {
return taskKey === undefined
? this.query<WorkerRunRecord>(
"SELECT id, task_key, status, branch, repo, pr_url, outcome_reason, attempt FROM runs ORDER BY id",
"SELECT id, task_key, origin, status, branch, repo, pr_url, outcome_reason, attempt FROM runs ORDER BY id",
)
: this.query<WorkerRunRecord>(
"SELECT id, task_key, status, branch, repo, pr_url, outcome_reason, attempt FROM runs WHERE task_key = ? ORDER BY id",
"SELECT id, task_key, origin, status, branch, repo, pr_url, outcome_reason, attempt FROM runs WHERE task_key = ? ORDER BY id",
[taskKey],
);
}
Expand Down
Loading
Loading