Skip to content

fix(cli): concurrent deploy telemetry events share an event id - #2129

Merged
9pace merged 8 commits into
mainfrom
paceben/fix/telemetry-event-id-race
Nov 6, 2026
Merged

9pace merged 8 commits into
mainfrom
paceben/fix/telemetry-event-id-race

Conversation

@9pace

@9pace 9pace commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

TelemetrySession.emit() increments this.count, then for DEPLOY events awaits trackDeployStatistics() (a read and write of the telemetry state file), and only then builds eventId from this.count. With --concurrency, other events are emitted during that await, so several DEPLOY events read the same final count and are sent with the same eventId.

Rn, we see telemetry data has repeated event IDs on CLI versions after the subprocess sender, mostly DEPLOY events in the same request with different contents.

This takes the event id and timestamp at the start of emit(), before the await.

Checklist

  • This change contains a major version upgrade for a dependency and I confirm all breaking changes are addressed
    • Release notes for the new version:

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

@github-actions

github-actions Bot commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

Scanned Files

None

@maintainer-for-aws maintainer-for-aws Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See the review summary comment for the overview; the notes below are inline.

Comment on lines +128 to +172
});

test('concurrent failed DEPLOY events count every sequential failure', async () => {
// GIVEN: an in-memory state file whose reads and writes take a while, so updates overlap
let state: telemetryState.TelemetryState = {};
const stateSpy = jest.spyOn(telemetryState, 'withTelemetryState').mockImplementation(async (block) => {
const copy = { ...state };
await new Promise((resolve) => setTimeout(resolve, 10));
const ret = block(copy);
state = copy;
return ret;
});

// WHEN
const error = { name: ToolkitError.name };
const first = session.emit({ eventType: 'DEPLOY', duration: 1234, error });
const second = session.emit({ eventType: 'DEPLOY', duration: 1234, error });
const third = session.emit({ eventType: 'DEPLOY', duration: 1234, error });
await first;
await second;
await third;
stateSpy.mockRestore();

// THEN
const failures = clientEmitSpy.mock.calls.map(([e]) => e.counters?.sequentialDeploymentFailures);
expect(failures.sort()).toEqual([1, 2, 3]);
expect(state.sequentialDeploymentFailures).toEqual(3);
});

test('DEPLOY event is still sent when the state file cannot be updated', async () => {
// GIVEN
const stateSpy = jest.spyOn(telemetryState, 'withTelemetryState').mockRejectedValue(new Error('EACCES'));

// WHEN
await session.emit({ eventType: 'DEPLOY', duration: 1234 });
stateSpy.mockRestore();

// THEN
expect(clientEmitSpy).toHaveBeenCalledTimes(1);
expect(clientEmitSpy.mock.calls[0][0]).toEqual(expect.objectContaining({
event: expect.objectContaining({ eventType: 'DEPLOY' }),
}));
expect(clientEmitSpy.mock.calls[0][0].counters?.sequentialDeploymentFailures).toBeUndefined();
});

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚪ Optional — Both new tests install a jest.spyOn(telemetryState, 'withTelemetryState') mock and call stateSpy.mockRestore() inline in the happy path — before the expect(...) assertions — with no afterEach/finally safety net. If an awaited emit() rejects, or an assertion placed before the restore throws, mockRestore() never runs and the mock persists into whichever test the randomized runner executes next (this repo runs tests randomized by default). That leaks global state and can cause confusing cross-test failures that do not reproduce deterministically.

Suggested change: Restore mocks in a guaranteed path: add afterEach(() => { jest.restoreAllMocks(); }); to the describe block, or wrap each test body in try { ... } finally { stateSpy.mockRestore(); }.

This branch had an error being deployed

1 failed (outdated) and 2 inactive deployments
run-tests — 0730b0da Deployed Nov 5, 2026 by 9pace via integ_cli (cli-integ-tests, 24.19, 5) #7641
no-approval — 0730b0da Deployed Nov 5, 2026 by 9pace via prepare #7641
automation — 0d9e90b7 Deployed Nov 5, 2026 by 9pace via Set AutoQueue on PR #2129 #3957
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants