fix(cli): deploy hangs forever when an AWS call gets no response by abhiramshaji-cyber · Pull Request #2105 · aws/aws-cdk-cli

abhiramshaji-cyber · 2026-10-29T06:47:33Z

A deploy that got no answer from AWS used to wait until the job was killed, and now each silent call is dropped after 5 minutes and retried.

What this code is for

sdkRequestHandler in packages/@aws-cdk/toolkit-lib/lib/api/aws-auth/awscli-compatible.ts builds the HTTP settings that every AWS SDK client in the CLI and in the toolkit library uses. Both packages/aws-cdk/lib/cli/cli.ts (with the proxy agent, if any) and Toolkit.sdkProvider in toolkit.ts (with sdkConfig.httpOptions.agent, if any) call it, and the result is passed to every client next to a ConfiguredRetryStrategy that retries timeouts and other network errors. The one exception after this PR is invokeCommand in SDK.lambda() in sdk.ts, which uses a longer idle timeout.

Before this PR

A GitHub Actions job runs cdk deploy, and the connection under one DescribeStacks call goes silent after the request was sent, as described in #2097.

The CloudFormation client sends DescribeStacks through NodeHttpHandler with connectionTimeout: 10000 and requestTimeout: 300000.
The connection was already open, so connectionTimeout never applies.
After 300000 ms the requestTimeout passes. throwOnRequestTimeout is not set, so Smithy only logs [WARN] a request has exceeded the configured 300000 ms requestTimeout.
The request is never failed, so ConfiguredRetryStrategy never sees an error and never retries.
The CLI waits on that same request with no further output. In the report on (cli): deploy waits forever when an AWS API call gets no response, because requestTimeout only logs a warning #2097 that was almost 6 hours, until the GitHub job timeout cancelled it. Against a local server that accepts and never replies, I saw the same thing on main: one connection, the warning at 300 s, and still waiting at 330 s.

After this PR

Same job, same silent DescribeStacks call.

Same as above, but the settings now also carry socketTimeout: 300000.
Once no data has moved in either direction for 300 seconds, Smithy destroys the request and rejects it with a TimeoutError.
ConfiguredRetryStrategy treats TimeoutError as a temporary error and sends the call again on a new connection.
If the new connection answers, the deploy carries on. If every attempt stays silent, the command fails with that TimeoutError instead of waiting. There are 7 attempts in total, so that is about 35 minutes plus the short retry delays. Against the same silent local server with a 60 s value, I saw the call retried on connections 2 to 7 and then failed at about 467 s, which is the same path with a shorter timer.
A slow call that keeps sending or receiving data, such as a large asset upload, never goes 300 seconds without data, so it is not cut off.
A hotswap of a BucketDeployment invokes its Lambda function synchronously, and Lambda sends nothing back until the function returns, which can take up to 15 minutes. Invoke therefore gets socketTimeout: 960000, which covers that maximum plus cold start init, so a long s3 sync is never dropped and started again while it is still running.

The diagram

flowchart LR

classDef box fill:#eef1ff,stroke:#4c5bd4,color:#111827
classDef bad fill:#ffd6d6,stroke:#cc0000,color:#111827,stroke-width:3px
classDef good fill:#c8f0c8,stroke:#008800,color:#111827,stroke-width:3px

subgraph BEFORE["BEFORE: waits until the job is killed, 1 connection"]
direction TB
  B1["cdk deploy calls DescribeStacks"] --> B2["NodeHttpHandler sends the request"]
  B2 --> B3["connection goes silent"]
  B3 --> B4{"requestTimeout 300000 passes"}
  B4 -->|"throwOnRequestTimeout not set"| B5["logs WARN and keeps waiting forever"]
  B4 -->|"response arrives"| B6["deploy continues"]
end

subgraph AFTER["AFTER: retried every 300 s, then fails with TimeoutError"]
direction TB
  A1["cdk deploy calls DescribeStacks"] --> A2["NodeHttpHandler sends the request"]
  A2 --> A3["connection goes silent"]
  A3 --> A4{"THE CHANGE: socketTimeout 300000"}
  A4 -->|"no data for 300 s"| A5["TimeoutError, retried on a new connection"]
  A4 -->|"data still moving"| A6["slow upload keeps going"]
end

B1 ~~~ A1

class B1,B2,B3,B6 box
class A1,A2,A3,A6 box
class B4,B5 bad
class A4,A5 good

linkStyle 3 stroke:#cc0000,stroke-width:5px
linkStyle 8 stroke:#008800,stroke-width:5px

style BEFORE fill:#fff5f5,stroke:#cc0000,color:#111827
style AFTER fill:#f3fff3,stroke:#008800,color:#111827

The change

sdkRequestHandler now returns socketTimeout: 300000 next to the existing connectionTimeout and requestTimeout. This is the fix. I reused the 300 seconds that requestTimeout already has, because the Smithy docs for throwOnRequestTimeout say requestTimeout "was for a long time incorrectly being set as a socket idle timeout". So the CLI already ran with a 5 minute idle timeout that threw for years, and this brings that behavior back on purpose.
SDK.lambda() in sdk.ts sends Invoke through a second client, created on the first invoke, with socketTimeout: 960000 and every other setting unchanged. If the caller already set socketTimeout to 0 (no timeout) or to something longer, Invoke keeps that value. The other Lambda calls, such as UpdateFunctionCode and the hotswap waiters, keep the shared 300 s value, so a silent one of those is not held for 16 minutes per attempt. The hotswap of a BucketDeployment calls Invoke synchronously and the function may run for the full 15 minutes, so a shorter idle timeout would drop it and the retries would start overlapping s3 sync --delete runs.
RequestHandlerSettings in base-credentials.ts gets an optional socketTimeout field, so the value type checks and a custom IBaseCredentialsProvider receives it in clientConfig.requestHandler too.
requestTimeout and its warning are left as they are. I did not turn on throwOnRequestTimeout, because that timer runs from the start of the request until the response headers arrive, so it would fail a healthy upload that takes longer than 5 minutes. @JimNero009 pointed this out in the issue, and their suggestion of a socketTimeout is what this PR follows. Thank you for the very clear report and reproduction.
Tests in awscli-compatible.test.ts: one checks the settings sdkRequestHandler returns, including the agent on both httpAgent and httpsAgent, and fails without this change. Two more run a real STSClient against a local TCP server with a 1 s socketTimeout. To be honest about what these two prove: they pass on main too, because they check how Smithy handles socketTimeout rather than this change. I kept them as a guard in case a future Smithy version changes that behavior, but I am very happy to drop them if you would rather not have a TCP server in a unit suite: a server that never answers gets a second connection and the call rejects with TimeoutError; a server that sends its whole response, headers included, in small pieces every 50 ms still succeeds on one connection. That second test fails if the settings use requestTimeout with throwOnRequestTimeout instead, which is why I went with socketTimeout.
Tests in sdk.test.ts: one checks that invokeCommand is sent through a client with socketTimeout: 960000 and the other settings unchanged, and fails without this change; four check that a disabled or longer caller value is kept and a shorter or unset one is raised to 960000; one checks the Invoke client is created only on the first invoke; two check that other Lambda calls and the CloudFormation client still get the shared settings.

I would really value your view on a few choices, and I am happy to change any of them:

300 seconds restores the old behavior. Would you prefer something shorter, such as the 60 to 120 seconds the issue suggested? If so, I would be glad to check the other clients for calls that can stay silent that long, the way the Lambda one does.
The issue also asks for these timeouts to be configurable. I left that out to keep this small, but I can add it here or in a follow up if you think it belongs.
The standalone cdk-assets CLI builds its clients in cdk-assets-lib/lib/aws.ts without any of these timeouts. Would you like the same change there as a separate PR?

What changes for a user

A deploy, bootstrap or any other command whose AWS call goes silent no longer hangs. Each silent attempt is dropped after 5 minutes and retried.
If AWS never answers at all, the command now fails with a TimeoutError after the existing retries, about 35 minutes, instead of running until the CI job is killed.
Calls that answer normally, and slow calls that keep moving data such as large asset uploads, behave exactly as before.
cdk deploy --hotswap and cdk watch with a BucketDeployment that takes many minutes keep working, because Invoke waits up to 16 minutes.
A call that changes state, such as ExecuteChangeSet or Invoke, is now retried after a silent timeout instead of hanging. If AWS did process the first attempt, the retry can fail with a clearer error such as an existing change set. This is the same tradeoff the SDK already makes for a reset connection.
Proxy users are covered, because the proxy agent goes through the same sdkRequestHandler.
The requestTimeout warning after 5 minutes still appears for very long uploads, as it does today.

Fixes #2097

Checklist

This change contains a major version upgrade for a dependency and I confirm all breaking changes are addressed
- Release notes for the new version:

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

Every SDK client gets a requestTimeout of 5 minutes, but without throwOnRequestTimeout Smithy only logs a warning when it passes. The request is never failed, so the retry strategy never sees it, and a connection that dies silently keeps the CLI waiting until something outside kills it. Add a 60 second socketTimeout instead. It fires only when no data has moved in either direction for that long, so slow but healthy calls such as large asset uploads keep working, which a total request timeout would cut off. The resulting TimeoutError is classified as transient, so the existing retry strategy retries it on a new connection and fails the command once retries run out. 60 seconds matches the AWS CLI read timeout default. Both the CLI and the toolkit library build their request handler through sdkRequestHandler, with or without a proxy agent, so this covers both. The field is added to RequestHandlerSettings so custom base credential providers receive it too. Closes aws#2097

A 60 second idle timeout on every client would cut off a hotswap of a BucketDeployment. Hotswap invokes its custom resource function synchronously, and Lambda sends nothing on the connection until the function returns, which can take up to 15 minutes. The retry strategy would then start up to 7 overlapping runs of the same sync before failing a command that works today. The shared idle timeout is now 300 seconds, the same value requestTimeout already had. Older Smithy versions applied requestTimeout as an idle timeout, so this restores the behavior the CLI ran with for years while still turning an endless hang into a bounded failure. The Lambda client gets 905 seconds, just above the Lambda maximum, so a long running invoke is never dropped and retried while it is still working.

# Conflicts: # packages/@aws-cdk/toolkit-lib/test/api/aws-auth/sdk.test.ts

abhiramshaji-cyber · 2026-11-05T15:22:41Z

Hi, hope you're doing well! I merged main into this branch in dc58f94 to clear the conflict in packages/@aws-cdk/toolkit-lib/test/api/aws-auth/sdk.test.ts, and the fix itself is unchanged. Whenever someone has a moment, could the integ workflow be approved so it can run? Happy to change anything you'd prefer.

Raise the Lambda client socket timeout from 905s to 960s so a function that runs to the 15 minute maximum after a cold start is not dropped and invoked again. Widen the margin in the slow response test so timer delay on busy CI runners cannot flake it, while the full response still takes longer than the timeout.

maintainer-for-aws · 2026-11-05T15:29:10Z

Automated review

A maintainer will still review this — treat the notes below as a starting point.

📌 Commit 9e50f96

This PR fixes cdk deploy (and other commands) hanging forever when an AWS API call is sent but never answered. In the toolkit-lib AWS-auth layer, sdkRequestHandler now sets a socketTimeout on every SDK client so a silently dead connection surfaces as a retryable TimeoutError that plugs into the existing retry strategy instead of waiting indefinitely. Because a synchronous Lambda Invoke (used by BucketDeployment hotswap) can legitimately sit idle for up to the function's full runtime, SDK.lambda() lazily builds a second client with a longer floor via invokeSocketTimeout, leaving all other calls on the shorter shared timeout. A new optional socketTimeout field on RequestHandlerSettings and a stray TODO removal round out the change.

The change is well-scoped and well-tested: the floor logic is covered by a matrix over disabled/longer/shorter/unset values, the lazy second client is pinned to first-invoke only, and the doc comment matches the implemented semantics. One optional note remains on the test suite's use of a live TCP listener. No other issues found on this change.

🔴 0 blocking · 🟡 0 recommended · ⚪ 1 optional

Files with findings (1)

File	Findings
packages/`@aws-cdk`/toolkit-lib/test/api/aws-auth/awscli-compatible.test.ts	⚪ 1

Generated automatically. React 👍 or 👎 to tell us whether this review helped, so we can improve these reviews.

maintainer-for-aws

See the review summary comment for the overview; the notes below are inline.

maintainer-for-aws · 2026-11-05T15:29:11Z


 // Also add notes to the PR about why you imported everything individually and used 'type' so reviewers don't have to ask.

+// a synchronous Invoke sends no data until the function returns: up to the 15 minute Lambda maximum plus cold start init


🟡 Recommended — The PR description describes the Lambda idle timeout as 905000 ms ("just above" the 900-second / 15-minute maximum), but the constant here, its use in SDK.lambda(), and the sdk.test.ts assertion all use 960_000 ms. The code is internally consistent and the adjacent comment justifies 960 s as the 15-minute maximum plus cold-start init, so this reads as stale description text rather than a code defect. The mismatch can mislead reviewers and future readers about the intended value.

Suggested change: Update the PR body (and any diagram) to state 960000 ms, or confirm 960_000 is intended and align the prose.

maintainer-for-aws · 2026-11-05T15:29:11Z

 // Also add notes to the PR about why you imported everything individually and used 'type' so reviewers don't have to ask.



⚪ Optional — The new LAMBDA_SOCKET_TIMEOUT constant is inserted directly beneath a pre-existing scratch comment ("Also add notes to the PR about why you imported everything individually..."). That comment predates this change and is not a defect introduced here, but since the diff now edits this exact region, it is a low-cost chance to drop the stale line.

Suggested change: While editing this region, consider removing the stale "add notes to the PR" scratch comment above the new constant.

maintainer-for-aws · 2026-11-05T15:29:11Z

+      httpsAgent: agent,
+    });
+  });
+
+  describe('with a short socket timeout', () => {
+    let server: net.Server;
+    let sockets: net.Socket[];
+    let respond: (socket: net.Socket) => void;
+
+    beforeEach(async () => {
+      sockets = [];
+      server = net.createServer((socket) => {
+        sockets.push(socket);
+        socket.on('error', () => undefined);
+        respond(socket);
+      });
+      await new Promise<void>((resolve) => server.listen(0, '127.0.0.1', resolve));
+    });
+
+    afterEach(async () => {
+      sockets.forEach((socket) => socket.destroy());
+      await new Promise((resolve) => server.close(resolve));
+    });
+
+    function client() {
+      return new STSClient({
+        region: 'us-east-1',
+        endpoint: `http://127.0.0.1:${(server.address() as net.AddressInfo).port}`,
+        credentials: { accessKeyId: 'AKIAEXAMPLE', secretAccessKey: 'dummy' },
+        requestHandler: { ...sdkRequestHandler(), socketTimeout: 1_000 },
+        retryStrategy: new ConfiguredRetryStrategy(2, () => 0),
+      });
+    }
+
+    test('retries a request that gets no response, then fails', async () => {
+      respond = (socket) => socket.resume();
+
+      await expect(client().send(new GetCallerIdentityCommand({}))).rejects.toMatchObject({ name: 'TimeoutError' });
+      expect(sockets).toHaveLength(2);
+    });
+
+    test('keeps a slow response that is still sending data', async () => {
+      const body = '<GetCallerIdentityResponse><GetCallerIdentityResult><Account>123456789012</Account></GetCallerIdentityResult></GetCallerIdentityResponse>';
+      const response = `HTTP/1.1 200 OK\r\nContent-Type: text/xml\r\nContent-Length: ${body.length}\r\nConnection: close\r\n\r\n${body}`;
+      respond = (socket) => {
+        socket.resume();
+        const chunks = response.match(/.{1,5}/gs)!;
+        const timer = setInterval(() => {
+          socket.write(chunks.shift()!);
+          if (chunks.length === 0) {
+            clearInterval(timer);
+            socket.end();
+          }
+        }, 50);
+      };
+
+      await expect(client().send(new GetCallerIdentityCommand({}))).resolves.toMatchObject({ Account: '123456789012' });
+      expect(sockets).toHaveLength(1);
+    });
+  });
+});


⚪ Optional — The two STSClient-against-net.Server tests in the "with a short socket timeout" block set an explicit socketTimeout: 1_000 on a locally constructed client, so they validate Smithy's general handling of socketTimeout rather than this PR's change to sdkRequestHandler and SDK.lambda(), and pass on main as well. Coverage of the actual change is already provided by the settings-shape test and the sdk.test.ts assertions, so these are optional. The fixtures are cleaned up in afterEach, so no resources leak.

Suggested change: Consider whether a real TCP listener belongs in the unit suite, or drop these two in favor of the settings assertions that fail without the change.

Only a synchronous Invoke stays silent for the whole function run. Other Lambda calls, such as UpdateFunctionCode and the hotswap waiters, now keep the shared 300s idle timeout instead of waiting 16 minutes per attempt.

abhiramshaji-cyber · 2026-11-05T15:29:42Z

A small follow up after a second look at my own change, sorry for the extra noise! In 0f50cb8 and c8300d5 I raised the Lambda Invoke idle timeout from 905 s to 960 s to leave room for cold start init, moved that longer timeout onto Invoke only so other Lambda calls like the hotswap waiters keep the shared 300 s, and widened the margin in the slow response test so it should not flake on a busy runner. The PR description is updated to match. Does scoping it to Invoke like that seem right to you?

Also drop a stale scratch comment above LAMBDA_SOCKET_TIMEOUT.

abhiramshaji-cyber · 2026-11-06T15:09:29Z

Sorry for the noise, the build caught a type error in my own test. The LambdaClient mock in sdk.test.ts declared its argument in a way the spy's type doesn't accept, so tsc failed on line 68. That's fixed in ef74b31, and the toolkit-lib build and tests pass locally. While I was there I also took the automated review's suggestion and dropped the old scratch comment above LAMBDA_SOCKET_TIMEOUT. On its other note, the two STS tests against a local server mostly confirm that Smithy honors socketTimeout, and the sdk.test.ts cases cover the change itself. Would you like me to keep them as an extra guard or drop them? Happy either way.

… Invoke Invoke only raises the socket timeout to LAMBDA_SOCKET_TIMEOUT when the caller's value is shorter or unset, and its client is created on the first invoke.

codecov-commenter · 2026-11-06T15:26:58Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 92.07%. Comparing base (611e948) to head (9e50f96).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2105      +/-   ##
==========================================
+ Coverage   92.05%   92.07%   +0.02%     
==========================================
  Files          82       82              
  Lines       13195    13206      +11     
  Branches     1894     1900       +6     
==========================================
+ Hits        12146    12160      +14     
+ Misses       1011     1008       -3     
  Partials       38       38

Flag	Coverage Δ
suite.unit	`92.07% <ø> (+0.02%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

abhiramshaji-cyber added 2 commits October 29, 2026 02:30

abhiramshaji-cyber temporarily deployed to automation October 29, 2026 06:47 — with GitHub Actions Inactive

abhiramshaji-cyber had a problem deploying to automation October 29, 2026 06:47 — with GitHub Actions Failure

abhiramshaji-cyber requested a deployment to integ-approval October 29, 2026 06:47 — with GitHub Actions Waiting

aws-cdk-automation enabled auto-merge October 29, 2026 06:47

github-actions Bot added bug p2 labels Oct 29, 2026

Merge remote-tracking branch 'upstream/main' into fix/sdk-socket-timeout

dc58f94

# Conflicts: # packages/@aws-cdk/toolkit-lib/test/api/aws-auth/sdk.test.ts

auto-merge was automatically disabled November 5, 2026 15:22
Head branch was pushed to by a user without write access

abhiramshaji-cyber requested a review from a team as a code owner November 5, 2026 15:22

abhiramshaji-cyber requested a deployment to integ-approval November 5, 2026 15:25 — with GitHub Actions Waiting

abhiramshaji-cyber requested a deployment to integ-approval November 5, 2026 15:26 — with GitHub Actions Waiting

maintainer-for-aws Bot reviewed Nov 5, 2026

View reviewed changes

fix(cli): apply the long Lambda socket timeout to Invoke only

c8300d5

Only a synchronous Invoke stays silent for the whole function run. Other Lambda calls, such as UpdateFunctionCode and the hotswap waiters, now keep the shared 300s idle timeout instead of waiting 16 minutes per attempt.

abhiramshaji-cyber requested a deployment to integ-approval November 5, 2026 15:30 — with GitHub Actions Waiting

fix(cli): type the LambdaClient mock to match the spy signature

ef74b31

Also drop a stale scratch comment above LAMBDA_SOCKET_TIMEOUT.

abhiramshaji-cyber requested a deployment to integ-approval November 6, 2026 15:11 — with GitHub Actions Waiting

fix(cli): keep a caller's disabled or longer socket timeout on Lambda…

6f34ad9

… Invoke Invoke only raises the socket timeout to LAMBDA_SOCKET_TIMEOUT when the caller's value is shorter or unset, and its client is created on the first invoke.

abhiramshaji-cyber requested a deployment to integ-approval November 6, 2026 15:16 — with GitHub Actions Waiting

docs(cli): describe the socket timeout default accurately

9e50f96

abhiramshaji-cyber requested a deployment to integ-approval November 6, 2026 15:32 — with GitHub Actions Waiting

Navigation Menu

Repository navigation

fix(cli): deploy hangs forever when an AWS call gets no response - #2105

fix(cli): deploy hangs forever when an AWS call gets no response#2105
abhiramshaji-cyber wants to merge 8 commits into
aws:mainfrom
abhiramshaji-cyber:fix/sdk-socket-timeout

abhiramshaji-cyber commented Oct 29, 2026 •

edited

Loading

Uh oh!

abhiramshaji-cyber commented Nov 5, 2026

Uh oh!

maintainer-for-aws Bot commented Nov 5, 2026 •

edited

Loading

Uh oh!

maintainer-for-aws Bot left a comment

Uh oh!

maintainer-for-aws Bot Nov 5, 2026

Uh oh!

maintainer-for-aws Bot Nov 5, 2026

Uh oh!

maintainer-for-aws Bot Nov 5, 2026

Uh oh!

abhiramshaji-cyber commented Nov 5, 2026

Uh oh!

abhiramshaji-cyber commented Nov 6, 2026

Uh oh!

codecov-commenter commented Nov 6, 2026 •

edited

Loading

Uh oh!

This branch is waiting to be deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants


		// Also add notes to the PR about why you imported everything individually and used 'type' so reviewers don't have to ask.

		// a synchronous Invoke sends no data until the function returns: up to the 15 minute Lambda maximum plus cold start init

Conversation

abhiramshaji-cyber commented Oct 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Checklist

Uh oh!

abhiramshaji-cyber commented Nov 5, 2026

Uh oh!

maintainer-for-aws Bot commented Nov 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

maintainer-for-aws Bot left a comment

Choose a reason for hiding this comment

Uh oh!

maintainer-for-aws Bot Nov 5, 2026

Choose a reason for hiding this comment

Uh oh!

maintainer-for-aws Bot Nov 5, 2026

Choose a reason for hiding this comment

Uh oh!

maintainer-for-aws Bot Nov 5, 2026

Choose a reason for hiding this comment

Uh oh!

abhiramshaji-cyber commented Nov 5, 2026

Uh oh!

abhiramshaji-cyber commented Nov 6, 2026

Uh oh!

codecov-commenter commented Nov 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

This branch is waiting to be deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

abhiramshaji-cyber commented Oct 29, 2026 •

edited

Loading

maintainer-for-aws Bot commented Nov 5, 2026 •

edited

Loading

codecov-commenter commented Nov 6, 2026 •

edited

Loading