feat: add AWS IAM database authentication for MySQL and PostgreSQL by michielvha · Pull Request #396 · crossplane-contrib/provider-sql

michielvha · 2026-06-20T04:19:59Z

Description of your changes

Adds AWS IAM database authentication for Aurora/RDS MySQL and PostgreSQL. Select AWSIAMAuth as the credentials source and provide a connection Secret with endpoint, port and username. The provider signs a fresh token for every physical connection attempt, including database/sql retries, using the AWS SDK's refreshable credentials. The signing region comes from the ProviderConfig, the connection Secret, or the SDK configuration, in that order.

Both engines support a per-ProviderConfig CA bundle through tlsConfig.caCert.secretRef. PostgreSQL accepts the selector with verify-ca or verify-full; the examples use verify-full. Namespaced PostgreSQL ProviderConfigs read the CA from their own namespace, while cluster-scoped configurations specify the Secret namespace. CA Secret updates reload on reconciliation and apply to subsequent connections without a provider restart. MySQL's custom TLS loaders accept a CA alone; an optional client certificate and key must be supplied together.

PostgreSQL IAM preserves require, verify-ca and verify-full, and upgrades disable to require. The API default remains verify-full. MySQL IAM defaults to verified tls=true; explicit skip-verify remains available. Existing password-authentication TLS defaults are preserved.

The docs and examples use CA Secret selectors. Creating IAM-enabled database users remains a separate bootstrap step.

Fixes #347

How has this code been tested

make reviewable: generation, module tidy, zero lint issues and the full unit suite passed.
Targeted race tests passed for the IAM credential cache, connection wrapper, PostgreSQL client and CA configuration paths.
The complete local kind E2E suite passed all eight variants across MariaDB, PostgreSQL 18 and MSSQL. PostgreSQL checks cover both APIs, all three ProviderConfig paths, CA rejection and recovery, Secret rotation, hostname verification, config isolation and DefaultPrivileges. The provider pod did not restart during these checks.
The earlier IAM connection changes were tested in EKS with Pod Identity against RDS PostgreSQL and MySQL across all three ProviderConfig paths per engine. The new PostgreSQL CA selector was tested in kind.

Temporary test resources were removed.

I have:

Read and followed Crossplane's contribution process.
Run make reviewable against the final revision.

phmcder · 2026-10-06T11:14:24Z

Any updates on this? is it waiting for #393 to be resolved?

michielvha · 2026-10-12T23:05:29Z

@phmcder No this isn’t waiting on #393, I just lost track of this PR, It’s ready for review.

kkendzia

Thanks for this great contribution! LGTM. IMHO it's a well written addition to the provider. :)

michielvha · 2026-10-18T22:58:17Z

@fernandezcuesta, @chlunde

Would you mind reviewing this PR when you have a moment? Thanks in advance!

krogon-bt · 2026-10-21T01:37:23Z

+		}
+		// IAM auth requires TLS. Cert verification depends on the operator
+		// supplying the RDS CA (e.g. via trust-manager or a mounted secret).
+		sslMode = "require"


sample place where sslMode is overwritten

I've corrected this in b8d6121 in all six cluster reconcilers and the shared namespaced provider. IAM now preserves require, verify-ca, and verify-full, and upgrades only an empty mode or disable to require. The Kubernetes default stays verify-full.

The regression tests cover mode preservation and both namespaced ProviderConfig kinds. I also checked against RDS: verify-full fails when the RDS CA is missing.

krogon-bt · 2026-10-21T01:39:30Z

This MR overwrites Postgres sslMode with require on every IAM connect, in all six cluster reconcilers and in namespaced GetProviderConfig. In lib/pq, require mode sets InsecureSkipVerify, so the connection is encrypted and the server certificate is not checked. The docs say an operator can set sslMode: verify-full and supply the Amazon RDS CA, but that field never reaches the driver.

krogon-bt · 2026-10-21T01:46:56Z

+- **[trust-manager](https://cert-manager.io/docs/trust/trust-manager/)** - add the
+  [Amazon RDS CA bundle](https://truststore.pki.rds.amazonaws.com/global/global-bundle.pem)
+  to the provider pod's trust store. Then set `tls: "true"` (MySQL) or
+  `sslMode: verify-full` (PostgreSQL) and the certificate verifies normally.


This is not accurate. Trust-manager does not change pod runtime trust store.

Trust-manager only publishes the bundle as a ConfigMap/Secret and pod still needs that file mounted where Go reads the system pool (/etc/ssl/certs/ca-certificates.crt). This makes the solution similar to mounting secret as below.

I've rewritten this in b8d6121 to include the missing mount and runtime configuration. The example mounts a ConfigMap containing public roots and the RDS CA, sets SSL_CERT_FILE, and documents restarting the provider after updates. trust-manager is described as distributing the bundle to a ConfigMap or Secret; the pod still needs to consume it.

I tested the mount in kind against RDS for both engines. Verified connections worked with the CA, failed without it, and recovered when it was restored.

krogon-bt · 2026-10-21T01:47:43Z

+- **Mounted secret** - mount the RDS CA and reference it through the provider's
+  existing custom TLS mechanism (`tls: custom` + `tlsConfig` for MySQL;
+  `sslMode: verify-full` with the CA for PostgreSQL).


MySQL has tlsConfig.caCert.secretRef available in ProviderConfig but noting equivalent for Postgres

I've removed that claim from the docs. The PostgreSQL example now uses the mounted trust bundle through SSL_CERT_FILE with sslMode: verify-full. This works with the existing ProviderConfig API, so I haven't added a CA Secret selector. Would you prefer adding a CA Secret selector to the PostgreSQL ProviderConfig?

krogon-bt · 2026-10-21T02:03:54Z

+// with AWS IAM authentication. If mode is unset or non-encrypting
+// ("preferred"/"false") it returns "skip-verify" (encrypted but unverified);
+// otherwise it returns mode unchanged so an operator-configured verifying mode
+// ("true"/"custom", backed by the RDS CA) is preserved.


tls=custom is not backed by the RDS CA on its own. LoadConfig rejects a tlsConfig that has caCert without clientCert and clientKey, and IAM database auth has no client certificate. The mode this function can actually verify with is tls=true, which reads the pod trust store. An omitted tls field still becomes skip-verify, so the connection is encrypted and the server certificate is not checked.

I've corrected the comment, docs, and examples in b8d6121. They now use tls: "true" with the mounted trust bundle and explain that the custom loader requires caCert, clientCert, and clientKey.

The loader and the omitted-field skip-verify fallback are unchanged. I checked tls: "true" against RDS: it connects with the RDS CA and rejects the connection without it.

krogon-bt · 2026-10-21T17:04:43Z

The commit from today, b8d6121 (fix: preserve IAM TLS verification and document CA mounts), fixes the PostgreSQL overwrite. verify-full and verify-ca are kept. Only an empty mode or disable is raised to require. The examples now set sslMode: verify-full and MySQL tls: "true", and the RDS CA is meant to arrive through SSL_CERT_FILE on the provider pod. That mount is actually used for those two settings.

What is still wrong is the MySQL fallback and one PostgreSQL claim in the docs.

MySQL still turns a missing or weak tls into skip-verify. EnsureTLS rewrites an omitted mode, "false", and preferred to skip-verify, and the tests assert that. skip-verify encrypts and does not check the server certificate, so the mounted bundle is ignored. tls: "true" and tls: custom are preserved. An IAM ProviderConfig that forgets tls: "true" connects without verification. The upgrade target that matches their own mount is true.

tls: custom still cannot take only the RDS CA. The loader still requires caCert, clientCert, and clientKey. The docs now say so. Aurora IAM has no client certificate, so a per-database CA secret is still impossible. Every IAM database shares the one pod bundle.

The PostgreSQL require sentence does not describe their mount. lib/pq does CA-only checking in require when sslrootcert or ~/.postgresql/root.crt is present. SSL_CERT_FILE is not that file. With their runtime config, require stays unverified. verify-full uses the mount. verify-ca does too, and it skips the hostname check.

The note that the 15-minute lifetime is never an issue is too strong. The token is baked into the DSN at connect time. A reconcile that lasts longer than 15 minutes and then opens another connection uses the expired token.

michielvha · 2026-10-22T07:06:42Z

I've addressed the remaining points:

MySQL IAM now upgrades omitted or weak TLS settings to verified tls=true. Explicit skip-verify remains opt-in.
Both MySQL TLS loaders accept a CA Secret alone. Client certificates and keys are optional and must be supplied together.
The PostgreSQL docs now state that require remains unverified with the SSL_CERT_FILE mount. The examples use verify-full.
Both drivers generate a fresh IAM token for every physical connection attempt, including database/sql retries.

Tested in EKS with Pod Identity against RDS PostgreSQL and MySQL across all three ProviderConfig paths per engine. CA rejection, rotation, password compatibility and fresh signing after a 16-minute wait passed. make reviewable and targeted race tests also passed.

krogon-bt · 2026-10-23T10:31:05Z

PostgreSQL in this PR can only verify the server through one SSL_CERT_FILE bundle mounted on the provider pod. MySQL already has a per-ProviderConfig CA via tlsConfig.caCert.secretRef. PostgreSQL needs the same selector as one provider may serve many databases.

michielvha · 2026-10-23T15:30:23Z

I've added tlsConfig.caCert.secretRef across the PostgreSQL ProviderConfig types. The examples use it with verify-full, and CA Secret updates reload on reconciliation without restarting the provider.

make reviewable, targeted race tests and all eight local E2E variants passed, including PostgreSQL CA rotation and isolation. I also tested IAM authentication manually from EKS against RDS PostgreSQL and MySQL.

Add pkg/clients/awsiam: generates short-lived RDS IAM auth tokens and injects them as the database password, with region resolution and a test seam (injectable TokenBuilder) so the logic is unit-tested without AWS credentials or network. Part of crossplane-contrib#347. Signed-off-by: michielvha <michielvh@outlook.com>

Add the AWSIAMAuth credential source value and an optional Region field to the MySQL and PostgreSQL ProviderConfig (and namespaced ClusterProviderConfig) credential types, for both cluster and namespaced API groups. Regenerate deepcopy and CRDs. Part of crossplane-contrib#347. Signed-off-by: michielvha <michielvh@outlook.com>

Add a cleartext parameter to mysql.New/DSN that appends allowCleartextPasswords=true to the DSN when set. AWS RDS IAM authentication needs this because the AWSAuthenticationPlugin uses MySQL's cleartext client plugin, which go-sql-driver refuses by default. This commit only plumbs the flag (all existing callers pass false, so behaviour is unchanged); the IAM reconciler changes that set it follow. Part of crossplane-contrib#347. Signed-off-by: michielvha <michielvh@outlook.com>

Add awsiam.Inject (load AWS config, resolve region, generate and inject the IAM token) and call it from each cluster reconciler's Connect() when the credentials source is AWSIAMAuth. For MySQL this also enables the cleartext flag and ensures TLS is on; for PostgreSQL it forces sslmode=require. Certificate verification relies on the operator supplying the RDS CA (e.g. via trust-manager or a mounted secret). Part of crossplane-contrib#347. Signed-off-by: michielvha <michielvh@outlook.com>

Inject the IAM token centrally in GetProviderConfig() for the namespaced MySQL and PostgreSQL providers when the credentials source is AWSIAMAuth. MySQL additionally surfaces a Cleartext flag (forwarded by its three reconcilers) and forces TLS on; PostgreSQL forces sslmode=require. The six namespaced PostgreSQL reconcilers need no change. Part of crossplane-contrib#347. Signed-off-by: michielvha <michielvh@outlook.com>

Add example AWSIAMAuth ProviderConfigs for cluster and namespaced MySQL and PostgreSQL, and docs/aws-iam-auth.md covering prerequisites, the operator-supplied RDS CA trust options (trust-manager / mounted secret), region resolution and the TLS behaviour. Part of crossplane-contrib#347. Signed-off-by: michielvha <michielvh@outlook.com>

Signed-off-by: michielvha <michielvh@outlook.com>

fernandezcuesta

Thanks for sticking with this one. I built the branch and ran go test ./pkg/..., and both pass. I did not run the E2E suite. I read the IAM client, both database clients, the TLS registry and garbage collector, the namespaced providers and a sample of the cluster reconcilers.

Per-connection token signing through PasswordConnector looks right, and the sslMode handling now keeps verify-ca and verify-full. I left three inline comments I'd like addressed before merge. The rest are smaller.

Separately, the PR now includes the MySQL custom-TLS rework (optional client cert, renamed registrations), the PostgreSQL DSN rewrite and the ExecTx close fix. They look correct to me. I'd lean towards splitting them out, but I'll defer to the maintainers on that.

For reference, the provider binary grows by about 6% with the AWS SDK (72.9 MB to 77.1 MB default build, 51.0 MB to 54.0 MB stripped). I did not measure memory.

Signed-off-by: michielvha <michielvh@outlook.com>

michielvha · 2026-11-01T19:51:03Z

Thanks for the review. I've addressed all six comments, including the shared connection helpers, the MySQL registration identity and PostgreSQL no-CA fast path.

make reviewable, targeted race tests and all eight local E2E variants passed. I also tested both engines in EKS with Pod Identity against temporary RDS instances, including CA rotation, namespace isolation and deletion.

I measured memory too. Local idle RSS was 49.7 MiB upstream and 51.6 MiB on this branch. In EKS with 100 additional consumers sharing one MySQL ClusterProviderConfig, median provider RSS was 114.4 MiB before these fixes and 71.8 MiB after, across three runs per version. CPU was about 0.17 core in both versions for that workload.

On scope, I'd suggest keeping the CA-only MySQL support and DSN handling here, since they're used by the IAM connection paths. I'm happy to split out the independent ExecTx close fix if you prefer.

michielvha mentioned this pull request Oct 16, 2026

feat: Add AWS IAM database authentication support for MySQL and PostgreSQL #347

Open

kkendzia approved these changes Oct 16, 2026

View reviewed changes

krogon-bt reviewed Oct 21, 2026

View reviewed changes

michielvha force-pushed the feat/aws-iam-auth branch from b75ef57 to 8ae3b03 Compare October 23, 2026 15:37

michielvha added 9 commits October 30, 2026 02:37

fix: preserve IAM TLS verification and document CA mounts

8c03173

Signed-off-by: michielvha <michielvh@outlook.com>

fix: verify IAM TLS and refresh tokens per connection

b525631

Signed-off-by: michielvha <michielvh@outlook.com>

feat: support PostgreSQL CA Secret selectors

bf4e6c2

Signed-off-by: michielvha <michielvh@outlook.com>

michielvha force-pushed the feat/aws-iam-auth branch from 8ae3b03 to bf4e6c2 Compare October 30, 2026 01:57

fernandezcuesta reviewed Oct 30, 2026

View reviewed changes

fernandezcuesta mentioned this pull request Oct 30, 2026

feat(postgres,mysql): IAM auth implementation #490

Open

2 of 8 tasks

michielvha added 2 commits October 30, 2026 16:57

fix: address IAM connection and TLS lifecycle review

dd2c7e4

Signed-off-by: michielvha <michielvh@outlook.com>

test: isolate AWS settings and guard dynamic PostgreSQL DSNs

b5cfba7

Signed-off-by: michielvha <michielvh@outlook.com>

Conversation

michielvha commented Jun 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of your changes

How has this code been tested

Uh oh!

phmcder commented Oct 6, 2026

Uh oh!

michielvha commented Oct 12, 2026

Uh oh!

kkendzia left a comment

Choose a reason for hiding this comment

Uh oh!

michielvha commented Oct 18, 2026

Uh oh!

krogon-bt Oct 21, 2026

Choose a reason for hiding this comment

Uh oh!

michielvha Oct 21, 2026

Choose a reason for hiding this comment

Uh oh!

krogon-bt commented Oct 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

krogon-bt Oct 21, 2026

Choose a reason for hiding this comment

Uh oh!

michielvha Oct 21, 2026

Choose a reason for hiding this comment

Uh oh!

krogon-bt Oct 21, 2026

Choose a reason for hiding this comment

Uh oh!

michielvha Oct 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

krogon-bt Oct 21, 2026

Choose a reason for hiding this comment

Uh oh!

michielvha Oct 21, 2026

Choose a reason for hiding this comment

Uh oh!

krogon-bt commented Oct 21, 2026

Uh oh!

michielvha commented Oct 22, 2026

Uh oh!

krogon-bt commented Oct 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

michielvha commented Oct 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fernandezcuesta left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

michielvha commented Nov 1, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

michielvha commented Jun 20, 2026 •

edited

Loading

krogon-bt commented Oct 21, 2026 •

edited

Loading

michielvha Oct 21, 2026 •

edited

Loading

krogon-bt commented Oct 23, 2026 •

edited

Loading

michielvha commented Oct 23, 2026 •

edited

Loading