Skip to content

feat(e2e): add an upgrade test for a configured installation - #1440

Draft
AdamSaleh wants to merge 4 commits into
redhat-developer:masterfrom
rh-gitops-release-qa:upgrade-suite
Draft

AdamSaleh wants to merge 4 commits into
redhat-developer:masterfrom
rh-gitops-release-qa:upgrade-suite

Conversation

@AdamSaleh

Copy link
Copy Markdown
Member

What does this PR do?

Adds an e2e suite that upgrades a configured installation of the operator, not a clean one.

Our e2e suites, and the upgrade tests built on them, all start from a clean installation: install, upgrade to another clean installation, run the tests. Customers don't upgrade clean installations, and upgrades are where we keep getting hit.

The new Ginkgo suite, test/openshift/e2e/ginkgo/upgrade, runs one ordered scenario:

  1. Install the operator from PREVIOUS_CHANNEL.
  2. Apply config/*.yaml, which turns on as many features as it can.
  3. Check that every feature works.
  4. Upgrade to NEW_CHANNEL.
  5. Run the same checks again, then check that:
    • no value in the .spec of any object in config/ was removed or changed;
    • the pods of the deployed applications were not restarted.
PREVIOUS_CHANNEL=gitops-1.21 NEW_CHANNEL=gitops-1.22 make e2e-tests-upgrade-ginkgo

It needs a cluster-admin login to an OpenShift cluster with three nodes, a CatalogSource that contains both channels, and no GitOps Operator installed, because the suite installs it itself.

Variable Use
CATALOG_SOURCE Catalog to install from (default redhat-operators)
PREVIOUS_VERSION, NEW_VERSION Optional: pin exact versions
SUBSCRIPTION_NAME Optional: the name for the Subscription the suite creates, for harnesses whose other suites look it up by name

Where to look

File What it is
upgrade_test.go The scenario: seven steps, top to bottom
config/*.yaml The configuration under test, as plain manifests
features_test.go verifyFeatures: one check per feature, run before and after the upgrade
olm_test.go Install and upgrade through OLM
README.md How to run it, what it covers, how to add a feature

Features covered today:

  • A customised default instance.
  • A second Argo CD instance with Redis HA.
  • The ApplicationSet controller, notifications, Dex SSO and the server Route.
  • An RBAC policy, kustomize build options, extra config and a custom health check.
  • A cluster-scoped instance via ARGOCD_CLUSTER_CONFIG_NAMESPACES, and managed namespaces.
  • An AppProject, an Application and an ApplicationSet.
  • Argo Rollouts outside openshift-gitops.

Adding a feature takes one YAML change and one check.

PR acceptance criteria:

  • Documentation was updated and verified using make serve-docs: not applicable. The suite's README lives next to it, like the other e2e suites.
  • Unit tests were updated: not applicable, this is test code only.
  • E2E tests were updated: this PR is a new e2e suite.

What type of PR is this?

/kind enhancement

Special notes to the reviewer:

These are the design choices most worth arguing about:

  • InstallPlans are always approved manually. The suite approves each plan itself. That's the only way to stop at a pinned version, and doing it every time keeps a single code path. It stops approving once the plan that installs the target has been approved.

  • The upgrade target comes from the catalog. The suite reads the channel head from the PackageManifest and approves plans until that CSV is installed. Following the Subscription's state stopped one hop early (1.21.5 → 1.22.0, when the head was 1.22.1).

  • "Configuration unchanged" allows additions. 1.22 adds empty commitServer and sourceHydrator fields to the default instance's spec, and a strict comparison fails on those. So the check fails only on a removed or changed value, and prints any additions.

  • "Rolled out" means running the operator's own image. After the CSV succeeds, the suite waits until both instances run the image named in the operator Deployment's ARGOCD_IMAGE.

  • The suite never cleans up. A cluster where the suite failed stays as it is, for inspection. The README has the commands to reset one, in an order that matters.

    A harness that runs other suites afterwards on the same cluster has to remove the fixture itself. Our downstream release testing does this. Without that step, the cluster-scoped RolloutManager the suite leaves behind made 1-100, 1-101 and 1-121 fail.

  • The suite stops at the first failure, so a failed step skips the steps after it.

How it has been run, always on OpenShift 4.22 on GCP:

Run From To Result
Channel heads, redhat-operators 1.21.5 1.22.1 (via 1.22.0) 7 of 7 passed, about 9 minutes
Both versions pinned, redhat-operators 1.21.4 1.22.0 (via 1.21.5) 7 of 7 passed, about 9 minutes
Downstream release testing, 1.20.9 release candidate 1.20.8 1.20.9 7 of 7 passed
Downstream release testing, 1.20.9 release candidate gitops-1.19 head (1.19.7) 1.20.9 7 of 7 passed
  • Release-candidate runs: in the two downstream runs, the suite was the first step on each cluster, and the regular parallel and sequential suites then ran against the upgraded operator.
  • Rebase: this branch is rebased onto current master. The last commit only follows the inner module rename from fix: Rename inner module from argoproj-labs to redhat-developer #1334.
  • Checks: go vet and gofmt are clean on the rebased branch. The suite has not been run again since the rebase.
  • Limits: it has only run on 4.22 on GCP, it isn't wired into CI here, and it hasn't yet caught a real regression.

🤖 Generated with Claude Code

AdamSaleh and others added 4 commits October 20, 2026 14:50
The existing suites, and the upgrade scenarios built on them, test a clean
installation. This adds a small Ginkgo suite, test/openshift/e2e/ginkgo/upgrade,
that upgrades an installation which is in use:

  1. install the operator from PREVIOUS_CHANNEL
  2. apply config/*.yaml, which turns on as many features as it can
  3. check every feature works
  4. upgrade to NEW_CHANNEL
  5. run the same checks again, and check that no configured value was
     removed or changed and that workload pods were not restarted

The suite installs the operator itself, so it needs a cluster without it.
PREVIOUS_VERSION and NEW_VERSION optionally pin exact versions; the
Subscription uses manual approval and the test approves each InstallPlan.

Run with:
  PREVIOUS_CHANNEL=gitops-1.21 NEW_CHANNEL=gitops-1.22 make e2e-tests-upgrade-ginkgo

This is a proof of concept: simple on purpose, with little reuse.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With PREVIOUS_VERSION=1.21.4 the suite approved the plan for 1.21.4 and then,
before that CSV had finished installing, the plan OLM offered next for 1.21.5.
The operator went past the pin and the install step timed out waiting for
1.21.4. Stop approving once the plan that installs the target is approved.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
In CI mode the e2e fixture finds the operator Subscription by the prefix
gitops-operator-. Under the suite's own name, openshift-gitops-operator,
every suite run on the cluster after the upgrade found nothing: on the
1.20.9 upgrade run, 17 sequential and 2 parallel specs failed at
fixture.go:456 with a nil Subscription. SUBSCRIPTION_NAME lets the
harness pick a name its other suites find; the default is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
redhat-developer#1334 renamed github.com/argoproj-labs/gitops-operator/argocd-operator to
github.com/redhat-developer/gitops-operator/argocd-operator; the upgrade suite
was written before it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@openshift-ci openshift-ci Bot added the kind/enhancement New feature or request label Oct 20, 2026
@openshift-ci

openshift-ci Bot commented Oct 20, 2026

Copy link
Copy Markdown

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@openshift-ci

openshift-ci Bot commented Oct 20, 2026

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign chengfang for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@coderabbitai

coderabbitai Bot commented Oct 20, 2026

Copy link
Copy Markdown
Contributor

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

Comment @coderabbitai help to get the list of available commands.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant