Skip to content

Give each package its own version and release only what changed - #63

Draft
jwrosewell wants to merge 3 commits into
rcfrom
ci/version-per-package
Draft

jwrosewell wants to merge 3 commits into
rcfrom
ci/version-per-package

Conversation

@jwrosewell

@jwrosewell jwrosewell commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

Problem

Step 3.5 of the run book in common-ci#306. ci/build-package.ps1 line 31 ran mvn versions:set in each package tree, so every one of the 45 artifacts this repository publishes to Maven Central got a new version whenever anything changed. The last release, 4.5.130 on 5 November, published all 45 at that version.

The run book's decisions are that each package has its own version, tagged <package>/<version> and continuing from the version it last published, that a package is released when a file under its folders changed since its last tag together with every package that depends on it, and that only the patch number moves.

Change

  • ci/packages.json lists the 45 published modules, each named by its artifactId, with its folder, the other folders its build takes files from, and the packages it depends on. A parent's folder is its pom.xml alone, because the rest of its folder is its modules. DependsOn holds the parent and every dependency on another package in any scope, test jars included. The two on-premise engines and ip-intelligence.translation take files from the cxx submodule, and pipeline.did from pipeline/owid-java, so a change to either submodule releases them.
  • ci/release.ps1 loads common-ci's steps/release-plan.ps1, which works out the plan, as 51degrees-python#47 does. It is kept apart from ci/packages.ps1, which every script loads, so building and testing a pull request does not need the shared step. It adds what Maven needs.
    • Get-BuildPlan gives the plan, or every package at its released version when nothing changed, so the nightly publish still has packages to test.
    • Set-PomVersions writes each published module's own version into its pom.xml, with the versions of its parent and of every package it depends on. A module that is not released stays at its last release, so a released module requires what is on Maven Central. The poms in the repository keep one development version.
  • ci/build-package.ps1 sets those versions and deploys only the planned modules, with -pl, so the one publishing bundle holds only them. It passes common-ci no folder to run versions:set in. The link it made at target/central-publishing is gone, because common-ci's java/build-package.ps1 has found the bundle under any module since common-ci#274.
  • ci/publish-package.ps1 uploads the bundle, then pushes a tag and makes a GitHub release for each planned package, and tells common-ci not to add its own single tag. The release notes list the commits that touched the package's files.
  • ci/get-next-package-version.ps1 gives common-ci the highest planned version as a label for the run, and ci/package-update-required.ps1 prints the plan and says whether there is anything to publish. Both are as in 51degrees-python.
  • ci/check-version-bump.ps1 and ci/check-release-interval.ps1 run common-ci's shared checks with -TagPrefix for each planned package, so each is checked against its own tags for a major or minor move and for a release less than a week after its last. They are modelled on 51degrees-dotnet's ci/check-version-bump.ps1. common-ci's nightly-publish.yml runs them once this repository is in its $OwnCheck lists (run book step 2). Until then it runs the shared checks against the label.
  • ci/run-integration-tests.ps1 and ci/run-performance-tests.ps1 point the examples at the version of device-detection, ip-intelligence and pipeline.jsonbuilder the publish built, in place of the one version of the run.
  • ci/build-project.ps1 fails a pull request whose poms refer to a package its DependsOn does not name, or that adds a module with no entry, because a dependent left out would not be released with the package it depends on.
  • ci/packages.ps1 calls its list of trees $Trees, which ci/update-packages.ps1 follows, so the word package means one thing. The root pom.xml, README.md, AGENTS.md and ci/README.md describe the new scheme.

No commit carries a version directive, and no major or minor number moves.

Checks

The checks ran on GitHub runners with common-ci at rc checked out at the root and this branch inside it, as common-ci's workflows lay them out. Each temporary branch was deleted when its run finished.

The run book's checks 1 to 4 (run 42368761295, passed). The workflow tagged every package locally at this branch's head, made one local commit for each case, and ran the scripts the publish runs. Nothing was pushed.

  1. A change to a package nothing depends on, one line in ip-intelligence/ip-intelligence, planned ip-intelligence 4.5.130 to 4.5.131 alone.
  2. A change to a package others depend on planned it and every package depending on it, each after the packages it depends on, which the workflow checked against DependsOn with code of its own rather than the shared functions.
    • One line in pipeline/pipeline.core planned 34 packages, pipeline.core first and geo-location.examples last.
    • One line in location/pom.xml planned the 9 packages of the location tree.
    • A new commit for the cxx submodule planned the two on-premise engines and ip-intelligence.translation as changed, and device-detection, ip-intelligence and geo-location.examples with them, 6 in all.
  3. Every planned version was the package's last tag with the patch number moved, 4.5.130 to 4.5.131, and ci/check-version-bump.ps1 passed each package against its own tags.
  4. A change to ci/options.json, a workflow, the root pom.xml, AGENTS.md, pipeline/README.md, two Doxyfiles and three files under examples/ planned nothing.

The same run also showed the following.

  • With nothing changed, the plan was empty, ci/package-update-required.ps1 exited 1, and the build plan held all 45 packages at 4.5.130 for the tests.
  • With no package tags at all, the plan stopped with pipeline has never been released and packages.json gives it no FirstVersion, so a publish before the first-release tags exist releases nothing.
  • From the first-release tags below, the next release is 29 packages at 4.5.131, the label for the run is 4.5.131, and both checks passed. pipeline.engines.fiftyone, pipeline.javascriptbuilder and pipeline.geo-location changed since 4.5.130, and the other 26 depend on them. The 16 left at 4.5.130 are pipeline, pipeline.common, pipeline.core, pipeline.did, pipeline.engines, pipeline.caching, pipeline.cloudrequestengine, pipeline.translation, pipeline.web.packages, pipeline.developer-examples and six of its eight examples.
  • Set-PomVersions wrote 4.5.131 into the 29 planned poms and 4.5.130 into the other 16, with each parent and dependency at its own package's version. pipeline.engines.fiftyone 4.5.131, for example, requires pipeline.engines 4.5.130, and pipeline.device-detection 4.5.131 manages pipeline-aggregate 4.5.131. mvn -pl <the 29> validate then read the rewritten reactor and the 29 modules at 4.5.131, BUILD SUCCESS.
  • The packages.json check found nothing missing as committed. With pipeline.core taken out of the DependsOn of pipeline.engines and the entry for geo-location.web.examples.shared removed, it named both.

A dry run of the publish, through common-ci's nightly-publish.yml at rc with dryrun set, with the first-release tags made in each job's checkout by a temporary step, and with PublishAnyBranch set so the Package job runs ci/publish-package.ps1 off main (run 42369689241). The Configure job printed the same plan of 29 packages at 4.5.131 and the label 4.5.131. With this repository not yet in $OwnCheck, it ran the shared checks against the label. The version check passed, and the weekly check dated 4.5.130 by its GitHub release, 5 November, and reported that the next may go from 12 November, as a notice because WEEKLY_RELEASES_FROM is not set. The seven Pre Build jobs built the native libraries. The Build job deployed exactly the 29 planned modules, each at 4.5.131, signed them, and the publishing plugin wrote one bundle under pipeline/pipeline.engines.fiftyone/target/central-publishing, the first planned module, which common-ci's java/build-package.ps1 found. The seven Test jobs, which install the built packages and test the examples against device-detection, ip-intelligence and pipeline.jsonbuilder at their built versions, and the Package job, which runs ci/publish-package.ps1 in dry run, all passed.

The full pull request build, common-ci's nightly-pull-request.yml run as a dry run from a temporary branch, built 70901e7, this branch's head, merged with rc (run 42423355619). All 15 build and test jobs passed, on Ubuntu x64 with Java 8, 11, 17 and 21, Ubuntu arm64 with Java 17, Windows with Java 8, 11, 17 and 21, and macOS arm64 and x64 with Java 11, 17 and 21. The first attempt failed only in MacOS_X64_Java_11, where ShareUsageTrackerTests.ShareUsageTracker_Session_DoNotTrack expected 1 and got 2, the timing failure that #61 fixes, and the second attempt of that job passed.

Documentation, tests, examples and pom metadata release nothing

James Rosewell decided on 7 November 2026 that a package is not released when the only changes under its folders since its last tag are to documentation, tests or examples, or only to metadata such as links in its manifest. common-ci#310 does that in steps/release-plan.ps1. It leaves out Markdown, LICENSE files and files under folders named docs, tests, test or examples, in any case, and then calls a repository's own ci/release-plan-filter.ps1, if it has one, with the files that are left.

Dry run

Run 42395465949 checked out common-ci at feature/release-plan-ignore (2a31d28) at the root and this branch inside it, as common-ci's workflows lay them out, ran the filter's tests and then the plan through ci/release.ps1. Every tag and commit it made was local, and nothing was pushed.

Tests Passed: 20, Failed: 0, Skipped: 0, Inconclusive: 0, NotRun: 0

From the 4.5.130 first-release tags the plan is 29 packages at 4.5.131, as before, with the reasons now counting files, such as pipeline.engines.fiftyone 4.5.130 4.5.131 changed since 4.5.130 (3 files). Each case below then started from every package tagged 4.5.131 at the branch head, which plans nothing, and added one commit.

Case Files changed Planned
1. pipeline.core pom, description and url pipeline/pipeline.core/pom.xml 0
2. pipeline parent pom, developers, licenses and scm pipeline/pom.xml 0
3. pipeline.core pom, a comment and a blank line pipeline/pipeline.core/pom.xml 0
4. pipeline.core, a file under src/test .../src/test/java/.../OptionsTest.java 0
5. pipeline.core, its README.md pipeline/pipeline.core/README.md 0
6. Cases 1, 4 and 5 together the three files 0
7. pipeline.core pom, the jar plugin's Automatic-Module-Name pipeline/pipeline.core/pom.xml 34, pipeline.core 4.5.131 4.5.132 changed since 4.5.131 (1 file) first
8. pipeline.core pom, a dependency version and the description pipeline/pipeline.core/pom.xml 34
9. pipeline.core, a source file under src/main .../src/main/java/.../BuildArg.java 34
10. Case 1 with ci/release-plan-filter.ps1 removed the pom and the filter 34

Case 10 shows that the filter, rather than the globs, is what leaves out case 1. The 34 are pipeline.core and every package that depends on it, as in the checks above.

For James to decide

  1. The defaults match folders inside a package's src/main too. location/geo-location.core/src/main/java/fiftyone/geolocation/core/test/MavenRunListener.java is in the published geo-location.core jar but sits in a test folder, and the main code of geo-location.examples, geo-location.web.examples.mvc, geo-location.web.examples.servlet and geo-location.web.examples.shared sits in examples folders. A change only there releases nothing, so those example packages are released only when a package they depend on is. The way to count them again is IgnoreDefaults set to false on those entries with their own Ignore list. This branch leaves the defaults on.
  2. pipeline.common, pipeline.core, pipeline.engines, device-detection.shared and ip-intelligence.shared also publish a test jar built from src/test, and the examples use the device-detection.shared one. A change only under src/test no longer releases them, so their published test jars stay as they were until something else changes.
  3. Comments and layout in a pom.xml do not count. The decision named metadata such as links, and a comment changes neither the build nor what a project using the package gets.

First release

Run book step 4. Every one of the 45 packages last published 4.5.130, on 5 November 2026, from f69da52e, the commit of the plain tag 4.5.130 and the head of main today. Publish run 41934429106 built it from that commit, and Maven Central's maven-metadata.xml for each of the 45 lists 4.5.130 as its newest 4.5 version, updated at 01:03 UTC on 5 November. So each tag is <artifactId>/4.5.130 on that commit. The 45 tags were pushed on 7 November 2026, and git ls-remote --tags shows each of them on f69da52e.

git fetch origin
c=f69da52e52104687206d1ca1c3595d2935790f34
tags=
for name in \
  pipeline pipeline.common pipeline.core pipeline.did pipeline.engines \
  pipeline.caching pipeline.cloudrequestengine pipeline.engines.fiftyone \
  pipeline.translation pipeline.javascriptbuilder pipeline.jsonbuilder \
  pipeline.web.packages pipeline.web.shared pipeline.web pipeline.web.mvc \
  pipeline-aggregate pipeline.developer-examples \
  pipeline.developerexamples.web.shared pipeline.developerexamples.flowelement \
  pipeline.developerexamples.onpremiseengine pipeline.developerexamples.cloudengine \
  pipeline.developerexamples.clientsideelement \
  pipeline.developerexamples.clientsideelement.mvc \
  pipeline.developerexamples.usagesharing pipeline.developerexamples.fodid \
  pipeline.device-detection device-detection.shared \
  device-detection.hash.engine.on-premise device-detection.cloud device-detection \
  pipeline.ip-intelligence ip-intelligence.shared ip-intelligence.translation \
  ip-intelligence.engine.on-premise ip-intelligence.cloud ip-intelligence \
  pipeline.geo-location geo-location.core geo-location.cloud geo-location \
  geo-location.examples geo-location.web.examples geo-location.web.examples.mvc \
  geo-location.web.examples.servlet geo-location.web.examples.shared; do
  git tag "$name/4.5.130" "$c"
  tags="$tags $name/4.5.130"
done
git push origin $tags

The tags are lightweight, as ci/publish-package.ps1 makes them, so the weekly check dates each by its commit, 23:31 UTC on 4 November. The plain 4.5.130 tag and its GitHub release stay as they are. The scripts on main read only plain version tags, so the new tags can be pushed before this reaches main.

Maven Central

The 25 artifacts of the pipeline tree show <release>4.6.0</release> in their maven-metadata.xml, from a minor version published on 18 September 2026 and withdrawn. Their newest 4.5 version is 4.5.130, and Central kept 4.6.0 as the release when 4.5.130 was published. The plan reads only this repository's tags, so 4.6.0 does not change it, and every package continues from 4.5.130. ci/update-packages.ps1 takes patch updates only, so it does not move a requirement to 4.6.0 either. A project that asks Maven for the newest release of one of the 25, or gives a range that includes 4.6, still gets 4.6.0. Nothing here changes that.

Order and what remains

  1. This needs common-ci's steps/release-plan.ps1 on common-ci's main, which is on rc only today, from common-ci#307, with common-ci#310, a draft, for the defaults and the filter. The publish workflows here take common-ci from main, and without the step ci/release.ps1 stops the publish. Without #310 the step ignores ci/release-plan-filter.ps1 and counts every file, as before. Building and testing a pull request does not use it.
  2. The 45 first-release tags are pushed, so nothing more is needed before the first publish from main after this merges.
  3. common-ci's nightly-publish.yml runs ci/check-version-bump.ps1 and ci/check-release-interval.ps1 only once 51degrees-java is in its two $OwnCheck lists.
  4. WEEKLY_RELEASES_FROM is not set where this repository's workflows read it, so the weekly check reports and passes. Each package's last release is dated 4 November, so with the rule in force a release before 11 November would need allow-early-release.
  5. Each publish makes one GitHub release per package, 29 for the next release as rc stands.
  6. With #310, a change only to documentation, tests, examples or a pom.xml's metadata releases nothing. The three points for James in the section above say where that reaches further than it may be meant to.
  7. The dry run does not upload, so Maven Central's check of a bundle whose modules' parents are not in it, but are already on Central, has not been tried.
  8. device-detection/Dockerfile.example sets the version from git describe --tags, which can now name a package tag. It is not used by CI and is left alone.
  9. pipeline.developer-examples with its eight examples, and the five location example modules, are published to Maven Central today, as they have been since 4.1, although their poms set maven.deploy.skip, which the publishing plugin does not read. They stay packages here, so their streams continue.

Every published module is a package in ci/packages.json, named by its
artifactId and tagged <package>/<version>. A publish takes its plan from
common-ci's steps/release-plan.ps1, which releases a package when a file
under its folders changed since its last tag, and every package that
depends on it, each at its last tag with the patch number moved.

build-package.ps1 writes each module's own version into its pom.xml, with
the versions of its parent and of the packages it depends on, and deploys
only the planned modules. publish-package.ps1 tags and releases each one.
check-version-bump.ps1 and check-release-interval.ps1 check each planned
package against its own tags. build-project.ps1 fails a pull request whose
poms do not match ci/packages.json.

Step 3.5 of 51Degrees/common-ci#306.
common-ci's release plan calls ci/release-plan-filter.ps1 with the files under a package's folders that changed since its last tag, once its Ignore globs and the default globs for documentation, tests and examples have left theirs out. The filter leaves out a pom.xml whose only change is to its name, description, url, licenses, developers, scm or issueManagement, or to its comments and layout, so a new link does not release the package. A version, dependency, plugin or build change still counts, as does every other file. Its Pester tests are ci/release-plan-filter.Tests.ps1.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant