Repository navigation
Give each package its own version and release only what changed - #63
Draft
jwrosewell wants to merge 3 commits into
Draft
jwrosewell wants to merge 3 commits into
jwrosewell wants to merge 3 commits into
Conversation
Every published module is a package in ci/packages.json, named by its artifactId and tagged <package>/<version>. A publish takes its plan from common-ci's steps/release-plan.ps1, which releases a package when a file under its folders changed since its last tag, and every package that depends on it, each at its last tag with the patch number moved. build-package.ps1 writes each module's own version into its pom.xml, with the versions of its parent and of the packages it depends on, and deploys only the planned modules. publish-package.ps1 tags and releases each one. check-version-bump.ps1 and check-release-interval.ps1 check each planned package against its own tags. build-project.ps1 fails a pull request whose poms do not match ci/packages.json. Step 3.5 of 51Degrees/common-ci#306.
common-ci's release plan calls ci/release-plan-filter.ps1 with the files under a package's folders that changed since its last tag, once its Ignore globs and the default globs for documentation, tests and examples have left theirs out. The filter leaves out a pom.xml whose only change is to its name, description, url, licenses, developers, scm or issueManagement, or to its comments and layout, so a new link does not release the package. A version, dependency, plugin or build change still counts, as does every other file. Its Pester tests are ci/release-plan-filter.Tests.ps1.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Step 3.5 of the run book in common-ci#306.
ci/build-package.ps1line 31 ranmvn versions:setin each package tree, so every one of the 45 artifacts this repository publishes to Maven Central got a new version whenever anything changed. The last release, 4.5.130 on 5 November, published all 45 at that version.The run book's decisions are that each package has its own version, tagged
<package>/<version>and continuing from the version it last published, that a package is released when a file under its folders changed since its last tag together with every package that depends on it, and that only the patch number moves.Change
ci/packages.jsonlists the 45 published modules, each named by its artifactId, with its folder, the other folders its build takes files from, and the packages it depends on. A parent's folder is itspom.xmlalone, because the rest of its folder is its modules.DependsOnholds the parent and every dependency on another package in any scope, test jars included. The two on-premise engines andip-intelligence.translationtake files from thecxxsubmodule, andpipeline.didfrompipeline/owid-java, so a change to either submodule releases them.ci/release.ps1loads common-ci'ssteps/release-plan.ps1, which works out the plan, as 51degrees-python#47 does. It is kept apart fromci/packages.ps1, which every script loads, so building and testing a pull request does not need the shared step. It adds what Maven needs.Get-BuildPlangives the plan, or every package at its released version when nothing changed, so the nightly publish still has packages to test.Set-PomVersionswrites each published module's own version into itspom.xml, with the versions of its parent and of every package it depends on. A module that is not released stays at its last release, so a released module requires what is on Maven Central. The poms in the repository keep one development version.ci/build-package.ps1sets those versions and deploys only the planned modules, with-pl, so the one publishing bundle holds only them. It passes common-ci no folder to runversions:setin. The link it made attarget/central-publishingis gone, because common-ci'sjava/build-package.ps1has found the bundle under any module since common-ci#274.ci/publish-package.ps1uploads the bundle, then pushes a tag and makes a GitHub release for each planned package, and tells common-ci not to add its own single tag. The release notes list the commits that touched the package's files.ci/get-next-package-version.ps1gives common-ci the highest planned version as a label for the run, andci/package-update-required.ps1prints the plan and says whether there is anything to publish. Both are as in 51degrees-python.ci/check-version-bump.ps1andci/check-release-interval.ps1run common-ci's shared checks with-TagPrefixfor each planned package, so each is checked against its own tags for a major or minor move and for a release less than a week after its last. They are modelled on 51degrees-dotnet'sci/check-version-bump.ps1. common-ci'snightly-publish.ymlruns them once this repository is in its$OwnChecklists (run book step 2). Until then it runs the shared checks against the label.ci/run-integration-tests.ps1andci/run-performance-tests.ps1point the examples at the version ofdevice-detection,ip-intelligenceandpipeline.jsonbuilderthe publish built, in place of the one version of the run.ci/build-project.ps1fails a pull request whose poms refer to a package itsDependsOndoes not name, or that adds a module with no entry, because a dependent left out would not be released with the package it depends on.ci/packages.ps1calls its list of trees$Trees, whichci/update-packages.ps1follows, so the word package means one thing. The rootpom.xml,README.md,AGENTS.mdandci/README.mddescribe the new scheme.No commit carries a version directive, and no major or minor number moves.
Checks
The checks ran on GitHub runners with common-ci at
rcchecked out at the root and this branch inside it, as common-ci's workflows lay them out. Each temporary branch was deleted when its run finished.The run book's checks 1 to 4 (run 42368761295, passed). The workflow tagged every package locally at this branch's head, made one local commit for each case, and ran the scripts the publish runs. Nothing was pushed.
ip-intelligence/ip-intelligence, plannedip-intelligence4.5.130 to 4.5.131 alone.DependsOnwith code of its own rather than the shared functions.pipeline/pipeline.coreplanned 34 packages,pipeline.corefirst andgeo-location.exampleslast.location/pom.xmlplanned the 9 packages of the location tree.cxxsubmodule planned the two on-premise engines andip-intelligence.translationas changed, anddevice-detection,ip-intelligenceandgeo-location.exampleswith them, 6 in all.ci/check-version-bump.ps1passed each package against its own tags.ci/options.json, a workflow, the rootpom.xml,AGENTS.md,pipeline/README.md, two Doxyfiles and three files underexamples/planned nothing.The same run also showed the following.
ci/package-update-required.ps1exited 1, and the build plan held all 45 packages at 4.5.130 for the tests.pipeline has never been released and packages.json gives it no FirstVersion, so a publish before the first-release tags exist releases nothing.pipeline.engines.fiftyone,pipeline.javascriptbuilderandpipeline.geo-locationchanged since 4.5.130, and the other 26 depend on them. The 16 left at 4.5.130 arepipeline,pipeline.common,pipeline.core,pipeline.did,pipeline.engines,pipeline.caching,pipeline.cloudrequestengine,pipeline.translation,pipeline.web.packages,pipeline.developer-examplesand six of its eight examples.Set-PomVersionswrote 4.5.131 into the 29 planned poms and 4.5.130 into the other 16, with each parent and dependency at its own package's version.pipeline.engines.fiftyone4.5.131, for example, requirespipeline.engines4.5.130, andpipeline.device-detection4.5.131 managespipeline-aggregate4.5.131.mvn -pl <the 29> validatethen read the rewritten reactor and the 29 modules at 4.5.131, BUILD SUCCESS.packages.jsoncheck found nothing missing as committed. Withpipeline.coretaken out of theDependsOnofpipeline.enginesand the entry forgeo-location.web.examples.sharedremoved, it named both.A dry run of the publish, through common-ci's
nightly-publish.ymlatrcwithdryrunset, with the first-release tags made in each job's checkout by a temporary step, and withPublishAnyBranchset so the Package job runsci/publish-package.ps1offmain(run 42369689241). The Configure job printed the same plan of 29 packages at 4.5.131 and the label 4.5.131. With this repository not yet in$OwnCheck, it ran the shared checks against the label. The version check passed, and the weekly check dated 4.5.130 by its GitHub release, 5 November, and reported that the next may go from 12 November, as a notice becauseWEEKLY_RELEASES_FROMis not set. The seven Pre Build jobs built the native libraries. The Build job deployed exactly the 29 planned modules, each at 4.5.131, signed them, and the publishing plugin wrote one bundle underpipeline/pipeline.engines.fiftyone/target/central-publishing, the first planned module, which common-ci'sjava/build-package.ps1found. The seven Test jobs, which install the built packages and test the examples againstdevice-detection,ip-intelligenceandpipeline.jsonbuilderat their built versions, and the Package job, which runsci/publish-package.ps1in dry run, all passed.The full pull request build, common-ci's
nightly-pull-request.ymlrun as a dry run from a temporary branch, built 70901e7, this branch's head, merged withrc(run 42423355619). All 15 build and test jobs passed, on Ubuntu x64 with Java 8, 11, 17 and 21, Ubuntu arm64 with Java 17, Windows with Java 8, 11, 17 and 21, and macOS arm64 and x64 with Java 11, 17 and 21. The first attempt failed only inMacOS_X64_Java_11, whereShareUsageTrackerTests.ShareUsageTracker_Session_DoNotTrackexpected 1 and got 2, the timing failure that #61 fixes, and the second attempt of that job passed.Documentation, tests, examples and pom metadata release nothing
James Rosewell decided on 7 November 2026 that a package is not released when the only changes under its folders since its last tag are to documentation, tests or examples, or only to metadata such as links in its manifest. common-ci#310 does that in
steps/release-plan.ps1. It leaves out Markdown,LICENSEfiles and files under folders nameddocs,tests,testorexamples, in any case, and then calls a repository's ownci/release-plan-filter.ps1, if it has one, with the files that are left.src/test. The default**/test/**matches a folder namedtestanywhere in the path, which takes in every module'ssrc/test, so no package needs anIgnorelist of its own. Case 4 below shows it.ci/release-plan-filter.ps1passes every file through except apom.xml. It reads thepom.xmlat the package's last tag and atHEAD, takes out the comments, the layout and thename,description,url,licenses,developers,scmandissueManagementchildren of<project>, and counts the file only when what is left differs. A version, dependency, plugin, build or repository change therefore still counts, and so does apom.xmlthat is new or removed.ci/release-plan-filter.Tests.ps1is a Pester 5 suite of 20 tests, each in a throwaway git repository with a taggedpom.xml. No workflow here runs it yet.ci/README.mdandREADME.mdsay which changes release nothing.rcmerged in at 2299507, which brings Run the copyright update once a year #52, Run the Selenium contract tests against the on-premise web example too #54 and Run the IP Intelligence cloud tests again, and let a property have no value #57. The only file both sides changed isci/run-integration-tests.ps1, which merged without conflict, and the Selenium step Run the Selenium contract tests against the on-premise web example too #54 added passes the same per-package version arguments to Maven.Dry run
Run 42395465949 checked out common-ci at
feature/release-plan-ignore(2a31d28) at the root and this branch inside it, as common-ci's workflows lay them out, ran the filter's tests and then the plan throughci/release.ps1. Every tag and commit it made was local, and nothing was pushed.From the 4.5.130 first-release tags the plan is 29 packages at 4.5.131, as before, with the reasons now counting files, such as
pipeline.engines.fiftyone 4.5.130 4.5.131 changed since 4.5.130 (3 files). Each case below then started from every package tagged 4.5.131 at the branch head, which plans nothing, and added one commit.pipeline.corepom,descriptionandurlpipeline/pipeline.core/pom.xmlpipelineparent pom,developers,licensesandscmpipeline/pom.xmlpipeline.corepom, a comment and a blank linepipeline/pipeline.core/pom.xmlpipeline.core, a file undersrc/test.../src/test/java/.../OptionsTest.javapipeline.core, itsREADME.mdpipeline/pipeline.core/README.mdpipeline.corepom, the jar plugin'sAutomatic-Module-Namepipeline/pipeline.core/pom.xmlpipeline.core 4.5.131 4.5.132 changed since 4.5.131 (1 file)firstpipeline.corepom, a dependency version and thedescriptionpipeline/pipeline.core/pom.xmlpipeline.core, a source file undersrc/main.../src/main/java/.../BuildArg.javaci/release-plan-filter.ps1removedCase 10 shows that the filter, rather than the globs, is what leaves out case 1. The 34 are
pipeline.coreand every package that depends on it, as in the checks above.For James to decide
src/maintoo.location/geo-location.core/src/main/java/fiftyone/geolocation/core/test/MavenRunListener.javais in the publishedgeo-location.corejar but sits in atestfolder, and the main code ofgeo-location.examples,geo-location.web.examples.mvc,geo-location.web.examples.servletandgeo-location.web.examples.sharedsits inexamplesfolders. A change only there releases nothing, so those example packages are released only when a package they depend on is. The way to count them again isIgnoreDefaultsset tofalseon those entries with their ownIgnorelist. This branch leaves the defaults on.pipeline.common,pipeline.core,pipeline.engines,device-detection.sharedandip-intelligence.sharedalso publish a test jar built fromsrc/test, and the examples use thedevice-detection.sharedone. A change only undersrc/testno longer releases them, so their published test jars stay as they were until something else changes.pom.xmldo not count. The decision named metadata such as links, and a comment changes neither the build nor what a project using the package gets.First release
Run book step 4. Every one of the 45 packages last published 4.5.130, on 5 November 2026, from
f69da52e, the commit of the plain tag4.5.130and the head ofmaintoday. Publish run 41934429106 built it from that commit, and Maven Central'smaven-metadata.xmlfor each of the 45 lists 4.5.130 as its newest 4.5 version, updated at 01:03 UTC on 5 November. So each tag is<artifactId>/4.5.130on that commit. The 45 tags were pushed on 7 November 2026, andgit ls-remote --tagsshows each of them onf69da52e.The tags are lightweight, as
ci/publish-package.ps1makes them, so the weekly check dates each by its commit, 23:31 UTC on 4 November. The plain4.5.130tag and its GitHub release stay as they are. The scripts onmainread only plain version tags, so the new tags can be pushed before this reachesmain.Maven Central
The 25 artifacts of the
pipelinetree show<release>4.6.0</release>in theirmaven-metadata.xml, from a minor version published on 18 September 2026 and withdrawn. Their newest 4.5 version is 4.5.130, and Central kept 4.6.0 as the release when 4.5.130 was published. The plan reads only this repository's tags, so 4.6.0 does not change it, and every package continues from 4.5.130.ci/update-packages.ps1takes patch updates only, so it does not move a requirement to 4.6.0 either. A project that asks Maven for the newest release of one of the 25, or gives a range that includes 4.6, still gets 4.6.0. Nothing here changes that.Order and what remains
steps/release-plan.ps1on common-ci'smain, which is onrconly today, from common-ci#307, with common-ci#310, a draft, for the defaults and the filter. The publish workflows here take common-ci frommain, and without the stepci/release.ps1stops the publish. Without #310 the step ignoresci/release-plan-filter.ps1and counts every file, as before. Building and testing a pull request does not use it.mainafter this merges.nightly-publish.ymlrunsci/check-version-bump.ps1andci/check-release-interval.ps1only once 51degrees-java is in its two$OwnChecklists.WEEKLY_RELEASES_FROMis not set where this repository's workflows read it, so the weekly check reports and passes. Each package's last release is dated 4 November, so with the rule in force a release before 11 November would needallow-early-release.rcstands.pom.xml's metadata releases nothing. The three points for James in the section above say where that reaches further than it may be meant to.device-detection/Dockerfile.examplesets the version fromgit describe --tags, which can now name a package tag. It is not used by CI and is left alone.pipeline.developer-exampleswith its eight examples, and the five location example modules, are published to Maven Central today, as they have been since 4.1, although their poms setmaven.deploy.skip, which the publishing plugin does not read. They stay packages here, so their streams continue.