[jwrosewell]

Java's usage sharing has no back-off, so every message is sent, and lost, while the service is failing. This adds the back-off that 51degrees-dotnet#88 restores in .NET, with the same five settings and defaults.

Problem

ShareUsageElement.sendUsageData (lines 341 to 379) sends every batch it takes from the queue, and a failed send is logged and the batch dropped. During an outage every batch is lost, and each one is another request to a failing service. .NET had a back-off from pipeline-dotnet#218, but it blocked processing as well as sending and was removed in pipeline-dotnet#261. The back-off is now to be part of the specification and of every usage sharing element.

Change

1. A send fails when it throws or the answer is not 200, as sendAsXML already reports.
2. When setFailuresToEnterRecovery sends fail within setFailuresWindowSeconds, sending pauses for setInitialRecoveryDelaySeconds. Successful sends in between do not reset the count.
3. When a pause ends, one send is tried. If it fails, the pause is multiplied by setRecoveryMultiplier, up to setMaxRecoveryDelaySeconds. If it succeeds, the back-off ends and the next pause starts again from the initial delay.
4. Processing does not change. Evidence is queued during a pause, and new evidence is dropped only when the queue is full, as now.
5. The consumer waits out the pause itself, on a latch that closing releases, and then sends until fewer than the minimum entries per message are queued, so the queue drains when a pause ends without any more requests being processed.
6. Closing ends a pause at once and sends nothing while paused. It logs how many entries were not sent. When not paused, closing drains the queue as now.
7. A warning is logged when sending pauses or a pause is lengthened, and information when sending resumes.

Setting	Default	Accepted values
setFailuresToEnterRecovery	10	1 to 100
setFailuresWindowSeconds	100	1 or more
setInitialRecoveryDelaySeconds	2.0	more than 0
setMaxRecoveryDelaySeconds	300.0	more than 0
setRecoveryMultiplier	2.0	1 or more

How it is built:

• ShareUsageBackOff (ShareUsageBackOff.java) holds the rules. Each method takes the time from the caller, so its tests set the time.
• ShareUsageBase gains the protected waitWhilePaused and recordSendResult, which ShareUsageElement.sendUsageData calls before and after each message. A class derived from ShareUsageBase with its own consumer gets no back-off until it calls them.
• New constructors on ShareUsageBase and ShareUsageElement take the five settings. The existing ones pass the defaults, so code built against them is unchanged.
• The defaults are in Constants under the .NET names.

Tests

The element tests in ShareUsageElementBackOffTests.java were written first. The first two use only the default settings and the constructor rc already has, so they compiled on rc and were run there with only the tests added (run 42347851766). Both failed on Ubuntu and Windows, and the other 27 usage sharing tests passed, with one skipped as before.

Test	Failure on rc
ShareUsageElement_BackOff_PausesThenDrainsQueue	Thirteen sends had been made 0.7 seconds after the tenth failed send, where ten were expected.
ShareUsageElement_BackOff_CloseWhilePausedDoesNotWaitOrSend	A send was made during the pause.

The two tests that set the delays, ShareUsageElement_BackOff_DelayGrowsToMaximum and ShareUsageElement_BackOff_FullQueueDropsNewEvidence, need the new constructor, so they were added with the change, as were ShareUsageBackOffTests.java, which checks the rules with set times, and the setter checks.

With the change (run 42348489883) all 39 usage sharing tests pass on Ubuntu and Windows, with the same one skipped.

The full pull request build, common-ci's nightly-pull-request.yml run as a dry run from a temporary branch, built 033155f, this branch's head, merged with rc (run 42350514885). All 15 build and test jobs passed on the first attempt, on Ubuntu x64 with Java 8, 11, 17 and 21, Ubuntu arm64 with Java 17, Windows with Java 8, 11, 17 and 21, and macOS arm64 and x64 with Java 11, 17 and 21.

Found, not changed

ShareUsageBuilderBaseTest uses JUnit 4 annotations, and its tests did not run in the runs above, which ran every other usage sharing test class. Whether the build runs JUnit 4 tests at all has not been checked.

Related

• specifications#97 makes this back-off and its settings part of the specification for every language.
• #61 makes Java share a repeated evidence set at most once in each interval.