[jwrosewell]

Follows the draft specification change 51Degrees/specifications#99, "Usage sharing accepts no Evidence, and reads the web request instead".

The problem

The usage sharing element's filter, EvidenceKeyFilterShareUsage, accepted every header except Cookie, and the Pipeline's filter is the union of its elements' filters, so it accepted every header too. The client-side endpoints' ETag is a hash of the Evidence in that filter, so a revalidation carrying If-None-Match hashed differently and never got a 304, and the Vary header named every header.

What changed

1. The usage sharing element accepts no Evidence. ShareUsageBase.getEvidenceKeyFilter returns an empty list. EvidenceKeyFilterShareUsage is unchanged as the rules for which values are shared (never-shared keys, the cookie header, the block list, 51D_ cookies and session values, the query string allow list).
2. It shares what it finds. The new ShareUsageEvidence.find gives every Evidence value and, when the Evidence holds server.http-request, every value of the web request, keyed without regard to case. A key in both keeps the Evidence value, and server.http-request itself is never one of the values. The ignore-data filter and the values shared both read these.
3. The session tracking hash is of what would be shared. ShareUsageTracker keys its cache on the found values its filter includes, rather than on the Evidence alone, which no longer holds headers no other element accepts. It keeps its own cache for that, because the base class's cache is keyed by FlowData.generateKey, and changing the base class would release pipeline.engines.
4. The web integration adds server.http-request on every request. WebRequestEvidenceServiceCore records every value it offers, accepted or not, in a new WebRequestEvidence, and adds it directly rather than through the accepted-Evidence check. That covers headers, cookies, query string and form parameters, session values, server.client-ip, header.protocol, and the Spring service's server.contextroot, so usage sharing sees the same values as before.
   • The form chosen is a read-only map keyed the way Evidence is keyed (header.<name>, cookie.<name>, query.<name>, session.<name>), not the HttpServletRequest. pipeline.engines.fiftyone, where the usage sharing element lives, does not depend on the servlet API. The values are copied so that they stay the same once the container recycles the request. Its toString gives a count rather than the values, so a debug log of Evidence, or an example page that lists all Evidence, does not print every header and cookie.
5. The cloud request engine leaves the web request out. CloudRequestEngineDefault.getFormData sends every Evidence value whose prefix is not header, cookie or query, whether or not the engine accepts it, so it would have sent server.http-request to the cloud service as http-request. It now leaves that key out. The key is repeated as a private constant there, because pipeline.cloudrequestengine does not depend on pipeline.engines.fiftyone.
6. The sequence element accepts the session id and sequence. SequenceElement accepted no Evidence and relied on usage sharing accepting query.session-id and query.sequence. With usage sharing accepting nothing, every client-side callback would start a new session. It now accepts both, as its specification lists. It also reads a sequence that arrives as text, which the cast to int failed on, as the red run below shows happens on rc today.

How to verify

Both runs are on ubuntu-latest with JDK 17, with the JavaScript templates submodule checked out, running ShareUsage*Tests, EvidenceKeyFilterShareUsageTests, CloudRequestEngineTests, UsageSharingEvidenceTests, ClientsidePropertyServiceTests, PipelineFilterTests, UACHServiceCoreTests and FiftyOneInterceptorTests through mvn -f pipeline/pom.xml -pl pipeline.engines.fiftyone,pipeline.cloudrequestengine,pipeline.web.packages/pipeline.web,pipeline.web.packages/pipeline.web.mvc -am test.

• Red, rc at 2299507 plus the three test commits (8977586, a645d1f, 2b1e52a): run 42424560368, TOTALS {'passed': 76, 'failure': 7, 'skipped': 1, 'error': 2}. The skip is ShareUsageBuilder_IgnoreData_InvalidFilter, which is @Disabled. The run that printed the red errors' causes in full, without the third test commit, is run 42419662468.
• Green, this branch at 2b1e52a: run 42424572596, TOTALS {'passed': 85, 'skipped': 1}.
• The two web request tracker tests then got a tracker of their own with an hour's interval (60b5550), because the class's 50 millisecond interval lets a busy runner count their second request as new, which is how Share a repeated evidence set at most once in each interval #61's tests were seen to fail on macOS. Green at 60b5550: run 42425700016, TOTALS {'passed': 85, 'skipped': 1}. It merges with Share a repeated evidence set at most once in each interval #61 without conflict.

The brief's four checks, each red on rc unless noted.

1. The Pipeline does not accept header.if-none-match or an arbitrary header. ShareUsageAcceptedEvidenceTests builds a Pipeline of the test engine EmptyEngine and the usage sharing element.

FAILURE  ...ShareUsageAcceptedEvidenceTests.Pipeline_WithShareUsage_AcceptsOnlyTheOtherElementsEvidence
         If-None-Match is accepted, so a revalidation's ETag differs from the one it sends. ==> expected: <false> but was: <true>

2. With the web request supplied, a header no other element accepts is still shared, and the cookie header and a blocked header are not. ShareUsageElement_WebRequest_SharesWhatNoOtherElementAccepts also checks that User-Agent, in both places, is shared once with the Evidence value, that a 51D_ cookie and the client IP from the request are shared, and that the web request itself is not.

FAILURE  ...ShareUsageElementTests.ShareUsageElement_WebRequest_SharesWhatNoOtherElementAccepts
         A header only the web request holds was not shared. ==> expected: <shared> but was: <>

The web integration side is addEvidenceFromRequest_AddsTheWebRequest.

FAILURE  ...UsageSharingEvidenceTests.addEvidenceFromRequest_AddsTheWebRequest
         The web request was not added.

3. A revalidation's ETag is the one it sends, so it gets a 304. Revalidation_SameETag_Gets304 serves the JSON endpoint from a real Pipeline of the sequence, JSON builder and usage sharing elements, with a request carrying the session id and sequence as a callback does, then sends If-None-Match with the ETag it got. On rc the Pipeline cannot process that request at all.

ERROR    ...UsageSharingEvidenceTests.Revalidation_SameETag_Gets304
           fiftyone.pipeline.exceptions.AggregateException: Exception(s) occurred processing evidence.
           Suppressed: java.lang.ClassCastException: class java.lang.String cannot be cast to class java.lang.Integer ...
           Suppressed: fiftyone.pipeline.core.exceptions.PipelineConfigurationException: Make sure there is a SequenceElement placed before this JsonBuilderElement in the pipeline

An earlier red run with the first test commit only, before the requests carried a session id and sequence (run 42418221825), failed on the ETag itself: The revalidation's ETag differs from the one it sends. expected:<113796253> but was:<1812379018>. Callback_SessionAndSequence_CarriedOn fails on rc with the same ClassCastException, and passes here with the session id kept and the sequence at 3.

4. Without a web request, Evidence added by hand is shared as before. The existing ShareUsageElementTests, which add Evidence by hand, pass unchanged on both sides, apart from the one below. Two new tracker tests show the session tracking hash. ShareUsageTracker_WebRequest_SessionCookieTracked passes on both sides, and ShareUsageTracker_WebRequest_UnsharedValueIgnored, two requests differing only in a cookie that is not shared, fails on rc with expected: <1> but was: <2>.

ShareUsageElement_TrackSession_SessionCookie asserted that the element accepted the session cookie. It now asserts that it accepts nothing, and still checks that the session cookie keeps two sessions apart and is never sent, so it is red on rc by design. CloudRequestEngineTests.getFormData_LeavesOutTheWebRequest fails on rc with The web request was sent to the cloud service.

Every element's accepted Evidence

Under #99's rule that an element accepts only the Evidence that can change its results, I read the filter of every flow element in the repository's main code. Usage sharing was the only one that accepted every header, and the sequence element relied on it. Every other filter is an EvidenceKeyFilterWhitelist, which matches only the keys listed, so none accepts every key or every header.

Element	Accepts
ShareUsageElement	nothing (this pull request)
SequenceElement	query.session-id, query.sequence (this pull request)
SetHeadersElement, JsonBuilderElement	nothing
JavaScriptBuilderElement	header.Host, header.protocol, query.fod-js-object-name, query.fod-js-enable-cookies, server.contextroot
CloudRequestEngineDefault	the keys the cloud service lists at https://cloud.51degrees.com/api/v4/evidencekeys. On 7 November that list was 138 named keys, with no If-None-Match and no wildcard
DeviceDetectionCloudEngine, IPIntelligenceCloudEngine, GeoLocationCloudEngine, the hardware profile engines (PropertyKeyedCloudEngineBase)	nothing
DeviceDetectionHashEngine, IPIntelligenceOnPremiseEngine	the keys the native engine lists for its data file (engine.getKeys())
TranslationEngine, CountriesTranslationEngine, CountryCodeTranslationEngine	query.translation, query.accept-language, header.accept-language
ParallelElements, the Pipeline	the union of their elements'
The developer examples' SimpleFlowElement, SimpleOnPremiseEngine, SimpleClientSideElement, SimpleCloudEngine	date-of-birth, date-of-birth, cookie.date-of-birth, nothing

WebPipeline_AcceptsOnlyWhatItsElementsUse builds a Pipeline of the sequence, set headers, JSON builder, JavaScript builder and usage sharing elements, and checks that it accepts neither header.if-none-match nor header.x-test nor server.http-request, while it still accepts header.host and query.session-id. On rc it fails with If-None-Match is accepted., and here it passes.

Other drafts touching the same files

• Share a repeated evidence set at most once in each interval #61 changes ShareUsageTracker.java and ShareUsageTrackerTests.java.
• Pause usage sharing sends after repeated failures #62 changes ShareUsageBase.java.
• git merge-tree of this branch with each of them merges without conflict. Cache the client-side endpoints' answers as the specification and RFC 9110 ask #67 changes ClientsidePropertyServiceCore.java, which this does not, and the revalidation test here sends a GET, so it holds under Cache the client-side endpoints' answers as the specification and RFC 9110 ask #67's 412 rule for other methods.

The JavaScript file and the session id

A request for 51Degrees.core.js carries no session id, so the sequence element creates one and writes it into Evidence, where, as an accepted key, it is part of the ETag, and the file gets a new ETag every time. James Rosewell decided on 7 November 2026 that this is correct, because the session id is rightly part of the ETag and the result changes as the sequence increases, so nothing changes there. A callback carries its session id and sequence, so its revalidation gets a 304.

For James to decide

1. Two more packages change. The cloud request engine fix changes pipeline.cloudrequestengine, which with pipeline.developerexamples.cloudengine is not in the 29 packages Monday's plan already releases. Without it, the cloud service would receive every header and cookie of each request as one form field.
2. Other languages' cloud request engines may send the web request too, if they also send Evidence they do not accept.
3. Query string parameters starting 51D_ that only usage sharing accepted no longer reach Evidence, so the JavaScript builder no longer echoes them into its callback URL. Usage sharing still shares them from the web request.

The full pull request build, common-ci's nightly-pull-request.yml run as a dry run from a temporary branch, built 60b5550, this branch's head, merged with rc (run 42455459082). All 15 build and test jobs passed on the second attempt, on Ubuntu x64 with Java 8, 11, 17 and 21, Ubuntu arm64 with Java 17, Windows with Java 8, 11, 17 and 21, and macOS arm64 and x64 with Java 11, 17 and 21. The first attempt failed only in MacOS_X64_Java_11, in DataUpdateServiceTests.DataUpdateService_UpdateFromFile_FileUpdated, a race in the data update service that this change does not touch (#70), and MacOS_X64_Java_21 was cancelled with it. An earlier full build of 2b1e52a, before the tracker interval commit, passed all 15 (run 42424866504).

What remains

• specifications#99 is a draft.