[jwrosewell]

Follows specifications#99, the draft that has the usage sharing element accept no Evidence, as James decided on 7 November for Monday's release.

The problem

ShareUsageEvidenceKeyFilter was both what the usage sharing element accepted and what it shared, so it accepted every header except Cookie, every server. key and the other keys its rules share (share_usage_evidencekeyfilter.py, share_usage.py line 129). A pipeline accepts a key when any element does, so a pipeline holding usage sharing accepted If-None-Match and every other header. The client-side ETag (#59) and the Vary header are built from what the pipeline accepts, so they named every header and the ETag of a revalidation could never match. The engines' own caches here use each engine's own filter, so they did not depend on it.

The change

1. The element accepts no evidence. ShareUsage.get_evidence_key_filter() returns an empty BasicListEvidenceKeyFilter. ShareUsageEvidenceKeyFilter and ShareUsageTrackerEvidenceKeyFilter are unchanged, and are now only the rules for which found keys are shared and which tell a repeat apart.
2. It shares what it finds. The new ShareUsage.found_evidence() returns the values of the web request under server.http-request, then every evidence value in the flow data, whether an element accepted it or not, with a later source replacing an earlier one, so a key found in both places is shared once with the evidence value. server.http-request itself is never among them. Both the data sent and the tracker's hash are now built from these found values, so the hash is of the values that would be shared.
3. Evidence no element accepts is set aside, not discarded. Python's Evidence.filter(), which FlowData.process() runs before any element, used to delete each key no element accepts. With usage sharing accepting none, that would delete every header only usage sharing read, before it could read it, including evidence a caller adds by hand, which the brief says must still be shared. So filter() now moves those keys to a new Evidence.get_unaccepted(), and get_all() holds what it held before for every other element.
4. The web integration adds server.http-request. fiftyone_pipeline_core.web.webevidence() adds a read-only view, a types.MappingProxyType, of the headers, cookies, query string and form fields it read, keyed as evidence is (header.<name>, cookie.<name>, query.<name>). It does not add the Flask request object itself, because the other elements read get_all() as a dictionary of strings. The Device Detection and IP Intelligence on-premise engines pass every value to their native evidence map, and the cloud request engine sends every key that is not a header, cookie or query value, so a request object there would break the first and be sent by the second. A view keyed as evidence is also what the element already knows how to read. Session values are not in the view, because webevidence() has never read the Flask session and usage sharing has never shared one, and the brief says nothing else about what is shared changes.
5. server.http-request reaches no other element. Evidence.filter() always sets it aside, whatever the filters accept, because three elements the builder adds accept every key until Three elements the pipeline builder adds accept every evidence key #52 is fixed. HTTP_REQUEST_KEY in fiftyone_pipeline_core.evidence names it. share_usage.py writes the key out rather than importing it, and reads get_unaccepted() only when it exists, so a newer fiftyone_pipeline_engines_fiftyone still loads beside an older fiftyone_pipeline_core, sharing less.

What is shared is the same as before for the same request. A header only usage sharing read moves from get_all() to get_unaccepted(), and the element reads both.

The other elements that accept every key

#52 found that the sequence element, the JavaScript builder and the set header element accept every key, and the JSON bundler does too. With usage sharing accepting none, they were the only reason a default pipeline still accepted every header. James decided on 7 November that they narrow, which #63 does as a draft of its own, and its check with #59 merged in gives Vary: Host, User-Agent for a default pipeline. CloudEngine and the cloud aspect engines built on it still accept every key, which #63 asks about.

How to verify

Red is rc at 0507eef with only the tests added (0f6be78), and green is this branch. Both ran the same temporary workflow on Ubuntu and Windows with Python 3.12, running every test of fiftyone_pipeline_core, fiftyone_pipeline_engines_fiftyone and fiftyone_pipeline_cloudrequestengine, and tests/test_cloud_mixed_web_offline.py of the pipeline examples.

1. Red (run 42417663963, failure on Ubuntu and Windows). The pipeline accepted If-None-Match for usage sharing, so a revalidation's accepted evidence differed.

   E       AssertionError: True is not false : header.if-none-match
   E       +  'header.if-none-match': '"a-tag"',
   E       AttributeError: 'Evidence' object has no attribute 'get_unaccepted'
   FAILED tests/test_share_usage_accepts_no_evidence.py::ShareUsageAcceptsNoEvidenceTests::test_a_revalidation_has_the_same_accepted_evidence
   FAILED tests/test_share_usage_accepts_no_evidence.py::ShareUsageAcceptsNoEvidenceTests::test_the_pipeline_accepts_no_header_for_usage_sharing
   FAILED tests/test_evidence_unaccepted.py::UnacceptedEvidenceTests::test_evidence_no_element_accepts_is_set_aside
   FAILED tests/test_evidence_unaccepted.py::UnacceptedEvidenceTests::test_the_web_request_is_added_and_set_aside
   

   The other new tests passed on rc as well, because they check that what is shared does not change.
2. Green (run 42418385363, on 352c3f9). fiftyone_pipeline_core 287 passed, fiftyone_pipeline_engines_fiftyone 40 passed and the pipeline examples' offline tests 12 passed, on both. The run reads as a failure only because 4 tests in the cloud request engine's test_cloudengine.py failed with "You need to create a resource key", on rc in the red run as on this branch. They call the live cloud service and the workflow gave them no key. The full pull request build, which gives them one, is below.
   A first green run on 89347b9 (run 42418000569) failed test_session_cookie in test_usage.py, which asserted that the element accepted the session cookie. 352c3f9 changes that assertion, as described below.
3. Full build. common-ci's nightly-pull-request.yml run as a dry run on this pull request at 352c3f9 over all 15 entries in ci/options.json (run 42418841312, success, 17 jobs passed and the two performance comparison jobs skipped). On Ubuntu with Python 3.12 (job) the cloud request engine's tests, given the Bespoke key, gave 40 passed and 2 skipped (the licence key tests), the Device Detection cloud tests 14 passed, the Device Detection examples 31 passed, the IP Intelligence examples 20 passed and 3 skipped, and the Selenium contract tests passed 9 of 9 against each Device Detection web example, the on-premise one holding usage sharing. The on-premise engines, which pass every evidence value to native code, ran with webevidence() adding the web request.

The brief's four tests are these.

1. The pipeline accepts no header for usage sharing. test_the_pipeline_accepts_no_header_for_usage_sharing builds a pipeline of the usage sharing element and an element accepting header.user-agent alone, and checks that neither header.if-none-match nor header.x-test is accepted.
2. The web request is still shared by the same rules. test_the_web_request_is_shared_by_the_same_rules shares X-Test, which no other element accepts, a 51D_ cookie and a 51D_ query parameter, and leaves out the Cookie header, a blocked header, an ordinary cookie, a query parameter not on the list and server.http-request.
3. A revalidation keeps the same accepted evidence. Python's client-side endpoints have no ETag on rc, and Set the client-side cache headers, with an ETag that is the same in every process and none after errors #59 adds one, so the test here, test_a_revalidation_has_the_same_accepted_evidence, checks the evidence the ETag hashes, which is what the pipeline accepts. A request and its revalidation carrying If-None-Match give the same accepted evidence. With Set the client-side cache headers, with an ETag that is the same in every process and none after errors #59, test_revalidation_with_the_etag_is_answered_not_modified there checks the 304 itself, and test_the_web_request_value_is_not_in_the_etag there checks that server.http-request does not change the ETag.
4. Evidence added by hand is shared as before. test_evidence_added_by_hand_is_shared_as_before adds a header no element accepts, a 51D_ cookie, an ordinary cookie and the client IP by hand, with no web request, and they are shared as before. The existing tests in test_usage.py, which add evidence by hand to a default pipeline, all pass.

test_a_key_in_both_places_is_shared_once_with_the_evidence_value and test_a_repeat_is_told_apart_by_what_would_be_shared check the rest of the brief. test_evidence_unaccepted.py checks that filter() sets aside what no element accepts, and that server.http-request is added and set aside even when an element accepts every key. test_the_web_request_is_not_sent_to_the_cloud in the pipeline examples checks that the cloud request engine never sends it.

test_session_cookie in test_usage.py asserted that the element accepted the session cookie. It now asserts that the element does not, and that the tracker's filter reads it.

Other drafts on the same files

• Pause usage sharing sends after repeated failures #53 changes share_usage.py and test_usage.py.
• Set the client-side cache headers, with an ETag that is the same in every process and none after errors #59 changes fiftyone_pipeline_core/web.py and adds tests at the end of test_cloud_mixed_web_offline.py, as this does. Set the client-side cache headers, with an ETag that is the same in every process and none after errors #59 leaves server.http-request out of its ETag, so the two work in either order.

What remains

• The three elements of Three elements the pipeline builder adds accept every evidence key #52.
• fiftyone_pipeline_core and fiftyone_pipeline_engines_fiftyone change, so they are released, and so are the packages that depend on them.