[jwrosewell]

This is the fix for #48 and #49, which can close once it reaches main. Session tracking follows .NET, which is the model for usage sharing.

Problem

1. With no options, which is how the device detection and IP intelligence builders add it, the usage sharing element sent every piece of evidence to 51Degrees, including every cookie, every form field and every query string parameter (#48). ShareUsageEvidenceKeyFilter switched on sharing everything whenever no cookie and no lists were given.
2. Once any option was set, every cookie not starting 51d_ raised AttributeError, because the filter read include_session and nothing set it (#49). The two on-premise web examples set "cookie": "", so they reached this on any request carrying an ordinary cookie.
3. Evidence.filter() deleted keys from the dictionary it was looping over, which raises dictionary changed size during iteration whenever a key is removed.
4. Python had only cookie, where .NET has two settings. SetTrackSession, false by default (Constants.cs lines 88 to 95), decides whether the session cookie is used to tell sessions apart, and SetAspSessionCookieName only names the cookie and "has no effect if TrackSession is false". The first version of this pull request made naming cookie turn tracking on.

Change

The filter follows the accepted evidence section and the configuration options of the usage sharing specification, and the .NET filter.

1. ShareUsage takes share_all, False by default, as the specification's "Share all Evidence" is. Only share_all=True shares everything.
2. Otherwise headers are shared unless blocked, cookies and session values only when their names start 51d_, query string and form parameters only when they start 51d_ or are on query_whitelist, and any other evidence. An empty query_whitelist shares only 51d_ parameters, and None shares every parameter, as .NET's SetShareAllQueryStringParameters does.
3. id.email, id.salt, request-query and request-path are never shared, in any mode, matched without regard to case as the whole key or after a ., as in .NET's EvidenceKeyFilterShareUsage.
4. A key is split at its first ., so query.valid.email is checked as valid.email. A key with no . is shared, where it used to raise IndexError, and ShareUsageData.try_add uses find so it no longer raises ValueError on such a key.
5. The header and parameter lists are matched without regard to case.
6. ShareUsage builds three filters once instead of a new filter for every key, as .NET's ShareUsageBase.cs lines 636 to 666 does. The filter for the data sent leaves the session cookie out, and the tracker's filter leaves out query.session-id and query.sequence, which differ on every request and made every request look new to the tracker. The session id and sequence are added to the parameters kept, as .NET does, because the usage data carries them separately.
7. ShareUsage takes track_session, False by default, as .NET's SetTrackSession is. Only when it is True do the pipeline's filter and the tracker's filter include the session cookie named by cookie. A repeat from a new session is then shared again, while a repeat within the same session is not. cookie only names the session cookie, and has no effect unless track_session is True. The session cookie is not sent unless share_all is True.
8. Evidence.filter() loops over a copy of the keys.
9. The usage sharing example's 51d.json, which the example says lists all the options, now lists track_session and share_all with their defaults.

For review

1. The Cookie header is no longer shared unless share_all is set. The specification said the default block list "SHOULD include the cookies HTTP header". HTTP has no header called Cookies, and the request header that carries cookies is Cookie (RFC 6265 section 5.4). The web integration adds it as header.cookie with every cookie in it (web.py), so without this the session and sign-in cookies the cookie rule leaves out would still be sent. The 51d_ cookies are still shared, as their own cookie. entries. specifications#89, now merged into rc, put the Cookie header in the specification, and .NET now blocks it too (51degrees-dotnet#81).
2. cookie keeps its default of None. .NET's default, asp.net_sessionid, is the ASP.NET session cookie, and Java's, JSESSIONID, is the servlet one. Python web frameworks have no common session cookie name, as Flask uses session and Django sessionid, so no name would be right by default. track_session=True therefore needs cookie set to have any effect, and the docstring says so.
3. The pipeline keeps the session cookie only when tracking is on. .NET's filter for the evidence the element accepts includes the session cookie whether or not tracking is on (ShareUsageBase.cs lines 646 to 648), though it never sends it and only compares it when tracking is on. Here the pipeline keeps it only when tracking is on, so that cookie has no effect at all otherwise. What is sent is the same.
4. share_all overrides session tracking, as in .NET. With share_all=True, every cookie is shared and compared, the session cookie included, whatever track_session says. .NET's share all filters do the same (ShareUsageBase.cs lines 659 to 666).
5. specifications#89, now merged into rc, added the four never-shared keys to the specification. specifications#97 adds the session tracking options.

Tests

• tests/test_share_usage_evidencekeyfilter.py is new. It ports the .NET filter tests and adds the cases the issues list: query.id.email, header.id.email, cookie.a.b, session.org.example.attribute, query.valid.email, the session cookie, and lists in mixed case.
• tests/test_usage.py sends through a real receiver and checks what reaches it: the default settings, the never-shared keys, share_all, a parameter name with a dot, an empty and a None parameter list, an ordinary cookie with a header list set, a repeat that differs only in its session id and sequence, and a key with no prefix.
• Three tests in tests/test_usage.py cover session tracking. test_session_cookie turns tracking on and checks the session cookie is kept but not sent. test_session_cookie_without_session_tracking names the cookie without turning tracking on, and checks that tracking is off by default, the cookie is not kept, and the same evidence from a second session is not sent again. test_session_tracking turns tracking on and checks that the same evidence is sent again for a new session but not for a repeat in the same session, and that the cookie is never sent.
• tests/test_core.py checks that evidence no element accepts is removed without an error. The JavaScript and set header elements accept every key, so the test leaves them out.

Checks

• The filter tests alone, at d92184d, failed on Python 3.12 and 3.14 for the reasons above. That covered 18 usage sharing tests and the new core test, including AttributeError: 'ShareUsageEvidenceKeyFilter' object has no attribute 'include_session', RuntimeError: dictionary changed size during iteration, the session cookie and email address reaching the receiver, and 2 != 1 for the repeat. Every existing test passed (run 42192496992, first attempt).
• With the filter fix, at a97de64, all 285 core tests and all 32 usage sharing tests passed on both versions (run 42192496992, second attempt), and the full build passed on all 15 configurations (run 42193217540).
• The session tracking tests alone, at b885b9e, failed on Python 3.12 and 3.14. test_session_cookie and test_session_tracking raised TypeError: ShareUsage.__init__() got an unexpected keyword argument 'track_session', and test_session_cookie_without_session_tracking failed with 2 != 1, because naming the cookie turned tracking on. The other 31 usage sharing tests passed, and the usage sharing example ran from its configuration (run 42320909238).
• With session tracking, at c7385d4, all 34 usage sharing tests passed on both versions, and the usage sharing example ran from its configuration with the two new options (run 42321432587).
• The full build of this pull request, at c7385d4, passed on all 15 configurations, Python 3.10 to 3.14 on Ubuntu, macOS and Windows (run 42322019143).

Found, not changed

• SequenceElementEvidenceKeyFilter and JavaScriptBuilderEvidenceKeyFilter define filter() rather than filter_evidence_key(), and SetHeaderElement has no filter of its own, so all three accept every key. A pipeline built with the default settings therefore keeps all evidence, whatever the other elements accept. What usage sharing sends is decided by its own filter, so this change does not depend on it. That is #52.
• test_usage.py defines test_includes_header twice, so the first one, which checks a header is shared, never runs.
• The usage sharing example in fiftyone_devicedetection_examples/onpremise/shareusage.py turns off certificate checks for every requests.post in the process. That is #51.
• The specification calls the repeat evidence interval a sliding window, reset on every match. Python stores a new time only when it shares a repeat again, and .NET never updates the time once stored. specifications#97 records the difference for a decision.

Release note

Usage sharing now shares only the evidence the specification accepts unless share_all=True is given, never shares an email address, its salt, the query string or the path, and no longer raises an error for ordinary cookies once an option is set. A new track_session option, False by default, uses the session cookie named by cookie to tell sessions apart, as .NET's SetTrackSession does.