[jwrosewell]

Python's usage sharing has no back-off, so every package is sent, and lost, while the service is failing. This adds the back-off that 51degrees-dotnet#88 restores in .NET, with the same five settings and defaults as constructor arguments.

Problem

ShareUsage.share_send_usage (lines 225 to 249) hands each full package to a thread pool with no limit on its work queue, and send_thread ignores the status of the answer. While the service is failing, every package is sent and lost, the requests to the failing service are not limited, and nothing records that sends are failing. The queue of entries waiting to be sent has no size limit either. .NET had a back-off from pipeline-dotnet#218, but it blocked processing as well as sending and was removed in pipeline-dotnet#261. The back-off is now to be part of the specification and of every usage sharing element.

Change

1. A send fails when the request raises or the answer is not 200.
2. When failures_to_enter_recovery sends fail within failures_window_seconds, sending pauses for initial_recovery_delay_seconds. Successful sends in between do not reset the count.
3. When a pause ends, one send is tried. If it fails, the pause is multiplied by recovery_multiplier, up to max_recovery_delay_seconds. If it succeeds, the back-off ends and the next pause starts again from the initial delay.
4. Processing does not change. Evidence is queued during a pause, and new evidence is dropped only when the queue is full. The new max_queue_size argument sets the size, by default 1000 or ten times requested_package_size if that is larger, as in .NET.
5. One thread sends at a time, as in .NET and Java. It sends packages of requested_package_size entries until fewer are queued. A timer that does not keep the process running starts sending when a pause ends, so the queue drains without any more requests being processed.
6. Closing cancels the timer and sends nothing while paused. It logs how many entries were not sent.
7. A warning is logged when sending pauses or a pause is lengthened, and information when sending resumes. An answer other than 200 is now logged as a warning.

Argument	Default	Accepted values
max_queue_size	1000, or ten times requested_package_size if larger	at least requested_package_size
failures_to_enter_recovery	10	1 to 100
failures_window_seconds	100	1 or more
initial_recovery_delay_seconds	2.0	more than 0
max_recovery_delay_seconds	300.0	more than 0
recovery_multiplier	2.0	1 or more

How it is built:

• ShareUsageBackOff in share_usage_backoff.py holds the rules. Each method takes the time from the caller, so its tests set the time. The element uses time.monotonic(), so a change to the system clock does not change a pause.
• share_send_usage now starts the sending thread if it is not running, enough entries are queued and sending is not paused. send_thread returns whether the service accepted the message.
• Arguments the pause cannot use raise ValueError.

Tests

The first two tests in test_usage.py use only the default settings, so they ran on rc with only the tests added (run 42348732712). Both failed on Ubuntu and Windows, and the other 34 tests passed.

Test	Failure on rc
test_back_off_pauses_then_drains_queue	Fifteen sends had been made 0.7 seconds after the tenth failed send, where ten were expected.
test_back_off_close_while_paused	Twelve sends were made, where ten were expected.

The test receiver can now answer with any status, and records every request as well as the data it accepts. test_back_off_delay_grows_to_maximum, test_back_off_full_queue_drops_new_evidence and test_back_off_invalid_settings use the new arguments, so they were added with the change, as was test_share_usage_backoff.py, which checks the rules with set times.

With the change (run 42348961812) all 46 tests in the package pass on Ubuntu and Windows.

The full pull request build (run 42349561927) passed on every configuration, at head 94d89a8.

For review

1. Sending now uses one thread at a time, where before each full package was sent on its own thread as soon as it filled. With a slow service, entries now wait in the queue rather than as requests in flight, and the queue limit applies.
2. Closing still does not send a package smaller than requested_package_size, as before. The specification's cleanup section asks for one, which is a separate change.

Related

• specifications#97 makes this back-off and its settings part of the specification for every language.
• The same back-off for Java is 51degrees-java#62 and for Node 51degrees-node#36.