Repository navigation
feat(automation): add a low-disk line to the foreman's health digest - #167
chriscoveries wants to merge 6 commits into
Conversation
…otice A Mail notice is sent once, and its delivery proves nothing: a notice typed into a busy terminal is accepted and can still be lost, and then nobody hears about that Mail again. With the new group delivery setting `health_digest_interval_seconds` above zero, the unread tick also sends the foreman one `system.notify(kind="health_digest")` that lists, one line per Actor, Mail still unread `mail_overdue_after_seconds` (default 7200) after its first notice: how many items, how long the oldest has waited, and how many notices went out. A problem the last digest listed is not sent again unless `health_digest_repeat_seconds` is set; a new one goes out at most once per interval. The digest copies no message bodies. The default, 0, sends no digest.
Nothing in CCCC watches the volume that holds CCCC_HOME, where the ledgers, blobs and runtime state of every group grow. With the health digest on, the unread tick now samples that volume (statvfs) and lists it as a problem while its available space is under `disk_critical_percent` (default 5, key `disk:critical`) or under `disk_warning_percent` or `disk_warning_bytes` (defaults 15 and 20 GiB, key `disk:warning`); zero turns a mark off, and with all three off the volume is not sampled. A level the last digest listed holds until the volume is one point and 1 GiB clear of its mark, so a volume hovering at a mark does not flap; a worse level is a new key and is sent. Not knowing is not clear: a sample that fails is listed as `disk:unavailable` with its error, and the level the last digest listed is listed again beside it, so that level keeps holding until a sample shows the volume clear of it. A low disk can recover and come back under the same key, which the digest could not tell from a problem it already listed. So once nothing the last digest listed remains, one more digest says so, with no problem keys; a problem after that is new. This also tells the foreman when overdue Mail has been read.
|
I am deferring this and its #166 dependency. Disk capacity belongs to the CCCC_HOME host: reporting the same volume through every Group's foreman duplicates notifications and couples host monitoring to Mail delivery policy. A smaller useful direction is one passive host-level disk diagnostic, sampled independently of Group notice ticks. Any notification policy should have one host-level owner and explicit recovery behavior. Please also account for current main's NoticeState production path rather than extending only the legacy test reducer. The tests currently assert that the machine has more than 3 GiB available. That makes correctness tests depend on CI/deployment disk capacity. Test threshold/recovery decisions with supplied sample values, and keep a separate small filesystem probe for successful and failed sampling; it should not require gigabytes of free space. |
Main now finds due Mail notices with an incremental projection (inbox/notices.rs) and keeps the ordered reducer only as a test-time cross-check. The health digest moves into the projection: the notice state remembers the last digest (its time and problem keys) and the time of each Mail notice, and the projection lists Mail still unread mail_overdue_after_seconds after its first notice and sends the digest to the foreman under the same rules. The ordered reducer keeps the digest, so the cross-check covers it.
Main comes in through the health digest branch it builds on, already merged with main. Main now finds due Mail notices with an incremental projection (inbox/notices.rs) and keeps the ordered reducer only as a test-time cross-check. The low-disk line moves into the projection with the rest of the digest: the sampled CCCC home volume adds its problems there, and a digest saying everything listed is resolved goes out under the same rules.
The unread pass returned before it looked at the digest when Mail and reply notices were both off, so a group that wants only the digest never got one. The early return now also requires the digest to be off. The notice paths are unchanged: a delay of 0 still turns its notice off.
Brings the digest-only fix and its test. This branch had already moved the early return after the digest with the same condition, so its code stands; the new test turns the disk marks off, as the other digest tests here do.
Nothing in CCCC watches the volume that holds
CCCC_HOME, where everygroup's ledger, blobs and runtime state grow. Nobody hears that it is
filling until writes start to fail.
This builds on the foreman's health digest (
feat/foreman-health-digest-v2on our fork, the PR before this one; until that lands this branch carries
its commit too). With the digest on, the unread tick also samples the
CCCC_HOMEvolume (statvfs) and lists it as a problem while its availablespace is under:
disk_critical_percent(default 5): keydisk:critical,CCCC home volume critically low: <available> available of <total> (<percent>%; critical mark 5%).disk_warning_percentordisk_warning_bytes(defaults 15 and 20 GiB):key
disk:warning,CCCC home volume low: <available> available of <total> (<percent>%; warning mark 15% or 20.0 GiB).Zero turns a mark off; with all three off the volume is not sampled. A
level the last digest listed holds until the volume is one percentage
point and 1 GiB clear of its mark, so a volume that hovers at a mark does
not send a digest on every tick; a worse level is a new key and is sent.
A sample that fails is not a clear volume. It is listed as
disk:unavailable(CCCC home volume unavailable: its free space could not be read: <error>.), and the level the last digest listed is listed againbeside it (
CCCC home volume low at its last sample; listed until a sample shows it clear.). That level keeps holding through the failed samples, soonly a sample can end it.
A volume can recover and fall again under the same key, which the digest
alone cannot tell from a problem it already listed. So once nothing the
last digest listed remains, one more digest says so ("Everything the last
health digest listed is resolved.", no problem keys); a problem after that
is new. The same all-clear tells the foreman when overdue Mail has been
read.
How we built it. On
51d9b5ba. The tick now reads the digest settingsbefore its early return, so a group with only the digest on is still
ticked, and samples the volume once per scheduled pass with
fs2, alreadya dependency of the core crate. The pass takes the path it samples (the
CCCC_HOMEroot), so a test can name one that does not exist.disk_problemspicks the level with the hold described above and turns afailed sample into
disk:unavailablebeside the held level; the digestcondition adds the all-clear (
automation.rs).group_settings_updateaccepts the three keys under
delivery(ops/settings.rs), and the IPCstandard documents them, the failed sample and the all-clear. 4 files,
+362/−28, 191 of the added lines the tests.
How we tested it.
CCCC_HOME.It invents no disk figures: the warning mark is placed around that
volume's own available bytes, so the test runs on whatever disk it finds.
A mark 1 GiB above the available space gives one digest to the foreman
keyed
disk:warningwith the line, and not again; a mark 0.5 GiB belowholds the warning; 2 GiB below gives one all-clear digest with no keys;
above again gives a new
disk:warningdigest.failure is the real error of sampling a path that does not exist. After
the
disk:warningdigest, the failed sample gives one digest keyeddisk:unavailableanddisk:warning, and not again while it lasts; asample back within the warning's recovery band (0.5 GiB clear of the
mark) resolves nothing; 2 GiB clear gives the all-clear.
read its Inbox, and nothing after it.
red reference run with only these tests on
51d9b5ba(plus the pathparameter the unit test passes, unused there),
https://github.com/chriscoveries/cccc/actions/runs/39679841612
the core and daemon automation, Mail notice, settings, digest and disk tests
(89 tests).
-D warnings, nextest, Windows smoke):https://github.com/chriscoveries/cccc/actions/runs/39679837909
Tested on: claw, Debian 13, Skylake 8vCPU AVX2/AVX-512, 15G RAM.