Skip to content

feat(automation): add a low-disk line to the foreman's health digest - #167

Closed
chriscoveries wants to merge 6 commits into
ChesterRa:mainfrom
chriscoveries:feat/foreman-digest-low-disk-v6
Closed

chriscoveries wants to merge 6 commits into
ChesterRa:mainfrom
chriscoveries:feat/foreman-digest-low-disk-v6

Conversation

@chriscoveries

Copy link
Copy Markdown
Contributor

Nothing in CCCC watches the volume that holds CCCC_HOME, where every
group's ledger, blobs and runtime state grow. Nobody hears that it is
filling until writes start to fail.

This builds on the foreman's health digest (feat/foreman-health-digest-v2
on our fork, the PR before this one; until that lands this branch carries
its commit too). With the digest on, the unread tick also samples the
CCCC_HOME volume (statvfs) and lists it as a problem while its available
space is under:

  • disk_critical_percent (default 5): key disk:critical,
    CCCC home volume critically low: <available> available of <total> (<percent>%; critical mark 5%).
  • disk_warning_percent or disk_warning_bytes (defaults 15 and 20 GiB):
    key disk:warning,
    CCCC home volume low: <available> available of <total> (<percent>%; warning mark 15% or 20.0 GiB).

Zero turns a mark off; with all three off the volume is not sampled. A
level the last digest listed holds until the volume is one percentage
point and 1 GiB clear of its mark, so a volume that hovers at a mark does
not send a digest on every tick; a worse level is a new key and is sent.

A sample that fails is not a clear volume. It is listed as
disk:unavailable (CCCC home volume unavailable: its free space could not be read: <error>.), and the level the last digest listed is listed again
beside it (CCCC home volume low at its last sample; listed until a sample shows it clear.). That level keeps holding through the failed samples, so
only a sample can end it.

A volume can recover and fall again under the same key, which the digest
alone cannot tell from a problem it already listed. So once nothing the
last digest listed remains, one more digest says so ("Everything the last
health digest listed is resolved.", no problem keys); a problem after that
is new. The same all-clear tells the foreman when overdue Mail has been
read.

How we built it. On 51d9b5ba. The tick now reads the digest settings
before its early return, so a group with only the digest on is still
ticked, and samples the volume once per scheduled pass with fs2, already
a dependency of the core crate. The pass takes the path it samples (the
CCCC_HOME root), so a test can name one that does not exist.
disk_problems picks the level with the hold described above and turns a
failed sample into disk:unavailable beside the held level; the digest
condition adds the all-clear (automation.rs). group_settings_update
accepts the three keys under delivery (ops/settings.rs), and the IPC
standard documents them, the failed sample and the all-clear. 4 files,
+362/−28, 191 of the added lines the tests.

How we tested it.

  • New core test on the real volume under the test's temporary CCCC_HOME.
    It invents no disk figures: the warning mark is placed around that
    volume's own available bytes, so the test runs on whatever disk it finds.
    A mark 1 GiB above the available space gives one digest to the foreman
    keyed disk:warning with the line, and not again; a mark 0.5 GiB below
    holds the warning; 2 GiB below gives one all-clear digest with no keys;
    above again gives a new disk:warning digest.
  • New core unit test for a failed sample, on the same real volume: the
    failure is the real error of sampling a path that does not exist. After
    the disk:warning digest, the failed sample gives one digest keyed
    disk:unavailable and disk:warning, and not again while it lasts; a
    sample back within the warning's recovery band (0.5 GiB clear of the
    mark) resolves nothing; 2 GiB clear gives the all-clear.
  • The Mail digest test now also expects the all-clear once the peer has
    read its Inbox, and nothing after it.
  • Without this change, on the health digest branch, all three tests fail:
    red reference run with only these tests on 51d9b5ba (plus the path
    parameter the unit test passes, unused there),
    https://github.com/chriscoveries/cccc/actions/runs/39679841612
  • claw, 2026-10-21 23:20Z: the three tests pass at this head, with the rest of
    the core and daemon automation, Mail notice, settings, digest and disk tests
    (89 tests).
  • Fork CI green (fmt, clippy -D warnings, nextest, Windows smoke):
    https://github.com/chriscoveries/cccc/actions/runs/39679837909

Tested on: claw, Debian 13, Skylake 8vCPU AVX2/AVX-512, 15G RAM.

…otice

A Mail notice is sent once, and its delivery proves nothing: a notice
typed into a busy terminal is accepted and can still be lost, and then
nobody hears about that Mail again.

With the new group delivery setting `health_digest_interval_seconds`
above zero, the unread tick also sends the foreman one
`system.notify(kind="health_digest")` that lists, one line per Actor,
Mail still unread `mail_overdue_after_seconds` (default 7200) after its
first notice: how many items, how long the oldest has waited, and how
many notices went out. A problem the last digest listed is not sent
again unless `health_digest_repeat_seconds` is set; a new one goes out
at most once per interval. The digest copies no message bodies. The
default, 0, sends no digest.
Nothing in CCCC watches the volume that holds CCCC_HOME, where the ledgers,
blobs and runtime state of every group grow. With the health digest on,
the unread tick now samples that volume (statvfs) and lists it as a
problem while its available space is under `disk_critical_percent`
(default 5, key `disk:critical`) or under `disk_warning_percent` or
`disk_warning_bytes` (defaults 15 and 20 GiB, key `disk:warning`); zero
turns a mark off, and with all three off the volume is not sampled. A
level the last digest listed holds until the volume is one point and
1 GiB clear of its mark, so a volume hovering at a mark does not flap; a
worse level is a new key and is sent.

Not knowing is not clear: a sample that fails is listed as
`disk:unavailable` with its error, and the level the last digest listed
is listed again beside it, so that level keeps holding until a sample
shows the volume clear of it.

A low disk can recover and come back under the same key, which the
digest could not tell from a problem it already listed. So once nothing
the last digest listed remains, one more digest says so, with no problem
keys; a problem after that is new. This also tells the foreman when
overdue Mail has been read.

Copy link
Copy Markdown
Owner

I am deferring this and its #166 dependency. Disk capacity belongs to the CCCC_HOME host: reporting the same volume through every Group's foreman duplicates notifications and couples host monitoring to Mail delivery policy.

A smaller useful direction is one passive host-level disk diagnostic, sampled independently of Group notice ticks. Any notification policy should have one host-level owner and explicit recovery behavior. Please also account for current main's NoticeState production path rather than extending only the legacy test reducer.

The tests currently assert that the machine has more than 3 GiB available. That makes correctness tests depend on CI/deployment disk capacity. Test threshold/recovery decisions with supplied sample values, and keep a separate small filesystem probe for successful and failed sampling; it should not require gigabytes of free space.

Main now finds due Mail notices with an incremental projection
(inbox/notices.rs) and keeps the ordered reducer only as a test-time
cross-check. The health digest moves into the projection: the notice state
remembers the last digest (its time and problem keys) and the time of each
Mail notice, and the projection lists Mail still unread
mail_overdue_after_seconds after its first notice and sends the digest to
the foreman under the same rules. The ordered reducer keeps the digest, so
the cross-check covers it.
Main comes in through the health digest branch it builds on, already merged
with main. Main now finds due Mail notices with an incremental projection
(inbox/notices.rs) and keeps the ordered reducer only as a test-time
cross-check. The low-disk line moves into the projection with the rest of
the digest: the sampled CCCC home volume adds its problems there, and a
digest saying everything listed is resolved goes out under the same rules.
The unread pass returned before it looked at the digest when Mail and
reply notices were both off, so a group that wants only the digest never
got one. The early return now also requires the digest to be off. The
notice paths are unchanged: a delay of 0 still turns its notice off.
Brings the digest-only fix and its test. This branch had already moved the
early return after the digest with the same condition, so its code stands;
the new test turns the disk marks off, as the other digest tests here do.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants