diff --git a/.changeset/admission-overload.md b/.changeset/admission-overload.md new file mode 100644 index 000000000..54915b07e --- /dev/null +++ b/.changeset/admission-overload.md @@ -0,0 +1,24 @@ +--- +default: minor +--- + +# zingod-grpc, zingod: one adaptive work limit governs load; fixed caps, pressure signals and overrides removed + +The adaptive work-stream limit is now the single authority on load. It always adapts, with +`max_work_streams` as its ceiling; disk reads run inside it with no cap of their own, and +`GetMempoolStream` takes no slot (the connection caps bound it). `/readyz` reports `down` with +reason `overloaded` once at least 10% of work requests are refused (at one or more a second) for +5 minutes, and clears after 5 minutes under 2%. CPU and IO saturation reach it as latency, which +lowers the limit, so the Linux PSI signals are gone. A stream's only time bound is its client's +`grpc-timeout` (and `unread_timeout_secs` for a client that stops reading). + +- Remove from your config, or zingod refuses to start: `[grpc.overload]`, and under `[grpc]` + `adaptive_work_streams`, `max_subscriptions`, `max_point_reads`, `max_range_reads`, + `max_scan_reads`, `silent_timeout_secs` +- `zingod overload` is removed; delete `/overload/` if it exists +- Metrics: `zingod_grpc_overload_latched{cause}` is now `zingod_grpc_overloaded` (no labels); + `zingod_grpc_active_subscriptions`, `zingod_grpc_subscriptions_rejected_total` and + `zingod_grpc_disk_read_wait_seconds` are gone +- `/readyz` message is `Overloaded: refusing work` +- The `Serving` log line carries `rps`, `conns`, `refused` and `failed` only; `Request + unavailable` is no longer logged diff --git a/book/src/SUMMARY.md b/book/src/SUMMARY.md index 46d3d7b3c..db9e25437 100644 --- a/book/src/SUMMARY.md +++ b/book/src/SUMMARY.md @@ -14,9 +14,10 @@ - [Quick start](operators/quick-start.md) - [Configuration](operators/configuration.md) - [First sync](operators/first-sync.md) +- [Network exposure](operators/network.md) +- [DoS protection](operators/dos-protection.md) - [Monitoring](operators/monitoring.md) - [Maintenance](operators/maintenance.md) -- [Network exposure](operators/network.md) - [Docker](operators/docker.md) - [Migrating from Zaino](operators/migrating-from-zaino.md) diff --git a/book/src/operators/configuration.md b/book/src/operators/configuration.md index 3be508fbd..b2597a08b 100644 --- a/book/src/operators/configuration.md +++ b/book/src/operators/configuration.md @@ -26,7 +26,7 @@ every enabled index, and `p2p.cache_dir` if `[p2p]` is enabled. | `[submission]` | how `SendTransaction` retries relaying | | `[p2p]` | Zingod's own Zcash p2p peers (off by default) | | `[serve]` | gRPC listen address, TLS, gRPC-web, state directory, drain delay | -| `[grpc]` | connection and stream limits, timeouts, trusted proxies, overload thresholds | +| `[grpc]` | [DoS protection](dos-protection.md) limits and timeouts, trusted proxies, shutdown grace | | `[metrics]` | admin listener for metrics and probes (off by default) | | `[sync]` | block fetching and finality depth | | `[index.]` | per index: `enabled` and `path` | diff --git a/book/src/operators/dos-protection.md b/book/src/operators/dos-protection.md new file mode 100644 index 000000000..908d48647 --- /dev/null +++ b/book/src/operators/dos-protection.md @@ -0,0 +1,128 @@ +# DoS protection + +Zingod protects itself with a fixed set of limits, checked in the order a request meets them. A +request over a limit is refused immediately, never queued, so an overloaded node stays responsive +and clients know at once to back off or try another server. + +| Layer | Bounds | Settings (`[grpc]`) | When exceeded | +| ------------- | ------------------------------------------ | -------------------------------------------------------- | ----------------------------------- | +| Connections | open sockets, in total and per client | `max_connections`, `max_connections_per_ip` | connection closed at accept | +| Streams | concurrent requests on one connection | `max_streams_per_connection` | HTTP/2 refuses the stream | +| Work | requests being served across all clients | `max_work_streams` (ceiling of the adaptive limit) | `UNAVAILABLE` with a retry hint | +| Request size | request body and address rows | `[serve] max_address_rows` | `RESOURCE_EXHAUSTED` | +| Progress | clients that stop reading | `unread_timeout_secs` | connection closed | +| Origins | browser sites you block | `zingod origin block` | `PERMISSION_DENIED` | + +The defaults suit a public server on one machine. Every value is in the +[configuration reference](configuration.md#reference). + +## Connections + +- `max_connections` (4096) caps open connections in total. Each one uses a file descriptor: + Zingod raises its soft `RLIMIT_NOFILE` to the hard limit at startup and refuses to start if + `max_connections` plus 1024 does not fit. Raise the hard limit (`LimitNOFILE=` in systemd, + `--ulimit nofile=` in Docker) or lower `max_connections`. +- `max_connections_per_ip` (128) caps connections from one client address. It is high because + mobile carriers put many wallets behind one address (CGNAT). Behind a proxy, every connection + comes from the proxy's address unless it sends a PROXY header; see + [Behind a proxy](network.md#behind-a-proxy). +- `max_streams_per_connection` (100) caps concurrent requests on one connection, matching + lightwalletd. A zingolib wallet uses up to 7. + +Idle and long-lived connections are recycled: a connection with no open stream for +`max_connection_idle_secs` (30 minutes) gets a GOAWAY, which frees its per-client slot. Every +connection also gets one after `max_connection_age_secs` (30 minutes, ±10%), and its in-flight +streams get `max_connection_age_grace_secs` (10 minutes) to finish. + +## Work streams + +Every method except `GetMempoolStream` takes a work slot for as long as it runs, including a long +`GetBlockRange`. `GetMempoolStream` stays open until the next block and is idle most of that time, +so it takes none; the connection limits bound it. + +The number of slots is not fixed. Zingod measures how long it takes to produce each response +message and adjusts the limit from that, using Netflix's Gradient2 algorithm: + +- While the node is busy and response times stay steady, the limit grows slowly. +- When response times rise past 1.5 times their 5-minute average, the limit shrinks within + seconds: to under half in about 5 seconds at 4 times the usual response time. +- While the node is quiet (fewer than half the slots in use) the limit does not move. +- It starts at 256, never drops below 32, and never exceeds `max_work_streams` (2048). +- Time spent waiting on a slow client does not count. Only time Zingod itself spends counts. + +CPU or disk saturation shows up as slower responses, so the limit tracks what the machine can +actually serve. Disk reads have no separate limit: each runs inside a work stream, so a slow disk +lowers the work limit. `max_work_streams` is a safety ceiling for costs that do not show up as +latency, such as memory. You rarely need to change it. + +A request over the limit gets `UNAVAILABLE` (`work stream limit reached`) with a +`grpc-retry-pushback-ms: 250` header. Wallets retry after that delay, or move to another server. + +The work limit is shared by all clients and filled first come, first served. A single address can +hold up to `max_connections_per_ip × max_streams_per_connection` streams (12,800 by default), +more than the whole work limit, so one aggressive client can take every slot until its streams +finish. On a public server, rate-limit per client address at your proxy or edge, or lower those +two settings if your clients do not share addresses. + +## Request size + +- A request body over 64 KiB is refused with `RESOURCE_EXHAUSTED`. `SendTransaction` allows + 2 MB + 1 KiB, the largest valid transaction. +- An address request that would walk more than `[serve] max_address_rows` (100,000) received + transparent outputs, across all its addresses, is refused with `RESOURCE_EXHAUSTED`. + +## Slow clients + +Streams have no total time limit; a full sync takes as long as it takes. A client's own +`grpc-timeout` is honoured as its deadline. A client that stops reading data Zingod has sent for +`unread_timeout_secs` (300) has its whole connection closed. + +`/readyz` and `/status` on the gRPC listener never take a work slot. Each connection may send 10 +at once, refilling at one per second; beyond that it gets `429`. + +## Browser origins + +With [gRPC-web](network.md#browser-clients-grpc-web) enabled, any website can make its visitors' +browsers call your server. `zingod origin` refuses traffic from sites you choose. Changes apply +within a second and persist until changed: + +| Command | Effect | +| ------------------------------- | -------------------------------------------------------------- | +| `zingod origin block ` | refuse requests from the origin with `PERMISSION_DENIED` | +| `zingod origin allow ` | serve the origin even when a parent domain is blocked | +| `zingod origin reset ` | remove the origin's policy | +| `zingod origin` | list the policies in effect | + +- Policies apply to the registrable domain and its subdomains. The most specific policy wins, so + `allow app.example.com` can override `block example.com`. +- Every origin that is not blocked meets the same limits as any other client. Clients can set any + `Origin` header, so an origin never lifts a limit. +- `zingod_grpc_origin_requests_total{verdict}` counts browser requests by policy. To find an + abusive origin, check your proxy or edge logs. + +## Overload + +The work limit decides when a node is overloaded. If at least 10% of work requests are refused, +at one or more refusals a second, for 5 minutes, `/readyz` reports `down` with reason +`overloaded` and the log shows `Overload latched` (WARN). After 5 minutes with fewer than 2% +refused, it clears (`Overload cleared`, INFO). + +An overloaded node keeps serving up to its limit. The `down` state tells a load balancer to send +new clients elsewhere until the node recovers. There is nothing to tune: if a node is often +overloaded, add capacity or spread clients across more nodes. + +## What to watch + +| Metric | Meaning | +| ---------------------------------------------- | ------------------------------------------------------- | +| `zingod_grpc_adaptive_stream_limit` | the work limit now | +| `zingod_grpc_active_streams` | work streams being served | +| `zingod_grpc_admission_rejected_total` | work requests refused at the limit | +| `zingod_grpc_connections_rejected_total` | connections refused at a connection limit | +| `zingod_grpc_stalled_connections_total` | connections closed by `unread_timeout_secs` | +| `zingod_grpc_overloaded` | 1 while the node reports `overloaded` | + +A work limit that sits well below `max_work_streams` while refusals climb means the machine is the +bottleneck. Refusals with the limit at its ceiling mean `max_work_streams` itself is the +bottleneck. The once-a-minute [`Serving` line](monitoring.md#the-serving-line) summarises +refusals in the log. diff --git a/book/src/operators/maintenance.md b/book/src/operators/maintenance.md index cad4d46c2..833ee0e5f 100644 --- a/book/src/operators/maintenance.md +++ b/book/src/operators/maintenance.md @@ -29,30 +29,6 @@ so dashboards can tell planned maintenance from a crash. Both commands are idempotent. -## Overload - -A node that is persistently short of capacity reports `down` with reason `overloaded`, so load -balancers move traffic away from it. Thresholds are set under `[grpc.overload]` (0 disables a -cause). A cause is raised after its signal stays above the threshold for the enter time, and -cleared after it stays below half the threshold for the exit time. - -| Cause | Signal, per minute | Enter / exit | -| ---------------------------- | ------------------------------------------------------------- | ------------ | -| `CPU pressure`, `IO pressure` | share of time Zingod's cgroup waited (Linux PSI) | 2 / 3 min | -| `refusing work` | at least 10% of requests refused, at 1/s or more (exit: < 2%) | 5 / 5 min | - -Override the detector on a running node, without a restart: - -```sh -zingod overload off # never report overloaded -zingod overload on # back to automatic -zingod overload set # override a threshold, e.g. cpu_pressure_percent 40 -zingod overload unset # back to the config's value -``` - -Overrides are stored in `state_dir` and persist across restarts. Zingod applies them within a -second and logs a warning while any is in effect. - ## systemd ```ini diff --git a/book/src/operators/monitoring.md b/book/src/operators/monitoring.md index 36b960ce6..0e259b798 100644 --- a/book/src/operators/monitoring.md +++ b/book/src/operators/monitoring.md @@ -8,7 +8,7 @@ Zingod reports a single state. The same value appears in `/readyz`, `/status`, t | State | Meaning | `/readyz` | | -------------- | ----------------------------------------------------------------------------------------- | --------- | | `held` | an operator ran [`zingod hold`](maintenance.md#operator-hold) | `503` | -| `down` | cannot serve: the server is stalled or [overloaded](maintenance.md#overload), startup is waiting for a validator, or the tip is unverified and stale | `503` | +| `down` | cannot serve: the server is stalled or [overloaded](dos-protection.md#overload), startup is waiting for a validator, or the tip is unverified and stale | `503` | | `provisioning` | starting up, or installing an [index snapshot](first-sync.md#index-snapshot) | `503` | | `degraded` | serving a recent tip that no trusted validator has confirmed (p2p peers supplied it) | `200` | | `syncing` | serving, but behind the best block; or no validator's mempool is available yet | `503` | @@ -107,13 +107,8 @@ ts=… level=info target=zingod::telemetry::progress message="Syncing blocks" he ### The `Serving` line -Once a minute, the gRPC server logs a `Serving` summary: `rps` (streams completed per second) and -`conns` (open connections). It is logged at WARN if, during that minute, any request failed or -was refused, took more than a second to its first response (`slow`), stalled (`stalled`), or a -connection was refused at a limit (`conns_refused`). These fields appear only when non-zero. - -- `INTERNAL`, `UNKNOWN` and `DATA_LOSS` count as failures. `UNAVAILABLE` and - `RESOURCE_EXHAUSTED` count as refusals. Other codes are client errors, counted in - `zingod_grpc_requests_total{code}`. -- The first failure of each minute is also logged as `Request failed` (ERROR), and the first - refusal not caused by capacity as `Request unavailable` (WARN). +Once a minute, the gRPC server logs a `Serving` summary: `rps` (requests completed per second) +and `conns` (open connections). If any request was refused (`UNAVAILABLE`, `RESOURCE_EXHAUSTED`) +or failed (`INTERNAL`, `UNKNOWN`, `DATA_LOSS`) that minute, it is logged at WARN with `refused` +and `failed` counts. The first failure of each minute is also logged as `Request failed` (ERROR) +with its message. Everything else is in the [metrics](#metrics). diff --git a/book/src/operators/network.md b/book/src/operators/network.md index e65efa997..0035ced4c 100644 --- a/book/src/operators/network.md +++ b/book/src/operators/network.md @@ -10,9 +10,8 @@ Zingod has two listeners: Connections to validators are plain HTTP JSON-RPC. -Limits on connections, streams and reads are set under `[grpc]`; see the -[reference](configuration.md#reference). When a limit is reached, the request is refused, not -queued. +Limits on connections, requests and reads are set under `[grpc]`; see +[DoS protection](dos-protection.md). ## TLS @@ -46,36 +45,12 @@ after changing the listen address, delete `cert.pem`. allowing any origin. Native gRPC is unaffected. Enabling it makes the server usable from any website. A popular site embedding it can send load -proportional to its own traffic. - -`zingod origin` manages per-origin policy. Changes apply within a second and persist until -changed: - -| Command | Effect | -| ------------------------------- | -------------------------------------------------------------- | -| `zingod origin block ` | refuse requests from the origin with `PERMISSION_DENIED` | -| `zingod origin allow ` | serve the origin even when a parent domain is blocked | -| `zingod origin reset ` | remove the origin's policy | -| `zingod origin` | list the policies in effect | - -- Policies apply to the registrable domain and its subdomains. The most specific policy wins, so - `allow app.example.com` can override `block example.com`. -- Every origin that is not blocked meets the same limits as any other client. Clients can set any - `Origin` header, so an origin never lifts a limit. -- `zingod_grpc_origin_requests_total{verdict}` counts browser requests by policy. To find an - abusive origin, check your proxy or edge logs. +proportional to its own traffic. To refuse a site, see [Browser +origins](dos-protection.md#browser-origins). Over HTTP/1.1, browsers open one connection per in-flight stream, each counting against `max_connections_per_ip`. Serve over TLS so browsers use HTTP/2. -## Stream timeouts - -Streams have no total time limit. Two settings close streams that stop making progress: - -- `unread_timeout_secs`: the client stopped reading. The connection is closed. -- `silent_timeout_secs`: the server has had nothing to send. The stream ends with `UNAVAILABLE`. - `GetMempoolStream` is exempt. - ## Behind a proxy Behind a proxy, every connection comes from the proxy's address, so `max_connections_per_ip` would @@ -114,9 +89,3 @@ For HAProxy, use `server zingod 127.0.0.1:8137 send-proxy-v2` in a `mode tcp` ba If your proxy cannot send the header, leave `trusted_proxies` empty and raise `max_connections_per_ip` to `max_connections`. - -## Open files - -Each connection uses a file descriptor. At startup, Zingod raises its soft `RLIMIT_NOFILE` to the -hard limit and refuses to start if `max_connections` plus 1024 does not fit. Raise the hard limit -(`LimitNOFILE=` in systemd, `--ulimit nofile=` in Docker) or lower `max_connections`. diff --git a/book/src/operators/zingod.toml b/book/src/operators/zingod.toml index 60297aab1..e4885ee6e 100644 --- a/book/src/operators/zingod.toml +++ b/book/src/operators/zingod.toml @@ -106,7 +106,7 @@ hold_goaway_delay_secs = 0 # client that pins it connects) # tls = { cert_path = "/etc/zingod/fullchain.pem", key_path = "/etc/zingod/key.pem" } -# Serving caps (defaults shown); every cap refuses rather than queues, except the read lanes +# Serving caps (defaults shown); every cap refuses rather than queues [grpc] # excess accepts closed; + 1024 must fit the open-file limit or boot fails max_connections = 4096 @@ -115,25 +115,13 @@ max_connections_per_ip = 128 # HTTP/2 SETTINGS_MAX_CONCURRENT_STREAMS: every stream on one connection, of any method (as # lightwalletd's grpc-max-concurrent-streams) max_streams_per_connection = 100 -# work streams (every method but GetMempoolStream) across all connections; excess = UNAVAILABLE + -# retry hint +# ceiling of the adaptive work-stream limit (every method but GetMempoolStream, across all +# connections; the limit follows measured latency); excess = UNAVAILABLE + retry hint +# - https://zingolabs.github.io/zingod/operators/dos-protection.html#work-streams max_work_streams = 2048 -# work streams admitted at once follow measured latency, max_work_streams the ceiling: a saturated -# node refuses new work within seconds (UNAVAILABLE + retry hint); false = always max_work_streams -adaptive_work_streams = true -# GetMempoolStream subscriptions (idle until the next block), counted apart from max_work_streams -max_subscriptions = 4096 -# index reads in flight on the blocking pool, per lane; excess waits in its own lane -# - point: one block / tree state / root list; range: one GetBlockRange window; scan: one address -# history -max_point_reads = 64 -max_range_reads = 32 -max_scan_reads = 4 -# no-progress bounds, never a cap on a stream's length (the client's grpc-timeout is its deadline) -# data its client does not read for this long closes its connection +# data its client does not read for this long closes its connection (no cap on a stream's length: +# the client's grpc-timeout is its deadline) unread_timeout_secs = 300 -# a stream with nothing to send for this long ends UNAVAILABLE (GetMempoolStream exempt) -silent_timeout_secs = 300 # GOAWAY after this long (±10% per connection): the client reconnects and re-resolves DNS, so a # load balancer or a replaced node moves long-held wallet connections; 0 = never max_connection_age_secs = 1800 @@ -149,12 +137,6 @@ trusted_proxies = [] # - https://zingolabs.github.io/zingod/operators/maintenance.html#operator-hold shutdown_grace_secs = 10 -# When /readyz reports `Overloaded: ` (0 = that cause off; `zingod overload` overrides live) -[grpc.overload] -# share of each minute zingod's cgroup waited (Linux PSI) -cpu_pressure_percent = 25 -io_pressure_percent = 25 - # One block-fetch pipeline, shared by every index (defaults shown; all non-zero) [sync] # Blocks below the tip kept reorg-able (in memory, served, written once buried); >= 1000 off regtest diff --git a/packages/zingod-grpc/src/adaptive.rs b/packages/zingod-grpc/src/adaptive.rs index 2fba48e28..6d87075e9 100644 --- a/packages/zingod-grpc/src/adaptive.rs +++ b/packages/zingod-grpc/src/adaptive.rs @@ -39,15 +39,12 @@ const FLOOR: usize = 32; /// Starting limit (clamped to `FLOOR..=ceiling`): growth needs load, so a cold start admits a burst const INITIAL: usize = 256; -/// `adaptive` off = the limit stays the ceiling -/// -/// - `sample` = relaxed atomics only; the one sample that closes a window takes `gradient` +/// `sample` = relaxed atomics only; the one sample that closes a window takes `gradient` #[derive(Debug)] pub(crate) struct Limiter { limit: AtomicUsize, floor: usize, ceiling: usize, - adaptive: bool, window: OpenWindow, gradient: Mutex, } @@ -72,9 +69,9 @@ struct Gradient { } impl Limiter { - pub(crate) fn new(ceiling: usize, adaptive: bool) -> Self { + pub(crate) fn new(ceiling: usize) -> Self { let floor = FLOOR.min(ceiling); - let initial = if adaptive { INITIAL.clamp(floor, ceiling) } else { ceiling }; + let initial = INITIAL.clamp(floor, ceiling); let window = OpenWindow { epoch: Instant::now(), opened_us: AtomicU64::new(0), @@ -86,7 +83,6 @@ impl Limiter { limit: AtomicUsize::new(initial), floor, ceiling, - adaptive, window, gradient: Mutex::new(Gradient { estimated_milli: initial.saturating_mul(MILLI), @@ -101,9 +97,6 @@ impl Limiter { /// One message's production time, `in_flight` work streams as it landed pub(crate) fn sample(&self, took: Duration, in_flight: usize) { - if !self.adaptive { - return; - } let window = &self.window; let micros = |duration: Duration| u64::try_from(duration.as_micros()).unwrap_or(u64::MAX); window.sum_us.fetch_add(micros(took), Relaxed); @@ -184,7 +177,7 @@ mod tests { /// ≈ +√limit / 5 per window #[tokio::test(start_paused = true)] async fn a_loaded_limiter_at_a_steady_rtt_grows_from_initial() { - let limiter = Limiter::new(CEILING, true); + let limiter = Limiter::new(CEILING); assert_eq!(limiter.limit(), INITIAL, "cold start"); let grown = windows(&limiter, 20, 10, LOADED).await; @@ -198,7 +191,7 @@ mod tests { /// Baseline 10 ms; floor = `FLOOR` #[tokio::test(start_paused = true)] async fn rtt_past_the_tolerance_shrinks_the_limit_to_the_floor_and_recovery_grows_it() { - let limiter = Limiter::new(CEILING, true); + let limiter = Limiter::new(CEILING); let before = *windows(&limiter, 20, 10, LOADED).await.last().expect("20 windows"); let shrunk = windows(&limiter, 10, 40, LOADED).await; @@ -215,20 +208,11 @@ mod tests { /// Peak in flight < limit / 2: a quiet node never inflates or deflates #[tokio::test(start_paused = true)] async fn an_app_limited_limiter_holds_its_limit_as_rtt_rises() { - let limiter = Limiter::new(CEILING, true); + let limiter = Limiter::new(CEILING); let quiet = windows(&limiter, 1, 10, 1).await; let slow = windows(&limiter, 20, 400, 1).await; assert_eq!(slow, vec![quiet[0]; 20], "1 in flight under {}", quiet[0]); } - - #[tokio::test(start_paused = true)] - async fn a_fixed_limiter_stays_at_its_ceiling() { - let fixed = Limiter::new(CEILING, false); - - let limits = windows(&fixed, 20, 400, LOADED).await; - - assert_eq!(limits, vec![CEILING; 20], "adaptive off, 40× RTT under load"); - } } diff --git a/packages/zingod-grpc/src/admission.rs b/packages/zingod-grpc/src/admission.rs index 12ceda573..07f16b7d8 100644 --- a/packages/zingod-grpc/src/admission.rs +++ b/packages/zingod-grpc/src/admission.rs @@ -1,10 +1,10 @@ -//! Stream admission: a [`Slot`], or `UNAVAILABLE` + pushback, never a queue +//! Work-stream admission: a [`Slot`] under the adaptive limit, or `UNAVAILABLE` + pushback, never a +//! queue //! //! - Slot owned by the response *body* (`guard.rs`), not the handler (released at handler return //! = unbounded streams) -//! - Two pools by [`Class`]: a subscription idles for a block, so it never holds a work slot (one -//! mempool stream per wallet would otherwise fill `max_work_streams` with idlers) -//! - Work held exactly to the adaptive limit (`adaptive.rs`), fed each message's production time +//! - `GetMempoolStream` takes none (idle until the next block: bounded by the connection caps) +//! - each slot feeds its messages' production time to the limiter (`adaptive.rs`) use std::sync::atomic::{AtomicUsize, Ordering}; use std::sync::Arc; @@ -16,8 +16,7 @@ use tonic::Status; use crate::adaptive::Limiter; use crate::metrics; -use crate::overload::{Overload, OverloadCause, Signals}; -use crate::OverloadLimits; +use crate::overload::Overload; /// Refusal retry hint, ms (held streams = wallet syncs: a slot frees well under a second) const PUSHBACK_MS: &str = "250"; @@ -25,24 +24,11 @@ const PUSHBACK_MS: &str = "250"; /// gRPC's own "come back later" header (grpc-core) const PUSHBACK_HEADER: &str = "grpc-retry-pushback-ms"; -/// Cap a request counts against (each method's in [`Method::route`](crate::service::Method::route), -/// unknown path = `Work`) -/// -/// - `Work` = unary, ranges, scans -/// - `Subscription` = open until the next block, idle between arrivals -#[derive(Clone, Copy, Debug, PartialEq, Eq)] -pub(crate) enum Class { - Work, - Subscription, -} - -/// The two admission pools, the adaptive work limit and their outcomes, shared by every connection +/// Work streams held, the adaptive limit and its outcomes, shared by every connection /// ([`GrpcService::admission`](crate::GrpcService::admission) = its readiness view) #[derive(Debug)] pub struct AdmissionState { - work: AtomicUsize, - subscriptions: AtomicUsize, - max_subscriptions: usize, + held: AtomicUsize, limiter: Limiter, overload: Overload, } @@ -50,17 +36,15 @@ pub struct AdmissionState { impl AdmissionState { pub(crate) fn new(limits: &crate::GrpcLimits) -> Self { Self { - work: AtomicUsize::new(0), - subscriptions: AtomicUsize::new(0), - max_subscriptions: limits.max_subscriptions.get(), - limiter: Limiter::new(limits.max_work_streams.get(), limits.adaptive_work_streams), - overload: Overload::new(limits.overload.clone()), + held: AtomicUsize::new(0), + limiter: Limiter::new(limits.max_work_streams.get()), + overload: Overload::default(), } } - /// First latched cause as of the last serving summary (rules: `overload.rs`) - pub fn overload(&self) -> Option { - self.overload.cause() + /// Refusing work persistently, as of the last serving summary (rules: `overload.rs`) + pub fn overloaded(&self) -> bool { + self.overload.latched() } #[cfg(test)] @@ -68,51 +52,36 @@ impl AdmissionState { self.limiter.limit() } - /// The operator's overload thresholds + whether a latched cause is reported (next window on) - pub fn set_overload(&self, limits: OverloadLimits, detection: bool) { - self.overload.set(limits, detection); - } - - /// Closes a serving summary's window (`elapsed` long, on its `signals`): its refusals, every - /// class - pub(crate) fn tick(&self, elapsed: Duration, signals: Signals) -> u64 { + /// Closes a serving summary's window (`elapsed` long) + pub(crate) fn tick(&self, elapsed: Duration) { metrics::adaptive_stream_limit(self.limiter.limit()); - self.overload.tick(elapsed, signals) - } - - /// `class`'s pool + its limit now (work: the adaptive one) - fn pool(&self, class: Class) -> (&AtomicUsize, usize) { - match class { - Class::Work => (&self.work, self.limiter.limit()), - Class::Subscription => (&self.subscriptions, self.max_subscriptions), - } + self.overload.tick(elapsed); } - /// Streams of `class` held now - pub(crate) fn held(&self, class: Class) -> usize { - self.pool(class).0.load(Ordering::Relaxed) + pub(crate) fn held(&self) -> usize { + self.held.load(Ordering::Relaxed) } - /// A slot, or `None` = refused at once at the class's limit - pub(crate) fn admit(self: &Arc, class: Class) -> Option { - let (held, limit) = self.pool(class); + /// A slot, or `None` = refused at once at the limit + pub(crate) fn admit(self: &Arc) -> Option { + let limit = self.limiter.limit(); let taken = |held: usize| (held < limit).then_some(held + 1); - if held.fetch_update(Ordering::Relaxed, Ordering::Relaxed, taken).is_err() { - self.overload.refused(class); + if self.held.fetch_update(Ordering::Relaxed, Ordering::Relaxed, taken).is_err() { + self.overload.refused(); + metrics::stream_rejected(); return None; } - self.overload.admitted(class); - metrics::stream_admitted(class); - Some(Slot { class, admission: Arc::clone(self), busy_since: Some(Instant::now()) }) + self.overload.admitted(); + metrics::stream_admitted(); + Some(Slot { admission: Arc::clone(self), busy_since: Some(Instant::now()) }) } } -/// One admitted stream's place in its pool, back on drop +/// One admitted stream's place, back on drop /// /// - `busy_since` = producing its next message since (`None` = handed one, not polled since) #[derive(Debug)] pub(crate) struct Slot { - class: Class, admission: Arc, busy_since: Option, } @@ -122,30 +91,24 @@ impl Slot { self.busy_since.get_or_insert_with(Instant::now); } - /// A data frame out: one work message's production time to the limiter + /// A data frame out: its production time to the limiter pub(crate) fn handed(&mut self) { - let since = self.busy_since.take(); - if let (Class::Work, Some(since)) = (self.class, since) { - let in_flight = self.admission.held(Class::Work); - self.admission.limiter.sample(since.elapsed(), in_flight); + if let Some(since) = self.busy_since.take() { + self.admission.limiter.sample(since.elapsed(), self.admission.held()); } } } impl Drop for Slot { fn drop(&mut self) { - self.admission.pool(self.class).0.fetch_sub(1, Ordering::Relaxed); - metrics::stream_released(self.class); + self.admission.held.fetch_sub(1, Ordering::Relaxed); + metrics::stream_released(); } } /// `UNAVAILABLE` + pushback hint, before any handler runs -pub(crate) fn refused(class: Class) -> Response { - let message = match class { - Class::Work => "work stream limit reached", - Class::Subscription => "subscription limit reached", - }; - let mut response = Status::unavailable(message).into_http(); +pub(crate) fn refused() -> Response { + let mut response = Status::unavailable("work stream limit reached").into_http(); response.headers_mut().insert(PUSHBACK_HEADER, HeaderValue::from_static(PUSHBACK_MS)); response } diff --git a/packages/zingod-grpc/src/deadline.rs b/packages/zingod-grpc/src/deadline.rs index adcc0eb33..7f727b25d 100644 --- a/packages/zingod-grpc/src/deadline.rs +++ b/packages/zingod-grpc/src/deadline.rs @@ -1,12 +1,8 @@ -//! Time bounds on one request: the client's `grpc-timeout`, and a silence bound, never a total cap +//! Client's `grpc-timeout` = a request's only time bound (never a total cap) //! -//! - deadline = `grpc-timeout` at arrival: past it, a pending handler = `DEADLINE_EXCEEDED`, a -//! stream ends in `DEADLINE_EXCEEDED` trailers -//! - silence = nothing to send for `silent_timeout` while hyper waits on the handler or the body -//! → `UNAVAILABLE` (retry) -//! - client-side twin = `stall.rs` (`unread_timeout` → connection closed) -//! - `GetMempoolStream` exempt (silent until the next block by design) -//! - a long stream that keeps delivering lives as long as the client wants (Z#1537) +//! - past it: pending handler = `DEADLINE_EXCEEDED`, a stream ends in `DEADLINE_EXCEEDED` trailers +//! - a stream that keeps delivering lives as long as the client wants (Z#1537) +//! - client stopped reading = `stall.rs` (`unread_timeout` → connection closed) use std::future::Future; use std::pin::Pin; @@ -42,80 +38,36 @@ fn requested(headers: &HeaderMap) -> Option { } } -/// One request's bounds, fixed at arrival (`silent_limit` = `None`: never cut for silence) -#[derive(Clone, Copy, Debug)] -pub(crate) struct Bounds { - deadline: Option, - silent_limit: Option, +/// `headers`' deadline, fixed at arrival +pub(crate) fn of(headers: &HeaderMap) -> Option { + requested(headers).map(|timeout| Instant::now() + timeout) } -impl Bounds { - pub(crate) fn of(headers: &HeaderMap, silent_limit: Option) -> Self { - Self { deadline: requested(headers).map(|timeout| Instant::now() + timeout), silent_limit } - } - - /// `answer` cut at the first bound - /// - /// - stream's body (status in trailers) bounded the same - pub(crate) async fn apply( - self, - answer: impl Future>, - ) -> Response { - let first_message = self.silent_limit.map(|limit| Instant::now() + limit); - let cut = match (self.deadline, first_message) { - (Some(deadline), Some(silent)) => Some(deadline.min(silent)), - (deadline, silent) => deadline.or(silent), - }; - let response = match cut { - Some(at) => match tokio::time::timeout_at(at, answer).await { - Ok(response) => response, - Err(_) => return self.cut_at(at).into_http(), - }, - None => answer.await, - }; - // grpc-status in the headers = complete (unary or a refusal): nothing left to bound - if response.headers().contains_key("grpc-status") { - return response; - } - let deadline = self.deadline.map(|at| Box::pin(tokio::time::sleep_until(at))); - let (silent_limit, silent_timer) = (self.silent_limit, None); - response.map(|inner| { - Body::new(Bounded { inner: Some(inner), deadline, silent_limit, silent_timer }) - }) - } - - /// Why a cut at `at` ended the request - fn cut_at(&self, at: Instant) -> Status { - match (self.deadline == Some(at), self.silent_limit) { - (false, Some(limit)) => silenced(limit), - _ => expired(), - } +/// `answer` cut at `deadline`, its stream body too (`None` = never cut) +pub(crate) async fn apply( + deadline: Option, + answer: impl Future>, +) -> Response { + let Some(deadline) = deadline else { return answer.await }; + let Ok(response) = tokio::time::timeout_at(deadline, answer).await else { + return expired().into_http(); + }; + // grpc-status in the headers = complete (unary or a refusal): nothing left to bound + if response.headers().contains_key("grpc-status") { + return response; } + let deadline = Box::pin(tokio::time::sleep_until(deadline)); + response.map(|inner| Body::new(Bounded { inner: Some(inner), deadline })) } fn expired() -> Status { Status::deadline_exceeded("grpc-timeout passed") } -fn silenced(silent: Duration) -> Status { - Status::unavailable(format!("nothing to send for {silent:?}")) -} - -/// A stream body under its [`Bounds`] (`inner` dropped at a cut: its reads and permits go with it) -/// -/// - `silent_timer` runs from the first `Pending` since the last frame, never while hyper holds one +/// Stream body under its deadline (`inner` dropped at the cut: its slot goes with it) struct Bounded { inner: Option, - deadline: Option>>, - silent_limit: Option, - silent_timer: Option>>, -} - -impl Bounded { - fn cut(&mut self, status: Status) -> Poll, Status>>> { - self.inner = None; - Poll::Ready(Some(Ok(Frame::trailers(trailers(&status))))) - } + deadline: Pin>, } impl http_body::Body for Bounded { @@ -128,36 +80,22 @@ impl http_body::Body for Bounded { ) -> Poll, Status>>> { let this = self.get_mut(); let Some(inner) = this.inner.as_mut() else { return Poll::Ready(None) }; - if this.deadline.as_ref().is_some_and(|deadline| deadline.is_elapsed()) { - return this.cut(expired()); - } - match Pin::new(inner).poll_frame(cx) { - Poll::Ready(Some(Ok(frame))) => { - this.silent_timer = None; - if frame.is_trailers() { - this.inner = None; - } + let polled = match this.deadline.is_elapsed() { + true => None, + false => Some(Pin::new(inner).poll_frame(cx)), + }; + match polled { + Some(Poll::Ready(Some(Ok(frame)))) if !frame.is_trailers() => { Poll::Ready(Some(Ok(frame))) } - Poll::Ready(other) => { + Some(Poll::Ready(done)) => { this.inner = None; - Poll::Ready(other) + Poll::Ready(done) } - Poll::Pending => { - if let Some(deadline) = this.deadline.as_mut() { - if deadline.as_mut().poll(cx).is_ready() { - return this.cut(expired()); - } - } - if let Some(limit) = this.silent_limit { - let timer = this - .silent_timer - .get_or_insert_with(|| Box::pin(tokio::time::sleep(limit))); - if timer.as_mut().poll(cx).is_ready() { - return this.cut(silenced(limit)); - } - } - Poll::Pending + Some(Poll::Pending) if this.deadline.as_mut().poll(cx).is_pending() => Poll::Pending, + None | Some(Poll::Pending) => { + this.inner = None; + Poll::Ready(Some(Ok(Frame::trailers(trailers(&expired()))))) } } } @@ -174,7 +112,7 @@ mod tests { use tokio::time::Instant; use tonic::{body::Body, Status}; - use super::{requested, Bounds}; + use super::requested; use crate::wire::trailers; fn timeout(value: &'static str) -> HeaderMap { @@ -229,17 +167,17 @@ mod tests { frames } - /// `Stream`: `count` records, one every `period`, then `OK` trailers (`silent` = pending) + /// `Stream`: `count` records, one every `period`, then `OK` trailers #[derive(Clone, Copy, Debug)] enum Handler { Pending, - Stream { count: u64, period: Duration, silent: bool }, + Stream { count: u64, period: Duration }, } async fn handle(handler: Handler) -> Response { use std::cmp::Ordering::{Equal, Greater, Less}; - let Handler::Stream { count, period, silent } = handler else { + let Handler::Stream { count, period } = handler else { return std::future::pending().await; }; let frames = futures::stream::unfold(0, move |sent: u64| async move { @@ -248,7 +186,6 @@ mod tests { tokio::time::sleep(period).await; Frame::data(Bytes::from_static(b"\0\0\0\0\0")) } - Equal if silent => return std::future::pending().await, Equal => Frame::trailers(trailers(&Status::ok(""))), Greater => return None, }; @@ -257,38 +194,31 @@ mod tests { Response::new(Body::new(StreamBody::new(Box::pin(frames)))) } - /// Z#1537 (paused clock): no total cap, only the client's deadline or the silence bound + /// Z#1537 (paused clock): no total cap, no silence cap, only the client's deadline #[tokio::test(start_paused = true)] - async fn a_stream_lives_while_it_delivers_until_the_clients_deadline_or_a_silence_bound() { - const SILENT: Option = Some(Duration::from_secs(300)); + async fn a_stream_lives_while_it_delivers_until_the_clients_deadline() { let s100 = Duration::from_secs(100); let data = |at: u64| (at, "data".to_owned()); let ended = |at: u64, code: &str| (at, code.to_owned()); let every_100s_for_1000s: Vec<_> = (1..=10).map(|n| data(n * 100)).collect(); - let delivering = Handler::Stream { count: 10, period: s100, silent: false }; + let delivering = Handler::Stream { count: 10, period: s100 }; - // case, grpc-timeout, silence bound, handler, frames as (second, data / status) + // case, grpc-timeout, handler, frames as (second, data / status) #[rustfmt::skip] let cases = [ - ("delivering 1,000 s", HeaderMap::new(), SILENT, delivering, + ("delivering 1,000 s", HeaderMap::new(), delivering, [every_100s_for_1000s, vec![ended(1_000, "Ok")]].concat()), - ("client deadline 250 s", timeout("250S"), SILENT, delivering, + ("client deadline 250 s", timeout("250S"), delivering, vec![data(100), data(200), ended(250, "DeadlineExceeded")]), - ("silent after one record", HeaderMap::new(), SILENT, Handler::Stream { count: 1, period: s100, silent: true }, - vec![data(100), ended(400, "Unavailable")]), - ("subscription, 1,000 s gaps", HeaderMap::new(), None, Handler::Stream { count: 2, period: s100 * 10, silent: false }, + ("1,000 s gaps, no deadline", HeaderMap::new(), Handler::Stream { count: 2, period: s100 * 10 }, vec![data(1_000), data(2_000), ended(2_000, "Ok")]), - ("handler pending, 2 s deadline", timeout("2S"), SILENT, Handler::Pending, + ("handler pending, 2 s deadline", timeout("2S"), Handler::Pending, vec![ended(2, "DeadlineExceeded")]), - ("handler pending, no deadline", HeaderMap::new(), SILENT, Handler::Pending, - vec![ended(300, "Unavailable")]), - ("handler pending, 1 h deadline", timeout("1H"), SILENT, Handler::Pending, - vec![ended(300, "Unavailable")]), ]; - for (case, headers, silent, handler, expected) in cases { + for (case, headers, handler, expected) in cases { let start = Instant::now(); - let response = Bounds::of(&headers, silent).apply(handle(handler)).await; + let response = super::apply(super::of(&headers), handle(handler)).await; assert_eq!(drained(start, response).await, expected, "{case}: {handler:?}"); } diff --git a/packages/zingod-grpc/src/guard.rs b/packages/zingod-grpc/src/guard.rs index d281aa416..0eddfc410 100644 --- a/packages/zingod-grpc/src/guard.rs +++ b/packages/zingod-grpc/src/guard.rs @@ -8,7 +8,7 @@ use std::pin::Pin; use std::sync::Arc; use std::task::{Context, Poll}; -use std::time::{Duration, Instant}; +use std::time::Instant; use bytes::Bytes; use futures::future::BoxFuture; @@ -17,7 +17,7 @@ use http_body::{Body, Frame, SizeHint}; use tonic::{Code, Status}; use tower::ServiceExt as _; -use crate::admission::{self, AdmissionState, Class, Slot}; +use crate::admission::{self, AdmissionState, Slot}; use crate::lifetime::{Activity, Open}; use crate::origin::{OriginGate, OriginVerdict}; use crate::service::Method; @@ -82,15 +82,13 @@ where fn call(&mut self, request: Request) -> Self::Future { let method = Method::of_request(&request); - let class = method.map_or(Class::Work, |method| method.route().class); let mut close = CloseOut { method, started: Instant::now(), - first_message: None, + answered: false, sent: 0, code: None, message: None, - at_capacity: false, ended: false, _open: self.activity.opened(), }; @@ -105,20 +103,18 @@ where None => OriginVerdict::Default, }; if verdict == OriginVerdict::Blocked { - close.code = Some(Code::PermissionDenied); let refusal = Status::permission_denied("origin blocked").into_http(); - let refusal = refusal.map(|inner| Guarded { inner, slot: None, ticket, close }); - return Box::pin(std::future::ready(Ok(refusal))); + return Box::pin(std::future::ready(Ok(refused(refusal, ticket, close)))); } // every unblocked origin = one admission path (Origin client-settable, never lifts a cap) - let slot = match self.admission.admit(class) { - Some(slot) => slot, - None => { - (close.code, close.at_capacity) = (Some(Code::Unavailable), true); - let refusal = admission::refused(class); - let refusal = refusal.map(|inner| Guarded { inner, slot: None, ticket, close }); + let slot = if method.is_some_and(Method::is_subscription) { + None + } else { + let Some(slot) = self.admission.admit() else { + let refusal = refused(admission::refused(), ticket, close); return Box::pin(std::future::ready(Ok(refusal))); - } + }; + Some(slot) }; let inner = self.inner.clone(); @@ -127,12 +123,22 @@ where // header-carried `grpc-status` = unary answer or error, already complete let (code, message) = status_of(response.headers()).unzip(); (close.code, close.message) = (code, message.flatten()); - Ok(response.map(|inner| Guarded { inner, slot: Some(slot), ticket, close })) + Ok(response.map(|inner| Guarded { inner, slot, ticket, close })) }) } } -/// Response body owning its stream's guards (`slot` = `None` on a refusal or once ended) +/// A refusal before any handler ran, measured as the request it was (`code` = its `grpc-status`) +fn refused( + refusal: Response, + ticket: Ticket, + mut close: CloseOut, +) -> Response> { + close.code = status_of(refusal.headers()).map(|(code, _)| code); + refusal.map(|inner| Guarded { inner, slot: None, ticket, close }) +} + +/// Response body owning its stream's guards (`slot` = `None`: a subscription, a refusal, or ended) #[derive(Debug)] pub(crate) struct Guarded { inner: B, @@ -143,17 +149,16 @@ pub(crate) struct Guarded { /// One stream's measurement to its end, recorded on drop /// -/// - `first_message` = admission → first DATA frame +/// - `answered` = first DATA frame sent (its time = `first_message_seconds`) /// - `message` = non-`Ok` status message #[derive(Debug)] struct CloseOut { method: Option, started: Instant, - first_message: Option, + answered: bool, sent: u64, code: Option, message: Option, - at_capacity: bool, ended: bool, _open: Open, } @@ -180,10 +185,9 @@ impl + Unpin> Body for Guarded { if let Some(slot) = this.slot.as_mut() { slot.handed(); } - if close.first_message.is_none() { - let elapsed = close.started.elapsed(); - close.first_message = Some(elapsed); - metrics::first_message(close.method, elapsed); + if !close.answered { + close.answered = true; + metrics::first_message(close.method, close.started.elapsed()); } close.sent += u64::try_from(chunk.len()).expect("usize fits u64"); } @@ -220,16 +224,8 @@ impl Drop for CloseOut { true => Code::Ok, false => Code::Cancelled, }); - let elapsed = self.started.elapsed(); - // answered work request's latency = its client's wait for the first message - let subscription = self.method.is_some_and(|method| method.route().is_subscription()); - let latency = - (code == Code::Ok && !subscription).then(|| self.first_message.unwrap_or(elapsed)); - - metrics::stream_finished(self.method, code, elapsed, latency, self.sent); - if code != Code::Ok && !self.at_capacity { - summary::answered_badly(self.method, code, self.message.as_deref().unwrap_or_default()); - } + metrics::stream_finished(self.method, code, self.started.elapsed(), self.sent); + summary::finished(self.method, code, self.message.as_deref().unwrap_or_default()); } } @@ -243,7 +239,6 @@ mod tests { use tower::Service as _; use super::*; - use crate::overload::Signals; /// - `/open` + `GetMempoolStream` never end /// - `/data` = one frame then waits @@ -306,7 +301,6 @@ mod tests { async fn the_origin_gate_blocks_and_otherwise_defers_to_the_limit() { let limits = crate::GrpcLimits { max_work_streams: NonZeroUsize::MIN, - adaptive_work_streams: false, ..crate::GrpcLimits::default() }; let state = Arc::new(AdmissionState::new(&limits)); @@ -318,7 +312,7 @@ mod tests { // fill the one work slot (held open, never ends) let held = guard.call(request("/open")).await.expect("infallible"); let _held = std::pin::pin!(held.into_body()); - assert_eq!(state.held(Class::Work), 1, "slot taken"); + assert_eq!(state.held(), 1, "slot taken"); let verdict = |response: &Response>| { status_of(response.headers()).map(|(code, _)| code) @@ -332,7 +326,7 @@ mod tests { assert_eq!(verdict(&blocked), Some(Code::PermissionDenied), "blocked origin refused"); assert_eq!(verdict(&allowed), Some(Code::Unavailable), "carve-out refused at the limit"); assert_eq!(verdict(&other), Some(Code::Unavailable), "default origin refused at the limit"); - assert_eq!(state.held(Class::Work), 1, "no refusal took a slot"); + assert_eq!(state.held(), 1, "no refusal took a slot"); } /// Guard → ticket wiring (stall timing itself: `stall.rs`) @@ -376,7 +370,7 @@ mod tests { Arc::default(), OriginGate::new(), ); - let in_flight = || state.held(Class::Work); + let in_flight = || state.held(); let open = guard.call(request("/open")).await.expect("infallible"); assert_eq!(in_flight(), 1, "handler returned, stream still open"); @@ -390,14 +384,11 @@ mod tests { assert_eq!(in_flight(), 0, "read to its end: returned before the drop"); } + /// `GetMempoolStream` idles until the next block: never takes or waits on a work slot #[tokio::test] - async fn subscriptions_have_their_own_pool_beside_work() { - let one = NonZeroUsize::MIN; - let limits = crate::GrpcLimits { - max_work_streams: one, - max_subscriptions: one, - ..Default::default() - }; + async fn subscriptions_open_with_every_work_slot_taken() { + let limits = + crate::GrpcLimits { max_work_streams: NonZeroUsize::MIN, ..Default::default() }; let state = Arc::new(AdmissionState::new(&limits)); let mut guard = Guard::new( Answers, @@ -407,31 +398,26 @@ mod tests { OriginGate::new(), ); let mempool = &Method::GetMempoolStream.path(); - fn refusal(response: &Response) -> Option { - Status::from_header_map(response.headers()) - } - let _work = guard.call(request("/open")).await.expect("infallible"); + let code = |response: &Response<_>| status_of(response.headers()).map(|(code, _)| code); - let subscribed = guard.call(request(mempool)).await.expect("infallible"); - assert_eq!( - refusal(&subscribed).map(|s| s.code()), - None, - "work pool full: subscription admitted" - ); - let second = guard.call(request(mempool)).await.expect("infallible"); - let refused = refusal(&second).map(|status| (status.code(), status.message().to_owned())); - let full = Some((Code::Unavailable, "subscription limit reached".to_owned())); - assert_eq!(refused, full, "second subscription over max_subscriptions 1"); - drop(subscribed); - let third = guard.call(request(mempool)).await.expect("infallible"); - assert_eq!(refusal(&third).map(|s| s.code()), None, "slot back on disconnect"); + let _work = guard.call(request("/open")).await.expect("infallible"); + let refused = guard.call(request("/open")).await.expect("infallible"); + let subscribed: Vec<_> = + futures::future::join_all((0..3).map(|_| guard.call(request(mempool)))) + .await + .into_iter() + .map(|response| code(&response.expect("infallible"))) + .collect(); + + assert_eq!(code(&refused), Some(Code::Unavailable), "work pool of 1 full"); + assert_eq!(subscribed, vec![None; 3], "every subscription opened"); + assert_eq!(state.held(), 1, "subscriptions hold no work slot"); } /// Default limits (`max_work_streams` 2048, adaptive from 256): /// /// - 257th open work stream refused at once (a queue would hang a wallet behind the open /// streams), below the ceiling, `UNAVAILABLE` + pushback, taking no slot - /// - counted once the window closes #[tokio::test] async fn work_past_the_adaptive_limit_is_refused_at_once_below_the_ceiling() { use futures::FutureExt as _; @@ -457,9 +443,7 @@ mod tests { let retry_after = HeaderValue::from_static("250"); assert_eq!(refusal, (Code::Unavailable, "work stream limit reached", Some(&retry_after))); - let refusals = state.tick(Duration::from_secs(1), Signals::default()); - assert_eq!(refusals, 1, "the window's one refusal"); - let work = (state.work_limit(), state.held(Class::Work)); + let work = (state.work_limit(), state.held()); assert_eq!(work, (256, 256), "refused below the 2048 ceiling, no slot of its own"); } } diff --git a/packages/zingod-grpc/src/lib.rs b/packages/zingod-grpc/src/lib.rs index 82ea86cdb..fe9510490 100644 --- a/packages/zingod-grpc/src/lib.rs +++ b/packages/zingod-grpc/src/lib.rs @@ -20,9 +20,7 @@ mod memo; mod metrics; mod origin; mod overload; -mod pressure; mod proxy_protocol; -mod read_lanes; mod routes; mod service; mod stall; @@ -37,9 +35,8 @@ pub use crate::metrics::{sent_bytes_total, METRICS}; pub use admission::AdmissionState; pub use health::Health; pub use lifetime::{Hold, Retire}; -pub use limits::{GrpcLimits, OverloadLimits}; +pub use limits::GrpcLimits; pub use origin::{OriginGate, OriginVerdict}; -pub use overload::OverloadCause; pub use proxy_protocol::TrustedProxies; pub use service::{RouteDeps, Routing}; diff --git a/packages/zingod-grpc/src/limits.rs b/packages/zingod-grpc/src/limits.rs index e3a33362c..3bc438e37 100644 --- a/packages/zingod-grpc/src/limits.rs +++ b/packages/zingod-grpc/src/limits.rs @@ -1,7 +1,6 @@ //! [`GrpcLimits`]: serving caps (zingod's `[grpc]`) //! -//! - Every cap refuses, never queues, except the read lanes (`read_lanes.rs`: a page-faulting -//! read waits for its lane's permit, at most `max_work_streams` waiting) +//! - every cap refuses, never queues use std::num::{NonZeroU32, NonZeroUsize}; use std::time::Duration; @@ -15,7 +14,6 @@ use crate::proxy_protocol::TrustedProxies; /// /// - each field's contract: usage.md "Serve stack" /// - `max_streams_per_connection` = every h2 stream, `max_work_streams` = admitted work only -/// - `unread_timeout` (client stopped reading) != `silent_timeout` (server has nothing to send) #[derive(Clone, Debug, PartialEq, Eq, Deserialize, Serialize)] #[serde(deny_unknown_fields, default)] pub struct GrpcLimits { @@ -23,39 +21,17 @@ pub struct GrpcLimits { pub max_connections_per_ip: NonZeroUsize, pub max_streams_per_connection: NonZeroU32, pub max_work_streams: NonZeroUsize, - pub max_subscriptions: NonZeroUsize, - pub max_point_reads: NonZeroUsize, - pub max_range_reads: NonZeroUsize, - pub max_scan_reads: NonZeroUsize, #[serde(rename = "unread_timeout_secs", with = "serde_secs::nonzero")] pub unread_timeout: Duration, - #[serde(rename = "silent_timeout_secs", with = "serde_secs::nonzero")] - pub silent_timeout: Duration, #[serde(rename = "max_connection_age_secs", with = "serde_secs::zero_is_never")] pub max_connection_age: Option, #[serde(rename = "max_connection_age_grace_secs", with = "serde_secs::any")] pub max_connection_age_grace: Duration, #[serde(rename = "max_connection_idle_secs", with = "serde_secs::zero_is_never")] pub max_connection_idle: Option, - pub adaptive_work_streams: bool, pub trusted_proxies: TrustedProxies, #[serde(rename = "shutdown_grace_secs", with = "serde_secs::any")] pub drain_timeout: Duration, - pub overload: OverloadLimits, -} - -/// When `/readyz` reports `Overloaded` (zingod's `[grpc.overload]`, usage.md "Overload"; 0 = off) -#[derive(Clone, Debug, PartialEq, Eq, Deserialize, Serialize)] -#[serde(deny_unknown_fields, default)] -pub struct OverloadLimits { - pub cpu_pressure_percent: u8, - pub io_pressure_percent: u8, -} - -impl Default for OverloadLimits { - fn default() -> Self { - Self { cpu_pressure_percent: 25, io_pressure_percent: 25 } - } } impl Default for GrpcLimits { @@ -70,26 +46,15 @@ impl Default for GrpcLimits { // lightwalletd's `grpc-max-concurrent-streams` (clients tuned against it) max_streams_per_connection: const { NonZeroU32::new(100).expect("100 is non-zero") }, max_work_streams: const { non_zero(2048) }, - // one mempool stream per connected wallet - max_subscriptions: const { non_zero(4096) }, - // µs, warm: CPU-bound, so ~cores in parallel is the useful ceiling - max_point_reads: const { non_zero(64) }, - // 1 MiB windows: device queue depth - max_range_reads: const { non_zero(32) }, - // whole address histories, capped per request by rows (few in flight, never many) - max_scan_reads: const { non_zero(4) }, // past pepper-sync's longest legitimate backpressure (scanning a fetched batch) unread_timeout: Duration::from_secs(300), - silent_timeout: Duration::from_secs(300), // long-held wallet connections re-resolve DNS (gRPC A9) max_connection_age: Some(Duration::from_secs(30 * 60)), // a whole `GetBlockRange` sync max_connection_age_grace: Duration::from_secs(600), max_connection_idle: Some(Duration::from_secs(30 * 60)), - adaptive_work_streams: true, trusted_proxies: TrustedProxies::default(), drain_timeout: Duration::from_secs(10), - overload: OverloadLimits::default(), } } } diff --git a/packages/zingod-grpc/src/metrics.rs b/packages/zingod-grpc/src/metrics.rs index f71d11663..328f3549f 100644 --- a/packages/zingod-grpc/src/metrics.rs +++ b/packages/zingod-grpc/src/metrics.rs @@ -7,13 +7,9 @@ use metrics::Unit; use tonic::Code; use zingod_primitives::metric::Metric; -use crate::admission::Class; use crate::lifetime::RetireReason; use crate::origin::OriginVerdict; -use crate::overload::OverloadCause; -use crate::read_lanes::Lane; use crate::service::{Method, Reply}; -use crate::summary; const REQUESTS_TOTAL: &str = "zingod.grpc.requests_total"; const FIRST_MESSAGE_SECONDS: &str = "zingod.grpc.first_message_seconds"; @@ -21,16 +17,13 @@ const DURATION_SECONDS: &str = "zingod.grpc.duration_seconds"; const SENT_BYTES_TOTAL: &str = "zingod.grpc.sent_bytes_total"; const ACTIVE_STREAMS: &str = "zingod.grpc.active_streams"; const ADMISSION_REJECTED_TOTAL: &str = "zingod.grpc.admission_rejected_total"; -const ACTIVE_SUBSCRIPTIONS: &str = "zingod.grpc.active_subscriptions"; -const SUBSCRIPTIONS_REJECTED_TOTAL: &str = "zingod.grpc.subscriptions_rejected_total"; const ACCEPT_ERRORS_TOTAL: &str = "zingod.grpc.accept_errors_total"; -const DISK_READ_WAIT_SECONDS: &str = "zingod.grpc.disk_read_wait_seconds"; const CONNECTIONS_ACTIVE: &str = "zingod.grpc.connections_active"; const CONNECTIONS_REJECTED_TOTAL: &str = "zingod.grpc.connections_rejected_total"; const STALLED_CONNECTIONS_TOTAL: &str = "zingod.grpc.stalled_connections_total"; const RETIRED_CONNECTIONS_TOTAL: &str = "zingod.grpc.retired_connections_total"; const ADAPTIVE_STREAM_LIMIT: &str = "zingod.grpc.adaptive_stream_limit"; -const OVERLOAD_LATCHED: &str = "zingod.grpc.overload_latched"; +const OVERLOADED: &str = "zingod.grpc.overloaded"; const ORIGIN_REQUESTS_TOTAL: &str = "zingod.grpc.origin_requests_total"; /// Σ `SENT_BYTES_TOTAL` over methods, readable without the metrics recorder (zingod's egress) @@ -91,20 +84,10 @@ pub const METRICS: &[Metric] = &[ ADMISSION_REJECTED_TOTAL, "Work streams refused UNAVAILABLE at the adaptive work-stream limit", ), - Metric::gauge(ACTIVE_SUBSCRIPTIONS, "GetMempoolStream subscriptions holding a slot"), - Metric::counter( - SUBSCRIPTIONS_REJECTED_TOTAL, - "Subscriptions refused UNAVAILABLE at max_subscriptions", - ), Metric::counter( ACCEPT_ERRORS_TOTAL, "accept() failures the listener backed off from (EMFILE, ENFILE, ENOBUFS, ...)", ), - Metric::histogram( - DISK_READ_WAIT_SECONDS, - "Wait for an index-read permit, by lane (point, range, scan)", - ) - .unit(Unit::Seconds), Metric::gauge(CONNECTIONS_ACTIVE, "Open client connections"), Metric::counter( STALLED_CONNECTIONS_TOTAL, @@ -117,12 +100,11 @@ pub const METRICS: &[Metric] = &[ ), Metric::gauge( ADAPTIVE_STREAM_LIMIT, - "Work streams admitted at once now (latency-adaptive; max_work_streams when off)", + "Work streams admitted at once now (latency-adaptive, max_work_streams the ceiling)", ), Metric::gauge( - OVERLOAD_LATCHED, - "Overload causes latched, by cause (1 = latched, also while detection is off; /readyz names \ - the first only while on)", + OVERLOADED, + "1 while admission has refused work persistently (= /readyz down, reason overloaded)", ), Metric::counter( ORIGIN_REQUESTS_TOTAL, @@ -150,7 +132,6 @@ pub(crate) fn connection_closed() { /// A connection closed for holding data unread past `unread_timeout` pub(crate) fn connection_stalled() { metrics::counter!(STALLED_CONNECTIONS_TOTAL).increment(1); - summary::connection_stalled(); } pub(crate) fn accept_failed() { @@ -159,29 +140,18 @@ pub(crate) fn accept_failed() { pub(crate) fn connection_rejected() { metrics::counter!(CONNECTIONS_REJECTED_TOTAL).increment(1); - summary::connection_refused(); } -fn active(class: Class) -> &'static str { - match class { - Class::Work => ACTIVE_STREAMS, - Class::Subscription => ACTIVE_SUBSCRIPTIONS, - } +pub(crate) fn stream_admitted() { + metrics::gauge!(ACTIVE_STREAMS).increment(1.0); } -pub(crate) fn stream_admitted(class: Class) { - metrics::gauge!(active(class)).increment(1.0); +pub(crate) fn stream_released() { + metrics::gauge!(ACTIVE_STREAMS).decrement(1.0); } -pub(crate) fn stream_released(class: Class) { - metrics::gauge!(active(class)).decrement(1.0); -} - -pub(crate) fn stream_rejected(class: Class) { - match class { - Class::Work => metrics::counter!(ADMISSION_REJECTED_TOTAL).increment(1), - Class::Subscription => metrics::counter!(SUBSCRIPTIONS_REJECTED_TOTAL).increment(1), - } +pub(crate) fn stream_rejected() { + metrics::counter!(ADMISSION_REJECTED_TOTAL).increment(1); } fn verdict_label(verdict: OriginVerdict) -> &'static str { @@ -198,13 +168,8 @@ pub(crate) fn origin_request(verdict: OriginVerdict) { } /// Set at each overload window -pub(crate) fn overload_latched(cause: OverloadCause, held: bool) { - metrics::gauge!(OVERLOAD_LATCHED, "cause" => cause.label()).set(f64::from(u8::from(held))); -} - -pub(crate) fn disk_read_waited(lane: Lane, waited: Duration) { - metrics::histogram!(DISK_READ_WAIT_SECONDS, "lane" => lane.label()) - .record(waited.as_secs_f64()); +pub(crate) fn overloaded(latched: bool) { + metrics::gauge!(OVERLOADED).set(f64::from(u8::from(latched))); } /// A stream's first message (a one-reply RPC's = its whole answer: [`stream_finished`] times it) @@ -216,16 +181,7 @@ pub(crate) fn first_message(method: Option, elapsed: Duration) { } /// One stream's close-out (recorded where the body drops) -/// -/// - `latency` = time to its first message, for an answered work request -pub(crate) fn stream_finished( - method: Option, - code: Code, - elapsed: Duration, - latency: Option, - sent: u64, -) { - summary::finished(code, latency); +pub(crate) fn stream_finished(method: Option, code: Code, elapsed: Duration, sent: u64) { let name = method_label(method); metrics::counter!(REQUESTS_TOTAL, "method" => name, "code" => code_label(code)).increment(1); // answers only (refusal = µs of no work) diff --git a/packages/zingod-grpc/src/overload.rs b/packages/zingod-grpc/src/overload.rs index bc82a0d01..a8c2332e3 100644 --- a/packages/zingod-grpc/src/overload.rs +++ b/packages/zingod-grpc/src/overload.rs @@ -1,9 +1,7 @@ -//! Overload latch per summary window (`summary::INTERVAL`) → readiness +//! Overload latch: admission refusing work, persistently → readiness `overloaded` //! -//! - one latch per [`OverloadCause`]: over its threshold for its `enter` → latched, under half of -//! it (refusals: under 2 %) for its `exit` → cleared; reported = the first latched -//! - signals: admission refusals (counted here), CPU / IO pressure (`pressure.rs`) -//! - operator settings live (`set`): while detection is off the latches still step, none reported +//! - one signal = the adaptive limiter's refusals (CPU / IO saturation → latency → lower limit) +//! - stepped once per serving summary (`summary::INTERVAL`); no operator thresholds use std::sync::atomic::{AtomicU64, Ordering}; use std::sync::Mutex; @@ -11,126 +9,42 @@ use std::time::Duration; use tracing::{info, warn}; -use crate::admission::Class; use crate::metrics; -use crate::OverloadLimits; -/// Refusals: >= ENTER_PERCENT of work (at >= 1 refusal/s) / < EXIT_PERCENT +/// Refusals: >= ENTER_PERCENT of work (at >= 1 refusal/s) latches, < EXIT_PERCENT clears const ENTER_PERCENT: u64 = 10; const EXIT_PERCENT: u64 = 2; -#[derive(Clone, Copy, Debug, PartialEq, Eq)] -pub enum OverloadCause { - CpuPressure, - IoPressure, - RefusingWork, -} - -impl OverloadCause { - const ALL: [Self; 3] = [Self::CpuPressure, Self::IoPressure, Self::RefusingWork]; - - /// Readiness `detail.message` - pub const fn detail(self) -> &'static str { - match self { - Self::CpuPressure => "Overloaded: CPU pressure", - Self::IoPressure => "Overloaded: IO pressure", - Self::RefusingWork => "Overloaded: refusing work", - } - } - - /// `zingod_grpc_overload_latched{cause}` - pub(crate) const fn label(self) -> &'static str { - match self { - Self::CpuPressure => "cpu_pressure", - Self::IoPressure => "io_pressure", - Self::RefusingWork => "refusing_work", - } - } +/// Time either condition must hold before the latch flips (admission already sheds in seconds) +const FLIP_AFTER: Duration = Duration::from_secs(5 * 60); - /// (enter, exit) - fn rule(self) -> (Duration, Duration) { - let minutes = |m: u64| Duration::from_secs(m * 60); - match self { - Self::CpuPressure | Self::IoPressure => (minutes(2), minutes(3)), - // admission already sheds within seconds: only a persistent shortfall counts - Self::RefusingWork => (minutes(5), minutes(5)), - } - } -} - -/// One window's signals (`None` = not measured here) -#[derive(Clone, Copy, Debug, Default)] -pub(crate) struct Signals { - pub(crate) cpu: Option, - pub(crate) io: Option, -} - -/// One cause's state (`flipping_for` = how long the opposite condition has held) -#[derive(Clone, Copy, Debug, Default)] +/// `flipping_for` = how long the opposite condition has held +#[derive(Debug, Default)] struct Latch { held: bool, flipping_for: Duration, } -/// `value` against `limit` (0 = off): (over, under half) -fn against(value: Option, limit: f64) -> (bool, bool) { - match value { - Some(value) if limit > 0.0 => (value >= limit, value < limit / 2.0), - _ => (false, true), - } -} - -#[derive(Debug)] +#[derive(Debug, Default)] pub(crate) struct Overload { - admitted_work: AtomicU64, - refused_work: AtomicU64, - refused_subscriptions: AtomicU64, - closed: Mutex, -} - -/// As of the last closed window + the operator's settings -#[derive(Debug)] -struct Closed { - limits: OverloadLimits, - detection: bool, - latches: [Latch; OverloadCause::ALL.len()], + admitted: AtomicU64, + refused: AtomicU64, + latch: Mutex, } impl Overload { - pub(crate) fn new(limits: OverloadLimits) -> Self { - let closed = Closed { limits, detection: true, latches: Default::default() }; - Self { - admitted_work: AtomicU64::new(0), - refused_work: AtomicU64::new(0), - refused_subscriptions: AtomicU64::new(0), - closed: Mutex::new(closed), - } - } - - pub(crate) fn set(&self, limits: OverloadLimits, detection: bool) { - let mut closed = self.closed.lock().expect("overload latch poisoned"); - (closed.limits, closed.detection) = (limits, detection); + pub(crate) fn admitted(&self) { + self.admitted.fetch_add(1, Ordering::Relaxed); } - pub(crate) fn admitted(&self, class: Class) { - if class == Class::Work { - self.admitted_work.fetch_add(1, Ordering::Relaxed); - } - } - - pub(crate) fn refused(&self, class: Class) { - metrics::stream_rejected(class); - let refused = match class { - Class::Work => &self.refused_work, - Class::Subscription => &self.refused_subscriptions, - }; - refused.fetch_add(1, Ordering::Relaxed); + pub(crate) fn refused(&self) { + self.refused.fetch_add(1, Ordering::Relaxed); } - /// Closes the window (`elapsed` long) on its `signals`: its refusals, every class - pub(crate) fn tick(&self, elapsed: Duration, signals: Signals) -> u64 { + /// Closes the window (`elapsed` long) + pub(crate) fn tick(&self, elapsed: Duration) { let take = |counter: &AtomicU64| counter.swap(0, Ordering::Relaxed); - let (admitted, refused) = (take(&self.admitted_work), take(&self.refused_work)); + let (admitted, refused) = (take(&self.admitted), take(&self.refused)); let attempts = admitted.saturating_add(refused); let percent_of_attempts = |percent: u64| attempts.saturating_mul(percent); let refused_percent = refused.saturating_mul(100); @@ -138,45 +52,21 @@ impl Overload { && Duration::from_secs(refused) >= elapsed; let quiet = refused == 0 || refused_percent < percent_of_attempts(EXIT_PERCENT); - let mut closed = self.closed.lock().expect("overload latch poisoned"); - let limits = &closed.limits; - let percent = |p: u8| f64::from(p) / 100.0; - let conditions = OverloadCause::ALL.map(|cause| match cause { - OverloadCause::CpuPressure => { - against(signals.cpu, percent(limits.cpu_pressure_percent)) + let mut latch = self.latch.lock().expect("overload latch poisoned"); + let flipping = if latch.held { quiet } else { refusing }; + latch.flipping_for = if flipping { latch.flipping_for + elapsed } else { Duration::ZERO }; + if latch.flipping_for >= FLIP_AFTER { + (latch.held, latch.flipping_for) = (!latch.held, Duration::ZERO); + match latch.held { + true => warn!(refused, attempts, "Overload latched"), + false => info!(refused, attempts, "Overload cleared"), } - OverloadCause::IoPressure => against(signals.io, percent(limits.io_pressure_percent)), - OverloadCause::RefusingWork => (refusing, quiet), - }); - let detection = closed.detection; - let causes = OverloadCause::ALL.into_iter().zip(conditions); - for ((cause, (over, under)), latch) in causes.zip(&mut closed.latches) { - let (enter, exit) = cause.rule(); - let (flipping, needed) = if latch.held { (under, exit) } else { (over, enter) }; - latch.flipping_for = - if flipping { latch.flipping_for + elapsed } else { Duration::ZERO }; - if latch.flipping_for >= needed { - (latch.held, latch.flipping_for) = (!latch.held, Duration::ZERO); - let cause = cause.detail(); - match latch.held { - true => { - warn!(cause, detection, ?signals, refused, attempts, "Overload latched") - } - false => info!(cause, ?signals, refused, attempts, "Overload cleared"), - } - } - metrics::overload_latched(cause, latch.held); } - - refused.saturating_add(take(&self.refused_subscriptions)) + metrics::overloaded(latch.held); } - /// The first latched cause (`None` while detection is off) - pub(crate) fn cause(&self) -> Option { - let closed = self.closed.lock().expect("overload latch poisoned"); - let mut latched = OverloadCause::ALL.into_iter().zip(&closed.latches); - let first = latched.find_map(|(cause, latch)| latch.held.then_some(cause)); - first.filter(|_| closed.detection) + pub(crate) fn latched(&self) -> bool { + self.latch.lock().expect("overload latch poisoned").held } } @@ -186,119 +76,40 @@ mod tests { const MINUTE: Duration = Duration::from_secs(60); - /// One minute of `per_second` work admissions, `refused_percent` of them refused → the cause - fn refusal_minute( - overload: &Overload, - per_second: u64, - refused_percent: u64, - ) -> Option { - let total = per_second * 60; - for _ in 0..total * (100 - refused_percent) / 100 { - overload.admitted(Class::Work); - } - for _ in 0..total * refused_percent / 100 { - overload.refused(Class::Work); - } - overload.tick(MINUTE, Signals::default()); - overload.cause() - } - /// Enter: >= 10 % refused at >= 1 refusal/s for 5 min; exit: < 2 % for 5 min #[test] - fn refusing_work_latches_after_five_sustained_minutes_and_clears_after_five_quiet() { - let (none, refusing) = (None, Some(OverloadCause::RefusingWork)); - let latched_5 = [vec![none; 4], vec![refusing]].concat(); - // case, work admissions/s, % refused each minute, cause after each minute + fn refusals_latch_after_five_sustained_minutes_and_clear_after_five_quiet() { + let latched_5 = [vec![false; 4], vec![true]].concat(); + // case, work admissions/s, % refused each minute, latched after each minute #[rustfmt::skip] let cases = [ - ("9 % for 10 min", 100, vec![9; 10], vec![none; 10]), + ("9 % for 10 min", 100, vec![9; 10], vec![false; 10]), ("10 % for 5 min", 100, vec![10; 5], latched_5.clone()), ("30 %, one minute at 25 %", 100, vec![30, 30, 25, 30, 30], latched_5.clone()), - ("50 % of 1/s = 0.5 refusals/s", 1, vec![50; 10], vec![none; 10]), - ("latched, then 2 % for 5 min", 100, [vec![10; 5], vec![2; 5]].concat(), [latched_5.clone(), vec![refusing; 5]].concat()), - ("latched, then 1 % for 5 min", 100, [vec![10; 5], vec![1; 5]].concat(), [latched_5.clone(), vec![refusing; 4], vec![none]].concat()), + ("30 %, one minute at 5 %", 100, vec![30, 30, 5, 30, 30, 30], vec![false; 6]), + ("50 % of 1/s = 0.5 refusals/s", 1, vec![50; 10], vec![false; 10]), + ("latched, then 2 % for 5 min", 100, [vec![10; 5], vec![2; 5]].concat(), [latched_5.clone(), vec![true; 5]].concat()), + ("latched, then 1 % for 5 min", 100, [vec![10; 5], vec![1; 5]].concat(), [latched_5.clone(), vec![true; 4], vec![false]].concat()), ]; - for (case, per_second, refused_percent, causes) in cases { - let overload = Overload::new(OverloadLimits::default()); - - let latched: Vec<_> = - refused_percent.iter().map(|&p| refusal_minute(&overload, per_second, p)).collect(); - - assert_eq!(latched, causes, "{case}: cause after each minute"); - } - } - - #[test] - fn a_window_counts_every_refusal_of_every_class() { - let overload = Overload::new(OverloadLimits::default()); - for _ in 0..120 { - overload.refused(Class::Work); - } - overload.refused(Class::Subscription); - - let refused = overload.tick(MINUTE, Signals::default()); - - assert_eq!(refused, 121, "120 work + 1 subscription refused"); - assert_eq!(overload.tick(MINUTE, Signals::default()), 0, "next window starts at zero"); - } - - /// One minute's pressure shares (`None` = unmeasured) - fn signals(cpu: Option, io: Option) -> Signals { - Signals { cpu, io } - } - - /// CPU / IO 25 %: enter over the limit for 2 min, exit under half of it for 3 - #[test] - fn each_pressure_cause_latches_over_its_limit_and_clears_under_half_of_it() { - let (none, cpu, io) = - (None, Some(OverloadCause::CpuPressure), Some(OverloadCause::IoPressure)); - let limits = OverloadLimits::default(); - #[rustfmt::skip] - let cases = [ - ("CPU 30 %; 15 % (not under 12.5) holds; unmeasured 3 min clears", - [signals(Some(0.3), None), signals(Some(0.3), None), signals(Some(0.15), None), - signals(None, None), signals(None, None), signals(None, None)], - [none, cpu, cpu, cpu, cpu, none]), - ("IO 25 % exactly latches; 20 % holds; 10 % for 3 min clears", - [signals(None, Some(0.25)), signals(None, Some(0.25)), signals(None, Some(0.2)), - signals(None, Some(0.1)), signals(None, Some(0.1)), signals(None, Some(0.1))], - [none, io, io, io, io, none]), - ("CPU + IO both latch: CPU named first, then IO", - [signals(Some(0.3), Some(0.3)), signals(Some(0.3), Some(0.3)), signals(Some(0.15), Some(0.3)), - signals(Some(0.1), Some(0.3)), signals(Some(0.1), Some(0.3)), signals(Some(0.1), Some(0.3))], - [none, cpu, cpu, cpu, cpu, io]), - ]; - for (case, minutes, causes) in cases { - let overload = Overload::new(limits.clone()); - - let latched = minutes.map(|minute| { - overload.tick(MINUTE, minute); - overload.cause() - }); - - assert_eq!(latched, causes, "{case}: cause after each minute"); - } - } + for (case, per_second, refused_percent, want) in cases { + let overload = Overload::default(); + + let latched: Vec = refused_percent + .iter() + .map(|&percent| { + let total = per_second * 60; + for _ in 0..total * (100 - percent) / 100 { + overload.admitted(); + } + for _ in 0..total * percent / 100 { + overload.refused(); + } + overload.tick(MINUTE); + overload.latched() + }) + .collect(); - #[test] - fn operator_settings_apply_to_a_latched_cause_live() { - let cpu = Some(OverloadCause::CpuPressure); - let limits = OverloadLimits { cpu_pressure_percent: 25, ..OverloadLimits::default() }; - let overload = Overload::new(limits.clone()); - for _ in 0..2 { - overload.tick(MINUTE, signals(Some(0.3), None)); + assert_eq!(latched, want, "{case}: latched after each minute"); } - assert_eq!(overload.cause(), cpu, "CPU 30 % for 2 min: latched"); - - overload.set(limits.clone(), false); - let detection_off = overload.cause(); - overload.set(OverloadLimits { cpu_pressure_percent: 0, ..limits }, true); - let limit_off = [0; 3].map(|_| { - overload.tick(MINUTE, signals(Some(1.0), None)); - overload.cause() - }); - - assert_eq!(detection_off, None, "detection off: latched, never reported"); - assert_eq!(limit_off, [cpu, cpu, None], "CPU limit 0 = off: clears by its 3 min exit"); } } diff --git a/packages/zingod-grpc/src/pressure.rs b/packages/zingod-grpc/src/pressure.rs deleted file mode 100644 index ca5e33bd6..000000000 --- a/packages/zingod-grpc/src/pressure.rs +++ /dev/null @@ -1,95 +0,0 @@ -//! CPU / IO pressure (Linux PSI): share of a window this process's cgroup waited, else unmeasured -//! -//! - share = Δ `some total=` µs / window (exact, not the kernel's 10 s / 60 s averages) -//! - never `/proc/pressure` (other processes' stalls would latch zingod's overload) - -use std::path::PathBuf; -use std::time::Duration; - -/// `some … total=<µs>` of a PSI file -fn some_total_us(psi: &str) -> Option { - let some = psi.lines().find(|line| line.starts_with("some "))?; - some.split_whitespace().find_map(|field| field.strip_prefix("total="))?.parse().ok() -} - -#[derive(Debug, Clone, Copy)] -pub(crate) enum Resource { - Cpu, - Io, -} - -impl Resource { - /// PSI file name, under `/proc/pressure` and as `.pressure` in a cgroup - fn name(self) -> &'static str { - match self { - Self::Cpu => "cpu", - Self::Io => "io", - } - } -} - -/// One resource's PSI file + its total at the last window -#[derive(Debug)] -pub(crate) struct Pressure { - file: Option, - last_us: Option, -} - -impl Pressure { - pub(crate) fn new(resource: Resource) -> Self { - let resource = resource.name(); - // cgroup v2: `0::/system.slice/zingod.service` - let cgroup = std::fs::read_to_string("/proc/self/cgroup").ok().and_then(|text| { - let path = text.lines().find_map(|line| line.strip_prefix("0::"))?.to_owned(); - Some(PathBuf::from("/sys/fs/cgroup").join(path.trim_start_matches('/'))) - }); - let file = - cgroup.map(|dir| dir.join(format!("{resource}.pressure"))).filter(|file| file.exists()); - let mut pressure = Self { file, last_us: None }; - pressure.last_us = pressure.read(); - pressure - } - - fn read(&self) -> Option { - some_total_us(&std::fs::read_to_string(self.file.as_ref()?).ok()?) - } - - /// Stalled share since the last call (`elapsed` long) - pub(crate) fn share(&mut self, elapsed: Duration) -> Option { - let now = self.read()?; - let stalled = now.checked_sub(self.last_us.replace(now)?)?; - let window = elapsed.max(Duration::from_micros(1)); - Some(Duration::from_micros(stalled).div_duration_f64(window).min(1.0)) - } -} - -#[cfg(test)] -mod tests { - use super::*; - - fn psi(some_total_us: u64) -> String { - format!("some avg10=1.00 total={some_total_us}\nfull avg10=0.00 total=9\n") - } - - /// Window = Δ `some` total (never `full`'s) / its length, capped at 1; no PSI file or no - /// `some` line = unmeasured - #[test] - fn a_window_is_the_stalled_share_of_its_length() { - let dir = tempfile::tempdir().expect("tempdir"); - let file = dir.path().join("cpu.pressure"); - let mut cpu = Pressure { file: Some(file.clone()), last_us: Some(1_000_000) }; - let minute = Duration::from_secs(60); - - std::fs::write(&file, psi(16_000_000)).expect("write"); - let quarter = cpu.share(minute); - std::fs::write(&file, psi(136_000_000)).expect("write"); - let over = cpu.share(minute); - std::fs::write(&file, "full avg10=0.00 total=1\n").expect("write"); - let no_some_line = cpu.share(minute); - - assert_eq!(quarter, Some(0.25), "15 s stalled of 60 s"); - assert_eq!(over, Some(1.0), "120 s stalled of 60 s: capped"); - assert_eq!(no_some_line, None, "no `some` line"); - assert_eq!(Pressure { file: None, last_us: None }.share(minute), None, "no PSI file"); - } -} diff --git a/packages/zingod-grpc/src/read_lanes.rs b/packages/zingod-grpc/src/read_lanes.rs deleted file mode 100644 index 16907816f..000000000 --- a/packages/zingod-grpc/src/read_lanes.rs +++ /dev/null @@ -1,75 +0,0 @@ -//! Index read lanes: semaphore gate around `spawn_blocking` disk reads -//! -//! - the one bounded wait in the serve path (a page-faulting read waits for its lane's permit) - -use std::num::NonZeroUsize; -use std::sync::Arc; -use std::time::Instant; - -use tokio::sync::Semaphore; - -use crate::limits::GrpcLimits; -use crate::metrics; - -/// Index read kind, own permit pool each (heavy never queues light): `Point` = one record / tree -/// state (µs warm), `Range` = one `GetBlockRange` window (<= 1 MiB), `Scan` = an address history -/// (bounded by the row budget) -#[derive(Clone, Copy, Debug, PartialEq, Eq)] -pub(crate) enum Lane { - Point, - Range, - Scan, -} - -impl Lane { - pub(crate) fn label(self) -> &'static str { - match self { - Self::Point => "point", - Self::Range => "range", - Self::Scan => "scan", - } - } -} - -/// Process-wide (clone = share) -#[derive(Clone, Debug)] -pub(crate) struct ReadLanes { - point: Arc, - range: Arc, - scan: Arc, -} - -impl ReadLanes { - pub(crate) fn new(limits: &GrpcLimits) -> Self { - let lane = |max: NonZeroUsize| Arc::new(Semaphore::new(max.get())); - Self { - point: lane(limits.max_point_reads), - range: lane(limits.max_range_reads), - scan: lane(limits.max_scan_reads), - } - } - - /// `read` on the blocking pool under a `lane` permit (faults mmapped pages: never on a runtime - /// worker, shared by metrics + other RPCs) - /// - /// - a panic in `read` resumes here (a broken invariant: zingod aborts) - /// - runtime shutting down → `unavailable` - pub(crate) async fn read( - &self, - lane: Lane, - read: impl FnOnce() -> T + Send + 'static, - ) -> Result { - let pool = match lane { - Lane::Point => &self.point, - Lane::Range => &self.range, - Lane::Scan => &self.scan, - }; - let waiting = Instant::now(); - let _permit = pool.acquire().await.expect("read lanes are never closed"); - metrics::disk_read_waited(lane, waiting.elapsed()); - tokio::task::spawn_blocking(read).await.map_err(|join| match join.try_into_panic() { - Ok(panic) => std::panic::resume_unwind(panic), - Err(_) => tonic::Status::unavailable("shutting down"), - }) - } -} diff --git a/packages/zingod-grpc/src/routes.rs b/packages/zingod-grpc/src/routes.rs index c3d12a1b8..b94715439 100644 --- a/packages/zingod-grpc/src/routes.rs +++ b/packages/zingod-grpc/src/routes.rs @@ -8,3 +8,18 @@ pub(super) mod mempool; pub(super) mod transactions; pub(super) mod transparent_address; pub(super) mod tree_state; + +use tonic::Status; + +/// `read` on the blocking pool (faults mmapped pages: never on a runtime worker) +/// +/// - panic in `read` resumes here (broken invariant: zingod aborts) +/// - runtime shutting down → `UNAVAILABLE` +async fn blocking( + read: impl FnOnce() -> T + Send + 'static, +) -> Result { + tokio::task::spawn_blocking(read).await.map_err(|join| match join.try_into_panic() { + Ok(panic) => std::panic::resume_unwind(panic), + Err(_) => Status::unavailable("shutting down"), + }) +} diff --git a/packages/zingod-grpc/src/routes/blocks.rs b/packages/zingod-grpc/src/routes/blocks.rs index 6f8ea89cb..bbf9afcff 100644 --- a/packages/zingod-grpc/src/routes/blocks.rs +++ b/packages/zingod-grpc/src/routes/blocks.rs @@ -12,7 +12,7 @@ use zingod_persistence::{OverlayView, PinnedView}; use zingod_sync::IndexKind; use super::args; -use crate::read_lanes::{Lane, ReadLanes}; +use super::blocking; use crate::wire::{self, unary_response}; use zingod_proto::proto::service; @@ -37,14 +37,13 @@ pub(crate) fn get_latest_block(served: &IndexesAtBlock) -> Response pub(crate) async fn get_block( compact_blocks: CompactBlocks<'_, V>, block_id: service::BlockId, - reads: &ReadLanes, ) -> Result, Status> { let block = if block_id.hash.is_empty() { let height = args::height(block_id.height, "height")?; - compact_blocks.read_by_height(height, reads).await? + compact_blocks.read_by_height(height).await? } else { let (height, hash) = compact_blocks.locate(&block_id.hash, "GetBlock")?; - compact_blocks.read_by_hash(height, hash, reads).await? + compact_blocks.read_by_hash(height, hash).await? }; Ok(unary_response(block)) } @@ -53,11 +52,10 @@ pub(crate) async fn get_block( pub(crate) fn get_block_range( compact_blocks: CompactBlocks<'_, V>, range: service::BlockRange, - reads: ReadLanes, ) -> Result, Status> { let pools = args::pools(&range.pool_types)?; let blocks = compact_blocks.open_range(&range, pools)?; - Ok(blocks.into_response(reads, std::convert::identity)) + Ok(blocks.into_response(std::convert::identity)) } impl CompactBlocks<'_, V> { @@ -71,8 +69,8 @@ impl CompactBlocks<'_, V> { args::locate(self.served, self.served_through(), hash, method) } - /// NFS layer (pepper-sync's reorg check at the tip) = RAM, answered inline; else the point lane - async fn read_by_height(self, height: BlockHeight, reads: &ReadLanes) -> Result { + /// NFS layer (pepper-sync's reorg check at the tip) = RAM, answered inline; else the blocking pool + async fn read_by_height(self, height: BlockHeight) -> Result { if height > self.served_through() { return Err(serve_error_status(ServeError::NotFound { height })); } @@ -84,21 +82,16 @@ impl CompactBlocks<'_, V> { let held = reader.block(height); held.unwrap_or_else(|| panic!("compact-block view holds {height} <= answers_through")) }; - reads.read(Lane::Point, read).await + blocking(read).await } /// Only where this index holds that same block (a located height can lag a reorg) - async fn read_by_hash( - self, - height: BlockHeight, - hash: [u8; 32], - reads: &ReadLanes, - ) -> Result { + async fn read_by_hash(self, height: BlockHeight, hash: [u8; 32]) -> Result { let reader = self.reader; let read = move || { reader.block_at(height, &zcash_protocol::BlockHash(hash)).map_err(serve_error_status) }; - reads.read(Lane::Point, read).await? + blocking(read).await? } /// Walk order up to the first height past [`Self::served_through`] @@ -147,17 +140,12 @@ impl BlockStream { /// Server-streaming response, cursor chunk by chunk /// /// - `reproject` = per-chunk rewrite (`identity` for `GetBlockRange`: records stay slices) - pub(super) fn into_response( - self, - reads: ReadLanes, - reproject: fn(Bytes) -> Bytes, - ) -> Response { + pub(super) fn into_response(self, reproject: fn(Bytes) -> Bytes) -> Response { let BlockStream { cursor, past_tip } = self; - let chunks = - futures::stream::try_unfold((cursor, reads), move |(cursor, reads)| async move { - let (cursor, chunk) = next_chunk(cursor, &reads).await?; - Ok(chunk.map(|chunk| (reproject(chunk), (cursor, reads)))) - }); + let chunks = futures::stream::try_unfold(cursor, move |cursor| async move { + let (cursor, chunk) = next_chunk(cursor).await?; + Ok(chunk.map(|chunk| (reproject(chunk), (cursor)))) + }); wire::streaming(chunks.chain(futures::stream::iter(past_tip.map(Err)))) } @@ -165,23 +153,21 @@ impl BlockStream { /// Cursor's next chunk (`None` = spent) /// -/// - disk step = blocking pool, range lane per step not per request (cold mmap refill ~11.7 ms +/// - disk step = blocking pool, per step not per request (cold mmap refill ~11.7 ms /// would stall the worker; layer reads never queue) /// - in-memory step = inline (else a task per block on the highest-volume RPC) async fn next_chunk( mut cursor: RangeCursor, - reads: &ReadLanes, ) -> Result<(RangeCursor, Option), Status> { if !cursor.next_touches_disk() { let chunk = cursor.next_chunk(); return Ok((cursor, chunk)); } - reads - .read(Lane::Range, move || { - let chunk = cursor.next_chunk(); - (cursor, chunk) - }) - .await + blocking(move || { + let chunk = cursor.next_chunk(); + (cursor, chunk) + }) + .await } fn serve_error_status(error: ServeError) -> Status { diff --git a/packages/zingod-grpc/src/routes/deprecated_nullifiers.rs b/packages/zingod-grpc/src/routes/deprecated_nullifiers.rs index e81a4668f..a5810de6f 100644 --- a/packages/zingod-grpc/src/routes/deprecated_nullifiers.rs +++ b/packages/zingod-grpc/src/routes/deprecated_nullifiers.rs @@ -18,7 +18,6 @@ use zingod_proto::proto::service; use super::args; use super::blocks::CompactBlocks; -use crate::read_lanes::ReadLanes; use crate::wire; /// `GetBlockRangeNullifiers` @@ -28,13 +27,12 @@ use crate::wire; pub(crate) fn get_block_range_nullifiers( compact_blocks: CompactBlocks<'_, V>, mut range: service::BlockRange, - reads: ReadLanes, ) -> Result, Status> { range.pool_types.retain(|&pool| pool != i32::from(service::PoolType::Transparent)); let pools = args::pools(&range.pool_types)?; let blocks = compact_blocks.open_range(&range, pools)?; - Ok(blocks.into_response(reads, nullifiers_only_chunk)) + Ok(blocks.into_response(nullifiers_only_chunk)) } /// Framed records → framed nullifier-only records, one frame per block diff --git a/packages/zingod-grpc/src/routes/transactions.rs b/packages/zingod-grpc/src/routes/transactions.rs index e84fcb439..fb623391f 100644 --- a/packages/zingod-grpc/src/routes/transactions.rs +++ b/packages/zingod-grpc/src/routes/transactions.rs @@ -13,8 +13,8 @@ use zingod_proto::proto::service::{self, RawTransaction}; use zingod_source::{ChainDataSource, GetTransactionError, QueryError}; use zingod_traffic::TrafficBalancer; +use super::blocking; use super::mempool::unmined; -use crate::read_lanes::{Lane, ReadLanes}; use crate::wire::{frame, unary_response}; /// `SendTransaction`, answered as lightwalletd does (a rejection = a reply, only unreachable = a @@ -47,7 +47,6 @@ pub(crate) async fn send_transaction( pub(crate) async fn get_transaction( validators: &TrafficBalancer, grpc_snapshot: &GrpcSnapshot, - reads: &ReadLanes, tx_filter: service::TxFilter, ) -> Result, Status> { let txid: [u8; 32] = tx_filter @@ -57,7 +56,7 @@ pub(crate) async fn get_transaction( .map_err(|_| Status::invalid_argument("txid must be 32 bytes, in protocol order"))?; let transaction = - fetch_raw_transaction(validators, grpc_snapshot, reads, TxId::from_bytes(txid)).await?; + fetch_raw_transaction(validators, grpc_snapshot, TxId::from_bytes(txid)).await?; Ok(unary_response(frame(&transaction))) } @@ -68,7 +67,6 @@ pub(crate) async fn get_transaction( pub(super) async fn fetch_raw_transaction( validators: &TrafficBalancer, grpc_snapshot: &GrpcSnapshot, - reads: &ReadLanes, txid: TxId, ) -> Result { let found = validators.lookup(move |source| async move { source.get_transaction(txid).await }); @@ -80,7 +78,7 @@ pub(super) async fn fetch_raw_transaction( } _ => { let unavailable = || Status::unavailable(unanswered.to_string()); - fetch_without_validators(validators, grpc_snapshot, reads, txid) + fetch_without_validators(validators, grpc_snapshot, txid) .await? .ok_or_else(unavailable) } @@ -106,7 +104,6 @@ fn raw_transaction_message(found: zingod_source::TransactionResponse) -> RawTran async fn fetch_without_validators( validators: &TrafficBalancer, grpc_snapshot: &GrpcSnapshot, - reads: &ReadLanes, txid: TxId, ) -> Result, Status> { if let Some(entry) = grpc_snapshot.mempool().ok().and_then(|mempool| mempool.get(&txid)) { @@ -116,7 +113,7 @@ async fn fetch_without_validators( else { return Ok(None); }; - let Some(mined) = reads.read(Lane::Point, move || index.mined(&txid)).await? else { + let Some(mined) = blocking(move || index.mined(&txid)).await? else { return Ok(None); }; let Some(raw) = validators.peer_block_bytes(mined.block.hash).await else { return Ok(None) }; @@ -227,11 +224,10 @@ mod tests { let ours = [(txid, bytes::Bytes::from(raw.clone()))]; let held = fixed_snapshots(&chain, tip, Vec::new(), TrustedValidatorSet::at([0]), &ours); let held = held.load(); - let reads = ReadLanes::new(&crate::GrpcLimits::default()); - let served = fetch_raw_transaction(&routes.validators, &held, &reads, txid).await; + let served = fetch_raw_transaction(&routes.validators, &held, txid).await; let unheld = TxId::from_bytes([9; 32]); - let other = fetch_raw_transaction(&routes.validators, &held, &reads, unheld).await; + let other = fetch_raw_transaction(&routes.validators, &held, unheld).await; let served = served.map(|tx| (tx.data.to_vec(), tx.height)); assert_eq!( diff --git a/packages/zingod-grpc/src/routes/transparent_address.rs b/packages/zingod-grpc/src/routes/transparent_address.rs index 7c11a8281..180108b77 100644 --- a/packages/zingod-grpc/src/routes/transparent_address.rs +++ b/packages/zingod-grpc/src/routes/transparent_address.rs @@ -21,8 +21,8 @@ use http::Response; use tonic::{body::Body, Status}; use super::args; +use super::blocking; use super::info::chain_name; -use crate::read_lanes::{Lane, ReadLanes}; use crate::wire::{self, frame, streamed_response, unary_response}; /// One request's transparent-address index: the reader (as of the served tip, row-capped) + the @@ -36,9 +36,8 @@ pub(crate) struct AddressIndex { pub(crate) async fn get_address_utxos( address_index: AddressIndex, request: service::GetAddressUtxosArg, - reads: &ReadLanes, ) -> Result, Status> { - let address_utxos = address_utxos(address_index, &request, reads).await?; + let address_utxos = address_utxos(address_index, &request).await?; let reply = service::GetAddressUtxosReplyList { address_utxos }; Ok(unary_response(wire::frame(&reply))) } @@ -47,9 +46,8 @@ pub(crate) async fn get_address_utxos( pub(crate) async fn get_address_utxos_stream( address_index: AddressIndex, request: service::GetAddressUtxosArg, - reads: &ReadLanes, ) -> Result, Status> { - let address_utxos = address_utxos(address_index, &request, reads).await?; + let address_utxos = address_utxos(address_index, &request).await?; Ok(streamed_response(address_utxos.iter().map(wire::frame).collect())) } @@ -57,9 +55,8 @@ pub(crate) async fn get_address_utxos_stream( pub(crate) async fn get_taddress_balance( address_index: AddressIndex, address_list: service::AddressList, - reads: &ReadLanes, ) -> Result, Status> { - let balance = total_balance(address_index, &address_list.addresses, reads).await?; + let balance = total_balance(address_index, &address_list.addresses).await?; Ok(unary_response(wire::frame(&balance))) } @@ -67,11 +64,10 @@ pub(crate) async fn get_taddress_balance( pub(crate) async fn get_taddress_balance_stream( address_index: AddressIndex, streamed: Vec, - reads: &ReadLanes, ) -> Result, Status> { let addresses: Vec = streamed.into_iter().map(|one| one.address).collect(); - let balance = total_balance(address_index, &addresses, reads).await?; + let balance = total_balance(address_index, &addresses).await?; Ok(unary_response(wire::frame(&balance))) } @@ -82,10 +78,9 @@ pub(crate) async fn get_taddress_transactions validators: TrafficBalancer, grpc_snapshot: Arc>, filter: service::TransparentAddressBlockFilter, - reads: &ReadLanes, ) -> Result, Status> { - let txids = address_txids(address_index, filter, reads).await?; - Ok(streamed_raw_transactions(txids, validators, grpc_snapshot, reads.clone())) + let txids = address_txids(address_index, filter).await?; + Ok(streamed_raw_transactions(txids, validators, grpc_snapshot)) } /// Utxos of every requested address at or above `startHeight`, height order, capped at @@ -93,7 +88,6 @@ pub(crate) async fn get_taddress_transactions async fn address_utxos( address_index: AddressIndex, request: &service::GetAddressUtxosArg, - reads: &ReadLanes, ) -> Result, Status> { let from = args::height(request.start_height, "startHeight")?; let addresses = request @@ -103,12 +97,11 @@ async fn address_utxos( .collect::, _>>()?; let reader = address_index.reader; - let (addresses, utxos_per_address) = reads - .read(Lane::Scan, move || { - let utxos = reader.utxos_of(&addresses, from); - (addresses, utxos) - }) - .await?; + let (addresses, utxos_per_address) = blocking(move || { + let utxos = reader.utxos_of(&addresses, from); + (addresses, utxos) + }) + .await?; let utxos_per_address = utxos_per_address.map_err(serve_error_status)?; let mut address_utxos = Vec::new(); @@ -137,7 +130,6 @@ async fn address_utxos( async fn total_balance( address_index: AddressIndex, addresses: &[String], - reads: &ReadLanes, ) -> Result { let distinct: Vec = addresses .iter() @@ -147,7 +139,7 @@ async fn total_balance( .collect(); let reader = address_index.reader; - let balances = reads.read(Lane::Scan, move || reader.balances(&distinct)).await?; + let balances = blocking(move || reader.balances(&distinct)).await?; let total = balances.map_err(serve_error_status)?.into_iter().sum::>(); let total = total.ok_or_else(|| { Status::failed_precondition("these addresses' unspent total exceeds the money supply") @@ -159,7 +151,6 @@ async fn total_balance( async fn address_txids( address_index: AddressIndex, filter: service::TransparentAddressBlockFilter, - reads: &ReadLanes, ) -> Result, Status> { let address = parse_transparent_address(&filter.address, address_index.network)?; let range = filter.range.ok_or_else(|| Status::invalid_argument("range is required"))?; @@ -171,7 +162,7 @@ async fn address_txids( let (start, end) = args::ordered(bound(range.start, "start")?, bound(range.end, "end")?)?; let reader = address_index.reader; - let touching = reads.read(Lane::Scan, move || reader.transactions(&address, start, end)); + let touching = blocking(move || reader.transactions(&address, start, end)); touching.await?.map_err(serve_error_status) } @@ -183,16 +174,13 @@ fn streamed_raw_transactions( txids: Vec, validators: TrafficBalancer, grpc_snapshot: Arc>, - reads: ReadLanes, ) -> Response { let transactions = futures::stream::iter(txids).then(move |txid| { let (validators, grpc_snapshot) = (validators.clone(), Arc::clone(&grpc_snapshot)); - let reads = reads.clone(); async move { let snapshot = &grpc_snapshot; let fetched = - super::transactions::fetch_raw_transaction(&validators, snapshot, &reads, txid) - .await; + super::transactions::fetch_raw_transaction(&validators, snapshot, txid).await; fetched.map(|transaction| frame(&transaction)) } }); diff --git a/packages/zingod-grpc/src/routes/tree_state.rs b/packages/zingod-grpc/src/routes/tree_state.rs index 4cc33ef4d..3d77e5cc7 100644 --- a/packages/zingod-grpc/src/routes/tree_state.rs +++ b/packages/zingod-grpc/src/routes/tree_state.rs @@ -18,9 +18,9 @@ use zingod_proto::proto::service; use zingod_sync::IndexKind; use super::args; +use super::blocking; use super::info::chain_name; use crate::memo::PerView; -use crate::read_lanes::{Lane, ReadLanes}; use crate::wire::{self, streamed_response, unary_response}; /// Framed answers per NFS publish: the layer heights every synced wallet asks (the tip among @@ -45,7 +45,6 @@ impl Default for Memos { pub(crate) struct TreeStateReads { pub(crate) indexed: Arc>, pub(crate) reader: TreeStateReader>, - pub(crate) reads: ReadLanes, pub(crate) memos: Arc>, } @@ -66,12 +65,12 @@ impl TreeStateReads { self.memoized(|memos| &memos.tree_states, height, frame).await? } - /// Framed tree state at `height`, read on the point lane (not memoized) + /// Framed tree state at `height`, read on the blocking pool (not memoized) async fn read_tree_state(&self, height: BlockHeight) -> Result { let (reader, through, params) = (self.reader.clone(), self.served_through(), self.served().params()); let read = move || framed_tree_state(&reader, height, through, params); - self.reads.read(Lane::Point, read).await? + blocking(read).await? } /// Framed tree state at `height` only if this index holds `hash` there (a located height can @@ -91,7 +90,7 @@ impl TreeStateReads { } Ok(tree_state_message(&tree_state, params)) }; - self.reads.read(Lane::Point, read).await? + blocking(read).await? } /// Every root of `pool` completed at or below the served tip, framed once per NFS publish @@ -105,7 +104,7 @@ impl TreeStateReads { self.memoized(|memos| &memos.subtree_roots, pool, frame).await } - /// `compute` on first ask per NFS publish, on the point lane (single flight), then the cached + /// `compute` on first ask per NFS publish, on the blocking pool (single flight), then the cached /// answer inline async fn memoized( &self, @@ -123,7 +122,7 @@ impl TreeStateReads { let (memos, indexed, reader) = (Arc::clone(&self.memos), Arc::clone(&self.indexed), self.reader.clone()); let compute = move || memo(&memos).get_or_compute(&indexed, key, || compute(&reader)); - self.reads.read(Lane::Point, compute).await + blocking(compute).await } } diff --git a/packages/zingod-grpc/src/service.rs b/packages/zingod-grpc/src/service.rs index 4cbec7bf5..0c4701c4b 100644 --- a/packages/zingod-grpc/src/service.rs +++ b/packages/zingod-grpc/src/service.rs @@ -13,7 +13,6 @@ use std::{ num::NonZeroUsize, sync::{Arc, OnceLock}, task::{Context, Poll}, - time::Duration, }; use futures::future::BoxFuture; @@ -27,14 +26,11 @@ use zingod_source::ChainDataSource; use zingod_sync::IndexKind; use zingod_traffic::TrafficBalancer; -use crate::admission::Class; -use crate::deadline::Bounds; -use crate::read_lanes::ReadLanes; +use crate::deadline; use crate::routes::{ blocks, deprecated_nullifiers, info, mempool, transactions, transparent_address, tree_state, }; use crate::wire; -use crate::GrpcLimits; /// Every method's path prefix (`/{service}/{method}`) const SERVICE: &str = "/cash.z.wallet.sdk.rpc.CompactTxStreamer/"; @@ -67,11 +63,10 @@ pub(crate) enum Method { GetStandardFee, } -/// One method's policy: admission pool, reply shape (its latency metric), the index whose view -/// its handler reads (`None` = reads none: always served) +/// One method's policy: reply shape (its latency metric), the index whose view its handler reads +/// (`None` = reads none: always served) #[derive(Clone, Copy, Debug)] pub(crate) struct Route { - pub(crate) class: Class, pub(crate) reply: Reply, pub(crate) index: Option, } @@ -87,40 +82,38 @@ pub(crate) enum Reply { impl Method { /// The registry's policy columns: one arm per method pub(crate) const fn route(self) -> Route { - use Class::{Subscription, Work}; use IndexKind::{CompactBlock, TransparentAddress, TreeState}; use Method::*; use Reply::{One, Stream}; - let (class, reply, index) = match self { - GetLatestBlock => (Work, One, None), - GetBlock => (Work, One, Some(CompactBlock)), - GetBlockRange => (Work, Stream, Some(CompactBlock)), + let (reply, index) = match self { + GetLatestBlock => (One, None), + GetBlock => (One, Some(CompactBlock)), + GetBlockRange => (Stream, Some(CompactBlock)), // TODO: REMOVE — deprecated alias of `GetBlockRange` (pepper-sync still calls it) - GetBlockRangeNullifiers => (Work, Stream, Some(CompactBlock)), - GetTreeState => (Work, One, Some(TreeState)), - GetLatestTreeState => (Work, One, Some(TreeState)), - GetSubtreeRoots => (Work, Stream, Some(TreeState)), - GetAddressUtxos => (Work, One, Some(TransparentAddress)), - GetAddressUtxosStream => (Work, Stream, Some(TransparentAddress)), - GetTaddressBalance => (Work, One, Some(TransparentAddress)), - GetTaddressBalanceStream => (Work, One, Some(TransparentAddress)), + GetBlockRangeNullifiers => (Stream, Some(CompactBlock)), + GetTreeState => (One, Some(TreeState)), + GetLatestTreeState => (One, Some(TreeState)), + GetSubtreeRoots => (Stream, Some(TreeState)), + GetAddressUtxos => (One, Some(TransparentAddress)), + GetAddressUtxosStream => (Stream, Some(TransparentAddress)), + GetTaddressBalance => (One, Some(TransparentAddress)), + GetTaddressBalanceStream => (One, Some(TransparentAddress)), // index names the txs, a validator holds the bytes // (`book/src/contributors/architecture.md` §1) - GetTaddressTransactions => (Work, Stream, Some(TransparentAddress)), + GetTaddressTransactions => (Stream, Some(TransparentAddress)), // TODO: REMOVE — deprecated alias of `GetTaddressTransactions` (pepper-sync calls it) - GetTaddressTxids => (Work, Stream, Some(TransparentAddress)), + GetTaddressTxids => (Stream, Some(TransparentAddress)), // one random entry per attempt, watched until it spreads // (`book/src/contributors/architecture.md` §6) - SendTransaction => (Work, One, None), - GetMempoolTx => (Work, Stream, None), - // open until the served tip moves, silent between arrivals - GetMempoolStream => (Subscription, Stream, None), - GetTransaction => (Work, One, None), - GetLightdInfo => (Work, One, None), - GetStandardFee => (Work, One, None), + SendTransaction => (One, None), + GetMempoolTx => (Stream, None), + GetMempoolStream => (Stream, None), + GetTransaction => (One, None), + GetLightdInfo => (One, None), + GetStandardFee => (One, None), }; - Route { class, reply, index } + Route { reply, index } } /// Wire name (= the `method` metric label) @@ -145,19 +138,17 @@ impl Method { request.extensions().get::().copied() } + /// Open until the served tip moves, idle between arrivals: no work slot, no latency to time + pub(crate) fn is_subscription(self) -> bool { + matches!(self, Self::GetMempoolStream) + } + #[cfg(test)] pub(crate) fn path(self) -> String { format!("{SERVICE}{}", self.name()) } } -impl Route { - /// Open until the next block, silent between arrivals: never idles out, no latency to time - pub(crate) fn is_subscription(self) -> bool { - self.class == Class::Subscription - } -} - /// What one `GrpcService` answers from /// /// - `grpc_snapshots` = every read (chain, indexes, mempool, validator facts) @@ -183,12 +174,9 @@ pub(crate) const STARTING: &str = "starting: not routed yet"; /// [`RouteDeps`] + what every request shares, one `Arc` (a request clones one pointer) /// /// - `routes` set once, after bind (empty = every method `UNAVAILABLE` [`STARTING`]) -/// - `reads` process-wide (every index read on the blocking pool under its lane's permit) struct Wired { routes: OnceLock>, - reads: ReadLanes, tree_states: Arc>, - silent_timeout: Duration, } /// Every `CompactTxStreamer` method, dispatched by path (the tower service each connection runs; @@ -202,10 +190,8 @@ impl Clone for Dispatch { } impl Dispatch { - pub(crate) fn new(limits: &GrpcLimits) -> Self { - let (reads, silent_timeout) = (ReadLanes::new(limits), limits.silent_timeout); - let routes = OnceLock::new(); - Self(Arc::new(Wired { routes, reads, tree_states: Arc::default(), silent_timeout })) + pub(crate) fn new() -> Self { + Self(Arc::new(Wired { routes: OnceLock::new(), tree_states: Arc::default() })) } pub(crate) fn routing(&self) -> Routing { @@ -267,7 +253,6 @@ impl Wired { use Method::*; let request = wire::collect_request(body, method).await?; - let reads = self.reads.clone(); let grpc_snapshot = routes.grpc_snapshots.load(); let served = || grpc_snapshot.served().map_err(wire::unavailable); let compact_blocks = || { @@ -281,8 +266,7 @@ impl Wired { let indexed = grpc_snapshot.indexed().ok_or_else(nothing)?; let reader = view(indexed.served().views().tree_state(), method); let (indexed, memos) = (Arc::clone(indexed), Arc::clone(&self.tree_states)); - let reads = reads.clone(); - Ok::<_, Status>(tree_state::TreeStateReads { indexed, reader, reads, memos }) + Ok::<_, Status>(tree_state::TreeStateReads { indexed, reader, memos }) }; let address_index = || { let served = served()?; @@ -294,12 +278,11 @@ impl Wired { match method { GetLatestBlock => Ok(blocks::get_latest_block(served()?)), - GetBlock => blocks::get_block(compact_blocks()?, decode(&request)?, &reads).await, - GetBlockRange => blocks::get_block_range(compact_blocks()?, decode(&request)?, reads), + GetBlock => blocks::get_block(compact_blocks()?, decode(&request)?).await, + GetBlockRange => blocks::get_block_range(compact_blocks()?, decode(&request)?), GetBlockRangeNullifiers => deprecated_nullifiers::get_block_range_nullifiers( compact_blocks()?, decode(&request)?, - reads, ), GetTreeState => tree_state::get_tree_state(&tree_states()?, decode(&request)?).await, GetLatestTreeState => tree_state::get_latest_tree_state(&tree_states()?).await, @@ -308,20 +291,19 @@ impl Wired { } GetAddressUtxos => { let index = address_index()?; - transparent_address::get_address_utxos(index, decode(&request)?, &reads).await + transparent_address::get_address_utxos(index, decode(&request)?).await } GetAddressUtxosStream => { let index = address_index()?; - transparent_address::get_address_utxos_stream(index, decode(&request)?, &reads) - .await + transparent_address::get_address_utxos_stream(index, decode(&request)?).await } GetTaddressBalance => { let index = address_index()?; - transparent_address::get_taddress_balance(index, decode(&request)?, &reads).await + transparent_address::get_taddress_balance(index, decode(&request)?).await } GetTaddressBalanceStream => { let (index, request) = (address_index()?, decode_each(&request)?); - transparent_address::get_taddress_balance_stream(index, request, &reads).await + transparent_address::get_taddress_balance_stream(index, request).await } GetTaddressTransactions | GetTaddressTxids => { transparent_address::get_taddress_transactions( @@ -329,7 +311,6 @@ impl Wired { routes.validators.clone(), Arc::clone(&grpc_snapshot), decode(&request)?, - &reads, ) .await } @@ -341,7 +322,7 @@ impl Wired { GetTransaction => { let validators = &routes.validators; let filter = decode(&request)?; - transactions::get_transaction(validators, &grpc_snapshot, &reads, filter).await + transactions::get_transaction(validators, &grpc_snapshot, filter).await } GetLightdInfo => info::get_lightd_info(&grpc_snapshot, routes.network, routes.version), GetStandardFee => info::get_standard_fee(&grpc_snapshot, routes.network), @@ -383,11 +364,10 @@ where let wired = Arc::clone(&self.0); let path = request.uri().path().to_owned(); let method = Method::of_request(&request); - let bounded = method.is_none_or(|method| !method.route().is_subscription()); - let bounds = Bounds::of(request.headers(), bounded.then_some(wired.silent_timeout)); - Box::pin( - async move { Ok(bounds.apply(wired.answer(method, &path, request.into_body())).await) }, - ) + let deadline = deadline::of(request.headers()); + Box::pin(async move { + Ok(deadline::apply(deadline, wired.answer(method, &path, request.into_body())).await) + }) } } diff --git a/packages/zingod-grpc/src/summary.rs b/packages/zingod-grpc/src/summary.rs index 880c03747..fd4fa0944 100644 --- a/packages/zingod-grpc/src/summary.rs +++ b/packages/zingod-grpc/src/summary.rs @@ -1,11 +1,7 @@ -//! Serving summary every [`INTERVAL`] (per-method latency, bytes, codes = the `zingod.grpc.*` -//! metrics) +//! `Serving` log line every [`INTERVAL`]: rate, connections, refused + failed (the rest = metrics) //! -//! - `Serving`: rate, connections held, then the window's problems by count (silent while nothing -//! served or held) -//! - `Serving` level: WARN once any request failed, was refused, stalled or ran slow, else INFO -//! - first server fault (ERROR `Request failed`) + first unavailable answer (WARN `Request -//! unavailable`) of a window logged as they happen, the rest only counted +//! - WARN once any request was refused or failed, else INFO; silent while idle +//! - first server fault of a window logged as it happens (ERROR `Request failed`), the rest counted use std::sync::atomic::{AtomicBool, AtomicU64, Ordering}; use std::time::Duration; @@ -18,156 +14,50 @@ use crate::service::Method; pub(crate) const INTERVAL: Duration = Duration::from_secs(60); -/// Time to first message past this = a slow request -const SLOW: Duration = Duration::from_secs(1); +static REQUESTS: AtomicU64 = AtomicU64::new(0); +static REFUSED: AtomicU64 = AtomicU64::new(0); +static FAILED: AtomicU64 = AtomicU64::new(0); +static FAILURE_LOGGED: AtomicBool = AtomicBool::new(false); -/// Code this server answers for (any other = the client's: bad argument, not found, cancelled) -/// -/// - `Refused` = not now (capacity, nothing served yet, validator down), `Failed` = this server -#[derive(Clone, Copy, Debug, PartialEq, Eq)] -enum Blame { - Refused, - Failed, +/// This server's fault (any other non-`Ok` = the client's or a refusal) +fn is_failure(code: Code) -> bool { + matches!(code, Code::Unknown | Code::Internal | Code::DataLoss) } -fn blame(code: Code) -> Option { - match code { - Code::Unavailable | Code::ResourceExhausted => Some(Blame::Refused), - Code::Unknown | Code::Internal | Code::DataLoss => Some(Blame::Failed), - _ => None, +/// Stream close-out; a window's first failure logged with the message its client got +pub(crate) fn finished(method: Option, code: Code, message: &str) { + REQUESTS.fetch_add(1, Ordering::Relaxed); + if matches!(code, Code::Unavailable | Code::ResourceExhausted) { + REFUSED.fetch_add(1, Ordering::Relaxed); } -} - -/// Everything counted since the last summary (`*_logged` = this window's first already logged) -struct Window { - requests: AtomicU64, - refused: AtomicU64, - failed: AtomicU64, - slow: AtomicU64, - connections_refused: AtomicU64, - stalled: AtomicU64, - failure_logged: AtomicBool, - unavailable_logged: AtomicBool, -} - -#[derive(Debug, Default, PartialEq, Eq)] -struct Counts { - requests: u64, - refused: u64, - failed: u64, - slow: u64, - connections_refused: u64, - stalled: u64, -} - -impl Window { - const fn new() -> Self { - Self { - requests: AtomicU64::new(0), - refused: AtomicU64::new(0), - failed: AtomicU64::new(0), - slow: AtomicU64::new(0), - connections_refused: AtomicU64::new(0), - stalled: AtomicU64::new(0), - failure_logged: AtomicBool::new(false), - unavailable_logged: AtomicBool::new(false), - } - } - - fn finished(&self, code: Code, latency: Option) { - let count = |counter: &AtomicU64| counter.fetch_add(1, Ordering::Relaxed); - count(&self.requests); - match blame(code) { - Some(Blame::Refused) => count(&self.refused), - Some(Blame::Failed) => count(&self.failed), - None => 0, - }; - if latency.is_some_and(|latency| latency > SLOW) { - count(&self.slow); - } - } - - /// Every counter read and zeroed (a request landing mid-swap counts in one window or the next) - fn take(&self) -> Counts { - let take = |counter: &AtomicU64| counter.swap(0, Ordering::Relaxed); - self.failure_logged.store(false, Ordering::Relaxed); - self.unavailable_logged.store(false, Ordering::Relaxed); - Counts { - requests: take(&self.requests), - refused: take(&self.refused), - failed: take(&self.failed), - slow: take(&self.slow), - connections_refused: take(&self.connections_refused), - stalled: take(&self.stalled), + if is_failure(code) { + FAILED.fetch_add(1, Ordering::Relaxed); + if !FAILURE_LOGGED.swap(true, Ordering::Relaxed) { + let (method, code) = (method_label(method), code_label(code)); + error!(method, code, error = message, "Request failed"); } } } -static WINDOW: Window = Window::new(); - -/// Stream close-out (`latency` = time to its first message, for an answered work request) -pub(crate) fn finished(code: Code, latency: Option) { - WINDOW.finished(code, latency); +/// (requests, refused, failed) since the last call, zeroed +fn take() -> (u64, u64, u64) { + let take = |counter: &AtomicU64| counter.swap(0, Ordering::Relaxed); + FAILURE_LOGGED.store(false, Ordering::Relaxed); + (take(&REQUESTS), take(&REFUSED), take(&FAILED)) } -pub(crate) fn connection_refused() { - WINDOW.connections_refused.fetch_add(1, Ordering::Relaxed); -} - -pub(crate) fn connection_stalled() { - WINDOW.stalled.fetch_add(1, Ordering::Relaxed); -} - -/// A server fault or unavailable answer (never an admission refusal): the window's first of each -/// is logged now, with the message the client got -pub(crate) fn answered_badly(method: Option, code: Code, message: &str) { - let first = |logged: &AtomicBool| !logged.swap(true, Ordering::Relaxed); - let method = method_label(method); - match blame(code) { - Some(Blame::Failed) if first(&WINDOW.failure_logged) => { - error!(method, code = code_label(code), error = message, "Request failed"); - } - Some(Blame::Refused) if first(&WINDOW.unavailable_logged) => { - warn!(method, error = message, "Request unavailable"); - } - _ => {} - } -} - -/// Logs the window since the last call (`elapsed` long, `connections` held now, `at_capacity` -/// admission refusals) and starts the next -pub(crate) fn summarise(elapsed: Duration, connections: usize, at_capacity: u64) { - let counts = WINDOW.take(); - let problems = - counts.refused + counts.failed + counts.slow + counts.stalled + counts.connections_refused; - // no connection = no stream held either - if counts.requests == 0 && connections == 0 && problems == 0 { +/// Logs the window since the last call (`elapsed` long, `connections` held now) +pub(crate) fn summarise(elapsed: Duration, connections: usize) { + let (requests, refused, failed) = take(); + if requests == 0 && connections == 0 { return; } - // requests × 1 s / window = per second - let window = elapsed.max(Duration::from_nanos(1)); - let per_s = Duration::from_secs(counts.requests).div_duration_f64(window); + let per_s = + Duration::from_secs(requests).div_duration_f64(elapsed.max(Duration::from_nanos(1))); let rps = (per_s * 100.0).round() / 100.0; - let nonzero = |count: u64| (count > 0).then_some(count); - - macro_rules! serving { - ($level:ident) => { - $level!( - rps, - conns = connections, - failed = nonzero(counts.failed), - refused = nonzero(counts.refused), - at_capacity = nonzero(at_capacity), - slow = nonzero(counts.slow), - stalled = nonzero(counts.stalled), - conns_refused = nonzero(counts.connections_refused), - "Serving" - ) - }; - } - match problems { - 0 => serving!(info), - _ => serving!(warn), + match refused + failed { + 0 => info!(rps, conns = connections, "Serving"), + _ => warn!(rps, conns = connections, refused, failed, "Serving"), } } @@ -175,28 +65,15 @@ pub(crate) fn summarise(elapsed: Duration, connections: usize, at_capacity: u64) mod tests { use super::*; - /// - codes split by who they blame (`NotFound`, `Cancelled` = the client's: uncounted) - /// - slow = past [`SLOW`] to the first message - /// - taken = zeroed + /// Refusals + server faults counted apart; client errors only as requests; taken = zeroed #[test] - fn a_window_counts_refusals_failures_and_slow_requests() { - let window = Window::new(); - window.finished(Code::Ok, Some(SLOW)); - window.finished(Code::Ok, Some(SLOW + Duration::from_millis(1))); - for code in [Code::NotFound, Code::Cancelled, Code::Unavailable, Code::Internal] { - window.finished(code, None); + fn a_window_counts_refusals_and_failures_apart_from_client_errors() { + let codes = [Code::Ok, Code::NotFound, Code::Cancelled, Code::Unavailable, Code::Internal]; + for code in codes { + finished(None, code, ""); } - let taken = window.take(); - - let expected = Counts { - requests: 6, - refused: 1, // Unavailable - failed: 1, // Internal - slow: 1, // SLOW + 1 ms (SLOW itself is not) - ..Counts::default() - }; - assert_eq!(taken, expected, "NotFound + Cancelled = the client's, uncounted"); - assert_eq!(window.take(), Counts::default(), "taken = zeroed"); + assert_eq!(take(), (5, 1, 1), "(requests, refused, failed)"); + assert_eq!(take(), (0, 0, 0), "taken = zeroed"); } } diff --git a/packages/zingod-grpc/src/testing.rs b/packages/zingod-grpc/src/testing.rs index e52ef0b19..73d7c6240 100644 --- a/packages/zingod-grpc/src/testing.rs +++ b/packages/zingod-grpc/src/testing.rs @@ -170,7 +170,7 @@ impl Drop for LiveRoutes { pub(super) fn dispatch( routes: RouteDeps, ) -> Dispatch { - let dispatch = Dispatch::new(&crate::GrpcLimits::default()); + let dispatch = Dispatch::new(); dispatch.routing().route(routes); dispatch } diff --git a/packages/zingod-grpc/src/transport.rs b/packages/zingod-grpc/src/transport.rs index 4fc6ed927..b2b1d2c71 100644 --- a/packages/zingod-grpc/src/transport.rs +++ b/packages/zingod-grpc/src/transport.rs @@ -27,8 +27,6 @@ use crate::connections::{ConnectionCaps, Reserved}; use crate::guard::Guard; use crate::health::{Health, HealthRoute}; use crate::lifetime::{Activity, Retire, RetireReason, Retirement}; -use crate::overload::Signals; -use crate::pressure::{Pressure, Resource}; use crate::proxy_protocol; use crate::service::{Dispatch, Routing}; use crate::summary::{self, INTERVAL}; @@ -87,7 +85,7 @@ pub enum GrpcServeError { impl GrpcService { /// Unrouted: health answered, every method `UNAVAILABLE` until [`Routing::route`] pub fn new(bind: SocketAddr, limits: GrpcLimits) -> Self { - let dispatch = Dispatch::new(&limits); + let dispatch = Dispatch::new(); let admission = Arc::new(AdmissionState::new(&limits)); let retire = Retire::default(); let origin_gate = crate::OriginGate::new(); @@ -258,7 +256,6 @@ impl BoundGrpcService { if let Some(tls) = &server.tls { tokio::spawn(tls.reload(cancel.clone()).instrument(Span::current())); } - let (mut cpu, mut io) = (Pressure::new(Resource::Cpu), Pressure::new(Resource::Io)); let shared = Arc::new(Shared { caps: ConnectionCaps::new(&server.limits), tls: server.tls.as_ref().map(crate::Tls::acceptor), @@ -283,9 +280,8 @@ impl BoundGrpcService { accepted = listener.accept() => accepted, now = summaries.tick() => { let elapsed = now.saturating_duration_since(window_opened); - let signals = Signals { cpu: cpu.share(elapsed), io: io.share(elapsed) }; - let at_capacity = shared.admission.tick(elapsed, signals); - summary::summarise(elapsed, shared.caps.held(), at_capacity); + shared.admission.tick(elapsed); + summary::summarise(elapsed, shared.caps.held()); window_opened = now; continue; } @@ -951,11 +947,7 @@ mod tests { let healthy = ("200".to_owned(), r#"{"state":"serving","detail":null}"#.to_owned()); assert_eq!(readyz().await, healthy, "ALPN http/1.1 /readyz, unrouted"); - assert_eq!( - admission.held(crate::admission::Class::Work), - 0, - "a monitor's /readyz never holds a work slot" - ); + assert_eq!(admission.held(), 0, "a monitor's /readyz never holds a work slot"); let starting = (Code::Unavailable, crate::service::STARTING.to_owned()); assert_eq!(lightd_info().await, starting, "ALPN h2 → the gRPC stack, unrouted"); diff --git a/packages/zingod-grpc/usage.md b/packages/zingod-grpc/usage.md index 75537a837..ba4671a5f 100644 --- a/packages/zingod-grpc/usage.md +++ b/packages/zingod-grpc/usage.md @@ -92,9 +92,9 @@ Every method is one `Method` variant in `service.rs`; its name is its wire name metric label. A new RPC is: 1. a `Method` variant, -2. its arm in `Method::route`: its admission `Class` (`Work`, or `Subscription` for a stream - that idles between arrivals: own pool, never cut for silence, no latency recorded) and the - index whose view its handler reads (`None` = reads none, so never disabled), +2. its arm in `Method::route`: its `Reply` and the index whose view its handler reads (`None` = + reads none, so never disabled); every method but `GetMempoolStream` + (`Method::is_subscription`: idle between arrivals) takes a work slot, 3. its arm in `Wired::handle`, which hands the handler its decoded request (`wire::decode`, or `wire::decode_each` for a client stream); the handler returns `Result, Status>` built with `wire::unary_response`, `wire::streamed_response` or `wire::streaming`, never by @@ -121,7 +121,7 @@ accept loop is zingod's. Outermost first, per connection after the first three: ```text accept ─ total cap ─ PROXY header (trusted peers) ─ per-client cap ─ TLS ─ h2 / HTTP/1.1 ─ health (resolves the RPC) ─ CORS + gRPC-web (with_grpc_web) ─ guard (metrics, admission, - origin gate, stall watch) ─ dispatch (grpc-timeout, silence, request body) ─ handler + origin gate, stall watch) ─ dispatch (grpc-timeout, request body) ─ handler ``` The guard (`guard.rs`) wraps each response body once: the body owns its admission slot, its @@ -141,8 +141,9 @@ stream ending or its client disconnecting). What each `[grpc]` cap bounds: - **Origin gate** (`with_origin_gate(Arc)`, read in `guard.rs`): a request's `Origin` → `block` (`PermissionDenied` before admission), else the normal admission path; `allow` only carves a host out of a blocked parent, never lifts a cap (`Origin` is client-settable). A - missing `Origin` (native clients) is default and uncounted. Markers are set operator-side (`zingod origin`) and swapped - whole each second; `verdict` folds an incoming host to the most-specific marker covering it. + missing `Origin` (native clients) is default and uncounted. Markers are set operator-side + (`zingod origin`) and swapped whole each second; `verdict` folds an incoming host to the + most-specific marker covering it. - **TLS** (`with_tls(Tls::load(files)?)` or `with_tls(Tls::self_signed(dir, listen)?)`): rustls with the aws-lc-rs provider, ALPN `h2` then `http/1.1` (load-balancer monitors connect over HTTPS/1.1). A missing or mismatched pair fails `Tls::load`, never plaintext. The files are @@ -165,9 +166,9 @@ stream ending or its client disconnecting). What each `[grpc]` cap bounds: to the first DATA frame, then poll to poll), so a client's backpressure never counts. Each window (≥ 500 ms, ≥ 10 samples) sets `limit × clamp(1.5 × baseline / mean, 0.5, 1) + √limit`, smoothed at 0.2, between 32 and `max_work_streams`; a window that peaked below half the limit - changes nothing. -- **Overload report.** `GrpcService::admission()` → `AdmissionState::overload()` gives the - first latched [overload](#overload) cause, moved once a minute. + changes nothing. Always on: the one authority on load. +- **Overload report.** `GrpcService::admission()` → `AdmissionState::overloaded()`, the + [overload](#overload) latch, stepped once a minute. - **Trusted proxies.** A peer inside `trusted_proxies` must open with a PROXY header (v1 or v2, read under 5 s, ≤ 4 KiB), and the per-client cap counts the address it names; `LOCAL` / `UNKNOWN` count the proxy itself. @@ -181,21 +182,18 @@ stream ending or its client disconnecting). What each `[grpc]` cap bounds: - **Unread.** A body that handed hyper a frame and is not polled again (flow control stalled on a client that stopped reading) owes that frame. Once any stream on a connection has owed for `unread_timeout`, the connection is closed. An idle stream (a quiet mempool subscription, a - read waiting for its lane) never owes. -- **Silence.** A handler or stream body with nothing to send for `silent_timeout` ends with - `UNAVAILABLE`; the clock restarts at every frame. `GetMempoolStream` is exempt. + pending read) never owes. - **Frames.** Records ready back to back are joined into DATA chunks of up to 16 KiB, and a record of 16 KiB or more passes through uncopied. One small frame per record makes an h2 0.4.16+ client that reads late answer `GOAWAY too_many_data_frames` (lightwalletd #593). - `method` and `code` metric labels come from fixed sets, so a misbehaving client cannot grow the series count. -### Read lanes +### Disk reads -`ReadLanes` runs every read that touches index files on the blocking pool, under a permit of its -lane, never on a runtime worker: **point** (block, tree state, subtree roots), **range** (each -`GetBlockRange` window below the NFS layer), **scan** (address UTXOs, balances, transactions). A -heavy scan queues behind the other scans, never behind or ahead of a tip read. +Every read that touches index files runs on the blocking pool (`routes::blocking`), never on a +runtime worker. No separate cap: each read sits inside a work stream, so the adaptive limit +bounds them, and a slow disk lowers it. ### Answered without a read @@ -212,15 +210,14 @@ The answers every synced wallet asks for right after each block never queue: The tree-state memos (`memo::PerView`) are keyed on the NFS publish (the `NfsPublication` `Arc`), not the `GrpcSnapshot`, which moves on every chain-view fold. A new publish starts empty, so -there is no invalidation rule. The first ask per publish computes on the point lane, +there is no invalidation rule. The first ask per publish computes on the blocking pool, single-flight; every later ask is answered inline. ### Overload -`AdmissionState::overload()` names the first cause latched (`/readyz`: `Overloaded: `); -`AdmissionState::set_overload` changes the `[grpc.overload]` thresholds (`OverloadLimits`) live. -Causes, signals and enter / exit times: -[operators: overload](../../book/src/operators/maintenance.md#overload). +`AdmissionState::overloaded()` latches on work refused at the adaptive limit, and nothing else: +CPU or IO saturation reaches it as latency, which lowers the limit. Enter / exit rules: +[operators: overload](../../book/src/operators/dos-protection.md#overload). ### Serving log diff --git a/packages/zingod-index-transparent-address/usage.md b/packages/zingod-index-transparent-address/usage.md index b29a25637..80b5dda3c 100644 --- a/packages/zingod-index-transparent-address/usage.md +++ b/packages/zingod-index-transparent-address/usage.md @@ -100,7 +100,7 @@ let touched = reader.transactions(&address, start, end)?; // Vec, (height, request fails with `ServeError::TooManyRows`. It never returns a short list or a partial balance. - Every method is synchronous and reads mmapped pages, so a transport runs it - off its async workers (`zingod-grpc`'s scan lane). + off its async workers (`zingod-grpc`'s blocking pool). - Blocks above the durable tip are `zingod-nfs` layers keyed exactly like the maps. An `NfsPublication`'s reader merges them over the durable store by key. diff --git a/packages/zingod-persistence/usage.md b/packages/zingod-persistence/usage.md index ecdc26d13..4d4d16568 100644 --- a/packages/zingod-persistence/usage.md +++ b/packages/zingod-persistence/usage.md @@ -190,7 +190,7 @@ append-only file stays unfragmented where a copy-on-write B-tree under random in interrupted commit or the zeroed reserve. Every read goes through `Pages`, sliced to the sealed length. - **Page faults on async threads.** A cold read can block for milliseconds. Callers run reads on the - blocking pool (`zingod-grpc`'s read lanes), and multi-record reads issue `MADV_WILLNEED` first. + blocking pool (`zingod-grpc`), and multi-record reads issue `MADV_WILLNEED` first. A batch of point lookups advises in two rounds, every candidate segment's fence page and then its record block, so the device sees the whole batch at once instead of two serial faults per segment per key. Point-lookup segments are mapped `MADV_RANDOM`, merge inputs diff --git a/packages/zingod/src/cli.rs b/packages/zingod/src/cli.rs index c90f6bd6a..c02036d8a 100644 --- a/packages/zingod/src/cli.rs +++ b/packages/zingod/src/cli.rs @@ -42,15 +42,6 @@ pub(super) enum Command { Verify, #[command(flatten)] Hold(crate::command::hold::Action), - /// Override when this node reports `Overloaded`, across restarts: markers in `[serve] - /// state_dir`/overload over `[grpc.overload]`, read by the running zingod within a second. - /// - /// Prints the settings in force. Exits 0 once the marker matches, 1 on an invalid key or - /// value, or an unreadable config or state directory. - Overload { - #[command(subcommand)] - action: crate::command::overload::Action, - }, /// Origin policy for browser (gRPC-web) clients, across restarts: markers in `[serve] /// state_dir`/origins, read by the running zingod within a second. /// diff --git a/packages/zingod/src/command.rs b/packages/zingod/src/command.rs index ea753d535..74958e938 100644 --- a/packages/zingod/src/command.rs +++ b/packages/zingod/src/command.rs @@ -1,11 +1,10 @@ -//! One-shot subcommands: `verify`, `hold` / `resume`, `overload` +//! One-shot subcommands: `verify`, `hold` / `resume`, `origin` //! -//! - hold + overload = marker files in `[serve] state_dir` (CLI writes, running zingod polls each +//! - hold + origin = marker files in `[serve] state_dir` (CLI writes, running zingod polls each //! second: both halves of each contract live in its file) pub(super) mod hold; pub(super) mod origin; -pub(super) mod overload; pub(super) mod verify; use std::io; diff --git a/packages/zingod/src/command/overload.rs b/packages/zingod/src/command/overload.rs deleted file mode 100644 index b0958336c..000000000 --- a/packages/zingod/src/command/overload.rs +++ /dev/null @@ -1,254 +0,0 @@ -//! `zingod overload on|off|set|unset`: operator markers in `/overload/` over -//! `[grpc.overload]` -//! -//! - `off` = never `Overloaded`; `` = that key's value; both survive restarts (as `held`) -//! - the running zingod reads them every second → the gRPC overload latch (a change = WARN; never -//! on public `/status`) - -use std::path::{Path, PathBuf}; -use std::process::ExitCode; -use std::sync::Arc; -use std::time::Duration; - -use tracing::warn; -use zingod_grpc::{AdmissionState, OverloadLimits}; - -use super::{remove_marker, write_marker, MarkerError}; - -const OFF: &str = "off"; - -#[derive(Debug, Clone, PartialEq, Eq, clap::Subcommand)] -pub(crate) enum Action { - /// Report `Overloaded` again when a cause latches. - On, - /// Never report `Overloaded` (a misfiring detector keeps the node in rotation). - Off, - /// Override one `[grpc.overload]` key, e.g. `cpu_pressure_percent 40`. - Set { key: String, value: String }, - /// Back to the config's value for one key. - Unset { key: String }, -} - -/// The config with every marker applied -#[derive(Debug, Clone, PartialEq, Eq)] -struct Settings { - limits: OverloadLimits, - detection: bool, - overrides: Vec, -} - -impl Settings { - fn unmarked(config: &OverloadLimits) -> Self { - Self { limits: config.clone(), detection: true, overrides: Vec::new() } - } - - /// Log notes (none while nothing is overridden) - fn notes(&self) -> Vec { - let off = - (!self.detection).then(|| "Overload detection off (zingod overload on)".to_owned()); - let set = (!self.overrides.is_empty()).then(|| { - format!("Overload overridden: {} (zingod overload unset)", self.overrides.join(", ")) - }); - off.into_iter().chain(set).collect() - } -} - -/// `config` with `markers` (`key`, `value`) applied, by `[grpc.overload]`'s own rules -/// -/// - each marker checked as it lands: a refusal names its `key = value` -fn merged(config: &OverloadLimits, markers: &[(String, String)]) -> Result { - let mut table = - toml::Table::try_from(config).expect("OverloadLimits serializes to a TOML table"); - let mut limits = config.clone(); - for (key, value) in markers { - let refused = |why: &str| format!("{key} = {value}: {why}"); - let parsed = toml::from_str::(&format!("value = {value}")) - .ok() - .and_then(|mut parsed| parsed.remove("value")) - .ok_or_else(|| refused("not a value"))?; - table.insert(key.clone(), parsed); - limits = table.clone().try_into().map_err(|e: toml::de::Error| refused(e.message()))?; - } - Ok(limits) -} - -/// `dir`'s markers over `config` (no `dir` = no markers) -fn read(dir: &Path, config: &OverloadLimits) -> Result { - let io = |path: &Path, source| MarkerError::Io { path: path.to_owned(), source }; - let entries = match std::fs::read_dir(dir) { - Ok(entries) => entries, - Err(error) if error.kind() == std::io::ErrorKind::NotFound => { - return Ok(Settings::unmarked(config)) - } - Err(error) => return Err(io(dir, error)), - }; - let mut markers = Vec::new(); - let mut detection = true; - for entry in entries { - let entry = entry.map_err(|error| io(dir, error))?; - let key = entry.file_name().to_string_lossy().into_owned(); - match key.as_str() { - OFF => detection = false, - _ => { - let path = entry.path(); - let value = std::fs::read_to_string(&path).map_err(|error| io(&path, error))?; - markers.push((key, value.trim().to_owned())); - } - } - } - markers.sort(); - let limits = merged(config, &markers).map_err(MarkerError::Invalid)?; - let overrides = markers.iter().map(|(key, value)| format!("{key} = {value}")).collect(); - Ok(Settings { limits, detection, overrides }) -} - -/// The marker `action` names written or removed (`Set` validated first) -fn apply(dir: &Path, config: &OverloadLimits, action: &Action) -> Result<(), MarkerError> { - match action { - Action::Off => write_marker(dir, OFF, ""), - Action::Set { key, value } => { - merged(config, &[(key.clone(), value.clone())]).map_err(MarkerError::Invalid)?; - write_marker(dir, key, value) - } - Action::On => remove_marker(dir, OFF).map(drop), - Action::Unset { key } => remove_marker(dir, key).map(drop), - } -} - -/// `zingod overload …`: prints the settings now in force -pub(crate) fn command(config_path: &Path, action: Action) -> Result { - let config = crate::config::load_config(config_path)?; - let dir = config.serve.state_dir.join("overload"); - let limits = &config.grpc.overload; - apply(&dir, limits, &action)?; - let settings = read(&dir, limits)?; - let printed = toml::to_string(&settings.limits).expect("OverloadLimits serializes to TOML"); - println!("detection = {}", settings.detection); - print!("{printed}"); - eprintln!( - "overridden: [{}] in {} (a running zingod applies within 1 s)", - settings.overrides.join(", "), - dir.display() - ); - Ok(ExitCode::SUCCESS) -} - -/// Every second for the process's life: the markers → `admission` (a change = WARN; unreadable or -/// invalid = WARN, settings kept) -pub(crate) fn spawn(admission: Arc, state_dir: &Path, config: OverloadLimits) { - let dir: PathBuf = state_dir.join("overload"); - tokio::spawn(async move { - let mut current = Settings::unmarked(&config); - let mut tick = tokio::time::interval(Duration::from_secs(1)); - loop { - tick.tick().await; - let settings = match read(&dir, &config) { - Ok(settings) if settings == current => continue, - Ok(settings) => settings, - Err(error) => { - warn!(%error, "Overload markers ignored, settings kept"); - continue; - } - }; - admission.set_overload(settings.limits.clone(), settings.detection); - warn!(notes = ?settings.notes(), "Overload settings changed by operator"); - current = settings; - } - }); -} - -#[cfg(test)] -mod tests { - use super::*; - - /// A session of commands, read back as the running zingod reads it: - /// - off / on toggle detection; set / unset one key over the config's (twice = once) - /// - notes name what is in force, none once all is undone - #[test] - fn markers_override_the_config_until_undone() { - let root = tempfile::tempdir().expect("tempdir"); - let dir = root.path().join("overload"); - let config = OverloadLimits::default(); - let run = |action: Action| apply(&dir, &config, &action); - let set = - |key: &str, value: &str| Action::Set { key: key.to_owned(), value: value.to_owned() }; - let defaults = Settings { limits: config.clone(), detection: true, overrides: vec![] }; - - assert_eq!(read(&dir, &config).expect("reads"), defaults, "no markers: the config"); - for action in [ - Action::Off, - Action::Off, - set("io_pressure_percent", "10"), - set("cpu_pressure_percent", "40"), - ] { - run(action).expect("applies"); - } - let expected = Settings { - limits: OverloadLimits { cpu_pressure_percent: 40, io_pressure_percent: 10 }, - detection: false, - overrides: vec!["cpu_pressure_percent = 40".into(), "io_pressure_percent = 10".into()], - }; - assert_eq!(read(&dir, &config).expect("reads"), expected, "off twice + two keys set"); - assert_eq!( - expected.notes(), - [ - "Overload detection off (zingod overload on)", - "Overload overridden: cpu_pressure_percent = 40, io_pressure_percent = 10 \ - (zingod overload unset)", - ], - "notes while overridden" - ); - - for action in [ - Action::On, - Action::On, - Action::Unset { key: "cpu_pressure_percent".into() }, - Action::Unset { key: "io_pressure_percent".into() }, - ] { - run(action).expect("applies"); - } - assert_eq!(read(&dir, &config).expect("reads"), defaults, "all undone"); - assert_eq!(defaults.notes(), Vec::::new(), "nothing overridden: no notes"); - } - - /// Invalid key or value refused naming it, before anything is written - #[test] - fn a_bad_override_is_refused_naming_it_and_never_lands() { - let root = tempfile::tempdir().expect("tempdir"); - let dir = root.path().join("overload"); - let config = OverloadLimits::default(); - - #[rustfmt::skip] - let refusals = [ - // key, value, refusal names - ("nope", "1", "nope = 1: unknown field `nope`"), - ("cpu_pressure_percent", "300", "cpu_pressure_percent = 300: invalid value"), - ("io_pressure_percent", "x", "io_pressure_percent = x: not a value"), - ]; - for (key, value, named) in refusals { - let set = Action::Set { key: key.to_owned(), value: value.to_owned() }; - let refused = apply(&dir, &config, &set).map_err(|error| error.to_string()); - assert!( - refused.as_ref().is_err_and(|refused| refused.contains(named)), - "{key} = {value}: refused naming {named:?}, got {refused:?}" - ); - } - let untouched = Settings { limits: config.clone(), detection: true, overrides: vec![] }; - assert_eq!(read(&dir, &config).expect("reads"), untouched, "nothing refused landed"); - } - - /// Marker dir present but unlistable (a file here) = error naming it, never "no markers" - #[test] - fn an_unreadable_marker_dir_is_an_error_not_the_config() { - let root = tempfile::tempdir().expect("tempdir"); - let dir = root.path().join("overload"); - std::fs::write(&dir, "").expect("file where the dir goes"); - - let read = read(&dir, &OverloadLimits::default()); - assert!( - matches!(&read, Err(MarkerError::Io { path, .. }) if *path == dir), - "Io error naming {}, got {read:?}", - dir.display() - ); - } -} diff --git a/packages/zingod/src/config.rs b/packages/zingod/src/config.rs index b4c0f606d..627c1e102 100644 --- a/packages/zingod/src/config.rs +++ b/packages/zingod/src/config.rs @@ -679,7 +679,7 @@ fn resolve_validators( tables.into_iter().map(resolve).collect() } -/// [`DaemonConfigFile`], resolved: what the daemon, `verify`, `hold` and `overload` run on +/// [`DaemonConfigFile`], resolved: what the daemon, `verify`, `hold` and `origin` run on #[derive(Debug)] pub(crate) struct DaemonConfig { pub(crate) network: NetworkType, @@ -1097,11 +1097,9 @@ key_path = "/k.pem" [grpc] max_work_streams = 64 unread_timeout_secs = 30 -silent_timeout_secs = 45 max_connection_age_secs = 0 max_connection_age_grace_secs = 60 max_connection_idle_secs = 120 -adaptive_work_streams = false shutdown_grace_secs = 600 trusted_proxies = ["10.0.0.0/8"] @@ -1128,11 +1126,9 @@ cache_dir = "/var/cache/zingod/peers" let grpc = GrpcLimits { max_work_streams: NonZeroUsize::new(64).expect("non-zero"), unread_timeout: secs(30), - silent_timeout: secs(45), max_connection_age: None, max_connection_age_grace: secs(60), max_connection_idle: Some(secs(120)), - adaptive_work_streams: false, drain_timeout: secs(600), trusted_proxies: zingod_grpc::TrustedProxies::new(vec!["10.0.0.0/8" .parse() diff --git a/packages/zingod/src/daemon.rs b/packages/zingod/src/daemon.rs index 9bb86c5b4..a602f8109 100644 --- a/packages/zingod/src/daemon.rs +++ b/packages/zingod/src/daemon.rs @@ -49,7 +49,7 @@ use zingod_source::{BlockchainInfo, ChainDataSource}; use zingod_sync::{FinalizedFollower, IndexKind}; use zingod_traffic::{Push, TrafficBalancer, TrustedValidatorId}; -use crate::command::{origin, overload}; +use crate::command::origin; use crate::config::{DaemonConfig, IndexConfig, ServeTls, TrustedValidatorConfig, ValidatorClient}; use crate::error::IndexerError; use crate::health::{admin, DaemonStatus, Sources}; @@ -159,7 +159,6 @@ async fn listen( server = server.with_tls(tls); } let serving = Serving { routing: server.routing(), admission: server.admission() }; - overload::spawn(server.admission(), &config.serve.state_dir, config.grpc.overload.clone()); origin::spawn(origin_gate, &config.serve.state_dir); let server = server.bind().await?; span.in_scope(|| info!(%endpoint, network, tls = mode, "Listening")); diff --git a/packages/zingod/src/health.rs b/packages/zingod/src/health.rs index 0cafc4f04..5afb6b592 100644 --- a/packages/zingod/src/health.rs +++ b/packages/zingod/src/health.rs @@ -13,7 +13,7 @@ use std::sync::{ use serde::Serialize; use tokio::sync::watch; use zcash_protocol::consensus::NetworkType; -use zingod_grpc::{Hold, OverloadCause, Retire}; +use zingod_grpc::{Hold, Retire}; use zingod_grpc_snapshot::{GrpcSnapshot, GrpcSnapshots, ServingStatus}; use zingod_persistence::DiskView; use zingod_primitives::metric::Metric; @@ -68,8 +68,7 @@ const METRICS: &[Metric] = &[ ), Metric::gauge( REASON, - "Why down, degraded or held, one-hot by reason (= /readyz's detail.reason; overloaded = \ - its cause in zingod_grpc_overload_latched)", + "Why down, degraded or held, one-hot by reason (= /readyz's detail.reason)", ), ]; @@ -149,7 +148,7 @@ struct Catchup { /// Why `down`, `degraded` or `held` /// -/// - `Overloaded` = `zingod_grpc::AdmissionState::overload` +/// - `Overloaded` = `zingod_grpc::AdmissionState::overloaded` /// - `GoawaySent` = held past `hold_goaway_delay_secs`, every connection GOAWAY'd (`zingod hold` /// waits for this) #[derive(Debug, Clone, Copy, PartialEq, Eq)] @@ -157,7 +156,7 @@ pub(crate) enum DaemonReason { NoValidatorReachable, TipUnvouched, RuntimeStalled, - Overloaded(OverloadCause), + Overloaded, GoawayPending, GoawaySent, } @@ -178,7 +177,7 @@ impl DaemonReason { Self::NoValidatorReachable => "no_validator_reachable", Self::TipUnvouched => "tip_unvouched", Self::RuntimeStalled => "runtime_stalled", - Self::Overloaded(_) => "overloaded", + Self::Overloaded => "overloaded", Self::GoawayPending => "goaway_pending", Self::GoawaySent => "goaway_sent", } @@ -189,7 +188,7 @@ impl DaemonReason { Self::NoValidatorReachable => "No trusted validator reachable", Self::TipUnvouched => "No trusted validator vouches for the tip", Self::RuntimeStalled => "Runtime stalled", - Self::Overloaded(cause) => cause.detail(), + Self::Overloaded => "Overloaded: refusing work", Self::GoawayPending => "GOAWAY pending", Self::GoawaySent => "GOAWAY sent", } @@ -217,7 +216,7 @@ enum Stage { /// One snapshot's facts: `height` = served (else handed), `target` = [`GrpcSnapshot::sync_target`] struct Booted { serving_status: ServingStatus, - overload: Option, + overloaded: bool, height: Option, target: Option, rate: Option, @@ -231,7 +230,7 @@ impl Booted { let now = now.map_or(0, |since| i64::try_from(since.as_secs()).unwrap_or(i64::MAX)); Self { serving_status: grpc_snapshot.serving_status(now), - overload: sources.grpc_admission.overload(), + overloaded: sources.grpc_admission.overloaded(), height: served.or(sources.progress.handed()).map(u32::from), target: grpc_snapshot.sync_target().map(u32::from), rate: *sources.block_rate.borrow(), @@ -253,8 +252,8 @@ fn readiness(hold: Hold, live: bool, stage: Stage) -> Readiness { Readiness::Down(DaemonReason::NoValidatorReachable) } (_, Stage::Starting { bootstrap, .. }) => Readiness::Provisioning(bootstrap), - (true, Stage::Booted(Booted { overload: Some(cause), .. })) => { - Readiness::Down(DaemonReason::Overloaded(cause)) + (true, Stage::Booted(Booted { overloaded: true, .. })) => { + Readiness::Down(DaemonReason::Overloaded) } (true, Stage::Booted(booted)) => match booted.serving_status { ServingStatus::Down => Readiness::Down(DaemonReason::TipUnvouched), @@ -490,20 +489,18 @@ mod tests { fn the_state_is_the_first_matching_rule_with_its_own_detail() { const LIVE: bool = true; const STALLED: bool = false; - let stage = |serving_status, overload, target: Option, rate: Option| { + let stage = |serving_status, overloaded, target: Option, rate: Option| { let (height, egress) = (Some(3_514_194), Some(Egress { start: 1_790_000_400, mbps: 12.5 })); - Stage::Booted(Booted { serving_status, overload, height, target, rate, egress }) + Stage::Booted(Booted { serving_status, overloaded, height, target, rate, egress }) }; - let booted = |serving_status, target, rate| stage(serving_status, None, target, rate); - let saturated = Some(OverloadCause::CpuPressure); - let overloaded = - |serving_status| stage(serving_status, saturated, Some(3_515_406), Some(6.2)); + let booted = |serving_status, target, rate| stage(serving_status, false, target, rate); + let overloaded = |serving_status| stage(serving_status, true, Some(3_515_406), Some(6.2)); let starting = |bootstrap, awaiting_validator| Stage::Starting { bootstrap, awaiting_validator }; let because = |reason: &str, message: &str| serde_json::json!({ "reason": reason, "message": message }); let stalled = because("runtime_stalled", "Runtime stalled"); - let saturated = because("overloaded", "Overloaded: CPU pressure"); + let saturated = because("overloaded", "Overloaded: refusing work"); let unvouched = because("tip_unvouched", "No trusted validator vouches for the tip"); let downloading = SnapshotProgress { phase: SnapshotPhase::Downloading, diff --git a/packages/zingod/src/health/state_log.rs b/packages/zingod/src/health/state_log.rs index bc0a1e9df..151d94d45 100644 --- a/packages/zingod/src/health/state_log.rs +++ b/packages/zingod/src/health/state_log.rs @@ -109,8 +109,6 @@ pub(crate) fn spawn(status: Arc) { #[cfg(test)] mod tests { - use zingod_grpc::OverloadCause; - use super::*; /// One node's day, second by second where it matters: @@ -125,8 +123,7 @@ mod tests { let (secs, mins) = (Duration::from_secs, |m: u64| Duration::from_secs(m * 60)); let changed = |from, after| Some(Line::Changed { from, after }); let not_ready = |elapsed| Some(Line::NotReady { elapsed }); - let io = DaemonReason::Overloaded(OverloadCause::IoPressure); - let cpu = DaemonReason::Overloaded(OverloadCause::CpuPressure); + let overloaded = DaemonReason::Overloaded; let (pending, sent) = (DaemonReason::GoawayPending, DaemonReason::GoawaySent); let unvouched = DaemonReason::TipUnvouched; @@ -140,13 +137,13 @@ mod tests { ("syncing: never repeats", at(1_200), Syncing(None), None), ("serving", at(1_200), Serving(None), changed(Some("syncing"), Some(mins(10)))), ("serving: never repeats", at(4_800), Serving(None), None), - ("overloaded", at(4_800), Down(io), changed(Some("serving"), Some(mins(60)))), - ("down: under REPEAT", at(4_830), Down(io), None), - ("down: REPEAT after the change", at(4_860), Down(io), not_ready(mins(1))), - ("down: under REPEAT since the last", at(4_890), Down(io), None), - ("down: second REPEAT", at(4_920), Down(io), not_ready(mins(2))), - ("down, new reason = a change", at(4_940), Down(cpu), changed(Some("down"), Some(secs(140)))), - ("down: REPEAT after the new reason", at(5_000), Down(cpu), not_ready(secs(60))), + ("overloaded", at(4_800), Down(overloaded), changed(Some("serving"), Some(mins(60)))), + ("down: under REPEAT", at(4_830), Down(overloaded), None), + ("down: REPEAT after the change", at(4_860), Down(overloaded), not_ready(mins(1))), + ("down: under REPEAT since the last", at(4_890), Down(overloaded), None), + ("down: second REPEAT", at(4_920), Down(overloaded), not_ready(mins(2))), + ("down, new reason = a change", at(4_940), Down(unvouched), changed(Some("down"), Some(secs(140)))), + ("down: REPEAT after the new reason", at(5_000), Down(unvouched), not_ready(secs(60))), ("held", at(5_000), Held(pending), changed(Some("down"), Some(mins(1)))), ("held, GOAWAYs sent = a change", at(5_010), Held(sent), changed(Some("held"), Some(secs(10)))), ("held: REPEAT", at(5_070), Held(sent), not_ready(mins(1))), diff --git a/packages/zingod/src/main.rs b/packages/zingod/src/main.rs index e50173a01..cce31b4d9 100644 --- a/packages/zingod/src/main.rs +++ b/packages/zingod/src/main.rs @@ -11,7 +11,7 @@ use clap::Parser; use tracing::{error, info, Instrument as _}; use crate::cli::{Cli, Command}; -use crate::command::{hold, origin, overload, verify}; +use crate::command::{hold, origin, verify}; use crate::config::load_config; use crate::error::IndexerError; use crate::health::{admin, notify, state_log, DaemonStatus}; @@ -41,11 +41,8 @@ fn main() -> ExitCode { let cli = Cli::parse(); // logging → stdout by default (reserved for `verify`'s JSON, `hold`/`resume`'s and - // `overload`'s state) - if !matches!( - cli.command, - Command::Verify | Command::Hold(_) | Command::Overload { .. } | Command::Origin { .. } - ) { + // `origin`'s state) + if !matches!(cli.command, Command::Verify | Command::Hold(_) | Command::Origin { .. }) { if let Err(error) = logging::init() { eprintln!("Error: RUST_LOG: {error}"); return ExitCode::from(2); @@ -74,7 +71,6 @@ fn main() -> ExitCode { } Command::Verify => verify::run(&config), Command::Hold(action) => exit(hold::command(&config, action), 1), - Command::Overload { action } => exit(overload::command(&config, action), 1), Command::Origin { action } => exit(origin::command(&config, action), 1), } }); diff --git a/packages/zingod/usage.md b/packages/zingod/usage.md index 2e8d18fe2..218a1ccce 100644 --- a/packages/zingod/usage.md +++ b/packages/zingod/usage.md @@ -15,7 +15,7 @@ hold, systemd, index snapshots, `verify`, network exposure): - `health/`: the state rules (`readiness`, pure), `/readyz`, `/status`, the admin listener, systemd notify and the state log. - `telemetry/`: logging, the metrics recorder, disk sampling, sync progress, gRPC egress. -- `command/`: `verify`, `hold` / `resume`, `overload`. +- `command/`: `verify`, `hold` / `resume`, `origin`. ## The admin listener @@ -55,4 +55,4 @@ What zingod logs and what to watch: [operators: logging](../../book/src/operator `daemon.rs` tests the task policy (a task ending first cancels the rest and fails the daemon, a signal stops it clean, during boot too) and the whole pipeline over a `MockValidator` and gRPC on localhost (`a_reorg_serves_the_heavier_chain_and_every_rpc_agrees_on_the_served_tip`). The state -rules, `/status`, systemd notify, hold, overload markers and `verify` test beside their code. +rules, `/status`, systemd notify, hold, origin markers and `verify` test beside their code.