Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 24 additions & 0 deletions .changeset/admission-overload.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
---
default: minor
---

# zingod-grpc, zingod: one adaptive work limit governs load; fixed caps, pressure signals and overrides removed

The adaptive work-stream limit is now the single authority on load. It always adapts, with
`max_work_streams` as its ceiling; disk reads run inside it with no cap of their own, and
`GetMempoolStream` takes no slot (the connection caps bound it). `/readyz` reports `down` with
reason `overloaded` once at least 10% of work requests are refused (at one or more a second) for
5 minutes, and clears after 5 minutes under 2%. CPU and IO saturation reach it as latency, which
lowers the limit, so the Linux PSI signals are gone. A stream's only time bound is its client's
`grpc-timeout` (and `unread_timeout_secs` for a client that stops reading).

- Remove from your config, or zingod refuses to start: `[grpc.overload]`, and under `[grpc]`
`adaptive_work_streams`, `max_subscriptions`, `max_point_reads`, `max_range_reads`,
`max_scan_reads`, `silent_timeout_secs`
- `zingod overload` is removed; delete `<state_dir>/overload/` if it exists
- Metrics: `zingod_grpc_overload_latched{cause}` is now `zingod_grpc_overloaded` (no labels);
`zingod_grpc_active_subscriptions`, `zingod_grpc_subscriptions_rejected_total` and
`zingod_grpc_disk_read_wait_seconds` are gone
- `/readyz` message is `Overloaded: refusing work`
- The `Serving` log line carries `rps`, `conns`, `refused` and `failed` only; `Request
unavailable` is no longer logged
3 changes: 2 additions & 1 deletion book/src/SUMMARY.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,9 +14,10 @@
- [Quick start](operators/quick-start.md)
- [Configuration](operators/configuration.md)
- [First sync](operators/first-sync.md)
- [Network exposure](operators/network.md)
- [DoS protection](operators/dos-protection.md)
- [Monitoring](operators/monitoring.md)
- [Maintenance](operators/maintenance.md)
- [Network exposure](operators/network.md)
- [Docker](operators/docker.md)
- [Migrating from Zaino](operators/migrating-from-zaino.md)

Expand Down
2 changes: 1 addition & 1 deletion book/src/operators/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ every enabled index, and `p2p.cache_dir` if `[p2p]` is enabled.
| `[submission]` | how `SendTransaction` retries relaying |
| `[p2p]` | Zingod's own Zcash p2p peers (off by default) |
| `[serve]` | gRPC listen address, TLS, gRPC-web, state directory, drain delay |
| `[grpc]` | connection and stream limits, timeouts, trusted proxies, overload thresholds |
| `[grpc]` | [DoS protection](dos-protection.md) limits and timeouts, trusted proxies, shutdown grace |
| `[metrics]` | admin listener for metrics and probes (off by default) |
| `[sync]` | block fetching and finality depth |
| `[index.<name>]` | per index: `enabled` and `path` |
Expand Down
128 changes: 128 additions & 0 deletions book/src/operators/dos-protection.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,128 @@
# DoS protection

Zingod protects itself with a fixed set of limits, checked in the order a request meets them. A
request over a limit is refused immediately, never queued, so an overloaded node stays responsive
and clients know at once to back off or try another server.

| Layer | Bounds | Settings (`[grpc]`) | When exceeded |
| ------------- | ------------------------------------------ | -------------------------------------------------------- | ----------------------------------- |
| Connections | open sockets, in total and per client | `max_connections`, `max_connections_per_ip` | connection closed at accept |
| Streams | concurrent requests on one connection | `max_streams_per_connection` | HTTP/2 refuses the stream |
| Work | requests being served across all clients | `max_work_streams` (ceiling of the adaptive limit) | `UNAVAILABLE` with a retry hint |
| Request size | request body and address rows | `[serve] max_address_rows` | `RESOURCE_EXHAUSTED` |
| Progress | clients that stop reading | `unread_timeout_secs` | connection closed |
| Origins | browser sites you block | `zingod origin block` | `PERMISSION_DENIED` |

The defaults suit a public server on one machine. Every value is in the
[configuration reference](configuration.md#reference).

## Connections

- `max_connections` (4096) caps open connections in total. Each one uses a file descriptor:
Zingod raises its soft `RLIMIT_NOFILE` to the hard limit at startup and refuses to start if
`max_connections` plus 1024 does not fit. Raise the hard limit (`LimitNOFILE=` in systemd,
`--ulimit nofile=` in Docker) or lower `max_connections`.
- `max_connections_per_ip` (128) caps connections from one client address. It is high because
mobile carriers put many wallets behind one address (CGNAT). Behind a proxy, every connection
comes from the proxy's address unless it sends a PROXY header; see
[Behind a proxy](network.md#behind-a-proxy).
- `max_streams_per_connection` (100) caps concurrent requests on one connection, matching
lightwalletd. A zingolib wallet uses up to 7.

Idle and long-lived connections are recycled: a connection with no open stream for
`max_connection_idle_secs` (30 minutes) gets a GOAWAY, which frees its per-client slot. Every
connection also gets one after `max_connection_age_secs` (30 minutes, ±10%), and its in-flight
streams get `max_connection_age_grace_secs` (10 minutes) to finish.

## Work streams

Every method except `GetMempoolStream` takes a work slot for as long as it runs, including a long
`GetBlockRange`. `GetMempoolStream` stays open until the next block and is idle most of that time,
so it takes none; the connection limits bound it.

The number of slots is not fixed. Zingod measures how long it takes to produce each response
message and adjusts the limit from that, using Netflix's Gradient2 algorithm:

- While the node is busy and response times stay steady, the limit grows slowly.
- When response times rise past 1.5 times their 5-minute average, the limit shrinks within
seconds: to under half in about 5 seconds at 4 times the usual response time.
- While the node is quiet (fewer than half the slots in use) the limit does not move.
- It starts at 256, never drops below 32, and never exceeds `max_work_streams` (2048).
- Time spent waiting on a slow client does not count. Only time Zingod itself spends counts.

CPU or disk saturation shows up as slower responses, so the limit tracks what the machine can
actually serve. Disk reads have no separate limit: each runs inside a work stream, so a slow disk
lowers the work limit. `max_work_streams` is a safety ceiling for costs that do not show up as
latency, such as memory. You rarely need to change it.

A request over the limit gets `UNAVAILABLE` (`work stream limit reached`) with a
`grpc-retry-pushback-ms: 250` header. Wallets retry after that delay, or move to another server.

The work limit is shared by all clients and filled first come, first served. A single address can
hold up to `max_connections_per_ip × max_streams_per_connection` streams (12,800 by default),
more than the whole work limit, so one aggressive client can take every slot until its streams
finish. On a public server, rate-limit per client address at your proxy or edge, or lower those
two settings if your clients do not share addresses.

## Request size

- A request body over 64 KiB is refused with `RESOURCE_EXHAUSTED`. `SendTransaction` allows
2 MB + 1 KiB, the largest valid transaction.
- An address request that would walk more than `[serve] max_address_rows` (100,000) received
transparent outputs, across all its addresses, is refused with `RESOURCE_EXHAUSTED`.

## Slow clients

Streams have no total time limit; a full sync takes as long as it takes. A client's own
`grpc-timeout` is honoured as its deadline. A client that stops reading data Zingod has sent for
`unread_timeout_secs` (300) has its whole connection closed.

`/readyz` and `/status` on the gRPC listener never take a work slot. Each connection may send 10
at once, refilling at one per second; beyond that it gets `429`.

## Browser origins

With [gRPC-web](network.md#browser-clients-grpc-web) enabled, any website can make its visitors'
browsers call your server. `zingod origin` refuses traffic from sites you choose. Changes apply
within a second and persist until changed:

| Command | Effect |
| ------------------------------- | -------------------------------------------------------------- |
| `zingod origin block <domain>` | refuse requests from the origin with `PERMISSION_DENIED` |
| `zingod origin allow <domain>` | serve the origin even when a parent domain is blocked |
| `zingod origin reset <domain>` | remove the origin's policy |
| `zingod origin` | list the policies in effect |

- Policies apply to the registrable domain and its subdomains. The most specific policy wins, so
`allow app.example.com` can override `block example.com`.
- Every origin that is not blocked meets the same limits as any other client. Clients can set any
`Origin` header, so an origin never lifts a limit.
- `zingod_grpc_origin_requests_total{verdict}` counts browser requests by policy. To find an
abusive origin, check your proxy or edge logs.

## Overload

The work limit decides when a node is overloaded. If at least 10% of work requests are refused,
at one or more refusals a second, for 5 minutes, `/readyz` reports `down` with reason
`overloaded` and the log shows `Overload latched` (WARN). After 5 minutes with fewer than 2%
refused, it clears (`Overload cleared`, INFO).

An overloaded node keeps serving up to its limit. The `down` state tells a load balancer to send
new clients elsewhere until the node recovers. There is nothing to tune: if a node is often
overloaded, add capacity or spread clients across more nodes.

## What to watch

| Metric | Meaning |
| ---------------------------------------------- | ------------------------------------------------------- |
| `zingod_grpc_adaptive_stream_limit` | the work limit now |
| `zingod_grpc_active_streams` | work streams being served |
| `zingod_grpc_admission_rejected_total` | work requests refused at the limit |
| `zingod_grpc_connections_rejected_total` | connections refused at a connection limit |
| `zingod_grpc_stalled_connections_total` | connections closed by `unread_timeout_secs` |
| `zingod_grpc_overloaded` | 1 while the node reports `overloaded` |

A work limit that sits well below `max_work_streams` while refusals climb means the machine is the
bottleneck. Refusals with the limit at its ceiling mean `max_work_streams` itself is the
bottleneck. The once-a-minute [`Serving` line](monitoring.md#the-serving-line) summarises
refusals in the log.
24 changes: 0 additions & 24 deletions book/src/operators/maintenance.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,30 +29,6 @@ so dashboards can tell planned maintenance from a crash.

Both commands are idempotent.

## Overload

A node that is persistently short of capacity reports `down` with reason `overloaded`, so load
balancers move traffic away from it. Thresholds are set under `[grpc.overload]` (0 disables a
cause). A cause is raised after its signal stays above the threshold for the enter time, and
cleared after it stays below half the threshold for the exit time.

| Cause | Signal, per minute | Enter / exit |
| ---------------------------- | ------------------------------------------------------------- | ------------ |
| `CPU pressure`, `IO pressure` | share of time Zingod's cgroup waited (Linux PSI) | 2 / 3 min |
| `refusing work` | at least 10% of requests refused, at 1/s or more (exit: < 2%) | 5 / 5 min |

Override the detector on a running node, without a restart:

```sh
zingod overload off # never report overloaded
zingod overload on # back to automatic
zingod overload set <key> <value> # override a threshold, e.g. cpu_pressure_percent 40
zingod overload unset <key> # back to the config's value
```

Overrides are stored in `state_dir` and persist across restarts. Zingod applies them within a
second and logs a warning while any is in effect.

## systemd

```ini
Expand Down
17 changes: 6 additions & 11 deletions book/src/operators/monitoring.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ Zingod reports a single state. The same value appears in `/readyz`, `/status`, t
| State | Meaning | `/readyz` |
| -------------- | ----------------------------------------------------------------------------------------- | --------- |
| `held` | an operator ran [`zingod hold`](maintenance.md#operator-hold) | `503` |
| `down` | cannot serve: the server is stalled or [overloaded](maintenance.md#overload), startup is waiting for a validator, or the tip is unverified and stale | `503` |
| `down` | cannot serve: the server is stalled or [overloaded](dos-protection.md#overload), startup is waiting for a validator, or the tip is unverified and stale | `503` |
| `provisioning` | starting up, or installing an [index snapshot](first-sync.md#index-snapshot) | `503` |
| `degraded` | serving a recent tip that no trusted validator has confirmed (p2p peers supplied it) | `200` |
| `syncing` | serving, but behind the best block; or no validator's mempool is available yet | `503` |
Expand Down Expand Up @@ -107,13 +107,8 @@ ts=… level=info target=zingod::telemetry::progress message="Syncing blocks" he

### The `Serving` line

Once a minute, the gRPC server logs a `Serving` summary: `rps` (streams completed per second) and
`conns` (open connections). It is logged at WARN if, during that minute, any request failed or
was refused, took more than a second to its first response (`slow`), stalled (`stalled`), or a
connection was refused at a limit (`conns_refused`). These fields appear only when non-zero.

- `INTERNAL`, `UNKNOWN` and `DATA_LOSS` count as failures. `UNAVAILABLE` and
`RESOURCE_EXHAUSTED` count as refusals. Other codes are client errors, counted in
`zingod_grpc_requests_total{code}`.
- The first failure of each minute is also logged as `Request failed` (ERROR), and the first
refusal not caused by capacity as `Request unavailable` (WARN).
Once a minute, the gRPC server logs a `Serving` summary: `rps` (requests completed per second)
and `conns` (open connections). If any request was refused (`UNAVAILABLE`, `RESOURCE_EXHAUSTED`)
or failed (`INTERNAL`, `UNKNOWN`, `DATA_LOSS`) that minute, it is logged at WARN with `refused`
and `failed` counts. The first failure of each minute is also logged as `Request failed` (ERROR)
with its message. Everything else is in the [metrics](#metrics).
39 changes: 4 additions & 35 deletions book/src/operators/network.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,8 @@ Zingod has two listeners:

Connections to validators are plain HTTP JSON-RPC.

Limits on connections, streams and reads are set under `[grpc]`; see the
[reference](configuration.md#reference). When a limit is reached, the request is refused, not
queued.
Limits on connections, requests and reads are set under `[grpc]`; see
[DoS protection](dos-protection.md).

## TLS

Expand Down Expand Up @@ -46,36 +45,12 @@ after changing the listen address, delete `cert.pem`.
allowing any origin. Native gRPC is unaffected.

Enabling it makes the server usable from any website. A popular site embedding it can send load
proportional to its own traffic.

`zingod origin` manages per-origin policy. Changes apply within a second and persist until
changed:

| Command | Effect |
| ------------------------------- | -------------------------------------------------------------- |
| `zingod origin block <domain>` | refuse requests from the origin with `PERMISSION_DENIED` |
| `zingod origin allow <domain>` | serve the origin even when a parent domain is blocked |
| `zingod origin reset <domain>` | remove the origin's policy |
| `zingod origin` | list the policies in effect |

- Policies apply to the registrable domain and its subdomains. The most specific policy wins, so
`allow app.example.com` can override `block example.com`.
- Every origin that is not blocked meets the same limits as any other client. Clients can set any
`Origin` header, so an origin never lifts a limit.
- `zingod_grpc_origin_requests_total{verdict}` counts browser requests by policy. To find an
abusive origin, check your proxy or edge logs.
proportional to its own traffic. To refuse a site, see [Browser
origins](dos-protection.md#browser-origins).

Over HTTP/1.1, browsers open one connection per in-flight stream, each counting against
`max_connections_per_ip`. Serve over TLS so browsers use HTTP/2.

## Stream timeouts

Streams have no total time limit. Two settings close streams that stop making progress:

- `unread_timeout_secs`: the client stopped reading. The connection is closed.
- `silent_timeout_secs`: the server has had nothing to send. The stream ends with `UNAVAILABLE`.
`GetMempoolStream` is exempt.

## Behind a proxy

Behind a proxy, every connection comes from the proxy's address, so `max_connections_per_ip` would
Expand Down Expand Up @@ -114,9 +89,3 @@ For HAProxy, use `server zingod 127.0.0.1:8137 send-proxy-v2` in a `mode tcp` ba

If your proxy cannot send the header, leave `trusted_proxies` empty and raise
`max_connections_per_ip` to `max_connections`.

## Open files

Each connection uses a file descriptor. At startup, Zingod raises its soft `RLIMIT_NOFILE` to the
hard limit and refuses to start if `max_connections` plus 1024 does not fit. Raise the hard limit
(`LimitNOFILE=` in systemd, `--ulimit nofile=` in Docker) or lower `max_connections`.
Loading
Loading