feat(alerts): Telegram alerts when a node stops or runs programs

deployments/alerts/README.md

@@ -0,0 +1,59 @@
+# Node alerts
+
+Two small alerts send Telegram messages from each node. Both read the same settings file and run every 5 minutes from a systemd timer.
+
+## tn-pg-alert: Postgres runs a program
+
+`tn-pg-alert` sends a Telegram message when a node's Postgres container logs a statement that runs a server program, such as `COPY ... TO PROGRAM`. kwild's database role may not run programs (see `deployments/postgres-roles.sql`), so Postgres refuses the statement and logs it. Any such line means something tried, and it is worth a look.
+
+Each run reads the container's log lines written since the last run that sent its alert. If a send fails, the next run reads the same lines again, so the alert is retried.
+
+## tn-node-alert: the node stops or crashes
+
+`tn-node-alert` sends a message when:
+
+- the node's block height has not changed for 10 minutes (`STALL_MINUTES`), including when its RPC stops answering. The message repeats every hour (`REPEAT_MINUTES`) while the stall lasts, and one more message says when blocks move again.
+- Docker restarted the kwild container on its own, which means it crashed. A manual restart or a recreate does not count.
+- the kwild container is not running.
+
+## Install on a node
+
+From a checkout of this repository on the node:
+
+```bash
+cd deployments/alerts
+sudo ./install.sh
+```
+
+The script installs `/usr/local/bin/tn-pg-alert` and `/usr/local/bin/tn-node-alert`, their services and timers, and an empty settings file `/etc/tn-alert/telegram.env` (mode 600). Until the token is set, alerts go to the system journal only (`journalctl -t tn-pg-alert -t tn-node-alert`).
+
+## Set the Telegram token
+
+1. Create a bot with [@BotFather](https://t.me/BotFather) and copy its token.
+2. Add the bot to the chat that should receive alerts and find the chat id, for example from `https://api.telegram.org/bot<token>/getUpdates` after posting a message in the chat.
+3. On each node, put both in `/etc/tn-alert/telegram.env`:
+
+   ```bash
+   TELEGRAM_BOT_TOKEN=123456:ABC...
+   TELEGRAM_CHAT_ID=-1001234567890
+   # only for a chat with topics:
+   TELEGRAM_TOPIC_ID=42
+   ```
+
+4. Check that messages arrive: `sudo tn-pg-alert --test` and `sudo tn-node-alert --test`
+
+No restart is needed. The next timer run reads the file.
+
+## Settings
+
+| Variable | Default | Meaning |
+|---|---|---|
+| `TELEGRAM_BOT_TOKEN` | empty | Bot token from @BotFather |
+| `TELEGRAM_CHAT_ID` | empty | Chat that receives the alerts |
+| `TELEGRAM_TOPIC_ID` | empty | Topic in a chat with topics, where the alerts go |
+| `PG_CONTAINER` | `tn-postgres` | Name of the node's Postgres container |
+| `NODE_CONTAINER` | `tn-node` | Name of the node's kwild container |
+| `NODE_RPC` | `http://127.0.0.1:8484/rpc/v1` | The node's JSON-RPC address |
+| `STALL_MINUTES` | `10` | Minutes without a new block before the first alert |
+| `REPEAT_MINUTES` | `60` | Minutes between repeats while a stall lasts |
+| `NODE_LABEL` | the hostname | Node name shown in the message |

deployments/alerts/install.sh

@@ -0,0 +1,23 @@
+#!/usr/bin/env bash
+# Installs tn-pg-alert and tn-node-alert on a node. Run it as root from this directory:
+#   sudo ./install.sh
+# It is safe to run again and keeps an existing /etc/tn-alert/telegram.env.
+set -euo pipefail
+cd "$(dirname "$0")"
+
+install -m 755 pg-program-alert.sh /usr/local/bin/tn-pg-alert
+install -m 755 node-health-alert.sh /usr/local/bin/tn-node-alert
+install -m 644 tn-pg-alert.service tn-pg-alert.timer tn-node-alert.service tn-node-alert.timer /etc/systemd/system/
+install -d -m 700 -o root -g root /etc/tn-alert
+chown root:root /etc/tn-alert
+chmod 700 /etc/tn-alert
+if [ -e /etc/tn-alert/telegram.env ]; then
+	chown root:root /etc/tn-alert/telegram.env
+	chmod 600 /etc/tn-alert/telegram.env
+else
+	install -m 600 -o root -g root /dev/null /etc/tn-alert/telegram.env
+	printf '%s\n' 'TELEGRAM_BOT_TOKEN=' 'TELEGRAM_CHAT_ID=' '# TELEGRAM_TOPIC_ID=' '# PG_CONTAINER=tn-postgres' '# NODE_LABEL=' >/etc/tn-alert/telegram.env
+fi
+systemctl daemon-reload
+systemctl enable --now tn-pg-alert.timer tn-node-alert.timer
+echo "tn-pg-alert and tn-node-alert installed. Put the bot token and chat id in /etc/tn-alert/telegram.env, then run: sudo tn-pg-alert --test && sudo tn-node-alert --test"

deployments/alerts/node-health-alert.sh

@@ -0,0 +1,135 @@
+#!/usr/bin/env bash
+# Sends a Telegram message when this node stops moving: its block height has
+# not changed for STALL_MINUTES, or Docker restarted its kwild container on its
+# own (a crash), or the container is not running. A stall alert repeats every
+# REPEAT_MINUTES while it lasts, and one message says when blocks move again.
+#
+# Settings come from /etc/tn-alert/telegram.env, shared with tn-pg-alert:
+#   TELEGRAM_BOT_TOKEN, TELEGRAM_CHAT_ID  where the message goes
+#   TELEGRAM_TOPIC_ID                     topic (thread) in a forum chat, optional
+#   NODE_CONTAINER                        kwild container, default tn-node
+#   NODE_RPC                              JSON-RPC URL, default http://127.0.0.1:8484/rpc/v1
+#   STALL_MINUTES                         default 10
+#   REPEAT_MINUTES                        default 60
+#   NODE_LABEL                            node name in the message, default the hostname
+# Without a token and a chat id, the alert goes to the system journal only.
+#
+# Usage: tn-node-alert          check the node
+#        tn-node-alert --test   send a test message
+set -euo pipefail
+
+ENV_FILE=${TN_ALERT_ENV:-/etc/tn-alert/telegram.env}
+STATE_DIR=${TN_ALERT_STATE:-/var/lib/tn-alert}
+if [ -r "$ENV_FILE" ]; then
+	# shellcheck disable=SC1090
+	. "$ENV_FILE"
+fi
+NODE_CONTAINER=${NODE_CONTAINER:-tn-node}
+NODE_RPC=${NODE_RPC:-http://127.0.0.1:8484/rpc/v1}
+STALL_MINUTES=${STALL_MINUTES:-10}
+REPEAT_MINUTES=${REPEAT_MINUTES:-60}
+NODE_LABEL=${NODE_LABEL:-$(hostname)}
+
+send() {
+	local text=$1 code topic=() logged=0
+	logger -t tn-node-alert -- "$text" || logged=1
+	if [ -z "${TELEGRAM_BOT_TOKEN:-}" ] || [ -z "${TELEGRAM_CHAT_ID:-}" ]; then
+		echo "tn-node-alert: no Telegram token or chat id set, wrote the alert to the journal only" >&2
+		return "$logged"
+	fi
+	if [ -n "${TELEGRAM_TOPIC_ID:-}" ]; then
+		topic=(--data-urlencode "message_thread_id=${TELEGRAM_TOPIC_ID}")
+	fi
+	# The URL holds the token, so it goes through stdin, not the command line.
+	code=$(printf 'url = "https://api.telegram.org/bot%s/sendMessage"\n' "$TELEGRAM_BOT_TOKEN" |
+		curl -sS -m 20 -o /dev/null -w '%{http_code}' -K - \
+			--data-urlencode "chat_id=${TELEGRAM_CHAT_ID}" ${topic[@]+"${topic[@]}"} \
+			--data-urlencode "text=${text}") || code=000
+	if [ "$code" != 200 ]; then
+		echo "tn-node-alert: Telegram answered HTTP $code" >&2
+		return 1
+	fi
+}
+
+if [ "${1:-}" = "--test" ]; then
+	send "${NODE_LABEL}: test message from tn-node-alert"
+	exit
+fi
+
+mkdir -p "$STATE_DIR"
+state="$STATE_DIR/node-state"
+now=$(date -u +%s)
+
+# Saved state: height, when it last changed, restart count, when the last stall
+# alert went out (0 when none is open), and the minutes of a stall whose
+# recovery message has not gone out yet (0 when none).
+last_height=0 changed_at=$now last_restarts=-1 alerted_at=0 recovery=0
+if [ -r "$state" ]; then
+	read -r last_height changed_at last_restarts alerted_at recovery <"$state" || true
+fi
+for v in last_height changed_at last_restarts alerted_at recovery; do
+	[[ ${!v} =~ ^-?[0-9]+$ ]] || printf -v "$v" '%s' 0
+done
+[ "$changed_at" -gt 0 ] || changed_at=$now
+
+failed=0
+
+# Crashes: Docker counts only the restarts its restart policy made, so a
+# manual restart or a recreate does not alert. A recreate resets the count.
+if info=$(docker inspect -f '{{.RestartCount}} {{.State.Status}} {{.State.ExitCode}}' "$NODE_CONTAINER" 2>/dev/null); then
+	read -r restarts status exit_code <<<"$info"
+	if [ "$status" != running ]; then
+		send "${NODE_LABEL}: ${NODE_CONTAINER} is not running (status ${status}, last exit code ${exit_code})." || failed=1
+	elif [ "$last_restarts" -ge 0 ] && [ "$restarts" -gt "$last_restarts" ]; then
+		send "${NODE_LABEL}: ${NODE_CONTAINER} restarted by itself $((restarts - last_restarts)) time(s) since the last check (${restarts} in total). Check: docker logs --tail 50 ${NODE_CONTAINER}" || failed=1
+	fi
+else
+	restarts=$last_restarts
+	send "${NODE_LABEL}: cannot inspect the container ${NODE_CONTAINER}." || failed=1
+fi
+
+# Height: a node that does not answer keeps its last height, so it counts as
+# stalled too.
+height=$(curl -s -m 10 -X POST "$NODE_RPC" -H 'content-type: application/json' \
+	-d '{"jsonrpc":"2.0","id":1,"method":"user.health","params":{}}' 2>/dev/null |
+	grep -o '"block_height"[[:space:]]*:[[:space:]]*[0-9]*' | head -1 | grep -o '[0-9]*$') || true
+answering=yes
+if [ -z "$height" ]; then
+	answering=no
+	height=$last_height
+fi
+
+if [ "$height" -gt "$last_height" ]; then
+	if [ "$alerted_at" -gt 0 ]; then
+		recovery=$(((now - changed_at) / 60))
+		[ "$recovery" -gt 0 ] || recovery=1
+	fi
+	changed_at=$now
+	alerted_at=0
+else
+	stalled=$(((now - changed_at) / 60))
+	if [ "$stalled" -ge "$STALL_MINUTES" ] &&
+		{ [ "$alerted_at" -eq 0 ] || [ $(((now - alerted_at) / 60)) -ge "$REPEAT_MINUTES" ]; }; then
+		why=""
+		[ "$answering" = no ] && why=" The node RPC is not answering."
+		if send "${NODE_LABEL}: no new block for ${stalled} min, still at height ${height}.${why}"; then
+			alerted_at=$now
+		else
+			failed=1
+		fi
+	fi
+fi
+
+# A recovery message that fails is kept and sent again on the next run.
+if [ "$recovery" -gt 0 ]; then
+	if send "${NODE_LABEL}: blocks are moving again, height ${height}, after ${recovery} min without a new block."; then
+		recovery=0
+	else
+		failed=1
+	fi
+fi
+
+printf '%s %s %s %s %s\n' "$height" "$changed_at" "${restarts:--1}" "$alerted_at" "$recovery" >"$state.tmp"
+sync "$state.tmp" 2>/dev/null || true
+mv -f "$state.tmp" "$state"
+exit "$failed"

deployments/alerts/pg-program-alert.sh

@@ -0,0 +1,79 @@
+#!/usr/bin/env bash
+# Sends a Telegram message when the node's Postgres container logs a statement
+# that runs a server program (COPY ... TO PROGRAM or FROM PROGRAM). kwild's
+# database role may not run programs, so Postgres refuses such a statement and
+# logs it. Any such line means something tried.
+#
+# Settings come from /etc/tn-alert/telegram.env (see README.md):
+#   TELEGRAM_BOT_TOKEN, TELEGRAM_CHAT_ID  where the message goes
+#   TELEGRAM_TOPIC_ID                     topic (thread) in a forum chat, optional
+#   PG_CONTAINER                          Postgres container, default tn-postgres
+#   NODE_LABEL                            node name in the message, default the hostname
+# Without a token and a chat id, the alert goes to the system journal only.
+#
+# Usage: tn-pg-alert          check the logs written since the last run
+#        tn-pg-alert --test   send a test message
+set -euo pipefail
+
+ENV_FILE=${TN_ALERT_ENV:-/etc/tn-alert/telegram.env}
+STATE_DIR=${TN_ALERT_STATE:-/var/lib/tn-alert}
+if [ -r "$ENV_FILE" ]; then
+	# shellcheck disable=SC1090
+	. "$ENV_FILE"
+fi
+PG_CONTAINER=${PG_CONTAINER:-tn-postgres}
+NODE_LABEL=${NODE_LABEL:-$(hostname)}
+
+send() {
+	local text=$1 code topic=() logged=0
+	logger -t tn-pg-alert -- "$text" || logged=1
+	if [ -z "${TELEGRAM_BOT_TOKEN:-}" ] || [ -z "${TELEGRAM_CHAT_ID:-}" ]; then
+		echo "tn-pg-alert: no Telegram token or chat id set, wrote the alert to the journal only" >&2
+		return "$logged"
+	fi
+	if [ -n "${TELEGRAM_TOPIC_ID:-}" ]; then
+		topic=(--data-urlencode "message_thread_id=${TELEGRAM_TOPIC_ID}")
+	fi
+	# The URL holds the token, so it goes through stdin, not the command line.
+	code=$(printf 'url = "https://api.telegram.org/bot%s/sendMessage"\n' "$TELEGRAM_BOT_TOKEN" |
+		curl -sS -m 20 -o /dev/null -w '%{http_code}' -K - \
+			--data-urlencode "chat_id=${TELEGRAM_CHAT_ID}" ${topic[@]+"${topic[@]}"} \
+			--data-urlencode "text=${text}") || code=000
+	if [ "$code" != 200 ]; then
+		echo "tn-pg-alert: Telegram answered HTTP $code" >&2
+		return 1
+	fi
+}
+
+if [ "${1:-}" = "--test" ]; then
+	send "${NODE_LABEL}: test message from tn-pg-alert"
+	exit
+fi
+
+mkdir -p "$STATE_DIR"
+now=$(date -u +%Y-%m-%dT%H:%M:%SZ)
+since=$(cat "$STATE_DIR/last-run" 2>/dev/null || true)
+if ! [[ $since =~ ^[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}Z$ ]]; then
+	since=$(date -u -d '-10 min' +%Y-%m-%dT%H:%M:%SZ)
+fi
+
+# A statement can be split over several log lines, so the scan also matches the
+# fixed message Postgres logs when it refuses one. Only matching lines are kept.
+pattern='(to|from)[[:space:]]+program|external program|pg_execute_server_program'
+if ! matches=$(
+	docker logs --since "$since" --until "$now" "$PG_CONTAINER" 2>&1 | grep -i -E "$pattern"
+	exit "${PIPESTATUS[0]}"
+); then
+	send "${NODE_LABEL}: tn-pg-alert cannot read the logs of ${PG_CONTAINER}"
+	exit 1
+fi
+
+count=$(printf '%s' "$matches" | grep -c . || true)
+if [ "$count" -gt 0 ]; then
+	# The last run time only moves once the message is out, so a failed send
+	# is retried on the next run.
+	send "${NODE_LABEL}: Postgres logged ${count} line(s) about running a server program between ${since} and ${now}. Read them with: docker logs --since ${since} --until ${now} ${PG_CONTAINER} 2>&1 | grep -i program"
+fi
+printf '%s\n' "$now" >"$STATE_DIR/last-run.tmp"
+sync "$STATE_DIR/last-run.tmp" 2>/dev/null || true
+mv -f "$STATE_DIR/last-run.tmp" "$STATE_DIR/last-run"

deployments/alerts/tn-node-alert.service

@@ -0,0 +1,10 @@
+[Unit]
+Description=Alert when the node stops making blocks or crashes
+After=docker.service
+Wants=docker.service
+
+[Service]
+Type=oneshot
+ExecStart=/usr/local/bin/tn-node-alert
+# A stuck run would block every later one, so it gets a time limit.
+TimeoutStartSec=4min

deployments/alerts/tn-node-alert.timer

@@ -0,0 +1,9 @@
+[Unit]
+Description=Run tn-node-alert every 5 minutes
+
+[Timer]
+OnCalendar=*:0/5
+Persistent=true
+
+[Install]
+WantedBy=timers.target

deployments/alerts/tn-pg-alert.service

@@ -0,0 +1,10 @@
+[Unit]
+Description=Alert when Postgres logs a statement that runs a server program
+After=docker.service
+Wants=docker.service
+
+[Service]
+Type=oneshot
+ExecStart=/usr/local/bin/tn-pg-alert
+# A stuck run would block every later one, so it gets a time limit.
+TimeoutStartSec=4min

deployments/alerts/tn-pg-alert.timer

@@ -0,0 +1,9 @@
+[Unit]
+Description=Run tn-pg-alert every 5 minutes
+
+[Timer]
+OnCalendar=*:0/5
+Persistent=true
+
+[Install]
+WantedBy=timers.target