Services deploy

Self-hosted MySQL and PostgreSQL with replicas, point-in-time backups, VPN, HTTPS gateway and monitoring — one repository, one .env, one docker compose up -d.

                           Internet
                  ┌───────────┴───────────┐
              UDP 51820              TCP 80 / 443
            ┌─────▼─────┐           ┌─────▼─────┐
            │    VPN    │           │   HTTP    │
            │ WireGuard │           │   nginx   │
            └─────┬─────┘           └─────┬─────┘
                  │ admins                │ public apps
 ─────────────────┴──── internal network ─┴─────────────────────────
  mysql-primary     ── replication ──►  mysql-replica     (read only)
  postgres-primary  ── streaming ────►  postgres-replica  (read only)
  backups                         PMM · Dozzle · Portainer · admin panels

✨ Key features

One file per server. Every setting lives in .env; COMPOSE_PROFILES decides what a server runs. All servers share the same committed files.
Replicas out of the box. Both databases clone their primary on first start and replicate continuously (GTID for MySQL, streaming for PostgreSQL).
Point-in-time recovery. XtraBackup full + incremental and pgBackRest full + differential, with binary logs / WAL. Restore any moment of the last 14 days into a temporary server.
Backups where you want them. Take them on the primary's server, the replica's server, or both, with no load on the primary when the replica does it.
Private by default. Only the VPN and the HTTP gateway face the internet; everything else is reached through WireGuard.
Monitoring and alerts provisioned. PMM dashboards, Query Analytics, and alert rules (databases down, replication, WAL archiving, disk, CPU, memory) created on startup.
Storage you choose. Each service keeps its data in a Docker volume or in any folder you set.

🧱 Tech stack

Layer	Technology	Role
Orchestration	Docker Compose v2.24+ (`include`, profiles)	One project, one network, per-server roles
Databases	Percona Server for MySQL 8.4, Percona Distribution for PostgreSQL 17	Primaries and read-only replicas
Backups	Percona XtraBackup 8.4, pgBackRest 2.59, supercronic	Physical backups, log archiving, schedules
Gateway	WireGuard (linuxserver/wireguard), nginx	Private access, TLS termination for apps
Monitoring	PMM 3 (Grafana, VictoriaMetrics, ClickHouse), cAdvisor, Dozzle, Portainer	Metrics, alerts, logs, container management
Admin	phpMyAdmin, pgAdmin 4	Database UIs through the VPN
Apps	Java (own JRE), Node.js 24 (Angular SSR, Express)	Example application containers
Scripting	Bash	Entrypoints, backups, restores

🚀 Quick start

Prerequisites

Docker Engine with Docker Compose v2.24 or later
Linux server (Ubuntu, Alpine); Docker Desktop works for development
2+ CPU cores and 4+ GB RAM (PMM Server)

git clone <repository> services-deploy
cd services-deploy
cp .env.example .env          # every setting, documented; edit passwords and VPN_HOST
docker compose config --quiet # validate the configuration

▶️ Basic usage

docker compose up -d
docker compose ps

Versioned deployment

Deployments can follow immutable Git tags while keeping apps/, .env, the Compose override, and active nginx configuration local to the deployment. The scripts use the latest vMAJOR.MINOR.PATCH tag and never run git clean.

The deployment directory must be a Git clone. Do not use GitHub's Code > Download ZIP option for this workflow because a ZIP does not include the .git metadata required to fetch tags and switch versions.

Clone the repository instead:

git clone https://github.com/projectocolibri/services-deploy.git
cd services-deploy

For a private repository, authenticate GitHub before cloning. SSH is usually the best option for servers:

git clone git@github.com:projectocolibri/services-deploy.git

.version       local deployed tag, ignored by Git
deploy.ps1     Windows deployment commands
deploy.sh      Linux deployment commands

Commands:

./deploy.sh check
./deploy.sh update
./deploy.sh run

.\deploy.ps1 check
.\deploy.ps1 update
.\deploy.ps1 run

check reports the latest tag without changing files. update switches tracked infrastructure files to the latest tag and validates Compose, but does not start containers. run checks for a newer tag, asks for confirmation before updating, validates the configuration, and runs docker compose up -d. Ignored deployment files are preserved. Do not run git clean -fdx because it can remove application source, secrets, and local Compose configuration.

The replicas clone their primary on the first start: allow a minute (the MySQL replica restarts once after cloning). Before going to production, replace every default password in .env, set VPN_HOST and go through the production checklist.

🏗️ 1. Technical architecture

One project, many roles

The root compose.yaml includes the infrastructure stacks (gateway/, monitoring/, and databases/) and declares the shared network. Applications are deployment-specific and are added through a local or private Compose override.
Every service has its own Compose profile. COMPOSE_PROFILES in .env lists the services of this server, and docker compose up -d starts exactly those. All servers run the same committed files; only .env differs.
No service depends on another stack. Each one waits for what it needs by itself (entrypoint wrappers poll their dependencies), so any subset of profiles works.

Network and access

See the diagram at the top.

Only the VPN (51820/udp) and the HTTP gateway (80, 443) are published. Everything else is internal and reached through the VPN by service name (http://pmm-server:8080).
Across servers, a replica server joins the VPN host as a WireGuard peer. The VPN host container has a fixed address (VPN_DNS_IP) whose DNS answers with the VPN host's service names. Replicas and backups use it as their DNS server: Docker DNS answers local names first and forwards the rest (e.g. mysql-primary) through the tunnel.

Data flow

 primary ──replication──► replica
    │                        │
    │ binlog / WAL           │ binlog / WAL (the replica's own)
    ▼                        ▼
 *-backup-primary        *-backup-replica        full + incremental/differential
 (primary's server)      (replica's server)      + continuous logs, 14 days
    │                        │
 every server ──► pmm-client + cAdvisor ──► PMM Server ──► alert rules ──► email / Slack

Runtime lifecycle

Primaries start through entrypoint wrappers that create application users and databases on every start (*_DATABASES), and install what backups need (MySQL page tracking).
Replicas clone their primary on the first start (CLONE INSTANCE / pg_basebackup), then replicate.
Backup containers wait for their local server, take a full backup if none exists, then run their schedule with supercronic. PostgreSQL archives WAL itself (archive_command → pgBackRest); MySQL binary logs are streamed by mysqlbinlog.
PMM Server starts, then its entrypoint creates the Slack contact point (if enabled) and every missing alert rule of monitoring/pmm/alerting/rules/.
pmm-client registers the node, cAdvisor and, with MONITOR_DATABASES=true, the running databases.

⚙️ 2. Environment configuration

2.1 The `.env` file

.env at the root is the only settings file. Compose loads it automatically and passes it to every stack. It is not committed: each server has its own.
.env.example is the committed template: development defaults and a comment for every variable.
COMPOSE_ENV_FILES must not be set in the shell, or Compose reads the files it names instead of .env.
When .env.example gets a new variable, copy it into each server's .env. Compare them with diff .env.example .env.
TZ: UTC is recommended; PMM, Grafana and Dozzle show browser time anyway. Images without time zone data (pmm-client, cAdvisor, Dozzle, the MySQL backups) stay on UTC; PostgreSQL keeps the zone it was initialized with.

Section of `.env.example`	Contents
1. Project	`PROJECT_NAME`: prefix of containers and volumes. Renaming it orphans existing volumes.
2. Profiles	`COMPOSE_PROFILES`: what this server runs
3. Network	`NETWORK_NAME`, `NETWORK_SUBNET` (also routed through the VPN), `VPN_DNS_IP`
4. Services	`TZ` and storage, then one block per infrastructure service: VPN host, VPN client, HTTP, PMM Server, PMM client, Dozzle, Portainer, MySQL, and PostgreSQL

2.2 Values to set for production

Service	Variables
Gateway	`VPN_HOST`, `VPN_PEERS`
MySQL	`MYSQL_ROOT_PASSWORD`, `MYSQL_USER`, `MYSQL_USER_PASSWORD`, `MYSQL_REPLICATION_PASSWORD`, `MYSQL_MONITOR_PASSWORD`, `MYSQL_DATABASES`
PostgreSQL	`POSTGRES_PASSWORD`, `POSTGRES_REPLICATION_PASSWORD`, `POSTGRES_MONITOR_PASSWORD`, `POSTGRES_APP_USER`, `POSTGRES_APP_PASSWORD`, `POSTGRES_DATABASES`, `POSTGRES_ADMIN_EMAIL`, `POSTGRES_ADMIN_PASSWORD`
Monitoring	`PMM_ADMIN_PASSWORD`, `PORTAINER_ADMIN_PASSWORD`, `SMTP_*`, `SLACK_WEBHOOK_URL`
Storage	`_DATA_VOLUME`, `_BACKUP_VOLUME` (see 2.4)

Passwords and users are applied when a data volume is first initialized: changing them later does not change an existing database. The PMM client uses the same MYSQL_MONITOR_* and POSTGRES_MONITOR_* accounts the primaries create.

2.3 Server roles (profiles)

Group	Profile	Service
Gateway	`vpn-host`	WireGuard server (on the server with the primaries)
	`vpn-client`	WireGuard peer of another server's VPN host
	`http`	nginx gateway
Monitoring	`pmm`	PMM Server
	`metrics`	PMM client + cAdvisor
	`logs`	Dozzle
	`portainer`	Portainer
MySQL	`mysql-primary`	Primary, read/write
	`mysql-replica`	Read-only replica
	`mysql-backup-primary`	Backups of the primary (on its server)
	`mysql-backup-replica`	Backups of the replica (on its server)
	`mysql-admin`	phpMyAdmin
PostgreSQL	`postgres-primary`	Primary, read/write (also starts `postgres-backup-server`)
	`postgres-replica`	Read-only hot standby
	`postgres-backup-primary`	Backups of the primary (on its server)
	`postgres-backup-replica`	Backups of the replica (on its server)
	`postgres-admin`	pgAdmin

# One server with everything (default), backups of the replicas
COMPOSE_PROFILES=vpn-host,http,pmm,metrics,logs,portainer,mysql-primary,mysql-replica,mysql-backup-replica,mysql-admin,postgres-primary,postgres-replica,postgres-backup-replica,postgres-admin

# Two servers: the primaries...
COMPOSE_PROFILES=vpn-host,http,pmm,metrics,logs,portainer,mysql-primary,mysql-admin,postgres-primary,postgres-admin
# ...and the replicas with their backups
COMPOSE_PROFILES=vpn-client,mysql-replica,mysql-backup-replica,postgres-replica,postgres-backup-replica

# Backups on both servers (two independent sets): add to the primaries' list
...,mysql-backup-primary,postgres-backup-primary

# Only MySQL, backups of the primary
COMPOSE_PROFILES=mysql-primary,mysql-backup-primary

After changing roles, docker compose up -d starts the new services.
Commands only see active profiles: stop a removed role explicitly (docker compose --profile mysql-admin down, or --profile '*' for all).
With no profile set, Compose starts nothing (no service selected).
Applications are not part of this repository or its default profiles. Add them through a deployment-specific Compose file.

2.4 Storage

Every service keeps its data in a Docker volume by default. Set its variable to an absolute folder path to keep the data there instead.

Variable	Default volume	Data	Folder owner (Linux)
`MYSQL_PRIMARY_DATA_VOLUME`	`mysql-data`	MySQL primary	`1001:1001`
`MYSQL_REPLICA_DATA_VOLUME`	`mysql-replica-data`	MySQL replica	`1001:1001`
`MYSQL_PRIMARY_BACKUP_VOLUME`	`mysql-primary-backup`	MySQL backups of the primary	`root`
`MYSQL_REPLICA_BACKUP_VOLUME`	`mysql-replica-backup`	MySQL backups of the replica	`root`
`POSTGRES_PRIMARY_DATA_VOLUME`	`postgres-data`	PostgreSQL primary	`26:26`
`POSTGRES_REPLICA_DATA_VOLUME`	`postgres-replica-data`	PostgreSQL replica	`26:26`
`POSTGRES_PRIMARY_BACKUP_VOLUME`	`postgres-primary-backup`	pgBackRest repository of the primary	`26:26`
`POSTGRES_REPLICA_BACKUP_VOLUME`	`postgres-replica-backup`	pgBackRest repository of the replica	`26:26`
`POSTGRES_ADMIN_DATA_VOLUME`	`pgadmin-data`	pgAdmin	`5050:0`
`PMM_DATA_VOLUME`	`pmm-data`	PMM Server	`1000:0`
`PORTAINER_DATA_VOLUME`	`portainer-data`	Portainer	`root`

# .env — everything of this server under /srv/mydata (c:/mydata on Windows)
POSTGRES_PRIMARY_DATA_VOLUME=/srv/mydata/databases/primary/postgres/data
POSTGRES_PRIMARY_BACKUP_VOLUME=/srv/mydata/databases/primary/postgres/backup
MYSQL_PRIMARY_DATA_VOLUME=/srv/mydata/databases/primary/mysql/data

Linux (Ubuntu, Alpine): create each folder before the first start, owned by its service's user:

sudo install -d -o 26 -g 26 /srv/mydata/databases/primary/postgres/data
sudo install -d -o 26 -g 26 /srv/mydata/databases/primary/postgres/backup
sudo install -d -o 1001 -g 1001 /srv/mydata/databases/primary/mysql/data

Docker Desktop on Windows: folders such as c:/mydata/... work for everything except PostgreSQL data (POSTGRES_PRIMARY_DATA_VOLUME, POSTGRES_REPLICA_DATA_VOLUME): PostgreSQL requires a data folder only its user can read, which Windows folders cannot provide. Keep those in volumes. Windows folders are slower than volumes.
Socket volumes (postgres-socket, postgres-replica-socket) hold no data and are not configurable.

Moving existing data: a new value starts the service empty. Stop the services using it, copy the data, set the variable, start again:

docker compose stop postgres-db postgres-backup-primary postgres-backup-server
docker run --rm -v <PROJECT_NAME>_postgres-data:/from \
    -v /srv/mydata/databases/primary/postgres/data:/to alpine cp -a /from/. /to/
# set POSTGRES_PRIMARY_DATA_VOLUME=/srv/mydata/databases/primary/postgres/data in .env
docker compose up -d

2.5 VPN

VPN host (server with the primaries) — private access to every internal service:

Set VPN_HOST (public IP or domain) and list the users in VPN_PEERS (comma separated, letters and digits only).
Start the stack. Each user gets a config and a QR code in gateway/vpn/host/config/peer_<name>/.
Import peer_<name>.conf into the WireGuard app, or scan peer_<name>.png.

Only the VPN subnet and the internal network go through the tunnel. After changing VPN_HOST, delete gateway/vpn/host/config/peer_* to regenerate client configs.

VPN client (replicas on another server) — the tunnel comes up on the host network, so its containers reach the primaries with no other change:

On the VPN host: add a peer, e.g. VPN_PEERS=admin,replica1, run docker compose up -d vpn, and copy gateway/vpn/host/config/peer_replica1/peer_replica1.conf to the new server (it holds that server's private key).
On the replica server (a clone of this repository): save it as gateway/vpn/client/config/peer.conf.
cp .env.example .env and use a different subnet, e.g. NETWORK_SUBNET=172.31.0.0/16 (the client refuses overlapping routes).
Set the profiles, e.g. vpn-client,mysql-replica,mysql-backup-replica,postgres-replica,postgres-backup-replica, and remove *-replica / *-backup-replica from the VPN host's list.
Use the same database passwords and VPN_DNS_IP as the VPN host.
docker compose up -d; check docker compose ps vpn-client is healthy and the replication state (section 7).

The replica server needs no open inbound port. Not covered yet: the VPN host's admin panels and PMM cannot reach containers on the replica server, and pmm-client-register.sh only registers databases on its own network.

2.6 HTTP gateway

TLS ends at nginx; every service behind it serves plain HTTP on its *_PORT (8080 by default). Requests for unknown hosts are refused.

Public app

Copy gateway/http/conf.d/sites.conf.example to sites.conf, remove default.conf, and configure the public app section with the domain and proxy_pass http://<service>:<port>. This gives the deployment one nginx configuration file.
Put the certificate in gateway/http/certs/<domain>/fullchain.pem and privkey.pem.
Reload: docker compose exec http nginx -s reload.

Internal sites (e.g. https://pmm.monitor.example.lan)

Use the internal sites section in gateway/http/conf.d/sites.conf; replace example.lan. Its map lists one host name per service on port 8080; update a line if you changed that service's *_PORT (nginx does not read .env).
Issue a certificate from the internal CA (created on the first run):
```
cd gateway/http
./internal-ca.sh example.lan '*.apps.example.lan' '*.mysql.example.lan' \
  '*.postgres.example.lan' '*.monitor.example.lan'
```
Install gateway/http/certs/internal-ca/ca.crt as a trusted root CA on every client. Never share ca.key.
Point the names at this server in your internal DNS (or the VPN clients' hosts file), then reload nginx.

Only private networks (LAN, VPN, Docker) are allowed on internal sites.

2.7 Databases

Engine	Primary (read/write)	Replica (read only)
MySQL (Percona Server 8.4)	`mysql-primary:3306`	`mysql-replica:3306`
PostgreSQL (Percona Distribution 17)	`postgres-primary:5432`	`postgres-replica:5432`

Apps connect by role alias (*-primary, *-replica), never by service name.
MySQL: the replica clones the primary (CLONE INSTANCE), then follows it with GTID replication under super_read_only.
PostgreSQL: the replica clones the primary (pg_basebackup), then streams its WAL as a hot standby.
Replication is asynchronous. A replica is not a backup: it replays mistakes within seconds.

Application databases are created by the primary on every start; existing ones are never changed or dropped.

	MySQL	PostgreSQL
Databases	`MYSQL_DATABASES` (empty: none)	`POSTGRES_DATABASES` (empty: none)
Owner / grants	all privileges to `MYSQL_USER` on configured databases only	`POSTGRES_APP_USER` can create databases and gets access to all current databases
Names	lowercase, digits, `_`; `%` patterns grant matching databases (`%` = all)	lowercase, digits, `_`

Fixed at initialization (a new volume is needed to change them): MySQL --lower-case-table-names=1 (must match on both servers), PostgreSQL --encoding=UTF8 --locale=C.UTF-8.

Admin panels: phpMyAdmin (http://mysql-admin:8080) and pgAdmin (http://postgres-admin:8080) list the servers in MYSQL_ADMIN_HOSTS / POSTGRES_ADMIN_HOSTS.

2.8 Backups

Enable a backup profile on the server where the backup should run. Each side backs up only the server on its own machine, into its own storage: with both sides you get two independent sets.

	MySQL (`databases/mysql/backup`)	PostgreSQL (`databases/postgres/backup`)
Profiles	`mysql-backup-primary`, `mysql-backup-replica`	`postgres-backup-primary`, `postgres-backup-replica`
Tool	Percona XtraBackup, zstd	pgBackRest, zstd
Schedule	full `MYSQL_BACKUP_FULL_SCHEDULE` (Sun 02:00), incremental `MYSQL_BACKUP_INCREMENTAL_SCHEDULE` (other days), UTC	full `POSTGRES_BACKUP_FULL_SCHEDULE` (Sun 01:00), differential `POSTGRES_BACKUP_DIFF_SCHEDULE` (other days), `TZ`
Point in time	binary logs, streamed continuously (`mysqlbinlog`)	WAL, archived by PostgreSQL at least every minute while there are writes
Verification	decompress and prepare a copy of the chain	`pgbackrest verify`
Retention	`MYSQL_BACKUP_RETENTION_DAYS` (14)	`POSTGRES_BACKUP_RETENTION_DAYS` (14)
Storage	`MYSQL_*_BACKUP_VOLUME`: `chains/`, `binlog/`	`POSTGRES_*_BACKUP_VOLUME`: pgBackRest repository

First start: a full backup at once, then the schedule. An incremental/differential without a full backup is taken as full.
Incrementals (MySQL) copy pages changed since the previous backup; both servers install page tracking (component_mysqlbackup) so only changed pages are read. Differentials (PostgreSQL) copy files changed since the last full.
Retention: any moment of the last N days stays restorable; older backups and the logs only they need are removed. The newest is never removed.
Replica side, MySQL: the replica's own binary logs carry the primary's GTIDs; nothing is needed from the primary.
Replica side, PostgreSQL: the standby archives the WAL it receives (archive_mode=always). pgBackRest still starts and stops each backup on the primary, through postgres-backup-server (TLS, port 8432, runs with postgres-primary). It creates the TLS certificates in databases/postgres/backup/certs/ on its first start. On one server, nothing else is needed. With the replica elsewhere, copy three files; postgres-backup-replica waits for them:
```
# from the primary's server to the replica's server
scp databases/postgres/backup/certs/{ca.crt,client.crt,client.key} \
    replica-server:services-deploy/databases/postgres/backup/certs/
```
To replace them: delete every file in certs/ on the primary's server, docker compose restart postgres-backup-server, copy again, docker compose restart postgres-backup-replica.
Health checks: unhealthy until the first backup exists, and when the last backup failed, the MySQL binlog stream stopped, or PostgreSQL's last WAL archive attempt failed.
Recreated server (new data volume): MySQL detects the new server_uuid, takes a full backup and keeps the old chains apart. pgBackRest refuses another cluster's repository (stanza-create keeps retrying): remove the old repository if needed (section 7).
Give the backups enough disk: put *_BACKUP_VOLUME on a folder of a separate disk.

2.9 Monitoring and alerts

Tool	What it shows	URL (through the VPN)
PMM	Server resources, MySQL / PostgreSQL dashboards, replication, Query Analytics, containers, alerts	`http://pmm-server:8080` (user `admin`)
Dozzle	Live logs of every container	`http://dozzle:8080`
Portainer	Containers, images, volumes, networks; start/stop, console, stats	`http://portainer:8080` (user `admin`)

The PMM client registers the node (its hostname) and cadvisor-<hostname>; with MONITOR_DATABASES=true it also registers the running databases. Restart pmm-client to register a database started later.
Portainer has full control of Docker. Change the deployment in this repository: docker compose up -d replaces containers edited in Portainer. Its admin password comes from PORTAINER_ADMIN_PASSWORD on the first start only (12+ characters).

Alert channel	`.env`	Result
Email	`ALERT_EMAIL_ENABLED=true` and `SMTP_*`	PMM can send email; create the contact point in the UI
Slack	`ALERT_SLACK_ENABLED=true` and `SLACK_WEBHOOK_URL`	Contact point Slack created on start (read-only in the UI)

Restart pmm-server after changing them, then route alerts in Alerting → Notification policies and use Test on the contact point. Before setting ALERT_SLACK_ENABLED=false, remove Slack from the policies; otherwise it is kept and a warning is logged.

Alert rules in monitoring/pmm/alerting/rules/ are created on every start if their uid does not exist yet, in the folder Custom alerts, and stay editable in the UI.

Rule	Fires when	For	Severity
MySQL down / PostgreSQL down	PMM cannot connect to the server	2m	critical
MySQL replication stopped	replica I/O or SQL thread not running	2m	critical
MySQL / PostgreSQL replica lag	replica more than 5 minutes behind	5m	warning
PostgreSQL WAL archiving failing	`archive_command` keeps failing	10m	critical
Disk almost full	less than 10% free	10m	warning
High CPU	CPU more than 90% busy	15m	warning
High memory	less than 10% available (free + reclaimable cache)	10m	warning
Network errors	more than 1 errored packet per second on an interface	15m	warning
Container restarting	more than 2 restarts in 15 minutes	—	warning

There is no network throughput rule: normal varies too much (backups use all the bandwidth they get). Each rule has a severity label for routing; alerts go to the default contact point until a policy says otherwise.

🛠️ 3. Development workflow

3.1 Directory map

.env.example                 every setting, documented (copy to .env)
compose.yaml                 includes every stack, defines the network
gateway/
  vpn/host/                  WireGuard server (server with the primaries)
  vpn/client/                WireGuard peer for replica-only servers
  http/                      nginx: conf.d/, certs/, internal-ca.sh
monitoring/
  pmm/                       PMM Server: dashboards/Custom/, alerting/ (Slack, rules/)
  client/                    PMM client, cAdvisor, custom-queries/
  logs/                      Dozzle
  portainer/                 Portainer
databases/
  mysql/
    master/                  primary (mysql-db): entrypoint, init script
    replica/                 replica (mysql-replica-db): clone + replication
    admin/                   phpMyAdmin
    backup/                  XtraBackup image, backup.sh, restore.sh
  postgres/
    master/                  primary (postgres-db): entrypoint, init script
    replica/                 replica (postgres-replica-db): clone + streaming
    admin/                   pgAdmin
    backup/                  pgBackRest image, backup.sh, restore.sh, certs/

3.2 Repository conventions

Header on every file: Purpose, Role, Prerequisites, Variables — keep it current when you change a file.
One service, one profile. Add the profile to the tables in .env.example (section 2) and this tutorial.
Every setting comes from .env, passed explicitly in compose.yaml and documented in .env.example with Format and Default. Nothing is hard-coded in Dockerfiles.
Storage is configurable: mount ${<NAME>_VOLUME:-<volume>}:/path and declare <volume> under volumes:.
No depends_on across stacks: wait in the entrypoint instead.
LF line endings everywhere (.gitattributes): CRLF breaks shell scripts and .env values.
Scripts are Bash with set -uo pipefail (or -euo), log lines prefixed [component], and a documented variable list.

3.3 Add a monitoring item

Alert rule: create it in the UI and export it (GET /graph/api/v1/provisioning/alert-rules/<uid>), or copy a file in monitoring/pmm/alerting/rules/, give it a new uid, and save it as a new .json. The metrics data source is PA58DA793C7250F1B. It is created on the next pmm-server start. Edit existing rules in the UI; to remove one, delete both the file and the rule.
Dashboard: add JSON to monitoring/pmm/dashboards/Custom/; PMM re-reads it every 60 seconds.
Custom metric: add a query to monitoring/client/custom-queries/ (see postgresql/replica-lag.yml), then restart pmm-client.

3.4 Local overrides

A compose.override.yaml (not committed) publishes internal tools on 127.0.0.1 for local work:

services:
  pmm-server:
    ports: ["127.0.0.1:8081:8080"]
  dozzle:
    ports: ["127.0.0.1:8082:8080"]
  portainer:
    ports: ["127.0.0.1:8083:8080"]

The same ignored or private override can define deployment-specific applications. Keep application source and application-only variables outside this repository; the infrastructure .env should contain only stable infrastructure settings.

services:
  customer-api:
    build:
      context: ../customer-apps/customer-api
    restart: always
    expose:
      - "8080"
    environment:
      NODE_ENV: production
      PORT: 8080

The application service joins this Compose project's default network, so the HTTP gateway can proxy to http://customer-api:8080. Add its nginx site configuration in the deployment-specific files under gateway/http/conf.d/, or keep that configuration in the private deployment repository as well. Validate the combined deployment with:

docker compose config --quiet
docker compose up -d --build

3.5 Application examples

The repository includes generic application packaging examples under apps/, but not application source or Java runtime archives:

apps/
├── .example/                  # the only versioned application directory
│   ├── java-service-example/  # Linux export + jre.tar.gz
│   ├── angular-app-example/   # Angular SSR export
│   └── nodejs-app-example/    # package.json + package-lock.json + app
└── <your-app>/                # local/private, ignored by Git

Copy the override template and add the application profiles to the local .env:

cp compose.override.yaml.example compose.override.yaml

COMPOSE_PROFILES=...,java-service-example,angular-app-example,nodejs-app-example

The override is intentionally an example and is not loaded automatically. Only apps/.example/ is kept in Git. Add real applications directly under apps/; those directories are ignored and are preserved on each deployment. Keep real application source, jre.tar.gz, application environment variables, and active nginx routes in the deployment's private files. The Java example expects an exported launcher named services; change the CMD in its Dockerfile for another launcher.

3.6 Configuration ownership

Reusable infrastructure configuration is committed and updated with the repository. Deployment-owned active configuration and generated state are ignored so an infrastructure update cannot replace server-specific settings.

Category	Examples	Git status
Compose definitions	`compose.yaml` files	Tracked
Reusable service configuration	PMM rules/dashboards, custom queries, VPN templates	Tracked
Configuration templates	`*.example` files	Tracked
Active nginx sites	`gateway/http/conf.d/*.conf`	Ignored
Environment and secrets	`.env`, certificates, database backup data	Ignored
Generated VPN state	keys, peer configs, QR codes, `wg0.conf`	Ignored

For nginx, copy the committed template once and customize the local file:

cp gateway/http/conf.d/sites.conf.example gateway/http/conf.d/sites.conf

The active sites.conf is ignored, while sites.conf.example can continue to receive improvements from infrastructure updates. If a deployment only needs the catch-all behavior, copy default.conf.example to default.conf.

✅ 4. Testing and validation

The repository is infrastructure as code: there is no unit test suite or linter configuration. Validate changes with these checks, in order.

1. Configuration — resolves every file, variable and profile:

docker compose config --quiet                  # this server's profiles
docker compose --profile '*' config --quiet    # every service

2. Shell syntax and lint:

for f in $(git ls-files '*.sh'); do bash -n "$f" || echo "FAIL $f"; done
docker run --rm -v "$PWD:/mnt" -w /mnt koalaman/shellcheck:v0.10.0 $(git ls-files '*.sh')

3. Isolated integration run — a throwaway project that never touches your volumes or network. Later --env-file values override earlier ones:

cat > /tmp/test.env <<'EOF'
PROJECT_NAME=bkptest
NETWORK_NAME=bkptest-net
NETWORK_SUBNET=172.29.0.0/16
VPN_DNS_IP=127.0.0.11
COMPOSE_PROFILES=mysql-primary,mysql-replica,mysql-backup-replica,postgres-primary,postgres-replica,postgres-backup-replica
EOF
docker compose --env-file .env.example --env-file /tmp/test.env up -d --build
docker compose -p bkptest ps                   # backup containers turn healthy after their first backup
PROJECT_NAME=bkptest bash databases/postgres/backup/restore.sh replica
docker compose --env-file .env.example --env-file /tmp/test.env --profile '*' down -v --rmi local

4. Restore drill — the real test of backups. Run it monthly in production (5.1).

5. Alert rules — after a pmm-server start, the log lists what was created:

docker compose logs pmm-server | grep 'alert rule'

💡 5. Practical examples

5.1 Point-in-time restore after a bad `DELETE`

A table was emptied at 11:04:20 UTC. Restore to just before it, into a temporary server:

bash databases/mysql/backup/restore.sh replica "2026-10-15 11:04:18"

[mysql-restore] preparing from services_mysql-replica-backup for 2026-10-15 11:04:18 UTC into volume mysql-restore
[mysql-restore] backups: 20261015-105957/20261015-105957-full 20261015-105957/20261015-110400-incr
[mysql-restore] starting temporary server mysql-restore
[mysql-restore] replaying binary logs of server 81762171-c887-11f1-92fe-8ac3a68eac8d from mysql-bin.000004 up to 2026-10-15 11:04:18 UTC
[mysql-restore] done. The temporary server mysql-restore holds the restored data:

Take the rows back, then remove the temporary server:

docker exec mysql-restore mysqldump -uroot -p shop orders > orders.sql
docker rm -f mysql-restore; docker volume rm mysql-restore

The PostgreSQL equivalent picks the right full + differential backup and stops before the bad transaction:

bash databases/postgres/backup/restore.sh replica "2026-10-15 10:51:34"

[postgres-restore]   INFO: repo1: restore backup set 20261015-105000F_20261015-105100D, recovery will start at 2026-10-15 10:51:00
[postgres-restore]   LOG:  recovery stopping before commit of transaction 758, time 2026-10-15 10:51:37.395886+00
[postgres-restore]   LOG:  archive recovery complete

Without a time, both scripts restore everything up to the last copied change.

5.2 Check the backups

docker compose exec postgres-backup-replica pgbackrest info

stanza: main
    status: ok

    db (current)
        wal archive min/max (17): 000000010000000000000003/000000010000000000000016

        full backup: 20261015-105000F
            timestamp start/stop: 2026-10-15 10:50:00+00 / 2026-10-15 10:50:04+00
            database size: 22.4MB, database backup size: 22.4MB
            repo1: backup set size: 2.6MB, backup size: 2.6MB

docker compose exec mysql-backup-replica ls /backups/chains/*

/backups/chains/20261015-105957:
20261015-105957-full  20261015-110400-incr  server_uuid  started

5.3 Recover after losing a server

Bring a backup copy back into a volume, then restore from it:

REPO_VOLUME=pg-recovered bash databases/postgres/backup/restore.sh

BACKUP_VOLUME=mysql-recovered bash databases/mysql/backup/restore.sh

[postgres-restore]   INFO: repo1: restore backup set 20261015-110000F_20261015-110600D, recovery will start at 2026-10-15 11:06:00
[postgres-restore]   LOG:  archive recovery complete

🧯 6. Troubleshooting

Symptom	Cause	Fix
`no service selected`, or settings from `.env` ignored	`COMPOSE_PROFILES` is empty, or `COMPOSE_ENV_FILES` is set in the shell	Set the profiles in `.env`; `unset COMPOSE_ENV_FILES`
PostgreSQL exits with `chmod: changing permissions of '/data/db': Operation not permitted`	`POSTGRES__DATA_VOLUME` points to a Windows folder*	Keep PostgreSQL data in a volume on Docker Desktop (the default value)
A service with a folder path fails with `Permission denied` (Linux)	Docker created the folder owned by root	`sudo install -d -o <uid> -g <gid> <folder>` with the owner from 2.4, then restart it

More cases:

Symptom	Fix
`postgres-backup-replica` logs `waiting for ca.crt, client.crt and client.key`	Copy the three files from the primary's server (2.8)
`postgres-backup-replica` keeps retrying `stanza-create`	The repository belongs to another cluster (recreated primary): move or delete it (section 7)
The VPN client refuses to start	`NETWORK_SUBNET` overlaps the VPN host's: use another subnet (e.g. `172.31.0.0/16`)
A replica keeps restarting after its first start	The MySQL replica restarts once after cloning (normal); otherwise check that its passwords and `lower-case-table-names` match the primary
A database started later is missing in PMM	`docker compose restart pmm-client`

🔧 7. Operations

Replication state

docker compose exec mysql-replica-db mysql -uroot -p -e "SHOW REPLICA STATUS\G"
docker compose exec postgres-replica-db psql -U postgres -c "SELECT status, sender_host FROM pg_stat_wal_receiver"
docker compose exec postgres-db psql -U postgres -c "SELECT client_addr, state, replay_lag FROM pg_stat_replication"

Rebuild a replica

docker compose rm -sf mysql-replica-db
docker volume rm <PROJECT_NAME>_mysql-replica-data   # or empty its folder
docker compose up -d mysql-replica-db
docker compose restart pmm-client

Same for postgres-replica-db / <PROJECT_NAME>_postgres-replica-data.

Turn a backup side off for good — remove its profile. For PostgreSQL also delete its repository, or the server keeps archiving WAL into it:

docker compose rm -sf postgres-backup-primary postgres-db
docker volume rm <PROJECT_NAME>_postgres-primary-backup   # or empty its folder
docker compose up -d postgres-db

On the replica's server: postgres-backup-replica, postgres-replica-db and <PROJECT_NAME>_postgres-replica-backup.

Docker log rotation (/etc/docker/daemon.json):

{ "log-driver": "json-file", "log-opts": { "max-size": "50m", "max-file": "5" } }

Name		Name	Last commit message	Last commit date
apps/.example		apps/.example
databases		databases
gateway		gateway
monitoring		monitoring
.env.example		.env.example
.gitattributes		.gitattributes
.gitignore		.gitignore
README.md		README.md
compose.override.yaml.example		compose.override.yaml.example
compose.yaml		compose.yaml
deploy.ps1		deploy.ps1
deploy.sh		deploy.sh

Latest commit

History

Folders and files

Repository files navigation