Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 1 addition & 2 deletions Makefile
Original file line number Diff line number Diff line change
@@ -1,8 +1,7 @@
LOCAL_ENV = \
DATABASE_URL=sqlite+aiosqlite:///./drukbox.db \
SERVICE_TOKENS=dev-token \
DEFAULT_HOST_PROVIDER=docker \
TAILSCALE_ENABLED=false
DEFAULT_HOST_PROVIDER=docker

.PHONY: dev
dev:
Expand Down
4 changes: 4 additions & 0 deletions docs/add-a-provider.md
Original file line number Diff line number Diff line change
Expand Up @@ -98,6 +98,10 @@ instead.
Set `max_lifetime` when the provider stops every VM after a fixed time. No
lease then ends later than that time.

Set `tailnet = True` when the provider's hosts have no public SSH address.
A provider that offers both paths reads a `<NAME>_TAILNET` setting into
`self.tailnet`, like `providers/hetzner/`.

## 7. Tests

- Unit-test the provider with a mocked api object
Expand Down
6 changes: 3 additions & 3 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -221,9 +221,9 @@ customizes its host — `image`, `env`, `template`, `instance_type`, or
## Diagnostics

`GET /doctor` runs one cheap, non-mutating probe per dependency —
database, active provider, secrets exchange, Tailscale when enabled — in
parallel with a
per-probe timeout. Providers own their probe (`diagnose()`) and their
database, active provider, secrets exchange, and Tailscale when the
active provider joins the tailnet — in parallel with a per-probe
timeout. Providers own their probe (`diagnose()`) and their
remediation slug (`diagnose_hint`); the endpoint stays a thin
orchestrator. It always returns 200; health is the `ok` field in the
body.
4 changes: 2 additions & 2 deletions docs/boat.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,8 @@ direct SSH coordinates. The response contains a per-host private key once.
The provider uses the public Boat HTTP API. It needs no Boat CLI or SDK.

Boat supplies an Ubuntu VM with passwordless sudo. The provider uses this
external SSH path even when `TAILSCALE_ENABLED` is true. The provider does not
join the tailnet. The secrets proxy must accept connections from the sandbox.
external SSH path and does not join the tailnet. The secrets proxy must accept
connections from the sandbox.

## Configuration

Expand Down
3 changes: 1 addition & 2 deletions docs/cloudflare.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

`POST /hosts` with `{"provider": "cloudflare"}` starts a Cloudflare container,
joins it to Tailscale, and returns `internal_ssh_host`. The container has no
public SSH address, so this provider requires `TAILSCALE_ENABLED=true`.
public SSH address, so this provider requires the Tailscale settings.

Cloudflare starts containers only from a Durable Object in a Worker. Drukbox
calls the Worker in `deploy/cloudflare`, which you deploy to your account.
Expand All @@ -25,7 +25,6 @@ callers reach the sandbox tag as `root`.
```dotenv
CLOUDFLARE_WORKER_URL=https://drukbox-sandboxes.YOUR-SUBDOMAIN.workers.dev
CLOUDFLARE_WORKER_TOKEN=YOUR-DEDICATED-TOKEN
TAILSCALE_ENABLED=true
```

The token can also come from `/run/secrets/CLOUDFLARE_WORKER_TOKEN`.
Expand Down
1 change: 0 additions & 1 deletion docs/daytona.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,6 @@ Image builds, stored snapshots, and sandboxes use provider resources.
| `DAYTONA_API_TIMEOUT` | API request timeout in seconds; default `150` |
| `DAYTONA_LIFECYCLE_TIMEOUT_SECONDS` | Start or deletion polling budget; default `180` |
| `DAYTONA_BOOTSTRAP_SSH_TIMEOUT_SECONDS` | SSH readiness budget; default `120` |
| `TAILSCALE_ENABLED` | Required: `true` |

Configure the tailnet and Tailscale credentials as described in
[Networking](networking.md). Hosts use Tailscale SSH as `root`; they have no
Expand Down
49 changes: 29 additions & 20 deletions docs/deploy.md
Original file line number Diff line number Diff line change
Expand Up @@ -116,9 +116,8 @@ all provider extras.

The `docker` provider runs each sandbox as a local container with sshd,
so you can try drukbox with no cloud account or API token.
Set `DEFAULT_HOST_PROVIDER=docker`,
`TAILSCALE_ENABLED=false`, and `UVICORN_HOST=127.0.0.1` in `drukbox.env`,
then run:
Set `DEFAULT_HOST_PROVIDER=docker` and `UVICORN_HOST=127.0.0.1` in
`drukbox.env`, then run:

```bash
# Linux: match the host socket's group. macOS: use group 0 (see below).
Expand Down Expand Up @@ -346,16 +345,22 @@ reads the same `drukbox.env` and connects to the same database.

## Choose a networking mode

`TAILSCALE_ENABLED=false` (default): callers reach sandboxes over the
provider's public path. On AWS this means per-VM keypairs and the
managed security group — see
[Networking](networking.md#tailscale-off-public-path-key-only-auth).
Each provider decides whether its hosts join your tailnet. Cloudflare,
Daytona, E2B, and Vercel always join it. Docker, docker-sbx, and Boat
never do. AWS, Hetzner, Exoscale, and exe.dev join it when
`AWS_TAILNET`, `HETZNER_TAILNET`, `EXOSCALE_TAILNET`, or `EXE_TAILNET`
is `true` (default `false`).

`TAILSCALE_ENABLED=true`: sandboxes join your tailnet at boot and
callers connect over the overlay. Requires a Tailscale OAuth client
with auth-key write scope, and tailnet ACLs that (a) own the tags in
`TAILSCALE_AUTH_TAGS` and (b) permit tailscaled-SSH to the tagged
nodes.
Off the tailnet, callers reach sandboxes over the provider's public
path. On AWS this means per-VM keypairs and the managed security
group — see
[Networking](networking.md#off-the-tailnet-public-path-key-only-auth).

On the tailnet, sandboxes join at boot and callers connect over the
overlay. This requires the `TAILSCALE_` settings, a Tailscale OAuth
client with auth-key write scope, and tailnet ACLs that (a) own the
tags in `TAILSCALE_AUTH_TAGS` and (b) permit tailscaled-SSH to the
tagged nodes.

## Private image registry

Expand Down Expand Up @@ -393,7 +398,7 @@ AWS credentials come from the SDK's default chain (instance profile,
`~/.aws`, or env) — drukbox never plumbs them through its own
settings. The policy needs `ec2:RunInstances`,
`ec2:TerminateInstances`, `ec2:DescribeInstances`, `ec2:CreateTags`,
`sts:GetCallerIdentity`, plus — with Tailscale off —
`sts:GetCallerIdentity`, plus — off the tailnet —
`ec2:ImportKeyPair`, `ec2:DeleteKeyPair`, `ec2:CreateSecurityGroup`,
`ec2:DescribeSecurityGroups`, `ec2:AuthorizeSecurityGroupIngress`,
`ec2:DescribeSubnets` (when `AWS_SUBNET_ID` is set), and
Expand Down Expand Up @@ -547,10 +552,10 @@ curl -fsS -H "Authorization: Bearer $TOKEN" http://localhost:8780/doctor
```

`/doctor` runs one read-only probe per dependency: database, active
provider, secrets exchange, and Tailscale when enabled. It reports per-check
status, latency, and a remediation hint on failures. It always returns 200 —
health is the `ok` field. `GET /healthz` is the unauthenticated
liveness probe.
provider, secrets exchange, and Tailscale when the active provider joins
the tailnet. It reports per-check status, latency, and a remediation hint
on failures. It always returns 200 — health is the `ok` field.
`GET /healthz` is the unauthenticated liveness probe.

For a full end-to-end check, run the black-box suite against the
deployment (it provisions and destroys a real host — disposable
Expand Down Expand Up @@ -605,11 +610,11 @@ Secrets exchange:
| `SECRETS_EXCHANGE_BIND_HOST` | `127.0.0.1` | Loopback listener for the exchange. The API reads it to reach the exchange. |
| `SECRETS_EXCHANGE_PORT` | `8781` | Port the exchange process listens on. The API reads it to reach the exchange. |

Tailscale (required when `TAILSCALE_ENABLED=true`):
Tailscale (required when a provider joins the tailnet; set the first four
together):

| Variable | Default | Purpose |
| --- | --- | --- |
| `TAILSCALE_ENABLED` | `false` | Provision hosts onto a tailnet. |
| `TAILSCALE_TAILNET` | — | Tailnet DNS suffix for sandbox MagicDNS hostnames. |
| `TAILSCALE_AUTH_TAGS` | — | Comma-separated tags applied to minted auth keys. |
| `TAILSCALE_OAUTH_CLIENT_ID` | — | OAuth client ID. |
Expand All @@ -632,6 +637,7 @@ exe.dev provider:
| `EXE_API_TIMEOUT` | `30.0` | Timeout for exe.dev API calls. |
| `EXE_BOOTSTRAP_SSH_TIMEOUT_SECONDS` | `30.0` | ssh-keyscan retry budget for a fresh exe.dev sandbox. |
| `EXE_SSH_USERNAME` | `exedev` | In-VM user callers SSH as. |
| `EXE_TAILNET` | `false` | Join each host to the tailnet. The host keeps its exe.dev SSH address. |

AWS provider:

Expand All @@ -647,6 +653,7 @@ AWS provider:
| `AWS_INSTANCE_PROFILE` | — | Optional IAM instance profile attached to sandboxes. |
| `AWS_BOOTSTRAP_SSH_TIMEOUT_SECONDS` | `120.0` | ssh-keyscan retry budget for a fresh EC2 instance. |
| `AWS_SSH_USERNAME` | `ubuntu` | In-VM user callers SSH as. |
| `AWS_TAILNET` | `false` | Join each host to the tailnet. The host then has no key pair, security group, or public IP. |

Hetzner provider:

Expand All @@ -659,9 +666,10 @@ Hetzner provider:
| `HETZNER_API_TIMEOUT` | `30.0` | Timeout for Hetzner API calls. |
| `HETZNER_BOOTSTRAP_SSH_TIMEOUT_SECONDS` | `120.0` | ssh-keyscan retry budget for a fresh server. |
| `HETZNER_SSH_USERNAME` | `root` | In-VM user callers SSH as. |
| `HETZNER_TAILNET` | `false` | Join each host to the tailnet. The host keeps its key-locked public IP. |

A fresh Hetzner server has no firewall — port 22 is open and SSH is
key-only. Drukbox mints a per-VM ed25519 key in both networking modes;
key-only. Drukbox mints a per-VM ed25519 key on or off the tailnet;
there is no security-group or ingress-CIDR configuration to manage.

Exoscale provider:
Expand All @@ -677,6 +685,7 @@ Exoscale provider:
| `EXOSCALE_API_TIMEOUT` | `30.0` | Timeout for Exoscale API calls. |
| `EXOSCALE_BOOTSTRAP_SSH_TIMEOUT_SECONDS` | `120.0` | ssh-keyscan retry budget for a fresh instance. |
| `EXOSCALE_SSH_USERNAME` | `ubuntu` | In-VM user callers SSH as. Exoscale Ubuntu templates default to `ubuntu`. |
| `EXOSCALE_TAILNET` | `false` | Join each host to the tailnet. The host keeps its key-locked public IP. |

The API key's IAM role must allow the compute operations `list-templates` and
`list-instance-types` in addition to the instance and SSH-key operations:
Expand Down
3 changes: 1 addition & 2 deletions docs/e2b.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

`POST /hosts` with `{"provider": "e2b"}` starts an E2B sandbox, joins it to
Tailscale, and returns `internal_ssh_host`. The sandbox has no public SSH
address, so this provider requires `TAILSCALE_ENABLED=true`.
address, so this provider requires the Tailscale settings.

## Prepare the template

Expand All @@ -24,7 +24,6 @@ callers reach the sandbox tag as `root`.
```dotenv
E2B_API_KEY=YOUR-API-KEY
E2B_DEFAULT_IMAGE=drukbox
TAILSCALE_ENABLED=true
```

The key can also come from `/run/secrets/E2B_API_KEY`.
Expand Down
47 changes: 22 additions & 25 deletions docs/networking.md
Original file line number Diff line number Diff line change
@@ -1,43 +1,39 @@
# Networking and reachability

This page explains how a caller reaches a sandbox and why each path is
shaped the way it is. For turning these modes on, read
[Deploy](deploy.md).
shaped the way it is. For the settings, read
[Deploy](deploy.md#choose-a-networking-mode).

## Two modes
## Two paths

`TAILSCALE_ENABLED` selects between two networking models. A provider
whose hosts cannot join a tailnet (docker — containers published on the
daemon host,
docker-sbx — local microVMs) always takes the external path,
whatever the mode. The API response carries
both addresses; which is populated depends on the mode and the
Each provider decides whether its hosts join the tailnet. The API
response carries both addresses; which is populated depends on the
provider:

- `external_ssh_host` / `external_ssh_port` — the provider-given
public path. Always present (empty for an AWS host with Tailscale
on, which has no public path at all).
- `internal_ssh_host` — the tailnet MagicDNS name, populated only with
Tailscale on. The internal path is always port 22 by Tailscale
convention, so there is no internal port field.
public path. Always present (empty for a host with no public path,
such as an AWS host on the tailnet).
- `internal_ssh_host` — the tailnet MagicDNS name, populated only for
a host on the tailnet. The internal path is always port 22 by
Tailscale convention, so there is no internal port field.

Callers pick whichever path they can reach and dial it themselves.

## Tailscale on: the overlay is the security model
## On the tailnet: the overlay is the security model

Per sandbox, drukbox mints an ephemeral, tag-scoped Tailscale auth key
and injects it at boot. The sandbox joins the tailnet, drukbox
discovers the device by hostname (one shared poller serves all
concurrent provisions), stores the device ID, and ssh-keyscans over
the tailnet path.

On AWS this mode has no public surface at all: no keypair is imported,
no security group is managed, and no public IP is attached —
authentication and reachability are tailnet ACLs plus tailscaled-SSH,
and `external_ssh_host` comes back empty.
On AWS a host on the tailnet has no public surface at all: no keypair
is imported, no security group is managed, and no public IP is
attached — authentication and reachability are tailnet ACLs plus
tailscaled-SSH, and `external_ssh_host` comes back empty.

Hetzner can't drop the public IP that cheaply (a server needs one to
reach the tailnet), so a tailscale-on Hetzner box still has a public
reach the tailnet), so a Hetzner box on the tailnet still has a public
IPv4 — but it is key-locked with the per-VM key drukbox always mints,
and `external_ssh_host` carries that public path alongside the tailnet
one. The tailnet remains the intended path; the public port just isn't
Expand All @@ -47,9 +43,9 @@ Teardown releases the Tailscale device by its stored ID — never by
hostname — and treats a 404 as success, because ephemeral devices
self-delete and operator cleanup can race the API.

## Tailscale off: public path, key-only auth
## Off the tailnet: public path, key-only auth

With Tailscale off the provider's public path is the only path.
Off the tailnet the provider's public path is the only path.

On exe.dev, SSH terminates at exe's edge (`ssh_dest`), and exe owns
authentication; drukbox returns no key material.
Expand Down Expand Up @@ -95,7 +91,8 @@ key material, scanned against the preferred path (`internal_ssh_host`
when present, else `external_ssh_host`) and keyed to whichever address
was scanned.

With Tailscale off, the first-time keyscan runs over the public
Off the tailnet, the first-time keyscan runs over the public
network and carries the same TCP-session MITM window any first-time
keyscan has. Operators who need stronger guarantees should enable
Tailscale and let the scan run over the authenticated overlay.
keyscan has. Operators who need stronger guarantees should put the
provider on the tailnet and let the scan run over the authenticated
overlay.
8 changes: 4 additions & 4 deletions docs/security.md
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ no rate limiting (see [Resource exhaustion](#resource-exhaustion)).
How a caller reaches a sandbox, and the tradeoffs of each path, are
covered in [Networking](networking.md). The security-relevant summary:

- **Per-VM keys.** On AWS (Tailscale off), Hetzner, docker, and
- **Per-VM keys.** On AWS (off the tailnet), Hetzner, docker, and
docker-sbx, drukbox mints a fresh ed25519 keypair per VM, returns
the private half **once** in the create response, and never persists
it — a later `GET /hosts/{id}` returns `private_key: null`. The key
Expand All @@ -72,10 +72,10 @@ covered in [Networking](networking.md). The security-relevant summary:
non-loopback `DOCKER_SSH_HOST` is reachable by everything that
reaches that address. On a tailnet it has no device of its own, so ACL
tags cannot scope it. Each sandbox's key is the only boundary.
- **First-keyscan MITM window.** With Tailscale off, the `known_hosts`
- **First-keyscan MITM window.** Off the tailnet, the `known_hosts`
material is scanned over the public network and carries the usual
trust-on-first-use window. Enable Tailscale to run the scan over the
authenticated overlay.
trust-on-first-use window. Put the provider on the tailnet to run the
scan over the authenticated overlay.

## Secrets: which field to use

Expand Down
3 changes: 1 addition & 2 deletions docs/vercel.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

`POST /hosts` with `{"provider": "vercel"}` starts a Vercel sandbox, joins it
to Tailscale, and returns `internal_ssh_host`. The sandbox has no public SSH
address, so this provider requires `TAILSCALE_ENABLED=true`.
address, so this provider requires the Tailscale settings.

## Prepare the image

Expand All @@ -29,7 +29,6 @@ VERCEL_TOKEN=YOUR-ACCESS-TOKEN
VERCEL_TEAM_ID=team_YOUR_TEAM
VERCEL_PROJECT_ID=prj_YOUR_PROJECT
VERCEL_DEFAULT_IMAGE=drukbox-sandbox:v1
TAILSCALE_ENABLED=true
```

The token can also come from `/run/secrets/VERCEL_TOKEN`.
Expand Down
2 changes: 1 addition & 1 deletion env/test.env
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ SECRETS_KEY=MDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDA=
DEVICE_DISCOVERY_TIMEOUT_SECONDS=2
EXE_API_TOKEN=exe-token
EXE_DEFAULT_IMAGE=ghcr.io/drukbox/sandbox:exeuntu-latest
TAILSCALE_ENABLED=true
EXE_TAILNET=true
TAILSCALE_AUTH_TAGS=tag:sandbox
TAILSCALE_OAUTH_CLIENT_ID=tailscale-client-id
TAILSCALE_OAUTH_CLIENT_SECRET=tailscale-client-secret
Expand Down
12 changes: 4 additions & 8 deletions src/api/app.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,6 @@

from core.database import close_database
from core.exceptions import AppException
from core.settings import get_settings
from diagnostics.api import router as diagnostics_router
from hosts.api import router as hosts_router
from http_proxies.api import router as http_proxies_router
Expand Down Expand Up @@ -39,13 +38,10 @@ def filter(self, record: logging.LogRecord) -> bool:

@asynccontextmanager
async def lifespan(app: FastAPI) -> AsyncIterator[None]:
# Process-wide Tailscale client when enabled: reusing one
# httpx.AsyncClient amortises the TLS handshake and OAuth token cache
# across every provision instead of paying both per request. When
# Tailscale is disabled the slot stays None and request handlers skip
# the network layer entirely.
settings = get_settings()
tailscale: Tailscale | None = Tailscale.from_settings() if settings.tailscale_enabled else None
# Process-wide Tailscale client: reusing one httpx.AsyncClient amortises
# the TLS handshake and OAuth token cache across every provision instead
# of paying both per request. Without a tailnet the slot stays None.
tailscale = Tailscale.from_settings()
app.state.tailscale = tailscale
try:
yield
Expand Down
14 changes: 12 additions & 2 deletions src/conftest.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@

import pytest
from httpx import ASGITransport, AsyncClient
from sqlalchemy import insert

TEST_ENV_PATH = Path(__file__).resolve().parents[1] / "env" / "test.env"

Expand Down Expand Up @@ -39,8 +40,6 @@ def load_test_env() -> None:

@pytest.fixture(autouse=True)
async def reset_database() -> AsyncGenerator[None]:
from sqlalchemy import insert

from core.database import Base, engine
from hosts import models # noqa: F401
from service_accounts.models import ServiceAccount
Expand All @@ -64,3 +63,14 @@ async def client() -> AsyncGenerator[AsyncClient]:
base_url="https://drukbox.example",
) as client:
yield client


@pytest.fixture
def no_tailscale_settings(monkeypatch: pytest.MonkeyPatch) -> None:
for key in (
"TAILSCALE_TAILNET",
"TAILSCALE_AUTH_TAGS",
"TAILSCALE_OAUTH_CLIENT_ID",
"TAILSCALE_OAUTH_CLIENT_SECRET",
):
monkeypatch.delenv(key)
Loading
Loading