From 9e233ccbfa531b30f95d213ee980545b91f17b38 Mon Sep 17 00:00:00 2001 From: Felipe Arce Date: Mon, 19 Oct 2026 18:54:45 +0000 Subject: [PATCH 1/9] Canvas link: serve canvases from a linked Canvas Cloud; remove the canvas crate copper-cloud no longer stores canvases itself. crates/canvas (now the engine of the standalone copper-browser/canvas-cloud server, history preserved there) is removed and replaced by crates/canvas-link: - reverse proxy for the unchanged /v1 canvas API (canvases, invites, canvas-links, mentions, people) inside the instance gate, acting as the signed-in Copper user via the app key + X-Canvas-User; upstream 401s are never forwarded (clients would sign out), outages are 502 canvas_upstream / 503 canvas_unavailable, never 404 or empty lists - WebSocket pipe for /v1/canvases/{id}/ws: upstream handshake first, binary-exact frames, close codes and reasons forwarded both ways, 1001 on shutdown - event bridge from /v1/apps/self/events into the per-user SSE (event: canvas) - user provisioning (batch on link, JIT on unknown_user, 10 min reconcile, outbox for disable/delete), optional SHA-256 leaf pin for a self-signed Canvas Cloud - admin API /admin/api/canvas-link (+ /test), canvases list/delete and overview counts sourced from the link, doctor check, CLI copper-cloud canvas link|unlink|status|sync-users - migration 0030_canvas_link.sql; 0100-0103 stay byte-identical for existing installs - /v1/info features gains canvas_link --- .github/workflows/ci.yml | 4 +- Cargo.lock | 179 +- Cargo.toml | 15 +- crates/{canvas => canvas-link}/Cargo.toml | 19 +- crates/canvas-link/src/admin.rs | 388 ++++ crates/canvas-link/src/bridge.rs | 361 ++++ crates/canvas-link/src/info.rs | 34 + crates/canvas-link/src/landing.rs | 207 +++ crates/canvas-link/src/lib.rs | 338 ++++ crates/canvas-link/src/provision.rs | 509 ++++++ crates/canvas-link/src/proxy.rs | 325 ++++ crates/canvas-link/src/settings.rs | 351 ++++ crates/canvas-link/src/tls.rs | 94 + crates/canvas-link/src/upstream.rs | 296 ++++ crates/canvas-link/src/ws.rs | 362 ++++ crates/canvas-link/tests/support/mock.rs | 700 ++++++++ crates/canvas-link/tests/upstream.rs | 136 ++ crates/canvas/src/admin.rs | 57 - crates/canvas/src/chat.rs | 175 -- crates/canvas/src/checklist.rs | 744 -------- crates/canvas/src/geometry.rs | 411 ----- crates/canvas/src/lib.rs | 128 -- crates/canvas/src/mentions.rs | 471 ----- crates/canvas/src/ops.rs | 1372 -------------- crates/canvas/src/ops/tests.rs | 790 --------- crates/canvas/src/read.rs | 479 ----- crates/canvas/src/rest.rs | 1575 ----------------- crates/canvas/src/room.rs | 649 ------- crates/canvas/src/schema.rs | 310 ---- crates/canvas/src/shape.rs | 298 ---- crates/canvas/src/store.rs | 124 -- crates/canvas/src/ws.rs | 257 --- crates/canvas/tests/chat.rs | 244 --- crates/canvas/tests/checklist.rs | 291 --- crates/canvas/tests/common/mod.rs | 514 ------ crates/canvas/tests/js_interop.rs | 236 --- crates/canvas/tests/mentions.rs | 432 ----- crates/canvas/tests/rest.rs | 938 ---------- crates/canvas/tests/ws.rs | 458 ----- crates/copper-cloud/Cargo.toml | 6 +- .../migrations/0030_canvas_link.sql | 58 + crates/copper-cloud/src/admin_api.rs | 216 ++- crates/copper-cloud/src/cli.rs | 304 ++++ crates/copper-cloud/src/doctor.rs | 54 + crates/copper-cloud/src/lib.rs | 30 +- crates/copper-cloud/src/serve.rs | 13 +- crates/copper-cloud/tests/admin.rs | 104 +- crates/copper-cloud/tests/api.rs | 8 +- crates/copper-cloud/tests/canvas_link.rs | 1064 +++++++++++ crates/copper-cloud/tests/common/mod.rs | 5 +- crates/copper-cloud/tests/history_delete.rs | 3 +- crates/core/src/app.rs | 2 + 52 files changed, 6003 insertions(+), 11135 deletions(-) rename crates/{canvas => canvas-link}/Cargo.toml (61%) create mode 100644 crates/canvas-link/src/admin.rs create mode 100644 crates/canvas-link/src/bridge.rs create mode 100644 crates/canvas-link/src/info.rs create mode 100644 crates/canvas-link/src/landing.rs create mode 100644 crates/canvas-link/src/lib.rs create mode 100644 crates/canvas-link/src/provision.rs create mode 100644 crates/canvas-link/src/proxy.rs create mode 100644 crates/canvas-link/src/settings.rs create mode 100644 crates/canvas-link/src/tls.rs create mode 100644 crates/canvas-link/src/upstream.rs create mode 100644 crates/canvas-link/src/ws.rs create mode 100644 crates/canvas-link/tests/support/mock.rs create mode 100644 crates/canvas-link/tests/upstream.rs delete mode 100644 crates/canvas/src/admin.rs delete mode 100644 crates/canvas/src/chat.rs delete mode 100644 crates/canvas/src/checklist.rs delete mode 100644 crates/canvas/src/geometry.rs delete mode 100644 crates/canvas/src/lib.rs delete mode 100644 crates/canvas/src/mentions.rs delete mode 100644 crates/canvas/src/ops.rs delete mode 100644 crates/canvas/src/ops/tests.rs delete mode 100644 crates/canvas/src/read.rs delete mode 100644 crates/canvas/src/rest.rs delete mode 100644 crates/canvas/src/room.rs delete mode 100644 crates/canvas/src/schema.rs delete mode 100644 crates/canvas/src/shape.rs delete mode 100644 crates/canvas/src/store.rs delete mode 100644 crates/canvas/src/ws.rs delete mode 100644 crates/canvas/tests/chat.rs delete mode 100644 crates/canvas/tests/checklist.rs delete mode 100644 crates/canvas/tests/common/mod.rs delete mode 100644 crates/canvas/tests/js_interop.rs delete mode 100644 crates/canvas/tests/mentions.rs delete mode 100644 crates/canvas/tests/rest.rs delete mode 100644 crates/canvas/tests/ws.rs create mode 100644 crates/copper-cloud/migrations/0030_canvas_link.sql create mode 100644 crates/copper-cloud/tests/canvas_link.rs diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index dc3ee82..e51e1a3 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -63,7 +63,7 @@ jobs: sudo sed -i '1i host all all 127.0.0.1/32 trust\nhost all all ::1/128 trust' "$HBA" sudo -u postgres psql -qc 'SELECT pg_reload_conf()' createdb -h localhost copper_cloud_test_core - createdb -h localhost copper_cloud_test_canvas + createdb -h localhost ccmig_test psql -h localhost -d copper_cloud_test_core -tAc 'SELECT version()' # Build the portal first so the portal tests run against the real export. - uses: oven-sh/setup-bun@v2 @@ -77,6 +77,8 @@ jobs: echo "no portal/package.json; the binary serves the 'portal not built' page" fi - name: cargo test + env: + CCMIG_TEST_DATABASE_URL: postgres://localhost:5432/ccmig_test run: cargo test --workspace --locked -j4 -- --test-threads=4 - name: CLI smoke test run: | diff --git a/Cargo.lock b/Cargo.lock index aa3322e..5132d8e 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -531,6 +531,23 @@ version = "1.0.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "4e7648175b45a9a48536d676f68d918270699102aa8dab5496df06904c914600" +[[package]] +name = "cfg_aliases" +version = "0.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f079e83a288787bcd14a6aea84cee5c87a67c5a3e660c30f557a3d24761b3527" + +[[package]] +name = "chacha20" +version = "0.10.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "65c35e4b699c7e15ccbe7ee35c005e4fc0a278d22238a2857e6ce2dadeda1b06" +dependencies = [ + "cfg-if", + "cpufeatures 0.3.1", + "rand_core 0.10.1", +] + [[package]] name = "chrono" version = "0.4.45" @@ -627,7 +644,8 @@ dependencies = [ "axum", "axum-server", "clap", - "copper-cloud-canvas", + "copper-cloud-canvas-link", + "copper-cloud-canvas-migrate", "copper-cloud-core", "futures", "hex", @@ -640,24 +658,28 @@ dependencies = [ "time", "tokio", "tokio-rustls", + "tokio-tungstenite", "tracing", + "url", "uuid", "zeroize", ] [[package]] -name = "copper-cloud-canvas" +name = "copper-cloud-canvas-link" version = "0.8.1" dependencies = [ "anyhow", "axum", - "base64 0.22.1", + "axum-server", "bytes", "copper-cloud-core", "futures", - "governor", + "hex", "http-body-util", "metrics", + "reqwest", + "rustls", "serde", "serde_json", "sqlx", @@ -668,6 +690,31 @@ dependencies = [ "tracing", "url", "uuid", + "webpki-roots 1.0.9", + "zeroize", +] + +[[package]] +name = "copper-cloud-canvas-migrate" +version = "0.8.1" +dependencies = [ + "anyhow", + "axum", + "base64 0.22.1", + "bytes", + "copper-cloud-core", + "futures", + "hex", + "reqwest", + "serde", + "serde_json", + "sha2 0.10.9", + "sqlx", + "time", + "tokio", + "tracing", + "url", + "uuid", "yrs", ] @@ -1209,8 +1256,10 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ff2abc00be7fca6ebc474524697ae276ad847ad0a6b3faa4bcb027e9a4614ad0" dependencies = [ "cfg-if", + "js-sys", "libc", "wasi", + "wasm-bindgen", ] [[package]] @@ -1237,6 +1286,7 @@ dependencies = [ "js-sys", "libc", "r-efi 6.0.0", + "rand_core 0.10.1", "wasm-bindgen", ] @@ -1475,6 +1525,21 @@ dependencies = [ "want", ] +[[package]] +name = "hyper-rustls" +version = "0.27.10" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "dfa8e654703247911e29c23fbeaa261834bd9bb74efba2f9acddc37bfb127f53" +dependencies = [ + "http", + "hyper", + "hyper-util", + "rustls", + "tokio", + "tokio-rustls", + "tower-service", +] + [[package]] name = "hyper-util" version = "0.1.21" @@ -1769,6 +1834,12 @@ dependencies = [ "tracing-subscriber", ] +[[package]] +name = "lru-slab" +version = "0.1.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4050469837a6ff301cd14c1f8f24f88549e6d548f24f64e2148eb0f72cebc51f" + [[package]] name = "matchers" version = "0.2.0" @@ -2199,6 +2270,62 @@ dependencies = [ "winapi", ] +[[package]] +name = "quinn" +version = "0.11.12" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4051e23e9185c255a7e33ef59cdbca87a22d359052eecd22fc6b901fb37d9d11" +dependencies = [ + "bytes", + "cfg_aliases", + "pin-project-lite", + "quinn-proto", + "quinn-udp", + "rustc-hash", + "rustls", + "socket2", + "thiserror 2.0.21", + "tokio", + "tracing", + "web-time", +] + +[[package]] +name = "quinn-proto" +version = "0.11.19" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0e750cca55fe4f0439a15d0bb529da9651e79993e8e72c61a899a36d462befbe" +dependencies = [ + "bytes", + "getrandom 0.4.3", + "lru-slab", + "rand 0.10.3", + "rand_pcg", + "ring", + "rustc-hash", + "rustls", + "rustls-pki-types", + "slab", + "thiserror 2.0.21", + "tinyvec", + "tracing", + "web-time", +] + +[[package]] +name = "quinn-udp" +version = "0.5.16" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "af66907df18639dcf4db56ca65490cabc4b27a97dbadd96f2926cca73298f016" +dependencies = [ + "cfg_aliases", + "libc", + "once_cell", + "socket2", + "tracing", + "windows-sys 0.52.0", +] + [[package]] name = "quote" version = "1.0.47" @@ -2241,6 +2368,17 @@ dependencies = [ "rand_core 0.9.5", ] +[[package]] +name = "rand" +version = "0.10.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "65c9fb96cbc91e3478eaae79a69fcd3f1ae4ad052e471fe6732fff548984b4af" +dependencies = [ + "chacha20", + "getrandom 0.4.3", + "rand_core 0.10.1", +] + [[package]] name = "rand_chacha" version = "0.3.1" @@ -2279,6 +2417,21 @@ dependencies = [ "getrandom 0.3.4", ] +[[package]] +name = "rand_core" +version = "0.10.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "63b8176103e19a2643978565ca18b50549f6101881c443590420e4dc998a3c69" + +[[package]] +name = "rand_pcg" +version = "0.10.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "caa0f4137e1c0a72f4c651489402276c8e8e1cf081f3b0ba156d2cbeef09e86a" +dependencies = [ + "rand_core 0.10.1", +] + [[package]] name = "rand_xoshiro" version = "0.7.0" @@ -2392,16 +2545,21 @@ dependencies = [ "http-body", "http-body-util", "hyper", + "hyper-rustls", "hyper-util", "js-sys", "log", "percent-encoding", "pin-project-lite", + "quinn", + "rustls", + "rustls-pki-types", "serde", "serde_json", "serde_urlencoded", "sync_wrapper", "tokio", + "tokio-rustls", "tokio-util", "tower", "tower-http", @@ -2485,6 +2643,12 @@ dependencies = [ "walkdir", ] +[[package]] +name = "rustc-hash" +version = "2.1.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6b1e7f9a428571be2dc5bc0505c13fb6bf936822b894ec87abf8a08a4e51742d" + [[package]] name = "rusticata-macros" version = "4.1.0" @@ -2558,6 +2722,7 @@ version = "1.15.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "2f4925028c7eb5d1fcdaf196971378ed9d2c1c4efc7dc5d011256f76c99c0a96" dependencies = [ + "web-time", "zeroize", ] @@ -3257,8 +3422,12 @@ checksum = "8f72a05e828585856dacd553fba484c242c46e391fb0e58917c942ee9202915c" dependencies = [ "futures-util", "log", + "rustls", + "rustls-pki-types", "tokio", + "tokio-rustls", "tungstenite", + "webpki-roots 0.26.11", ] [[package]] @@ -3454,6 +3623,8 @@ dependencies = [ "httparse", "log", "rand 0.9.5", + "rustls", + "rustls-pki-types", "sha1", "thiserror 2.0.21", ] diff --git a/Cargo.toml b/Cargo.toml index d7e9c19..b728501 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -1,6 +1,11 @@ [workspace] resolver = "2" -members = ["crates/core", "crates/canvas", "crates/copper-cloud"] +members = [ + "crates/core", + "crates/canvas-link", + "crates/canvas-migrate", + "crates/copper-cloud", +] [workspace.package] version = "0.8.1" @@ -28,7 +33,8 @@ must_use_candidate = "allow" # which keeps x86_64 + aarch64 cross builds simple. [workspace.dependencies] copper-cloud-core = { path = "crates/core" } -copper-cloud-canvas = { path = "crates/canvas" } +copper-cloud-canvas-link = { path = "crates/canvas-link" } +copper-cloud-canvas-migrate = { path = "crates/canvas-migrate" } anyhow = "1" thiserror = "2" @@ -88,8 +94,11 @@ clap = { version = "4", features = ["derive", "env"] } # portal (static export embedded at build time; debug builds read the folder at run time) rust-embed = { version = "8", features = ["interpolate-folder-path", "mime-guess", "include-exclude"] } -# test-only +# canvas link (HTTP + WebSocket client to a linked Canvas Cloud; rustls `ring`, no aws-lc) reqwest = { version = "0.12", default-features = false, features = ["json", "stream"] } +tokio-tungstenite = { version = "0.29", default-features = false, features = ["connect", "handshake", "rustls-tls-webpki-roots"] } +url = "2" +webpki-roots = "1" [profile.release] lto = "thin" diff --git a/crates/canvas/Cargo.toml b/crates/canvas-link/Cargo.toml similarity index 61% rename from crates/canvas/Cargo.toml rename to crates/canvas-link/Cargo.toml index f9aa29c..3b34c62 100644 --- a/crates/canvas/Cargo.toml +++ b/crates/canvas-link/Cargo.toml @@ -1,6 +1,6 @@ [package] -name = "copper-cloud-canvas" -description = "Canvases REST API and Yjs WebSocket rooms for copper-cloud" +name = "copper-cloud-canvas-link" +description = "Canvas link for copper-cloud: proxies the canvas API to a linked Canvas Cloud, bridges its events and provisions users" version.workspace = true edition.workspace = true license.workspace = true @@ -18,24 +18,25 @@ workspace = true copper-cloud-core.workspace = true anyhow.workspace = true axum.workspace = true -base64.workspace = true bytes.workspace = true futures.workspace = true -governor.workspace = true +hex.workspace = true metrics.workspace = true +reqwest = { workspace = true, features = ["rustls-tls-manual-roots"] } +rustls.workspace = true serde.workspace = true serde_json.workspace = true sqlx.workspace = true time.workspace = true tokio.workspace = true +tokio-tungstenite.workspace = true tracing.workspace = true -url = "2" +url.workspace = true uuid.workspace = true -# yrs ≥ 0.18 ships the y-sync protocol (DefaultProtocol, Awareness, sync::Message) as -# `yrs::sync`; the standalone `y-sync` crate is frozen on yrs 0.17. -yrs = { version = "0.28", features = ["sync"] } +webpki-roots.workspace = true +zeroize.workspace = true [dev-dependencies] -tokio-tungstenite = "0.29" +axum-server.workspace = true tower.workspace = true http-body-util.workspace = true diff --git a/crates/canvas-link/src/admin.rs b/crates/canvas-link/src/admin.rs new file mode 100644 index 0000000..4253d77 --- /dev/null +++ b/crates/canvas-link/src/admin.rs @@ -0,0 +1,388 @@ +//! Admin API / CLI / doctor helpers: link status and changes (audited), the canvas listing and +//! deletion, overview stats and per-user canvas counts — all sourced from Canvas Cloud's +//! `/v1/apps/self/*`. + +use std::collections::HashMap; +use std::time::{Duration, Instant}; + +use axum::response::{IntoResponse, Response}; +use copper_cloud_core::error::{ApiError, ApiResult}; +use reqwest::{Method, StatusCode}; +use serde_json::{json, Value}; +use time::OffsetDateTime; +use uuid::Uuid; + +use crate::settings::{self, NewLink}; +use crate::upstream::{Probe, Stats, Upstream, UpstreamError, PROBE_TIMEOUT}; +use crate::{upstream_failed, Active, CanvasLink, Health}; + +/// A cached reachability result older than this is refreshed by `GET /admin/api/canvas-link`. +pub const HEALTH_TTL: Duration = Duration::from_secs(15); +/// How long per-user canvas counts are reused for the people list. +pub const COUNTS_TTL: Duration = Duration::from_secs(30); + +/// Who changed the link (audit + `updated_by`). +#[derive(Clone, Debug)] +pub enum Actor { + Admin { id: Uuid, email: String }, + Cli, +} + +impl Actor { + fn label(&self) -> &str { + match self { + Self::Admin { email, .. } => email, + Self::Cli => "cli", + } + } + + fn admin_id(&self) -> Option { + match self { + Self::Admin { id, .. } => Some(*id), + Self::Cli => None, + } + } +} + +/// Error of a link-backed admin call. +#[derive(Debug)] +pub enum LinkError { + Api(ApiError), + /// No Canvas Cloud linked (`503 canvas_unavailable`). + Unlinked, + /// Canvas Cloud failed (`502 canvas_upstream`). + Upstream(String), +} + +impl From for LinkError { + fn from(e: ApiError) -> Self { + Self::Api(e) + } +} + +impl From for LinkError { + fn from(e: UpstreamError) -> Self { + Self::Upstream(e.to_string()) + } +} + +impl IntoResponse for LinkError { + fn into_response(self) -> Response { + match self { + Self::Api(e) => e.into_response(), + Self::Unlinked => crate::unavailable(), + Self::Upstream(msg) => upstream_failed(msg), + } + } +} + +/// Write an `admin_audit` row (`admin_id` NULL for the CLI, with `detail.via = "cli"`). +pub async fn audit( + db: &sqlx::PgPool, + actor: &Actor, + action: &str, + target: &str, + detail: Value, +) -> ApiResult<()> { + let mut detail = detail; + if let (Actor::Cli, Value::Object(map)) = (actor, &mut detail) { + map.insert("via".into(), Value::from("cli")); + } + sqlx::query( + "INSERT INTO admin_audit (admin_id, action, target, detail) VALUES ($1, $2, $3, $4)", + ) + .bind(actor.admin_id()) + .bind(action) + .bind(target.chars().take(256).collect::()) + .bind(detail) + .execute(db) + .await?; + tracing::info!( + actor = actor.label(), + action, + audit_target = target, + "admin action" + ); + Ok(()) +} + +fn rfc3339(t: OffsetDateTime) -> String { + t.format(&time::format_description::well_known::Rfc3339) + .unwrap_or_default() +} + +/// `{migrated, failed, legacy_canvases}` from the migration ledger and the legacy table. +pub async fn migrations_summary(db: &sqlx::PgPool) -> ApiResult { + let (migrated, failed, legacy): (i64, i64, i64) = sqlx::query_as( + "SELECT (SELECT count(*) FROM canvas_migrations WHERE status = 'migrated'), + (SELECT count(*) FROM canvas_migrations WHERE status = 'failed'), + (SELECT count(*) FROM canvases)", + ) + .fetch_one(db) + .await?; + Ok(json!({ "migrated": migrated, "failed": failed, "legacy_canvases": legacy })) +} + +fn health_from(result: &Result) -> Health { + match result { + Ok(p) => Health { + reachable: Some(true), + version: p.version.clone(), + error: None, + checked_at: Some(OffsetDateTime::now_utc()), + }, + Err(e) => Health { + reachable: Some(false), + version: None, + error: Some(e.to_string()), + checked_at: Some(OffsetDateTime::now_utc()), + }, + } +} + +/// Probe the linked Canvas Cloud now and remember the result. +pub async fn check(link: &CanvasLink, active: &Active) -> Result { + let result = active.upstream.check(PROBE_TIMEOUT).await; + link.set_health(health_from(&result)); + result +} + +/// The `GET /admin/api/canvas-link` body (re-probes when the last check is stale). +pub async fn view(link: &CanvasLink) -> ApiResult { + let db = &link.state().db; + let current = link.current().await?; + let migrations = migrations_summary(db).await?; + let Some(active) = current else { + return Ok(json!({ + "linked": false, + "url": null, + "enabled": false, + "tls_fingerprint": null, + "linked_at": null, + "updated_at": null, + "updated_by": null, + "status": { "reachable": null, "version": null, "error": null, "checked_at": null }, + "bridge": { "connected": false, "last_event_at": null, "last_error": null }, + "migrations": migrations, + })); + }; + let stale = link.health().checked_at.is_none_or(|at| { + (OffsetDateTime::now_utc() - at) > time::Duration::try_from(HEALTH_TTL).unwrap_or_default() + }); + if stale { + let _ = check(link, &active).await; + } + let s = &active.settings; + Ok(json!({ + "linked": true, + "url": s.base(), + "enabled": s.enabled, + "tls_fingerprint": s.tls_fingerprint, + "linked_at": rfc3339(s.linked_at), + "updated_at": rfc3339(s.updated_at), + "updated_by": s.updated_by, + "status": link.health(), + "bridge": link.bridge_status(), + "migrations": migrations, + })) +} + +fn probe_failure(url: &str, err: &UpstreamError) -> ApiError { + let msg = match err { + UpstreamError::Transport(m) => format!("could not reach Canvas Cloud at {url}: {m}"), + UpstreamError::Status { status, .. } + if *status == StatusCode::UNAUTHORIZED || *status == StatusCode::FORBIDDEN => + { + format!("Canvas Cloud at {url} rejected the app key ({err})") + } + other => format!("Canvas Cloud at {url} did not pass the check ({other})"), + }; + ApiError::bad_request(msg) +} + +/// Validate, test (`/v1/info` + `/v1/apps/self/stats`) and store a link; audited as +/// `canvas_link.set`. Users are provisioned in the background by the reconcile loop (the +/// CLI provisions them itself unless `--no-sync`). Returns the stored settings. +pub async fn set( + link: &CanvasLink, + actor: &Actor, + url: &str, + app_key: &str, + tls_fingerprint: Option<&str>, +) -> ApiResult> { + let input = NewLink::parse(url, app_key, tls_fingerprint)?; + let base = settings::base_string(&input.url); + let probe_settings = settings::LinkSettings { + url: input.url.clone(), + app_key: input.app_key.clone(), + tls_fingerprint: input.tls_fingerprint.clone(), + enabled: true, + linked_at: OffsetDateTime::now_utc(), + updated_at: OffsetDateTime::now_utc(), + updated_by: None, + }; + let upstream = Upstream::new(&probe_settings).map_err(ApiError::Internal)?; + let probe = upstream + .check(PROBE_TIMEOUT) + .await + .map_err(|e| probe_failure(&base, &e))?; + let state = link.state(); + settings::save(&state.db, &state.crypto, &input, actor.label()).await?; + audit( + &state.db, + actor, + "canvas_link.set", + &base, + json!({ + "url": base, + "tls_fingerprint": input.tls_fingerprint, + "key_last4": settings::last4(&input.app_key), + "version": probe.version, + }), + ) + .await?; + let active = link + .reload() + .await? + .ok_or_else(|| ApiError::internal("canvas link vanished after save"))?; + link.set_health(health_from(&Ok(probe))); + Ok(active) +} + +/// Remove the link (`canvas_link.remove`). Returns the URL that was linked. +pub async fn remove(link: &CanvasLink, actor: &Actor) -> ApiResult> { + let db = &link.state().db; + let url = settings::remove(db).await?; + if let Some(u) = &url { + audit(db, actor, "canvas_link.remove", u, json!({ "url": u })).await?; + } + link.reload().await?; + Ok(url) +} + +/// `POST /admin/api/canvas-link/test` body (`canvas_link.test` audited). +pub async fn test(link: &CanvasLink, actor: &Actor) -> ApiResult { + let Some(active) = link.current().await? else { + return Ok(json!({ + "ok": false, "version": null, "latency_ms": null, + "error": "no Canvas Cloud is linked", + })); + }; + let result = check(link, &active).await; + let body = match &result { + Ok(p) => json!({ + "ok": true, + "version": p.version, + "latency_ms": u64::try_from(p.latency.as_millis()).unwrap_or(u64::MAX), + "error": null, + }), + Err(e) => { + json!({ "ok": false, "version": null, "latency_ms": null, "error": e.to_string() }) + } + }; + audit( + &link.state().db, + actor, + "canvas_link.test", + &active.settings.base(), + json!({ "ok": body["ok"], "error": body["error"] }), + ) + .await?; + Ok(body) +} + +/// Canvas Cloud stats for the overview (`None` when unlinked or unreachable). +pub async fn stats(link: &CanvasLink) -> Option { + let active = link.enabled().await.ok()??; + active.upstream.stats(PROBE_TIMEOUT).await.ok() +} + +/// Copper user id → canvas count (`None` when unlinked or unreachable). Cached briefly. +pub async fn canvas_counts(link: &CanvasLink) -> Option> { + if let Some((counts, at)) = link.counts_cache().as_ref() { + if at.elapsed() < COUNTS_TTL { + return Some(counts.clone()); + } + } + let active = link.enabled().await.ok()??; + let remote = crate::provision::remote_users(&active.upstream, PROBE_TIMEOUT) + .await + .ok()?; + let counts: HashMap = remote + .into_iter() + .filter_map(|(s, n)| Some((s.parse::().ok()?, n))) + .collect(); + *link.counts_cache() = Some((counts.clone(), Instant::now())); + Some(counts) +} + +/// One row of the admin canvas list (`GET /admin/api/canvases`), the shape the portal knows. +fn admin_canvas(c: &Value) -> Value { + let owner = c.get("owner").cloned().unwrap_or(Value::Null); + let owner_id = owner + .get("user_id") + .cloned() + .filter(|v| !v.is_null()) + .or_else(|| owner.get("subject").cloned()) + .unwrap_or(Value::Null); + json!({ + "id": c.get("id"), + "name": c.get("name"), + "kind": c.get("kind"), + "owner_id": owner_id, + "owner_email": owner.get("email"), + "member_count": c.get("member_count").and_then(Value::as_i64).unwrap_or(0), + "share_link_count": c.get("share_link_count").and_then(Value::as_i64).unwrap_or(0), + "created_at": c.get("created_at"), + "updated_at": c.get("updated_at"), + }) +} + +/// Canvases owned by Copper users, from `GET /v1/apps/self/canvases`. `Ok(None)` = unlinked. +pub async fn list_canvases( + link: &CanvasLink, + kind: Option<&str>, + limit: i64, + offset: i64, +) -> Result, i64)>, LinkError> { + let Some(active) = link.enabled().await? else { + return Ok(None); + }; + let mut path = format!("/v1/apps/self/canvases?limit={limit}&offset={offset}"); + if let Some(k) = kind { + path.push_str("&kind="); + path.push_str(k); + } + let resp: Value = Upstream::json(active.upstream.request(Method::GET, &path)).await?; + let items: Vec = resp + .get("canvases") + .or_else(|| resp.get("items")) + .and_then(Value::as_array) + .map(|a| a.iter().map(admin_canvas).collect()) + .unwrap_or_default(); + let total = resp + .get("total") + .and_then(Value::as_i64) + .unwrap_or_else(|| i64::try_from(items.len()).unwrap_or(i64::MAX)); + Ok(Some((items, total))) +} + +/// Delete one canvas on Canvas Cloud (`DELETE /v1/apps/self/canvases/{id}`; Canvas Cloud +/// closes its room with 4404 and notifies members). `Ok(None)` = not found; the returned +/// value is the canvas's name when Canvas Cloud reports it. +pub async fn delete_canvas( + link: &CanvasLink, + id: Uuid, +) -> Result>, LinkError> { + let Some(active) = link.enabled().await? else { + return Err(LinkError::Unlinked); + }; + let path = format!("/v1/apps/self/canvases/{id}"); + match Upstream::json::(active.upstream.request(Method::DELETE, &path)).await { + Ok(v) => Ok(Some( + v.get("name").and_then(Value::as_str).map(str::to_owned), + )), + Err(e) if e.status() == Some(StatusCode::NOT_FOUND) => Ok(None), + Err(e) => Err(e.into()), + } +} diff --git a/crates/canvas-link/src/bridge.rs b/crates/canvas-link/src/bridge.rs new file mode 100644 index 0000000..32c51c7 --- /dev/null +++ b/crates/canvas-link/src/bridge.rs @@ -0,0 +1,361 @@ +//! Event bridge (CONTRACT §7.3): while a Canvas Cloud is linked, one SSE client on +//! `GET {url}/v1/apps/self/events` (`Last-Event-ID` resume, exponential backoff 1 s → 30 s). +//! Each `canvas` event is re-published to the Copper user whose id is the event's `subject` +//! as `Event::Canvas {canvas_id, kind}`, so Copper's `/v1/sync/events` stream carries the same +//! `event: canvas` frames as before the split. `resync` is logged (clients refetch on their +//! own); keep-alive comments only feed the idle watchdog. + +use std::time::Duration; + +use copper_cloud_core::events::Event; +use futures::StreamExt as _; +use reqwest::header::{ACCEPT, AUTHORIZATION}; +use serde::Deserialize; +use time::OffsetDateTime; +use uuid::Uuid; + +use crate::upstream::{error_fields, root_cause, CONNECT_TIMEOUT}; +use crate::{Active, CanvasLink}; + +/// Reconnect backoff bounds. +pub const BACKOFF_MIN: Duration = Duration::from_secs(1); +pub const BACKOFF_MAX: Duration = Duration::from_secs(30); +/// Canvas Cloud sends a keep-alive every 15 s; reconnect after this much silence. +pub const IDLE_TIMEOUT: Duration = Duration::from_secs(45); +/// How often a live session re-checks the settings (CLI edits from another process). +const SETTINGS_POLL: Duration = Duration::from_secs(5); +/// Largest single SSE event we buffer. +const MAX_EVENT_BYTES: usize = 256 * 1024; + +/// One dispatched SSE event. +#[derive(Debug, Default, PartialEq, Eq)] +pub struct SseEvent { + pub event: String, + pub data: String, + pub id: Option, +} + +/// Incremental `text/event-stream` parser (WHATWG rules: `event`, `data` (joined by `\n`), +/// `id`; comments and `retry` ignored; CR, LF and CRLF line ends). +#[derive(Debug, Default)] +pub struct SseParser { + buf: Vec, + event: String, + data: Vec, + id: Option, + has_fields: bool, +} + +impl SseParser { + /// Feed bytes; returns the events completed by them. + pub fn feed(&mut self, chunk: &[u8]) -> Vec { + self.buf.extend_from_slice(chunk); + let mut out = Vec::new(); + while let Some(pos) = self.buf.iter().position(|b| *b == b'\n' || *b == b'\r') { + // A lone trailing `\r` might be the first half of `\r\n`: wait for more. + if self.buf[pos] == b'\r' && pos + 1 == self.buf.len() { + break; + } + let skip = if self.buf[pos] == b'\r' && self.buf.get(pos + 1) == Some(&b'\n') { + 2 + } else { + 1 + }; + let line: Vec = self.buf.drain(..pos + skip).take(pos).collect(); + let line = String::from_utf8_lossy(&line); + if let Some(ev) = self.line(&line) { + out.push(ev); + } + } + if self.buf.len() > MAX_EVENT_BYTES { + self.buf.clear(); + } + out + } + + fn line(&mut self, line: &str) -> Option { + if line.is_empty() { + if !self.has_fields { + return None; + } + self.has_fields = false; + let ev = SseEvent { + event: if self.event.is_empty() { + "message".to_owned() + } else { + std::mem::take(&mut self.event) + }, + data: std::mem::take(&mut self.data).join("\n"), + id: self.id.clone(), + }; + return Some(ev); + } + if line.starts_with(':') { + return None; + } + let (field, value) = match line.split_once(':') { + Some((f, v)) => (f, v.strip_prefix(' ').unwrap_or(v)), + None => (line, ""), + }; + match field { + "event" => { + value.clone_into(&mut self.event); + self.has_fields = true; + } + "data" => { + if self.data.iter().map(String::len).sum::() < MAX_EVENT_BYTES { + self.data.push(value.to_owned()); + } + self.has_fields = true; + } + "id" if !value.contains('\0') => { + self.id = Some(value.to_owned()); + self.has_fields = true; + } + _ => {} + } + None + } +} + +#[derive(Deserialize)] +struct CanvasEventData { + subject: Option, + user_id: Option, + canvas_id: Uuid, + kind: String, +} + +/// Map one SSE event to `(copper user id, Event::Canvas)`. The subject is the Copper user id +/// (provisioning uses it as both subject and id hint); `user_id` is the fallback. +pub fn canvas_event(ev: &SseEvent) -> Option<(Uuid, Event)> { + if ev.event != "canvas" { + return None; + } + let data: CanvasEventData = serde_json::from_str(&ev.data).ok()?; + let user = data + .subject + .as_deref() + .and_then(|s| s.parse::().ok()) + .or(data.user_id)?; + Some(( + user, + Event::Canvas { + canvas_id: data.canvas_id, + kind: data.kind, + }, + )) +} + +/// The bridge supervisor: runs for the life of the process. +pub async fn run(link: CanvasLink) { + let mut generation = link.subscribe(); + let mut shutdown = copper_cloud_core::shutdown::subscribe(); + let mut last_id: Option = None; + let mut last_base: Option = None; + let mut backoff = BACKOFF_MIN; + loop { + if copper_cloud_core::shutdown::is_shutting_down() { + break; + } + let active = match link.enabled().await { + Ok(a) => a, + Err(err) => { + tracing::warn!(error = %err, "canvas link settings unavailable"); + None + } + }; + let Some(active) = active else { + link.update_bridge(|b| { + b.connected = false; + b.last_error = None; + }); + tokio::select! { + _ = generation.changed() => {} + () = tokio::time::sleep(SETTINGS_POLL) => {} + _ = shutdown.changed() => {} + } + continue; + }; + if last_base.as_deref() != Some(active.upstream.base()) { + last_id = None; + last_base = Some(active.upstream.base().to_owned()); + } + let result = tokio::select! { + r = session(&link, &active, &mut last_id, &mut backoff) => Some(r), + () = settings_changed(&link, &active) => None, + _ = shutdown.changed() => None, + }; + link.update_bridge(|b| b.connected = false); + match result { + None => backoff = BACKOFF_MIN, + Some(Err(err)) => { + tracing::warn!(error = %err, retry_in_s = backoff.as_secs(), "canvas event bridge disconnected"); + link.update_bridge(|b| b.last_error = Some(err)); + tokio::select! { + () = tokio::time::sleep(backoff) => {} + () = settings_changed(&link, &active) => {} + _ = shutdown.changed() => {} + } + backoff = (backoff * 2).min(BACKOFF_MAX); + } + Some(Ok(())) => {} + } + } + link.update_bridge(|b| b.connected = false); +} + +/// Resolves when the effective link is no longer `active` (unlinked, re-keyed, …). +async fn settings_changed(link: &CanvasLink, active: &std::sync::Arc) { + loop { + tokio::time::sleep(SETTINGS_POLL).await; + match link.enabled().await { + Ok(Some(now)) if std::sync::Arc::ptr_eq(&now, active) => {} + Ok(_) => return, + Err(_) => {} + } + } +} + +/// One SSE connection until it ends. `Ok(())` = the server closed the stream cleanly. +async fn session( + link: &CanvasLink, + active: &Active, + last_id: &mut Option, + backoff: &mut Duration, +) -> Result<(), String> { + let up = &active.upstream; + let mut rb = up + .client() + .get(up.url("/v1/apps/self/events")) + .header(AUTHORIZATION, up.auth_header()) + .header(ACCEPT, "text/event-stream"); + if let Some(id) = last_id.as_deref() { + rb = rb.header("last-event-id", id); + } + let resp = tokio::time::timeout(CONNECT_TIMEOUT * 3, rb.send()) + .await + .map_err(|_| "Canvas Cloud event stream did not answer in time".to_owned())? + .map_err(|e| format!("Canvas Cloud event stream unreachable ({})", root_cause(&e)))?; + let status = resp.status(); + if !status.is_success() { + let body = resp.bytes().await.unwrap_or_default(); + let (code, message) = error_fields(&body); + return Err(format!( + "Canvas Cloud event stream answered {}{}{}", + status.as_u16(), + code.map(|c| format!(" {c}")).unwrap_or_default(), + message.map(|m| format!(": {m}")).unwrap_or_default() + )); + } + *backoff = BACKOFF_MIN; + link.update_bridge(|b| { + b.connected = true; + b.last_error = None; + }); + tracing::info!(url = up.base(), "canvas event bridge connected"); + let state = link.state(); + let mut parser = SseParser::default(); + let mut stream = resp.bytes_stream(); + loop { + let chunk = match tokio::time::timeout(IDLE_TIMEOUT, stream.next()).await { + Err(_) => return Err("Canvas Cloud event stream went silent".into()), + Ok(None) => return Ok(()), + Ok(Some(Err(e))) => { + return Err(format!( + "Canvas Cloud event stream broke ({})", + root_cause(&e) + )) + } + Ok(Some(Ok(c))) => c, + }; + for ev in parser.feed(&chunk) { + if ev.id.is_some() { + last_id.clone_from(&ev.id); + } + match ev.event.as_str() { + "canvas" => { + if let Some((user, event)) = canvas_event(&ev) { + state.events.publish(user, event); + metrics::counter!("canvas_link_events_total").increment(1); + } else { + tracing::debug!(data = %ev.data, "ignoring undecodable canvas event"); + } + link.update_bridge(|b| b.last_event_at = Some(OffsetDateTime::now_utc())); + } + "resync" => { + tracing::info!("canvas event bridge: resync (events were missed)"); + link.update_bridge(|b| b.last_event_at = Some(OffsetDateTime::now_utc())); + } + _ => {} + } + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn parses_sse() { + let mut p = SseParser::default(); + let evs = p.feed(b": keepalive\n\nevent: canvas\nid: 7\ndata: {\"a\":1}\n\nevent: resync\r\ndata\r\n\r\n"); + assert_eq!( + evs, + vec![ + SseEvent { + event: "canvas".into(), + data: "{\"a\":1}".into(), + id: Some("7".into()) + }, + SseEvent { + event: "resync".into(), + data: String::new(), + id: Some("7".into()) + }, + ] + ); + // Split across chunks, including inside a CRLF. + let mut p = SseParser::default(); + assert_eq!(p.feed(b"event: can"), Vec::::new()); + assert_eq!(p.feed(b"vas\r"), Vec::::new()); + assert_eq!(p.feed(b"\ndata: x\ndata: y\r"), Vec::::new()); + let evs = p.feed(b"\n\r\n"); + assert_eq!(evs.len(), 1); + assert_eq!(evs[0].data, "x\ny"); + assert_eq!(evs[0].id, None); + } + + #[test] + fn maps_canvas_events() { + let user = Uuid::now_v7(); + let canvas = Uuid::now_v7(); + let ev = SseEvent { + event: "canvas".into(), + data: format!( + r#"{{"subject":"{user}","user_id":"{}","canvas_id":"{canvas}","kind":"renamed"}}"#, + Uuid::nil() + ), + id: Some("1".into()), + }; + let (u, e) = canvas_event(&ev).unwrap(); + assert_eq!(u, user); + assert!( + matches!(e, Event::Canvas { canvas_id, ref kind } if canvas_id == canvas && kind == "renamed") + ); + let ev = SseEvent { + event: "canvas".into(), + data: format!( + r#"{{"subject":"x","user_id":"{user}","canvas_id":"{canvas}","kind":"update"}}"# + ), + id: None, + }; + assert_eq!(canvas_event(&ev).unwrap().0, user); + assert!(canvas_event(&SseEvent { + event: "resync".into(), + ..SseEvent::default() + }) + .is_none()); + } +} diff --git a/crates/canvas-link/src/info.rs b/crates/canvas-link/src/info.rs new file mode 100644 index 0000000..f48bec6 --- /dev/null +++ b/crates/canvas-link/src/info.rs @@ -0,0 +1,34 @@ +//! `GET /v1/info` gains `"canvas": {"linked": bool}` (additive) without touching core: an outer +//! middleware rewrites the JSON body of a successful `/v1/info` response. + +use axum::body::Body; +use axum::extract::{Request, State}; +use axum::http::{header, Method, StatusCode}; +use axum::middleware::Next; +use axum::response::Response; +use serde_json::{json, Value}; + +use crate::CanvasLink; + +const MAX_INFO_BYTES: usize = 64 * 1024; + +/// Middleware (`axum::middleware::from_fn_with_state(link, info_layer)`) for the whole app. +pub async fn info_layer(State(link): State, req: Request, next: Next) -> Response { + let is_info = req.method() == Method::GET && req.uri().path() == "/v1/info"; + let resp = next.run(req).await; + if !is_info || resp.status() != StatusCode::OK { + return resp; + } + let (mut parts, body) = resp.into_parts(); + let Ok(bytes) = axum::body::to_bytes(body, MAX_INFO_BYTES).await else { + return Response::from_parts(parts, Body::empty()); + }; + let Ok(Value::Object(mut map)) = serde_json::from_slice::(&bytes) else { + return Response::from_parts(parts, Body::from(bytes)); + }; + let linked = matches!(link.enabled().await, Ok(Some(_))); + map.insert("canvas".into(), json!({ "linked": linked })); + let out = serde_json::to_vec(&Value::Object(map)).unwrap_or_else(|_| bytes.to_vec()); + parts.headers.remove(header::CONTENT_LENGTH); + Response::from_parts(parts, Body::from(out)) +} diff --git a/crates/canvas-link/src/landing.rs b/crates/canvas-link/src/landing.rs new file mode 100644 index 0000000..8a97410 --- /dev/null +++ b/crates/canvas-link/src/landing.rs @@ -0,0 +1,207 @@ +//! `GET /join/{token}` — the public landing page for canvas share links, moved unchanged from +//! the old canvas crate. It is merged at the application root (outside `/v1`, so no instance +//! key is needed) and deliberately touches neither the database nor Canvas Cloud: it only +//! checks the token's shape and hands it to Copper as +//! `copper://canvas/join/{token}?cloud={host}`. + +use axum::extract::Path; +use axum::http::{header, HeaderMap, HeaderValue, Uri}; +use axum::response::{Html, IntoResponse, Response}; +use copper_cloud_core::ids; + +fn token_digest(token: &str) -> Option<[u8; 32]> { + ids::is_token_shape(token).then(|| ids::sha256(token.as_bytes())) +} + +fn escape_html(value: &str) -> String { + value + .replace('&', "&") + .replace('<', "<") + .replace('>', ">") + .replace('"', """) + .replace('\'', "'") +} + +fn percent_encode_component(value: &str) -> String { + let mut out = String::with_capacity(value.len()); + for byte in value.bytes() { + if byte.is_ascii_alphanumeric() || matches!(byte, b'-' | b'.' | b'_' | b'~') { + out.push(char::from(byte)); + } else { + use std::fmt::Write as _; + let _ = write!(out, "%{byte:02X}"); + } + } + out +} + +/// Accept only a host name/IP and an optional numeric port. In particular, reject user-info, +/// paths and control characters before putting the value in a custom URL scheme. +fn validated_host(headers: &HeaderMap, uri: &Uri) -> Option { + // HTTP/1.1 carries the target in a literal `Host` header; HTTP/2 (our + // only transport once TLS is on, since rustls negotiates h2 by ALPN) + // forbids that header and sends `:authority` instead, which hyper/h2 + // surface as the request URI's authority, not a header — so a host + // taken from headers alone is always absent on this listener. + let from_header = headers + .get(header::HOST) + .and_then(|v| v.to_str().ok()) + .map(str::to_owned); + let from_authority = uri.authority().map(|a| a.as_str().to_owned()); + let raw = from_header.or(from_authority)?; + let raw = raw.as_str(); + if raw.is_empty() + || raw.trim() != raw + || raw.chars().any(|c| c.is_control() || c.is_whitespace()) + { + return None; + } + if raw.contains(['@', '/', '?', '#', '%']) { + return None; + } + let host = if let Some(rest) = raw.strip_prefix('[') { + let end = rest.find(']')?; + let host = &rest[..end]; + let suffix = &rest[end + 1..]; + if suffix.is_empty() { + host + } else { + let port = suffix.strip_prefix(':').filter(|p| !p.is_empty())?; + if port.parse::().is_err() { + return None; + } + host + } + } else { + if raw.matches(':').count() > 1 { + return None; + } + match raw.rsplit_once(':') { + Some((host, port)) if !port.is_empty() => { + if port.parse::().is_err() { + return None; + } + host + } + Some((_, _)) => return None, + None => raw, + } + }; + if host.is_empty() + || host.parse::().is_err() && url::Host::parse(host).is_err() + { + return None; + } + Some(raw.to_owned()) +} + +/// `GET /join/:token` deliberately does not consult the database. It is outside `/v1`, so it is +/// reachable before a client has an instance key and can hand the opaque token to Copper. +pub async fn join_landing(Path(token): Path, uri: Uri, headers: HeaderMap) -> Response { + let href = token_digest(&token).and_then(|_| { + validated_host(&headers, &uri).map(|host| { + format!( + "copper://canvas/join/{}?cloud={}", + percent_encode_component(&token), + host + ) + }) + }); + let html = match href { + Some(href) => format!( + "Open canvas in Copper

Canvas share link

Open this canvas in Copper to continue.

Open this canvas in Copper
", + escape_html(&href) + ), + None => "Open canvas in Copper

Canvas share link

Open this link in Copper to continue.

".to_owned(), + }; + let mut response = Html(html).into_response(); + let headers = response.headers_mut(); + headers.insert(header::CACHE_CONTROL, HeaderValue::from_static("no-store")); + headers.insert( + header::REFERRER_POLICY, + HeaderValue::from_static("no-referrer"), + ); + headers.insert( + header::X_CONTENT_TYPE_OPTIONS, + HeaderValue::from_static("nosniff"), + ); + headers.insert( + header::CONTENT_SECURITY_POLICY, + HeaderValue::from_static("default-src 'none'; style-src 'unsafe-inline'"), + ); + response +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn host_falls_back_to_authority_when_no_host_header() { + // HTTP/2 requests never carry a literal `Host` header (RFC 9113 §8.3.1 + // forbids it); hyper/h2 surface the `:authority` pseudo-header as the + // request URI's authority instead. The join landing page is only ever + // served over h2 once TLS is on, so this path must work from the URI + // alone, not just from headers. + let empty = HeaderMap::new(); + let uri: Uri = "https://127.0.0.1:8445/join/abc".parse().unwrap(); + assert_eq!( + validated_host(&empty, &uri).as_deref(), + Some("127.0.0.1:8445") + ); + + // A literal Host header (HTTP/1.1) still wins when present. + let mut headers = HeaderMap::new(); + headers.insert(header::HOST, HeaderValue::from_static("example.com")); + assert_eq!( + validated_host(&headers, &uri).as_deref(), + Some("example.com") + ); + + // Neither a header nor a URI authority: no host to offer. + let relative: Uri = "/join/abc".parse().unwrap(); + assert_eq!(validated_host(&empty, &relative), None); + } + + async fn landing(token: &str, host: Option<&str>) -> (axum::http::HeaderMap, String) { + let mut headers = HeaderMap::new(); + if let Some(h) = host { + headers.insert(header::HOST, HeaderValue::from_str(h).unwrap()); + } + let uri: Uri = format!("/join/{token}").parse().unwrap(); + let resp = join_landing(Path(token.to_owned()), uri, headers).await; + assert_eq!(resp.status(), axum::http::StatusCode::OK); + let headers = resp.headers().clone(); + let body = axum::body::to_bytes(resp.into_body(), 1 << 20) + .await + .unwrap(); + (headers, String::from_utf8(body.to_vec()).unwrap()) + } + + // Moved from the old canvas crate's `tests/rest.rs` share-link test. + #[tokio::test] + async fn landing_page_links_to_copper() { + let token = ids::random_token(); + let (headers, html) = landing(&token, Some("cloud.example:443")).await; + assert_eq!(headers["cache-control"], "no-store"); + assert_eq!(headers["referrer-policy"], "no-referrer"); + assert_eq!(headers["x-content-type-options"], "nosniff"); + assert_eq!( + headers["content-security-policy"], + "default-src 'none'; style-src 'unsafe-inline'" + ); + assert!(html.contains("copper://canvas/join/")); + assert!(html.contains("cloud=cloud.example:443")); + + let (_, html) = landing(&token, Some("[::1]")).await; + assert!(html.contains("cloud=[::1]")); + + let (_, html) = landing(&token, Some("")).await; + assert!(!html.contains("copper://")); + assert!(!html.contains("")); + + // Not a token: no link either. + let (_, html) = landing("not-a-token", Some("cloud.example")).await; + assert!(!html.contains("copper://")); + } +} diff --git a/crates/canvas-link/src/lib.rs b/crates/canvas-link/src/lib.rs new file mode 100644 index 0000000..a6ee187 --- /dev/null +++ b/crates/canvas-link/src/lib.rs @@ -0,0 +1,338 @@ +//! Canvas link: copper-cloud's thin client for a linked **Canvas Cloud** server. +//! +//! When an instance admin links a Canvas Cloud (URL + app key, see [`settings`]), copper-cloud +//! serves the unchanged canvas API to Copper by proxying it there, acting as the signed-in +//! Copper user (`Authorization: Bearer ` + `X-Canvas-User: `): +//! +//! * [`CanvasLink::router`] — `/v1/canvases…` (incl. the y-websocket pipe, see [`ws`]), +//! `/v1/invites…`, `/v1/canvas-links/…`, `/v1/mentions…`, `/v1/people` (see [`proxy`]); +//! * [`bridge`] — Canvas Cloud's app event stream re-published as `canvas` events on each +//! user's `/v1/sync/events`; +//! * [`provision`] — Copper users mirrored as Canvas Cloud app users (same ids), JIT on +//! `unknown_user`, a reconcile loop and an outbox for failures; +//! * [`admin`] — the admin API / CLI / doctor helpers; +//! * [`landing`] — the `copper://` share-link landing page (`/join/{token}`); +//! * [`info_layer`] — adds `"canvas": {"linked": bool}` to `GET /v1/info`. +//! +//! Unlinked, every canvas route answers `503 canvas_unavailable`; Canvas Cloud unreachable, +//! `502 canvas_upstream`. + +use std::collections::HashMap; +use std::sync::atomic::{AtomicBool, Ordering}; +use std::sync::{Arc, Mutex, PoisonError, RwLock}; +use std::time::{Duration, Instant}; + +use axum::http::StatusCode; +use axum::response::{IntoResponse, Response}; +use axum::routing::{any, get}; +use axum::{Extension, Router}; +use copper_cloud_core::error::{ApiError, ApiResult, ErrorLog}; +use copper_cloud_core::state::SharedState; +use serde_json::json; +use time::OffsetDateTime; +use tokio::sync::{watch, Notify}; +use uuid::Uuid; + +pub mod admin; +pub mod bridge; +mod info; +pub mod landing; +pub mod provision; +mod proxy; +pub mod settings; +pub mod tls; +pub mod upstream; +pub mod ws; + +pub use info::info_layer; +pub use settings::LinkSettings; +pub use upstream::{Stats, Upstream, UpstreamError}; + +/// Largest accepted REST request body (except ops) — same as the old canvas router. +pub const MAX_BODY_BYTES: usize = 64 * 1024; +/// Largest accepted `POST /canvases/{id}/ops` body (a 2 MiB `data:` image plus JSON). +pub const MAX_OPS_BODY_BYTES: usize = 4 << 20; +/// How long the decrypted settings are cached before re-reading the row (CLI edits are +/// picked up within this window; admin API edits immediately). +pub const SETTINGS_TTL: Duration = Duration::from_secs(5); + +/// The link as currently configured: settings + a client built for them. +#[derive(Debug)] +pub struct Active { + pub settings: LinkSettings, + pub upstream: Upstream, +} + +impl Active { + pub fn new(settings: LinkSettings) -> anyhow::Result { + let upstream = Upstream::new(&settings)?; + Ok(Self { settings, upstream }) + } +} + +/// Event-bridge status (admin API / doctor). +#[derive(Clone, Debug, Default, serde::Serialize)] +pub struct BridgeStatus { + pub connected: bool, + #[serde(with = "time::serde::rfc3339::option")] + pub last_event_at: Option, + pub last_error: Option, +} + +/// Last reachability check of the linked Canvas Cloud. +#[derive(Clone, Debug, Default, serde::Serialize)] +pub struct Health { + pub reachable: Option, + pub version: Option, + pub error: Option, + #[serde(with = "time::serde::rfc3339::option")] + pub checked_at: Option, +} + +type Cached = (Option>, Instant); + +struct Inner { + state: SharedState, + cache: RwLock>, + refresh: tokio::sync::Mutex<()>, + generation: watch::Sender, + closing: watch::Sender, + bridge: Mutex, + health: Mutex, + outbox_kick: Notify, + /// Per-user hash of what was last pushed to Canvas Cloud (reconcile). + synced: Mutex>, + /// `subject → canvas_count` from `/v1/apps/self/users`, briefly cached for the people list. + counts: Mutex, Instant)>>, + started: AtomicBool, +} + +/// Handle to the canvas link of one copper-cloud process. Cheap to clone. +#[derive(Clone)] +pub struct CanvasLink { + inner: Arc, +} + +impl std::fmt::Debug for CanvasLink { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_struct("CanvasLink").finish_non_exhaustive() + } +} + +fn lock(m: &Mutex) -> std::sync::MutexGuard<'_, T> { + m.lock().unwrap_or_else(PoisonError::into_inner) +} + +impl CanvasLink { + pub fn new(state: SharedState) -> Self { + Self { + inner: Arc::new(Inner { + state, + cache: RwLock::new(None), + refresh: tokio::sync::Mutex::new(()), + generation: watch::channel(0).0, + closing: watch::channel(false).0, + bridge: Mutex::new(BridgeStatus::default()), + health: Mutex::new(Health::default()), + outbox_kick: Notify::new(), + synced: Mutex::new(HashMap::new()), + counts: Mutex::new(None), + started: AtomicBool::new(false), + }), + } + } + + pub fn state(&self) -> &SharedState { + &self.inner.state + } + + /// The `/v1` canvas routes (mount under `/v1`, where the canvas router used to be, so the + /// instance gate, tracing/metrics and rate limits run first). Any method. + pub fn router(&self) -> Router { + Router::new() + .route("/canvases", any(proxy::proxy)) + .route("/canvases/{*rest}", any(proxy::proxy)) + .route("/people", any(proxy::proxy)) + .route("/invites", any(proxy::proxy)) + .route("/invites/{*rest}", any(proxy::proxy)) + .route("/canvas-links/{*rest}", any(proxy::proxy)) + .route("/mentions", any(proxy::proxy)) + .route("/mentions/{*rest}", any(proxy::proxy)) + .layer(Extension(self.clone())) + } + + /// The public `copper://` landing page for share links (merge at the application root — + /// outside `/v1`, so the instance gate does not run for it). + pub fn landing_router() -> Router { + Router::new().route("/join/{token}", get(landing::join_landing)) + } + + fn peek(&self) -> Option { + self.inner + .cache + .read() + .unwrap_or_else(PoisonError::into_inner) + .clone() + } + + async fn load(&self, force: bool) -> ApiResult>> { + if !force { + if let Some((v, at)) = self.peek() { + if at.elapsed() < SETTINGS_TTL { + return Ok(v); + } + } + } + let _single = self.inner.refresh.lock().await; + let prev = self.peek(); + if !force { + if let Some((v, at)) = &prev { + if at.elapsed() < SETTINGS_TTL { + return Ok(v.clone()); + } + } + } + let state = &self.inner.state; + let loaded = settings::load(&state.db, &state.crypto).await?; + let prev_active = prev.and_then(|(v, _)| v); + let next = match (loaded, &prev_active) { + (None, _) => None, + (Some(s), Some(p)) + if p.settings.same_target(&s) + && p.settings.enabled == s.enabled + && p.settings.updated_at == s.updated_at => + { + Some(p.clone()) + } + (Some(s), _) => Some(Arc::new(Active::new(s).map_err(ApiError::Internal)?)), + }; + let changed = match (&prev_active, &next) { + (None, None) => false, + (Some(a), Some(b)) => !Arc::ptr_eq(a, b), + _ => true, + }; + *self + .inner + .cache + .write() + .unwrap_or_else(PoisonError::into_inner) = Some((next.clone(), Instant::now())); + if changed { + self.inner.generation.send_modify(|g| *g += 1); + *lock(&self.inner.counts) = None; + // A new target (or none) starts from scratch: everyone is pushed again. + lock(&self.inner.synced).clear(); + if next.is_none() { + *lock(&self.inner.health) = Health::default(); + } + } + Ok(next) + } + + /// The stored link (enabled or not), cached for [`SETTINGS_TTL`]. + pub async fn current(&self) -> ApiResult>> { + self.load(false).await + } + + /// The link if one is stored **and** enabled. + pub async fn enabled(&self) -> ApiResult>> { + Ok(self.current().await?.filter(|a| a.settings.enabled)) + } + + /// Re-read the settings now (after this process changed them) and wake the background + /// tasks. + pub async fn reload(&self) -> ApiResult>> { + let v = self.load(true).await?; + self.inner.outbox_kick.notify_one(); + Ok(v) + } + + /// Bumps whenever the effective link changes (linked, unlinked, re-keyed). + pub fn subscribe(&self) -> watch::Receiver { + self.inner.generation.subscribe() + } + + /// Close every proxied WebSocket with `1001` (server shutting down). + pub fn close_all(&self) { + self.inner.closing.send_replace(true); + } + + pub(crate) fn closing(&self) -> watch::Receiver { + self.inner.closing.subscribe() + } + + pub fn bridge_status(&self) -> BridgeStatus { + lock(&self.inner.bridge).clone() + } + + pub(crate) fn update_bridge(&self, f: impl FnOnce(&mut BridgeStatus)) { + f(&mut lock(&self.inner.bridge)); + } + + pub fn health(&self) -> Health { + lock(&self.inner.health).clone() + } + + pub(crate) fn set_health(&self, h: Health) { + *lock(&self.inner.health) = h; + } + + /// Ask the outbox worker to run now (after enqueueing work). + pub fn kick_outbox(&self) { + self.inner.outbox_kick.notify_one(); + } + + pub(crate) async fn outbox_wait(&self, max: Duration) { + let _ = tokio::time::timeout(max, self.inner.outbox_kick.notified()).await; + } + + pub(crate) fn synced(&self) -> std::sync::MutexGuard<'_, HashMap> { + lock(&self.inner.synced) + } + + pub(crate) fn counts_cache( + &self, + ) -> std::sync::MutexGuard<'_, Option<(HashMap, Instant)>> { + lock(&self.inner.counts) + } + + /// Start the background tasks once: the event bridge, the reconcile loop and the outbox + /// worker. No-op outside a tokio runtime or when already started. + pub fn start(&self) { + if tokio::runtime::Handle::try_current().is_err() + || self.inner.started.swap(true, Ordering::SeqCst) + { + return; + } + tokio::spawn(bridge::run(self.clone())); + tokio::spawn(provision::reconcile_loop(self.clone())); + tokio::spawn(provision::outbox_loop(self.clone())); + } +} + +/// `503 {"error":"canvas_unavailable"}` — no Canvas Cloud is linked. +pub fn unavailable() -> Response { + error_response( + StatusCode::SERVICE_UNAVAILABLE, + "canvas_unavailable", + "no Canvas Cloud is linked to this Copper Cloud", + ) +} + +/// `502 {"error":"canvas_upstream"}` — the linked Canvas Cloud failed us. +pub fn upstream_failed(message: impl Into) -> Response { + error_response(StatusCode::BAD_GATEWAY, "canvas_upstream", message) +} + +fn error_response(status: StatusCode, code: &'static str, message: impl Into) -> Response { + let message = message.into(); + let mut resp = ( + status, + axum::Json(json!({ "error": code, "message": message })), + ) + .into_response(); + resp.extensions_mut().insert(ErrorLog { + code, + message: Some(copper_cloud_core::error::loggable_message(&message)), + }); + resp +} diff --git a/crates/canvas-link/src/provision.rs b/crates/canvas-link/src/provision.rs new file mode 100644 index 0000000..8148c02 --- /dev/null +++ b/crates/canvas-link/src/provision.rs @@ -0,0 +1,509 @@ +//! Provisioning (CONTRACT §7.4): every Copper user exists on Canvas Cloud as an app user whose +//! subject **and** id are the Copper `users.id`, so member lists, chat `authorId`s and mention +//! ids inside documents stay valid. +//! +//! * [`sync_all`] — bulk `users:batch` (on link, `canvas sync-users`, and the first reconcile). +//! * [`provision_user`] — one `PUT …/users/{subject}` (JIT after `401 unknown_user`). +//! * [`reconcile_loop`] — every 10 min (and whenever the link changes): push users whose +//! email / display name / disabled flag changed (a hash per user is kept in memory), +//! disconnect newly disabled users, delete app users whose Copper account is gone. +//! * The outbox (`canvas_link_outbox`): admin edits/deletes enqueue work in their own +//! transaction ([`enqueue_user_changed`], [`enqueue_user_deleted`]); [`outbox_loop`] pushes +//! it and retries failures with backoff, so nothing is lost while Canvas Cloud is down. + +use std::collections::{HashMap, HashSet}; +use std::hash::{Hash as _, Hasher as _}; +use std::time::Duration; + +use copper_cloud_core::error::ApiResult; +use reqwest::{Method, StatusCode}; +use serde::Serialize; +use serde_json::{json, Value}; +use uuid::Uuid; + +use crate::upstream::{Upstream, UpstreamError}; +use crate::{Active, CanvasLink}; + +/// Reconcile period. +pub const RECONCILE_EVERY: Duration = Duration::from_secs(600); +/// Idle outbox poll (work is also kicked right after it is enqueued). +pub const OUTBOX_POLL: Duration = Duration::from_secs(30); +/// Users per `users:batch` call (Canvas Cloud's limit). +pub const BATCH: usize = 1000; +/// Outbox items are dropped (with a warning) after this many failed attempts. +const MAX_ATTEMPTS: i32 = 50; +/// Page size when listing Canvas Cloud's app users. +const LIST_PAGE: i64 = 500; +/// Safety bound on pages listed per pass. +const MAX_PAGES: i64 = 1000; + +/// Outbox kinds. +pub mod kind { + pub const UPSERT_USER: &str = "upsert_user"; + pub const DELETE_USER: &str = "delete_user"; + pub const DISCONNECT_USER: &str = "disconnect_user"; +} + +/// A Copper account as mirrored to Canvas Cloud. +#[derive(Clone, Debug, sqlx::FromRow)] +pub struct CopperUser { + pub id: Uuid, + pub email: String, + pub display_name: Option, + pub disabled: bool, +} + +impl CopperUser { + fn digest(&self) -> u64 { + let mut h = std::collections::hash_map::DefaultHasher::new(); + self.email.hash(&mut h); + self.display_name.hash(&mut h); + self.disabled.hash(&mut h); + h.finish() + } + + fn upsert_body(&self) -> Value { + json!({ + "email": self.email, + "display_name": self.display_name, + "disabled": self.disabled, + "id": self.id, + }) + } + + fn batch_item(&self) -> Value { + json!({ + "subject": self.id.to_string(), + "email": self.email, + "display_name": self.display_name, + "disabled": self.disabled, + "id": self.id, + }) + } +} + +/// Copper stores a missing display name as `''`; Canvas Cloud takes `null`. +const USER_SQL: &str = + "SELECT id, email, NULLIF(display_name, '') AS display_name, disabled FROM users"; + +async fn copper_user(db: &sqlx::PgPool, id: Uuid) -> ApiResult> { + Ok( + sqlx::query_as::<_, CopperUser>(&format!("{USER_SQL} WHERE id = $1")) + .bind(id) + .fetch_optional(db) + .await?, + ) +} + +async fn copper_users(db: &sqlx::PgPool) -> ApiResult> { + Ok( + sqlx::query_as::<_, CopperUser>(&format!("{USER_SQL} ORDER BY created_at, id")) + .fetch_all(db) + .await?, + ) +} + +fn subject_path(id: Uuid) -> String { + format!("/v1/apps/self/users/{id}") +} + +/// Mark `users` as pushed (reconcile bookkeeping). +fn remember(link: &CanvasLink, users: &[&CopperUser]) { + let mut synced = link.synced(); + for u in users { + synced.insert(u.id, u.digest()); + } +} + +/// `PUT /v1/apps/self/users/{id}` for one Copper user (JIT, outbox). A user that no longer +/// exists in Copper is deleted from Canvas Cloud instead. +pub async fn provision_user( + link: &CanvasLink, + active: &Active, + user_id: Uuid, +) -> anyhow::Result<()> { + let Some(user) = copper_user(&link.state().db, user_id).await? else { + delete_remote_user(&active.upstream, user_id).await?; + return Ok(()); + }; + let resp: Value = Upstream::json( + active + .upstream + .request(Method::PUT, &subject_path(user_id)) + .json(&user.upsert_body()), + ) + .await?; + check_same_id(&resp, user_id); + remember(link, &[&user]); + tracing::info!(%user_id, "provisioned user on Canvas Cloud"); + Ok(()) +} + +fn check_same_id(item: &Value, expected: Uuid) -> bool { + let got = item + .get("user_id") + .and_then(Value::as_str) + .and_then(|s| s.parse::().ok()); + match got { + Some(id) if id != expected => { + tracing::warn!( + copper_user_id = %expected, + canvas_user_id = %id, + "Canvas Cloud linked this Copper user to an existing account with a different id; chat author ids will not match" + ); + false + } + _ => true, + } +} + +async fn delete_remote_user(up: &Upstream, user_id: Uuid) -> Result<(), UpstreamError> { + match Upstream::json::(up.request(Method::DELETE, &subject_path(user_id))).await { + Ok(_) => Ok(()), + Err(e) if e.status() == Some(StatusCode::NOT_FOUND) => Ok(()), + Err(e) => Err(e), + } +} + +async fn disconnect_remote_user(up: &Upstream, user_id: Uuid) -> Result<(), UpstreamError> { + let path = format!("{}/disconnect", subject_path(user_id)); + match Upstream::json::(up.request(Method::POST, &path)).await { + Ok(_) => Ok(()), + Err(e) if e.status() == Some(StatusCode::NOT_FOUND) => Ok(()), + Err(e) => Err(e), + } +} + +/// Outcome of a bulk sync. +#[derive(Debug, Default, Serialize)] +pub struct SyncReport { + pub total: usize, + pub provisioned: usize, + /// `(user id, error)` for users Canvas Cloud refused. + pub failed: Vec<(Uuid, String)>, + /// Users Canvas Cloud linked to an existing account with a different id. + pub id_mismatch: Vec, +} + +impl SyncReport { + pub fn ok(&self) -> bool { + self.failed.is_empty() && self.id_mismatch.is_empty() + } +} + +/// Push `users` with `POST /v1/apps/self/users:batch`, [`BATCH`] at a time. +async fn push_batch( + link: &CanvasLink, + active: &Active, + users: &[CopperUser], + report: &mut SyncReport, +) -> Result<(), UpstreamError> { + for chunk in users.chunks(BATCH) { + let body = json!({ "users": chunk.iter().map(CopperUser::batch_item).collect::>() }); + let resp: Value = Upstream::json( + active + .upstream + .request(Method::POST, "/v1/apps/self/users:batch") + .timeout(crate::upstream::LONG_REQUEST_TIMEOUT) + .json(&body), + ) + .await?; + let items = resp + .get("results") + .or_else(|| resp.get("users")) + .and_then(Value::as_array); + let mut ok: Vec<&CopperUser> = Vec::with_capacity(chunk.len()); + match items { + Some(items) => { + let by_subject: HashMap = items + .iter() + .filter_map(|i| Some((i.get("subject")?.as_str()?.to_owned(), i))) + .collect(); + for (n, user) in chunk.iter().enumerate() { + let item = by_subject + .get(&user.id.to_string()) + .copied() + .or_else(|| items.get(n)); + let err = item.and_then(|i| i.get("error")).filter(|e| !e.is_null()); + if let Some(err) = err { + let msg = item + .and_then(|i| i.get("message")) + .and_then(Value::as_str) + .map_or_else( + || err.as_str().unwrap_or("error").to_owned(), + |m| format!("{}: {m}", err.as_str().unwrap_or("error")), + ); + report.failed.push((user.id, msg)); + } else { + if item.is_some_and(|i| !check_same_id(i, user.id)) { + report.id_mismatch.push(user.id); + } + ok.push(user); + } + } + } + None => ok.extend(chunk.iter()), + } + report.provisioned += ok.len(); + remember(link, &ok); + } + Ok(()) +} + +/// Provision every Copper user (link time, `copper-cloud canvas sync-users`). +pub async fn sync_all(link: &CanvasLink, active: &Active) -> anyhow::Result { + let users = copper_users(&link.state().db).await?; + let mut report = SyncReport { + total: users.len(), + ..SyncReport::default() + }; + push_batch(link, active, &users, &mut report).await?; + for (id, err) in &report.failed { + tracing::warn!(user_id = %id, error = %err, "Canvas Cloud refused a user"); + } + Ok(report) +} + +/// List Canvas Cloud's users of this app: `subject → canvas_count`. +pub async fn remote_users( + up: &Upstream, + timeout: Duration, +) -> Result, UpstreamError> { + let mut out = HashMap::new(); + for page in 0..MAX_PAGES { + let path = format!( + "/v1/apps/self/users?limit={LIST_PAGE}&offset={}", + page * LIST_PAGE + ); + let resp: Value = Upstream::json(up.request(Method::GET, &path).timeout(timeout)).await?; + let items = resp + .get("users") + .or_else(|| resp.get("items")) + .and_then(Value::as_array) + .cloned() + .unwrap_or_default(); + let n = items.len(); + for item in items { + if let Some(subject) = item.get("subject").and_then(Value::as_str) { + let count = item + .get("canvas_count") + .and_then(Value::as_i64) + .unwrap_or(0); + out.insert(subject.to_owned(), count); + } + } + let total = resp.get("total").and_then(Value::as_i64); + let fetched = i64::try_from(out.len()).unwrap_or(i64::MAX); + if n < usize::try_from(LIST_PAGE).unwrap_or(usize::MAX) + || total.is_some_and(|t| fetched >= t) + { + break; + } + } + Ok(out) +} + +/// One reconcile pass. Returns `(pushed, disconnected, deleted)`. +pub async fn reconcile_once( + link: &CanvasLink, + active: &Active, +) -> anyhow::Result<(usize, usize, usize)> { + let users = copper_users(&link.state().db).await?; + let (changed, newly_disabled): (Vec, Vec) = { + let synced = link.synced(); + let mut changed = Vec::new(); + let mut disabled = Vec::new(); + for u in &users { + if synced.get(&u.id) != Some(&u.digest()) { + if u.disabled { + disabled.push(u.id); + } + changed.push(u.clone()); + } + } + (changed, disabled) + }; + let mut report = SyncReport::default(); + if !changed.is_empty() { + push_batch(link, active, &changed, &mut report).await?; + } + let mut disconnected = 0; + for id in newly_disabled { + if disconnect_remote_user(&active.upstream, id).await.is_ok() { + disconnected += 1; + } + } + // App users whose Copper account is gone (e.g. `copper-cloud admin delete-user`). + let ours: HashSet = users.iter().map(|u| u.id.to_string()).collect(); + let remote = remote_users(&active.upstream, crate::upstream::REQUEST_TIMEOUT).await?; + let mut deleted = 0; + for subject in remote.keys().filter(|s| !ours.contains(*s)) { + let Ok(id) = subject.parse::() else { + continue; + }; + match delete_remote_user(&active.upstream, id).await { + Ok(()) => { + deleted += 1; + link.synced().remove(&id); + } + Err(e) => { + tracing::warn!(user_id = %id, error = %e, "removing a deleted user from Canvas Cloud failed"); + } + } + } + Ok((report.provisioned, disconnected, deleted)) +} + +/// Background reconcile: right away, then every [`RECONCILE_EVERY`] and on link changes. +pub async fn reconcile_loop(link: CanvasLink) { + let mut generation = link.subscribe(); + let mut shutdown = copper_cloud_core::shutdown::subscribe(); + loop { + if copper_cloud_core::shutdown::is_shutting_down() { + break; + } + if let Ok(Some(active)) = link.enabled().await { + match reconcile_once(&link, &active).await { + Ok((0, 0, 0)) => {} + Ok((pushed, disconnected, deleted)) => { + tracing::info!(pushed, disconnected, deleted, "canvas users reconciled"); + } + Err(err) => { + tracing::warn!(error = format!("{err:#}"), "canvas user reconcile failed"); + } + } + } + tokio::select! { + () = tokio::time::sleep(RECONCILE_EVERY) => {} + r = generation.changed() => { if r.is_err() { break; } } + _ = shutdown.changed() => {} + } + } +} + +// --------------------------------------------------------------------------------------------- +// Outbox + +/// Queue `kind` for `subject` (in the caller's transaction) — only while a link exists. +async fn enqueue( + tx: &mut sqlx::PgConnection, + kind: &str, + subject: Uuid, + payload: Option, +) -> ApiResult { + let n = sqlx::query( + "INSERT INTO canvas_link_outbox (kind, subject, payload) + SELECT $1, $2, $3 WHERE EXISTS (SELECT 1 FROM canvas_link WHERE id = 1)", + ) + .bind(kind) + .bind(subject) + .bind(payload) + .execute(&mut *tx) + .await? + .rows_affected(); + Ok(n > 0) +} + +/// A user's email / name / disabled flag changed (admin API): push it, and disconnect their +/// canvas sockets when `disabled`. Call inside the change's transaction, then +/// [`CanvasLink::kick_outbox`] after commit. +pub async fn enqueue_user_changed( + tx: &mut sqlx::PgConnection, + user_id: Uuid, + disabled: bool, +) -> ApiResult<()> { + enqueue(tx, kind::UPSERT_USER, user_id, None).await?; + if disabled { + enqueue(tx, kind::DISCONNECT_USER, user_id, None).await?; + } + Ok(()) +} + +/// A user was deleted (admin API): delete them on Canvas Cloud (their canvases go with them +/// there). Call inside the delete's transaction, then [`CanvasLink::kick_outbox`]. +pub async fn enqueue_user_deleted(tx: &mut sqlx::PgConnection, user_id: Uuid) -> ApiResult<()> { + enqueue(tx, kind::DELETE_USER, user_id, None).await?; + Ok(()) +} + +#[derive(sqlx::FromRow)] +struct OutboxRow { + id: i64, + kind: String, + subject: Uuid, + attempts: i32, +} + +/// Run every due outbox item once. Returns `(done, failed)`. +pub async fn process_outbox(link: &CanvasLink, active: &Active) -> ApiResult<(usize, usize)> { + let db = &link.state().db; + let rows: Vec = sqlx::query_as( + "SELECT id, kind, subject, attempts FROM canvas_link_outbox + WHERE next_attempt_at <= now() ORDER BY id LIMIT 200", + ) + .fetch_all(db) + .await?; + let (mut done, mut failed) = (0, 0); + for row in rows { + let result: anyhow::Result<()> = match row.kind.as_str() { + kind::UPSERT_USER => provision_user(link, active, row.subject).await, + kind::DELETE_USER => { + let r = delete_remote_user(&active.upstream, row.subject).await; + link.synced().remove(&row.subject); + r.map_err(Into::into) + } + kind::DISCONNECT_USER => disconnect_remote_user(&active.upstream, row.subject) + .await + .map_err(Into::into), + other => Err(anyhow::anyhow!("unknown outbox kind {other}")), + }; + match result { + Ok(()) => { + done += 1; + sqlx::query("DELETE FROM canvas_link_outbox WHERE id = $1") + .bind(row.id) + .execute(db) + .await?; + } + Err(err) if row.attempts + 1 >= MAX_ATTEMPTS => { + failed += 1; + tracing::warn!(kind = %row.kind, user_id = %row.subject, error = format!("{err:#}"), "giving up on canvas link outbox item"); + sqlx::query("DELETE FROM canvas_link_outbox WHERE id = $1") + .bind(row.id) + .execute(db) + .await?; + } + Err(err) => { + failed += 1; + sqlx::query( + "UPDATE canvas_link_outbox + SET attempts = attempts + 1, last_error = $2, + next_attempt_at = now() + make_interval(secs => least(30 * power(2, least(attempts, 7)), 3600)) + WHERE id = $1", + ) + .bind(row.id) + .bind(format!("{err:#}")) + .execute(db) + .await?; + } + } + } + Ok((done, failed)) +} + +/// Background outbox worker: kicked after enqueueing, else every [`OUTBOX_POLL`]. +pub async fn outbox_loop(link: CanvasLink) { + loop { + if copper_cloud_core::shutdown::is_shutting_down() { + break; + } + if let Ok(Some(active)) = link.enabled().await { + match process_outbox(&link, &active).await { + Ok((0, 0)) => {} + Ok((done, failed)) => tracing::info!(done, failed, "canvas link outbox processed"), + Err(err) => tracing::warn!(error = %err, "canvas link outbox failed"), + } + } + link.outbox_wait(OUTBOX_POLL).await; + } +} diff --git a/crates/canvas-link/src/proxy.rs b/crates/canvas-link/src/proxy.rs new file mode 100644 index 0000000..aa8423d --- /dev/null +++ b/crates/canvas-link/src/proxy.rs @@ -0,0 +1,325 @@ +//! The canvas API reverse proxy (CONTRACT §7.2). +//! +//! * Caller: a signed-in Copper user ([`AuthUser`]); the instance gate already ran. +//! * Forwarded to `{canvas cloud}/v1{same path}{same query minus token=}` with the method, the +//! body (64 KiB, 4 MiB for `/ops`), `Content-Type`, `Accept` and `If-*`. Nothing else of the +//! client's headers is forwarded (no `Authorization`, `Cookie`, `X-Copper-Instance`, +//! hop-by-hop); added: `Authorization: Bearer `, `X-Canvas-User: `, +//! `X-Forwarded-For`. +//! * Response: status, body (streamed), `Content-Type`, `Cache-Control` (+ `ETag`, +//! `Last-Modified`, `Location`, `Retry-After`) verbatim — except **401**, which Copper +//! treats as "signed out": `401 unknown_user` provisions the user and retries once; any +//! other 401 (or a second `unknown_user`) becomes `502 canvas_upstream`. +//! * Canvas Cloud unreachable / timed out → `502 canvas_upstream`; unlinked → `503 +//! canvas_unavailable`. + +use std::net::IpAddr; + +use axum::body::{Body, Bytes}; +use axum::extract::ws::WebSocketUpgrade; +use axum::extract::{FromRequestParts as _, OriginalUri, Request, State}; +use axum::http::{header, HeaderMap, HeaderName, Method, StatusCode, Uri}; +use axum::response::{IntoResponse, Response}; +use axum::Extension; +use copper_cloud_core::auth::AuthUser; +use copper_cloud_core::error::ApiError; +use copper_cloud_core::extract::read_body; +use copper_cloud_core::state::SharedState; +use uuid::Uuid; + +use crate::upstream::{ + error_fields, UpstreamError, CANVAS_USER_HEADER, LONG_REQUEST_TIMEOUT, REQUEST_TIMEOUT, +}; +use crate::{upstream_failed, Active, CanvasLink, MAX_BODY_BYTES, MAX_OPS_BODY_BYTES}; + +/// Request headers forwarded to Canvas Cloud. +const FORWARD_REQUEST: &[HeaderName] = &[ + header::CONTENT_TYPE, + header::ACCEPT, + header::IF_MATCH, + header::IF_NONE_MATCH, + header::IF_MODIFIED_SINCE, + header::IF_UNMODIFIED_SINCE, + header::IF_RANGE, +]; + +/// Response headers passed back to Copper. +const FORWARD_RESPONSE: &[HeaderName] = &[ + header::CONTENT_TYPE, + header::CACHE_CONTROL, + header::ETAG, + header::LAST_MODIFIED, + header::LOCATION, + header::RETRY_AFTER, + header::CONTENT_DISPOSITION, +]; + +/// Largest 401 body we read to look for `unknown_user`. +const MAX_ERROR_BODY: usize = 16 * 1024; + +pub(crate) async fn proxy( + State(state): State, + Extension(link): Extension, + req: Request, +) -> Response { + match handle(&state, &link, req).await { + Ok(resp) | Err(resp) => resp, + } +} + +/// What the client asked for, ready to forward. +pub(crate) struct Forward { + pub user_id: Uuid, + pub client_ip: Option, + /// `/v1/...?query` (token removed). + pub path_and_query: String, +} + +// `Err` is the finished error response (same type as `Ok`). +#[allow(clippy::result_large_err)] +async fn handle( + state: &SharedState, + link: &CanvasLink, + req: Request, +) -> Result { + let client_ip = copper_cloud_core::observe::client_ip(&req, state.cfg.trust_proxy); + let (mut parts, body) = req.into_parts(); + let user = AuthUser::from_request_parts(&mut parts, state) + .await + .map_err(IntoResponse::into_response)?; + let Some(active) = link.enabled().await.map_err(IntoResponse::into_response)? else { + return Err(crate::unavailable()); + }; + let uri = parts + .extensions + .get::() + .map_or_else(|| parts.uri.clone(), |o| o.0.clone()); + let path = v1_path(&uri); + let fwd = Forward { + user_id: user.user_id, + client_ip, + path_and_query: match forwarded_query(uri.query()) { + Some(q) => format!("{path}?{q}"), + None => path.clone(), + }, + }; + + if is_ws_path(&path) { + if let Ok(ws) = WebSocketUpgrade::from_request_parts(&mut parts, state).await { + return Ok(crate::ws::proxy(link, &active, &fwd, ws).await); + } + } + + let ops = path.ends_with("/ops"); + let limit = if ops { + MAX_OPS_BODY_BYTES + } else { + MAX_BODY_BYTES + }; + check_declared_length(&parts.headers, limit).map_err(IntoResponse::into_response)?; + let bytes = read_body(body, limit) + .await + .map_err(IntoResponse::into_response)?; + let timeout = if ops { + LONG_REQUEST_TIMEOUT + } else { + REQUEST_TIMEOUT + }; + + let mut provisioned = false; + loop { + let resp = send( + &active, + &parts.method, + &parts.headers, + &fwd, + &bytes, + timeout, + ) + .await + .map_err(|e| { + tracing::warn!(error = %e, "canvas upstream request failed"); + upstream_failed(e.to_string()) + })?; + if resp.status() != StatusCode::UNAUTHORIZED { + return Ok(pass_through(resp)); + } + let body = limited_bytes(resp).await; + let (code, message) = error_fields(&body); + if code.as_deref() == Some("unknown_user") && !provisioned { + provisioned = true; + jit_provision(link, &active, fwd.user_id).await?; + continue; + } + return Err(rejected(code.as_deref(), message.as_deref(), fwd.user_id)); + } +} + +/// Provision `user_id` after `unknown_user`; failure → `502`. +#[allow(clippy::result_large_err)] +pub(crate) async fn jit_provision( + link: &CanvasLink, + active: &Active, + user_id: Uuid, +) -> Result<(), Response> { + crate::provision::provision_user(link, active, user_id) + .await + .map_err(|e| { + tracing::warn!(%user_id, error = %e, "just-in-time canvas user provisioning failed"); + upstream_failed(format!( + "could not provision this user on Canvas Cloud ({e})" + )) + }) +} + +/// Map an upstream 401 (never forwarded: Copper would sign the user out) to `502`. +pub(crate) fn rejected(code: Option<&str>, message: Option<&str>, user_id: Uuid) -> Response { + let detail = match (code, message) { + (Some(c), Some(m)) if !m.is_empty() => format!("{c}: {m}"), + (Some(c), _) => c.to_owned(), + (None, Some(m)) => m.to_owned(), + (None, None) => "401".to_owned(), + }; + tracing::error!( + %user_id, + upstream_error = %detail, + "Canvas Cloud answered 401 to the app; check the canvas link's app key" + ); + upstream_failed(format!( + "Canvas Cloud rejected this Copper Cloud's app key/user ({detail})" + )) +} + +async fn send( + active: &Active, + method: &Method, + headers: &HeaderMap, + fwd: &Forward, + body: &Bytes, + timeout: std::time::Duration, +) -> Result { + let up = &active.upstream; + let mut rb = up + .client() + .request(method.clone(), up.url(&fwd.path_and_query)) + .timeout(timeout) + .header(header::AUTHORIZATION, up.auth_header()) + .header(CANVAS_USER_HEADER, fwd.user_id.to_string()); + if let Some(ip) = fwd.client_ip { + rb = rb.header("x-forwarded-for", ip.to_string()); + } + for name in FORWARD_REQUEST { + for value in headers.get_all(name) { + rb = rb.header(name, value); + } + } + if !(body.is_empty() && matches!(*method, Method::GET | Method::HEAD)) { + rb = rb.body(body.clone()); + } + rb.send().await.map_err(|e| UpstreamError::from_reqwest(&e)) +} + +fn pass_through(resp: reqwest::Response) -> Response { + let status = resp.status(); + let mut out = Response::builder().status(status); + if let Some(h) = out.headers_mut() { + for name in FORWARD_RESPONSE { + for value in resp.headers().get_all(name) { + h.append(name, value.clone()); + } + } + } + let body = Body::from_stream(resp.bytes_stream()); + out.body(body) + .unwrap_or_else(|_| StatusCode::BAD_GATEWAY.into_response()) +} + +async fn limited_bytes(mut resp: reqwest::Response) -> Vec { + let mut out = Vec::new(); + while let Ok(Some(chunk)) = resp.chunk().await { + if out.len() + chunk.len() > MAX_ERROR_BODY { + break; + } + out.extend_from_slice(&chunk); + } + out +} + +fn check_declared_length(headers: &HeaderMap, limit: usize) -> Result<(), ApiError> { + let declared = headers + .get(header::CONTENT_LENGTH) + .and_then(|v| v.to_str().ok()) + .and_then(|v| v.parse::().ok()); + match declared { + Some(n) if n > limit as u64 => Err(ApiError::PayloadTooLarge), + _ => Ok(()), + } +} + +/// The request path as Canvas Cloud sees it: `/v1/...` (the router is nested under `/v1`). +fn v1_path(uri: &Uri) -> String { + let path = uri.path(); + if path.starts_with("/v1/") { + path.to_owned() + } else { + format!("/v1{path}") + } +} + +/// `/v1/canvases/{id}/ws`. +pub(crate) fn is_ws_path(path: &str) -> bool { + path.strip_prefix("/v1/canvases/") + .and_then(|rest| rest.strip_suffix("/ws")) + .is_some_and(|id| !id.is_empty() && !id.contains('/')) +} + +/// The query string minus every `token=` parameter (Copper's session token for WebSocket +/// tooling); `None` when nothing is left. +pub(crate) fn forwarded_query(query: Option<&str>) -> Option { + let kept: Vec<&str> = query? + .split('&') + .filter(|kv| !kv.is_empty() && *kv != "token" && !kv.starts_with("token=")) + .collect(); + (!kept.is_empty()).then(|| kept.join("&")) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn query_drops_token() { + assert_eq!(forwarded_query(None), None); + assert_eq!(forwarded_query(Some("")), None); + assert_eq!(forwarded_query(Some("token=abc")), None); + assert_eq!( + forwarded_query(Some("full=true&token=abc&ids=a,b")).as_deref(), + Some("full=true&ids=a,b") + ); + assert_eq!( + forwarded_query(Some("tokens=1&x=%20")).as_deref(), + Some("tokens=1&x=%20") + ); + assert_eq!(forwarded_query(Some("token&q=1")).as_deref(), Some("q=1")); + } + + #[test] + fn ws_paths() { + assert!(is_ws_path("/v1/canvases/personal/ws")); + assert!(is_ws_path( + "/v1/canvases/0190f0f0-0000-7000-8000-000000000000/ws" + )); + assert!(!is_ws_path("/v1/canvases/ws")); + assert!(!is_ws_path("/v1/canvases//ws")); + assert!(!is_ws_path("/v1/canvases/a/b/ws")); + assert!(!is_ws_path("/v1/canvases/a/state")); + } + + #[test] + fn v1_prefix() { + let u: Uri = "/v1/canvases?x=1".parse().unwrap(); + assert_eq!(v1_path(&u), "/v1/canvases"); + let u: Uri = "/canvases/a".parse().unwrap(); + assert_eq!(v1_path(&u), "/v1/canvases/a"); + } +} diff --git a/crates/canvas-link/src/settings.rs b/crates/canvas-link/src/settings.rs new file mode 100644 index 0000000..a68fa61 --- /dev/null +++ b/crates/canvas-link/src/settings.rs @@ -0,0 +1,351 @@ +//! The stored link (`canvas_link`, singleton row): Canvas Cloud base URL, the app key sealed +//! under a per-row data key wrapped by the KEK (AAD [`AAD_APP_KEY`]), the optional TLS leaf +//! pin, and bookkeeping. The key never reaches logs, the admin API or the CLI output (only +//! its last four characters). + +use std::net::IpAddr; + +use copper_cloud_core::crypto::Crypto; +use copper_cloud_core::error::{ApiError, ApiResult}; +use time::OffsetDateTime; +use url::Url; +use zeroize::Zeroizing; + +/// AAD binding the sealed app key to its column. +pub const AAD_APP_KEY: &[u8] = b"canvas_link:app_key"; +const MAX_KEY_LEN: usize = 4096; +const MAX_URL_LEN: usize = 2048; + +/// The decrypted link settings. +#[derive(Clone)] +pub struct LinkSettings { + /// Canvas Cloud base URL (scheme + authority + optional path prefix, no trailing `/`). + pub url: Url, + pub app_key: Zeroizing, + /// Lower-case hex SHA-256 of the pinned leaf certificate (self-signed Canvas Cloud). + pub tls_fingerprint: Option, + pub enabled: bool, + pub linked_at: OffsetDateTime, + pub updated_at: OffsetDateTime, + pub updated_by: Option, +} + +impl std::fmt::Debug for LinkSettings { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_struct("LinkSettings") + .field("url", &self.url.as_str()) + .field("app_key", &"") + .field("tls_fingerprint", &self.tls_fingerprint) + .field("enabled", &self.enabled) + .field("linked_at", &self.linked_at) + .field("updated_at", &self.updated_at) + .field("updated_by", &self.updated_by) + .finish() + } +} + +impl LinkSettings { + /// The base URL as text without a trailing `/` (what the admin typed, normalised). + pub fn base(&self) -> String { + base_string(&self.url) + } + + /// Whether `other` points at the same server with the same credentials (the HTTP client + /// and the event bridge can be kept). + pub fn same_target(&self, other: &Self) -> bool { + self.url == other.url + && *self.app_key == *other.app_key + && self.tls_fingerprint == other.tls_fingerprint + } +} + +/// `url` without a trailing `/`. +pub fn base_string(url: &Url) -> String { + url.as_str().trim_end_matches('/').to_owned() +} + +/// Validated input for a new / replaced link. +pub struct NewLink { + pub url: Url, + pub app_key: Zeroizing, + pub tls_fingerprint: Option, +} + +impl NewLink { + /// Validate raw admin/CLI input (see [`parse_base_url`], [`clean_app_key`], + /// [`clean_fingerprint`]). + pub fn parse(url: &str, app_key: &str, tls_fingerprint: Option<&str>) -> ApiResult { + Ok(Self { + url: parse_base_url(url)?, + app_key: clean_app_key(app_key)?, + tls_fingerprint: clean_fingerprint(tls_fingerprint)?, + }) + } +} + +/// Validate a Canvas Cloud base URL: `https://host[:port][/prefix]`, or `http://` for a +/// loopback host only (a Canvas Cloud on the same machine). No user info, query or fragment. +pub fn parse_base_url(raw: &str) -> ApiResult { + let raw = raw.trim(); + if raw.is_empty() || raw.len() > MAX_URL_LEN || raw.chars().any(char::is_whitespace) { + return Err(ApiError::bad_request( + "url must be the Canvas Cloud's https:// URL", + )); + } + let url = Url::parse(raw).map_err(|_| { + ApiError::bad_request( + "url must be the Canvas Cloud's https:// URL (e.g. https://canvas.example.com)", + ) + })?; + if !url.username().is_empty() || url.password().is_some() { + return Err(ApiError::bad_request("url must not contain credentials")); + } + if url.query().is_some() || url.fragment().is_some() { + return Err(ApiError::bad_request( + "url must not contain a query or fragment", + )); + } + let Some(host) = url.host() else { + return Err(ApiError::bad_request("url needs a host")); + }; + match url.scheme() { + "https" => {} + "http" if is_loopback(&host) => {} + "http" => return Err(ApiError::bad_request( + "http:// is only allowed for a Canvas Cloud on this machine (localhost); use https://", + )), + _ => { + return Err(ApiError::bad_request( + "url must start with https:// (or http:// for localhost)", + )) + } + } + let base = base_string(&url); + Url::parse(&base).map_err(|_| ApiError::bad_request("invalid url")) +} + +fn is_loopback(host: &url::Host<&str>) -> bool { + match host { + url::Host::Domain(d) => d.eq_ignore_ascii_case("localhost"), + url::Host::Ipv4(ip) => IpAddr::V4(*ip).is_loopback(), + url::Host::Ipv6(ip) => IpAddr::V6(*ip).is_loopback(), + } +} + +/// Validate an app key: trimmed, 8–4096 printable ASCII characters without spaces. +pub fn clean_app_key(raw: &str) -> ApiResult> { + let key = raw.trim(); + if key.len() < 8 || key.len() > MAX_KEY_LEN || !key.bytes().all(|b| b.is_ascii_graphic()) { + return Err(ApiError::bad_request( + "app_key must be the Canvas Cloud app key (cva_…), printable ASCII without spaces", + )); + } + Ok(Zeroizing::new(key.to_owned())) +} + +/// Normalise an optional SHA-256 leaf fingerprint (hex, any case, `:` separators allowed); +/// empty means none. +pub fn clean_fingerprint(raw: Option<&str>) -> ApiResult> { + match raw.map(str::trim).filter(|s| !s.is_empty()) { + None => Ok(None), + Some(fp) => copper_cloud_core::link::normalize_fingerprint(fp) + .map(Some) + .ok_or_else(|| { + ApiError::bad_request( + "tls_fingerprint must be the SHA-256 of the certificate: 64 hex characters (colons allowed)", + ) + }), + } +} + +/// Last four characters of a key (`""` when too short to reveal any). +pub fn last4(key: &str) -> String { + copper_cloud_core::intelligence::last4(key) +} + +type Row = ( + String, + Vec, + Vec, + Option, + bool, + OffsetDateTime, + OffsetDateTime, + Option, +); + +const SELECT_SQL: &str = "SELECT url, app_key_sealed, data_key_wrapped, tls_fingerprint, enabled, + linked_at, updated_at, updated_by +FROM canvas_link WHERE id = 1"; + +fn decode(crypto: &Crypto, row: Row) -> ApiResult { + let (url, sealed, wrapped, tls_fingerprint, enabled, linked_at, updated_at, updated_by) = row; + let data_key = Zeroizing::new(crypto.unwrap_key(&wrapped)?); + let plain = Zeroizing::new(Crypto::open(&data_key, AAD_APP_KEY, &sealed)?); + let app_key = Zeroizing::new( + String::from_utf8(plain.to_vec()) + .map_err(|_| ApiError::internal("stored canvas link key is not UTF-8"))?, + ); + let url = Url::parse(&url).map_err(|e| ApiError::internal(format!("stored url: {e}")))?; + Ok(LinkSettings { + url, + app_key, + tls_fingerprint, + enabled, + linked_at, + updated_at, + updated_by, + }) +} + +/// Load and decrypt the link (`None` when no Canvas Cloud is linked). +pub async fn load(db: &sqlx::PgPool, crypto: &Crypto) -> ApiResult> { + let row: Option = sqlx::query_as(SELECT_SQL).fetch_optional(db).await?; + row.map(|r| decode(crypto, r)).transpose() +} + +/// Whether a link row exists (cheap; no decryption). +pub async fn exists<'e, E: sqlx::PgExecutor<'e>>(db: E) -> ApiResult { + Ok( + sqlx::query_scalar::<_, i32>("SELECT 1 FROM canvas_link WHERE id = 1") + .fetch_optional(db) + .await? + .is_some(), + ) +} + +/// Store (insert or replace) the link, enabled. A replaced link keeps its `linked_at` when the +/// URL is unchanged (key rotation); a new URL counts as a new link. +pub async fn save( + db: &sqlx::PgPool, + crypto: &Crypto, + link: &NewLink, + updated_by: &str, +) -> ApiResult { + let data_key = Zeroizing::new(Crypto::new_key()); + let wrapped = crypto.wrap_key(&data_key); + let sealed = Crypto::seal(&data_key, AAD_APP_KEY, link.app_key.as_bytes()); + let base = base_string(&link.url); + sqlx::query( + "INSERT INTO canvas_link (id, url, app_key_sealed, data_key_wrapped, tls_fingerprint, + enabled, linked_at, updated_at, updated_by) + VALUES (1, $1, $2, $3, $4, true, now(), now(), $5) + ON CONFLICT (id) DO UPDATE SET + url = EXCLUDED.url, + app_key_sealed = EXCLUDED.app_key_sealed, + data_key_wrapped = EXCLUDED.data_key_wrapped, + tls_fingerprint = EXCLUDED.tls_fingerprint, + enabled = true, + linked_at = CASE WHEN canvas_link.url = EXCLUDED.url THEN canvas_link.linked_at + ELSE now() END, + updated_at = now(), + updated_by = EXCLUDED.updated_by", + ) + .bind(&base) + .bind(&sealed) + .bind(&wrapped) + .bind(&link.tls_fingerprint) + .bind(updated_by) + .execute(db) + .await?; + load(db, crypto) + .await? + .ok_or_else(|| ApiError::internal("canvas link vanished after save")) +} + +/// Remove the link. Returns the URL that was linked, if any. Queued outbox work goes too +/// (it targeted that server). +pub async fn remove(db: &sqlx::PgPool) -> ApiResult> { + let mut tx = db.begin().await?; + let url: Option = + sqlx::query_scalar("DELETE FROM canvas_link WHERE id = 1 RETURNING url") + .fetch_optional(&mut *tx) + .await?; + if url.is_some() { + sqlx::query("DELETE FROM canvas_link_outbox") + .execute(&mut *tx) + .await?; + } + tx.commit().await?; + Ok(url) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn urls() { + for ok in [ + "https://canvas.example.com", + "https://canvas.example.com/", + "https://canvas.example.com:8443/prefix/", + "http://127.0.0.1:8480", + "http://localhost:8480", + "http://[::1]:8480", + " https://c.example ", + ] { + let u = parse_base_url(ok).unwrap_or_else(|e| panic!("{ok}: {e}")); + assert!(!u.as_str().ends_with("//"), "{u}"); + } + assert_eq!( + base_string(&parse_base_url("https://canvas.example.com/").unwrap()), + "https://canvas.example.com" + ); + assert_eq!( + base_string(&parse_base_url("https://c.example:8443/p/").unwrap()), + "https://c.example:8443/p" + ); + for bad in [ + "", + "canvas.example.com", + "ftp://canvas.example.com", + "http://canvas.example.com", + "http://10.0.0.5:8480", + "https://user:pw@canvas.example.com", + "https://canvas.example.com/?x=1", + "https://canvas.example.com/#f", + "https://can vas.example.com", + ] { + assert!(parse_base_url(bad).is_err(), "{bad}"); + } + } + + #[test] + fn keys_and_fingerprints() { + assert_eq!( + clean_app_key(" cva_abcdefgh ").unwrap().as_str(), + "cva_abcdefgh" + ); + assert!(clean_app_key("short").is_err()); + assert!(clean_app_key("has space inside").is_err()); + let hex = "AB".repeat(32); + assert_eq!( + clean_fingerprint(Some(&hex)).unwrap().unwrap(), + "ab".repeat(32) + ); + let colons = vec!["Ab"; 32].join(":"); + assert_eq!( + clean_fingerprint(Some(&colons)).unwrap().unwrap(), + "ab".repeat(32) + ); + assert_eq!(clean_fingerprint(Some(" ")).unwrap(), None); + assert_eq!(clean_fingerprint(None).unwrap(), None); + assert!(clean_fingerprint(Some("abcd")).is_err()); + assert!(clean_fingerprint(Some(&"zz".repeat(32))).is_err()); + } + + #[test] + fn debug_redacts_key() { + let s = LinkSettings { + url: Url::parse("https://c.example").unwrap(), + app_key: Zeroizing::new("cva_supersecretkey".into()), + tls_fingerprint: None, + enabled: true, + linked_at: OffsetDateTime::UNIX_EPOCH, + updated_at: OffsetDateTime::UNIX_EPOCH, + updated_by: None, + }; + assert!(!format!("{s:?}").contains("supersecret")); + } +} diff --git a/crates/canvas-link/src/tls.rs b/crates/canvas-link/src/tls.rs new file mode 100644 index 0000000..1e5fa97 --- /dev/null +++ b/crates/canvas-link/src/tls.rs @@ -0,0 +1,94 @@ +//! TLS to the linked Canvas Cloud (rustls, `ring`): the web PKI (webpki roots) by default, or — +//! when a fingerprint is pinned — exactly one leaf certificate whose SHA-256 matches (a +//! self-signed Canvas Cloud), the same idea as Copper pinning copper-cloud. + +use std::sync::Arc; + +use copper_cloud_core::ids; +use rustls::client::danger::{HandshakeSignatureValid, ServerCertVerified, ServerCertVerifier}; +use rustls::crypto::CryptoProvider; +use rustls::pki_types::{CertificateDer, ServerName, UnixTime}; +use rustls::{DigitallySignedStruct, SignatureScheme}; + +/// A rustls client config for talking to Canvas Cloud over HTTP/1.1 (REST, SSE and the +/// WebSocket upgrade all use HTTP/1.1). +pub fn client_config(pin: Option<&str>) -> anyhow::Result { + let provider = copper_cloud_core::tls::provider(); + let builder = rustls::ClientConfig::builder_with_provider(provider.clone()) + .with_safe_default_protocol_versions()?; + let mut config = if let Some(fp) = pin { + builder + .dangerous() + .with_custom_certificate_verifier(Arc::new(PinnedVerifier { + expected: fp.to_ascii_lowercase(), + provider, + })) + .with_no_client_auth() + } else { + let roots: rustls::RootCertStore = webpki_roots::TLS_SERVER_ROOTS.iter().cloned().collect(); + builder.with_root_certificates(roots).with_no_client_auth() + }; + config.alpn_protocols = vec![b"http/1.1".to_vec()]; + Ok(config) +} + +/// Accepts exactly one leaf certificate (by SHA-256), whatever its names or issuer; the +/// handshake signature is still verified against that certificate's key. +#[derive(Debug)] +struct PinnedVerifier { + expected: String, + provider: Arc, +} + +impl ServerCertVerifier for PinnedVerifier { + fn verify_server_cert( + &self, + end_entity: &CertificateDer<'_>, + _intermediates: &[CertificateDer<'_>], + _server_name: &ServerName<'_>, + _ocsp_response: &[u8], + _now: UnixTime, + ) -> Result { + if ids::sha256_hex(end_entity) == self.expected { + Ok(ServerCertVerified::assertion()) + } else { + Err(rustls::Error::General( + "Canvas Cloud certificate does not match the pinned fingerprint".into(), + )) + } + } + + fn verify_tls12_signature( + &self, + message: &[u8], + cert: &CertificateDer<'_>, + dss: &DigitallySignedStruct, + ) -> Result { + rustls::crypto::verify_tls12_signature( + message, + cert, + dss, + &self.provider.signature_verification_algorithms, + ) + } + + fn verify_tls13_signature( + &self, + message: &[u8], + cert: &CertificateDer<'_>, + dss: &DigitallySignedStruct, + ) -> Result { + rustls::crypto::verify_tls13_signature( + message, + cert, + dss, + &self.provider.signature_verification_algorithms, + ) + } + + fn supported_verify_schemes(&self) -> Vec { + self.provider + .signature_verification_algorithms + .supported_schemes() + } +} diff --git a/crates/canvas-link/src/upstream.rs b/crates/canvas-link/src/upstream.rs new file mode 100644 index 0000000..8f4f020 --- /dev/null +++ b/crates/canvas-link/src/upstream.rs @@ -0,0 +1,296 @@ +//! The HTTP(S) / WebSocket client for one linked Canvas Cloud. + +use std::sync::Arc; +use std::time::{Duration, Instant}; + +use reqwest::header::{HeaderValue, AUTHORIZATION}; +use reqwest::{Method, RequestBuilder, StatusCode}; +use serde::Deserialize; +use serde_json::Value; +use tokio_tungstenite::Connector; +use zeroize::Zeroizing; + +use crate::settings::LinkSettings; + +/// TCP + TLS connect budget. +pub const CONNECT_TIMEOUT: Duration = Duration::from_secs(5); +/// Whole-request budget for ordinary calls. +pub const REQUEST_TIMEOUT: Duration = Duration::from_secs(30); +/// Whole-request budget for `/ops` and imports. +pub const LONG_REQUEST_TIMEOUT: Duration = Duration::from_secs(120); +/// Budget for admin/overview probes (the portal must stay responsive when Canvas Cloud is down). +pub const PROBE_TIMEOUT: Duration = Duration::from_secs(5); +/// Header naming the user an app request acts for. +pub const CANVAS_USER_HEADER: &str = "x-canvas-user"; + +/// A configured client for the linked Canvas Cloud. Cheap to clone. +#[derive(Clone)] +pub struct Upstream { + client: reqwest::Client, + base: String, + app_key: Arc>, + ws_tls: Option>, +} + +impl std::fmt::Debug for Upstream { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_struct("Upstream") + .field("base", &self.base) + .finish_non_exhaustive() + } +} + +/// Why a call to Canvas Cloud failed. +#[derive(Debug)] +pub enum UpstreamError { + /// Could not connect / TLS failed / timed out / connection dropped. + Transport(String), + /// Canvas Cloud answered with an unexpected status. + Status { + status: StatusCode, + code: Option, + message: Option, + }, + /// Canvas Cloud answered 2xx with a body we could not read. + Body(String), +} + +impl std::fmt::Display for UpstreamError { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + match self { + Self::Transport(msg) => write!(f, "{msg}"), + Self::Status { + status, + code, + message, + } => { + write!(f, "Canvas Cloud answered {}", status.as_u16())?; + if let Some(c) = code { + write!(f, " {c}")?; + } + if let Some(m) = message.as_deref().filter(|m| !m.is_empty()) { + write!(f, ": {m}")?; + } + Ok(()) + } + Self::Body(msg) => write!(f, "unreadable Canvas Cloud response: {msg}"), + } + } +} + +impl std::error::Error for UpstreamError {} + +impl UpstreamError { + /// Describe a reqwest failure without leaking the URL's query (none is secret, but keep + /// messages short) — "unreachable", "timed out", TLS errors. + pub fn from_reqwest(err: &reqwest::Error) -> Self { + let what = if err.is_timeout() { + "Canvas Cloud did not answer in time".to_owned() + } else if err.is_connect() { + format!("Canvas Cloud is unreachable ({})", root_cause(err)) + } else { + format!("Canvas Cloud request failed ({})", root_cause(err)) + }; + Self::Transport(what) + } + + pub fn status(&self) -> Option { + match self { + Self::Status { status, .. } => Some(*status), + _ => None, + } + } + + /// The upstream JSON `error` code, if any. + pub fn code(&self) -> Option<&str> { + match self { + Self::Status { code, .. } => code.as_deref(), + _ => None, + } + } +} + +/// The innermost error message of a chain (e.g. the rustls error under reqwest's wrapper). +pub fn root_cause(err: &(dyn std::error::Error + 'static)) -> String { + let mut cur: &(dyn std::error::Error + 'static) = err; + while let Some(next) = cur.source() { + cur = next; + } + cur.to_string() +} + +/// `{"error": …, "message": …}` of an error body, if it is one. +pub fn error_fields(body: &[u8]) -> (Option, Option) { + #[derive(Deserialize)] + struct E { + error: Option, + message: Option, + } + serde_json::from_slice::(body).map_or((None, None), |e| (e.error, e.message)) +} + +/// `GET /v1/apps/self/stats`. +#[derive(Clone, Copy, Debug, Default, Deserialize, serde::Serialize, PartialEq, Eq)] +pub struct Stats { + #[serde(default)] + pub users: i64, + #[serde(default)] + pub canvases: i64, + #[serde(default)] + pub rooms: Rooms, +} + +#[derive(Clone, Copy, Debug, Default, Deserialize, serde::Serialize, PartialEq, Eq)] +pub struct Rooms { + #[serde(default)] + pub rooms: i64, + #[serde(default)] + pub peers: i64, +} + +impl Upstream { + /// Build a client for `settings` (TLS pin, timeouts). + pub fn new(settings: &LinkSettings) -> anyhow::Result { + let https = settings.url.scheme() == "https"; + let pin = settings.tls_fingerprint.as_deref(); + let mut builder = reqwest::Client::builder() + .connect_timeout(CONNECT_TIMEOUT) + .pool_idle_timeout(Duration::from_secs(60)) + .redirect(reqwest::redirect::Policy::none()) + .user_agent(concat!("copper-cloud/", env!("CARGO_PKG_VERSION"))); + let ws_tls = if https { + builder = builder.use_preconfigured_tls(crate::tls::client_config(pin)?); + Some(Arc::new(crate::tls::client_config(pin)?)) + } else { + None + }; + Ok(Self { + client: builder.build()?, + base: settings.base(), + app_key: Arc::new(settings.app_key.clone()), + ws_tls, + }) + } + + /// The underlying client (TLS pin applied; no default auth header, no global timeout — + /// set one per request). + pub fn client(&self) -> &reqwest::Client { + &self.client + } + + /// Base URL without a trailing `/`. + pub fn base(&self) -> &str { + &self.base + } + + /// `Bearer ` header value. + pub(crate) fn auth_header(&self) -> HeaderValue { + let mut v = HeaderValue::from_str(&format!("Bearer {}", self.app_key.as_str())) + .unwrap_or_else(|_| HeaderValue::from_static("Bearer invalid")); + v.set_sensitive(true); + v + } + + /// `{base}{path}` where `path` starts with `/` (e.g. `/v1/info`). + pub fn url(&self, path: &str) -> String { + format!("{}{path}", self.base) + } + + /// The WebSocket URL for an absolute `/v1/...` path (`ws(s)://`). + pub fn ws_url(&self, path_and_query: &str) -> String { + let base = self + .base + .strip_prefix("https://") + .map(|r| format!("wss://{r}")) + .or_else(|| { + self.base + .strip_prefix("http://") + .map(|r| format!("ws://{r}")) + }) + .unwrap_or_else(|| self.base.clone()); + format!("{base}{path_and_query}") + } + + /// The tokio-tungstenite connector matching the HTTP client's TLS policy. + pub fn ws_connector(&self) -> Connector { + match &self.ws_tls { + Some(cfg) => Connector::Rustls(cfg.clone()), + None => Connector::Plain, + } + } + + /// An app-authenticated request (`Authorization: Bearer `, 30 s budget). + pub fn request(&self, method: Method, path: &str) -> RequestBuilder { + self.client + .request(method, self.url(path)) + .header(AUTHORIZATION, self.auth_header()) + .timeout(REQUEST_TIMEOUT) + } + + /// Send `rb` and decode a 2xx JSON body; any other status is an [`UpstreamError::Status`]. + pub async fn json( + rb: RequestBuilder, + ) -> Result { + let resp = rb + .send() + .await + .map_err(|e| UpstreamError::from_reqwest(&e))?; + let status = resp.status(); + let mut body = resp + .bytes() + .await + .map_err(|e| UpstreamError::from_reqwest(&e))?; + if !status.is_success() { + let (code, message) = error_fields(&body); + return Err(UpstreamError::Status { + status, + code, + message, + }); + } + if body.iter().all(u8::is_ascii_whitespace) { + // `204 No Content` and friends: decode as JSON `null`. + body = bytes::Bytes::from_static(b"null"); + } + serde_json::from_slice(&body).map_err(|e| UpstreamError::Body(e.to_string())) + } + + /// `GET /v1/info` (unauthenticated on Canvas Cloud; the app key is sent anyway). + pub async fn info(&self, timeout: Duration) -> Result { + Self::json(self.request(Method::GET, "/v1/info").timeout(timeout)).await + } + + /// `GET /v1/apps/self/stats` — also proves the app key is accepted. + pub async fn stats(&self, timeout: Duration) -> Result { + Self::json( + self.request(Method::GET, "/v1/apps/self/stats") + .timeout(timeout), + ) + .await + } + + /// Probe: `/v1/info` + `/v1/apps/self/stats`. Returns the server version and latency. + pub async fn check(&self, timeout: Duration) -> Result { + let started = Instant::now(); + let info = self.info(timeout).await?; + let stats = self.stats(timeout).await?; + Ok(Probe { + version: info + .get("version") + .and_then(Value::as_str) + .map(str::to_owned), + name: info.get("name").and_then(Value::as_str).map(str::to_owned), + latency: started.elapsed(), + stats, + }) + } +} + +/// Result of [`Upstream::check`]. +#[derive(Clone, Debug)] +pub struct Probe { + pub version: Option, + pub name: Option, + pub latency: Duration, + pub stats: Stats, +} diff --git a/crates/canvas-link/src/ws.rs b/crates/canvas-link/src/ws.rs new file mode 100644 index 0000000..1abc2bb --- /dev/null +++ b/crates/canvas-link/src/ws.rs @@ -0,0 +1,362 @@ +//! The y-websocket pipe: `/v1/canvases/{id}/ws` from Copper ↔ the same path on Canvas Cloud. +//! +//! 1. Open the upstream WebSocket **first** (same path, query minus `token=`, app key + +//! `X-Canvas-User`). If Canvas Cloud refuses the handshake with an HTTP status, answer the +//! client with that status/body without upgrading — except 401 (`unknown_user` → provision +//! + retry once; anything else → `502`, never a 401 that would sign Copper out). +//! 2. Upgrade the client (HTTP/1.1 `Upgrade` or HTTP/2 extended CONNECT) and pipe text/binary +//! frames both ways, byte-exact. Each side keeps its own ping/pong; there is no idle +//! timeout (rooms are long-lived). A close from either side is forwarded with the same +//! code and reason (4403, 4404, 1001, …); a lost upstream closes the client with 1011; +//! copper-cloud shutting down closes the client with 1001. + +use std::time::Duration; + +use axum::extract::ws::{ + CloseFrame as AxClose, Message as AxMessage, Utf8Bytes as AxUtf8, WebSocket, WebSocketUpgrade, +}; +use axum::http::{header, HeaderValue, StatusCode}; +use axum::response::{IntoResponse, Response}; +use bytes::Bytes; +use futures::{SinkExt as _, StreamExt as _}; +use tokio::net::TcpStream; +use tokio::sync::watch; +use tokio_tungstenite::tungstenite::client::IntoClientRequest as _; +use tokio_tungstenite::tungstenite::protocol::frame::coding::CloseCode; +use tokio_tungstenite::tungstenite::protocol::{CloseFrame as TClose, WebSocketConfig}; +use tokio_tungstenite::tungstenite::{self, Message as TMessage, Utf8Bytes as TUtf8}; +use tokio_tungstenite::{MaybeTlsStream, WebSocketStream}; + +use crate::proxy::{jit_provision, rejected, Forward}; +use crate::upstream::{error_fields, root_cause, CANVAS_USER_HEADER}; +use crate::{upstream_failed, Active, CanvasLink}; + +/// Largest WebSocket message accepted on either side (images travel as `data:` URLs inside +/// Yjs updates; never cap below what Canvas Cloud accepts). +pub const MAX_MESSAGE_BYTES: usize = 64 << 20; +/// Largest single WebSocket frame accepted on either side. +pub const MAX_FRAME_BYTES: usize = 16 << 20; +/// copper-cloud → Copper keep-alive ping interval (no idle timeout is enforced). +pub const PING_EVERY: Duration = Duration::from_secs(30); +/// Upstream TCP + TLS + WebSocket handshake budget. +pub const HANDSHAKE_TIMEOUT: Duration = Duration::from_secs(15); +/// A frame that cannot be written within this long ends the pipe (a wedged peer). +const SEND_TIMEOUT: Duration = Duration::from_secs(60); +/// How long a forwarded close waits for the other side's close reply. +const CLOSE_GRACE: Duration = Duration::from_secs(5); + +/// Close codes copper-cloud itself sends on the pipe. +pub mod close { + /// copper-cloud is shutting down. + pub const GOING_AWAY: u16 = 1001; + /// The upstream connection was lost; reconnect. + pub const SERVER_ERROR: u16 = 1011; +} + +type Upstream = WebSocketStream>; + +enum ConnectError { + Refused { + status: StatusCode, + content_type: Option, + body: Vec, + }, + Transport(String), +} + +pub(crate) async fn proxy( + link: &CanvasLink, + active: &Active, + fwd: &Forward, + ws: WebSocketUpgrade, +) -> Response { + let mut provisioned = false; + let upstream = loop { + match connect(active, fwd).await { + Ok(stream) => break stream, + Err(ConnectError::Refused { + status, + content_type, + body, + }) => { + if status == StatusCode::UNAUTHORIZED { + let (code, message) = error_fields(&body); + if code.as_deref() == Some("unknown_user") && !provisioned { + provisioned = true; + if let Err(resp) = jit_provision(link, active, fwd.user_id).await { + return resp; + } + continue; + } + return rejected(code.as_deref(), message.as_deref(), fwd.user_id); + } + let mut resp = (status, body).into_response(); + if let Some(ct) = content_type { + resp.headers_mut().insert(header::CONTENT_TYPE, ct); + } + return resp; + } + Err(ConnectError::Transport(msg)) => { + tracing::warn!(error = %msg, "canvas upstream WebSocket failed"); + return upstream_failed(msg); + } + } + }; + let link = link.clone(); + ws.max_message_size(MAX_MESSAGE_BYTES) + .max_frame_size(MAX_FRAME_BYTES) + .on_upgrade(move |socket| pipe(link, socket, upstream)) +} + +async fn connect(active: &Active, fwd: &Forward) -> Result { + let up = &active.upstream; + let mut req = up + .ws_url(&fwd.path_and_query) + .into_client_request() + .map_err(|e| ConnectError::Transport(format!("invalid Canvas Cloud URL ({e})")))?; + let headers = req.headers_mut(); + headers.insert(header::AUTHORIZATION, up.auth_header()); + if let Ok(v) = HeaderValue::from_str(&fwd.user_id.to_string()) { + headers.insert(CANVAS_USER_HEADER, v); + } + if let Some(ip) = fwd.client_ip { + if let Ok(v) = HeaderValue::from_str(&ip.to_string()) { + headers.insert("x-forwarded-for", v); + } + } + let config = WebSocketConfig::default() + .max_message_size(Some(MAX_MESSAGE_BYTES)) + .max_frame_size(Some(MAX_FRAME_BYTES)); + let attempt = tokio_tungstenite::connect_async_tls_with_config( + req, + Some(config), + true, + Some(up.ws_connector()), + ); + match tokio::time::timeout(HANDSHAKE_TIMEOUT, attempt).await { + Err(_) => Err(ConnectError::Transport( + "Canvas Cloud did not answer the WebSocket handshake in time".into(), + )), + Ok(Ok((stream, _))) => Ok(stream), + Ok(Err(tungstenite::Error::Http(resp))) => { + let status = resp.status(); + let content_type = resp.headers().get(header::CONTENT_TYPE).cloned(); + let body = resp.into_body().unwrap_or_default(); + Err(ConnectError::Refused { + status, + content_type, + body, + }) + } + Ok(Err(err)) => Err(ConnectError::Transport(format!( + "Canvas Cloud WebSocket is unreachable ({})", + root_cause(&err) + ))), + } +} + +/// Resolves once `rx` turns `true`; never if its sender is gone. +async fn flagged(rx: &mut watch::Receiver) { + if rx.wait_for(|v| *v).await.is_err() { + std::future::pending::<()>().await; + } +} + +fn to_upstream_close(frame: Option) -> Option { + let f = frame?; + sendable(f.code).then(|| TClose { + code: CloseCode::from(f.code), + reason: TUtf8::try_from(Bytes::from(f.reason)).unwrap_or_default(), + }) +} + +fn to_client_close(frame: Option) -> Option { + let f = frame?; + let code = u16::from(f.code); + sendable(code).then(|| AxClose { + code, + reason: AxUtf8::try_from(Bytes::from(f.reason)).unwrap_or_default(), + }) +} + +/// Codes that may appear on the wire (1005/1006/1015 are local-only). +fn sendable(code: u16) -> bool { + !matches!(code, 1005 | 1006 | 1015) +} + +fn own_close(code: u16, reason: &'static str) -> AxClose { + AxClose { + code, + reason: AxUtf8::from_static(reason), + } +} + +async fn timed>, E>(f: F) -> bool { + matches!(tokio::time::timeout(SEND_TIMEOUT, f).await, Ok(Ok(()))) +} + +#[allow(clippy::too_many_lines)] +async fn pipe(link: CanvasLink, client: WebSocket, upstream: Upstream) { + metrics::gauge!("canvas_link_ws_pipes").increment(1.0); + let mut shutdown = copper_cloud_core::shutdown::subscribe(); + let mut closing = link.closing(); + let (mut c_tx, mut c_rx) = client.split(); + let (mut u_tx, mut u_rx) = upstream.split(); + let mut ping = tokio::time::interval_at(tokio::time::Instant::now() + PING_EVERY, PING_EVERY); + loop { + tokio::select! { + m = c_rx.next() => match m { + Some(Ok(AxMessage::Binary(b))) => { + if !timed(u_tx.send(TMessage::Binary(b))).await { + let _ = c_tx.send(AxMessage::Close(Some(own_close( + close::SERVER_ERROR, "canvas server connection lost")))).await; + break; + } + } + Some(Ok(AxMessage::Text(t))) => { + let t = TUtf8::try_from(Bytes::from(t)).unwrap_or_default(); + if !timed(u_tx.send(TMessage::Text(t))).await { + let _ = c_tx.send(AxMessage::Close(Some(own_close( + close::SERVER_ERROR, "canvas server connection lost")))).await; + break; + } + } + Some(Ok(AxMessage::Ping(_) | AxMessage::Pong(_))) => {} + Some(Ok(AxMessage::Close(frame))) => { + let _ = timed(u_tx.send(TMessage::Close(to_upstream_close(frame)))).await; + // Wait (briefly) for Canvas Cloud's close reply so it sees a clean close. + let _ = tokio::time::timeout(CLOSE_GRACE, async { + while let Some(Ok(m)) = u_rx.next().await { + if matches!(m, TMessage::Close(_)) { + break; + } + } + }) + .await; + let _ = tokio::time::timeout(CLOSE_GRACE, c_tx.close()).await; + break; + } + Some(Err(_)) | None => { + let _ = timed(u_tx.send(TMessage::Close(Some(TClose { + code: CloseCode::Away, + reason: "client went away".into(), + })))) + .await; + break; + } + }, + m = u_rx.next() => match m { + Some(Ok(TMessage::Binary(b))) => { + if !timed(c_tx.send(AxMessage::Binary(b))).await { + let _ = timed(u_tx.send(TMessage::Close(Some(TClose { + code: CloseCode::Away, + reason: "client went away".into(), + })))).await; + break; + } + } + Some(Ok(TMessage::Text(t))) => { + let t = AxUtf8::try_from(Bytes::from(t)).unwrap_or_default(); + if !timed(c_tx.send(AxMessage::Text(t))).await { + let _ = timed(u_tx.send(TMessage::Close(Some(TClose { + code: CloseCode::Away, + reason: "client went away".into(), + })))).await; + break; + } + } + Some(Ok(TMessage::Close(frame))) => { + let _ = timed(c_tx.send(AxMessage::Close(to_client_close(frame)))).await; + // Complete the upstream close handshake (tungstenite queued the reply). + let _ = tokio::time::timeout(CLOSE_GRACE, u_tx.close()).await; + // Give the client a moment to answer our close. + let _ = tokio::time::timeout(CLOSE_GRACE, async { + while let Some(Ok(m)) = c_rx.next().await { + if matches!(m, AxMessage::Close(_)) { + break; + } + } + }) + .await; + break; + } + Some(Ok(_)) => {} + Some(Err(_)) | None => { + let _ = timed(c_tx.send(AxMessage::Close(Some(own_close( + close::SERVER_ERROR, "canvas server connection lost"))))).await; + break; + } + }, + _ = ping.tick() => { + if !timed(c_tx.send(AxMessage::Ping(Bytes::new()))).await { + let _ = timed(u_tx.send(TMessage::Close(Some(TClose { + code: CloseCode::Away, + reason: "client went away".into(), + })))).await; + break; + } + } + () = flagged(&mut shutdown) => { + going_away(&mut c_tx, &mut u_tx).await; + break; + } + () = flagged(&mut closing) => { + going_away(&mut c_tx, &mut u_tx).await; + break; + } + } + } + metrics::gauge!("canvas_link_ws_pipes").decrement(1.0); +} + +async fn going_away(c_tx: &mut C, u_tx: &mut U) +where + C: futures::Sink + Unpin, + U: futures::Sink + Unpin, +{ + let _ = tokio::time::timeout( + CLOSE_GRACE, + c_tx.send(AxMessage::Close(Some(own_close( + close::GOING_AWAY, + "server shutting down", + )))), + ) + .await; + let _ = tokio::time::timeout( + CLOSE_GRACE, + u_tx.send(TMessage::Close(Some(TClose { + code: CloseCode::Away, + reason: "server shutting down".into(), + }))), + ) + .await; +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn close_frames_round_trip() { + let up = to_upstream_close(Some(AxClose { + code: 4403, + reason: "removed".into(), + })) + .unwrap(); + assert_eq!(u16::from(up.code), 4403); + assert_eq!(up.reason.as_str(), "removed"); + let down = to_client_close(Some(TClose { + code: CloseCode::from(4404), + reason: "deleted".into(), + })) + .unwrap(); + assert_eq!(down.code, 4404); + assert_eq!(down.reason.as_str(), "deleted"); + assert!(to_client_close(Some(TClose { + code: CloseCode::from(1005), + reason: "".into(), + })) + .is_none()); + assert!(to_upstream_close(None).is_none()); + } +} diff --git a/crates/canvas-link/tests/support/mock.rs b/crates/canvas-link/tests/support/mock.rs new file mode 100644 index 0000000..6fc10f2 --- /dev/null +++ b/crates/canvas-link/tests/support/mock.rs @@ -0,0 +1,700 @@ +//! In-process mock Canvas Cloud (axum, ephemeral port) implementing just enough of CONTRACT +//! §5.1/§5.3 for the canvas-link tests: app-key + `X-Canvas-User` auth with `unknown_user`, +//! a canvas list, an echo route (what the proxy forwarded), status routes, a WebSocket echo +//! with scripted close codes, the app event stream (SSE), `/v1/info` and +//! `/v1/apps/self/{stats,users…,canvases…}`. +//! +//! Shared by `crates/canvas-link/tests` and `crates/copper-cloud/tests` (via `#[path]`). + +#![allow( + dead_code, + clippy::missing_panics_doc, + clippy::too_many_lines, + clippy::result_large_err +)] + +use std::collections::HashMap; +use std::convert::Infallible; +use std::net::SocketAddr; +use std::path::Path as FsPath; +use std::sync::atomic::{AtomicBool, AtomicU64, AtomicUsize, Ordering}; +use std::sync::{Arc, Mutex}; + +use axum::body::{Body, Bytes}; +use axum::extract::ws::rejection::WebSocketUpgradeRejection; +use axum::extract::ws::{CloseFrame, Message, WebSocket, WebSocketUpgrade}; +use axum::extract::{DefaultBodyLimit, Path, Query, Request, State}; +use axum::http::{header, HeaderMap, HeaderValue, Method, StatusCode, Uri}; +use axum::middleware::Next; +use axum::response::{IntoResponse, Response}; +use axum::routing::{any, delete, get, post, put}; +use axum::{Json, Router}; +use serde_json::{json, Map, Value}; +use tokio::sync::broadcast; + +/// The app key the mock accepts by default. +pub const APP_KEY: &str = "cva_mock_app_key_0123456789abcdef"; + +#[derive(Clone, Debug)] +pub struct MockUser { + pub subject: String, + pub user_id: String, + pub email: String, + pub display_name: Option, + pub disabled: bool, +} + +#[derive(Clone, Debug)] +pub struct MockCanvas { + pub id: String, + pub name: String, + pub kind: String, + /// Owner subject. + pub owner: String, + pub members: Vec, +} + +/// One request as received (no body). +#[derive(Clone, Debug)] +pub struct Recorded { + pub method: String, + pub path: String, + pub query: Option, + pub headers: HashMap, +} + +pub struct MockState { + pub app_key: Mutex, + pub users: Mutex>, + pub canvases: Mutex>, + pub requests: Mutex>, + /// Close frames received from the proxy on the WebSocket: `(code, reason)` + /// (`1005` = a close without a code). + pub ws_closes: Mutex>, + pub disconnects: Mutex>, + pub deleted_users: Mutex>, + pub last_event_ids: Mutex>>, + pub user_puts: AtomicUsize, + pub batch_calls: AtomicUsize, + /// `PUT users/{subject}` answers 500. + pub fail_provision: AtomicBool, + /// `PUT users/{subject}` answers 200 but does not store the user (a second + /// `unknown_user` follows). + pub sticky_unknown: AtomicBool, + pub live_peers: AtomicUsize, + pub event_seq: AtomicU64, + pub events: broadcast::Sender<(u64, String, String)>, +} + +impl MockState { + fn new() -> Self { + Self { + app_key: Mutex::new(APP_KEY.to_owned()), + users: Mutex::default(), + canvases: Mutex::default(), + requests: Mutex::default(), + ws_closes: Mutex::default(), + disconnects: Mutex::default(), + deleted_users: Mutex::default(), + last_event_ids: Mutex::default(), + user_puts: AtomicUsize::new(0), + batch_calls: AtomicUsize::new(0), + fail_provision: AtomicBool::new(false), + sticky_unknown: AtomicBool::new(false), + live_peers: AtomicUsize::new(0), + event_seq: AtomicU64::new(0), + events: broadcast::channel(256).0, + } + } + + /// Accept a different app key from now on (simulates a revoked/rotated key). + pub fn set_app_key(&self, key: &str) { + key.clone_into(&mut self.app_key.lock().unwrap()); + } + + pub fn has_user(&self, subject: &str) -> bool { + self.users.lock().unwrap().contains_key(subject) + } + + pub fn user(&self, subject: &str) -> Option { + self.users.lock().unwrap().get(subject).cloned() + } + + pub fn add_user(&self, subject: &str, email: &str) { + self.users.lock().unwrap().insert( + subject.to_owned(), + MockUser { + subject: subject.to_owned(), + user_id: subject.to_owned(), + email: email.to_owned(), + display_name: None, + disabled: false, + }, + ); + } + + /// Add a canvas owned by `owner` (subject), returning its id. + pub fn add_canvas(&self, owner: &str, name: &str, kind: &str) -> String { + let id = uuid::Uuid::now_v7().to_string(); + self.canvases.lock().unwrap().push(MockCanvas { + id: id.clone(), + name: name.to_owned(), + kind: kind.to_owned(), + owner: owner.to_owned(), + members: vec![owner.to_owned()], + }); + id + } + + pub fn canvas_count(&self) -> usize { + self.canvases.lock().unwrap().len() + } + + /// Requests received for `path` (exact). + pub fn requests_for(&self, path: &str) -> Vec { + self.requests + .lock() + .unwrap() + .iter() + .filter(|r| r.path == path) + .cloned() + .collect() + } + + /// Publish an app `canvas` event; returns its id. + pub fn publish_canvas(&self, subject: &str, canvas_id: &str, kind: &str) -> u64 { + let id = self.event_seq.fetch_add(1, Ordering::SeqCst) + 1; + let data = json!({ + "subject": subject, "user_id": subject, "canvas_id": canvas_id, "kind": kind, + }); + let _ = self.events.send((id, "canvas".into(), data.to_string())); + id + } + + /// Number of connected event-stream clients. + pub fn event_clients(&self) -> usize { + self.events.receiver_count() + } +} + +pub struct Mock { + pub addr: SocketAddr, + pub base: String, + pub st: Arc, +} + +fn router(st: Arc) -> Router { + Router::new() + .route("/v1/info", get(info)) + .route("/v1/canvases", get(list_canvases).post(create_canvas)) + .route("/v1/canvases/{id}/ws", any(ws)) + .route("/v1/canvases/{id}/status/{code}", any(status)) + .route("/v1/canvases/{id}/ops", post(ops)) + .route("/v1/apps/self/stats", get(stats)) + .route("/v1/apps/self/users", get(app_users)) + .route("/v1/apps/self/users:batch", post(batch)) + .route( + "/v1/apps/self/users/{subject}", + put(put_user).delete(delete_user), + ) + .route("/v1/apps/self/users/{subject}/disconnect", post(disconnect)) + .route("/v1/apps/self/events", get(events)) + .route("/v1/apps/self/canvases", get(app_canvases)) + .route("/v1/apps/self/canvases/{id}", delete(app_delete_canvas)) + .fallback(echo) + .layer(DefaultBodyLimit::max(64 << 20)) + .layer(axum::middleware::from_fn_with_state(st.clone(), record)) + .with_state(st) +} + +/// Plain HTTP on `127.0.0.1:0`. +pub async fn start() -> Mock { + let st = Arc::new(MockState::new()); + let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap(); + let addr = listener.local_addr().unwrap(); + let app = router(st.clone()); + tokio::spawn(async move { + axum::serve(listener, app).await.unwrap(); + }); + Mock { + addr, + base: format!("http://{addr}"), + st, + } +} + +/// HTTPS on `127.0.0.1:0` with the given PEM certificate/key. +pub async fn start_tls(cert: &FsPath, key: &FsPath) -> Mock { + copper_cloud_core::tls::install_default_provider(); + let st = Arc::new(MockState::new()); + let config = axum_server::tls_rustls::RustlsConfig::from_pem_file(cert, key) + .await + .unwrap(); + let listener = std::net::TcpListener::bind("127.0.0.1:0").unwrap(); + listener.set_nonblocking(true).unwrap(); + let addr = listener.local_addr().unwrap(); + let app = router(st.clone()); + let server = axum_server::from_tcp_rustls(listener, config).unwrap(); + tokio::spawn(async move { + server.serve(app.into_make_service()).await.unwrap(); + }); + Mock { + addr, + base: format!("https://127.0.0.1:{}", addr.port()), + st, + } +} + +async fn record(State(st): State>, req: Request, next: Next) -> Response { + let headers = req + .headers() + .iter() + .map(|(k, v)| { + ( + k.as_str().to_owned(), + v.to_str().unwrap_or("").to_owned(), + ) + }) + .collect(); + st.requests.lock().unwrap().push(Recorded { + method: req.method().to_string(), + path: req.uri().path().to_owned(), + query: req.uri().query().map(str::to_owned), + headers, + }); + next.run(req).await +} + +fn err(status: u16, code: &str, message: &str) -> Response { + ( + StatusCode::from_u16(status).unwrap(), + Json(json!({ "error": code, "message": message })), + ) + .into_response() +} + +fn app_ok(st: &MockState, h: &HeaderMap) -> bool { + let want = format!("Bearer {}", st.app_key.lock().unwrap()); + h.get(header::AUTHORIZATION).and_then(|v| v.to_str().ok()) == Some(want.as_str()) +} + +fn app_auth(st: &MockState, h: &HeaderMap) -> Result<(), Response> { + if app_ok(st, h) { + Ok(()) + } else { + Err(err(401, "unauthorized", "invalid app key")) + } +} + +fn user_auth(st: &MockState, h: &HeaderMap) -> Result { + app_auth(st, h)?; + let Some(subject) = h.get("x-canvas-user").and_then(|v| v.to_str().ok()) else { + return Err(err(401, "unauthorized", "X-Canvas-User is required")); + }; + match st.users.lock().unwrap().get(subject) { + None => Err(err( + 401, + "unknown_user", + "this app has no user with that subject", + )), + Some(u) if u.disabled => Err(err(403, "user_disabled", "user is disabled")), + Some(u) => Ok(u.clone()), + } +} + +async fn info() -> Json { + Json(json!({ + "name": "canvas-cloud", "version": "0.0.0-mock", "signup": false, + "features": [], "limits": {}, + })) +} + +async fn echo( + State(st): State>, + method: Method, + uri: Uri, + headers: HeaderMap, + body: Bytes, +) -> Response { + let user = match user_auth(&st, &headers) { + Ok(u) => u, + Err(r) => return r, + }; + let hdrs: Map = headers + .iter() + .map(|(k, v)| { + ( + k.as_str().to_owned(), + Value::from(v.to_str().unwrap_or("")), + ) + }) + .collect(); + let mut resp = Json(json!({ + "method": method.as_str(), + "path": uri.path(), + "query": uri.query(), + "headers": hdrs, + "body": String::from_utf8_lossy(&body), + "user": user.subject, + })) + .into_response(); + let h = resp.headers_mut(); + h.insert( + header::CACHE_CONTROL, + HeaderValue::from_static("private, max-age=5"), + ); + h.insert(header::ETAG, HeaderValue::from_static("\"mock-etag\"")); + h.insert( + "x-mock-internal", + HeaderValue::from_static("do-not-forward"), + ); + h.insert(header::SET_COOKIE, HeaderValue::from_static("mock=1")); + resp +} + +async fn status( + State(st): State>, + Path((_id, code)): Path<(String, u16)>, + headers: HeaderMap, +) -> Response { + if let Err(r) = user_auth(&st, &headers) { + return r; + } + let mut resp = err(code, &format!("mock_{code}"), "from the mock Canvas Cloud"); + if code == 429 { + resp.headers_mut() + .insert(header::RETRY_AFTER, HeaderValue::from_static("7")); + } + resp +} + +async fn ops(State(st): State>, headers: HeaderMap, body: Bytes) -> Response { + if let Err(r) = user_auth(&st, &headers) { + return r; + } + Json(json!({ "received": body.len() })).into_response() +} + +async fn list_canvases(State(st): State>, headers: HeaderMap) -> Response { + let user = match user_auth(&st, &headers) { + Ok(u) => u, + Err(r) => return r, + }; + let canvases: Vec = st + .canvases + .lock() + .unwrap() + .iter() + .filter(|c| c.members.contains(&user.subject)) + .map(|c| json!({ "id": c.id, "name": c.name, "kind": c.kind })) + .collect(); + Json(json!({ "canvases": canvases })).into_response() +} + +async fn create_canvas( + State(st): State>, + headers: HeaderMap, + Json(body): Json, +) -> Response { + let user = match user_auth(&st, &headers) { + Ok(u) => u, + Err(r) => return r, + }; + let name = body["name"].as_str().unwrap_or("Untitled"); + let id = st.add_canvas(&user.subject, name, "shared"); + Json(json!({ "id": id, "name": name, "kind": "shared", "role": "owner" })).into_response() +} + +async fn ws( + State(st): State>, + Path(id): Path, + headers: HeaderMap, + up: Result, +) -> Response { + if let Err(r) = user_auth(&st, &headers) { + return r; + } + match id.as_str() { + "forbidden" => return err(403, "forbidden", "not a member of this canvas"), + "missing" => return err(404, "not_found", "no such canvas"), + _ => {} + } + let Ok(up) = up else { + return err(400, "websocket", "WebSocket upgrade required"); + }; + up.max_message_size(64 << 20) + .max_frame_size(16 << 20) + .on_upgrade(move |sock| ws_session(st, sock)) +} + +/// Echo text/binary; text `close ` makes the server close with that code. +async fn ws_session(st: Arc, mut sock: WebSocket) { + st.live_peers.fetch_add(1, Ordering::SeqCst); + while let Some(Ok(msg)) = sock.recv().await { + match msg { + Message::Text(t) => { + if let Some(rest) = t.as_str().strip_prefix("close ") { + let (code, reason) = rest.split_once(' ').unwrap_or((rest, "")); + let _ = sock + .send(Message::Close(Some(CloseFrame { + code: code.parse().unwrap(), + reason: reason.into(), + }))) + .await; + } else if sock.send(Message::Text(t)).await.is_err() { + break; + } + } + Message::Binary(b) => { + if sock.send(Message::Binary(b)).await.is_err() { + break; + } + } + Message::Close(frame) => { + st.ws_closes + .lock() + .unwrap() + .push(frame.map_or((1005, String::new()), |f| (f.code, f.reason.to_string()))); + // Keep reading: that flushes our close reply and ends the stream. + } + Message::Ping(_) | Message::Pong(_) => {} + } + } + st.live_peers.fetch_sub(1, Ordering::SeqCst); +} + +async fn stats(State(st): State>, headers: HeaderMap) -> Response { + if let Err(r) = app_auth(&st, &headers) { + return r; + } + let peers = st.live_peers.load(Ordering::SeqCst); + Json(json!({ + "users": st.users.lock().unwrap().len(), + "canvases": st.canvas_count(), + "rooms": { "rooms": usize::from(peers > 0), "peers": peers }, + })) + .into_response() +} + +fn user_json(u: &MockUser, canvases: &[MockCanvas]) -> Value { + let count = canvases + .iter() + .filter(|c| c.members.contains(&u.subject)) + .count(); + json!({ + "subject": u.subject, "user_id": u.user_id, "email": u.email, + "display_name": u.display_name, "disabled": u.disabled, "canvas_count": count, + }) +} + +async fn app_users( + State(st): State>, + headers: HeaderMap, + Query(q): Query>, +) -> Response { + if let Err(r) = app_auth(&st, &headers) { + return r; + } + let limit: usize = q.get("limit").and_then(|v| v.parse().ok()).unwrap_or(100); + let offset: usize = q.get("offset").and_then(|v| v.parse().ok()).unwrap_or(0); + let canvases = st.canvases.lock().unwrap().clone(); + let mut users: Vec = st.users.lock().unwrap().values().cloned().collect(); + users.sort_by(|a, b| a.subject.cmp(&b.subject)); + let total = users.len(); + let page: Vec = users + .iter() + .skip(offset) + .take(limit) + .map(|u| user_json(u, &canvases)) + .collect(); + Json(json!({ "users": page, "total": total })).into_response() +} + +fn upsert(st: &MockState, subject: &str, body: &Value) -> (MockUser, bool) { + let mut users = st.users.lock().unwrap(); + let created = !users.contains_key(subject); + let user_id = users.get(subject).map_or_else( + || { + body["id"] + .as_str() + .map_or_else(|| subject.to_owned(), str::to_owned) + }, + |u| u.user_id.clone(), + ); + let u = MockUser { + subject: subject.to_owned(), + user_id, + email: body["email"].as_str().unwrap_or_default().to_owned(), + display_name: body["display_name"].as_str().map(str::to_owned), + disabled: body["disabled"].as_bool().unwrap_or(false), + }; + users.insert(subject.to_owned(), u.clone()); + (u, created) +} + +async fn put_user( + State(st): State>, + Path(subject): Path, + headers: HeaderMap, + Json(body): Json, +) -> Response { + if let Err(r) = app_auth(&st, &headers) { + return r; + } + st.user_puts.fetch_add(1, Ordering::SeqCst); + if st.fail_provision.load(Ordering::SeqCst) { + return err(500, "internal", "mock provisioning failure"); + } + if st.sticky_unknown.load(Ordering::SeqCst) { + return Json(json!({ "user_id": subject, "subject": subject, "created": true })) + .into_response(); + } + let (u, created) = upsert(&st, &subject, &body); + Json(json!({ + "user_id": u.user_id, "subject": u.subject, "email": u.email, + "display_name": u.display_name, "disabled": u.disabled, "created": created, + })) + .into_response() +} + +async fn batch( + State(st): State>, + headers: HeaderMap, + Json(body): Json, +) -> Response { + if let Err(r) = app_auth(&st, &headers) { + return r; + } + st.batch_calls.fetch_add(1, Ordering::SeqCst); + let mut results = Vec::new(); + for item in body["users"].as_array().cloned().unwrap_or_default() { + let subject = item["subject"].as_str().unwrap_or_default().to_owned(); + let (u, created) = upsert(&st, &subject, &item); + results.push(json!({ "subject": subject, "user_id": u.user_id, "created": created })); + } + Json(json!({ "results": results })).into_response() +} + +async fn delete_user( + State(st): State>, + Path(subject): Path, + headers: HeaderMap, +) -> Response { + if let Err(r) = app_auth(&st, &headers) { + return r; + } + if st.users.lock().unwrap().remove(&subject).is_none() { + return err(404, "not_found", "no such user"); + } + st.canvases.lock().unwrap().retain(|c| c.owner != subject); + st.deleted_users.lock().unwrap().push(subject); + Json(json!({ "ok": true })).into_response() +} + +async fn disconnect( + State(st): State>, + Path(subject): Path, + headers: HeaderMap, +) -> Response { + if let Err(r) = app_auth(&st, &headers) { + return r; + } + st.disconnects.lock().unwrap().push(subject); + Json(json!({ "ok": true })).into_response() +} + +async fn events(State(st): State>, headers: HeaderMap) -> Response { + if let Err(r) = app_auth(&st, &headers) { + return r; + } + st.last_event_ids.lock().unwrap().push( + headers + .get("last-event-id") + .and_then(|v| v.to_str().ok()) + .map(str::to_owned), + ); + let rx = st.events.subscribe(); + let live = futures::stream::unfold(rx, |mut rx| async move { + loop { + match rx.recv().await { + Ok((id, event, data)) => { + let frame = format!("event: {event}\nid: {id}\ndata: {data}\n\n"); + return Some((Ok::(Bytes::from(frame)), rx)); + } + Err(broadcast::error::RecvError::Lagged(_)) => {} + Err(broadcast::error::RecvError::Closed) => return None, + } + } + }); + let hello = futures::stream::once(async { + Ok::(Bytes::from_static(b": hello\n\n")) + }); + Response::builder() + .header(header::CONTENT_TYPE, "text/event-stream") + .header(header::CACHE_CONTROL, "no-store") + .body(Body::from_stream(futures::StreamExt::chain(hello, live))) + .unwrap() +} + +fn canvas_json(c: &MockCanvas, users: &HashMap) -> Value { + let owner = users.get(&c.owner); + json!({ + "id": c.id, "name": c.name, "kind": c.kind, + "owner": { + "subject": c.owner, + "user_id": owner.map_or(c.owner.as_str(), |u| u.user_id.as_str()), + "email": owner.map(|u| u.email.clone()), + "display_name": owner.and_then(|u| u.display_name.clone()), + }, + "member_count": c.members.len(), + "share_link_count": 0, + "created_at": "2026-10-19T00:00:00Z", + "updated_at": "2026-10-19T00:00:00Z", + }) +} + +async fn app_canvases( + State(st): State>, + headers: HeaderMap, + Query(q): Query>, +) -> Response { + if let Err(r) = app_auth(&st, &headers) { + return r; + } + let limit: usize = q.get("limit").and_then(|v| v.parse().ok()).unwrap_or(100); + let offset: usize = q.get("offset").and_then(|v| v.parse().ok()).unwrap_or(0); + let kind = q.get("kind").filter(|k| !k.is_empty()); + let users = st.users.lock().unwrap().clone(); + let all: Vec = st + .canvases + .lock() + .unwrap() + .iter() + .filter(|c| kind.is_none_or(|k| &c.kind == k)) + .cloned() + .collect(); + let page: Vec = all + .iter() + .skip(offset) + .take(limit) + .map(|c| canvas_json(c, &users)) + .collect(); + Json(json!({ "canvases": page, "total": all.len() })).into_response() +} + +async fn app_delete_canvas( + State(st): State>, + Path(id): Path, + headers: HeaderMap, +) -> Response { + if let Err(r) = app_auth(&st, &headers) { + return r; + } + let mut canvases = st.canvases.lock().unwrap(); + let before = canvases.len(); + canvases.retain(|c| c.id != id); + if canvases.len() == before { + return err(404, "not_found", "no such canvas"); + } + Json(json!({ "ok": true })).into_response() +} diff --git a/crates/canvas-link/tests/upstream.rs b/crates/canvas-link/tests/upstream.rs new file mode 100644 index 0000000..7a2c0c8 --- /dev/null +++ b/crates/canvas-link/tests/upstream.rs @@ -0,0 +1,136 @@ +//! The Canvas Cloud client against the in-process mock: probe, error mapping and TLS pinning +//! (no database needed; the proxy/bridge/admin tests live in `crates/copper-cloud/tests`). + +#![allow(clippy::missing_panics_doc)] + +#[path = "support/mock.rs"] +mod mock; + +use std::path::PathBuf; +use std::time::Duration; + +use copper_cloud_canvas_link::upstream::{Upstream, UpstreamError}; +use copper_cloud_canvas_link::LinkSettings; +use time::OffsetDateTime; +use zeroize::Zeroizing; + +fn settings(base: &str, key: &str, pin: Option<&str>) -> LinkSettings { + LinkSettings { + url: copper_cloud_canvas_link::settings::parse_base_url(base).unwrap(), + app_key: Zeroizing::new(key.to_owned()), + tls_fingerprint: pin.map(str::to_owned), + enabled: true, + linked_at: OffsetDateTime::now_utc(), + updated_at: OffsetDateTime::now_utc(), + updated_by: None, + } +} + +const T: Duration = Duration::from_secs(5); + +#[tokio::test] +async fn probe_and_error_mapping() { + copper_cloud_core::tls::install_default_provider(); + let m = mock::start().await; + m.st.add_user("u1", "u1@example.com"); + m.st.add_canvas("u1", "One", "shared"); + + let up = Upstream::new(&settings(&m.base, mock::APP_KEY, None)).unwrap(); + let probe = up.check(T).await.unwrap(); + assert_eq!(probe.version.as_deref(), Some("0.0.0-mock")); + assert_eq!(probe.name.as_deref(), Some("canvas-cloud")); + assert_eq!(probe.stats.users, 1); + assert_eq!(probe.stats.canvases, 1); + // The app key travels as a bearer token, never as X-Canvas-User. + let seen = m.st.requests_for("/v1/apps/self/stats"); + assert_eq!( + seen[0].headers["authorization"], + format!("Bearer {}", mock::APP_KEY) + ); + assert!(!seen[0].headers.contains_key("x-canvas-user")); + + // A wrong key: a Status error with the upstream code (the admin API turns it into 400). + let bad = Upstream::new(&settings(&m.base, "cva_wrong_key_123456", None)).unwrap(); + let e = bad.check(T).await.unwrap_err(); + assert_eq!(e.status().map(|s| s.as_u16()), Some(401)); + assert_eq!(e.code(), Some("unauthorized")); + assert!(e.to_string().contains("401"), "{e}"); + + // Nothing listening: a transport error. + let port = { + let l = std::net::TcpListener::bind("127.0.0.1:0").unwrap(); + l.local_addr().unwrap().port() + }; + let down = Upstream::new(&settings( + &format!("http://127.0.0.1:{port}"), + mock::APP_KEY, + None, + )) + .unwrap(); + let e = down.check(T).await.unwrap_err(); + assert!(matches!(e, UpstreamError::Transport(_)), "{e:?}"); + assert!(e.to_string().contains("unreachable"), "{e}"); +} + +fn temp_dir() -> PathBuf { + let dir = std::env::temp_dir().join(format!("cc-link-tls-{}", uuid::Uuid::now_v7())); + std::fs::create_dir_all(&dir).unwrap(); + dir +} + +#[tokio::test] +async fn tls_fingerprint_pinning() { + copper_cloud_core::tls::install_default_provider(); + let dir = temp_dir(); + let (cert, key) = (dir.join("cert.pem"), dir.join("key.pem")); + copper_cloud_core::tls::generate_self_signed( + &["canvas.internal".to_owned(), "127.0.0.1".to_owned()], + &cert, + &key, + ) + .unwrap(); + let fp = copper_cloud_core::tls::leaf_fingerprint(&cert).unwrap(); + let m = mock::start_tls(&cert, &key).await; + assert!(m.base.starts_with("https://")); + + // Pinned to the right certificate: accepted (names and issuer do not matter). + let pinned = Upstream::new(&settings(&m.base, mock::APP_KEY, Some(&fp))).unwrap(); + let probe = pinned.check(T).await.unwrap(); + assert_eq!(probe.version.as_deref(), Some("0.0.0-mock")); + + // Pinned to another certificate: refused. + let wrong = Upstream::new(&settings(&m.base, mock::APP_KEY, Some(&"ab".repeat(32)))).unwrap(); + let e = wrong.check(T).await.unwrap_err(); + assert!(matches!(e, UpstreamError::Transport(_)), "{e:?}"); + assert!(e.to_string().contains("pinned fingerprint"), "{e}"); + + // No pin: the web PKI does not know a self-signed certificate. + let unpinned = Upstream::new(&settings(&m.base, mock::APP_KEY, None)).unwrap(); + let e = unpinned.check(T).await.unwrap_err(); + assert!(matches!(e, UpstreamError::Transport(_)), "{e:?}"); + + // The WebSocket connector follows the same policy: pinned connects, wrong pin fails. + for (up, ok) in [(&pinned, true), (&wrong, false)] { + let mut req = + tokio_tungstenite::tungstenite::client::IntoClientRequest::into_client_request( + up.ws_url("/v1/canvases/c1/ws"), + ) + .unwrap(); + req.headers_mut().insert( + "authorization", + format!("Bearer {}", mock::APP_KEY).parse().unwrap(), + ); + m.st.add_user("ws-user", "ws@example.com"); + req.headers_mut() + .insert("x-canvas-user", "ws-user".parse().unwrap()); + let r = tokio_tungstenite::connect_async_tls_with_config( + req, + None, + true, + Some(up.ws_connector()), + ) + .await; + assert_eq!(r.is_ok(), ok, "{:?}", r.err()); + } + let _ = std::fs::remove_dir_all(dir); +} diff --git a/crates/canvas/src/admin.rs b/crates/canvas/src/admin.rs deleted file mode 100644 index dc45426..0000000 --- a/crates/canvas/src/admin.rs +++ /dev/null @@ -1,57 +0,0 @@ -//! Instance-admin operations used by the admin API (`/admin/api/*`, in the binary crate). -//! They bypass membership checks — callers must have authenticated an admin. - -use copper_cloud_core::error::ApiError; -use copper_cloud_core::state::AppState; -use uuid::Uuid; - -use crate::rest::{member_ids, publish}; -use crate::room::rooms; - -/// Delete any canvas (Personal ones too — the owner gets a fresh empty one on next use): -/// its updates, snapshot, members and invites go with it (FK cascade), connected peers are -/// disconnected (close 4404) and members get a `canvas` `deleted` event. Returns the -/// deleted canvas's name, `None` if it did not exist. -pub async fn delete_canvas(state: &AppState, canvas_id: Uuid) -> Result, ApiError> { - let members = member_ids(state, canvas_id).await?; - let name: Option = - sqlx::query_scalar("DELETE FROM canvases WHERE id = $1 RETURNING name") - .bind(canvas_id) - .fetch_optional(&state.db) - .await?; - if name.is_some() { - canvases_deleted(state, &[(canvas_id, members)]); - tracing::info!(%canvas_id, "canvas deleted by admin"); - } - Ok(name) -} - -/// The canvases `user_id` owns, each with its member ids — collect this **before** deleting -/// the user, then pass it to [`canvases_deleted`] after the delete commits. -pub async fn owned_canvases( - state: &AppState, - user_id: Uuid, -) -> Result)>, ApiError> { - let ids: Vec = sqlx::query_scalar("SELECT id FROM canvases WHERE owner_id = $1") - .bind(user_id) - .fetch_all(&state.db) - .await?; - let mut out = Vec::with_capacity(ids.len()); - for id in ids { - out.push((id, member_ids(state, id).await?)); - } - Ok(out) -} - -/// After canvases were deleted in the database: close their rooms and notify members. -pub fn canvases_deleted(state: &AppState, deleted: &[(Uuid, Vec)]) { - for (canvas_id, members) in deleted { - rooms().close(*canvas_id); - publish(state, members, *canvas_id, "deleted"); - } -} - -/// Disconnect every live canvas peer of `user_id` (account disabled or deleted). -pub fn disconnect_user(user_id: Uuid) { - rooms().kick_user_everywhere(user_id); -} diff --git a/crates/canvas/src/chat.rs b/crates/canvas/src/chat.rs deleted file mode 100644 index 8161f5b..0000000 --- a/crates/canvas/src/chat.rs +++ /dev/null @@ -1,175 +0,0 @@ -//! Canvas chat (0.6.0) as the server sees it: a top-level `Y.Array` named `chat` in the canvas -//! document, owned and written by Copper. Each element is a plain object -//! `{id, authorId, authorName, text, mentions: [userId], at, editedAt?, deleted?}` (`at` and -//! `editedAt` are ms epoch). The server never writes it: rooms persist and relay it like any -//! other part of the document, and `GET /canvases/{id}/read` includes the most recent messages -//! (see [`read_chat`]). Mention notifications are in [`crate::mentions`]. - -use serde::Serialize; -use serde_json::Value; -use yrs::{Array as _, ReadTxn}; - -use crate::read::{ReadResult, READ_TEXT_LIMIT}; -use crate::schema::{js_len, js_slice, json_f64, out_to_json, ser_num, ser_opt_num}; - -/// Name of the document root that holds the chat. -pub const CHAT: &str = "chat"; -/// Messages `GET /canvases/{id}/read` returns (the most recent ones). -pub const READ_CHAT_LIMIT: usize = 50; - -/// One chat message in the read format. -#[derive(Debug, Clone, PartialEq, Serialize)] -pub struct ChatSummary { - pub id: String, - #[serde(rename = "authorId")] - pub author_id: String, - #[serde(rename = "authorName")] - pub author_name: String, - /// Cut to 500 characters plus `…` unless `full`. - pub text: String, - /// User ids mentioned in the message. - pub mentions: Vec, - /// When it was sent (ms epoch). - #[serde(serialize_with = "ser_num")] - pub at: f64, - #[serde( - rename = "editedAt", - skip_serializing_if = "Option::is_none", - serialize_with = "ser_opt_num" - )] - pub edited_at: Option, -} - -/// `GET /canvases/{id}/read`: the [`ReadResult`] plus `chat` (oldest first). -#[derive(Debug, Clone, PartialEq, Serialize)] -pub struct ReadWithChat { - #[serde(flatten)] - pub read: ReadResult, - pub chat: Vec, -} - -fn string(v: &Value, key: &str) -> String { - v.get(key) - .and_then(Value::as_str) - .unwrap_or_default() - .to_owned() -} - -fn clip(s: &str, full: bool) -> String { - if full || js_len(s) <= READ_TEXT_LIMIT { - s.to_owned() - } else { - format!("{}…", js_slice(s, READ_TEXT_LIMIT)) - } -} - -/// One stored message, or `None` when it is deleted or not a message at all (readers are -/// tolerant: anything without a string `id` is skipped, missing fields read as empty). -fn summarize(v: &Value, full: bool) -> Option { - let id = v.get("id")?.as_str().filter(|s| !s.is_empty())?; - if v.get("deleted").and_then(Value::as_bool) == Some(true) { - return None; - } - Some(ChatSummary { - id: id.to_owned(), - author_id: string(v, "authorId"), - author_name: string(v, "authorName"), - text: clip(&string(v, "text"), full), - mentions: v - .get("mentions") - .and_then(Value::as_array) - .map(|m| { - m.iter() - .filter_map(Value::as_str) - .map(str::to_owned) - .collect() - }) - .unwrap_or_default(), - at: v.get("at").and_then(json_f64).unwrap_or(0.0), - edited_at: v.get("editedAt").and_then(json_f64), - }) -} - -/// The last `limit` messages of the document's `chat` array, oldest first, in array order -/// (the order every client shows). Deleted messages are left out; text is cut to 500 -/// characters unless `full`. Empty when the document has no chat. -pub fn read_chat(txn: &T, limit: usize, full: bool) -> Vec { - let Some(chat) = txn.get_array(CHAT) else { - return Vec::new(); - }; - let all: Vec = chat.iter(txn).map(|o| out_to_json(txn, &o)).collect(); - let mut out: Vec = all - .iter() - .rev() - .filter_map(|v| summarize(v, full)) - .take(limit) - .collect(); - out.reverse(); - out -} - -#[cfg(test)] -#[allow(clippy::float_cmp)] -mod tests { - use super::*; - use crate::schema::json_to_any; - use serde_json::json; - use yrs::{Doc, Transact as _, WriteTxn as _}; - - fn push(doc: &Doc, v: &Value) { - let mut txn = doc.transact_mut(); - let chat = txn.get_or_insert_array(CHAT); - chat.push_back(&mut txn, json_to_any(v)); - } - - #[test] - fn reads_the_tail_and_skips_deleted_and_junk() { - let doc = Doc::new(); - assert!( - read_chat(&doc.transact(), 50, false).is_empty(), - "no chat yet" - ); - for i in 0..60 { - push( - &doc, - &json!({"id": format!("m{i}"), "authorId": "u1", "authorName": "Ann", - "text": format!("hello {i}"), "mentions": ["u2"], "at": 1_000 + i}), - ); - } - push( - &doc, - &json!({"id": "gone", "text": "x", "at": 2000, "deleted": true}), - ); - push(&doc, &json!("not a message")); - push(&doc, &json!({"text": "no id"})); - push( - &doc, - &json!({"id": "long", "authorName": "Ben", "text": "y".repeat(800), "at": 3000, - "editedAt": 3001.5}), - ); - - let txn = doc.transact(); - let got = read_chat(&txn, 50, false); - assert_eq!(got.len(), 50); - assert_eq!(got[0].id, "m11", "oldest of the last 50"); - assert_eq!(got[48].id, "m59"); - let last = &got[49]; - assert_eq!(last.id, "long"); - assert_eq!(js_len(&last.text), READ_TEXT_LIMIT + 1); - assert_eq!(last.author_id, "", "missing fields read as empty"); - assert_eq!(last.mentions.len(), 0); - assert_eq!(last.edited_at, Some(3001.5)); - assert_eq!(got[48].mentions, vec!["u2".to_owned()]); - - let full = read_chat(&txn, 2, true); - assert_eq!(full.len(), 2); - assert_eq!(full[1].text.len(), 800); - - let v = serde_json::to_value(&got[48]).unwrap(); - assert_eq!( - v, - json!({"id": "m59", "authorId": "u1", "authorName": "Ann", "text": "hello 59", - "mentions": ["u2"], "at": 1059}) - ); - } -} diff --git a/crates/canvas/src/checklist.rs b/crates/canvas/src/checklist.rs deleted file mode 100644 index 042d4c4..0000000 --- a/crates/canvas/src/checklist.rs +++ /dev/null @@ -1,744 +0,0 @@ -//! The checklist (RSVP) shape — the page's `Canvas/src/canvas/checklist.ts`: a title, -//! 1–4 columns ("Yes", "No") and up to 60 rows, each row with at most one pick. -//! -//! Stored on the shape's own `Y.Map`, beside the usual box props: -//! -//! ```text -//! title string -//! columns string[] whole value, last writer wins -//! rows {id, label}[] whole value, last writer wins -//! pick: {col, by, byId, at} one key per row -//! ``` -//! -//! A pick lives under its row's own key, so two people picking different rows at the -//! same moment both stick. `col` is the column's label; a pick naming no current column -//! (or no current row) reads as no pick. - -use serde::Serialize; -use serde_json::{json, Map as JsonMap, Value}; -use yrs::{Any, Map, MapRef, ReadTxn, TransactionMut}; - -use crate::schema::{jnum, js_slice, json_f64, json_to_any}; - -/// Prefix of the per-row pick keys. -pub const PICK_PREFIX: &str = "pick:"; -pub const MAX_COLUMNS: usize = 4; -pub const MAX_ROWS: usize = 60; -/// Labels are cut to these many UTF-16 units (as JavaScript counts). -pub const MAX_COLUMN_LABEL: usize = 40; -pub const MAX_ROW_LABEL: usize = 200; -pub const DEFAULT_COLUMNS: [&str; 2] = ["Yes", "No"]; - -/// Layout numbers shared with the page (world px), for a fresh card's size. -const HEAD: f64 = 49.0; -const COLUMNS_HEAD: f64 = 28.0; -const ROW: f64 = 34.0; -const FOOT: f64 = 14.0; -const COL: f64 = 64.0; -const LABEL: f64 = 168.0; -const PAD: f64 = 14.0; - -/// The pick key of a row. -pub fn pick_key(row_id: &str) -> String { - format!("{PICK_PREFIX}{row_id}") -} - -/// A fresh card's width for this many columns. -#[allow(clippy::cast_precision_loss)] -pub fn checklist_width(columns: usize) -> f64 { - if columns <= 1 { - 280.0 - } else { - PAD * 2.0 + LABEL + COL * columns as f64 - } -} - -/// The height a card needs before anyone has measured it (one line per label). -#[allow(clippy::cast_precision_loss)] -pub fn checklist_height(columns: usize, rows: usize) -> f64 { - HEAD + if columns > 1 { COLUMNS_HEAD } else { 0.0 } + ROW * rows.max(1) as f64 + FOOT -} - -/// One row: a person or a task. -#[derive(Debug, Clone, PartialEq, Eq)] -pub struct Row { - pub id: String, - pub label: String, -} - -impl Row { - pub fn to_json(&self) -> Value { - json!({ "id": self.id, "label": self.label }) - } -} - -/// A row's pick: the column's label, who made it and when. -#[derive(Debug, Clone, PartialEq)] -pub struct Pick { - pub col: String, - pub by: String, - pub by_id: String, - pub at: f64, -} - -impl Pick { - pub fn to_json(&self) -> Value { - json!({ "col": self.col, "by": self.by, "byId": self.by_id, "at": jnum(self.at) }) - } -} - -/// A checklist as read from the document. -#[derive(Debug, Clone, PartialEq, Default)] -pub struct Checklist { - pub columns: Vec, - pub rows: Vec, - /// Picks that name a current row and column, in row order. - pub picks: Vec<(String, Pick)>, -} - -impl Checklist { - pub fn pick(&self, row_id: &str) -> Option<&Pick> { - self.picks.iter().find(|(r, _)| r == row_id).map(|(_, p)| p) - } -} - -/// `s.trim().toLowerCase()`. -fn fold(s: &str) -> String { - s.trim().to_lowercase() -} - -/// `JSON.stringify(s)`, for messages. -fn quoted(s: &str) -> String { - serde_json::to_string(s).unwrap_or_default() -} - -// ---- tolerant reads ----------------------------------------------------------------------- - -/// Columns as stored: strings, deduplicated, at most four; none → Yes / No. -pub fn read_columns(v: Option<&Value>) -> Vec { - let mut out: Vec = Vec::new(); - for c in v.and_then(Value::as_array).into_iter().flatten() { - let Some(c) = c.as_str() else { continue }; - if c.trim().is_empty() || out.iter().any(|o| fold(o) == fold(c)) { - continue; - } - out.push(c.to_owned()); - if out.len() == MAX_COLUMNS { - break; - } - } - if out.is_empty() { - DEFAULT_COLUMNS.iter().map(|c| (*c).to_owned()).collect() - } else { - out - } -} - -/// Rows as stored: `{id, label}` with unique ids, at most sixty. -pub fn read_rows(v: Option<&Value>) -> Vec { - let mut out: Vec = Vec::new(); - for r in v.and_then(Value::as_array).into_iter().flatten() { - let Some(o) = r.as_object() else { continue }; - let Some(id) = o - .get("id") - .and_then(Value::as_str) - .filter(|s| !s.is_empty()) - else { - continue; - }; - if out.iter().any(|r| r.id == id) { - continue; - } - out.push(Row { - id: id.to_owned(), - label: o - .get("label") - .and_then(Value::as_str) - .unwrap_or_default() - .to_owned(), - }); - if out.len() == MAX_ROWS { - break; - } - } - out -} - -/// One stored pick, if it names a column. -pub fn read_pick(v: Option<&Value>) -> Option { - let o = v?.as_object()?; - let col = o - .get("col") - .and_then(Value::as_str) - .filter(|s| !s.is_empty())?; - let s = |k: &str| { - o.get(k) - .and_then(Value::as_str) - .unwrap_or_default() - .to_owned() - }; - Some(Pick { - col: col.to_owned(), - by: s("by"), - by_id: s("byId"), - at: o.get("at").and_then(json_f64).unwrap_or(0.0), - }) -} - -/// The checklist on a shape map; `get` reads one prop as JSON. -pub fn read_checklist(get: &dyn Fn(&str) -> Option) -> Checklist { - let columns = read_columns(get("columns").as_ref()); - let rows = read_rows(get("rows").as_ref()); - let picks = rows - .iter() - .filter_map(|r| { - let p = read_pick(get(&pick_key(&r.id)).as_ref())?; - columns.contains(&p.col).then(|| (r.id.clone(), p)) - }) - .collect(); - Checklist { - columns, - rows, - picks, - } -} - -/// How many rows picked each column, in column order. -pub fn tally(c: &Checklist) -> Vec<(String, u64)> { - c.columns - .iter() - .map(|col| { - let n = c.picks.iter().filter(|(_, p)| &p.col == col).count(); - (col.clone(), n as u64) - }) - .collect() -} - -/// `{column: count}` in column order (a JSON object, as the page prints it). -#[allow(clippy::ref_option)] -pub fn ser_tally( - v: &Option>, - s: S, -) -> Result { - use serde::ser::SerializeMap as _; - let items = v.as_deref().unwrap_or_default(); - let mut m = s.serialize_map(Some(items.len()))?; - for (k, n) in items { - m.serialize_entry(k, n)?; - } - m.end() -} - -/// One row as `canvas_read` shows it: `{id, label, pick, by?, at?}`. -#[derive(Debug, Clone, PartialEq, Serialize)] -pub struct RowSummary { - pub id: String, - pub label: String, - pub pick: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub by: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub at: Option, -} - -/// The rows as `canvas_read` shows them. -pub fn row_summaries(c: &Checklist) -> Vec { - c.rows - .iter() - .map(|r| { - let p = c.pick(&r.id); - RowSummary { - id: r.id.clone(), - label: r.label.clone(), - pick: p.map(|p| p.col.clone()), - by: p.map(|p| p.by.clone()), - at: p.map(|p| jnum(p.at)), - } - }) - .collect() -} - -// ---- input (ops) -------------------------------------------------------------------------- - -/// `columns`: 1–4 distinct non-empty strings, each cut to 40 characters. -pub fn clean_columns(v: &Value) -> Result, String> { - let list = v - .as_array() - .filter(|l| (1..=MAX_COLUMNS).contains(&l.len())) - .ok_or("`columns` must be a list of 1 to 4 names")?; - let mut out: Vec = Vec::new(); - for c in list { - let name = c - .as_str() - .filter(|s| !s.trim().is_empty()) - .ok_or("`columns` names must be non-empty strings")?; - let name = js_slice(name.trim(), MAX_COLUMN_LABEL).to_owned(); - if out.iter().any(|o| fold(o) == fold(&name)) { - return Err(format!("`columns` has {} twice", quoted(&name))); - } - out.push(name); - } - Ok(out) -} - -/// `^[A-Za-z0-9_-]{1,32}$` -fn valid_row_id(s: &str) -> bool { - (1..=32).contains(&s.len()) - && s.bytes() - .all(|b| b.is_ascii_alphanumeric() || matches!(b, b'_' | b'-')) -} - -/// A fresh row id: `r` and eight base-36 characters. -pub fn new_row_id() -> String { - let mut n = uuid::Uuid::new_v4().as_u128(); - let mut id = String::from("r"); - for _ in 0..8 { - let d = u32::try_from(n % 36).unwrap_or(0); - id.push(char::from_digit(d, 36).unwrap_or('0')); - n /= 36; - } - id -} - -/// `rows`: up to 60 labels (`"Ann"`) or `{label, id?}`. A row keeps its id — and so -/// its pick — when it names an existing row's id, or failing that has the same label -/// as an existing row not yet taken. -pub fn clean_rows(v: &Value, existing: &[Row]) -> Result, String> { - const SHAPE: &str = "`rows` must be a list of labels or {label, id?}"; - let list = v.as_array().ok_or(SHAPE)?; - if list.len() > MAX_ROWS { - return Err(format!("`rows`: at most {MAX_ROWS} rows")); - } - let mut given: Vec<(String, Option)> = Vec::with_capacity(list.len()); - for r in list { - match r { - Value::String(s) => given.push((js_slice(s.trim(), MAX_ROW_LABEL).to_owned(), None)), - Value::Object(o) => { - let label = [o.get("label"), o.get("text"), o.get("title")] - .into_iter() - .flatten() - .find(|v| !v.is_null()) - .unwrap_or(&Value::Null); - let label = match label { - Value::Null => "", - Value::String(s) => s.as_str(), - _ => return Err("`rows`: a label must be a string".into()), - }; - let id = match o.get("id") { - None | Some(Value::Null) => None, - Some(Value::String(s)) if valid_row_id(s) => Some(s.clone()), - Some(_) => return Err("`rows`: an id must be 1–32 of A–Z a–z 0–9 _ -".into()), - }; - given.push((js_slice(label.trim(), MAX_ROW_LABEL).to_owned(), id)); - } - _ => return Err(SHAPE.into()), - } - } - let mut taken: Vec = Vec::new(); - for id in given.iter().filter_map(|(_, id)| id.as_ref()) { - if taken.contains(id) { - return Err(format!("`rows`: id {id} is used twice")); - } - taken.push(id.clone()); - } - let mut out = Vec::with_capacity(given.len()); - for (label, id) in given { - let id = if let Some(id) = id { - id - } else { - let mut id = existing - .iter() - .find(|r| r.label == label && !taken.contains(&r.id)) - .map(|r| r.id.clone()); - while id.as_ref().is_none_or(|i| taken.contains(i)) { - id = Some(new_row_id()); - } - let id = id.unwrap_or_default(); - taken.push(id.clone()); - id - }; - out.push(Row { id, label }); - } - Ok(out) -} - -/// The row `reference` names: its id, its label, or its label in any case. -pub fn find_row<'a>(rows: &'a [Row], reference: &str) -> Result<&'a Row, String> { - if let Some(r) = rows.iter().find(|r| r.id == reference) { - return Ok(r); - } - let exact: Vec<&Row> = rows.iter().filter(|r| r.label == reference).collect(); - let folded: Vec<&Row> = rows - .iter() - .filter(|r| fold(&r.label) == fold(reference)) - .collect(); - for hits in [exact, folded] { - match hits.len() { - 0 => {} - 1 => return Ok(hits[0]), - _ => { - return Err(format!( - "`picks`: more than one row is called {} — use its id", - quoted(reference) - )) - } - } - } - Err(format!("`picks`: no row {}", quoted(reference))) -} - -/// `picks`: `{rowIdOrLabel: column | true | null}` → `[(rowId, column | None)]`. A column -/// is named in any case; `true` is the first column; null, false or "" clears the row. -pub fn clean_picks( - v: &Value, - rows: &[Row], - columns: &[String], -) -> Result)>, String> { - let o = v - .as_object() - .ok_or("`picks` must be an object like {\"Ann\": \"Yes\"}")?; - let mut out: Vec<(String, Option)> = Vec::new(); - for (reference, value) in o { - let row = find_row(rows, reference)?; - let col = match value { - Value::Null | Value::Bool(false) => None, - Value::String(s) if s.is_empty() => None, - Value::Bool(true) => columns.first().cloned(), - Value::String(s) => Some( - columns - .iter() - .find(|c| fold(c) == fold(s)) - .cloned() - .ok_or_else(|| { - format!("`picks`: no column {} ({})", quoted(s), columns.join(", ")) - })?, - ), - _ => return Err("`picks` values must be a column name, true or null".into()), - }; - if let Some(at) = out.iter().position(|(r, _)| *r == row.id) { - out[at].1 = col; - } else { - out.push((row.id.clone(), col)); - } - } - Ok(out) -} - -// ---- writes (inside a transaction) --------------------------------------------------------- - -/// Who a pick is attributed to. -#[derive(Debug, Clone)] -pub struct Picker<'a> { - pub name: &'a str, - pub id: &'a str, - pub at: i64, -} - -fn map_json(txn: &T, m: &MapRef, k: &str) -> Option { - m.get(txn, k).map(|o| crate::schema::out_to_json(txn, &o)) -} - -/// Set (or with `col` `None`, clear) one row's pick. Touches no other row. -pub fn write_pick( - txn: &mut TransactionMut, - m: &MapRef, - row_id: &str, - col: Option<&str>, - by: &Picker, -) { - let key = pick_key(row_id); - match col { - None => { - if m.contains_key(txn, &key) { - m.remove(txn, &key); - } - } - Some(col) => { - let pick = Pick { - col: col.to_owned(), - by: by.name.to_owned(), - by_id: by.id.to_owned(), - #[allow(clippy::cast_precision_loss)] - at: by.at as f64, - }; - m.insert(txn, key, json_to_any(&pick.to_json())); - } - } -} - -/// New columns. A pick naming a column whose label is unchanged but for case follows -/// it; a pick naming a column that has gone is dropped. -pub fn write_columns(txn: &mut TransactionMut, m: &MapRef, next: &[String]) { - m.insert( - txn, - "columns", - Any::from( - next.iter() - .map(|c| Any::from(c.as_str())) - .collect::>(), - ), - ); - let keys: Vec = m - .keys(txn) - .filter(|k| k.starts_with(PICK_PREFIX)) - .map(str::to_owned) - .collect(); - for key in keys { - let Some(pick) = read_pick(map_json(txn, m, &key).as_ref()) else { - m.remove(txn, &key); - continue; - }; - match next.iter().find(|c| fold(c) == fold(&pick.col)) { - None => { - m.remove(txn, &key); - } - Some(to) if *to != pick.col => { - let moved = Pick { - col: to.clone(), - ..pick - }; - m.insert(txn, key, json_to_any(&moved.to_json())); - } - Some(_) => {} - } - } -} - -/// New rows; the picks of rows that are gone go with them. -pub fn write_rows(txn: &mut TransactionMut, m: &MapRef, next: &[Row]) { - let rows: Vec = next.iter().map(Row::to_json).collect(); - m.insert(txn, "rows", json_to_any(&Value::Array(rows))); - let keep: Vec = next.iter().map(|r| pick_key(&r.id)).collect(); - let gone: Vec = m - .keys(txn) - .filter(|k| k.starts_with(PICK_PREFIX) && !keep.iter().any(|kk| kk == k)) - .map(str::to_owned) - .collect(); - for key in gone { - m.remove(txn, &key); - } -} - -/// The checklist props of an add or update, split from the plain ones: `columns` and -/// `rows` (already cleaned) and the raw `picks`. -#[derive(Debug, Default)] -pub struct ChecklistInput { - pub columns: Option>, - pub rows: Option>, - pub picks: Option, -} - -impl ChecklistInput { - /// Takes `columns`, `rows` and `picks` out of cleaned props. - pub fn take(props: &mut JsonMap) -> Self { - let columns = props.remove("columns").map(|v| read_columns(Some(&v))); - let rows = props.remove("rows").map(|v| { - v.as_array() - .into_iter() - .flatten() - .filter_map(|r| { - Some(Row { - id: r.get("id")?.as_str()?.to_owned(), - label: r.get("label")?.as_str()?.to_owned(), - }) - }) - .collect() - }); - Self { - columns, - rows, - picks: props.remove("picks"), - } - } -} - -/// The page's `writeChecklist`: new rows and columns (dropping the picks they orphan), -/// then the op's picks; fresh lists also size the card unless the op sized it. -pub fn write_checklist( - txn: &mut TransactionMut, - m: &MapRef, - input: &ChecklistInput, - by: &Picker, - sized_w: bool, - sized_h: bool, -) -> Result<(), String> { - let live = read_checklist(&|k| map_json(txn, m, k)); - let rows = input.rows.as_deref().unwrap_or(&live.rows); - let columns = input.columns.as_deref().unwrap_or(&live.columns); - let resolved = match &input.picks { - None => Vec::new(), - Some(p) => clean_picks(p, rows, columns)?, - }; - let (rows, columns) = (rows.to_vec(), columns.to_vec()); - if input.columns.is_some() { - write_columns(txn, m, &columns); - } - if input.rows.is_some() { - write_rows(txn, m, &rows); - } - for (row, col) in &resolved { - write_pick(txn, m, row, col.as_deref(), by); - } - if input.columns.is_some() && !sized_w { - let w = map_json(txn, m, "w") - .as_ref() - .and_then(json_f64) - .unwrap_or(0.0); - let w = w.max(checklist_width(columns.len())); - m.insert(txn, "w", crate::schema::num_any(w)); - } - if (input.rows.is_some() || input.columns.is_some()) && !sized_h { - let h = checklist_height(columns.len(), rows.len()); - m.insert(txn, "h", crate::schema::num_any(h)); - } - Ok(()) -} - -#[cfg(test)] -mod tests { - use super::*; - - fn rows(labels: &[(&str, &str)]) -> Vec { - labels - .iter() - .map(|(id, label)| Row { - id: (*id).to_owned(), - label: (*label).to_owned(), - }) - .collect() - } - - #[test] - fn reads_tolerantly() { - assert_eq!(read_columns(None), ["Yes", "No"]); - assert_eq!( - read_columns(Some(&json!([ - "Yes", "yes", "", 3, "No", "Maybe", "Later", "Never" - ]))), - ["Yes", "No", "Maybe", "Later"] - ); - assert_eq!( - read_rows(Some( - &json!([{"id": "a", "label": "A"}, {"id": "a", "label": "dup"}, {"label": "no id"}, "str", {"id": "b"}]) - )), - rows(&[("a", "A"), ("b", "")]) - ); - assert_eq!(read_pick(Some(&json!({"col": ""}))), None); - assert_eq!( - read_pick(Some(&json!({"col": "Yes", "at": 5}))), - Some(Pick { - col: "Yes".into(), - by: String::new(), - by_id: String::new(), - at: 5.0 - }) - ); - } - - #[test] - fn cleans_columns() { - assert_eq!( - clean_columns(&json!([" Yes ", "No"])).unwrap(), - ["Yes", "No"] - ); - assert_eq!( - clean_columns(&json!(["x".repeat(60)])).unwrap()[0].len(), - 40 - ); - assert!(clean_columns(&json!([])).unwrap_err().contains("1 to 4")); - assert!(clean_columns(&json!(["a", "b", "c", "d", "e"])) - .unwrap_err() - .contains("1 to 4")); - assert_eq!( - clean_columns(&json!(["Yes", "yes"])).unwrap_err(), - "`columns` has \"yes\" twice" - ); - assert!(clean_columns(&json!(["Yes", ""])) - .unwrap_err() - .contains("non-empty")); - } - - #[test] - fn cleans_rows_keeping_ids() { - let existing = rows(&[("a", "Ann"), ("b", "Bob")]); - let out = clean_rows( - &json!(["Bob", {"label": "Ann", "id": "a"}, "Cy", {"text": "Dee"}]), - &existing, - ) - .unwrap(); - let labels: Vec<&str> = out.iter().map(|r| r.label.as_str()).collect(); - assert_eq!(labels, ["Bob", "Ann", "Cy", "Dee"]); - assert_eq!(out[0].id, "b"); - assert_eq!(out[1].id, "a"); - assert!(out.iter().all(|r| valid_row_id(&r.id))); - let mut ids: Vec<&str> = out.iter().map(|r| r.id.as_str()).collect(); - ids.dedup(); - assert_eq!(ids.len(), 4); - assert!(clean_rows(&json!("Ann"), &[]).unwrap_err().contains("list")); - let many: Vec = (0..61).map(|i| format!("p{i}")).collect(); - assert!(clean_rows(&json!(many), &[]).unwrap_err().contains("60")); - assert!(clean_rows(&json!([{"label": "x", "id": "bad id!"}]), &[]) - .unwrap_err() - .contains("id")); - assert!(clean_rows( - &json!([{"label": "x", "id": "a"}, {"label": "y", "id": "a"}]), - &[] - ) - .unwrap_err() - .contains("twice")); - assert!(clean_rows(&json!([{"label": 3}]), &[]) - .unwrap_err() - .contains("string")); - } - - #[test] - fn cleans_picks() { - let rs = rows(&[("a", "Ann"), ("b", "Bob"), ("c", "bob")]); - let cols = vec!["Yes".to_owned(), "No".to_owned()]; - let mut got = - clean_picks(&json!({"a": "yes", "Bob": "No", "c": null}), &rs, &cols).unwrap(); - got.sort(); - assert_eq!( - got, - [ - ("a".to_owned(), Some("Yes".to_owned())), - ("b".to_owned(), Some("No".to_owned())), - ("c".to_owned(), None) - ] - ); - assert_eq!( - clean_picks(&json!({"ann": true}), &rs, &["Done".to_owned()]).unwrap(), - [("a".to_owned(), Some("Done".to_owned()))] - ); - assert!(clean_picks(&json!({"BOB": "Yes"}), &rs, &cols) - .unwrap_err() - .contains("more than one")); - assert_eq!( - clean_picks(&json!({"Zed": "Yes"}), &rs, &cols).unwrap_err(), - "`picks`: no row \"Zed\"" - ); - assert_eq!( - clean_picks(&json!({"Ann": "Maybe"}), &rs, &cols).unwrap_err(), - "`picks`: no column \"Maybe\" (Yes, No)" - ); - assert!(clean_picks(&json!(["Ann"]), &rs, &cols) - .unwrap_err() - .contains("object")); - assert!(clean_picks(&json!({"Ann": 1}), &rs, &cols) - .unwrap_err() - .contains("values")); - } - - #[test] - fn sizes_like_the_page() { - assert!((checklist_width(2) - 324.0).abs() < f64::EPSILON); - assert!((checklist_width(1) - 280.0).abs() < f64::EPSILON); - assert!((checklist_height(2, 3) - (49.0 + 28.0 + 102.0 + 14.0)).abs() < f64::EPSILON); - assert!((checklist_height(1, 0) - 97.0).abs() < f64::EPSILON); - let id = new_row_id(); - assert!( - id.starts_with('r') && id.len() == 9 && valid_row_id(&id), - "{id}" - ); - } -} diff --git a/crates/canvas/src/geometry.rs b/crates/canvas/src/geometry.rs deleted file mode 100644 index 20d8b1d..0000000 --- a/crates/canvas/src/geometry.rs +++ /dev/null @@ -1,411 +0,0 @@ -//! Board geometry shared with the canvas page (`Canvas/src/canvas/geometry.ts`, -//! `arrows.ts`, `placement.ts`): boxes, side anchors, arrow paths and free-space placement. -//! Pure functions, no Yjs. - -/// A point in world coordinates. -#[derive(Debug, Clone, Copy, PartialEq)] -pub struct Point { - pub x: f64, - pub y: f64, -} - -/// An axis-aligned box (top-left + size) in world coordinates. -#[derive(Debug, Clone, Copy, PartialEq)] -pub struct BoxF { - pub x: f64, - pub y: f64, - pub w: f64, - pub h: f64, -} - -/// A side of a box an arrow end can be pinned to. -#[derive(Debug, Clone, Copy, PartialEq, Eq)] -pub enum Side { - Top, - Right, - Bottom, - Left, -} - -impl Side { - /// Parses `top|right|bottom|left`. - pub fn parse(s: &str) -> Option { - match s { - "top" => Some(Self::Top), - "right" => Some(Self::Right), - "bottom" => Some(Self::Bottom), - "left" => Some(Self::Left), - _ => None, - } - } - - pub fn as_str(self) -> &'static str { - match self { - Self::Top => "top", - Self::Right => "right", - Self::Bottom => "bottom", - Self::Left => "left", - } - } - - fn normal(self) -> Point { - match self { - Self::Top => Point { x: 0.0, y: -1.0 }, - Self::Right => Point { x: 1.0, y: 0.0 }, - Self::Bottom => Point { x: 0.0, y: 1.0 }, - Self::Left => Point { x: -1.0, y: 0.0 }, - } - } -} - -/// JavaScript `Math.round` (halves round toward +∞). -pub fn js_round(v: f64) -> f64 { - (v + 0.5).floor() -} - -pub fn center(b: &BoxF) -> Point { - Point { - x: b.x + b.w / 2.0, - y: b.y + b.h / 2.0, - } -} - -/// Where the ray from the box's centre toward `toward` leaves the box. -pub fn exit_point(b: &BoxF, toward: Point) -> Point { - let c = center(b); - let (dx, dy) = (toward.x - c.x, toward.y - c.y); - if dx == 0.0 && dy == 0.0 { - return c; - } - let tx = if dx == 0.0 { - f64::INFINITY - } else { - b.w / 2.0 / dx.abs() - }; - let ty = if dy == 0.0 { - f64::INFINITY - } else { - b.h / 2.0 / dy.abs() - }; - let t = tx.min(ty); - Point { - x: c.x + dx * t, - y: c.y + dy * t, - } -} - -/// The middle of one side of a box. -pub fn side_point(b: &BoxF, side: Side) -> Point { - match side { - Side::Top => Point { - x: b.x + b.w / 2.0, - y: b.y, - }, - Side::Right => Point { - x: b.x + b.w, - y: b.y + b.h / 2.0, - }, - Side::Bottom => Point { - x: b.x + b.w / 2.0, - y: b.y + b.h, - }, - Side::Left => Point { - x: b.x, - y: b.y + b.h / 2.0, - }, - } -} - -/// `inner` lies entirely inside `outer`. -pub fn contains_box(outer: &BoxF, inner: &BoxF) -> bool { - inner.x >= outer.x - && inner.y >= outer.y - && inner.x + inner.w <= outer.x + outer.w - && inner.y + inner.h <= outer.y + outer.h -} - -/// Two boxes closer than `gap` on both axes. -pub fn boxes_overlap(a: &BoxF, b: &BoxF, gap: f64) -> bool { - a.x < b.x + b.w + gap && b.x < a.x + a.w + gap && a.y < b.y + b.h + gap && b.y < a.y + a.h + gap -} - -/// Centre-to-centre segment between two boxes, clipped to both borders (`None` when they -/// overlap). -fn box_segment(a: &BoxF, b: &BoxF) -> Option<(Point, Point)> { - let (ca, cb) = (center(a), center(b)); - if (ca.x - cb.x).abs() < f64::midpoint(a.w, b.w) - && (ca.y - cb.y).abs() < f64::midpoint(a.h, b.h) - { - return None; - } - Some((exit_point(a, cb), exit_point(b, ca))) -} - -/// An arrow end resolved against the current boxes. -#[derive(Debug, Clone, Copy)] -pub enum End { - /// A free point. - Point(Point), - /// A shape's box, optionally pinned to a side. - Box(BoxF, Option), -} - -/// The drawn path of an arrow. -#[derive(Debug, Clone, Copy, PartialEq)] -pub struct ArrowPath { - pub p: Point, - pub q: Point, - /// Where the label sits. - pub mid: Point, - /// Bezier control points when an end is pinned to a side. - pub curve: Option<(Point, Point)>, -} - -fn cubic_at(p0: Point, c1: Point, c2: Point, p3: Point, t: f64) -> Point { - let u = 1.0 - t; - Point { - x: u * u * u * p0.x + 3.0 * u * u * t * c1.x + 3.0 * u * t * t * c2.x + t * t * t * p3.x, - y: u * u * u * p0.y + 3.0 * u * u * t * c1.y + 3.0 * u * t * t * c2.y + t * t * t * p3.y, - } -} - -/// The path of an arrow between two resolved ends; `None` when there is no line to draw. -#[allow(clippy::many_single_char_names)] -pub fn arrow_path(a: End, b: End) -> Option { - let point_of = |e: &End| match *e { - End::Point(p) => Some(p), - End::Box(bx, Some(side)) => Some(side_point(&bx, side)), - End::Box(_, None) => None, - }; - let (pa, pb) = (point_of(&a), point_of(&b)); - let (p, q) = match (pa, pb, a, b) { - (Some(p), Some(q), _, _) => (p, q), - (Some(p), None, _, End::Box(bb, _)) => (p, exit_point(&bb, p)), - (None, Some(q), End::Box(ba, _), _) => (exit_point(&ba, q), q), - (None, None, End::Box(ba, _), End::Box(bb, _)) => box_segment(&ba, &bb)?, - _ => return None, - }; - let len = (q.x - p.x).hypot(q.y - p.y); - if len < 1.0 { - return None; - } - let side_of = |e: &End| match *e { - End::Box(_, s) => s, - End::Point(_) => None, - }; - let (sa, sb) = (side_of(&a), side_of(&b)); - if sa.is_none() && sb.is_none() { - return Some(ArrowPath { - p, - q, - mid: Point { - x: f64::midpoint(p.x, q.x), - y: f64::midpoint(p.y, q.y), - }, - curve: None, - }); - } - let reach = (len / 2.2).clamp(24.0, 160.0); - let towards = |from: Point, to: Point| { - let d = (to.x - from.x).hypot(to.y - from.y); - let d = if d == 0.0 { 1.0 } else { d }; - Point { - x: (to.x - from.x) / d, - y: (to.y - from.y) / d, - } - }; - let na = sa.map_or_else(|| towards(p, q), Side::normal); - let nb = sb.map_or_else(|| towards(q, p), Side::normal); - let c1 = Point { - x: p.x + na.x * reach, - y: p.y + na.y * reach, - }; - let c2 = Point { - x: q.x + nb.x * reach, - y: q.y + nb.y * reach, - }; - Some(ArrowPath { - p, - q, - mid: cubic_at(p, c1, c2, q, 0.5), - curve: Some((c1, c2)), - }) -} - -/// Bounding box of an arrow's path (curves sampled, as the page does). -pub fn arrow_box(path: &ArrowPath) -> BoxF { - let mut xs = vec![path.p.x, path.q.x]; - let mut ys = vec![path.p.y, path.q.y]; - if let Some((c1, c2)) = path.curve { - for i in 1..8 { - let pt = cubic_at(path.p, c1, c2, path.q, f64::from(i) / 8.0); - xs.push(pt.x); - ys.push(pt.y); - } - } - let min = |v: &[f64]| v.iter().copied().fold(f64::INFINITY, f64::min); - let max = |v: &[f64]| v.iter().copied().fold(f64::NEG_INFINITY, f64::max); - let (x, y) = (min(&xs), min(&ys)); - BoxF { - x, - y, - w: max(&xs) - x, - h: max(&ys) - y, - } -} - -/// Clear space kept around an auto-placed shape. -pub const PLACEMENT_GAP: f64 = 24.0; -/// Grid step of the placement search. -pub const PLACEMENT_STEP: f64 = 32.0; -/// The search gives up this far from the centre (and uses the centre). -pub const PLACEMENT_MAX_RADIUS: f64 = 6000.0; - -/// Top-left of a `w`×`h` box near `around` that overlaps none of `taken` (with -/// [`PLACEMENT_GAP`]): the centred spot if free, else the nearest free spot ring by ring on -/// a [`PLACEMENT_STEP`] grid, ties going right, then down, then left, then up — the same -/// walk as the page's `findFreeSpot`. -pub fn find_free_spot(taken: &[BoxF], w: f64, h: f64, around: Point) -> Point { - let (w, h) = (w.max(1.0), h.max(1.0)); - let origin = Point { - x: js_round(around.x - w / 2.0), - y: js_round(around.y - h / 2.0), - }; - let free_in = |set: &[BoxF], p: Point| { - let b = BoxF { - x: p.x, - y: p.y, - w, - h, - }; - !set.iter().any(|t| boxes_overlap(&b, t, PLACEMENT_GAP)) - }; - if free_in(taken, origin) { - return origin; - } - let reach = PLACEMENT_MAX_RADIUS + w.max(h); - let near: Vec = taken - .iter() - .filter(|t| { - t.x < origin.x + w + reach - && t.x + t.w > origin.x - reach - && t.y < origin.y + h + reach - && t.y + t.h > origin.y - reach - }) - .copied() - .collect(); - #[allow(clippy::cast_possible_truncation)] - let rings = (PLACEMENT_MAX_RADIUS / PLACEMENT_STEP).ceil() as i32; - let mut cells: Vec<(Point, f64, f64)> = Vec::new(); - for k in 1..=rings { - cells.clear(); - for i in -k..=k { - for j in -k..=k { - if i.abs().max(j.abs()) != k { - continue; - } - let dx = f64::from(i) * PLACEMENT_STEP; - let dy = f64::from(j) * PLACEMENT_STEP; - let mut a = dy.atan2(dx); - if a < 0.0 { - a += std::f64::consts::TAU; - } - cells.push(( - Point { - x: origin.x + dx, - y: origin.y + dy, - }, - dx.hypot(dy), - a, - )); - } - } - cells.sort_by(|m, n| m.1.total_cmp(&n.1).then(m.2.total_cmp(&n.2))); - if let Some((p, _, _)) = cells.iter().find(|c| free_in(&near, c.0)) { - return *p; - } - } - origin -} - -#[cfg(test)] -#[allow(clippy::float_cmp)] -mod tests { - use super::*; - - const B: BoxF = BoxF { - x: 0.0, - y: 0.0, - w: 100.0, - h: 100.0, - }; - - #[test] - fn rounding_matches_js() { - assert_eq!(js_round(2.5), 3.0); - assert_eq!(js_round(-2.5), -2.0); - assert_eq!(js_round(-2.6), -3.0); - assert_eq!(js_round(1.2), 1.0); - } - - #[test] - fn straight_arrow_between_boxes() { - let other = BoxF { x: 300.0, ..B }; - let path = arrow_path(End::Box(B, None), End::Box(other, None)).unwrap(); - assert_eq!(path.p, Point { x: 100.0, y: 50.0 }); - assert_eq!(path.q, Point { x: 300.0, y: 50.0 }); - assert_eq!(path.mid, Point { x: 200.0, y: 50.0 }); - let bx = arrow_box(&path); - assert_eq!((bx.x, bx.w, bx.h), (100.0, 200.0, 0.0)); - // Overlapping boxes have no line. - assert!(arrow_path(End::Box(B, None), End::Box(BoxF { x: 50.0, ..B }, None)).is_none()); - } - - #[test] - fn side_pinned_arrow_curves() { - let other = BoxF { x: 300.0, ..B }; - let path = arrow_path( - End::Box(B, Some(Side::Bottom)), - End::Box(other, Some(Side::Top)), - ) - .unwrap(); - assert_eq!(path.p, Point { x: 50.0, y: 100.0 }); - assert_eq!(path.q, Point { x: 350.0, y: 0.0 }); - assert!(path.curve.is_some()); - let bx = arrow_box(&path); - assert!(bx.h > 100.0, "curve overshoots the chord"); - } - - #[test] - fn point_to_box() { - let path = arrow_path(End::Point(Point { x: -100.0, y: 50.0 }), End::Box(B, None)).unwrap(); - assert_eq!(path.q, Point { x: 0.0, y: 50.0 }); - } - - #[test] - fn placement() { - assert_eq!( - find_free_spot(&[], 200.0, 200.0, Point { x: 0.0, y: 0.0 }), - Point { - x: -100.0, - y: -100.0 - } - ); - let taken = [BoxF { - x: -100.0, - y: -100.0, - w: 200.0, - h: 200.0, - }]; - let p = find_free_spot(&taken, 200.0, 200.0, Point { x: 0.0, y: 0.0 }); - let b = BoxF { - x: p.x, - y: p.y, - w: 200.0, - h: 200.0, - }; - assert!(!boxes_overlap(&b, &taken[0], PLACEMENT_GAP)); - // Ties go right first. - assert!(p.x > 0.0 && p.y == -100.0, "{p:?}"); - } -} diff --git a/crates/canvas/src/lib.rs b/crates/canvas/src/lib.rs deleted file mode 100644 index 0aad20f..0000000 --- a/crates/canvas/src/lib.rs +++ /dev/null @@ -1,128 +0,0 @@ -//! Canvas: canvases REST API + Yjs (y-websocket) rooms for copper-cloud. -//! -//! * [`router`] — every canvas route, without the `/v1` prefix (the binary nests it). -//! * [`ensure_personal_canvas`] — the caller's Personal canvas (created on first use). -//! * [`ops`] / [`read`] — the spec §4 canvas ops and read format over a `yrs` document. -//! * [`rooms_metrics`] — live room / peer counts. -//! * [`admin`] — instance-admin operations (delete any canvas, disconnect a user). -//! * [`chat`] / [`mentions`] — the document's `chat` array as `/read` shows it, and chat -//! @mention notifications (0.6.0). -//! -//! See `docs/canvas.md` and `docs/canvas-protocol.md`. - -use axum::extract::DefaultBodyLimit; -use axum::routing::{any, delete, get, post}; -use axum::Router; -use copper_cloud_core::state::SharedState; - -pub mod admin; -pub mod chat; -pub mod checklist; -pub mod geometry; -pub mod mentions; -pub mod ops; -pub mod read; -mod rest; -mod room; -pub mod schema; -pub mod shape; -mod store; -mod ws; - -pub use rest::{ - ensure_personal_canvas, CanvasView, InviteView, MemberView, UserRef, - INVITE_NUDGE_INTERVAL_SECS, PERSONAL, -}; -pub use room::{ - close_all_rooms, evict_room_now, room_in_memory, rooms_metrics, RoomsMetrics, - BROADCAST_CAPACITY, COMPACT_EVERY, EVENT_THROTTLE, IDLE_EVICT, -}; -pub use ws::{close as close_codes, MAX_MESSAGE_BYTES, PING_EVERY, READ_TIMEOUT}; - -/// Largest accepted REST request body (except ops). -pub const MAX_BODY_BYTES: usize = 64 * 1024; -/// Largest accepted `POST /canvases/:id/ops` body: room for a 2 MiB `data:` image plus JSON. -pub const MAX_OPS_BODY_BYTES: usize = 4 << 20; - -/// All canvas routes (mount under `/v1`). `:id` may be a canvas UUID or `personal`. -/// -/// | Method | Path | -/// |---|---| -/// | GET, POST | `/canvases` | -/// | GET, PATCH, DELETE | `/canvases/{id}` | -/// | GET | `/canvases/{id}/members` | -/// | DELETE | `/canvases/{id}/members/{user_id}` | -/// | GET, POST | `/canvases/{id}/invites` | -/// | DELETE | `/canvases/{id}/invites/{invite_id}` | -/// | GET | `/invites` | -/// | POST | `/invites/{id}/accept`, `/invites/{id}/decline` | -/// | POST | `/canvases/{id}/links` | -/// | GET | `/canvases/{id}/links` | -/// | DELETE | `/canvases/{id}/links`, `/canvases/{id}/links/{link_id}` | -/// | GET, POST | `/canvas-links/{token}`, `/canvas-links/{token}/join` | -/// | GET (upgrade) | `/canvases/{id}/ws` (WebSocket; HTTP/2 CONNECT too) | -/// | GET | `/canvases/{id}/state`, `/canvases/{id}/read` | -/// | POST | `/canvases/{id}/ops` | -/// | POST | `/canvases/{id}/mentions` | -/// | GET | `/mentions` | -/// | POST | `/mentions/read` | -pub fn router() -> Router { - Router::new() - .route( - "/canvases", - get(rest::list_canvases).post(rest::create_canvas), - ) - .route("/people", get(rest::list_people)) - .route( - "/canvases/{id}", - get(rest::get_canvas) - .patch(rest::rename_canvas) - .delete(rest::delete_canvas), - ) - .route("/canvases/{id}/members", get(rest::list_members)) - .route( - "/canvases/{id}/members/{user_id}", - delete(rest::remove_member), - ) - .route( - "/canvases/{id}/invites", - get(rest::list_canvas_invites).post(rest::create_invite), - ) - .route( - "/canvases/{id}/invites/{invite_id}", - delete(rest::revoke_invite), - ) - .route("/invites", get(rest::my_invites)) - .route("/invites/{id}/accept", post(rest::accept_invite)) - .route("/invites/{id}/decline", post(rest::decline_invite)) - .route( - "/canvases/{id}/links", - get(rest::list_share_links) - .post(rest::create_share_link) - .delete(rest::revoke_all_share_links), - ) - .route( - "/canvases/{id}/links/{link_id}", - axum::routing::delete(rest::revoke_share_link), - ) - .route("/canvas-links/{token}", get(rest::preview_share_link)) - .route("/canvas-links/{token}/join", post(rest::join_share_link)) - // `any`: HTTP/1.1 GET upgrades and HTTP/2 extended CONNECT (RFC 8441) both work. - .route("/canvases/{id}/ws", any(rest::ws_upgrade)) - .route("/canvases/{id}/state", get(rest::get_state)) - .route("/canvases/{id}/read", get(rest::read_canvas)) - .route( - "/canvases/{id}/ops", - post(rest::post_ops).layer(DefaultBodyLimit::max(MAX_OPS_BODY_BYTES)), - ) - .route("/canvases/{id}/mentions", post(mentions::post_mentions)) - .route("/mentions", get(mentions::list_mentions)) - .route("/mentions/read", post(mentions::mark_read)) - .layer(DefaultBodyLimit::max(MAX_BODY_BYTES)) -} - -/// The public landing page for web-form share links. It is merged at the application root rather -/// than under `/v1`, so the instance-key gate does not run for it. -pub fn landing_router() -> Router { - Router::new().route("/join/{token}", get(rest::join_landing)) -} diff --git a/crates/canvas/src/mentions.rs b/crates/canvas/src/mentions.rs deleted file mode 100644 index 74ade26..0000000 --- a/crates/canvas/src/mentions.rs +++ /dev/null @@ -1,471 +0,0 @@ -//! Chat @mentions on shared canvases (0.6.0). -//! -//! The chat lives in the canvas document ([`crate::chat`]); the server only learns who was -//! mentioned when the sender's Copper reports it: -//! -//! * `POST /canvases/{id}/mentions {message_id, user_ids, excerpt}` — one row per recipient -//! that is a member of the canvas (the sender excluded), idempotent per -//! `(canvas, message_id, recipient)`; each new recipient gets a `canvas` event, kind -//! `mention`. -//! * `GET /mentions?unread=1&limit=n` — the caller's mentions on canvases they can still see, -//! newest first. -//! * `POST /mentions/read {ids}|{canvas_id}` — marks them read; the caller's other devices get -//! a `canvas` event, kind `mention_read`, per canvas touched. -//! -//! Excerpts are sealed under the canvas doc key (like the document itself), with AAD -//! [`EXCERPT_AAD_PREFIX`] ‖ the row id. - -use std::collections::{HashMap, HashSet}; -use std::num::NonZeroU32; -use std::sync::atomic::{AtomicU64, Ordering}; -use std::sync::LazyLock; - -use axum::extract::{Path, State}; -use axum::Json; -use copper_cloud_core::auth::AuthUser; -use copper_cloud_core::crypto::Crypto; -use copper_cloud_core::error::ApiError; -use copper_cloud_core::extract::QueryParams; -use copper_cloud_core::ids; -use copper_cloud_core::state::SharedState; -use governor::{DefaultKeyedRateLimiter, Quota, RateLimiter}; -use serde::{Deserialize, Serialize}; -use time::OffsetDateTime; -use uuid::Uuid; - -use crate::rest::{access, publish, Body, UserRef}; -use crate::store; - -/// Longest excerpt stored (characters); longer ones are cut. -pub const MAX_EXCERPT_CHARS: usize = 200; -/// Longest chat message id (characters). -pub const MAX_MESSAGE_ID_CHARS: usize = 128; -/// Most `user_ids` in one `POST /canvases/{id}/mentions`. -pub const MAX_MENTION_TARGETS: usize = 100; -/// `GET /mentions` returns at most this many (default [`DEFAULT_MENTIONS_LIMIT`]). -pub const MAX_MENTIONS_LIMIT: i64 = 100; -/// `GET /mentions` page size when `limit` is not given. -pub const DEFAULT_MENTIONS_LIMIT: i64 = 50; -/// Most `ids` in one `POST /mentions/read`. -pub const MAX_READ_IDS: usize = 500; -/// `POST /canvases/{id}/mentions` calls per user per minute (GCRA: bursts of this many, then -/// one every `60 / n` s). -pub const MENTION_POSTS_PER_MINUTE: u32 = 60; -/// AAD prefix of a sealed excerpt (followed by the mention row id's 16 raw bytes). -pub const EXCERPT_AAD_PREFIX: &[u8] = b"copper-cloud/v1/mention:"; - -/// SSE `canvas` event kind sent to each newly mentioned member. -pub const KIND_MENTION: &str = "mention"; -/// SSE `canvas` event kind sent to the caller after marking mentions read. -pub const KIND_MENTION_READ: &str = "mention_read"; - -// --------------------------------------------------------------------------------------------- -// Rate limit - -struct PostLimiter { - limiter: DefaultKeyedRateLimiter, - calls: AtomicU64, -} - -static POST_LIMITER: LazyLock = LazyLock::new(|| PostLimiter { - limiter: RateLimiter::keyed(Quota::per_minute( - NonZeroU32::new(MENTION_POSTS_PER_MINUTE).expect("non-zero"), - )), - calls: AtomicU64::new(0), -}); - -/// `true` if `user` may post mentions now (process-wide, per user). -fn allow_post(user: Uuid) -> bool { - let l = &*POST_LIMITER; - if l.calls.fetch_add(1, Ordering::Relaxed) % 4096 == 4095 { - l.limiter.retain_recent(); - l.limiter.shrink_to_fit(); - } - l.limiter.check_key(&user).is_ok() -} - -// --------------------------------------------------------------------------------------------- -// Helpers - -fn excerpt_aad(mention_id: Uuid) -> Vec { - let mut aad = Vec::with_capacity(EXCERPT_AAD_PREFIX.len() + 16); - aad.extend_from_slice(EXCERPT_AAD_PREFIX); - aad.extend_from_slice(mention_id.as_bytes()); - aad -} - -/// One line of at most [`MAX_EXCERPT_CHARS`]: control characters (newlines included) become -/// spaces, runs of whitespace collapse, ends are trimmed. -fn clean_excerpt(raw: &str) -> String { - let spaced: String = raw - .chars() - .map(|c| if c.is_control() { ' ' } else { c }) - .collect(); - spaced - .split_whitespace() - .collect::>() - .join(" ") - .chars() - .take(MAX_EXCERPT_CHARS) - .collect() -} - -fn message_id(raw: &str) -> Result { - let id = raw.trim(); - if id.is_empty() { - return Err(ApiError::bad_request("message_id must not be empty")); - } - if id.chars().count() > MAX_MESSAGE_ID_CHARS { - return Err(ApiError::bad_request(format!( - "message_id must be at most {MAX_MESSAGE_ID_CHARS} characters" - ))); - } - if id.chars().any(char::is_control) { - return Err(ApiError::bad_request( - "message_id must not contain control characters", - )); - } - Ok(id.to_owned()) -} - -// --------------------------------------------------------------------------------------------- -// POST /canvases/{id}/mentions - -#[derive(Deserialize)] -pub(crate) struct MentionBody { - message_id: String, - #[serde(default)] - user_ids: Vec, - #[serde(default)] - excerpt: String, -} - -/// `POST /canvases/{id}/mentions` response. -#[derive(Debug, Clone, Serialize)] -pub struct MentionsSent { - /// Members (other than the caller) who are now mentioned by this message — including on a - /// retry, when nobody is notified again. - pub notified: Vec, - /// Everyone else from `user_ids`: the caller, non-members, unknown ids. - pub skipped: Vec, -} - -pub(crate) async fn post_mentions( - State(state): State, - user: AuthUser, - Path(raw): Path, - Body(body): Body, -) -> Result, ApiError> { - let a = access(&state, user.user_id, &raw).await?; - if a.kind == "personal" { - return Err(ApiError::bad_request("the Personal canvas has no chat")); - } - let message_id = message_id(&body.message_id)?; - if body.user_ids.len() > MAX_MENTION_TARGETS { - return Err(ApiError::bad_request(format!( - "at most {MAX_MENTION_TARGETS} user_ids per message" - ))); - } - if !allow_post(user.user_id) { - metrics::counter!("canvas_mentions_rate_limited_total").increment(1); - tracing::warn!(user_id = %user.user_id, canvas_id = %a.canvas_id, "mention rate limit exceeded"); - return Err(ApiError::RateLimited); - } - let excerpt = clean_excerpt(&body.excerpt); - let mut seen = HashSet::new(); - let wanted: Vec = body - .user_ids - .into_iter() - .filter(|u| seen.insert(*u)) - .collect(); - if wanted.is_empty() { - return Ok(Json(MentionsSent { - notified: Vec::new(), - skipped: Vec::new(), - })); - } - - // Seal an excerpt for every candidate row up front (≤ 100, microseconds), so membership - // filtering, the idempotent insert and the result are one statement. - let key = store::doc_key(&state.db, &state.crypto, a.canvas_id).await?; - let row_ids: Vec = wanted.iter().map(|_| ids::uuid_v7()).collect(); - let sealed: Vec> = row_ids - .iter() - .map(|id| Crypto::seal(&key, &excerpt_aad(*id), excerpt.as_bytes())) - .collect(); - let rows: Vec<(Uuid, bool)> = sqlx::query_as( - "WITH t AS ( - SELECT * FROM UNNEST($4::uuid[], $5::uuid[], $6::bytea[]) AS t (id, to_user, excerpt) - ), ok AS ( - SELECT t.* FROM t - JOIN canvas_members m ON m.canvas_id = $1 AND m.user_id = t.to_user - WHERE t.to_user <> $3 - ), ins AS ( - INSERT INTO canvas_mentions (id, canvas_id, message_id, from_user, to_user, excerpt_sealed) - SELECT id, $1, $2, $3, to_user, excerpt FROM ok - ON CONFLICT (canvas_id, message_id, to_user) DO NOTHING - RETURNING to_user - ) - SELECT ok.to_user, (ins.to_user IS NOT NULL) AS fresh - FROM ok LEFT JOIN ins ON ins.to_user = ok.to_user", - ) - .bind(a.canvas_id) - .bind(&message_id) - .bind(user.user_id) - .bind(&row_ids) - .bind(&wanted) - .bind(&sealed) - .fetch_all(&state.db) - .await?; - - let mentioned: HashSet = rows.iter().map(|r| r.0).collect(); - let fresh: Vec = rows.iter().filter(|r| r.1).map(|r| r.0).collect(); - let (notified, skipped): (Vec, Vec) = - wanted.into_iter().partition(|u| mentioned.contains(u)); - publish(&state, &fresh, a.canvas_id, KIND_MENTION); - metrics::counter!("canvas_mentions_total").increment(fresh.len() as u64); - tracing::info!( - user_id = %user.user_id, - canvas_id = %a.canvas_id, - notified = fresh.len(), - mentioned = notified.len(), - skipped = skipped.len(), - "canvas chat mentions" - ); - Ok(Json(MentionsSent { notified, skipped })) -} - -// --------------------------------------------------------------------------------------------- -// GET /mentions - -#[derive(Debug, Default, Deserialize)] -pub(crate) struct MentionsQuery { - unread: Option, - limit: Option, -} - -/// `{id, name}` of the canvas a mention is on. -#[derive(Debug, Clone, Serialize)] -pub struct MentionCanvas { - pub id: Uuid, - pub name: String, -} - -/// One `GET /mentions` item. -#[derive(Debug, Clone, Serialize)] -pub struct MentionView { - pub id: Uuid, - pub canvas: MentionCanvas, - pub from: UserRef, - pub message_id: String, - pub excerpt: String, - #[serde(with = "time::serde::rfc3339")] - pub created_at: OffsetDateTime, - #[serde(with = "time::serde::rfc3339::option")] - pub read_at: Option, -} - -/// `GET /mentions` response. -#[derive(Debug, Clone, Serialize)] -pub struct MentionList { - pub mentions: Vec, - /// Every unread mention of the caller on canvases they can see (not capped by `limit`). - pub unread: i64, -} - -#[derive(sqlx::FromRow)] -struct MentionRow { - id: Uuid, - canvas_id: Uuid, - canvas_name: String, - doc_key_wrapped: Vec, - from_id: Uuid, - from_email: String, - from_display_name: String, - message_id: String, - excerpt_sealed: Vec, - created_at: OffsetDateTime, - read_at: Option, -} - -fn flag(raw: Option<&str>) -> Result { - match raw.map(str::trim) { - None | Some("" | "0" | "false" | "no") => Ok(false), - Some("1" | "true" | "yes") => Ok(true), - Some(_) => Err(ApiError::bad_request("unread must be 1 or 0")), - } -} - -pub(crate) async fn list_mentions( - State(state): State, - user: AuthUser, - QueryParams(q): QueryParams, -) -> Result, ApiError> { - let unread_only = flag(q.unread.as_deref())?; - let limit = q.limit.unwrap_or(DEFAULT_MENTIONS_LIMIT); - if limit < 1 { - return Err(ApiError::bad_request("limit must be at least 1")); - } - let limit = limit.min(MAX_MENTIONS_LIMIT); - // Only canvases the caller is still a member of. - let rows = sqlx::query_as::<_, MentionRow>( - "SELECT mt.id, mt.canvas_id, c.name AS canvas_name, c.doc_key_wrapped, - u.id AS from_id, u.email AS from_email, u.display_name AS from_display_name, - mt.message_id, mt.excerpt_sealed, mt.created_at, mt.read_at - FROM canvas_mentions mt - JOIN canvas_members m ON m.canvas_id = mt.canvas_id AND m.user_id = mt.to_user - JOIN canvases c ON c.id = mt.canvas_id - JOIN users u ON u.id = mt.from_user - WHERE mt.to_user = $1 AND (NOT $2 OR mt.read_at IS NULL) - ORDER BY mt.created_at DESC, mt.id DESC - LIMIT $3", - ) - .bind(user.user_id) - .bind(unread_only) - .bind(limit) - .fetch_all(&state.db) - .await?; - let unread: i64 = sqlx::query_scalar( - "SELECT count(*) FROM canvas_mentions mt - JOIN canvas_members m ON m.canvas_id = mt.canvas_id AND m.user_id = mt.to_user - WHERE mt.to_user = $1 AND mt.read_at IS NULL", - ) - .bind(user.user_id) - .fetch_one(&state.db) - .await?; - - let mut keys: HashMap> = HashMap::new(); - let mut mentions = Vec::with_capacity(rows.len()); - for r in rows { - let key = *keys.entry(r.canvas_id).or_insert_with(|| { - state - .crypto - .unwrap_key(&r.doc_key_wrapped) - .inspect_err(|e| tracing::warn!(canvas_id = %r.canvas_id, error = %e, "unwrapping canvas key for mentions failed")) - .ok() - }); - let opened = key.and_then(|k| Crypto::open(&k, &excerpt_aad(r.id), &r.excerpt_sealed).ok()); - let excerpt = if let Some(bytes) = opened { - String::from_utf8_lossy(&bytes).into_owned() - } else { - tracing::warn!(mention_id = %r.id, "mention excerpt does not decrypt; served empty"); - String::new() - }; - mentions.push(MentionView { - id: r.id, - canvas: MentionCanvas { - id: r.canvas_id, - name: r.canvas_name, - }, - from: UserRef { - id: r.from_id, - email: r.from_email, - display_name: r.from_display_name, - }, - message_id: r.message_id, - excerpt, - created_at: r.created_at, - read_at: r.read_at, - }); - } - Ok(Json(MentionList { mentions, unread })) -} - -// --------------------------------------------------------------------------------------------- -// POST /mentions/read - -#[derive(Deserialize)] -pub(crate) struct ReadBody { - #[serde(default)] - ids: Option>, - #[serde(default)] - canvas_id: Option, -} - -/// `POST /mentions/read` response. -#[derive(Debug, Clone, Serialize)] -pub struct MentionsRead { - pub ok: bool, - /// Mentions that went from unread to read. - pub updated: usize, -} - -pub(crate) async fn mark_read( - State(state): State, - user: AuthUser, - Body(body): Body, -) -> Result, ApiError> { - if body.ids.is_none() && body.canvas_id.is_none() { - return Err(ApiError::bad_request("send ids or canvas_id")); - } - let ids = body.ids.unwrap_or_default(); - if ids.len() > MAX_READ_IDS { - return Err(ApiError::bad_request(format!( - "at most {MAX_READ_IDS} ids per call" - ))); - } - let touched: Vec = sqlx::query_scalar( - "UPDATE canvas_mentions mt SET read_at = now() - WHERE mt.to_user = $1 AND mt.read_at IS NULL - AND (mt.id = ANY($2) OR mt.canvas_id = $3) - AND EXISTS (SELECT 1 FROM canvas_members m - WHERE m.canvas_id = mt.canvas_id AND m.user_id = $1) - RETURNING mt.canvas_id", - ) - .bind(user.user_id) - .bind(&ids) - .bind(body.canvas_id) - .fetch_all(&state.db) - .await?; - let canvases: HashSet = touched.iter().copied().collect(); - for canvas_id in canvases { - publish(&state, &[user.user_id], canvas_id, KIND_MENTION_READ); - } - Ok(Json(MentionsRead { - ok: true, - updated: touched.len(), - })) -} - -#[cfg(test)] -mod tests { - use super::*; - - #[test] - fn excerpts_are_one_trimmed_line() { - assert_eq!( - clean_excerpt(" hi\n\n@Ann see\tthis \u{7}"), - "hi @Ann see this" - ); - assert_eq!(clean_excerpt(&"é".repeat(300)).chars().count(), 200); - assert_eq!(clean_excerpt(""), ""); - } - - #[test] - fn message_ids() { - assert_eq!(message_id(" m1 ").unwrap(), "m1"); - assert!(message_id(" ").is_err()); - assert!(message_id(&"x".repeat(129)).is_err()); - assert!(message_id("a\u{0}b").is_err()); - } - - #[test] - fn unread_flag() { - assert!(!flag(None).unwrap()); - assert!(flag(Some("1")).unwrap() && flag(Some("true")).unwrap()); - assert!(!flag(Some("0")).unwrap()); - assert!(flag(Some("maybe")).is_err()); - } - - #[test] - fn aad_binds_the_row() { - let key = Crypto::new_key(); - let (a, b) = (Uuid::now_v7(), Uuid::now_v7()); - let sealed = Crypto::seal(&key, &excerpt_aad(a), b"hello"); - assert_eq!( - Crypto::open(&key, &excerpt_aad(a), &sealed).unwrap(), - b"hello" - ); - assert!(Crypto::open(&key, &excerpt_aad(b), &sealed).is_err()); - } -} diff --git a/crates/canvas/src/ops.rs b/crates/canvas/src/ops.rs deleted file mode 100644 index 6b0f6c0..0000000 --- a/crates/canvas/src/ops.rs +++ /dev/null @@ -1,1372 +0,0 @@ -//! Canvas ops (spec §4) with the same semantics as the canvas page's `Canvas/src/ops.ts` -//! (`copperCanvas.apply`), applied to a `yrs` document inside one transaction: -//! -//! ```text -//! {op:"add", shape:{type, id?, x?, y?, w?, h?, color?, …}} // no x/y → free space near `near` -//! {op:"update", id, patch:{…}} // merges props (`props` also accepted) -//! {op:"move", id, dx, dy} // a frame carries what sits inside it -//! {op:"resize", id, w, h} -//! {op:"delete", id} // arrows ending on it go too -//! {op:"connect", from:id, to:id, label?, color?, fromSide?, toSide?, id?} -//! {op:"clear", confirm:true} // server: needs confirmation -//! ``` -//! -//! A checklist (`crate::checklist`) takes `title`, `columns` (1–4, default Yes/No), `rows` -//! (labels or `{label, id?}`, ≤ 60) and `picks` (`{rowIdOrLabel: column | true | null}`) -//! on add and update; `rows` and `columns` replace the lists. -//! -//! A failing op changes nothing and is reported in `errors`; the rest still apply. -//! `ids[i]` is the id op `i` created or touched (`null` when it failed, and for `clear`). -//! -//! [`apply_ops`] is a pure function over a [`TransactionMut`] and the `shapes` root map. - -use serde::Serialize; -use serde_json::{json, Map as JsonMap, Value}; -use yrs::{Any, GetString, Map, MapPrelim, MapRef, Out, Text, TextPrelim, TextRef, TransactionMut}; - -use crate::checklist::{ - checklist_height, checklist_width, clean_columns, clean_picks, clean_rows, write_checklist, - ChecklistInput, Picker, Row, DEFAULT_COLUMNS, -}; -use crate::geometry::{center, contains_box, find_free_spot, js_round, BoxF, Point, Side}; -use crate::schema::{ - color_for, is_valid_color, jnum, js_len, js_slice, json_f64, json_to_any, ShapeType, - AGENT_STATUSES, MAX_DATA_URL, MAX_TEXT, MAX_TITLE, -}; -use crate::shape::{ - all_shapes, arrow_mid, bare_id, box_lookup, live_shape, max_z, store_value, Endpoint, ImageRef, - Shape, -}; - -/// Most ops accepted in one call (as on the page). -pub const MAX_OPS: usize = 500; -/// How long an agent shows `writing` after its last op. -pub const WRITING_MS: u64 = 1500; -const MAX_FAVICON: usize = 256 * 1024; -/// Accepted and dropped silently: the server owns these. -const IGNORED: [&str; 5] = ["id", "type", "by", "createdAt", "updatedAt"]; - -/// One canvas operation (parsed tolerantly, like the page). -#[derive(Debug, Clone, PartialEq)] -pub enum Op { - Add { - shape: JsonMap, - }, - Update { - id: String, - patch: JsonMap, - }, - Move { - id: String, - dx: f64, - dy: f64, - }, - Resize { - id: String, - w: Option, - h: Option, - }, - Delete { - id: String, - }, - Connect { - from: Value, - to: Value, - from_side: Option, - to_side: Option, - label: Option, - color: Option, - id: Option, - }, - Clear { - confirm: bool, - }, -} - -fn shape_id_of(o: &JsonMap) -> Result { - match o.get("id") { - Some(Value::String(s)) if !s.is_empty() => Ok(bare_id(s).to_owned()), - _ => Err("`id` must be a shape id".into()), - } -} - -/// `v ?? 0` as a finite number. -fn num_or_zero(v: Option<&Value>, err: &str) -> Result { - match v { - None | Some(Value::Null) => Ok(0.0), - Some(v) => json_f64(v).ok_or_else(|| err.to_owned()), - } -} - -impl Op { - /// Parses one JSON op; the error text matches the page's. - pub fn parse(v: &Value) -> Result { - let o = v - .as_object() - .ok_or("an op must be an object like {op:\"add\", …}")?; - match o.get("op").and_then(Value::as_str) { - Some("add") => Ok(Op::Add { - shape: o - .get("shape") - .and_then(Value::as_object) - .cloned() - .ok_or("`add` needs `shape:{type, …}`")?, - }), - Some("update") => { - let id = shape_id_of(o)?; - let patch = o - .get("patch") - .and_then(Value::as_object) - .or_else(|| o.get("props").and_then(Value::as_object)) - .cloned() - .ok_or("`update` needs `patch:{…}`")?; - Ok(Op::Update { id, patch }) - } - Some("move") => { - let err = "`dx` and `dy` must be finite numbers"; - Ok(Op::Move { - id: shape_id_of(o)?, - dx: num_or_zero(o.get("dx"), err)?, - dy: num_or_zero(o.get("dy"), err)?, - }) - } - Some("resize") => { - let err = || "`w` and `h` must be positive numbers".to_owned(); - let dim = |k: &str| match o.get(k) { - None | Some(Value::Null) => Ok(None), - Some(v) => json_f64(v).map(Some).ok_or_else(err), - }; - Ok(Op::Resize { - id: shape_id_of(o)?, - w: dim("w")?, - h: dim("h")?, - }) - } - Some("delete") => Ok(Op::Delete { - id: shape_id_of(o)?, - }), - Some("connect") => Ok(Op::Connect { - from: o.get("from").cloned().unwrap_or(Value::Null), - to: o.get("to").cloned().unwrap_or(Value::Null), - from_side: o.get("fromSide").cloned(), - to_side: o.get("toSide").cloned(), - label: o.get("label").cloned(), - color: o.get("color").cloned(), - id: o.get("id").cloned(), - }), - Some("clear") => Ok(Op::Clear { - confirm: o.get("confirm") == Some(&Value::Bool(true)), - }), - _ => Err(format!( - "unknown op {} (add, update, move, resize, delete, connect, clear)", - o.get("op").unwrap_or(&Value::Null) - )), - } - } - - /// The op's name as it appears on the wire. - pub fn name(&self) -> &'static str { - match self { - Op::Add { .. } => "add", - Op::Update { .. } => "update", - Op::Move { .. } => "move", - Op::Resize { .. } => "resize", - Op::Delete { .. } => "delete", - Op::Connect { .. } => "connect", - Op::Clear { .. } => "clear", - } - } -} - -/// Context for one batch of ops. -#[derive(Debug, Clone)] -pub struct OpsCtx { - /// Written to `by` on every shape the batch creates. - pub by: String, - /// The writer's id, on the checklist picks the batch makes (`byId`). - pub by_id: String, - /// Written to `createdAt`/`updatedAt` (ms since the Unix epoch). - pub now_ms: i64, - /// `confirm: true` was given at request level, which unlocks `clear`. - pub confirm_clear: bool, - /// Where auto-placement looks for free space (the viewport centre on the page). - pub origin: Point, -} - -impl OpsCtx { - /// A context stamped with the current time, origin `(0,0)` and `clear` locked. - pub fn new(by: impl Into) -> Self { - Self { - by: by.into(), - by_id: "agent".into(), - now_ms: crate::schema::now_ms(), - confirm_clear: false, - origin: Point { x: 0.0, y: 0.0 }, - } - } -} - -/// One failed op (`index` is `-1` for request-level errors). -#[derive(Debug, Clone, PartialEq, Serialize)] -pub struct OpError { - pub index: i64, - pub op: String, - pub error: String, -} - -/// Result of a batch: `{applied, ids, errors}`. -#[derive(Debug, Clone, Default, PartialEq, Serialize)] -pub struct OpsResult { - /// Number of ops that succeeded. - pub applied: usize, - /// `ids[i]`: the id op `i` created or touched; `null` when it failed, and for `clear`. - pub ids: Vec>, - pub errors: Vec, - /// Where an agent's cursor should rest: the last thing it touched. - #[serde(skip)] - pub anchor: Option, -} - -impl OpsResult { - /// A result carrying one request-level error. - pub fn request_error(error: impl Into) -> Self { - Self { - errors: vec![OpError { - index: -1, - op: String::new(), - error: error.into(), - }], - ..Self::default() - } - } -} - -/// Applies a batch of parsed ops (see the module docs). -pub fn apply_ops(txn: &mut TransactionMut, shapes: &MapRef, ops: &[Op], ctx: &OpsCtx) -> OpsResult { - let mut res = OpsResult::default(); - for (index, op) in ops.iter().enumerate() { - let out = apply_one(txn, shapes, op, ctx); - record(&mut res, index, op.name(), out); - } - res -} - -/// Parses and applies raw JSON ops; parse failures are reported at their index. -pub fn apply_json_ops( - txn: &mut TransactionMut, - shapes: &MapRef, - ops: &[Value], - ctx: &OpsCtx, -) -> OpsResult { - if ops.len() > MAX_OPS { - return OpsResult::request_error(format!( - "at most {MAX_OPS} ops per call (got {})", - ops.len() - )); - } - let mut res = OpsResult::default(); - for (index, raw) in ops.iter().enumerate() { - let name = raw.get("op").and_then(Value::as_str).unwrap_or(""); - let out = Op::parse(raw).and_then(|op| apply_one(txn, shapes, &op, ctx)); - record(&mut res, index, js_slice(name, 32), out); - } - res -} - -type OneResult = Result<(Option, Option), String>; - -fn record(res: &mut OpsResult, index: usize, name: &str, out: OneResult) { - match out { - Ok((id, anchor)) => { - res.applied += 1; - res.ids.push(id); - if anchor.is_some() { - res.anchor = anchor; - } - } - Err(error) => { - res.ids.push(None); - res.errors.push(OpError { - index: i64::try_from(index).unwrap_or(i64::MAX), - op: name.to_owned(), - error, - }); - } - } -} - -fn apply_one(txn: &mut TransactionMut, shapes: &MapRef, op: &Op, ctx: &OpsCtx) -> OneResult { - match op { - Op::Add { shape } => { - let (id, at) = op_add(txn, shapes, shape, ctx)?; - Ok((Some(id), Some(at))) - } - Op::Update { id, patch } => { - let id = op_update(txn, shapes, id, patch, ctx)?; - Ok((Some(id.clone()), Some(anchor_of(txn, shapes, &id, ctx)))) - } - Op::Move { id, dx, dy } => { - let id = op_move(txn, shapes, id, *dx, *dy, ctx)?; - Ok((Some(id.clone()), Some(anchor_of(txn, shapes, &id, ctx)))) - } - Op::Resize { id, w, h } => { - let id = op_resize(txn, shapes, id, *w, *h, ctx)?; - Ok((Some(id.clone()), Some(anchor_of(txn, shapes, &id, ctx)))) - } - Op::Delete { id } => { - let at = anchor_of(txn, shapes, id, ctx); - let id = op_delete(txn, shapes, id)?; - Ok((Some(id), Some(at))) - } - Op::Connect { .. } => { - let (id, at) = op_connect(txn, shapes, op, ctx)?; - Ok((Some(id), Some(at))) - } - Op::Clear { confirm } => { - if !(*confirm || ctx.confirm_clear) { - return Err("clear needs confirmation: send confirm: true".into()); - } - shapes.clear(txn); - Ok((None, None)) - } - } -} - -// ---- validation ------------------------------------------------------------------------- - -/// `^[A-Za-z0-9_:.-]{1,64}$` -fn valid_shape_id(s: &str) -> bool { - (1..=64).contains(&s.len()) - && s.bytes() - .all(|b| b.is_ascii_alphanumeric() || matches!(b, b'_' | b':' | b'.' | b'-')) -} - -fn starts_with_ci(s: &str, prefix: &str) -> bool { - s.len() >= prefix.len() && s.as_bytes()[..prefix.len()].eq_ignore_ascii_case(prefix.as_bytes()) -} - -/// A `data:image/…` (≤ 2 MiB) or `https:` URL. -pub fn valid_image_src(v: &Value) -> Result { - let err = || "`src` must be a data: or https: URL".to_owned(); - let src = v.as_str().filter(|s| !s.is_empty()).ok_or_else(err)?; - if src.starts_with("data:") { - // ^data:image\/[a-z0-9.+-]+[;,] - let ok = starts_with_ci(src, "data:image/") - && src["data:image/".len()..] - .bytes() - .position(|b| b == b';' || b == b',') - .is_some_and(|end| { - end > 0 - && src.as_bytes()["data:image/".len().."data:image/".len() + end] - .iter() - .all(|b| b.is_ascii_alphanumeric() || matches!(b, b'.' | b'+' | b'-')) - }); - if !ok { - return Err("`src` data URL must be an image".into()); - } - if src.len() > MAX_DATA_URL { - return Err("image data URL is over 2 MB".into()); - } - return Ok(src.to_owned()); - } - if starts_with_ci(src, "https://") - && src.len() > "https://".len() - && !src.chars().any(char::is_whitespace) - { - return Ok(src.to_owned()); - } - Err(err()) -} - -fn has_scheme(s: &str) -> bool { - // ^[a-z][a-z0-9+.-]*: - let mut chars = s.chars(); - if !chars.next().is_some_and(|c| c.is_ascii_alphabetic()) { - return false; - } - for c in chars { - if c == ':' { - return true; - } - if !(c.is_ascii_alphanumeric() || matches!(c, '+' | '.' | '-')) { - return false; - } - } - false -} - -/// An `http(s)`, `mailto` or `copper` URL, normalised (bare hosts get `https://`). -pub fn valid_url(v: &Value) -> Result { - let raw = v - .as_str() - .filter(|s| !s.trim().is_empty()) - .ok_or("`url` must be a non-empty string")?; - let t = raw.trim(); - let candidate = if has_scheme(t) { - t.to_owned() - } else { - format!("https://{t}") - }; - let parsed = url::Url::parse(&candidate) - .map_err(|_| format!("`url` is not a URL: {}", js_slice(t, 80)))?; - if !matches!(parsed.scheme(), "http" | "https" | "mailto" | "copper") { - return Err(format!("unsupported link scheme {}:", parsed.scheme())); - } - Ok(parsed.to_string()) -} - -/// Where a URL points, for a default link title. -pub fn host_of(u: &str) -> String { - url::Url::parse(u) - .ok() - .and_then(|p| { - p.host_str() - .map(|h| h.strip_prefix("www.").unwrap_or(h).to_owned()) - }) - .filter(|h| !h.is_empty()) - .unwrap_or_else(|| u.to_owned()) -} - -fn check_end( - txn: &TransactionMut, - shapes: &MapRef, - end: &Endpoint, - key: &str, - self_id: Option<&str>, -) -> Result<(), String> { - let Some(r) = end.ref_id() else { return Ok(()) }; - if Some(r) == self_id { - return Err(format!("`{key}` cannot be the arrow itself")); - } - match live_shape(txn, shapes, r) { - None => Err(format!("`{key}`: no shape {r}")), - Some(t) if t.is_arrow() => Err(format!("`{key}`: {r} is an arrow; arrows join boxes")), - Some(_) => Ok(()), - } -} - -/// The checked, normalised props a caller gave for a shape of `kind` (the page's -/// `cleanProps`). `image: null` survives as `Value::Null` (= remove). -fn clean_props( - kind: ShapeType, - raw: &JsonMap, - txn: &TransactionMut, - shapes: &MapRef, - self_id: Option<&str>, -) -> Result, String> { - let allowed = kind.allowed(); - let mut out = JsonMap::new(); - for (raw_key, value) in raw { - let key = kind.alias(raw_key); - if IGNORED.contains(&key) { - continue; - } - if !allowed.contains(&key) { - if kind == ShapeType::Arrow && matches!(key, "x" | "y" | "w" | "h") { - return Err( - "arrows have no box of their own: set `from`/`to` (or move the shapes they join)" - .into(), - ); - } - return Err(format!( - "unknown prop `{raw_key}` for {} (allowed: {})", - kind.as_str(), - allowed.join(", ") - )); - } - let cleaned = match key { - "x" | "y" | "z" => { - json_f64(value).ok_or_else(|| format!("`{key}` must be a finite number"))?; - value.clone() - } - "w" | "h" => { - json_f64(value) - .filter(|n| *n > 0.0) - .ok_or_else(|| format!("`{key}` must be a positive number"))?; - value.clone() - } - "naturalW" | "naturalH" => { - json_f64(value) - .filter(|n| *n >= 0.0) - .ok_or_else(|| format!("`{key}` must be a non-negative number"))?; - value.clone() - } - "color" => match value.as_str() { - Some(c) if is_valid_color(c) => value.clone(), - _ => { - return Err( - "`color` must be yellow, pink, blue, green, purple, gray, white or a #hex colour" - .into(), - ) - } - }, - "text" => { - let t = value.as_str().ok_or("`text` must be a string")?; - if js_len(t) > MAX_TEXT { - return Err(format!("`text` is over {MAX_TEXT} characters")); - } - value.clone() - } - "title" | "label" => { - let t = value - .as_str() - .ok_or_else(|| format!("`{key}` must be a string"))?; - Value::String(js_slice(t, MAX_TITLE).to_owned()) - } - "fontSize" => { - json_f64(value) - .filter(|n| (6.0..=240.0).contains(n)) - .ok_or("`fontSize` must be between 6 and 240")?; - value.clone() - } - "align" => match value.as_str() { - Some("left" | "center" | "right") => value.clone(), - _ => return Err("`align` must be left, center or right".into()), - }, - "image" => clean_image(value)?, - "src" => Value::String(valid_image_src(value)?), - "url" => Value::String(valid_url(value)?), - "favicon" => { - let f = value - .as_str() - .ok_or("`favicon` must be a data: or https: URL")?; - if !f.is_empty() { - if !(starts_with_ci(f, "data:image/") || starts_with_ci(f, "https://")) { - return Err("`favicon` must be a data: or https: URL".into()); - } - if f.len() > MAX_FAVICON { - return Err("`favicon` is over 256 KB".into()); - } - } - value.clone() - } - "columns" | "rows" => clean_checklist_list(key, value, txn, shapes, self_id)?, - "from" | "to" => { - let end = Endpoint::read(value).ok_or_else(|| { - format!("`{key}` must be a shape id, {{ref, side?}} or {{x, y}}") - })?; - check_end(txn, shapes, &end, key, self_id)?; - end.to_json() - } - _ => value.clone(), - }; - out.insert(key.to_owned(), cleaned); - } - Ok(out) -} - -/// A checklist's `columns` or `rows`, cleaned (rows keep the ids of `self_id`'s rows). -fn clean_checklist_list( - key: &str, - value: &Value, - txn: &TransactionMut, - shapes: &MapRef, - self_id: Option<&str>, -) -> Result { - if key == "columns" { - return Ok(json!(clean_columns(value)?)); - } - let existing = self_id - .and_then(|id| live_shape(txn, shapes, id)) - .and_then(|s| s.checklist) - .map(|c| c.rows) - .unwrap_or_default(); - Ok(Value::Array( - clean_rows(value, &existing)? - .iter() - .map(Row::to_json) - .collect(), - )) -} - -fn clean_image(value: &Value) -> Result { - match value { - Value::Null => Ok(Value::Null), - Value::String(s) if s.is_empty() => Ok(Value::Null), - Value::String(_) => Ok(ImageRef { - src: valid_image_src(value)?, - natural_w: 0.0, - natural_h: 0.0, - } - .to_json()), - Value::Object(o) => Ok(ImageRef { - src: valid_image_src(o.get("src").unwrap_or(&Value::Null))?, - natural_w: o.get("naturalW").and_then(json_f64).unwrap_or(0.0), - natural_h: o.get("naturalH").and_then(json_f64).unwrap_or(0.0), - } - .to_json()), - _ => Err("`image` must be {src, naturalW, naturalH} or null".into()), - } -} - -fn num(props: &JsonMap, k: &str) -> Option { - props.get(k).and_then(json_f64) -} - -/// An image shape's starting size: natural size, at most 480 wide. -fn image_size_for(width: f64, height: f64) -> (f64, f64) { - let nw = if width > 0.0 { width } else { 320.0 }; - let nh = if height > 0.0 { height } else { 240.0 }; - let w = js_round(nw.min(480.0)); - (w.max(16.0), js_round(w * nh / nw).max(16.0)) -} - -/// A frame sized for an image: 160–640 wide. -fn frame_size_for(width: f64, height: f64) -> (f64, f64) { - let w = js_round(width.clamp(160.0, 640.0)); - let body = if height > 0.0 && width > 0.0 { - w * height / width - } else { - w * 0.75 - }; - (w, js_round(body).max(120.0)) -} - -fn clamp_min(kind: ShapeType, props: &mut JsonMap) { - if let Some((mw, mh)) = kind.min_size() { - if let Some(w) = num(props, "w") { - props.insert("w".into(), w.max(mw).into()); - } - if let Some(h) = num(props, "h") { - props.insert("h".into(), h.max(mh).into()); - } - } -} - -// ---- writes ------------------------------------------------------------------------------- - -fn shape_map(txn: &TransactionMut, shapes: &MapRef, id: &str) -> Option { - match shapes.get(txn, id) { - Some(Out::YMap(m)) => Some(m), - _ => None, - } -} - -/// Byte offsets of the smallest single splice turning `prev` into `next`. -fn text_splice<'a>(prev: &str, next: &'a str) -> (usize, usize, &'a str) { - let start = prev - .char_indices() - .zip(next.chars()) - .find(|((_, a), b)| a != b) - .map_or_else(|| prev.len().min(next.len()), |((i, _), _)| i); - // The common prefix ends on a char boundary of both strings. - let start = if prev.is_char_boundary(start) && next.is_char_boundary(start) { - start - } else { - 0 - }; - let (pt, nt) = (&prev[start..], &next[start..]); - let suffix: usize = pt - .chars() - .rev() - .zip(nt.chars().rev()) - .take_while(|(a, b)| a == b) - .map(|(a, _)| a.len_utf8()) - .sum(); - let suffix = suffix.min(pt.len()).min(nt.len()); - (start, pt.len() - suffix, &nt[..nt.len() - suffix]) -} - -fn splice_into(txn: &mut TransactionMut, t: &TextRef, next: &str) { - let prev = t.get_string(txn); - let (index, remove, insert) = text_splice(&prev, next); - let index = u32::try_from(index).unwrap_or(u32::MAX); - if remove > 0 { - t.remove_range(txn, index, u32::try_from(remove).unwrap_or(u32::MAX)); - } - if !insert.is_empty() { - t.insert(txn, index, insert); - } -} - -/// Sets a shape's body text: a minimal splice into its `Y.Text` (created from a plain -/// string if needed) for sticky/text, a plain string otherwise. -fn set_text(txn: &mut TransactionMut, m: &MapRef, kind: ShapeType, next: &str) { - match m.get(txn, "text") { - Some(Out::YText(t)) => splice_into(txn, &t, next), - other => { - if !kind.has_body() { - m.insert(txn, "text", next); - return; - } - let current = match other { - Some(Out::Any(Any::String(s))) => s.to_string(), - _ => String::new(), - }; - let t = m.insert(txn, "text", TextPrelim::new(current)); - splice_into(txn, &t, next); - } - } -} - -/// Writes `props` onto an existing shape (the page's `CanvasStore.update`). -fn store_update( - txn: &mut TransactionMut, - m: &MapRef, - kind: ShapeType, - props: &JsonMap, - ctx: &OpsCtx, -) { - for (k, v) in props { - if k == "text" { - if let Some(t) = v.as_str() { - set_text(txn, m, kind, t); - continue; - } - } - if k == "image" && v.is_null() { - m.remove(txn, "image"); - continue; - } - if let Some(any) = store_value(k, v) { - m.try_update(txn, k.as_str(), any); - } - } - m.try_update(txn, "updatedAt", Any::from(ctx.now_ms)); -} - -/// Creates a shape (the page's `CanvasStore.create`). -fn create( - txn: &mut TransactionMut, - shapes: &MapRef, - kind: ShapeType, - id: &str, - props: &JsonMap, - ctx: &OpsCtx, -) { - let existing = all_shapes(txn, shapes); - let mut full = JsonMap::new(); - let (dw, dh) = kind.default_size(); - for (k, v) in [("x", 0.0), ("y", 0.0), ("w", dw), ("h", dh)] { - full.insert(k.into(), v.into()); - } - full.insert("color".into(), kind.default_color().into()); - full.insert("z".into(), (max_z(&existing) + 1.0).into()); - full.insert("createdAt".into(), ctx.now_ms.into()); - full.insert("updatedAt".into(), ctx.now_ms.into()); - for (k, v) in props { - full.insert(k.clone(), v.clone()); - } - let mut prelim: Vec<(String, Any)> = vec![ - ("id".into(), Any::from(id)), - ("type".into(), Any::from(kind.as_str())), - ]; - for (k, v) in &full { - if k == "text" || v.is_null() { - continue; - } - if let Some(any) = store_value(k, v) { - prelim.push((k.clone(), any)); - } - } - let m = shapes.insert(txn, id, MapPrelim::from_iter(prelim)); - let text = full.get("text").and_then(Value::as_str); - if kind.has_body() { - m.insert(txn, "text", TextPrelim::new(text.unwrap_or_default())); - } else if let Some(t) = text.filter(|t| !t.is_empty()) { - m.insert(txn, "text", t); - } -} - -fn need_shape(txn: &TransactionMut, shapes: &MapRef, id: &str) -> Result { - live_shape(txn, shapes, bare_id(id)).ok_or_else(|| format!("no shape {id}")) -} - -/// Where an agent's cursor goes after touching `id`. -fn anchor_of(txn: &TransactionMut, shapes: &MapRef, id: &str, ctx: &OpsCtx) -> Point { - let all = all_shapes(txn, shapes); - let Some(s) = all.iter().find(|s| s.id == id) else { - return ctx.origin; - }; - if s.is_arrow() { - return arrow_mid(s, &box_lookup(&all)).unwrap_or(ctx.origin); - } - center(&s.bbox()) -} - -// ---- the ops ------------------------------------------------------------------------------ - -/// Fills an image's missing size from its natural size (and vice versa), like the page. -fn size_image(props: &mut JsonMap) { - let nw = num(props, "naturalW").unwrap_or(0.0); - let nh = num(props, "naturalH").unwrap_or(0.0); - let (pw, ph) = (num(props, "w"), num(props, "h")); - if pw.is_none() || ph.is_none() { - let (sw, sh) = image_size_for( - if nw > 0.0 { nw } else { 320.0 }, - if nh > 0.0 { nh } else { 240.0 }, - ); - let (width, height) = match (pw, ph) { - (None, Some(height)) => (js_round(height * sw / sh), height), - (Some(width), None) => (width, js_round(width * sh / sw)), - _ => (sw, sh), - }; - props.insert("w".into(), width.into()); - props.insert("h".into(), height.into()); - } - let width = num(props, "w").unwrap_or(0.0); - let height = num(props, "h").unwrap_or(0.0); - if nw <= 0.0 || nh <= 0.0 { - props.insert( - "naturalW".into(), - (if nw > 0.0 { nw } else { width }).into(), - ); - props.insert( - "naturalH".into(), - (if nh > 0.0 { nh } else { height }).into(), - ); - } -} - -/// Per-type requirements and derived defaults of a shape being added. -fn type_defaults(kind: ShapeType, props: &mut JsonMap) -> Result<(), String> { - match kind { - ShapeType::Image => { - if !props.contains_key("src") { - return Err("an image needs `src`".into()); - } - size_image(props); - } - ShapeType::Frame => { - if let Some(img) = props.get("image").and_then(ImageRef::read) { - if !props.contains_key("w") && !props.contains_key("h") { - let nw = if img.natural_w > 0.0 { - img.natural_w - } else { - 640.0 - }; - let nh = if img.natural_h > 0.0 { - img.natural_h - } else { - 480.0 - }; - let (width, height) = frame_size_for(nw, nh); - props.insert("w".into(), width.into()); - props.insert("h".into(), height.into()); - } - } - } - ShapeType::Checklist => { - let input = ChecklistInput::take(props); - let columns = input - .columns - .unwrap_or_else(|| DEFAULT_COLUMNS.iter().map(|c| (*c).to_owned()).collect()); - let rows = input.rows.unwrap_or_default(); - if !props.contains_key("title") { - props.insert("title".into(), "".into()); - } - // Checked before anything is written: a bad pick fails the whole add. - if let Some(p) = &input.picks { - clean_picks(p, &rows, &columns)?; - } - if !props.contains_key("w") { - props.insert("w".into(), checklist_width(columns.len()).into()); - } - if !props.contains_key("h") { - props.insert( - "h".into(), - checklist_height(columns.len(), rows.len()).into(), - ); - } - props.insert("columns".into(), json!(columns)); - props.insert( - "rows".into(), - Value::Array(rows.iter().map(Row::to_json).collect()), - ); - if let Some(p) = input.picks { - props.insert("picks".into(), p); - } - } - ShapeType::Link => { - let Some(u) = props.get("url").and_then(Value::as_str).map(str::to_owned) else { - return Err("a link needs `url`".into()); - }; - if props - .get("title") - .and_then(Value::as_str) - .is_none_or(str::is_empty) - { - props.insert("title".into(), host_of(&u).into()); - } - } - _ => {} - } - Ok(()) -} - -#[allow(clippy::many_single_char_names)] -fn op_add( - txn: &mut TransactionMut, - shapes: &MapRef, - raw: &JsonMap, - ctx: &OpsCtx, -) -> Result<(String, Point), String> { - let kind = raw - .get("type") - .and_then(Value::as_str) - .and_then(ShapeType::parse) - .ok_or("`shape.type` must be sticky, text, frame, arrow, image, link or checklist")?; - let id = match raw.get("id") { - None => None, - Some(Value::String(s)) if valid_shape_id(s) => { - if shapes.contains_key(txn, s) { - return Err(format!("a shape with id {s} already exists")); - } - Some(s.clone()) - } - Some(_) => return Err("`shape.id` must be 1–64 of A–Z a–z 0–9 _ - : .".into()), - }; - let mut props = clean_props(kind, raw, txn, shapes, id.as_deref())?; - let id = id.unwrap_or_else(|| uuid::Uuid::now_v7().to_string()); - props.insert("by".into(), ctx.by.clone().into()); - - if kind == ShapeType::Arrow { - if !(props.contains_key("from") && props.contains_key("to")) { - return Err("an arrow needs `from` and `to`".into()); - } - create(txn, shapes, kind, &id, &props, ctx); - let all = all_shapes(txn, shapes); - let anchor = all - .iter() - .find(|s| s.id == id) - .and_then(|s| arrow_mid(s, &box_lookup(&all))) - .unwrap_or(Point { x: 0.0, y: 0.0 }); - return Ok((id, anchor)); - } - type_defaults(kind, &mut props)?; - let (dw, dh) = kind.default_size(); - let (mut w, mut h) = ( - num(&props, "w").unwrap_or(dw), - num(&props, "h").unwrap_or(dh), - ); - if let Some((mw, mh)) = kind.min_size() { - w = w.max(mw); - h = h.max(mh); - } - props.insert("w".into(), w.into()); - props.insert("h".into(), h.into()); - let (x, y) = (num(&props, "x"), num(&props, "y")); - if x.is_none() || y.is_none() { - let taken: Vec = all_shapes(txn, shapes) - .iter() - .filter(|s| !s.is_arrow()) - .map(Shape::bbox) - .collect(); - let spot = find_free_spot(&taken, w, h, ctx.origin); - if x.is_none() { - props.insert("x".into(), spot.x.into()); - } - if y.is_none() { - props.insert("y".into(), spot.y.into()); - } - } - let picks = props.remove("picks"); - create(txn, shapes, kind, &id, &props, ctx); - if let (ShapeType::Checklist, Some(picks)) = (kind, picks) { - if let Some(m) = shape_map(txn, shapes, &id) { - let input = ChecklistInput { - picks: Some(picks), - ..ChecklistInput::default() - }; - write_checklist(txn, &m, &input, &picker(ctx), true, true)?; - } - } - let b = BoxF { - x: num(&props, "x").unwrap_or(0.0), - y: num(&props, "y").unwrap_or(0.0), - w, - h, - }; - Ok((id, center(&b))) -} - -fn op_update( - txn: &mut TransactionMut, - shapes: &MapRef, - id: &str, - patch: &JsonMap, - ctx: &OpsCtx, -) -> Result { - let shape = need_shape(txn, shapes, id)?; - let mut props = clean_props(shape.kind, patch, txn, shapes, Some(&shape.id))?; - clamp_min(shape.kind, &mut props); - if shape.is_arrow() { - let pick = |k: &str, cur: &Option| { - props - .get(k) - .and_then(Endpoint::read) - .or_else(|| cur.clone()) - }; - let (from, to) = (pick("from", &shape.from), pick("to", &shape.to)); - if let (Some(a), Some(b)) = ( - from.as_ref().and_then(Endpoint::ref_id), - to.as_ref().and_then(Endpoint::ref_id), - ) { - if a == b { - return Err("an arrow cannot start and end on the same shape".into()); - } - } - } - let m = shape_map(txn, shapes, &shape.id).ok_or_else(|| format!("no shape {id}"))?; - if shape.kind == ShapeType::Checklist { - let input = ChecklistInput::take(&mut props); - if let Some(p) = &input.picks { - let live = shape.checklist.clone().unwrap_or_default(); - clean_picks( - p, - input.rows.as_deref().unwrap_or(&live.rows), - input.columns.as_deref().unwrap_or(&live.columns), - )?; - } - let sized = (props.contains_key("w"), props.contains_key("h")); - store_update(txn, &m, shape.kind, &props, ctx); - write_checklist(txn, &m, &input, &picker(ctx), sized.0, sized.1)?; - return Ok(shape.id); - } - store_update(txn, &m, shape.kind, &props, ctx); - Ok(shape.id) -} - -/// Who a checklist pick made by this batch is attributed to. -fn picker(ctx: &OpsCtx) -> Picker<'_> { - Picker { - name: &ctx.by, - id: &ctx.by_id, - at: ctx.now_ms, - } -} - -fn op_move( - txn: &mut TransactionMut, - shapes: &MapRef, - id: &str, - dx: f64, - dy: f64, - ctx: &OpsCtx, -) -> Result { - let shape = need_shape(txn, shapes, id)?; - if !dx.is_finite() || !dy.is_finite() { - return Err("`dx` and `dy` must be finite numbers".into()); - } - let mut moves: Vec<(String, JsonMap)> = Vec::new(); - if shape.is_arrow() { - let shift = |e: &Option| match e { - Some(Endpoint::Point { x, y }) => Some(json!({ "x": jnum(x + dx), "y": jnum(y + dy) })), - Some(e) => Some(e.to_json()), - None => None, - }; - let mut p = JsonMap::new(); - if let Some(f) = shift(&shape.from) { - p.insert("from".into(), f); - } - if let Some(t) = shift(&shape.to) { - p.insert("to".into(), t); - } - moves.push((shape.id.clone(), p)); - } else { - let at = |s: &Shape| { - let mut p = JsonMap::new(); - p.insert("x".into(), (s.x + dx).into()); - p.insert("y".into(), (s.y + dy).into()); - p - }; - moves.push((shape.id.clone(), at(&shape))); - if shape.kind == ShapeType::Frame { - let frame = shape.bbox(); - for child in all_shapes(txn, shapes) { - if child.id != shape.id && !child.is_arrow() && contains_box(&frame, &child.bbox()) - { - moves.push((child.id.clone(), at(&child))); - } - } - } - } - for (mid, p) in moves { - if let Some(m) = shape_map(txn, shapes, &mid) { - for (k, v) in &p { - if let Some(any) = store_value(k, v) { - m.try_update(txn, k.as_str(), any); - } - } - m.try_update(txn, "updatedAt", Any::from(ctx.now_ms)); - } - } - Ok(shape.id) -} - -fn op_resize( - txn: &mut TransactionMut, - shapes: &MapRef, - id: &str, - w: Option, - h: Option, - ctx: &OpsCtx, -) -> Result { - let shape = need_shape(txn, shapes, id)?; - let Some((mw, mh)) = shape.kind.min_size() else { - return Err(format!("{} shapes have no size", shape.kind.as_str())); - }; - let (w, h) = (w.unwrap_or(shape.w), h.unwrap_or(shape.h)); - if !(w.is_finite() && h.is_finite() && w > 0.0 && h > 0.0) { - return Err("`w` and `h` must be positive numbers".into()); - } - let mut props = JsonMap::new(); - props.insert("w".into(), w.max(mw).into()); - props.insert("h".into(), h.max(mh).into()); - let m = shape_map(txn, shapes, &shape.id).ok_or_else(|| format!("no shape {id}"))?; - store_update(txn, &m, shape.kind, &props, ctx); - Ok(shape.id) -} - -fn op_delete(txn: &mut TransactionMut, shapes: &MapRef, id: &str) -> Result { - let shape = need_shape(txn, shapes, id)?; - let cascade: Vec = all_shapes(txn, shapes) - .into_iter() - .filter(|s| s.touches(&[shape.id.as_str()])) - .map(|s| s.id) - .collect(); - shapes.remove(txn, &shape.id); - for a in cascade { - shapes.remove(txn, &a); - } - Ok(shape.id) -} - -fn op_connect( - txn: &mut TransactionMut, - shapes: &MapRef, - op: &Op, - ctx: &OpsCtx, -) -> Result<(String, Point), String> { - let Op::Connect { - from, - to, - from_side, - to_side, - label, - color, - id, - } = op - else { - return Err("not a connect op".into()); - }; - let end = |v: &Value, side: &Option, key: &str| -> Result { - let mut e = Endpoint::read(v).ok_or_else(|| format!("`{key}` must be a shape id"))?; - if let (Endpoint::Ref { side: s, .. }, Some(raw)) = (&mut e, side) { - *s = Some( - raw.as_str() - .and_then(Side::parse) - .ok_or_else(|| format!("`{key}Side` must be top, right, bottom or left"))?, - ); - } - Ok(e) - }; - let from = end(from, from_side, "from")?; - let to = end(to, to_side, "to")?; - if let (Some(a), Some(b)) = (from.ref_id(), to.ref_id()) { - if a == b { - return Err("`from` and `to` are the same shape".into()); - } - } - let mut raw = JsonMap::new(); - raw.insert("from".into(), from.to_json()); - raw.insert("to".into(), to.to_json()); - if let Some(l) = label { - raw.insert("label".into(), l.clone()); - } - if let Some(c) = color { - raw.insert("color".into(), c.clone()); - } - let id = match id { - None => None, - Some(Value::String(s)) if valid_shape_id(s) => { - if shapes.contains_key(txn, s) { - return Err(format!("a shape with id {s} already exists")); - } - Some(s.clone()) - } - Some(_) => return Err("`id` must be 1–64 of A–Z a–z 0–9 _ - : .".into()), - }; - let mut props = clean_props(ShapeType::Arrow, &raw, txn, shapes, id.as_deref())?; - props.insert("by".into(), ctx.by.clone().into()); - let id = id.unwrap_or_else(|| uuid::Uuid::now_v7().to_string()); - create(txn, shapes, ShapeType::Arrow, &id, &props, ctx); - let at = anchor_of(txn, shapes, &id, ctx); - Ok((id, at)) -} - -// ---- agents ------------------------------------------------------------------------------- - -/// A change to an agent's presence (unset fields keep their previous value). -#[derive(Debug, Clone, Default, PartialEq)] -pub struct AgentPatch { - pub name: Option, - pub color: Option, - /// `Some(None)` clears the cursor. - pub cursor: Option>, - /// `idle` | `thinking` | `writing`. - pub status: Option<&'static str>, -} - -/// One `agents` entry as read back. -#[derive(Debug, Clone, PartialEq)] -pub struct AgentPresence { - pub name: String, - pub color: String, - pub cursor: Option, - pub status: String, - pub updated_at: f64, -} - -/// Tolerant read of one `agents` value (the page's `readAgent`). -#[allow(clippy::cast_precision_loss)] -pub fn read_agent(v: &Value) -> Option { - let o = v.as_object()?; - let cursor = o.get("cursor").and_then(Value::as_object).and_then(|c| { - Some(Point { - x: c.get("x").and_then(json_f64)?, - y: c.get("y").and_then(json_f64)?, - }) - }); - let updated = match o.get("updatedAt") { - Some(Value::Number(n)) => n - .as_f64() - .map_or(0.0, |t| if t < 1e12 { t * 1000.0 } else { t }), - Some(Value::String(s)) => { - time::OffsetDateTime::parse(s, &time::format_description::well_known::Rfc3339) - .map_or(0.0, |t| (t.unix_timestamp_nanos() / 1_000_000) as f64) - } - _ => 0.0, - }; - Some(AgentPresence { - name: o - .get("name") - .and_then(Value::as_str) - .map(str::trim) - .filter(|n| !n.is_empty()) - .unwrap_or("Agent") - .to_owned(), - color: o - .get("color") - .and_then(Value::as_str) - .unwrap_or("") - .to_owned(), - cursor, - status: o - .get("status") - .and_then(Value::as_str) - .filter(|s| AGENT_STATUSES.contains(s)) - .unwrap_or("idle") - .to_owned(), - updated_at: updated, - }) -} - -/// Merges `patch` into agent `id`'s entry and writes it whole (the page's `writeAgent`). -pub fn write_agent( - txn: &mut TransactionMut, - agents: &MapRef, - id: &str, - patch: &AgentPatch, - now_ms: i64, -) -> Value { - let prev = agents - .get(txn, id) - .map(|o| crate::schema::out_to_json(txn, &o)) - .and_then(|v| read_agent(&v)); - let cursor = match patch.cursor { - None => prev.as_ref().and_then(|p| p.cursor), - Some(None) => None, - Some(Some(p)) => Some(Point { - x: js_round(p.x), - y: js_round(p.y), - }), - }; - let name = patch - .name - .as_deref() - .map(str::trim) - .filter(|n| !n.is_empty()) - .map(str::to_owned) - .or_else(|| prev.as_ref().map(|p| p.name.clone())) - .unwrap_or_else(|| "Agent".into()); - let color = patch - .color - .clone() - .filter(|c| !c.is_empty()) - .or_else(|| prev.as_ref().map(|p| p.color.clone())) - .unwrap_or_default(); - let status = patch - .status - .map(str::to_owned) - .or_else(|| prev.as_ref().map(|p| p.status.clone())) - .unwrap_or_else(|| "idle".into()); - let value = json!({ - "name": js_slice(&name, 60), - "color": js_slice(&color, 32), - "cursor": cursor.map(|c| json!({ "x": jnum(c.x), "y": jnum(c.y) })), - "status": status, - "updatedAt": now_ms, - }); - agents.insert(txn, id, json_to_any(&value)); - value -} - -/// Who an ops call acts as (`as` in the request), parsed like the page's `parseApplyInput`. -#[derive(Debug, Clone, PartialEq)] -pub struct Actor { - pub id: String, - pub name: String, - pub color: Option, -} - -impl Actor { - /// `"name"` or `{id?, name?, color?}`; `Ok(None)` when absent or null. - pub fn parse(v: Option<&Value>) -> Result, String> { - match v { - None | Some(Value::Null) => Ok(None), - Some(Value::String(s)) => Ok(Some(Self { - id: format!("agent:{s}"), - name: s.clone(), - color: None, - })), - Some(Value::Object(o)) => { - let name = o - .get("name") - .and_then(Value::as_str) - .map(str::trim) - .filter(|n| !n.is_empty()) - .map_or_else(|| "Agent".to_owned(), |n| js_slice(n, 60).to_owned()); - let id = o - .get("id") - .and_then(Value::as_str) - .map(str::trim) - .filter(|n| !n.is_empty()) - .map_or_else( - || format!("agent:{}", name.to_lowercase()), - |n| js_slice(n, 64).to_owned(), - ); - let color = o - .get("color") - .and_then(Value::as_str) - .filter(|c| !c.is_empty()) - .map(|c| js_slice(c, 32).to_owned()); - Ok(Some(Self { id, name, color })) - } - Some(_) => Err("`as` must be {id,name,color}".into()), - } - } - - /// The presence written right after this actor's ops landed. - pub fn writing(&self, at: Option) -> AgentPatch { - AgentPatch { - name: Some(self.name.clone()), - color: Some(self.color.clone().unwrap_or_else(|| color_for(&self.id))), - cursor: at.map(Some), - status: Some("writing"), - } - } -} - -#[cfg(test)] -mod tests; diff --git a/crates/canvas/src/ops/tests.rs b/crates/canvas/src/ops/tests.rs deleted file mode 100644 index 9175b42..0000000 --- a/crates/canvas/src/ops/tests.rs +++ /dev/null @@ -1,790 +0,0 @@ -use super::*; -use crate::schema::out_to_json; -use serde_json::json; -use yrs::updates::decoder::Decode; -use yrs::{Doc, GetString, ReadTxn, Transact}; - -fn ctx() -> OpsCtx { - OpsCtx { - by: "tester".into(), - by_id: "u-tester".into(), - now_ms: 1_000, - confirm_clear: false, - origin: Point { x: 0.0, y: 0.0 }, - } -} - -#[allow(clippy::needless_pass_by_value)] -fn run(doc: &Doc, ops: Value) -> OpsResult { - let shapes = doc.get_or_insert_map("shapes"); - let mut txn = doc.transact_mut(); - apply_json_ops(&mut txn, &shapes, ops.as_array().unwrap(), &ctx()) -} - -fn shape(doc: &Doc, id: &str) -> Value { - let shapes = doc.get_or_insert_map("shapes"); - let txn = doc.transact(); - shapes - .get(&txn, id) - .map_or(Value::Null, |o| out_to_json(&txn, &o)) -} - -fn count(doc: &Doc) -> u32 { - let shapes = doc.get_or_insert_map("shapes"); - let txn = doc.transact(); - shapes.len(&txn) -} - -fn ok_ids(r: &OpsResult) -> Vec { - r.ids.iter().flatten().cloned().collect() -} - -#[test] -fn parse_ops() { - assert_eq!( - Op::parse(&json!({"op":"move","id":"shape:a","dx":1})).unwrap(), - Op::Move { - id: "a".into(), - dx: 1.0, - dy: 0.0 - } - ); - assert_eq!( - Op::parse(&json!({"op":"clear"})).unwrap(), - Op::Clear { confirm: false } - ); - assert_eq!( - Op::parse(&json!({"op":"update","id":"a","props":{"text":"x"}})).unwrap(), - Op::Update { - id: "a".into(), - patch: json!({"text":"x"}).as_object().unwrap().clone() - } - ); - assert!(Op::parse(&json!({"op":"explode"})) - .unwrap_err() - .starts_with("unknown op \"explode\"")); - assert!(Op::parse(&json!("add")).is_err()); - assert!(Op::parse(&json!({"op":"move","id":"a","dx":"far"})).is_err()); - assert!(Op::parse(&json!({"op":"update","id":"a"})).is_err()); - assert!(Op::parse(&json!({"op":"delete"})).is_err()); -} - -#[test] -fn add_with_explicit_geometry_and_defaults() { - let doc = Doc::new(); - let r = run( - &doc, - json!([{"op":"add","shape":{"id":"s1","type":"sticky","x":10.4,"y":20.6,"text":"hi","by":"spoof"}}]), - ); - assert_eq!(r.applied, 1, "{r:?}"); - assert_eq!(r.ids, vec![Some("s1".to_owned())]); - let s = shape(&doc, "s1"); - assert_eq!(s["id"], "s1"); - assert_eq!(s["type"], "sticky"); - assert_eq!(s["x"], 10, "boxes are rounded"); - assert_eq!(s["y"], 21); - assert_eq!(s["w"], 200); - assert_eq!(s["h"], 200); - assert_eq!(s["text"], "hi"); - assert_eq!(s["color"], "yellow"); - assert_eq!(s["by"], "tester"); - assert_eq!(s["createdAt"], 1000); - assert_eq!(s["updatedAt"], 1000); - assert_eq!(s["z"], 1); - // Sticky bodies are a Y.Text. - let shapes = doc.get_or_insert_map("shapes"); - let txn = doc.transact(); - let Some(Out::YMap(m)) = shapes.get(&txn, "s1") else { - panic!() - }; - assert!(matches!(m.get(&txn, "text"), Some(Out::YText(_)))); - assert_eq!(r.anchor, Some(Point { x: 110.4, y: 120.6 })); -} - -#[test] -fn add_places_in_free_space_with_uuid_ids() { - let doc = Doc::new(); - let r = run( - &doc, - json!([ - {"op":"add","shape":{"type":"sticky"}}, - {"op":"add","shape":{"type":"sticky"}}, - {"op":"add","shape":{"type":"frame","title":"F"}}, - {"op":"add","shape":{"type":"text","text":"t"}} - ]), - ); - assert_eq!(r.applied, 4, "{r:?}"); - let ids = ok_ids(&r); - for id in &ids { - assert_eq!(uuid::Uuid::parse_str(id).unwrap().get_version_num(), 7); - } - let first = shape(&doc, &ids[0]); - assert_eq!( - (first["x"].clone(), first["y"].clone()), - (json!(-100), json!(-100)) - ); - let boxes: Vec = ids - .iter() - .map(|id| { - let s = shape(&doc, id); - BoxF { - x: s["x"].as_f64().unwrap(), - y: s["y"].as_f64().unwrap(), - w: s["w"].as_f64().unwrap(), - h: s["h"].as_f64().unwrap(), - } - }) - .collect(); - for (i, a) in boxes.iter().enumerate() { - for b in &boxes[i + 1..] { - assert!( - !crate::geometry::boxes_overlap(a, b, 0.0), - "{a:?} overlaps {b:?}" - ); - } - } - let z: Vec = ids - .iter() - .map(|id| shape(&doc, id)["z"].as_f64().unwrap()) - .collect(); - assert_eq!(z, vec![1.0, 2.0, 3.0, 4.0]); - assert_eq!(shape(&doc, &ids[3])["h"], 36); -} - -#[test] -fn add_validation_errors() { - let doc = Doc::new(); - let r = run( - &doc, - json!([ - {"op":"add","shape":{"x":1}}, - {"op":"add","shape":{"type":"blob"}}, - {"op":"add","shape":{"type":"sticky","color":"red"}}, - {"op":"add","shape":{"type":"sticky","w":-5}}, - {"op":"add","shape":{"type":"link"}}, - {"op":"add","shape":{"type":"link","url":"javascript:alert(1)"}}, - {"op":"add","shape":{"type":"image"}}, - {"op":"add","shape":{"type":"image","src":"http://insecure/x.png"}}, - {"op":"add","shape":{"type":"arrow","from":{"x":0,"y":0}}}, - {"op":"add","shape":{"type":"arrow","from":{"x":0,"y":0},"to":{"x":5,"y":5},"x":3}}, - {"op":"add","shape":{"type":"sticky","url":"https://x.y"}}, - {"op":"add","shape":{"id":"bad id!","type":"text"}}, - {"op":"add","shape":{"id":"dup","type":"text"}}, - {"op":"add","shape":{"id":"dup","type":"text"}}, - {"op":"add","shape":{"type":"text","fontSize":2}}, - {"op":"nope"}, - 42 - ]), - ); - assert_eq!(r.applied, 1, "{r:?}"); - assert_eq!(r.ids.len(), 17); - assert_eq!(r.ids[12], Some("dup".to_owned())); - let idx: Vec = r.errors.iter().map(|e| e.index).collect(); - assert_eq!( - idx, - vec![0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15, 16] - ); - assert_eq!(r.errors[14].op, "nope"); - assert_eq!(r.errors[15].op, ""); - assert!( - r.errors[9].error.contains("arrows have no box"), - "{:?}", - r.errors[9] - ); - assert!( - r.errors[10].error.contains("unknown prop `url` for sticky"), - "{:?}", - r.errors[10] - ); - assert_eq!(count(&doc), 1); -} - -#[test] -fn aliases_images_links_and_min_sizes() { - let doc = Doc::new(); - let r = run( - &doc, - json!([ - {"op":"add","shape":{"id":"i","type":"image","src":"https://x/y.png","naturalW":960,"naturalH":480,"x":0,"y":0}}, - {"op":"add","shape":{"id":"l","type":"link","url":"www.example.com/a","text":"Ex","color":"#FF00aa","x":0,"y":400}}, - {"op":"add","shape":{"id":"f","type":"frame","text":"Group","x":0,"y":800,"image":{"src":"data:image/png;base64,AA","naturalW":1000,"naturalH":500}}}, - {"op":"add","shape":{"id":"tiny","type":"sticky","w":10,"h":10,"x":0,"y":-400}}, - {"op":"add","shape":{"id":"lk","type":"link","url":"https://www.rust-lang.org","x":800,"y":0}} - ]), - ); - assert_eq!(r.applied, 5, "{r:?}"); - let i = shape(&doc, "i"); - assert_eq!((i["w"].clone(), i["h"].clone()), (json!(480), json!(240))); - let l = shape(&doc, "l"); - assert_eq!(l["url"], "https://www.example.com/a"); - assert_eq!(l["title"], "Ex", "text is a link's title"); - assert_eq!(l["color"], "#FF00aa"); - let f = shape(&doc, "f"); - assert_eq!(f["title"], "Group"); - assert_eq!((f["w"].clone(), f["h"].clone()), (json!(640), json!(320))); - assert_eq!(f["image"]["naturalW"], 1000); - let tiny = shape(&doc, "tiny"); - assert_eq!( - (tiny["w"].clone(), tiny["h"].clone()), - (json!(96), json!(64)) - ); - assert_eq!(shape(&doc, "lk")["title"], "rust-lang.org"); -} - -#[test] -fn update_merges_splices_text_and_ignores_server_keys() { - let doc = Doc::new(); - run( - &doc, - json!([{"op":"add","shape":{"id":"a","type":"sticky","x":0,"y":0,"text":"hello world"}}]), - ); - let r = run( - &doc, - json!([ - {"op":"update","id":"a","patch":{"text":"hello brave world","color":"pink","by":"spoof","type":"text"}}, - {"op":"update","id":"shape:a","props":{"w":50}}, - {"op":"update","id":"a","patch":{"x":null}}, - {"op":"update","id":"a","patch":{"label":"nope"}}, - {"op":"update","id":"missing","patch":{"text":"x"}} - ]), - ); - assert_eq!(r.applied, 2, "{r:?}"); - assert_eq!(r.ids[..2], [Some("a".to_owned()), Some("a".to_owned())]); - assert_eq!(r.errors.len(), 3); - let s = shape(&doc, "a"); - assert_eq!(s["text"], "hello brave world"); - assert_eq!(s["color"], "pink"); - assert_eq!(s["by"], "tester", "by is the creator"); - assert_eq!(s["type"], "sticky"); - assert_eq!(s["w"], 96, "clamped to the minimum"); - - // A plain-string body (from an older writer) becomes a Y.Text on edit. - let shapes = doc.get_or_insert_map("shapes"); - { - let mut txn = doc.transact_mut(); - let Some(Out::YMap(m)) = shapes.get(&txn, "a") else { - panic!() - }; - m.insert(&mut txn, "text", "plain ünïcode"); - } - run( - &doc, - json!([{"op":"update","id":"a","patch":{"text":"plain ünïcödé!"}}]), - ); - let txn = doc.transact(); - let Some(Out::YMap(m)) = shapes.get(&txn, "a") else { - panic!() - }; - let Some(Out::YText(t)) = m.get(&txn, "text") else { - panic!("text is a Y.Text") - }; - assert_eq!(t.get_string(&txn), "plain ünïcödé!"); -} - -#[test] -fn text_splice_is_minimal() { - assert_eq!( - text_splice("hello world", "hello brave world"), - (6, 0, "brave ") - ); - assert_eq!(text_splice("abc", "abc"), (3, 0, "")); - assert_eq!(text_splice("abcdef", "abef"), (2, 2, "")); - assert_eq!(text_splice("", "x"), (0, 0, "x")); - assert_eq!(text_splice("añb", "aöb"), (1, 2, "ö")); - assert_eq!(text_splice("aaa", "aa"), (2, 1, "")); -} - -#[test] -fn move_carries_frame_children_and_arrow_points() { - let doc = Doc::new(); - run( - &doc, - json!([ - {"op":"add","shape":{"id":"f","type":"frame","x":0,"y":0,"w":500,"h":500}}, - {"op":"add","shape":{"id":"in","type":"sticky","x":50,"y":50,"w":100,"h":100}}, - {"op":"add","shape":{"id":"out","type":"sticky","x":800,"y":0,"w":100,"h":100}}, - {"op":"add","shape":{"id":"ar","type":"arrow","from":{"x":0,"y":0},"to":{"ref":"out"}}} - ]), - ); - let r = run( - &doc, - json!([ - {"op":"move","id":"f","dx":10,"dy":20}, - {"op":"move","id":"ar","dx":5,"dy":5}, - {"op":"move","id":"zzz","dx":1,"dy":1} - ]), - ); - assert_eq!(r.applied, 2, "{r:?}"); - assert_eq!(shape(&doc, "f")["x"], 10); - assert_eq!(shape(&doc, "in")["y"], 70, "frame children move along"); - assert_eq!(shape(&doc, "out")["x"], 800, "others stay"); - let ar = shape(&doc, "ar"); - assert_eq!(ar["from"], json!({"x":5,"y":5})); - assert_eq!(ar["to"], json!({"ref":"out"})); - assert_eq!(ar["x"], 0, "arrows keep no box of their own"); -} - -#[test] -fn resize_rules() { - let doc = Doc::new(); - run( - &doc, - json!([ - {"op":"add","shape":{"id":"a","type":"sticky","x":0,"y":0}}, - {"op":"add","shape":{"id":"b","type":"sticky","x":400,"y":0}}, - {"op":"connect","id":"c","from":"a","to":"b"} - ]), - ); - let r = run( - &doc, - json!([ - {"op":"resize","id":"a","w":260}, - {"op":"resize","id":"a","w":0,"h":100}, - {"op":"resize","id":"c","w":10,"h":10}, - {"op":"resize","id":"b","w":1,"h":1} - ]), - ); - assert_eq!(r.applied, 2, "{r:?}"); - let a = shape(&doc, "a"); - assert_eq!((a["w"].clone(), a["h"].clone()), (json!(260), json!(200))); - let b = shape(&doc, "b"); - assert_eq!((b["w"].clone(), b["h"].clone()), (json!(96), json!(64))); - assert!(r.errors[1].error.contains("arrow shapes have no size")); -} - -#[test] -fn delete_cascades_to_arrows() { - let doc = Doc::new(); - run( - &doc, - json!([ - {"op":"add","shape":{"id":"a","type":"sticky","x":0,"y":0}}, - {"op":"add","shape":{"id":"b","type":"sticky","x":400,"y":0}}, - {"op":"add","shape":{"id":"c","type":"sticky","x":800,"y":0}}, - {"op":"connect","id":"ab","from":"a","to":"b"}, - {"op":"connect","id":"bc","from":"b","to":"c","fromSide":"right","toSide":"left"}, - {"op":"connect","id":"ac","from":"a","to":"c"} - ]), - ); - assert_eq!(shape(&doc, "bc")["from"], json!({"ref":"b","side":"right"})); - let r = run( - &doc, - json!([{"op":"delete","id":"b"},{"op":"delete","id":"b"}]), - ); - assert_eq!(r.applied, 1, "{r:?}"); - assert_eq!(r.ids, vec![Some("b".to_owned()), None]); - for gone in ["b", "ab", "bc"] { - assert_eq!(shape(&doc, gone), Value::Null, "{gone}"); - } - assert_eq!(shape(&doc, "ac")["type"], "arrow"); - assert_eq!(count(&doc), 3); -} - -#[test] -fn connect_validation_and_by() { - let doc = Doc::new(); - run( - &doc, - json!([ - {"op":"add","shape":{"id":"a","type":"sticky","x":0,"y":0}}, - {"op":"add","shape":{"id":"b","type":"sticky","x":400,"y":0}}, - {"op":"add","shape":{"id":"ar","type":"arrow","from":{"x":0,"y":0},"to":{"x":1,"y":1}}} - ]), - ); - let r = run( - &doc, - json!([ - {"op":"connect","from":"a","to":"a"}, - {"op":"connect","from":"a","to":"missing"}, - {"op":"connect","from":"a","to":"ar"}, - {"op":"connect","from":"a","to":"b","fromSide":"middle"}, - {"op":"connect","from":"a","to":"b","color":"blurple"}, - {"op":"connect","from":"shape:a","to":{"id":"b"},"label":"next","color":"blue"} - ]), - ); - assert_eq!(r.applied, 1, "{r:?}"); - assert_eq!(r.errors.len(), 5); - let id = r.ids[5].clone().unwrap(); - let arrow = shape(&doc, &id); - assert_eq!(arrow["from"], json!({"ref":"a"})); - assert_eq!(arrow["to"], json!({"ref":"b"})); - assert_eq!(arrow["label"], "next"); - assert_eq!(arrow["color"], "blue"); - assert_eq!(arrow["by"], "tester"); - // Anchor is the arrow's midpoint. - assert_eq!(r.anchor, Some(Point { x: 300.0, y: 100.0 })); - - let r = run( - &doc, - json!([{"op":"update","id":id,"patch":{"to":"a"}}, {"op":"update","id":id,"patch":{"to":{"x":9,"y":9}}}]), - ); - assert_eq!(r.applied, 1, "{r:?}"); - assert_eq!(shape(&doc, &id)["to"], json!({"x":9,"y":9})); -} - -#[test] -fn clear_requires_confirmation() { - let doc = Doc::new(); - run( - &doc, - json!([{"op":"add","shape":{"type":"sticky"}},{"op":"add","shape":{"type":"text"}}]), - ); - let r = run(&doc, json!([{"op":"clear"}])); - assert_eq!(r.applied, 0); - assert_eq!(count(&doc), 2); - let r = run(&doc, json!([{"op":"clear","confirm":true}])); - assert_eq!(r.applied, 1); - assert_eq!(r.ids, vec![None]); - assert_eq!(count(&doc), 0); - - run(&doc, json!([{"op":"add","shape":{"type":"sticky"}}])); - let shapes = doc.get_or_insert_map("shapes"); - let mut txn = doc.transact_mut(); - let mut c = ctx(); - c.confirm_clear = true; - let r = apply_ops(&mut txn, &shapes, &[Op::Clear { confirm: false }], &c); - assert_eq!(r.applied, 1); - assert_eq!(shapes.len(&txn), 0); -} - -#[test] -fn too_many_ops_is_a_request_error() { - let doc = Doc::new(); - let ops: Vec = (0..=MAX_OPS).map(|_| json!({"op":"clear"})).collect(); - let r = run(&doc, Value::Array(ops)); - assert_eq!(r.applied, 0); - assert_eq!(r.errors[0].index, -1); - assert_eq!(r.ids.len(), 0, "no ids for an empty op list"); -} - -#[test] -fn ops_produce_a_transferable_update() { - let doc = Doc::new(); - let shapes = doc.get_or_insert_map("shapes"); - let before = doc.transact().state_vector(); - { - let mut txn = doc.transact_mut(); - let op = Op::parse(&json!({"op":"add","shape":{"id":"s","type":"sticky","text":"sync"}})) - .unwrap(); - assert_eq!(apply_ops(&mut txn, &shapes, &[op], &ctx()).applied, 1); - } - let update = doc.transact().encode_state_as_update_v1(&before); - let other = Doc::new(); - other - .transact_mut() - .apply_update(yrs::Update::decode_v1(&update).unwrap()) - .unwrap(); - assert_eq!(shape(&other, "s")["text"], "sync"); -} - -#[test] -fn actors_and_presence() { - assert_eq!(Actor::parse(None).unwrap(), None); - assert_eq!( - Actor::parse(Some(&json!("Claude"))).unwrap().unwrap().id, - "agent:Claude" - ); - let a = Actor::parse(Some(&json!({"name":" Planner "}))) - .unwrap() - .unwrap(); - assert_eq!( - (a.id.as_str(), a.name.as_str()), - ("agent:planner", "Planner") - ); - let b = Actor::parse(Some(&json!({}))).unwrap().unwrap(); - assert_eq!(b.name, "Agent"); - assert!(Actor::parse(Some(&json!(3))).is_err()); - assert_eq!(color_for("agent:planner"), color_for("agent:planner")); - assert!(color_for("x").starts_with("hsl(")); - - let doc = Doc::new(); - let agents = doc.get_or_insert_map("agents"); - let mut txn = doc.transact_mut(); - write_agent( - &mut txn, - &agents, - "ag", - &a.writing(Some(Point { x: 1.4, y: 2.6 })), - 5, - ); - let v = out_to_json(&txn, &agents.get(&txn, "ag").unwrap()); - assert_eq!(v["name"], "Planner"); - assert_eq!(v["status"], "writing"); - assert_eq!(v["cursor"], json!({"x":1,"y":3})); - assert_eq!(v["updatedAt"], 5); - // A partial patch keeps the rest. - write_agent( - &mut txn, - &agents, - "ag", - &AgentPatch { - status: Some("idle"), - ..AgentPatch::default() - }, - 6, - ); - let v = out_to_json(&txn, &agents.get(&txn, "ag").unwrap()); - assert_eq!(v["status"], "idle"); - assert_eq!(v["name"], "Planner"); - assert_eq!(v["cursor"], json!({"x":1,"y":3})); -} - -#[test] -fn url_and_src_validation() { - assert_eq!( - valid_url(&json!("example.com")).unwrap(), - "https://example.com/" - ); - assert_eq!(valid_url(&json!("mailto:a@b.c")).unwrap(), "mailto:a@b.c"); - assert!(valid_url(&json!("ftp://x")).unwrap_err().contains("ftp:")); - assert!(valid_url(&json!(" ")).is_err()); - assert!(valid_image_src(&json!("data:image/png;base64,AA")).is_ok()); - assert!(valid_image_src(&json!("data:text/html,")).is_err()); - assert!(valid_image_src(&json!("https://a.b/c d.png")).is_err()); - let huge = format!("data:image/png;base64,{}", "A".repeat(MAX_DATA_URL)); - assert!(valid_image_src(&json!(huge)) - .unwrap_err() - .contains("over 2 MB")); -} - -// ---- checklist (the page's Canvas/src/__tests__/checklist-ops.test.ts) -------------------- - -fn read_one(doc: &Doc, id: &str) -> Value { - let txn = doc.transact(); - let out = crate::read::read_canvas_txn(&txn, &crate::read::ReadOpts::default()); - let shape = out - .shapes - .into_iter() - .find(|s| s.id == id) - .expect("shape in the read"); - serde_json::to_value(shape).unwrap() -} - -fn picks(v: &Value) -> Vec { - v["rows"] - .as_array() - .unwrap() - .iter() - .map(|r| r["pick"].clone()) - .collect() -} - -fn pick_keys(doc: &Doc, id: &str) -> usize { - shape(doc, id) - .as_object() - .unwrap() - .keys() - .filter(|k| k.starts_with("pick:")) - .count() -} - -#[test] -fn checklist_add_with_defaults_rows_picks_and_size() { - let doc = Doc::new(); - let r = run( - &doc, - json!([{ "op": "add", "shape": { "type": "checklist", "id": "tue", "x": 0, "y": 0, "title": "Tue", - "rows": ["Ann", "Bob", { "label": "Cy", "id": "cy" }], "picks": { "Ann": "yes", "cy": "No" } } }]), - ); - assert!(r.errors.is_empty(), "{:?}", r.errors); - let s = shape(&doc, "tue"); - assert_eq!(s["type"], "checklist"); - assert_eq!(s["title"], "Tue"); - assert_eq!(s["columns"], json!(["Yes", "No"])); - assert_eq!(s["color"], "green"); - assert_eq!(s["w"], 324); - assert_eq!(s["h"], 49 + 28 + 34 * 3 + 14); - assert_eq!(s["by"], "tester"); - assert_eq!(s["rows"][2], json!({ "id": "cy", "label": "Cy" })); - assert_eq!( - s["pick:cy"], - json!({ "col": "No", "by": "tester", "byId": "u-tester", "at": 1000 }) - ); - let read = read_one(&doc, "tue"); - assert_eq!(picks(&read), [json!("Yes"), Value::Null, json!("No")]); - assert_eq!(read["tally"], json!({ "Yes": 1, "No": 1 })); - assert_eq!(read["rows"][0]["by"], "tester"); - assert_eq!(read["rows"][0]["at"], 1000); - assert_eq!( - read["rows"][1], - json!({ "id": read["rows"][1]["id"], "label": "Bob", "pick": null }) - ); - assert!(read.get("picks").is_none()); -} - -#[test] -fn checklist_one_column_is_a_todo_list() { - let doc = Doc::new(); - run( - &doc, - json!([{ "op": "add", "shape": { "type": "checklist", "id": "todo", "columns": ["Done"], - "rows": ["Book table"], "picks": { "book table": true } } }]), - ); - let s = shape(&doc, "todo"); - assert_eq!(s["w"], 280); - assert_eq!(s["h"], 49 + 34 + 14); - let read = read_one(&doc, "todo"); - assert_eq!(read["rows"][0]["pick"], "Done"); - assert_eq!(read["tally"], json!({ "Done": 1 })); -} - -#[test] -fn checklist_bad_input_fails_the_whole_add() { - let doc = Doc::new(); - let r = run( - &doc, - json!([ - { "op": "add", "shape": { "type": "checklist", "rows": ["Ann"], "picks": { "Ann": "Maybe" } } }, - { "op": "add", "shape": { "type": "checklist", "columns": ["A", "B", "C", "D", "E"] } }, - { "op": "add", "shape": { "type": "checklist", "rows": "Ann" } }, - { "op": "add", "shape": { "type": "checklist", "rows": ["Ann"], "fontSize": 20 } } - ]), - ); - assert_eq!(r.applied, 0); - let at: Vec = r.errors.iter().map(|e| e.index).collect(); - assert_eq!(at, [0, 1, 2, 3]); - assert_eq!(r.errors[0].error, "`picks`: no column \"Maybe\" (Yes, No)"); - assert!( - r.errors[3].error.contains("unknown prop `fontSize`"), - "{}", - r.errors[3].error - ); - assert_eq!(count(&doc), 0); -} - -#[test] -fn checklist_update_picks_one_row_at_a_time() { - let doc = Doc::new(); - run( - &doc, - json!([{ "op": "add", "shape": { "type": "checklist", "id": "c", "rows": ["Ann", "Bob", "Cy"], - "picks": { "Ann": "Yes", "Bob": "No" } } }]), - ); - let r = run( - &doc, - json!([{ "op": "update", "id": "c", "patch": { "picks": { "Ann": null, "Cy": "yes" } } }]), - ); - assert!(r.errors.is_empty(), "{:?}", r.errors); - let read = read_one(&doc, "c"); - assert_eq!(picks(&read), [Value::Null, json!("No"), json!("Yes")]); - assert_eq!(read["tally"], json!({ "Yes": 1, "No": 1 })); - assert_eq!(pick_keys(&doc, "c"), 2); -} - -#[test] -fn checklist_rows_keep_their_picks_and_columns_drop_theirs() { - let doc = Doc::new(); - run( - &doc, - json!([{ "op": "add", "shape": { "type": "checklist", "id": "c", "rows": ["Ann", "Bob"], - "picks": { "Ann": "Yes", "Bob": "No" } } }]), - ); - let ann = shape(&doc, "c")["rows"][0]["id"].clone(); - run( - &doc, - json!([{ "op": "update", "id": "c", "patch": { "rows": ["Bob", "Dee", "Ann"], - "columns": ["Yes", "No", "Maybe"], "picks": { "Dee": "maybe" } } }]), - ); - let s = shape(&doc, "c"); - assert_eq!(s["rows"][2]["id"], ann); - assert_eq!(s["columns"], json!(["Yes", "No", "Maybe"])); - assert_eq!(s["w"], 14 * 2 + 168 + 64 * 3); - assert_eq!(s["h"], 49 + 28 + 34 * 3 + 14); - assert_eq!( - picks(&read_one(&doc, "c")), - [json!("No"), json!("Maybe"), json!("Yes")] - ); - run( - &doc, - json!([{ "op": "update", "id": "c", "patch": { "columns": ["Yes", "Maybe"], "rows": ["Bob", "Dee"] } }]), - ); - assert_eq!(picks(&read_one(&doc, "c")), [Value::Null, json!("Maybe")]); - assert_eq!(pick_keys(&doc, "c"), 1); - // A column renamed only in case keeps its picks, under the new spelling. - run( - &doc, - json!([{ "op": "update", "id": "c", "patch": { "columns": ["Yes", "MAYBE"] } }]), - ); - assert_eq!(picks(&read_one(&doc, "c")), [Value::Null, json!("MAYBE")]); -} - -#[test] -fn checklist_update_errors_change_nothing() { - let doc = Doc::new(); - run( - &doc, - json!([{ "op": "add", "shape": { "type": "checklist", "id": "c", "title": "T", "rows": ["Ann"] } }]), - ); - let r = run( - &doc, - json!([{ "op": "update", "id": "c", "patch": { "title": "New", "picks": { "Zed": "Yes" } } }]), - ); - assert_eq!(r.errors[0].error, "`picks`: no row \"Zed\""); - assert_eq!(shape(&doc, "c")["title"], "T"); -} - -#[test] -fn checklist_moves_resizes_with_a_floor_and_takes_text_as_title() { - let doc = Doc::new(); - run( - &doc, - json!([ - { "op": "add", "shape": { "type": "checklist", "id": "c", "x": 0, "y": 0, "rows": ["Ann"] } }, - { "op": "move", "id": "c", "dx": 10, "dy": 20 }, - { "op": "resize", "id": "c", "w": 50, "h": 50 }, - { "op": "add", "shape": { "type": "checklist", "id": "d", "text": "Wed dinner" } } - ]), - ); - let s = shape(&doc, "c"); - assert_eq!( - ( - s["x"].clone(), - s["y"].clone(), - s["w"].clone(), - s["h"].clone() - ), - (json!(10), json!(20), json!(200), json!(72)) - ); - assert_eq!(shape(&doc, "d")["title"], "Wed dinner"); - assert_eq!(read_one(&doc, "d")["columns"], json!(["Yes", "No"])); -} - -#[test] -fn checklist_concurrent_picks_on_different_rows_both_stick() { - let a = Doc::with_client_id(1); - run( - &a, - json!([{ "op": "add", "shape": { "type": "checklist", "id": "c", "rows": [{"label": "Ann", "id": "ann"}, {"label": "Bob", "id": "bob"}] } }]), - ); - let b = Doc::with_client_id(2); - let full = a - .transact() - .encode_state_as_update_v1(&yrs::StateVector::default()); - b.transact_mut() - .apply_update(yrs::Update::decode_v1(&full).unwrap()) - .unwrap(); - // Both write before either hears from the other. - run( - &a, - json!([{ "op": "update", "id": "c", "patch": { "picks": { "ann": "Yes" } } }]), - ); - run( - &b, - json!([{ "op": "update", "id": "c", "patch": { "picks": { "bob": "No" } } }]), - ); - let ua = a - .transact() - .encode_state_as_update_v1(&b.transact().state_vector()); - let ub = b - .transact() - .encode_state_as_update_v1(&a.transact().state_vector()); - a.transact_mut() - .apply_update(yrs::Update::decode_v1(&ub).unwrap()) - .unwrap(); - b.transact_mut() - .apply_update(yrs::Update::decode_v1(&ua).unwrap()) - .unwrap(); - for d in [&a, &b] { - assert_eq!(picks(&read_one(d, "c")), [json!("Yes"), json!("No")]); - } -} diff --git a/crates/canvas/src/read.rs b/crates/canvas/src/read.rs deleted file mode 100644 index a61bbe9..0000000 --- a/crates/canvas/src/read.rs +++ /dev/null @@ -1,479 +0,0 @@ -//! `canvas_read` format (spec §4), identical to the page's `readCanvas` -//! (`Canvas/src/ops.ts`): a compact, agent-friendly summary of a canvas document. - -use serde::Serialize; -use serde_json::Value; -use yrs::{Doc, Map as _, ReadTxn, Transact as _}; - -use crate::checklist::{row_summaries, ser_tally, tally, RowSummary}; -use crate::geometry::{contains_box, js_round}; -use crate::ops::read_agent; -use crate::schema::{jnum, js_len, js_slice, out_to_json, ser_num, ser_opt_num, ShapeType, AGENTS}; -use crate::shape::{arrow_bbox, box_lookup, doc_shapes, Endpoint, Shape}; - -/// Text, titles and labels are cut to this many characters unless `full`. -pub const READ_TEXT_LIMIT: usize = 500; -/// Agents not written to for this long are hidden. -pub const AGENT_TTL_MS: f64 = 2.0 * 60.0 * 1000.0; - -/// `{id, name, kind}` of the canvas being read. -#[derive(Debug, Clone, PartialEq, Serialize)] -pub struct CanvasRef { - pub id: String, - pub name: String, - pub kind: String, -} - -/// `{src, naturalW, naturalH}` with `src` described rather than echoed when it is data. -#[derive(Debug, Clone, PartialEq, Serialize)] -pub struct ImageSummary { - pub src: String, - #[serde(rename = "naturalW", serialize_with = "ser_num")] - pub natural_w: f64, - #[serde(rename = "naturalH", serialize_with = "ser_num")] - pub natural_h: f64, -} - -/// One shape in the read format. -#[derive(Debug, Clone, PartialEq, Serialize)] -pub struct ShapeSummary { - pub id: String, - #[serde(rename = "type")] - pub kind: String, - #[serde(serialize_with = "ser_num")] - pub x: f64, - #[serde(serialize_with = "ser_num")] - pub y: f64, - #[serde(serialize_with = "ser_num")] - pub w: f64, - #[serde(serialize_with = "ser_num")] - pub h: f64, - pub color: String, - #[serde(serialize_with = "ser_num")] - pub z: f64, - #[serde(skip_serializing_if = "Option::is_none")] - pub by: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub text: Option, - #[serde( - rename = "fontSize", - skip_serializing_if = "Option::is_none", - serialize_with = "ser_opt_num" - )] - pub font_size: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub align: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub title: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub image: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub label: Option, - /// Arrow ends: `{x,y}` or `{ref, side?}`. - #[serde(skip_serializing_if = "Option::is_none")] - pub from: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub to: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub src: Option, - #[serde( - rename = "naturalW", - skip_serializing_if = "Option::is_none", - serialize_with = "ser_opt_num" - )] - pub natural_w: Option, - #[serde( - rename = "naturalH", - skip_serializing_if = "Option::is_none", - serialize_with = "ser_opt_num" - )] - pub natural_h: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub url: Option, - /// A checklist's columns. - #[serde(skip_serializing_if = "Option::is_none")] - pub columns: Option>, - /// A checklist's rows, each with its pick (and who made it, when). - #[serde(skip_serializing_if = "Option::is_none")] - pub rows: Option>, - /// A checklist's count per column, in column order. - #[serde(skip_serializing_if = "Option::is_none", serialize_with = "ser_tally")] - pub tally: Option>, - /// The innermost frame this shape sits in. - #[serde(skip_serializing_if = "Option::is_none")] - pub frame: Option, -} - -/// One live agent. -#[derive(Debug, Clone, PartialEq, Serialize)] -pub struct AgentSummary { - pub id: String, - pub name: String, - pub color: String, - pub cursor: Option, - pub status: String, - #[serde(rename = "updatedAt", serialize_with = "ser_num")] - pub updated_at: f64, -} - -/// `{canvas, viewport?, shapes, agents, selection, count}`. -#[derive(Debug, Clone, PartialEq, Serialize)] -pub struct ReadResult { - #[serde(skip_serializing_if = "Option::is_none")] - pub canvas: Option, - /// The server has no viewport; always omitted. - #[serde(skip_serializing_if = "Option::is_none")] - pub viewport: Option, - pub shapes: Vec, - pub agents: Vec, - /// The server has no selection; always empty. - pub selection: Vec, - /// Every readable shape in the document (before `ids`/`types` filtering). - pub count: usize, -} - -/// Read options (`full`, `ids`, `types`). -#[derive(Debug, Clone, Default, PartialEq)] -pub struct ReadOpts { - /// Whole text instead of the first 500 characters. - pub full: bool, - /// Only these shapes. - pub ids: Option>, - /// Only these types. - pub types: Option>, - /// "Now" for the agent liveness window (ms epoch; 0 = current time). - pub now_ms: i64, -} - -fn clip(s: &str, full: bool) -> String { - if full || js_len(s) <= READ_TEXT_LIMIT { - s.to_owned() - } else { - format!("{}…", js_slice(s, READ_TEXT_LIMIT)) - } -} - -/// A data URL is never echoed whole: its kind and size are enough to reason about. -#[allow(clippy::cast_precision_loss)] -pub fn describe_src(src: &str, full: bool) -> String { - if !src.starts_with("data:") || (full && src.len() <= 4096) { - return src.to_owned(); - } - let kind = src["data:".len()..] - .split([';', ',']) - .next() - .filter(|k| !k.is_empty()) - .unwrap_or("data"); - let comma = src.find(',').map_or(-1.0, |i| i as f64); - let bytes = js_round((src.len() as f64 - comma - 1.0) * 3.0 / 4.0); - let size = if bytes >= 1024.0 * 1024.0 { - format!("{:.1} MB", bytes / 1024.0 / 1024.0) - } else { - format!("{} KB", js_round(bytes / 1024.0).max(1.0)) - }; - format!("data:{kind} ({size})") -} - -/// Reads every shape and agent from `doc` (text cut to 500 characters unless `full`). -pub fn read_shapes(doc: &Doc, full: bool) -> ReadResult { - let txn = doc.transact(); - read_canvas_txn( - &txn, - &ReadOpts { - full, - ..ReadOpts::default() - }, - ) -} - -fn summarize(s: &Shape, all: &[Shape], frames: &[&Shape], full: bool) -> ShapeSummary { - let bx = if s.is_arrow() { - arrow_bbox(s, &box_lookup(all)) - } else { - s.bbox() - }; - let mut sum = ShapeSummary { - id: s.id.clone(), - kind: s.kind.as_str().to_owned(), - x: js_round(bx.x), - y: js_round(bx.y), - w: js_round(bx.w), - h: js_round(bx.h), - color: s.color.clone(), - z: s.z, - by: s.by.clone(), - text: None, - font_size: None, - align: None, - title: None, - image: None, - label: None, - from: None, - to: None, - src: None, - natural_w: None, - natural_h: None, - url: None, - columns: None, - rows: None, - tally: None, - frame: None, - }; - match s.kind { - ShapeType::Sticky | ShapeType::Text => { - sum.text = Some(clip(&s.text, full)); - sum.font_size = s.font_size; - sum.align.clone_from(&s.align); - } - ShapeType::Frame => { - sum.title = Some(clip(&s.title, full)); - sum.image = s.image.as_ref().map(|i| ImageSummary { - src: describe_src(&i.src, full), - natural_w: i.natural_w, - natural_h: i.natural_h, - }); - } - ShapeType::Arrow => { - sum.label = Some(clip(&s.label, full)); - sum.from = s.from.as_ref().map(Endpoint::to_json); - sum.to = s.to.as_ref().map(Endpoint::to_json); - } - ShapeType::Image => { - sum.src = Some(describe_src(&s.src, full)); - sum.natural_w = Some(s.natural_w); - sum.natural_h = Some(s.natural_h); - } - ShapeType::Link => { - sum.url = Some(s.url.clone()); - sum.title = Some(clip(&s.title, full)); - } - ShapeType::Checklist => { - sum.title = Some(clip(&s.title, full)); - let c = s.checklist.clone().unwrap_or_default(); - sum.tally = Some(tally(&c)); - sum.rows = Some(row_summaries(&c)); - sum.columns = Some(c.columns); - } - } - if !s.is_arrow() { - let inner = s.bbox(); - let mut best: Option<&Shape> = None; - for f in frames { - if f.id == s.id || !contains_box(&f.bbox(), &inner) { - continue; - } - if best.is_none_or(|b| f.w * f.h < b.w * b.h) { - best = Some(f); - } - } - sum.frame = best.map(|f| f.id.clone()); - } - sum -} - -/// The board as an agent should see it, over an existing read transaction. -#[allow(clippy::cast_precision_loss)] -pub fn read_canvas_txn(txn: &T, opts: &ReadOpts) -> ReadResult { - let all = doc_shapes(txn); - let frames: Vec<&Shape> = all.iter().filter(|s| s.kind == ShapeType::Frame).collect(); - let mut shapes: Vec = all - .iter() - .filter(|s| opts.ids.as_ref().is_none_or(|ids| ids.contains(&s.id))) - .filter(|s| { - opts.types - .as_ref() - .is_none_or(|t| t.iter().any(|k| k == s.kind.as_str())) - }) - .map(|s| summarize(s, &all, &frames, opts.full)) - .collect(); - shapes.sort_by(|a, b| { - a.y.total_cmp(&b.y) - .then(a.x.total_cmp(&b.x)) - .then_with(|| a.id.cmp(&b.id)) - }); - - let now = if opts.now_ms > 0 { - opts.now_ms as f64 - } else { - crate::schema::now_ms() as f64 - }; - let mut agents: Vec = Vec::new(); - if let Some(map) = txn.get_map(AGENTS) { - for (id, value) in map.iter(txn) { - let Some(a) = read_agent(&out_to_json(txn, &value)) else { - continue; - }; - if a.updated_at <= 0.0 || now - a.updated_at > AGENT_TTL_MS { - continue; - } - agents.push(AgentSummary { - id: id.to_owned(), - name: a.name, - color: a.color, - cursor: a - .cursor - .map(|c| serde_json::json!({ "x": jnum(c.x), "y": jnum(c.y) })), - status: a.status, - updated_at: a.updated_at, - }); - } - } - let busy = |a: &AgentSummary| u8::from(a.status == "idle"); - agents.sort_by(|a, b| { - busy(a) - .cmp(&busy(b)) - .then_with(|| a.name.to_lowercase().cmp(&b.name.to_lowercase())) - .then_with(|| a.id.cmp(&b.id)) - }); - - ReadResult { - canvas: None, - viewport: None, - shapes, - agents, - selection: Vec::new(), - count: all.len(), - } -} - -/// Reads `meta.name` from a document, if set. -pub fn meta_name(txn: &T) -> Option { - let meta = txn.get_map(crate::schema::META)?; - crate::schema::out_string(txn, meta.get(txn, "name").as_ref()) -} - -#[cfg(test)] -#[allow(clippy::float_cmp)] -mod tests { - use super::*; - use crate::geometry::Point; - use crate::ops::{apply_json_ops, write_agent, AgentPatch, OpsCtx}; - use crate::schema::SHAPES; - use serde_json::json; - use yrs::Doc; - - fn ctx() -> OpsCtx { - OpsCtx { - by: "Ann".into(), - by_id: "u-ann".into(), - now_ms: 7, - confirm_clear: false, - origin: Point { x: 0.0, y: 0.0 }, - } - } - - #[test] - fn read_format() { - let doc = Doc::new(); - let shapes = doc.get_or_insert_map(SHAPES); - let agents = doc.get_or_insert_map(AGENTS); - let long = "x".repeat(600); - let now = crate::schema::now_ms(); - { - let mut txn = doc.transact_mut(); - let r = apply_json_ops( - &mut txn, - &shapes, - json!([ - {"op":"add","shape":{"id":"f","type":"frame","x":-50,"y":-50,"w":900,"h":400,"title":"Box"}}, - {"op":"add","shape":{"id":"s","type":"sticky","x":0,"y":0,"text":long}}, - {"op":"add","shape":{"id":"l","type":"link","x":400,"y":0,"url":"https://a.b"}}, - {"op":"add","shape":{"id":"i","type":"image","x":0,"y":600,"src":"data:image/png;base64,AAAA"}}, - {"op":"connect","from":"s","to":"l","label":"see"} - ]) - .as_array() - .unwrap(), - &ctx(), - ); - assert_eq!(r.applied, 5, "{r:?}"); - write_agent( - &mut txn, - &agents, - "bot", - &AgentPatch { - name: Some("Bot".into()), - status: Some("writing"), - ..AgentPatch::default() - }, - now, - ); - write_agent( - &mut txn, - &agents, - "old", - &AgentPatch { - name: Some("Old".into()), - ..AgentPatch::default() - }, - now - 10 * 60 * 1000, - ); - } - let r = read_shapes(&doc, false); - assert_eq!(r.count, 5); - let ids: Vec<&str> = r.shapes.iter().map(|s| s.id.as_str()).collect(); - // Sorted by y, then x. - assert_eq!(ids[0], "f"); - assert_eq!(*ids.last().unwrap(), "i"); - let s = r.shapes.iter().find(|s| s.id == "s").unwrap(); - assert_eq!( - s.text.as_ref().unwrap().chars().count(), - READ_TEXT_LIMIT + 1 - ); - assert_eq!(s.by.as_deref(), Some("Ann")); - assert_eq!(s.frame.as_deref(), Some("f")); - let l = r.shapes.iter().find(|s| s.id == "l").unwrap(); - assert_eq!( - l.title.as_deref(), - Some("a.b"), - "default link title is the host" - ); - assert_eq!(l.url.as_deref(), Some("https://a.b/")); - let arrow = r.shapes.iter().find(|s| s.kind == "arrow").unwrap(); - assert_eq!(arrow.label.as_deref(), Some("see")); - assert_eq!(arrow.from, Some(json!({"ref":"s"}))); - // Arrow box comes from the path between the two boxes. - assert_eq!((arrow.x, arrow.w), (200.0, 200.0)); - let img = r.shapes.iter().find(|s| s.id == "i").unwrap(); - assert_eq!(img.src.as_deref(), Some("data:image/png (1 KB)")); - assert_eq!(r.agents.len(), 1, "stale agents are hidden"); - assert_eq!(r.agents[0].name, "Bot"); - assert_eq!(r.agents[0].status, "writing"); - - let full = read_shapes(&doc, true); - let s = full.shapes.iter().find(|s| s.id == "s").unwrap(); - assert_eq!(s.text.as_ref().unwrap().len(), 600); - assert_eq!( - full.shapes - .iter() - .find(|s| s.id == "i") - .unwrap() - .src - .as_deref(), - Some("data:image/png;base64,AAAA") - ); - - let txn = doc.transact(); - let only = read_canvas_txn( - &txn, - &ReadOpts { - types: Some(vec!["sticky".into()]), - ..ReadOpts::default() - }, - ); - assert_eq!(only.shapes.len(), 1); - assert_eq!(only.count, 5); - - let v = serde_json::to_value(&r).unwrap(); - assert!(v.get("canvas").is_none()); - assert_eq!(v["selection"], json!([])); - assert!(v["shapes"][0].get("text").is_none()); - assert_eq!(v["shapes"][0]["type"], "frame"); - } - - #[test] - fn describe() { - assert_eq!(describe_src("https://x/y.png", false), "https://x/y.png"); - let big = format!("data:image/jpeg;base64,{}", "A".repeat(2_000_000)); - assert_eq!(describe_src(&big, true), "data:image/jpeg (1.4 MB)"); - } -} diff --git a/crates/canvas/src/rest.rs b/crates/canvas/src/rest.rs deleted file mode 100644 index a378025..0000000 --- a/crates/canvas/src/rest.rs +++ /dev/null @@ -1,1575 +0,0 @@ -//! Canvases REST API (mounted under `/v1` by the binary). -//! -//! Access control: every canvas lookup joins `canvas_members` on the caller, so a canvas the -//! caller is not a member of is indistinguishable from one that does not exist (404). - -use axum::body::Bytes; -use axum::extract::ws::rejection::WebSocketUpgradeRejection; -use axum::extract::{FromRequest, Path, Query, Request, State, WebSocketUpgrade}; -use axum::http::{header, HeaderMap, HeaderValue, StatusCode, Uri}; -use axum::response::{Html, IntoResponse, Response}; -use axum::Json; -use base64::engine::general_purpose::STANDARD; -use base64::Engine as _; -use copper_cloud_core::auth::{user_id_by_email, AuthUser}; -use copper_cloud_core::crypto::Crypto; -use copper_cloud_core::error::ApiError; -use copper_cloud_core::events::Event; -use copper_cloud_core::extract::{OptionalJsonBody, QueryParams}; -use copper_cloud_core::ids; -use copper_cloud_core::state::{AppState, SharedState}; -use serde::de::DeserializeOwned; -use serde::{Deserialize, Serialize}; -use std::sync::Arc; - -use serde_json::{json, Value}; -use time::OffsetDateTime; -use uuid::Uuid; -use yrs::updates::decoder::Decode as _; -use yrs::{Doc, Map as _, ReadTxn as _, StateVector, Transact as _, WriteTxn as _}; - -use crate::chat::{read_chat, ReadWithChat, READ_CHAT_LIMIT}; -use crate::geometry::Point; -use crate::ops::{ - apply_json_ops, read_agent, write_agent, Actor, AgentPatch, OpsCtx, OpsResult, WRITING_MS, -}; -use crate::read::{read_canvas_txn, CanvasRef, ReadOpts}; -use crate::room::{rooms, SERVER_ORIGIN}; -use crate::schema::{out_to_json, AGENTS, META, SHAPES}; -use crate::store; -use crate::ws::{serve_peer, MAX_MESSAGE_BYTES}; - -/// Path segment that resolves to the caller's Personal canvas. -pub const PERSONAL: &str = "personal"; -/// Longest canvas name (characters). -pub const MAX_NAME_CHARS: usize = 200; - -// --------------------------------------------------------------------------------------------- -// Extractors - -/// JSON body (any or no `Content-Type`, like core's extractor) whose rejections are reported -/// as [`ApiError`]: `413` over the router's body limit, `400` on malformed JSON. -pub(crate) struct Body(pub T); - -impl FromRequest for Body -where - T: DeserializeOwned, - S: Send + Sync, -{ - type Rejection = ApiError; - - async fn from_request(req: Request, state: &S) -> Result { - let bytes = Bytes::from_request(req, state).await.map_err(|e| { - if e.status() == StatusCode::PAYLOAD_TOO_LARGE { - ApiError::PayloadTooLarge - } else { - ApiError::bad_request("failed to read request body") - } - })?; - serde_json::from_slice(&bytes) - .map(Self) - .map_err(|e| ApiError::bad_request(format!("invalid JSON body: {e}"))) - } -} - -// --------------------------------------------------------------------------------------------- -// Views - -/// A user as shown inside canvas/member/invite views. -#[derive(Debug, Clone, Serialize)] -pub struct UserRef { - pub id: Uuid, - pub email: String, - pub display_name: String, -} - -/// `GET /canvases` item. -#[derive(Debug, Clone, Serialize)] -pub struct CanvasView { - pub id: Uuid, - pub name: String, - /// `personal` | `shared`. - pub kind: String, - /// The caller's role: `owner` | `editor`. - pub role: String, - pub owner: UserRef, - pub member_count: i64, - #[serde(with = "time::serde::rfc3339")] - pub created_at: OffsetDateTime, - #[serde(with = "time::serde::rfc3339")] - pub updated_at: OffsetDateTime, -} - -#[derive(sqlx::FromRow)] -struct CanvasRow { - id: Uuid, - name: String, - kind: String, - role: String, - owner_id: Uuid, - owner_email: String, - owner_display_name: String, - member_count: i64, - created_at: OffsetDateTime, - updated_at: OffsetDateTime, -} - -impl From for CanvasView { - fn from(r: CanvasRow) -> Self { - Self { - id: r.id, - name: r.name, - kind: r.kind, - role: r.role, - owner: UserRef { - id: r.owner_id, - email: r.owner_email, - display_name: r.owner_display_name, - }, - member_count: r.member_count, - created_at: r.created_at, - updated_at: r.updated_at, - } - } -} - -const CANVAS_VIEW_SQL: &str = " -SELECT c.id, c.name, c.kind, m.role, c.owner_id, - u.email AS owner_email, u.display_name AS owner_display_name, - (SELECT count(*) FROM canvas_members x WHERE x.canvas_id = c.id) AS member_count, - c.created_at, c.updated_at -FROM canvas_members m -JOIN canvases c ON c.id = m.canvas_id -JOIN users u ON u.id = c.owner_id -WHERE m.user_id = $1 AND ($2::uuid IS NULL OR c.id = $2) -ORDER BY (c.kind = 'personal') DESC, c.updated_at DESC, c.id"; - -/// `GET /canvases/:id/members` item. -#[derive(Debug, Clone, Serialize, sqlx::FromRow)] -pub struct MemberView { - pub user_id: Uuid, - pub email: String, - pub display_name: String, - pub role: String, - #[serde(with = "time::serde::rfc3339")] - pub added_at: OffsetDateTime, -} - -/// An invite as shown to the inviter's canvas and to the invitee. -#[derive(Debug, Clone, Serialize)] -pub struct InviteView { - pub id: Uuid, - pub canvas_id: Uuid, - pub canvas_name: String, - pub email: String, - /// `pending` | `accepted` | `declined` | `revoked`. - pub status: String, - pub invited_by: Option, - /// The account `email` belongs to, or `None` while nobody has signed up with it. - pub invitee: Option, - #[serde(with = "time::serde::rfc3339")] - pub created_at: OffsetDateTime, - /// When the invite was last re-sent as a reminder; `None` until the first one. - #[serde(with = "time::serde::rfc3339::option")] - pub nudged_at: Option, -} - -/// `POST /canvases/:id/invites` response: the invite plus whether this call reminded the -/// invitee of an invite that was already pending. -#[derive(Debug, Clone, Serialize)] -pub(crate) struct InviteSent { - #[serde(flatten)] - invite: InviteView, - /// `true` only on a `200` re-send that notified the invitee; `false` on a new invite - /// (`201`, which notifies on its own), when throttled, or when the email has no account yet. - nudged: bool, -} - -/// A share link as returned by the owner's link list. -#[derive(Debug, Clone, Serialize, sqlx::FromRow)] -pub struct ShareLinkView { - pub id: Uuid, - pub role: String, - pub created_by: Option, - #[serde(with = "time::serde::rfc3339")] - pub created_at: OffsetDateTime, - pub uses: i64, -} - -#[derive(Debug, Clone, Serialize)] -pub(crate) struct CreatedShareLink { - id: Uuid, - token: String, - canvas_id: Uuid, - role: String, - #[serde(with = "time::serde::rfc3339")] - created_at: OffsetDateTime, -} - -#[derive(Debug, Clone, sqlx::FromRow)] -struct ShareLinkPreviewRow { - canvas_id: Uuid, - name: String, - owner_id: Uuid, - owner_display_name: String, - role: String, -} - -/// `GET /people` item. This intentionally omits account state and other private fields. -#[derive(Debug, Clone, Serialize, sqlx::FromRow)] -pub struct PersonView { - pub id: Uuid, - pub display_name: String, - pub email: String, -} - -#[derive(sqlx::FromRow)] -struct InviteRow { - id: Uuid, - canvas_id: Uuid, - canvas_name: String, - email: String, - status: String, - created_at: OffsetDateTime, - nudged_at: Option, - inviter_id: Option, - inviter_email: Option, - inviter_display_name: Option, - invitee_id: Option, - invitee_email: Option, - invitee_display_name: Option, -} - -fn user_ref(id: Option, email: Option, name: Option) -> Option { - match (id, email) { - (Some(id), Some(email)) => Some(UserRef { - id, - email, - display_name: name.unwrap_or_default(), - }), - _ => None, - } -} - -impl From for InviteView { - fn from(r: InviteRow) -> Self { - Self { - id: r.id, - canvas_id: r.canvas_id, - canvas_name: r.canvas_name, - email: r.email, - status: r.status, - invited_by: user_ref(r.inviter_id, r.inviter_email, r.inviter_display_name), - invitee: user_ref(r.invitee_id, r.invitee_email, r.invitee_display_name), - created_at: r.created_at, - nudged_at: r.nudged_at, - } - } -} - -/// `invitee` joins on the lower-cased address (`users_email_lower_key`); invite emails are -/// stored lower-cased. -const INVITE_VIEW_SQL: &str = " -SELECT i.id, i.canvas_id, c.name AS canvas_name, i.email, i.status, i.created_at, i.nudged_at, - u.id AS inviter_id, u.email AS inviter_email, u.display_name AS inviter_display_name, - v.id AS invitee_id, v.email AS invitee_email, v.display_name AS invitee_display_name -FROM canvas_invites i -JOIN canvases c ON c.id = i.canvas_id -LEFT JOIN users u ON u.id = i.invited_by -LEFT JOIN users v ON lower(v.email) = i.email"; - -/// Re-sending a pending invite reminds the invitee at most once per this many seconds. -pub const INVITE_NUDGE_INTERVAL_SECS: u32 = 30; - -const SHARE_LINK_NOT_FOUND: &str = "link_not_found"; - -// --------------------------------------------------------------------------------------------- -// Helpers - -/// The caller's access to one canvas. -#[derive(Debug, Clone)] -pub(crate) struct Access { - pub canvas_id: Uuid, - pub name: String, - pub kind: String, - pub role: String, -} - -impl Access { - fn is_personal(&self) -> bool { - self.kind == "personal" - } - - fn is_owner(&self) -> bool { - self.role == "owner" - } -} - -/// Resolves `raw` (a UUID or `personal`) to a canvas the caller is a member of, else 404. -pub(crate) async fn access(state: &AppState, user_id: Uuid, raw: &str) -> Result { - let canvas_id = if raw == PERSONAL { - ensure_personal_canvas(state, user_id).await? - } else { - Uuid::parse_str(raw).map_err(|_| ApiError::NotFound)? - }; - let row: Option<(String, String, String)> = sqlx::query_as( - "SELECT c.name, c.kind, m.role FROM canvases c - JOIN canvas_members m ON m.canvas_id = c.id AND m.user_id = $2 - WHERE c.id = $1", - ) - .bind(canvas_id) - .bind(user_id) - .fetch_optional(&state.db) - .await?; - let (name, kind, role) = row.ok_or(ApiError::NotFound)?; - Ok(Access { - canvas_id, - name, - kind, - role, - }) -} - -fn clean_name(raw: &str, max_chars: usize) -> Option { - let s: String = raw - .trim() - .chars() - .filter(|c| !c.is_control()) - .take(max_chars) - .collect(); - let s = s.trim().to_owned(); - (!s.is_empty()).then_some(s) -} - -fn canvas_name(raw: &str) -> Result { - if raw.trim().chars().count() > MAX_NAME_CHARS { - return Err(ApiError::bad_request(format!( - "name must be at most {MAX_NAME_CHARS} characters" - ))); - } - clean_name(raw, MAX_NAME_CHARS).ok_or_else(|| ApiError::bad_request("name must not be empty")) -} - -/// Trimmed, lower-cased, plausibly-shaped email address. -fn invite_email(raw: &str) -> Result { - let email = raw.trim().to_lowercase(); - let valid = (3..=254).contains(&email.len()) - && !email.chars().any(|c| c.is_whitespace() || c.is_control()) - && email.split_once('@').is_some_and(|(local, domain)| { - !local.is_empty() - && domain.contains('.') - && !domain.starts_with('.') - && !domain.ends_with('.') - && !domain.contains('@') - }); - if valid { - Ok(email) - } else { - Err(ApiError::bad_request("invalid email address")) - } -} - -/// A fresh document update that sets `meta.name` / `meta.createdBy`. -fn initial_update(name: &str, created_by: Uuid) -> Vec { - let doc = Doc::new(); - let meta = doc.get_or_insert_map(META); - { - let mut txn = doc.transact_mut(); - meta.insert(&mut txn, "name", name); - meta.insert(&mut txn, "createdBy", created_by.to_string()); - } - let update = doc - .transact() - .encode_state_as_update_v1(&StateVector::default()); - update -} - -/// Inserts a canvas, its owner membership and its initial document inside `tx`. -async fn insert_canvas( - tx: &mut sqlx::PgConnection, - state: &AppState, - owner: Uuid, - name: &str, - kind: &str, -) -> Result, ApiError> { - let id = ids::uuid_v7(); - let key = Crypto::new_key(); - let wrapped = state.crypto.wrap_key(&key); - let inserted: Option = sqlx::query_scalar( - "INSERT INTO canvases (id, owner_id, name, kind, doc_key_wrapped) - VALUES ($1, $2, $3, $4, $5) - ON CONFLICT (owner_id) WHERE kind = 'personal' DO NOTHING - RETURNING id", - ) - .bind(id) - .bind(owner) - .bind(name) - .bind(kind) - .bind(&wrapped) - .fetch_optional(&mut *tx) - .await?; - if inserted.is_none() { - return Ok(None); - } - sqlx::query("INSERT INTO canvas_members (canvas_id, user_id, role) VALUES ($1, $2, 'owner')") - .bind(id) - .bind(owner) - .execute(&mut *tx) - .await?; - let sealed = Crypto::seal(&key, &store::aad(id), &initial_update(name, owner)); - sqlx::query(r#"INSERT INTO canvas_updates (canvas_id, "update") VALUES ($1, $2)"#) - .bind(id) - .bind(&sealed) - .execute(&mut *tx) - .await?; - Ok(Some(id)) -} - -/// Returns the caller's Personal canvas, creating it (kind `personal`, name `Personal`) on -/// first use. Safe under concurrency (unique partial index on `owner_id`). -pub async fn ensure_personal_canvas(state: &AppState, user_id: Uuid) -> Result { - let find = || { - sqlx::query_scalar::<_, Uuid>( - "SELECT id FROM canvases WHERE owner_id = $1 AND kind = 'personal'", - ) - .bind(user_id) - }; - if let Some(id) = find().fetch_optional(&state.db).await? { - return Ok(id); - } - let mut tx = state.db.begin().await?; - let created = insert_canvas(&mut tx, state, user_id, "Personal", "personal").await?; - if let Some(id) = created { - tx.commit().await?; - tracing::info!(user_id = %user_id, canvas_id = %id, "personal canvas created"); - return Ok(id); - } - tx.rollback().await?; - find() - .fetch_optional(&state.db) - .await? - .ok_or(ApiError::NotFound) -} - -async fn canvas_view(state: &AppState, user_id: Uuid, id: Uuid) -> Result { - sqlx::query_as::<_, CanvasRow>(CANVAS_VIEW_SQL) - .bind(user_id) - .bind(Some(id)) - .fetch_optional(&state.db) - .await? - .map(CanvasView::from) - .ok_or(ApiError::NotFound) -} - -pub(crate) async fn member_ids(state: &AppState, canvas_id: Uuid) -> Result, ApiError> { - Ok( - sqlx::query_scalar("SELECT user_id FROM canvas_members WHERE canvas_id = $1") - .bind(canvas_id) - .fetch_all(&state.db) - .await?, - ) -} - -pub(crate) fn publish(state: &AppState, users: &[Uuid], canvas_id: Uuid, kind: &str) { - for user in users { - state.events.publish( - *user, - Event::Canvas { - canvas_id, - kind: kind.to_owned(), - }, - ); - } -} - -async fn publish_members(state: &AppState, canvas_id: Uuid, kind: &str) { - match member_ids(state, canvas_id).await { - Ok(users) => publish(state, &users, canvas_id, kind), - Err(e) => { - tracing::warn!(%canvas_id, error = %e, "listing canvas members for an event failed"); - } - } -} - -/// The display name used for `by` when a REST caller does not say who they are. -async fn display_name(state: &AppState, user: &AuthUser) -> Result { - let name: Option = sqlx::query_scalar("SELECT display_name FROM users WHERE id = $1") - .bind(user.user_id) - .fetch_optional(&state.db) - .await?; - Ok(name - .filter(|n| !n.trim().is_empty()) - .unwrap_or_else(|| user.email.clone())) -} - -fn ok() -> Json { - Json(json!({ "ok": true })) -} - -fn link_not_found() -> ApiError { - ApiError::NotFoundCode { - code: SHARE_LINK_NOT_FOUND, - message: "share link not found", - } -} - -fn token_digest(token: &str) -> Option<[u8; 32]> { - ids::is_token_shape(token).then(|| ids::sha256(token.as_bytes())) -} - -// --------------------------------------------------------------------------------------------- -// Canvases - -pub(crate) async fn list_canvases( - State(state): State, - user: AuthUser, -) -> Result>, ApiError> { - ensure_personal_canvas(&state, user.user_id).await?; - let rows = sqlx::query_as::<_, CanvasRow>(CANVAS_VIEW_SQL) - .bind(user.user_id) - .bind(None::) - .fetch_all(&state.db) - .await?; - Ok(Json(rows.into_iter().map(CanvasView::from).collect())) -} - -#[derive(Deserialize)] -pub(crate) struct NameBody { - name: String, -} - -pub(crate) async fn create_canvas( - State(state): State, - user: AuthUser, - Body(body): Body, -) -> Result<(StatusCode, Json), ApiError> { - let name = canvas_name(&body.name)?; - let mut tx = state.db.begin().await?; - let id = insert_canvas(&mut tx, &state, user.user_id, &name, "shared") - .await? - .ok_or_else(|| ApiError::internal("canvas insert returned nothing"))?; - tx.commit().await?; - tracing::info!(user_id = %user.user_id, canvas_id = %id, "canvas created"); - publish(&state, &[user.user_id], id, "created"); - Ok(( - StatusCode::CREATED, - Json(canvas_view(&state, user.user_id, id).await?), - )) -} - -pub(crate) async fn get_canvas( - State(state): State, - user: AuthUser, - Path(raw): Path, -) -> Result, ApiError> { - let a = access(&state, user.user_id, &raw).await?; - Ok(Json(canvas_view(&state, user.user_id, a.canvas_id).await?)) -} - -pub(crate) async fn rename_canvas( - State(state): State, - user: AuthUser, - Path(raw): Path, - Body(body): Body, -) -> Result, ApiError> { - let a = access(&state, user.user_id, &raw).await?; - if a.is_personal() { - return Err(ApiError::bad_request( - "the Personal canvas cannot be renamed", - )); - } - if !a.is_owner() { - return Err(ApiError::Forbidden); - } - let name = canvas_name(&body.name)?; - sqlx::query("UPDATE canvases SET name = $2, updated_at = now() WHERE id = $1") - .bind(a.canvas_id) - .bind(&name) - .execute(&state.db) - .await?; - // Keep the document's `meta.name` in step (persisted + broadcast like any update). - let room = rooms().get(a.canvas_id); - room.transact(&state, SERVER_ORIGIN, |txn| { - let meta = txn.get_or_insert_map(META); - meta.try_update(txn, "name", name.as_str()); - }) - .await?; - publish_members(&state, a.canvas_id, "renamed").await; - Ok(Json(canvas_view(&state, user.user_id, a.canvas_id).await?)) -} - -pub(crate) async fn delete_canvas( - State(state): State, - user: AuthUser, - Path(raw): Path, -) -> Result, ApiError> { - let a = access(&state, user.user_id, &raw).await?; - if a.is_personal() { - return Err(ApiError::bad_request( - "the Personal canvas cannot be deleted", - )); - } - if !a.is_owner() { - return Err(ApiError::Forbidden); - } - let members = member_ids(&state, a.canvas_id).await?; - let deleted = sqlx::query("DELETE FROM canvases WHERE id = $1 AND owner_id = $2") - .bind(a.canvas_id) - .bind(user.user_id) - .execute(&state.db) - .await? - .rows_affected(); - if deleted == 0 { - return Err(ApiError::NotFound); - } - rooms().close(a.canvas_id); - tracing::info!(user_id = %user.user_id, canvas_id = %a.canvas_id, "canvas deleted"); - publish(&state, &members, a.canvas_id, "deleted"); - Ok(ok()) -} - -// --------------------------------------------------------------------------------------------- -// Members - -pub(crate) async fn list_members( - State(state): State, - user: AuthUser, - Path(raw): Path, -) -> Result>, ApiError> { - let a = access(&state, user.user_id, &raw).await?; - let rows = sqlx::query_as::<_, MemberView>( - "SELECT m.user_id, u.email, u.display_name, m.role, m.added_at - FROM canvas_members m JOIN users u ON u.id = m.user_id - WHERE m.canvas_id = $1 - ORDER BY (m.role = 'owner') DESC, m.added_at, m.user_id", - ) - .bind(a.canvas_id) - .fetch_all(&state.db) - .await?; - Ok(Json(rows)) -} - -pub(crate) async fn remove_member( - State(state): State, - user: AuthUser, - Path((raw, target)): Path<(String, String)>, -) -> Result, ApiError> { - let a = access(&state, user.user_id, &raw).await?; - let target = Uuid::parse_str(&target).map_err(|_| ApiError::NotFound)?; - if a.is_personal() { - return Err(ApiError::bad_request("the Personal canvas has no members")); - } - if target == user.user_id { - if a.is_owner() { - return Err(ApiError::bad_request( - "the owner cannot leave a canvas; delete it instead", - )); - } - } else if !a.is_owner() { - return Err(ApiError::Forbidden); - } - let removed = sqlx::query( - "DELETE FROM canvas_members WHERE canvas_id = $1 AND user_id = $2 AND role <> 'owner'", - ) - .bind(a.canvas_id) - .bind(target) - .execute(&state.db) - .await? - .rows_affected(); - if removed == 0 { - return Err(ApiError::NotFound); - } - rooms().kick_user(a.canvas_id, target); - publish(&state, &[target], a.canvas_id, "member_removed"); - publish_members(&state, a.canvas_id, "member_removed").await; - Ok(ok()) -} - -// --------------------------------------------------------------------------------------------- -// Invites - -#[derive(Deserialize)] -pub(crate) struct InviteBody { - email: String, -} - -async fn invite_view(state: &AppState, id: Uuid) -> Result { - sqlx::query_as::<_, InviteRow>(&format!("{INVITE_VIEW_SQL} WHERE i.id = $1")) - .bind(id) - .fetch_optional(&state.db) - .await? - .map(InviteView::from) - .ok_or(ApiError::NotFound) -} - -pub(crate) async fn create_invite( - State(state): State, - user: AuthUser, - Path(raw): Path, - Body(body): Body, -) -> Result<(StatusCode, Json), ApiError> { - let a = access(&state, user.user_id, &raw).await?; - if a.is_personal() { - return Err(ApiError::bad_request( - "the Personal canvas cannot be shared", - )); - } - let email = invite_email(&body.email)?; - if email == user.email.trim().to_lowercase() { - return Err(ApiError::bad_request( - "you are already a member of this canvas", - )); - } - let already_member: bool = sqlx::query_scalar( - "SELECT EXISTS (SELECT 1 FROM canvas_members m JOIN users u ON u.id = m.user_id - WHERE m.canvas_id = $1 AND lower(u.email) = $2)", - ) - .bind(a.canvas_id) - .bind(&email) - .fetch_one(&state.db) - .await?; - if already_member { - return Err(ApiError::Conflict( - json!({ "message": "that user is already a member of this canvas" }), - )); - } - let token = ids::sha256_hex(ids::random_token().as_bytes()); - let created: Option = sqlx::query_scalar( - "INSERT INTO canvas_invites (id, canvas_id, email, invited_by, token) - VALUES ($1, $2, $3, $4, $5) - ON CONFLICT (canvas_id, email) WHERE status = 'pending' DO NOTHING - RETURNING id", - ) - .bind(ids::uuid_v7()) - .bind(a.canvas_id) - .bind(&email) - .bind(user.user_id) - .bind(&token) - .fetch_optional(&state.db) - .await?; - let invitee = user_id_by_email(&state.db, &email).await?; - let (status, id, nudged) = if let Some(id) = created { - tracing::info!(user_id = %user.user_id, canvas_id = %a.canvas_id, invite_id = %id, "canvas invite created"); - if let Some(invitee) = invitee { - publish(&state, &[invitee], a.canvas_id, "invited"); - } - (StatusCode::CREATED, id, false) - } else { - let (id, nudged) = nudge_invite(&state, a.canvas_id, &email, invitee.is_some()).await?; - if nudged { - tracing::info!(user_id = %user.user_id, canvas_id = %a.canvas_id, invite_id = %id, "canvas invite re-sent"); - if let Some(invitee) = invitee { - publish(&state, &[invitee], a.canvas_id, "invited"); - } - } - (StatusCode::OK, id, nudged) - }; - let invite = invite_view(&state, id).await?; - Ok((status, Json(InviteSent { invite, nudged }))) -} - -/// The pending invite of `email` to `canvas_id`, re-sent: when the invitee has an account and -/// the last reminder (or the invite itself) is older than [`INVITE_NUDGE_INTERVAL_SECS`], -/// stamps `nudged_at` and returns `true` (the caller notifies the invitee). The conditional -/// `UPDATE` makes concurrent re-sends remind at most once per interval. -async fn nudge_invite( - state: &AppState, - canvas_id: Uuid, - email: &str, - has_account: bool, -) -> Result<(Uuid, bool), ApiError> { - if has_account { - let nudged: Option = sqlx::query_scalar( - "UPDATE canvas_invites SET nudged_at = now() - WHERE canvas_id = $1 AND email = $2 AND status = 'pending' - AND COALESCE(nudged_at, created_at) <= now() - make_interval(secs => $3) - RETURNING id", - ) - .bind(canvas_id) - .bind(email) - .bind(f64::from(INVITE_NUDGE_INTERVAL_SECS)) - .fetch_optional(&state.db) - .await?; - if let Some(id) = nudged { - return Ok((id, true)); - } - } - // Throttled or no account yet. (An invite accepted, declined or revoked in between makes - // this a 404.) - let id: Uuid = sqlx::query_scalar( - "SELECT id FROM canvas_invites - WHERE canvas_id = $1 AND email = $2 AND status = 'pending'", - ) - .bind(canvas_id) - .bind(email) - .fetch_one(&state.db) - .await?; - Ok((id, false)) -} - -/// `DELETE /canvases/:id/invites/:invite_id` — the canvas owner or the original inviter -/// withdraws a pending invite (`204`). Other members get `403`; an unknown invite, one of -/// another canvas, or one that is no longer pending is `404`. -pub(crate) async fn revoke_invite( - State(state): State, - user: AuthUser, - Path((raw, invite)): Path<(String, String)>, -) -> Result { - let a = access(&state, user.user_id, &raw).await?; - let invite_id = Uuid::parse_str(&invite).map_err(|_| ApiError::NotFound)?; - let row: Option<(Option, String)> = sqlx::query_as( - "SELECT invited_by, email FROM canvas_invites - WHERE id = $1 AND canvas_id = $2 AND status = 'pending'", - ) - .bind(invite_id) - .bind(a.canvas_id) - .fetch_optional(&state.db) - .await?; - let (inviter, email) = row.ok_or(ApiError::NotFound)?; - if !a.is_owner() && inviter != Some(user.user_id) { - return Err(ApiError::Forbidden); - } - let revoked = sqlx::query( - "UPDATE canvas_invites SET status = 'revoked' - WHERE id = $1 AND canvas_id = $2 AND status = 'pending'", - ) - .bind(invite_id) - .bind(a.canvas_id) - .execute(&state.db) - .await? - .rows_affected(); - if revoked == 0 { - // Accepted, declined or revoked between the two statements. - return Err(ApiError::NotFound); - } - tracing::info!(user_id = %user.user_id, canvas_id = %a.canvas_id, %invite_id, "canvas invite revoked"); - if let Some(invitee) = user_id_by_email(&state.db, &email).await? { - publish(&state, &[invitee], a.canvas_id, "invite_revoked"); - } - Ok(StatusCode::NO_CONTENT) -} - -pub(crate) async fn list_canvas_invites( - State(state): State, - user: AuthUser, - Path(raw): Path, -) -> Result>, ApiError> { - let a = access(&state, user.user_id, &raw).await?; - let rows = sqlx::query_as::<_, InviteRow>(&format!( - "{INVITE_VIEW_SQL} WHERE i.canvas_id = $1 AND i.status = 'pending' ORDER BY i.created_at DESC" - )) - .bind(a.canvas_id) - .fetch_all(&state.db) - .await?; - Ok(Json(rows.into_iter().map(InviteView::from).collect())) -} - -pub(crate) async fn my_invites( - State(state): State, - user: AuthUser, -) -> Result>, ApiError> { - let rows = sqlx::query_as::<_, InviteRow>(&format!( - "{INVITE_VIEW_SQL} WHERE i.email = lower($1) AND i.status = 'pending' - ORDER BY i.created_at DESC" - )) - .bind(user.email.trim()) - .fetch_all(&state.db) - .await?; - Ok(Json(rows.into_iter().map(InviteView::from).collect())) -} - -pub(crate) async fn accept_invite( - State(state): State, - user: AuthUser, - Path(raw): Path, -) -> Result, ApiError> { - let id = Uuid::parse_str(&raw).map_err(|_| ApiError::NotFound)?; - let mut tx = state.db.begin().await?; - let canvas_id: Uuid = sqlx::query_scalar( - "SELECT canvas_id FROM canvas_invites - WHERE id = $1 AND email = lower($2) AND status = 'pending' - FOR UPDATE", - ) - .bind(id) - .bind(user.email.trim()) - .fetch_optional(&mut *tx) - .await? - .ok_or(ApiError::NotFound)?; - sqlx::query( - "INSERT INTO canvas_members (canvas_id, user_id, role) VALUES ($1, $2, 'editor') - ON CONFLICT (canvas_id, user_id) DO NOTHING", - ) - .bind(canvas_id) - .bind(user.user_id) - .execute(&mut *tx) - .await?; - sqlx::query("UPDATE canvas_invites SET status = 'accepted', accepted_at = now() WHERE id = $1") - .bind(id) - .execute(&mut *tx) - .await?; - tx.commit().await?; - tracing::info!(user_id = %user.user_id, %canvas_id, invite_id = %id, "canvas invite accepted"); - publish_members(&state, canvas_id, "member_added").await; - Ok(Json(canvas_view(&state, user.user_id, canvas_id).await?)) -} - -pub(crate) async fn decline_invite( - State(state): State, - user: AuthUser, - Path(raw): Path, -) -> Result, ApiError> { - let id = Uuid::parse_str(&raw).map_err(|_| ApiError::NotFound)?; - let row: Option<(Uuid, Option)> = sqlx::query_as( - "UPDATE canvas_invites SET status = 'declined' - WHERE id = $1 AND email = lower($2) AND status = 'pending' - RETURNING canvas_id, invited_by", - ) - .bind(id) - .bind(user.email.trim()) - .fetch_optional(&state.db) - .await?; - let (canvas_id, inviter) = row.ok_or(ApiError::NotFound)?; - if let Some(inviter) = inviter { - publish(&state, &[inviter], canvas_id, "invite_declined"); - } - Ok(ok()) -} - -// --------------------------------------------------------------------------------------------- -// Share links - -#[derive(Debug, Default, Deserialize)] -pub(crate) struct ShareLinkBody { - role: Option, -} - -pub(crate) async fn create_share_link( - State(state): State, - user: AuthUser, - Path(raw): Path, - OptionalJsonBody(body): OptionalJsonBody, -) -> Result<(StatusCode, Json), ApiError> { - let access = access(&state, user.user_id, &raw).await?; - if access.is_personal() { - return Err(ApiError::bad_request( - "the Personal canvas cannot be shared", - )); - } - match body.role.as_deref() { - None | Some("editor") => {} - Some(_) => return Err(ApiError::bad_request("share link role must be editor")), - } - - let token = ids::random_token(); - let digest = ids::sha256(token.as_bytes()); - let (id, role, created_at): (Uuid, String, OffsetDateTime) = sqlx::query_as( - "INSERT INTO canvas_share_links (id, canvas_id, token_sha256, role, created_by) - VALUES ($1, $2, $3, 'editor', $4) - RETURNING id, role, created_at", - ) - .bind(ids::uuid_v7()) - .bind(access.canvas_id) - .bind(&digest[..]) - .bind(user.user_id) - .fetch_one(&state.db) - .await?; - tracing::info!(user_id = %user.user_id, canvas_id = %access.canvas_id, link_id = %id, "canvas share link created"); - Ok(( - StatusCode::CREATED, - Json(CreatedShareLink { - id, - token, - canvas_id: access.canvas_id, - role, - created_at, - }), - )) -} - -pub(crate) async fn list_share_links( - State(state): State, - user: AuthUser, - Path(raw): Path, -) -> Result, ApiError> { - let access = access(&state, user.user_id, &raw).await?; - if !access.is_owner() { - return Err(ApiError::Forbidden); - } - let links = sqlx::query_as::<_, ShareLinkView>( - "SELECT id, role, created_by, created_at, uses - FROM canvas_share_links WHERE canvas_id = $1 - ORDER BY created_at DESC, id DESC", - ) - .bind(access.canvas_id) - .fetch_all(&state.db) - .await?; - Ok(Json(json!({ "links": links }))) -} - -pub(crate) async fn revoke_share_link( - State(state): State, - user: AuthUser, - Path((raw, link_id)): Path<(String, String)>, -) -> Result { - let access = access(&state, user.user_id, &raw).await?; - if !access.is_owner() { - return Err(ApiError::Forbidden); - } - let link_id = Uuid::parse_str(&link_id).map_err(|_| ApiError::NotFound)?; - let deleted = sqlx::query("DELETE FROM canvas_share_links WHERE id = $1 AND canvas_id = $2") - .bind(link_id) - .bind(access.canvas_id) - .execute(&state.db) - .await? - .rows_affected(); - if deleted == 0 { - return Err(ApiError::NotFound); - } - Ok(StatusCode::NO_CONTENT) -} - -pub(crate) async fn revoke_all_share_links( - State(state): State, - user: AuthUser, - Path(raw): Path, -) -> Result { - let access = access(&state, user.user_id, &raw).await?; - if !access.is_owner() { - return Err(ApiError::Forbidden); - } - sqlx::query("DELETE FROM canvas_share_links WHERE canvas_id = $1") - .bind(access.canvas_id) - .execute(&state.db) - .await?; - Ok(StatusCode::NO_CONTENT) -} - -async fn share_link_preview_row( - state: &AppState, - token: &str, -) -> Result { - let digest = token_digest(token).ok_or_else(link_not_found)?; - sqlx::query_as::<_, ShareLinkPreviewRow>( - "SELECT l.canvas_id, c.name, c.owner_id, u.display_name AS owner_display_name, l.role - FROM canvas_share_links l - JOIN canvases c ON c.id = l.canvas_id - JOIN users u ON u.id = c.owner_id - WHERE l.token_sha256 = $1", - ) - .bind(&digest[..]) - .fetch_optional(&state.db) - .await? - .ok_or_else(link_not_found) -} - -pub(crate) async fn preview_share_link( - State(state): State, - user: AuthUser, - Path(token): Path, -) -> Result, ApiError> { - let row = share_link_preview_row(&state, &token).await?; - let member: bool = sqlx::query_scalar( - "SELECT EXISTS (SELECT 1 FROM canvas_members WHERE canvas_id = $1 AND user_id = $2)", - ) - .bind(row.canvas_id) - .bind(user.user_id) - .fetch_one(&state.db) - .await?; - Ok(Json(json!({ - "canvas_id": row.canvas_id, - "name": row.name, - "owner": { "id": row.owner_id, "display_name": row.owner_display_name }, - "role": row.role, - "member": member, - }))) -} - -pub(crate) async fn join_share_link( - State(state): State, - user: AuthUser, - Path(token): Path, -) -> Result, ApiError> { - let digest = token_digest(&token).ok_or_else(link_not_found)?; - let mut tx = state.db.begin().await?; - let row: Option<(Uuid, String, String)> = sqlx::query_as( - "SELECT l.canvas_id, l.role, c.kind FROM canvas_share_links l - JOIN canvases c ON c.id = l.canvas_id - WHERE l.token_sha256 = $1 FOR UPDATE", - ) - .bind(&digest[..]) - .fetch_optional(&mut *tx) - .await?; - let Some((canvas_id, role, kind)) = row else { - return Err(link_not_found()); - }; - if kind == "personal" { - return Err(link_not_found()); - } - let added: Option = sqlx::query_scalar( - "INSERT INTO canvas_members (canvas_id, user_id, role) VALUES ($1, $2, $3) - ON CONFLICT (canvas_id, user_id) DO NOTHING RETURNING user_id", - ) - .bind(canvas_id) - .bind(user.user_id) - .bind(&role) - .fetch_optional(&mut *tx) - .await?; - sqlx::query( - "UPDATE canvas_invites SET status = 'accepted', accepted_at = COALESCE(accepted_at, now()) - WHERE canvas_id = $1 AND lower(email) = lower($2) AND status = 'pending'", - ) - .bind(canvas_id) - .bind(user.email.trim()) - .execute(&mut *tx) - .await?; - sqlx::query("UPDATE canvas_share_links SET uses = uses + 1 WHERE token_sha256 = $1") - .bind(&digest[..]) - .execute(&mut *tx) - .await?; - tx.commit().await?; - publish_members(&state, canvas_id, "member_added").await; - tracing::info!(user_id = %user.user_id, %canvas_id, joined = added.is_some(), "canvas share link joined"); - Ok(Json(canvas_view(&state, user.user_id, canvas_id).await?)) -} - -// --------------------------------------------------------------------------------------------- -// People - -#[derive(Debug, Default, Deserialize)] -pub(crate) struct PeopleQuery { - q: Option, - limit: Option, -} - -pub(crate) async fn list_people( - State(state): State, - user: AuthUser, - QueryParams(q): QueryParams, -) -> Result, ApiError> { - let limit = q.limit.unwrap_or(20); - if limit < 1 { - return Err(ApiError::bad_request("limit must be at least 1")); - } - let limit = limit.min(50); - let needle = q.q.unwrap_or_default(); - let people = sqlx::query_as::<_, PersonView>( - "SELECT id, display_name, email FROM users - WHERE NOT disabled AND id <> $1 - AND ($2 = '' OR strpos(lower(display_name), lower($2)) > 0 - OR strpos(lower(email), lower($2)) > 0) - ORDER BY lower(display_name), id - LIMIT $3", - ) - .bind(user.user_id) - .bind(needle) - .bind(limit) - .fetch_all(&state.db) - .await?; - Ok(Json(json!({ "people": people }))) -} - -// --------------------------------------------------------------------------------------------- -// Documents - -/// `GET /canvases/:id/ws` — upgrade to a y-websocket room. -pub(crate) async fn ws_upgrade( - State(state): State, - user: AuthUser, - Path(raw): Path, - ws: Result, -) -> Result { - let a = access(&state, user.user_id, &raw).await?; - let ws = ws.map_err(|_| ApiError::bad_request("expected a WebSocket upgrade"))?; - let room = rooms().get(a.canvas_id); - room.ensure_loaded(&state).await?; - let user_id = user.user_id; - Ok(ws - .max_message_size(MAX_MESSAGE_BYTES) - .max_frame_size(MAX_MESSAGE_BYTES) - .on_upgrade(move |socket| serve_peer(state, room, user_id, socket)) - .into_response()) -} - -#[derive(Deserialize)] -pub(crate) struct StateQuery { - /// Optional base64 state vector: return only what the caller is missing. - sv: Option, -} - -pub(crate) async fn get_state( - State(state): State, - user: AuthUser, - Path(raw): Path, - Query(q): Query, -) -> Result, ApiError> { - let a = access(&state, user.user_id, &raw).await?; - let sv = match q.sv.as_deref().filter(|s| !s.is_empty()) { - Some(s) => { - let bytes = - ids::b64_decode_any(s).map_err(|_| ApiError::bad_request("sv is not base64"))?; - StateVector::decode_v1(&bytes) - .map_err(|_| ApiError::bad_request("sv is not a state vector"))? - } - None => StateVector::default(), - }; - let room = rooms().get(a.canvas_id); - let g = room.read(&state).await?; - let update = g.doc().transact().encode_state_as_update_v1(&sv); - Ok(Json(json!({ "state": STANDARD.encode(update) }))) -} - -#[derive(Deserialize)] -pub(crate) struct ReadQuery { - #[serde(default)] - full: bool, - /// Comma-separated shape ids. - ids: Option, - /// Comma-separated shape types. - types: Option, -} - -fn csv(s: Option<&str>) -> Option> { - s.map(|s| { - s.split(',') - .map(|p| crate::shape::bare_id(p.trim()).to_owned()) - .filter(|p| !p.is_empty()) - .collect() - }) -} - -pub(crate) async fn read_canvas( - State(state): State, - user: AuthUser, - Path(raw): Path, - Query(q): Query, -) -> Result, ApiError> { - let a = access(&state, user.user_id, &raw).await?; - let room = rooms().get(a.canvas_id); - let g = room.read(&state).await?; - let opts = ReadOpts { - full: q.full, - ids: csv(q.ids.as_deref()), - types: csv(q.types.as_deref()), - now_ms: 0, - }; - let txn = g.doc().transact(); - let mut out = read_canvas_txn(&txn, &opts); - out.canvas = Some(CanvasRef { - id: a.canvas_id.to_string(), - name: a.name, - kind: a.kind, - }); - // The most recent chat messages (0.6.0), so agents can read the conversation too. - let chat = read_chat(&txn, READ_CHAT_LIMIT, q.full); - Ok(Json(ReadWithChat { read: out, chat })) -} - -/// `POST /canvases/:id/ops` — `{ops:[…], as?:{id,name,color}|"name", confirm?, near?:{x,y}}` -/// or a bare array of ops. Envelope problems are reported like the page does: `200` with -/// `errors:[{index:-1,…}]`. -pub(crate) async fn post_ops( - State(state): State, - user: AuthUser, - Path(raw): Path, - Body(body): Body, -) -> Result, ApiError> { - let a = access(&state, user.user_id, &raw).await?; - let (ops, actor, confirm, near) = match body { - Value::Array(ops) => (ops, None, false, None), - Value::Object(mut o) => { - let Some(Value::Array(ops)) = o.remove("ops") else { - return Ok(Json(OpsResult::request_error("`ops` must be an array"))); - }; - let actor = match Actor::parse(o.get("as")) { - Ok(actor) => actor, - Err(e) => return Ok(Json(OpsResult::request_error(e))), - }; - let near = o.get("near").and_then(|n| { - Some(Point { - x: n.get("x")?.as_f64()?, - y: n.get("y")?.as_f64()?, - }) - }); - ( - ops, - actor, - o.get("confirm") == Some(&Value::Bool(true)), - near, - ) - } - _ => { - return Ok(Json(OpsResult::request_error( - "expected {ops:[...], as?:{id,name,color}} or an array of ops", - ))) - } - }; - let by = match &actor { - Some(actor) => actor.name.clone(), - None => display_name(&state, &user).await?, - }; - let mut ctx = OpsCtx::new(by); - // Checklist picks say who made them: the agent named in `as`, else the caller. - ctx.by_id = actor - .as_ref() - .map_or_else(|| user.user_id.to_string(), |a| a.id.clone()); - ctx.confirm_clear = confirm; - if let Some(p) = near { - ctx.origin = p; - } - - let room = rooms().get(a.canvas_id); - let (result, wrote) = room - .transact(&state, SERVER_ORIGIN, |txn| { - let shapes = txn.get_or_insert_map(SHAPES); - let result = apply_json_ops(txn, &shapes, &ops, &ctx); - let wrote = match &actor { - Some(actor) if result.applied > 0 => { - let agents = txn.get_or_insert_map(AGENTS); - write_agent( - txn, - &agents, - &actor.id, - &actor.writing(result.anchor), - ctx.now_ms, - ); - Some(ctx.now_ms) - } - _ => None, - }; - (result, wrote) - }) - .await?; - if let (Some(actor), Some(at)) = (actor, wrote) { - schedule_idle(Arc::clone(&state), a.canvas_id, actor.id, at); - } - tracing::debug!( - user_id = %user.user_id, - canvas_id = %a.canvas_id, - applied = result.applied, - errors = result.errors.len(), - "canvas ops applied" - ); - Ok(Json(result)) -} - -/// After [`WRITING_MS`], flips the agent back to `idle` unless it wrote again since. -fn schedule_idle(state: SharedState, canvas_id: Uuid, agent_id: String, written_at: i64) { - tokio::spawn(async move { - tokio::time::sleep(std::time::Duration::from_millis(WRITING_MS)).await; - let Some(room) = rooms().peek(canvas_id) else { - return; - }; - let res = room - .transact(&state, SERVER_ORIGIN, |txn| { - let agents = txn.get_or_insert_map(AGENTS); - let current = agents - .get(txn, &agent_id) - .map(|o| out_to_json(txn, &o)) - .and_then(|v| read_agent(&v)); - #[allow(clippy::cast_precision_loss)] - let still = current.is_some_and(|c| { - c.status == "writing" && (c.updated_at - written_at as f64).abs() < 0.5 - }); - if still { - let idle = AgentPatch { - status: Some("idle"), - ..AgentPatch::default() - }; - write_agent(txn, &agents, &agent_id, &idle, crate::schema::now_ms()); - } - }) - .await; - if let Err(e) = res { - tracing::debug!(%canvas_id, error = %e, "agent idle write skipped"); - } - }); -} - -// --------------------------------------------------------------------------------------------- -// Ungated web landing page - -fn escape_html(value: &str) -> String { - value - .replace('&', "&") - .replace('<', "<") - .replace('>', ">") - .replace('"', """) - .replace('\'', "'") -} - -fn percent_encode_component(value: &str) -> String { - let mut out = String::with_capacity(value.len()); - for byte in value.bytes() { - if byte.is_ascii_alphanumeric() || matches!(byte, b'-' | b'.' | b'_' | b'~') { - out.push(char::from(byte)); - } else { - use std::fmt::Write as _; - let _ = write!(out, "%{byte:02X}"); - } - } - out -} - -/// Accept only a host name/IP and an optional numeric port. In particular, reject user-info, -/// paths and control characters before putting the value in a custom URL scheme. -fn validated_host(headers: &HeaderMap, uri: &Uri) -> Option { - // HTTP/1.1 carries the target in a literal `Host` header; HTTP/2 (our - // only transport once TLS is on, since rustls negotiates h2 by ALPN) - // forbids that header and sends `:authority` instead, which hyper/h2 - // surface as the request URI's authority, not a header — so a host - // taken from headers alone is always absent on this listener. - let from_header = headers - .get(header::HOST) - .and_then(|v| v.to_str().ok()) - .map(str::to_owned); - let from_authority = uri.authority().map(|a| a.as_str().to_owned()); - let raw = from_header.or(from_authority)?; - let raw = raw.as_str(); - if raw.is_empty() - || raw.trim() != raw - || raw.chars().any(|c| c.is_control() || c.is_whitespace()) - { - return None; - } - if raw.contains(['@', '/', '?', '#', '%']) { - return None; - } - let host = if let Some(rest) = raw.strip_prefix('[') { - let end = rest.find(']')?; - let host = &rest[..end]; - let suffix = &rest[end + 1..]; - if suffix.is_empty() { - host - } else { - let port = suffix.strip_prefix(':').filter(|p| !p.is_empty())?; - if port.parse::().is_err() { - return None; - } - host - } - } else { - if raw.matches(':').count() > 1 { - return None; - } - match raw.rsplit_once(':') { - Some((host, port)) if !port.is_empty() => { - if port.parse::().is_err() { - return None; - } - host - } - Some((_, _)) => return None, - None => raw, - } - }; - if host.is_empty() - || host.parse::().is_err() && url::Host::parse(host).is_err() - { - return None; - } - Some(raw.to_owned()) -} - -/// `GET /join/:token` deliberately does not consult the database. It is outside `/v1`, so it is -/// reachable before a client has an instance key and can hand the opaque token to Copper. -pub(crate) async fn join_landing( - Path(token): Path, - uri: Uri, - headers: HeaderMap, -) -> Response { - let href = token_digest(&token).and_then(|_| { - validated_host(&headers, &uri).map(|host| { - format!( - "copper://canvas/join/{}?cloud={}", - percent_encode_component(&token), - host - ) - }) - }); - let html = match href { - Some(href) => format!( - "Open canvas in Copper

Canvas share link

Open this canvas in Copper to continue.

Open this canvas in Copper
", - escape_html(&href) - ), - None => "Open canvas in Copper

Canvas share link

Open this link in Copper to continue.

".to_owned(), - }; - let mut response = Html(html).into_response(); - let headers = response.headers_mut(); - headers.insert(header::CACHE_CONTROL, HeaderValue::from_static("no-store")); - headers.insert( - header::REFERRER_POLICY, - HeaderValue::from_static("no-referrer"), - ); - headers.insert( - header::X_CONTENT_TYPE_OPTIONS, - HeaderValue::from_static("nosniff"), - ); - headers.insert( - header::CONTENT_SECURITY_POLICY, - HeaderValue::from_static("default-src 'none'; style-src 'unsafe-inline'"), - ); - response -} - -#[cfg(test)] -mod tests { - use super::*; - - #[test] - fn host_falls_back_to_authority_when_no_host_header() { - // HTTP/2 requests never carry a literal `Host` header (RFC 9113 §8.3.1 - // forbids it); hyper/h2 surface the `:authority` pseudo-header as the - // request URI's authority instead. The join landing page is only ever - // served over h2 once TLS is on, so this path must work from the URI - // alone, not just from headers. - let empty = HeaderMap::new(); - let uri: Uri = "https://127.0.0.1:8445/join/abc".parse().unwrap(); - assert_eq!( - validated_host(&empty, &uri).as_deref(), - Some("127.0.0.1:8445") - ); - - // A literal Host header (HTTP/1.1) still wins when present. - let mut headers = HeaderMap::new(); - headers.insert(header::HOST, HeaderValue::from_static("example.com")); - assert_eq!( - validated_host(&headers, &uri).as_deref(), - Some("example.com") - ); - - // Neither a header nor a URI authority: no host to offer. - let relative: Uri = "/join/abc".parse().unwrap(); - assert_eq!(validated_host(&empty, &relative), None); - } - - #[test] - fn names() { - assert_eq!(canvas_name(" Plan \u{7}A ").unwrap(), "Plan A"); - assert!(canvas_name(" ").is_err()); - assert!(canvas_name(&"x".repeat(201)).is_err()); - assert_eq!(canvas_name(&"é".repeat(200)).unwrap().chars().count(), 200); - } - - #[test] - fn emails() { - assert_eq!( - invite_email(" Bob@Example.COM ").unwrap(), - "bob@example.com" - ); - for bad in ["", "bob", "bob@", "@x.io", "a b@c.io", "a@b", "a@.b.io"] { - assert!(invite_email(bad).is_err(), "{bad}"); - } - } - - #[test] - fn initial_update_sets_meta() { - let id = Uuid::now_v7(); - let update = initial_update("Board", id); - let doc = Doc::new(); - doc.transact_mut() - .apply_update(yrs::Update::decode_v1(&update).unwrap()) - .unwrap(); - let txn = doc.transact(); - assert_eq!(crate::read::meta_name(&txn).as_deref(), Some("Board")); - } -} diff --git a/crates/canvas/src/room.rs b/crates/canvas/src/room.rs deleted file mode 100644 index ca29ca3..0000000 --- a/crates/canvas/src/room.rs +++ /dev/null @@ -1,649 +0,0 @@ -//! In-memory canvas rooms: one `yrs::Doc` + awareness + broadcast channel per open canvas. -//! -//! Write path (WebSocket updates, REST ops, server-side meta edits): take the room's write -//! lock → apply in one `yrs` transaction → seal + INSERT the produced update → broadcast it → -//! compact every [`COMPACT_EVERY`] updates. The lock is held across the INSERT, so persisted -//! order == application order == broadcast order, and nobody ever observes an update that is -//! not durable. If persisting fails the in-memory doc is dropped (the room reloads from the -//! database on next use) and every peer is disconnected; y-websocket clients reconnect and -//! re-send whatever the server is missing during the sync handshake. -//! -//! Rooms with no peers and no REST activity for [`IDLE_EVICT`] are dropped from memory. - -use std::collections::{HashMap, HashSet}; -use std::panic::AssertUnwindSafe; -use std::sync::atomic::{AtomicBool, AtomicU64, AtomicUsize, Ordering}; -use std::sync::{Arc, LazyLock, Mutex as StdMutex, MutexGuard, PoisonError, Weak}; -use std::time::{Duration, Instant}; - -use bytes::Bytes; -use copper_cloud_core::error::ApiError; -use copper_cloud_core::events::Event; -use copper_cloud_core::state::{AppState, SharedState}; -use serde::Serialize; -use tokio::sync::{broadcast, RwLock, RwLockReadGuard, RwLockWriteGuard}; -use uuid::Uuid; -use yrs::sync::{Awareness, AwarenessUpdate, Message, SyncMessage}; -use yrs::updates::decoder::Decode; -use yrs::updates::encoder::Encode; -use yrs::{ClientID, Doc, ReadTxn, StateVector, Transact, TransactionMut, Update}; - -use crate::store; -use crate::ws::close; - -/// Rooms idle (no peers, no REST use) this long are evicted from memory. -pub const IDLE_EVICT: Duration = Duration::from_secs(60); -/// Compact the update log into a snapshot after this many updates. -pub const COMPACT_EVERY: usize = 200; -/// Broadcast frames buffered per room; a peer that falls further behind is disconnected. -pub const BROADCAST_CAPACITY: usize = 256; -/// `Event::Canvas { kind: "update" }` is published at most this often per room. -pub const EVENT_THROTTLE: Duration = Duration::from_secs(2); - -/// Awareness client ids one connection may publish (y-websocket uses one per document). -pub const MAX_AWARENESS_CLIENTS_PER_PEER: usize = 16; - -/// Origin id used for server-side writes (REST ops, renames); never equals a peer id. -pub(crate) const SERVER_ORIGIN: u64 = 0; - -static NEXT_PEER: AtomicU64 = AtomicU64::new(1); - -/// A fresh, process-unique peer id. -pub(crate) fn next_peer_id() -> u64 { - NEXT_PEER.fetch_add(1, Ordering::Relaxed) -} - -fn lock(m: &StdMutex) -> MutexGuard<'_, T> { - m.lock().unwrap_or_else(PoisonError::into_inner) -} - -/// One frame fanned out to every peer of a room. -#[derive(Clone, Debug)] -pub(crate) enum Frame { - /// An encoded y-websocket message. Peers skip document frames they - /// originated; awareness frames are echoed back to their origin as well, - /// like the reference y-websocket server does, so a lone client hears - /// something at least every 15 s (its own presence renewal) and never - /// trips the provider's 30 s silence timeout. - Data { - origin: u64, - bytes: Bytes, - echo: bool, - }, - /// Disconnect `user`'s peers (or everybody when `None`) with a close code. - Kick { - user: Option, - code: u16, - reason: &'static str, - }, -} - -/// The loaded document of a room. -pub(crate) struct DocState { - doc: Doc, - key: [u8; 32], - /// Updates produced by committed transactions (filled by `observe_update_v1`). - produced: Arc>>>, - since_snapshot: usize, - last_seq: i64, -} - -impl DocState { - async fn load(state: &AppState, canvas_id: Uuid) -> Result { - let stored = store::load(&state.db, &state.crypto, canvas_id).await?; - let doc = Doc::new(); - let mut last_seq = stored.snapshot.as_ref().map_or(0, |s| s.0); - { - let mut txn = doc.transact_mut(); - let blobs = stored - .snapshot - .iter() - .map(|(seq, b)| (*seq, b)) - .chain(stored.updates.iter().map(|(seq, b)| (*seq, b))); - for (seq, blob) in blobs { - last_seq = last_seq.max(seq); - let applied = Update::decode_v1(blob) - .map_err(|e| e.to_string()) - .and_then(|u| txn.apply_update(u).map_err(|e| e.to_string())); - if let Err(error) = applied { - tracing::error!(%canvas_id, seq, error, "skipping undecodable stored canvas update"); - } - } - } - let produced: Arc>>> = Arc::default(); - let sink = Arc::clone(&produced); - doc.observe_update_v1("copper-cloud-persist", move |_, e| { - lock(&sink).push(e.update.clone()); - }) - .map_err(|e| ApiError::internal(format!("observing canvas doc: {e}")))?; - tracing::debug!(%canvas_id, updates = stored.updates.len(), "canvas room loaded"); - Ok(Self { - doc, - key: stored.key, - produced, - since_snapshot: stored.updates.len(), - last_seq, - }) - } - - /// The room's document (read-only use). - pub(crate) fn doc(&self) -> &Doc { - &self.doc - } - - fn take_produced(&self) -> Option> { - let mut produced = std::mem::take(&mut *lock(&self.produced)); - match produced.len() { - 0 => None, - 1 => produced.pop(), - _ => match yrs::merge_updates_v1(&produced) { - Ok(merged) => Some(merged), - Err(e) => { - tracing::error!(error = %e, "merging canvas updates failed"); - None - } - }, - } - } -} - -#[derive(Default)] -struct Throttle { - last: Option, - trailing: bool, -} - -/// One open canvas. -pub(crate) struct Room { - pub id: Uuid, - doc: RwLock>, - awareness: StdMutex, - tx: broadcast::Sender, - peers: AtomicUsize, - last_active: StdMutex, - evict_scheduled: AtomicBool, - throttle: StdMutex, - closed: AtomicBool, -} - -impl Room { - fn new(id: Uuid) -> Self { - let (tx, _) = broadcast::channel(BROADCAST_CAPACITY); - Self { - id, - doc: RwLock::new(None), - // Awareness only stores client states here; its doc is never used. - awareness: StdMutex::new(Awareness::new(Doc::new())), - tx, - peers: AtomicUsize::new(0), - last_active: StdMutex::new(Instant::now()), - evict_scheduled: AtomicBool::new(false), - throttle: StdMutex::default(), - closed: AtomicBool::new(false), - } - } - - fn check_open(&self) -> Result<(), ApiError> { - if self.closed.load(Ordering::Acquire) { - Err(ApiError::NotFound) - } else { - Ok(()) - } - } - - /// Write guard over a loaded document (loads it on first use). - async fn loaded_write( - &self, - state: &AppState, - ) -> Result>, ApiError> { - let mut g = self.doc.write().await; - self.check_open()?; - if g.is_none() { - *g = Some(DocState::load(state, self.id).await?); - } - Ok(g) - } - - /// Read guard over a loaded document (loads it on first use). - pub(crate) async fn read( - &self, - state: &AppState, - ) -> Result, ApiError> { - self.touch(); - { - let g = self.doc.read().await; - self.check_open()?; - if let Ok(mapped) = RwLockReadGuard::try_map(g, Option::as_ref) { - return Ok(mapped); - } - } - let g = self.loaded_write(state).await?.downgrade(); - RwLockReadGuard::try_map(g, Option::as_ref) - .map_err(|_| ApiError::internal("canvas room failed to load")) - } - - /// Makes sure the document is loaded (used before a WebSocket upgrade). - pub(crate) async fn ensure_loaded(&self, state: &AppState) -> Result<(), ApiError> { - self.read(state).await.map(drop) - } - - /// Runs `f` in one write transaction, then persists and broadcasts what it produced. - /// - /// `origin` is the peer that caused the change (its own frame is not echoed back). - pub(crate) async fn transact( - self: &Arc, - state: &SharedState, - origin: u64, - f: impl FnOnce(&mut TransactionMut<'_>) -> R, - ) -> Result { - self.touch(); - let mut g = self.loaded_write(state).await?; - let Some(st) = g.as_mut() else { - return Err(ApiError::internal("canvas room not loaded")); - }; - let out = std::panic::catch_unwind(AssertUnwindSafe(|| { - let mut txn = st.doc.transact_mut(); - f(&mut txn) - })); - let Ok(out) = out else { - // The document may be half-applied: forget it and disconnect everyone. - *g = None; - drop(g); - self.kick(None, close::SERVER_ERROR, "server error"); - return Err(ApiError::internal("canvas transaction panicked")); - }; - let Some(update) = st.take_produced() else { - return Ok(out); - }; - match store::append(&state.db, &st.key, self.id, &update).await { - Ok(seq) => { - st.last_seq = seq; - st.since_snapshot += 1; - } - Err(e) => { - *g = None; - drop(g); - self.kick(None, close::SERVER_ERROR, "server error"); - return Err(e); - } - } - let msg = Message::Sync(SyncMessage::Update(update)).encode_v1(); - let _ = self.tx.send(Frame::Data { - origin, - bytes: Bytes::from(msg), - echo: false, - }); - if st.since_snapshot >= COMPACT_EVERY { - let snapshot = st - .doc - .transact() - .encode_state_as_update_v1(&StateVector::default()); - match store::compact(&state.db, &st.key, self.id, st.last_seq, &snapshot).await { - Ok(deleted) => { - tracing::debug!(canvas_id = %self.id, deleted, "canvas compacted"); - st.since_snapshot = 0; - } - Err(e) => { - tracing::warn!(canvas_id = %self.id, error = %e, "canvas compaction failed"); - } - } - } - drop(g); - self.notify_update(state); - Ok(out) - } - - /// Applies a remote (client) lib0-v1 update. - pub(crate) async fn apply_remote( - self: &Arc, - state: &SharedState, - origin: u64, - update: &[u8], - ) -> Result<(), ApiError> { - self.transact(state, origin, |txn| { - let decoded = Update::decode_v1(update).map_err(|e| format!("invalid update: {e}"))?; - txn.apply_update(decoded) - .map_err(|e| format!("update rejected: {e}")) - }) - .await? - .map_err(ApiError::BadRequest) - } - - /// The `SyncStep2` reply to a client's `SyncStep1`. - pub(crate) async fn sync_step2( - &self, - state: &AppState, - sv: &StateVector, - ) -> Result, ApiError> { - let g = self.read(state).await?; - let update = g.doc.transact().encode_state_as_update_v1(sv); - Ok(Message::Sync(SyncMessage::SyncStep2(update)).encode_v1()) - } - - /// Messages sent to a new peer: `SyncStep1` (our state vector) and, if anyone is present, - /// the current awareness states. - pub(crate) async fn greeting(&self, state: &AppState) -> Result, ApiError> { - let sv = self.read(state).await?.doc.transact().state_vector(); - let mut out = vec![Bytes::from( - Message::Sync(SyncMessage::SyncStep1(sv)).encode_v1(), - )]; - if let Some(msg) = self.awareness_message() { - out.push(Bytes::from(msg)); - } - Ok(out) - } - - /// The current awareness states as one message (`None` when empty). - pub(crate) fn awareness_message(&self) -> Option> { - let a = lock(&self.awareness); - let update = a.update().ok()?; - (!update.clients.is_empty()).then(|| Message::Awareness(update).encode_v1()) - } - - /// Applies an awareness update from peer `origin` and relays the changed states. - pub(crate) fn apply_awareness( - &self, - origin: u64, - update: AwarenessUpdate, - controlled: &mut HashSet, - ) -> Result<(), ApiError> { - let new_ids = update - .clients - .keys() - .filter(|id| !controlled.contains(id)) - .count(); - if controlled.len() + new_ids > MAX_AWARENESS_CLIENTS_PER_PEER { - return Err(ApiError::bad_request( - "too many awareness clients on one connection", - )); - } - let msg = { - let mut a = lock(&self.awareness); - let Some(summary) = a - .apply_update_summary(update) - .map_err(|e| ApiError::bad_request(format!("invalid awareness update: {e}")))? - else { - return Ok(()); - }; - for id in summary.added.iter().chain(&summary.updated) { - controlled.insert(*id); - } - for id in &summary.removed { - controlled.remove(id); - } - let changed = summary.all_changes(); - match a.update_with_clients(changed) { - Ok(u) => Message::Awareness(u).encode_v1(), - Err(_) => return Ok(()), - } - }; - let _ = self.tx.send(Frame::Data { - origin, - bytes: Bytes::from(msg), - echo: true, - }); - Ok(()) - } - - /// Registers a peer and returns its broadcast receiver. - pub(crate) fn join(&self) -> broadcast::Receiver { - self.peers.fetch_add(1, Ordering::AcqRel); - metrics::gauge!("canvas_peers").increment(1.0); - self.touch(); - self.tx.subscribe() - } - - /// Unregisters a peer: clears the awareness states it controlled (telling the others) and - /// schedules eviction once the room is idle. - pub(crate) fn leave(self: &Arc, origin: u64, controlled: &HashSet) { - if !controlled.is_empty() { - let msg = { - let mut a = lock(&self.awareness); - for id in controlled { - a.remove_state(*id); - } - a.update_with_clients(controlled.iter().copied()) - .ok() - .map(|u| Message::Awareness(u).encode_v1()) - }; - if let Some(msg) = msg { - let _ = self.tx.send(Frame::Data { - origin, - bytes: Bytes::from(msg), - echo: false, - }); - } - } - self.peers.fetch_sub(1, Ordering::AcqRel); - metrics::gauge!("canvas_peers").decrement(1.0); - self.touch(); - } - - /// Asks every peer of `user` (or every peer) to disconnect. - pub(crate) fn kick(&self, user: Option, code: u16, reason: &'static str) { - let _ = self.tx.send(Frame::Kick { user, code, reason }); - } - - pub(crate) fn peer_count(&self) -> usize { - self.peers.load(Ordering::Acquire) - } - - /// Records activity and makes sure an eviction check is pending. - fn touch(&self) { - *lock(&self.last_active) = Instant::now(); - if !self.evict_scheduled.swap(true, Ordering::AcqRel) { - let id = self.id; - if let Ok(rt) = tokio::runtime::Handle::try_current() { - rt.spawn(evict_loop(id)); - } else { - self.evict_scheduled.store(false, Ordering::Release); - } - } - } - - fn idle_for(&self) -> Duration { - lock(&self.last_active).elapsed() - } - - /// Publishes `Event::Canvas { kind: "update" }` to every member, at most once per - /// [`EVENT_THROTTLE`] (leading edge + one trailing publish). - fn notify_update(self: &Arc, state: &SharedState) { - let now = Instant::now(); - let delay = { - let mut t = lock(&self.throttle); - match t.last { - Some(last) if now.duration_since(last) < EVENT_THROTTLE => { - if t.trailing { - return; - } - t.trailing = true; - Some(EVENT_THROTTLE.saturating_sub(now.duration_since(last))) - } - _ => { - t.last = Some(now); - None - } - } - }; - let state = Arc::clone(state); - let weak: Weak = Arc::downgrade(self); - let canvas_id = self.id; - tokio::spawn(async move { - if let Some(delay) = delay { - tokio::time::sleep(delay).await; - if let Some(room) = weak.upgrade() { - let mut t = lock(&room.throttle); - t.trailing = false; - t.last = Some(Instant::now()); - } - } - if let Err(e) = publish_update(&state, canvas_id).await { - tracing::warn!(%canvas_id, error = %e, "publishing canvas update event failed"); - } - }); - } -} - -/// Bumps `canvases.updated_at` and tells every member something changed. -async fn publish_update(state: &AppState, canvas_id: Uuid) -> Result<(), ApiError> { - let members: Vec = sqlx::query_scalar( - "WITH bump AS (UPDATE canvases SET updated_at = now() WHERE id = $1) - SELECT user_id FROM canvas_members WHERE canvas_id = $1", - ) - .bind(canvas_id) - .fetch_all(&state.db) - .await?; - for user_id in members { - state.events.publish( - user_id, - Event::Canvas { - canvas_id, - kind: "update".into(), - }, - ); - } - Ok(()) -} - -async fn evict_loop(id: Uuid) { - let mut wait = IDLE_EVICT; - loop { - tokio::time::sleep(wait).await; - match rooms().try_evict(id, false) { - EvictOutcome::Evicted | EvictOutcome::Gone => return, - EvictOutcome::Busy(remaining) => { - wait = remaining.max(Duration::from_secs(1)); - } - } - } -} - -enum EvictOutcome { - Evicted, - Gone, - Busy(Duration), -} - -/// Live room counters. -#[derive(Debug, Clone, Copy, Default, PartialEq, Eq, Serialize)] -pub struct RoomsMetrics { - /// Rooms in memory. - pub rooms: usize, - /// Connected WebSocket peers across all rooms. - pub peers: usize, -} - -/// The process-wide room registry. -pub(crate) struct Rooms { - map: StdMutex>>, -} - -static ROOMS: LazyLock = LazyLock::new(|| Rooms { - map: StdMutex::new(HashMap::new()), -}); - -pub(crate) fn rooms() -> &'static Rooms { - &ROOMS -} - -impl Rooms { - /// The room for `id`, created (unloaded) if needed. - pub(crate) fn get(&self, id: Uuid) -> Arc { - let mut map = lock(&self.map); - let room = map.entry(id).or_insert_with(|| { - metrics::gauge!("canvas_rooms").increment(1.0); - Arc::new(Room::new(id)) - }); - Arc::clone(room) - } - - /// The room for `id` if it is in memory. - pub(crate) fn peek(&self, id: Uuid) -> Option> { - lock(&self.map).get(&id).cloned() - } - - /// Evicts `id` if nobody uses it. Only the registry may hold a reference: every user of a - /// room holds an `Arc`, so `strong_count == 1` under the registry lock means no handler - /// or peer can be mid-write. - fn try_evict(&self, id: Uuid, force: bool) -> EvictOutcome { - let mut map = lock(&self.map); - let Some(room) = map.get(&id) else { - return EvictOutcome::Gone; - }; - let idle = room.idle_for(); - let unused = Arc::strong_count(room) == 1 && room.peer_count() == 0; - if unused && (force || idle >= IDLE_EVICT) { - map.remove(&id); - metrics::gauge!("canvas_rooms").decrement(1.0); - tracing::debug!(canvas_id = %id, "canvas room evicted"); - return EvictOutcome::Evicted; - } - if !unused { - return EvictOutcome::Busy(IDLE_EVICT); - } - EvictOutcome::Busy(IDLE_EVICT.saturating_sub(idle)) - } - - /// Removes a room for good (canvas deleted) and disconnects its peers. - pub(crate) fn close(&self, id: Uuid) { - if let Some(room) = lock(&self.map).remove(&id) { - room.closed.store(true, Ordering::Release); - room.kick(None, close::DELETED, "canvas deleted"); - metrics::gauge!("canvas_rooms").decrement(1.0); - } - } - - /// Disconnects `user`'s peers from room `id` (membership revoked). - pub(crate) fn kick_user(&self, id: Uuid, user: Uuid) { - if let Some(room) = self.peek(id) { - room.kick(Some(user), close::REVOKED, "access revoked"); - } - } - - /// Disconnects `user`'s peers from every room (account disabled or deleted). - pub(crate) fn kick_user_everywhere(&self, user: Uuid) { - let all: Vec> = lock(&self.map).values().cloned().collect(); - for room in all { - room.kick(Some(user), close::REVOKED, "access revoked"); - } - } - - pub(crate) fn metrics(&self) -> RoomsMetrics { - let map = lock(&self.map); - RoomsMetrics { - rooms: map.len(), - peers: map.values().map(|r| r.peer_count()).sum(), - } - } -} - -/// Current room/peer counts. -pub fn rooms_metrics() -> RoomsMetrics { - rooms().metrics() -} - -/// Graceful shutdown: tells every connected peer the server is going away (close 1001) so -/// clients reconnect promptly. Nothing needs flushing — every update is persisted before it -/// is broadcast. -pub fn close_all_rooms() { - let all: Vec> = lock(&rooms().map).values().cloned().collect(); - for room in all { - room.kick(None, close::GOING_AWAY, "server shutting down"); - } -} - -/// Test hook: evicts the room for `canvas_id` now if no peer or request is using it. -/// Returns whether it was evicted (or was not in memory). -#[doc(hidden)] -pub fn evict_room_now(canvas_id: Uuid) -> bool { - matches!( - rooms().try_evict(canvas_id, true), - EvictOutcome::Evicted | EvictOutcome::Gone - ) -} - -/// Test hook: whether a room for `canvas_id` is in memory. -#[doc(hidden)] -pub fn room_in_memory(canvas_id: Uuid) -> bool { - rooms().peek(canvas_id).is_some() -} diff --git a/crates/canvas/src/schema.rs b/crates/canvas/src/schema.rs deleted file mode 100644 index e7b4a48..0000000 --- a/crates/canvas/src/schema.rs +++ /dev/null @@ -1,310 +0,0 @@ -//! The shared canvas Yjs schema (spec §4) and small value helpers. -//! -//! Document layout (identical on the TS page, the Swift host and this server): -//! -//! * `shapes`: `Y.Map>` — per-property last-writer-wins. -//! * `agents`: `Y.Map` (plain -//! JSON object values). -//! * `meta`: `Y.Map` with `name` and `createdBy`. -//! * `comments`: optional, untouched by the server. - -use serde_json::Value; -use yrs::types::ToJson; -use yrs::{Any, Number, Out, ReadTxn}; - -/// Root map holding every shape. -pub const SHAPES: &str = "shapes"; -/// Root map holding agent presence. -pub const AGENTS: &str = "agents"; -/// Root map holding canvas metadata. -pub const META: &str = "meta"; - -/// Every shape type the schema knows. -pub const SHAPE_TYPES: [&str; 7] = [ - "sticky", - "text", - "frame", - "arrow", - "image", - "link", - "checklist", -]; - -/// A shape type. -#[derive(Debug, Clone, Copy, PartialEq, Eq)] -pub enum ShapeType { - Sticky, - Text, - Frame, - Arrow, - Image, - Link, - /// A clickable RSVP / to-do card (`crate::checklist`). - Checklist, -} - -impl ShapeType { - pub fn parse(s: &str) -> Option { - Some(match s { - "sticky" => Self::Sticky, - "text" => Self::Text, - "frame" => Self::Frame, - "arrow" => Self::Arrow, - "image" => Self::Image, - "link" => Self::Link, - "checklist" => Self::Checklist, - _ => return None, - }) - } - - pub fn as_str(self) -> &'static str { - match self { - Self::Sticky => "sticky", - Self::Text => "text", - Self::Frame => "frame", - Self::Arrow => "arrow", - Self::Image => "image", - Self::Link => "link", - Self::Checklist => "checklist", - } - } - - /// Default size (the page's `SHAPE_SIZE`). - pub fn default_size(self) -> (f64, f64) { - match self { - Self::Sticky => (200.0, 200.0), - Self::Text => (240.0, 36.0), - Self::Frame => (480.0, 320.0), - Self::Arrow => (0.0, 0.0), - Self::Image => (320.0, 240.0), - Self::Link => (300.0, 84.0), - Self::Checklist => (324.0, 120.0), - } - } - - /// Smallest size of resizable types (`None` for arrows). - pub fn min_size(self) -> Option<(f64, f64)> { - match self { - Self::Sticky => Some((96.0, 64.0)), - Self::Text => Some((40.0, 24.0)), - Self::Frame => Some((160.0, 120.0)), - Self::Image => Some((32.0, 32.0)), - Self::Link => Some((180.0, 56.0)), - Self::Checklist => Some((200.0, 72.0)), - Self::Arrow => None, - } - } - - pub fn default_color(self) -> &'static str { - match self { - Self::Sticky => "yellow", - Self::Image | Self::Link => "white", - Self::Text | Self::Frame | Self::Arrow => "gray", - Self::Checklist => "green", - } - } - - /// Whether the body text is a `Y.Text` (concurrent typing merges). - pub fn has_body(self) -> bool { - matches!(self, Self::Sticky | Self::Text) - } - - /// Props an op may set on this type. - pub fn allowed(self) -> &'static [&'static str] { - match self { - Self::Sticky => &["x", "y", "w", "h", "color", "z", "text", "fontSize"], - Self::Text => &[ - "x", "y", "w", "h", "color", "z", "text", "fontSize", "align", - ], - Self::Frame => &["x", "y", "w", "h", "color", "z", "title", "image"], - Self::Arrow => &["color", "z", "from", "to", "label"], - Self::Image => &[ - "x", "y", "w", "h", "color", "z", "src", "naturalW", "naturalH", - ], - Self::Link => &["x", "y", "w", "h", "color", "z", "url", "title", "favicon"], - Self::Checklist => &[ - "x", "y", "w", "h", "color", "z", "title", "columns", "rows", "picks", - ], - } - } - - /// Friendly spellings: `text` on a frame or link is its title, on an arrow its label. - pub fn alias(self, key: &str) -> &str { - match (self, key) { - (Self::Frame | Self::Link | Self::Checklist, "text" | "label") => "title", - (Self::Arrow, "text" | "title") => "label", - (Self::Sticky | Self::Text, "title") => "text", - _ => key, - } - } -} - -/// Longest `data:` URL accepted for images (2 MiB, as on the page). -pub const MAX_DATA_URL: usize = 2 * 1024 * 1024; -/// Longest sticky/text body (UTF-16 units). -pub const MAX_TEXT: usize = 20_000; -/// Titles and labels are cut to this many UTF-16 units. -pub const MAX_TITLE: usize = 500; - -/// Length in UTF-16 code units (JavaScript `.length`). -pub fn js_len(s: &str) -> usize { - s.encode_utf16().count() -} - -/// The longest prefix of `s` that is at most `max` UTF-16 units (never splits a char). -pub fn js_slice(s: &str, max: usize) -> &str { - let mut units = 0; - for (i, c) in s.char_indices() { - units += c.len_utf16(); - if units > max { - return &s[..i]; - } - } - s -} - -/// A stable colour for an agent without one (the page's `colorFor`). -pub fn color_for(seed: &str) -> String { - let mut hash: i32 = 0; - for unit in seed.encode_utf16() { - hash = hash.wrapping_mul(31).wrapping_add(i32::from(unit)); - } - format!("hsl({} 62% 48%)", hash.unsigned_abs() % 360) -} - -/// Named palette colours (plus `#hex`). -pub const NAMED_COLORS: [&str; 7] = ["yellow", "pink", "blue", "green", "purple", "gray", "white"]; - -/// Agent presence statuses. -pub const AGENT_STATUSES: [&str; 3] = ["idle", "thinking", "writing"]; - -/// True when `s` is a palette colour or a `#rgb`, `#rgba`, `#rrggbb` or `#rrggbbaa` hex colour. -pub fn is_valid_color(s: &str) -> bool { - if NAMED_COLORS.contains(&s) { - return true; - } - match s.strip_prefix('#') { - Some(hex) => { - matches!(hex.len(), 3 | 4 | 6 | 8) && hex.bytes().all(|b| b.is_ascii_hexdigit()) - } - None => false, - } -} - -/// Converts a JSON value into a Yjs `Any`. -pub fn json_to_any(v: &Value) -> Any { - match v { - Value::Null => Any::Null, - Value::Bool(b) => Any::Bool(*b), - Value::Number(n) => n - .as_i64() - .map(|i| Any::Number(Number::Int(i))) - .or_else(|| n.as_f64().map(num_any)) - .unwrap_or(Any::Null), - Value::String(s) => Any::String(s.as_str().into()), - Value::Array(items) => Any::from(items.iter().map(json_to_any).collect::>()), - Value::Object(map) => Any::from( - map.iter() - .map(|(k, v)| (k.clone(), json_to_any(v))) - .collect::>(), - ), - } -} - -/// Converts a Yjs `Any` into JSON. -pub fn any_to_json(a: &Any) -> Value { - serde_json::to_value(a).unwrap_or(Value::Null) -} - -/// Converts any Yjs output value (including nested shared types) into JSON. -pub fn out_to_json(txn: &T, out: &Out) -> Value { - match out { - Out::Any(a) => any_to_json(a), - other => any_to_json(&other.to_json(txn)), - } -} - -/// A number as a compact Yjs value (integral values are stored as integers). -pub fn num_any(v: f64) -> Any { - Any::Number(Number::try_i64(v)) -} - -/// Reads a finite number from a Yjs output value. -pub fn out_f64(out: Option<&Out>) -> Option { - match out { - Some(Out::Any(Any::Number(n))) => n.as_f64().filter(|f| f.is_finite()), - _ => None, - } -} - -/// Reads a string from a Yjs output value (plain strings and `Y.Text`). -pub fn out_string(txn: &T, out: Option<&Out>) -> Option { - match out { - Some(Out::Any(Any::String(s))) => Some(s.to_string()), - Some(Out::YText(t)) => { - use yrs::GetString; - Some(t.get_string(txn)) - } - _ => None, - } -} - -/// A JSON number that is an integer when `v` is integral (as JavaScript prints it). -#[allow(clippy::cast_possible_truncation)] -pub fn jnum(v: f64) -> Value { - if v.is_finite() && v.fract() == 0.0 && v.abs() <= 9_007_199_254_740_991.0 { - Value::from(v as i64) - } else { - Value::from(v) - } -} - -/// `serialize_with` for `f64` fields: integral values print as integers. -pub fn ser_num(v: &f64, s: S) -> Result { - serde::Serialize::serialize(&jnum(*v), s) -} - -/// `serialize_with` for `Option` fields. -#[allow(clippy::ref_option)] -pub fn ser_opt_num(v: &Option, s: S) -> Result { - serde::Serialize::serialize(&v.map(jnum), s) -} - -/// Reads a finite number from a JSON value. -pub fn json_f64(v: &Value) -> Option { - v.as_f64().filter(|f| f.is_finite()) -} - -/// Current time in milliseconds since the Unix epoch. -pub fn now_ms() -> i64 { - let d = std::time::SystemTime::now() - .duration_since(std::time::UNIX_EPOCH) - .unwrap_or_default(); - i64::try_from(d.as_millis()).unwrap_or(i64::MAX) -} - -#[cfg(test)] -mod tests { - use super::*; - use serde_json::json; - - #[test] - fn colors() { - for ok in ["yellow", "white", "#fff", "#ffff", "#a0b1c2", "#A0B1C2ff"] { - assert!(is_valid_color(ok), "{ok}"); - } - for bad in [ - "", "red", "#", "#ff", "#ggg", "#12345", "yellow ", "#1234567", - ] { - assert!(!is_valid_color(bad), "{bad}"); - } - } - - #[test] - fn json_roundtrip() { - let v = json!({"a": 1, "b": 1.5, "c": "x", "d": [true, null], "e": {"f": -3}}); - assert_eq!(any_to_json(&json_to_any(&v)), v); - assert_eq!(num_any(2.0), Any::Number(Number::Int(2))); - assert_eq!(num_any(2.5), Any::Number(Number::Float(2.5))); - } -} diff --git a/crates/canvas/src/shape.rs b/crates/canvas/src/shape.rs deleted file mode 100644 index 8e45942..0000000 --- a/crates/canvas/src/shape.rs +++ /dev/null @@ -1,298 +0,0 @@ -//! Tolerant reads of shapes from the Yjs document (mirrors the page's `readShape`), plus -//! the value normalisation used when writing props (`storeValue`). - -use serde_json::{json, Value}; -use yrs::{Any, Map, MapRef, Out, ReadTxn}; - -use crate::checklist::{read_checklist, Checklist}; -use crate::geometry::{arrow_box, arrow_path, js_round, BoxF, End, Point, Side}; -use crate::schema::{ - is_valid_color, jnum, json_f64, out_f64, out_string, out_to_json, ShapeType, SHAPES, -}; - -/// `shape:` prefix some callers put on ids. -pub fn bare_id(s: &str) -> &str { - s.strip_prefix("shape:").unwrap_or(s) -} - -/// An arrow end as stored: a free point, or a shape optionally pinned to a side. -#[derive(Debug, Clone, PartialEq)] -pub enum Endpoint { - Point { x: f64, y: f64 }, - Ref { id: String, side: Option }, -} - -impl Endpoint { - /// Any accepted spelling — `{x,y}`, `{ref, side?}`, `{id, side?}`, `"id"` or `"shape:id"` - /// — or `None` (the page's `readEndpoint`). Points are rounded. - pub fn read(v: &Value) -> Option { - if let Some(s) = v.as_str() { - let t = s.trim(); - return (!t.is_empty()).then(|| Self::Ref { - id: bare_id(t).to_owned(), - side: None, - }); - } - let o = v.as_object()?; - let side = o.get("side").and_then(Value::as_str).and_then(Side::parse); - for key in ["ref", "id"] { - if let Some(r) = o.get(key).and_then(Value::as_str) { - let t = r.trim(); - if !t.is_empty() { - return Some(Self::Ref { - id: bare_id(t).to_owned(), - side, - }); - } - } - } - let x = o.get("x").and_then(json_f64)?; - let y = o.get("y").and_then(json_f64)?; - Some(Self::Point { - x: js_round(x), - y: js_round(y), - }) - } - - pub fn to_json(&self) -> Value { - match self { - Self::Point { x, y } => json!({ "x": jnum(*x), "y": jnum(*y) }), - Self::Ref { id, side: Some(s) } => json!({ "ref": id, "side": s.as_str() }), - Self::Ref { id, side: None } => json!({ "ref": id }), - } - } - - pub fn ref_id(&self) -> Option<&str> { - match self { - Self::Ref { id, .. } => Some(id), - Self::Point { .. } => None, - } - } -} - -/// `{src, naturalW, naturalH}` of a frame image. -#[derive(Debug, Clone, PartialEq)] -pub struct ImageRef { - pub src: String, - pub natural_w: f64, - pub natural_h: f64, -} - -impl ImageRef { - pub fn read(v: &Value) -> Option { - let o = v.as_object()?; - let src = o - .get("src") - .and_then(Value::as_str) - .filter(|s| !s.is_empty())?; - Some(Self { - src: src.to_owned(), - natural_w: o.get("naturalW").and_then(json_f64).unwrap_or(0.0), - natural_h: o.get("naturalH").and_then(json_f64).unwrap_or(0.0), - }) - } - - pub fn to_json(&self) -> Value { - json!({ "src": self.src, "naturalW": jnum(self.natural_w), "naturalH": jnum(self.natural_h) }) - } -} - -/// A plain read of one shape (defaults filled in like the page does). -#[derive(Debug, Clone, PartialEq)] -pub struct Shape { - pub id: String, - pub kind: ShapeType, - pub x: f64, - pub y: f64, - pub w: f64, - pub h: f64, - pub color: String, - pub by: Option, - pub created_at: f64, - pub updated_at: f64, - pub z: f64, - pub text: String, - pub title: String, - pub font_size: Option, - pub align: Option, - pub image: Option, - pub from: Option, - pub to: Option, - pub label: String, - pub src: String, - pub natural_w: f64, - pub natural_h: f64, - pub url: String, - /// A checklist's columns, rows and picks (`None` for other types). - pub checklist: Option, -} - -impl Shape { - pub fn bbox(&self) -> BoxF { - BoxF { - x: self.x, - y: self.y, - w: self.w, - h: self.h, - } - } - - pub fn is_arrow(&self) -> bool { - self.kind == ShapeType::Arrow - } - - /// Whether one of this arrow's ends is bound to a shape in `ids`. - pub fn touches(&self, ids: &[&str]) -> bool { - let hit = |e: &Option| { - e.as_ref() - .and_then(Endpoint::ref_id) - .is_some_and(|r| ids.contains(&r)) - }; - self.is_arrow() && (hit(&self.from) || hit(&self.to)) - } -} - -/// Reads one shape map; `None` when it is not a readable shape. -pub fn read_shape(txn: &T, id: &str, out: &Out) -> Option { - let Out::YMap(m) = out else { return None }; - let get = |k: &str| m.get(txn, k); - let num = |k: &str| out_f64(get(k).as_ref()); - let s = |k: &str| out_string(txn, get(k).as_ref()); - let kind = ShapeType::parse(&s("type")?)?; - let (dw, dh) = kind.default_size(); - let color = s("color") - .filter(|c| is_valid_color(c)) - .unwrap_or_else(|| kind.default_color().to_owned()); - let json = |k: &str| get(k).map(|o| out_to_json(txn, &o)); - let w = num("w").unwrap_or(dw).max(0.0); - let h = num("h").unwrap_or(dh).max(0.0); - let is = |t: ShapeType| kind == t; - Some(Shape { - id: id.to_owned(), - x: num("x").unwrap_or(0.0), - y: num("y").unwrap_or(0.0), - w, - h, - color, - by: s("by").filter(|b| !b.is_empty()), - created_at: num("createdAt").unwrap_or(0.0), - updated_at: num("updatedAt").unwrap_or(0.0), - z: num("z").unwrap_or(0.0), - text: s("text").unwrap_or_default(), - title: s("title").unwrap_or_default(), - font_size: num("fontSize").filter(|f| *f > 0.0), - align: s("align").filter(|a| matches!(a.as_str(), "left" | "center" | "right")), - image: if is(ShapeType::Frame) { - json("image").as_ref().and_then(ImageRef::read) - } else { - None - }, - from: if is(ShapeType::Arrow) { - json("from").as_ref().and_then(Endpoint::read) - } else { - None - }, - to: if is(ShapeType::Arrow) { - json("to").as_ref().and_then(Endpoint::read) - } else { - None - }, - label: s("label").unwrap_or_default(), - src: s("src").unwrap_or_default(), - natural_w: num("naturalW").unwrap_or(w), - natural_h: num("naturalH").unwrap_or(h), - url: s("url").unwrap_or_default(), - checklist: is(ShapeType::Checklist).then(|| read_checklist(&json)), - kind, - }) -} - -/// Every readable shape of the document, in map order. -pub fn all_shapes(txn: &T, shapes: &MapRef) -> Vec { - shapes - .iter(txn) - .filter_map(|(id, out)| read_shape(txn, id, &out)) - .collect() -} - -/// One shape by id. -pub fn live_shape(txn: &T, shapes: &MapRef, id: &str) -> Option { - shapes.get(txn, id).and_then(|o| read_shape(txn, id, &o)) -} - -/// Every readable shape (when the `shapes` root map exists). -pub fn doc_shapes(txn: &T) -> Vec { - txn.get_map(SHAPES) - .map(|m| all_shapes(txn, &m)) - .unwrap_or_default() -} - -/// Resolves an endpoint against the current boxes (arrows are not boxes). -pub fn resolve_end(e: Option<&Endpoint>, box_of: &dyn Fn(&str) -> Option) -> Option { - match e? { - Endpoint::Point { x, y } => Some(End::Point(Point { x: *x, y: *y })), - Endpoint::Ref { id, side } => box_of(id).map(|b| End::Box(b, *side)), - } -} - -/// Bounding box of an arrow given the current boxes (`0,0,0,0` when it cannot be drawn). -pub fn arrow_bbox(arrow: &Shape, box_of: &dyn Fn(&str) -> Option) -> BoxF { - let path = resolve_end(arrow.from.as_ref(), box_of) - .zip(resolve_end(arrow.to.as_ref(), box_of)) - .and_then(|(a, b)| arrow_path(a, b)); - path.as_ref().map_or( - BoxF { - x: 0.0, - y: 0.0, - w: 0.0, - h: 0.0, - }, - arrow_box, - ) -} - -/// Midpoint of an arrow's path, if it can be drawn. -pub fn arrow_mid(arrow: &Shape, box_of: &dyn Fn(&str) -> Option) -> Option { - let a = resolve_end(arrow.from.as_ref(), box_of)?; - let b = resolve_end(arrow.to.as_ref(), box_of)?; - arrow_path(a, b).map(|p| p.mid) -} - -/// A box lookup over a set of shapes (arrows excluded). -pub fn box_lookup(shapes: &[Shape]) -> impl Fn(&str) -> Option + '_ { - move |id: &str| { - shapes - .iter() - .find(|s| s.id == id && !s.is_arrow()) - .map(Shape::bbox) - } -} - -/// The value as stored for `key` (the page's `storeValue`): boxes rounded, endpoints and -/// images normalised. `None` means "do not store". -pub fn store_value(key: &str, value: &Value) -> Option { - match key { - "x" | "y" | "w" | "h" => match json_f64(value) { - Some(n) => Some(crate::schema::num_any(js_round(n))), - None => Some(crate::schema::json_to_any(value)), - }, - "from" | "to" => Endpoint::read(value).map(|e| crate::schema::json_to_any(&e.to_json())), - "image" => ImageRef::read(value).map(|i| crate::schema::json_to_any(&i.to_json())), - _ => Some(crate::schema::json_to_any(value)), - } -} - -/// The current `text` of a shape map as a plain string. -pub fn map_text(txn: &T, m: &MapRef) -> Option { - out_string(txn, m.get(txn, "text").as_ref()) -} - -/// Highest `z` (at least 0), like the page's `maxZ`. -pub fn max_z(shapes: &[Shape]) -> f64 { - shapes.iter().map(|s| s.z).fold(0.0, f64::max) -} - -/// Whether `Any` is a string equal to `s`. -pub fn any_is_str(a: &Any, s: &str) -> bool { - matches!(a, Any::String(v) if &**v == s) -} diff --git a/crates/canvas/src/store.rs b/crates/canvas/src/store.rs deleted file mode 100644 index 4ef6a07..0000000 --- a/crates/canvas/src/store.rs +++ /dev/null @@ -1,124 +0,0 @@ -//! Durable canvas document storage: sealed update log + compacted snapshot. - -use copper_cloud_core::crypto::Crypto; -use copper_cloud_core::error::ApiError; -use sqlx::PgPool; -use uuid::Uuid; - -/// What a room needs to rebuild its document. -pub(crate) struct Stored { - pub key: [u8; 32], - /// `(seq, state)` of the compacted snapshot, if any. - pub snapshot: Option<(i64, Vec)>, - /// `(seq, update)` of every update newer than the snapshot, in order. - pub updates: Vec<(i64, Vec)>, -} - -/// AAD binding a sealed update/snapshot to its canvas. -pub(crate) fn aad(canvas_id: Uuid) -> [u8; 16] { - *canvas_id.as_bytes() -} - -/// Unwraps a canvas's doc key. -pub(crate) async fn doc_key( - db: &PgPool, - crypto: &Crypto, - canvas_id: Uuid, -) -> Result<[u8; 32], ApiError> { - let wrapped: Vec = sqlx::query_scalar("SELECT doc_key_wrapped FROM canvases WHERE id = $1") - .bind(canvas_id) - .fetch_optional(db) - .await? - .ok_or(ApiError::NotFound)?; - crypto.unwrap_key(&wrapped) -} - -/// Loads and decrypts the snapshot and the update tail of a canvas. -pub(crate) async fn load( - db: &PgPool, - crypto: &Crypto, - canvas_id: Uuid, -) -> Result { - let key = doc_key(db, crypto, canvas_id).await?; - let aad = aad(canvas_id); - let snapshot: Option<(i64, Vec)> = - sqlx::query_as("SELECT seq, state FROM canvas_snapshots WHERE canvas_id = $1") - .bind(canvas_id) - .fetch_optional(db) - .await?; - let since = snapshot.as_ref().map_or(0, |s| s.0); - let snapshot = match snapshot { - Some((seq, sealed)) => Some((seq, Crypto::open(&key, &aad, &sealed)?)), - None => None, - }; - let rows: Vec<(i64, Vec)> = sqlx::query_as( - r#"SELECT seq, "update" FROM canvas_updates - WHERE canvas_id = $1 AND seq > $2 ORDER BY seq"#, - ) - .bind(canvas_id) - .bind(since) - .fetch_all(db) - .await?; - let mut updates = Vec::with_capacity(rows.len()); - for (seq, sealed) in rows { - updates.push((seq, Crypto::open(&key, &aad, &sealed)?)); - } - Ok(Stored { - key, - snapshot, - updates, - }) -} - -/// Appends one sealed update; returns its sequence number. -pub(crate) async fn append( - db: &PgPool, - key: &[u8; 32], - canvas_id: Uuid, - update: &[u8], -) -> Result { - let sealed = Crypto::seal(key, &aad(canvas_id), update); - let seq: i64 = sqlx::query_scalar( - r#"INSERT INTO canvas_updates (canvas_id, "update") VALUES ($1, $2) RETURNING seq"#, - ) - .bind(canvas_id) - .bind(&sealed) - .fetch_one(db) - .await?; - metrics::counter!("canvas_updates_persisted_total").increment(1); - metrics::counter!("canvas_update_bytes_total").increment(update.len() as u64); - Ok(seq) -} - -/// Replaces the snapshot with `state` (covering every update up to `upto_seq`) and deletes -/// the covered updates, in one transaction. -pub(crate) async fn compact( - db: &PgPool, - key: &[u8; 32], - canvas_id: Uuid, - upto_seq: i64, - state: &[u8], -) -> Result { - let sealed = Crypto::seal(key, &aad(canvas_id), state); - let mut tx = db.begin().await?; - sqlx::query( - "INSERT INTO canvas_snapshots (canvas_id, seq, state, updated_at) - VALUES ($1, $2, $3, now()) - ON CONFLICT (canvas_id) DO UPDATE - SET seq = EXCLUDED.seq, state = EXCLUDED.state, updated_at = now()", - ) - .bind(canvas_id) - .bind(upto_seq) - .bind(&sealed) - .execute(&mut *tx) - .await?; - let deleted = sqlx::query("DELETE FROM canvas_updates WHERE canvas_id = $1 AND seq <= $2") - .bind(canvas_id) - .bind(upto_seq) - .execute(&mut *tx) - .await? - .rows_affected(); - tx.commit().await?; - metrics::counter!("canvas_compactions_total").increment(1); - Ok(deleted) -} diff --git a/crates/canvas/src/ws.rs b/crates/canvas/src/ws.rs deleted file mode 100644 index c1587cf..0000000 --- a/crates/canvas/src/ws.rs +++ /dev/null @@ -1,257 +0,0 @@ -//! y-websocket peers: one read loop + one write task per connection. -//! -//! Wire format: every WebSocket *binary* message carries one or more y-protocol messages -//! (lib0 var-uint tag, then payload), exactly as `y-websocket` sends them: -//! `0` sync (`0` `SyncStep1` / `1` `SyncStep2` / `2` `Update`), `1` awareness, `2` auth, -//! `3` awareness query. Text frames are ignored. - -use std::collections::HashSet; -use std::sync::Arc; -use std::time::Duration; - -use axum::extract::ws::{CloseFrame, Message as WsMessage, Utf8Bytes, WebSocket}; -use bytes::Bytes; -use copper_cloud_core::error::ApiError; -use copper_cloud_core::state::SharedState; -use futures::stream::SplitSink; -use futures::{SinkExt as _, StreamExt as _}; -use tokio::sync::{broadcast, mpsc}; -use uuid::Uuid; -use yrs::encoding::read::Cursor; -use yrs::sync::{Message, MessageReader, SyncMessage}; -use yrs::updates::decoder::DecoderV1; -use yrs::ClientID; - -use crate::room::{next_peer_id, Frame, Room}; - -/// Largest accepted WebSocket message / frame. The page stores images as `data:` URLs of up -/// to 2 MiB, so one Yjs update can legitimately exceed 1 MiB; 8 MiB leaves room for a few. -pub const MAX_MESSAGE_BYTES: usize = 8 << 20; -/// Server → client ping interval. -pub const PING_EVERY: Duration = Duration::from_secs(30); -/// A peer that sends nothing (not even a pong) for this long is dropped. -pub const READ_TIMEOUT: Duration = Duration::from_secs(90); -/// A peer that does not accept a frame within this long is dropped. -const SEND_TIMEOUT: Duration = Duration::from_secs(15); -/// Replies queued for one peer (`SyncStep2`, awareness query answers). -const DIRECT_CAPACITY: usize = 32; - -/// Close codes sent by the server. -pub mod close { - /// Normal closure. - pub const NORMAL: u16 = 1000; - /// Server shutting down / room closed. - pub const GOING_AWAY: u16 = 1001; - /// Undecodable y-protocol message or rejected update. - pub const INVALID: u16 = 1007; - /// Message or frame over [`super::MAX_MESSAGE_BYTES`]. - pub const TOO_BIG: u16 = 1009; - /// Server-side failure; reconnect to resync. - pub const SERVER_ERROR: u16 = 1011; - /// The peer fell too far behind the room; reconnect to resync. - pub const TOO_SLOW: u16 = 1013; - /// The caller is no longer a member of the canvas (do not reconnect). - pub const REVOKED: u16 = 4403; - /// The canvas was deleted (do not reconnect). - pub const DELETED: u16 = 4404; -} - -enum Outgoing { - Data(Bytes), - Close(u16, &'static str), -} - -/// Unregisters a peer from its room however the peer task ends (including a panic). -struct PeerGuard { - room: Arc, - peer: u64, - controlled: HashSet, -} - -impl Drop for PeerGuard { - fn drop(&mut self) { - self.room.leave(self.peer, &self.controlled); - } -} - -/// Serves one authenticated member on `room` until either side hangs up. -pub(crate) async fn serve_peer( - state: SharedState, - room: Arc, - user_id: Uuid, - socket: WebSocket, -) { - let peer = next_peer_id(); - let bcast = room.join(); - let mut guard = PeerGuard { - room: Arc::clone(&room), - peer, - controlled: HashSet::new(), - }; - let (sink, mut stream) = socket.split(); - let (direct_tx, direct_rx) = mpsc::channel::(DIRECT_CAPACITY); - - match room.greeting(&state).await { - Ok(msgs) => { - for m in msgs { - let _ = direct_tx.try_send(Outgoing::Data(m)); - } - } - Err(e) => { - tracing::warn!(canvas_id = %room.id, error = %e, "canvas greeting failed"); - let _ = direct_tx.try_send(Outgoing::Close(close::SERVER_ERROR, "load failed")); - } - } - - let mut writer = tokio::spawn(write_loop(sink, direct_rx, bcast, peer, user_id)); - let mut writer_done = false; - loop { - tokio::select! { - _ = &mut writer => { - writer_done = true; - break; - } - next = tokio::time::timeout(READ_TIMEOUT, stream.next()) => { - let msg = match next { - Ok(Some(Ok(msg))) => msg, - Ok(Some(Err(e))) => { - // tungstenite's capacity error: the frame/message is over the limit. - if e.to_string().contains("Space limit exceeded") { - let _ = direct_tx.try_send(Outgoing::Close(close::TOO_BIG, "message too big")); - } - break; - } - _ => break, - }; - match msg { - WsMessage::Binary(bytes) => { - metrics::counter!("canvas_ws_bytes_in_total").increment(bytes.len() as u64); - if let Err(e) = handle_binary(&state, &room, peer, &bytes, &direct_tx, &mut guard.controlled).await { - let (code, reason) = match &e { - ApiError::BadRequest(_) => (close::INVALID, "invalid message"), - ApiError::NotFound => (close::DELETED, "canvas deleted"), - _ => (close::SERVER_ERROR, "server error"), - }; - tracing::debug!(canvas_id = %room.id, peer, code, error = %e, "closing canvas peer"); - let _ = direct_tx.send(Outgoing::Close(code, reason)).await; - break; - } - } - WsMessage::Close(_) => break, - WsMessage::Text(_) | WsMessage::Ping(_) | WsMessage::Pong(_) => {} - } - } - } - } - drop(guard); - drop(direct_tx); - if !writer_done - && tokio::time::timeout(Duration::from_secs(2), &mut writer) - .await - .is_err() - { - writer.abort(); - } -} - -/// Decodes every y-protocol message in one WebSocket frame. -fn decode_frame(bytes: &[u8]) -> Result, ApiError> { - std::panic::catch_unwind(|| { - let mut decoder = DecoderV1::new(Cursor::new(bytes)); - MessageReader::new(&mut decoder).collect::, _>>() - }) - .map_err(|_| ApiError::bad_request("undecodable message"))? - .map_err(|e| ApiError::bad_request(format!("undecodable message: {e}"))) -} - -async fn handle_binary( - state: &SharedState, - room: &Arc, - peer: u64, - bytes: &[u8], - direct: &mpsc::Sender, - controlled: &mut HashSet, -) -> Result<(), ApiError> { - for msg in decode_frame(bytes)? { - match msg { - Message::Sync(SyncMessage::SyncStep1(sv)) => { - let reply = room.sync_step2(state, &sv).await?; - send_direct(direct, reply).await?; - } - Message::Sync(SyncMessage::SyncStep2(update) | SyncMessage::Update(update)) => { - room.apply_remote(state, peer, &update).await?; - } - Message::Awareness(update) => room.apply_awareness(peer, update, controlled)?, - Message::AwarenessQuery => { - if let Some(reply) = room.awareness_message() { - send_direct(direct, reply).await?; - } - } - // Auth is decided at upgrade time; custom tags are ignored. - Message::Auth(_) | Message::Custom(..) => {} - } - } - Ok(()) -} - -async fn send_direct(direct: &mpsc::Sender, msg: Vec) -> Result<(), ApiError> { - direct - .send(Outgoing::Data(Bytes::from(msg))) - .await - .map_err(|_| ApiError::internal("peer writer gone")) -} - -async fn write_loop( - mut sink: SplitSink, - mut direct: mpsc::Receiver, - mut bcast: broadcast::Receiver, - peer: u64, - user: Uuid, -) { - let mut shutdown = copper_cloud_core::shutdown::subscribe(); - let mut ping = tokio::time::interval_at(tokio::time::Instant::now() + PING_EVERY, PING_EVERY); - ping.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Delay); - let close_with = loop { - let out = tokio::select! { - d = direct.recv() => match d { - Some(Outgoing::Data(b)) => WsMessage::Binary(b), - Some(Outgoing::Close(code, reason)) => break Some((code, reason)), - None => break Some((close::NORMAL, "")), - }, - f = bcast.recv() => match f { - Ok(Frame::Data { origin, bytes, echo }) => { - if origin == peer && !echo { - continue; - } - WsMessage::Binary(bytes) - } - Ok(Frame::Kick { user: target, code, reason }) => { - if target.is_none_or(|u| u == user) { - break Some((code, reason)); - } - continue; - } - Err(broadcast::error::RecvError::Lagged(_)) => break Some((close::TOO_SLOW, "too slow")), - Err(broadcast::error::RecvError::Closed) => break Some((close::GOING_AWAY, "")), - }, - _ = ping.tick() => WsMessage::Ping(Bytes::new()), - () = copper_cloud_core::shutdown::wait(&mut shutdown) => { - break Some((close::GOING_AWAY, "server shutting down")); - } - }; - if let WsMessage::Binary(b) = &out { - metrics::counter!("canvas_ws_bytes_out_total").increment(b.len() as u64); - } - match tokio::time::timeout(SEND_TIMEOUT, sink.send(out)).await { - Ok(Ok(())) => {} - _ => break None, - } - }; - if let Some((code, reason)) = close_with { - let frame = CloseFrame { - code, - reason: Utf8Bytes::from_static(reason), - }; - let _ = tokio::time::timeout(SEND_TIMEOUT, sink.send(WsMessage::Close(Some(frame)))).await; - } -} diff --git a/crates/canvas/tests/chat.rs b/crates/canvas/tests/chat.rs deleted file mode 100644 index d63f53d..0000000 --- a/crates/canvas/tests/chat.rs +++ /dev/null @@ -1,244 +0,0 @@ -//! Canvas chat (0.6.0): the client-owned top-level `chat` Y.Array is relayed live, persisted, -//! and reloaded untouched after the room is dropped from memory, after a fresh server state -//! (new pool + app, nothing cached), and across snapshot compaction; `/read` shows the last 50. - -#![allow(clippy::too_many_lines)] - -mod common; - -use std::time::Duration; - -use axum::http::StatusCode; -use common::{eventually, Peer, TestApp}; -use copper_cloud_canvas::schema::{json_to_any, out_to_json}; -use serde_json::{json, Value}; -use yrs::{Array as _, Doc, ReadTxn as _, Transact as _, TransactionMut, WriteTxn as _}; - -const T: Duration = Duration::from_secs(5); - -fn canvas_uuid(id: &str) -> uuid::Uuid { - uuid::Uuid::parse_str(id).unwrap() -} - -/// A message as the Copper page stores it (a plain object). -fn message(i: usize, author: &str, mentions: &[&str]) -> Value { - json!({ - "id": format!("m{i}"), - "authorId": format!("{author}-id"), - "authorName": author, - "text": format!("message {i} from {author}"), - "mentions": mentions, - "at": 1_790_000_000_000_i64 + i64::try_from(i).unwrap(), - }) -} - -fn push(txn: &mut TransactionMut, msg: &Value) { - let chat = txn.get_or_insert_array("chat"); - chat.push_back(txn, json_to_any(msg)); -} - -/// The `chat` array of `doc` as JSON (empty when absent). -fn chat_of(doc: &Doc) -> Vec { - let txn = doc.transact(); - txn.get_array("chat").map_or_else(Vec::new, |a| { - a.iter(&txn).map(|o| out_to_json(&txn, &o)).collect() - }) -} - -async fn send(p: &mut Peer, msg: Value) { - p.edit(|txn| push(txn, &msg)).await; -} - -#[tokio::test] -async fn chat_persists_across_eviction_and_restart() { - let app = TestApp::new().await; - let ann = app.user("ann").await; - let ben = app.user("ben").await; - let id = app.canvas(&ann, "Team dinners").await; - let cid = canvas_uuid(&id); - app.share(&ann, &id, &ben).await; - let ben_id = ben.id.to_string(); - let to_ben = [ben_id.as_str()]; - - // Two live peers: messages typed by one appear at the other. - let mut pa = app.peer(&ann, &id).await; - let mut pb = app.peer(&ben, &id).await; - send(&mut pa, message(1, "ann", &[])).await; - send(&mut pa, message(2, "ann", &to_ben)).await; - assert!(pb.pump_until(T, |p| chat_of(&p.doc).len() == 2).await); - send(&mut pb, message(3, "ben", &[])).await; - assert!( - pb.pump_until(T, |p| chat_of(&p.doc).len() == 3).await, - "ben sees all three" - ); - assert!(pa.pump_until(T, |p| chat_of(&p.doc).len() == 3).await); - let live = chat_of(&pa.doc); - assert_eq!(live, chat_of(&pb.doc)); - assert_eq!(live[1], message(2, "ann", &to_ben)); - - // A REST ops write (server-side transaction on `shapes`) leaves the chat alone. - let (st, r) = app - .post( - &ann, - &format!("/canvases/{id}/ops"), - json!({ "ops": [{ "op": "add", "shape": { "id": "s1", "type": "sticky", "text": "RSVP" } }] }), - ) - .await; - assert_eq!(st, StatusCode::OK, "{r}"); - assert!(pb.pump_until(T, |p| p.shape_count() == 1).await); - assert_eq!(chat_of(&pb.doc), live); - - // /read carries the chat (oldest first) next to the shapes. - let (st, read) = app.get(&ann, &format!("/canvases/{id}/read")).await; - assert_eq!(st, StatusCode::OK, "{read}"); - assert_eq!(read["count"], 1); - let chat = read["chat"].as_array().unwrap(); - let got: Vec<&str> = chat.iter().map(|m| m["id"].as_str().unwrap()).collect(); - assert_eq!(got, vec!["m1", "m2", "m3"]); - assert_eq!(chat[1], message(2, "ann", &to_ben)); - - // Drop the room from memory and reload it from Postgres. - pa.close().await; - pb.close().await; - assert!( - eventually(T, || async { copper_cloud_canvas::evict_room_now(cid) }).await, - "room evicts once the peers are gone" - ); - assert!(!copper_cloud_canvas::room_in_memory(cid)); - assert_eq!(chat_of(&app.state_doc(&ann, &id).await), live); - assert!(eventually(T, || async { copper_cloud_canvas::evict_room_now(cid) }).await); - - // A restarted server (same database and master key; fresh pool, state, router and - // listener; nothing in memory) serves the same chat over REST and over the socket. - let app2 = app.restart().await; - assert!(!copper_cloud_canvas::room_in_memory(cid)); - assert_eq!(chat_of(&app2.state_doc(&ben, &id).await), live); - let mut pc = app2.peer(&ben, &id).await; - assert_eq!(chat_of(&pc.doc), live, "reconnecting peer syncs the chat"); - assert_eq!(pc.shape_count(), 1); - - // The client caps the array by deleting the oldest when it appends; that persists too. - pc.edit(|txn| { - push(txn, &message(4, "ben", &[])); - let chat = txn.get_or_insert_array("chat"); - chat.remove(txn, 0); - }) - .await; - // Edit a message in place (replace the element), as an edit/delete would. - pc.edit(|txn| { - let chat = txn.get_or_insert_array("chat"); - chat.remove(txn, 0); - let mut edited = message(2, "ann", &[]); - edited["text"] = json!("message 2, edited"); - edited["editedAt"] = json!(1_790_000_000_500_i64); - chat.insert(txn, 0, json_to_any(&edited)); - }) - .await; - let after = chat_of(&pc.doc); - assert_eq!( - after - .iter() - .map(|m| m["id"].as_str().unwrap()) - .collect::>(), - vec!["m2", "m3", "m4"] - ); - assert!( - eventually(T, || async { - chat_of(&app2.state_doc(&ann, &id).await) == after - }) - .await, - "server applied the cap and the edit" - ); - pc.close().await; - assert!(eventually(T, || async { copper_cloud_canvas::evict_room_now(cid) }).await); - assert_eq!(chat_of(&app2.state_doc(&ann, &id).await), after); - let (_, read) = app2.get(&ann, &format!("/canvases/{id}/read")).await; - assert_eq!(read["chat"][0]["text"], "message 2, edited"); - assert_eq!(read["chat"][0]["editedAt"], 1_790_000_000_500_i64); - - // Chat bytes are sealed in the update log like the rest of the document. - let blobs: Vec> = - sqlx::query_scalar(r#"SELECT "update" FROM canvas_updates WHERE canvas_id = $1"#) - .bind(cid) - .fetch_all(app2.db()) - .await - .unwrap(); - assert_ne!(blobs.len(), 0); - for b in blobs { - assert!( - !b.windows(9).any(|w| w == b"message 3"), - "chat stored in plaintext" - ); - } -} - -#[tokio::test] -async fn chat_survives_compaction_and_read_shows_the_last_50() { - let app = TestApp::new().await; - let ann = app.user("ann").await; - let id = app.canvas(&ann, "Chatty").await; - let cid = canvas_uuid(&id); - - // No chat yet: `chat` is present and empty. - let (st, read) = app.get(&ann, &format!("/canvases/{id}/read")).await; - assert_eq!(st, StatusCode::OK, "{read}"); - assert_eq!(read["chat"], json!([])); - - // One update per message, past the compaction threshold (200). - let n = 230; - let mut p = app.peer(&ann, &id).await; - for i in 0..n { - send(&mut p, message(i, "ann", &[])).await; - } - // A deleted message and a long one at the end. - let mut gone = message(n, "ann", &[]); - gone["deleted"] = json!(true); - gone["text"] = json!(""); - send(&mut p, gone).await; - let mut long = message(n + 1, "ann", &[]); - long["text"] = json!("z".repeat(900)); - send(&mut p, long).await; - assert!( - eventually(Duration::from_secs(20), || async { - chat_of(&app.state_doc(&ann, &id).await).len() == n + 2 - }) - .await, - "all messages applied" - ); - p.close().await; - let snapshot: Option = - sqlx::query_scalar("SELECT seq FROM canvas_snapshots WHERE canvas_id = $1") - .bind(cid) - .fetch_optional(app.db()) - .await - .unwrap(); - assert!(snapshot.is_some(), "a snapshot exists after > 200 updates"); - - assert!(eventually(T, || async { copper_cloud_canvas::evict_room_now(cid) }).await); - let reloaded = chat_of(&app.state_doc(&ann, &id).await); - assert_eq!(reloaded.len(), n + 2); - assert_eq!(reloaded[0], message(0, "ann", &[])); - assert_eq!(reloaded[n - 1], message(n - 1, "ann", &[])); - - // /read: the last 50 non-deleted messages, oldest first, text cut unless `full`. - let (_, read) = app.get(&ann, &format!("/canvases/{id}/read")).await; - let chat = read["chat"].as_array().unwrap(); - assert_eq!(chat.len(), 50); - assert_eq!(chat[0]["id"], format!("m{}", n + 1 - 50)); - assert_eq!(chat[48]["id"], format!("m{}", n - 1)); - assert_eq!(chat[49]["id"], format!("m{}", n + 1)); - assert!( - chat.iter().all(|m| m["id"] != format!("m{n}")), - "deleted hidden" - ); - assert_eq!(chat[49]["text"].as_str().unwrap().chars().count(), 501); - let (_, full) = app - .get(&ann, &format!("/canvases/{id}/read?full=true")) - .await; - assert_eq!(full["chat"][49]["text"].as_str().unwrap().len(), 900); - // Filters narrow shapes, not the chat. - let (_, only) = app - .get(&ann, &format!("/canvases/{id}/read?types=frame")) - .await; - assert_eq!(only["chat"].as_array().unwrap().len(), 50); -} diff --git a/crates/canvas/tests/checklist.rs b/crates/canvas/tests/checklist.rs deleted file mode 100644 index 7a0da20..0000000 --- a/crates/canvas/tests/checklist.rs +++ /dev/null @@ -1,291 +0,0 @@ -//! The checklist (RSVP) shape over REST `/ops` + `/read` and the live room: the same -//! props, validation and limits as the canvas page (`Canvas/src/canvas/checklist.ts`), -//! and picks on different rows made at the same moment all stick. - -#![allow(clippy::too_many_lines)] - -mod common; - -use std::time::Duration; - -use axum::http::StatusCode; -use common::TestApp; -use serde_json::{json, Value}; -use yrs::{Any, Map as _, Out, TransactionMut, WriteTxn as _}; - -const T: Duration = Duration::from_secs(5); - -fn checklist(read: &Value, id: &str) -> Value { - read["shapes"] - .as_array() - .unwrap() - .iter() - .find(|s| s["id"] == id) - .cloned() - .unwrap_or(Value::Null) -} - -fn picks(c: &Value) -> Vec { - c["rows"] - .as_array() - .unwrap() - .iter() - .map(|r| r["pick"].clone()) - .collect() -} - -#[tokio::test] -async fn ops_and_read_speak_checklist() { - let app = TestApp::new().await; - let ann = app.user("ann").await; - let id = app.canvas(&ann, "Team dinners").await; - let base = format!("/canvases/{id}"); - - let (s, r) = app - .post( - &ann, - &format!("{base}/ops"), - json!({ "as": { "id": "agent:planner", "name": "Planner" }, "ops": [ - { "op": "add", "shape": { "id": "tue", "type": "checklist", "x": 0, "y": 0, "title": "Tue · Oct 7", - "rows": ["Ann", "Bob", { "label": "Cy", "id": "cy" }], "picks": { "Ann": "yes" } } }, - { "op": "add", "shape": { "id": "todo", "type": "checklist", "columns": ["Done"], "rows": ["Book a table"] } } - ]}), - ) - .await; - assert_eq!(s, StatusCode::OK, "{r}"); - assert_eq!(r["applied"], 2, "{r}"); - - let (s, read) = app.get(&ann, &format!("{base}/read")).await; - assert_eq!(s, StatusCode::OK, "{read}"); - let tue = checklist(&read, "tue"); - assert_eq!(tue["type"], "checklist"); - assert_eq!(tue["title"], "Tue · Oct 7"); - assert_eq!(tue["columns"], json!(["Yes", "No"])); - assert_eq!(tue["color"], "green"); - assert_eq!(tue["by"], "Planner"); - assert_eq!(picks(&tue), [json!("Yes"), Value::Null, Value::Null]); - assert_eq!(tue["rows"][0]["by"], "Planner"); - assert!(tue["rows"][0]["at"].as_f64().unwrap() > 1.7e12); - assert_eq!( - tue["rows"][2], - json!({ "id": "cy", "label": "Cy", "pick": null }) - ); - assert_eq!(tue["tally"], json!({ "Yes": 1, "No": 0 })); - assert!(tue.get("picks").is_none(), "{tue}"); - let todo = checklist(&read, "todo"); - assert_eq!( - (todo["w"].clone(), todo["h"].clone()), - (json!(280), json!(97)) - ); - - // The pick is stored on its own key, attributed to the agent. - let doc = app.state_doc(&ann, &id).await; - assert_eq!( - common::shape_prop( - &doc, - "tue", - &format!("pick:{}", tue["rows"][0]["id"].as_str().unwrap()) - ) - .unwrap()["byId"], - "agent:planner" - ); - - // Update: picks by label or id (null clears), rows and columns replace the lists. - let (_, r) = app - .post( - &ann, - &format!("{base}/ops"), - json!({ "ops": [ - { "op": "update", "id": "tue", "patch": { "picks": { "Ann": null, "cy": "No", "Bob": true } } }, - { "op": "update", "id": "todo", "patch": { "picks": { "book a table": "done" }, "title": "Before Friday" } } - ]}), - ) - .await; - assert!(r["errors"].as_array().unwrap().is_empty(), "{r}"); - let (_, read) = app.get(&ann, &format!("{base}/read?types=checklist")).await; - let tue = checklist(&read, "tue"); - assert_eq!(picks(&tue), [Value::Null, json!("Yes"), json!("No")]); - assert_eq!(tue["rows"][1]["by"], "ann", "the caller, without `as`"); - let todo = checklist(&read, "todo"); - assert_eq!(todo["title"], "Before Friday"); - assert_eq!(todo["tally"], json!({ "Done": 1 })); - - let (_, r) = app - .post( - &ann, - &format!("{base}/ops"), - json!({ "ops": [ - { "op": "update", "id": "tue", "patch": { "rows": ["Cy", "Dee", "Bob"], "columns": ["Yes", "No", "Maybe"] } } - ]}), - ) - .await; - assert!(r["errors"].as_array().unwrap().is_empty(), "{r}"); - let (_, read) = app.get(&ann, &format!("{base}/read")).await; - let tue = checklist(&read, "tue"); - let labels: Vec<&str> = tue["rows"] - .as_array() - .unwrap() - .iter() - .map(|r| r["label"].as_str().unwrap()) - .collect(); - assert_eq!(labels, ["Cy", "Dee", "Bob"]); - assert_eq!(picks(&tue), [json!("No"), Value::Null, json!("Yes")]); - assert_eq!(tue["tally"], json!({ "Yes": 1, "No": 1, "Maybe": 0 })); - assert_eq!(tue["w"], 14 * 2 + 168 + 64 * 3); - - // Validation and limits, reported per op like the page. - let too_many: Vec = (0..61).map(|i| format!("Person {i}")).collect(); - let (_, r) = app - .post( - &ann, - &format!("{base}/ops"), - json!({ "ops": [ - { "op": "update", "id": "tue", "patch": { "picks": { "Zed": "Yes" } } }, - { "op": "update", "id": "tue", "patch": { "picks": { "Cy": "Later" } } }, - { "op": "add", "shape": { "type": "checklist", "rows": too_many } }, - { "op": "add", "shape": { "type": "checklist", "columns": ["Yes", "YES"] } }, - { "op": "add", "shape": { "type": "checklist", "columns": [] } }, - { "op": "add", "shape": { "type": "checklist", "rows": [{ "label": "x", "id": "no spaces" }] } }, - { "op": "add", "shape": { "type": "checklist", "url": "https://example.com" } }, - { "op": "add", "shape": { "type": "poll" } } - ]}), - ) - .await; - let errors: Vec<&str> = r["errors"] - .as_array() - .unwrap() - .iter() - .map(|e| e["error"].as_str().unwrap()) - .collect(); - assert_eq!(r["applied"], 0, "{r}"); - assert_eq!(errors[0], "`picks`: no row \"Zed\""); - assert_eq!(errors[1], "`picks`: no column \"Later\" (Yes, No, Maybe)"); - assert_eq!(errors[2], "`rows`: at most 60 rows"); - assert_eq!(errors[3], "`columns` has \"YES\" twice"); - assert_eq!(errors[4], "`columns` must be a list of 1 to 4 names"); - assert_eq!(errors[5], "`rows`: an id must be 1–32 of A–Z a–z 0–9 _ -"); - assert!( - errors[6].starts_with("unknown prop `url` for checklist"), - "{}", - errors[6] - ); - assert_eq!( - errors[7], - "`shape.type` must be sticky, text, frame, arrow, image, link or checklist" - ); - let (_, read) = app.get(&ann, &format!("{base}/read")).await; - assert_eq!(read["count"], 2); -} - -/// Every row's pick through REST at once, one request per person: all of them stick. -#[tokio::test] -async fn concurrent_rest_picks_on_different_rows_all_stick() { - let app = TestApp::new().await; - let owner = app.user("host").await; - let id = app.canvas(&owner, "RSVP").await; - let people = ["ann", "bob", "cy", "dee", "eve", "fay"]; - let rows: Vec = people - .iter() - .map(|p| json!({ "label": p, "id": p })) - .collect(); - let (s, r) = app - .post( - &owner, - &format!("/canvases/{id}/ops"), - json!({ "ops": [{ "op": "add", "shape": { "id": "c", "type": "checklist", "rows": rows } }] }), - ) - .await; - assert_eq!(s, StatusCode::OK, "{r}"); - let mut users = Vec::new(); - for p in people { - let u = app.user(p).await; - app.share(&owner, &id, &u).await; - users.push(u); - } - let calls = users.iter().enumerate().map(|(i, u)| { - let app = &app; - let id = &id; - async move { - let col = if i % 2 == 0 { "Yes" } else { "No" }; - let body = json!({ "ops": [{ "op": "update", "id": "c", "patch": { "picks": { people[i]: col } } }] }); - app.post(u, &format!("/canvases/{id}/ops"), body).await - } - }); - for (s, r) in futures::future::join_all(calls).await { - assert_eq!(s, StatusCode::OK, "{r}"); - assert_eq!(r["applied"], 1, "{r}"); - } - let (_, read) = app.get(&owner, &format!("/canvases/{id}/read")).await; - let c = checklist(&read, "c"); - assert_eq!( - picks(&c), - [ - json!("Yes"), - json!("No"), - json!("Yes"), - json!("No"), - json!("Yes"), - json!("No") - ] - ); - assert_eq!(c["tally"], json!({ "Yes": 3, "No": 3 })); - for (row, p) in c["rows"].as_array().unwrap().iter().zip(people) { - assert_eq!(row["by"], p, "each pick is its picker's"); - } -} - -fn pick(txn: &mut TransactionMut, shape: &str, row: &str, col: &str, by: &str) { - let shapes = txn.get_or_insert_map("shapes"); - let Some(Out::YMap(m)) = shapes.get(txn, shape) else { - panic!("no shape {shape}") - }; - let value = Any::from(std::collections::HashMap::from([ - ("col".to_owned(), Any::from(col)), - ("by".to_owned(), Any::from(by)), - ("byId".to_owned(), Any::from(by)), - ("at".to_owned(), Any::from(1_700_000_000_000_i64)), - ])); - m.insert(txn, format!("pick:{row}"), value); -} - -/// Two editors in the live room click different rows at the same moment (each writes -/// before hearing the other): both picks stick, for both of them and on the server. -#[tokio::test] -async fn live_peers_picking_different_rows_at_once_both_stick() { - let app = TestApp::new().await; - let a = app.user("ada").await; - let b = app.user("grace").await; - let id = app.canvas(&a, "Live RSVP").await; - app.share(&a, &id, &b).await; - app.post( - &a, - &format!("/canvases/{id}/ops"), - json!({ "ops": [{ "op": "add", "shape": { "id": "c", "type": "checklist", - "rows": [{ "label": "Ada", "id": "ada" }, { "label": "Grace", "id": "grace" }] } }] }), - ) - .await; - let mut pa = app.peer(&a, &id).await; - let mut pb = app.peer(&b, &id).await; - assert!(pa.pump_until(T, |p| p.shape_count() == 1).await); - assert!(pb.pump_until(T, |p| p.shape_count() == 1).await); - // Neither has heard the other's click when it makes its own. - let (ea, eb) = tokio::join!( - pa.edit(|txn| pick(txn, "c", "ada", "Yes", "ada")), - pb.edit(|txn| pick(txn, "c", "grace", "No", "grace")), - ); - let ((), ()) = (ea, eb); - let both = |p: &common::Peer| { - p.shape_prop("c", "pick:ada") - .is_some_and(|v| v["col"] == "Yes") - && p.shape_prop("c", "pick:grace") - .is_some_and(|v| v["col"] == "No") - }; - assert!(pa.pump_until(T, both).await, "A sees both picks"); - assert!(pb.pump_until(T, both).await, "B sees both picks"); - let (_, read) = app.get(&a, &format!("/canvases/{id}/read")).await; - let c = checklist(&read, "c"); - assert_eq!(picks(&c), [json!("Yes"), json!("No")], "{c}"); - assert_eq!(c["tally"], json!({ "Yes": 1, "No": 1 })); - pa.close().await; - pb.close().await; -} diff --git a/crates/canvas/tests/common/mod.rs b/crates/canvas/tests/common/mod.rs deleted file mode 100644 index 012b314..0000000 --- a/crates/canvas/tests/common/mod.rs +++ /dev/null @@ -1,514 +0,0 @@ -//! Shared integration-test harness: real Postgres (`COPPER_CLOUD_TEST_CANVAS_DATABASE_URL`, -//! default `postgres://localhost:5432/copper_cloud_test_canvas`), the real core app + canvas -//! router on an ephemeral port, users/sessions inserted directly, and a minimal y-websocket -//! client built on `yrs` + tokio-tungstenite. - -#![allow(dead_code, clippy::missing_panics_doc)] - -use std::collections::HashMap; -use std::net::SocketAddr; -use std::path::Path; -use std::sync::Arc; -use std::time::Duration; - -use axum::body::Body; -use axum::http::{Method, Request, StatusCode}; -use base64::Engine as _; -use copper_cloud_core::config::Config; -use copper_cloud_core::crypto::Crypto; -use copper_cloud_core::ids; -use copper_cloud_core::state::{AppState, SharedState}; -use futures::{SinkExt as _, StreamExt as _}; -use http_body_util::BodyExt as _; -use serde_json::Value; -use sqlx::postgres::PgPoolOptions; -use sqlx::PgPool; -use tokio::net::TcpStream; -use tokio_tungstenite::tungstenite::client::IntoClientRequest as _; -use tokio_tungstenite::tungstenite::protocol::frame::coding::CloseCode; -use tokio_tungstenite::tungstenite::Message as WsMessage; -use tokio_tungstenite::{MaybeTlsStream, WebSocketStream}; -use tower::ServiceExt as _; -use uuid::Uuid; -use yrs::encoding::read::Cursor; -use yrs::sync::awareness::AwarenessUpdateEntry; -use yrs::sync::{AwarenessUpdate, Message, MessageReader, SyncMessage}; -use yrs::updates::decoder::{Decode as _, DecoderV1}; -use yrs::updates::encoder::Encode as _; -use yrs::{ - ClientID, Doc, Map as _, Out, ReadTxn as _, StateVector, Transact as _, TransactionMut, Update, -}; - -pub const DATABASE_URL: &str = "postgres://localhost:5432/copper_cloud_test_canvas"; - -/// The test database: `COPPER_CLOUD_TEST_CANVAS_DATABASE_URL`, else [`DATABASE_URL`] — so -/// parallel checkouts can each test against their own database. -pub fn database_url() -> String { - std::env::var("COPPER_CLOUD_TEST_CANVAS_DATABASE_URL") - .unwrap_or_else(|_| DATABASE_URL.to_owned()) -} - -static MIGRATED: tokio::sync::OnceCell<()> = tokio::sync::OnceCell::const_new(); - -async fn migrate(pool: &PgPool) { - MIGRATED - .get_or_init(|| async { - let migrator = sqlx::migrate::Migrator::new(Path::new("../copper-cloud/migrations")) - .await - .expect("reading migrations"); - if let Err(e) = migrator.run(pool).await { - // A migration changed during development: start from a clean schema. - eprintln!("migrations failed ({e}); resetting the test schema"); - sqlx::raw_sql("DROP SCHEMA public CASCADE; CREATE SCHEMA public;") - .execute(pool) - .await - .expect("resetting schema"); - migrator.run(pool).await.expect("running migrations"); - } - }) - .await; -} - -/// One running server. -pub struct TestApp { - pub state: SharedState, - pub router: axum::Router, - pub addr: SocketAddr, - pub instance_key: String, -} - -/// A signed-in user. -#[derive(Clone, Debug)] -pub struct User { - pub id: Uuid, - pub email: String, - pub token: String, -} - -impl TestApp { - pub async fn new() -> Self { - Self::with_config(Config::for_tests(&database_url())).await - } - - /// A second server over the same database and config (same master key): what a restart - /// of this one sees. Rooms are process-wide, so evict them first for a cold start. - pub async fn restart(&self) -> Self { - Self::with_config(Config::clone(&self.state.cfg)).await - } - - async fn with_config(cfg: Config) -> Self { - let pool = PgPoolOptions::new() - .max_connections(4) - .acquire_timeout(Duration::from_secs(30)) - .connect(&cfg.database_url) - .await - .expect("connecting to the canvas test database (createdb copper_cloud_test_canvas)"); - migrate(&pool).await; - let instance_key = cfg.instance_key.clone(); - let state = AppState::new(pool, cfg); - let router = copper_cloud_core::app_with( - state.clone(), - copper_cloud_canvas::router(), - copper_cloud_canvas::landing_router(), - ); - let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap(); - let addr = listener.local_addr().unwrap(); - let serve = router - .clone() - .into_make_service_with_connect_info::(); - tokio::spawn(async move { - let _ = axum::serve(listener, serve).await; - }); - Self { - state, - router, - addr, - instance_key, - } - } - - pub fn db(&self) -> &PgPool { - &self.state.db - } - - /// Inserts a user + device + session directly (core schema) and returns its bearer token. - pub async fn user(&self, name: &str) -> User { - let id = ids::uuid_v7(); - let email = format!("{name}-{}@example.test", &id.simple().to_string()[20..]); - let wrapped = self.state.crypto.wrap_key(&Crypto::new_key()); - sqlx::query( - "INSERT INTO users (id, email, display_name, password_hash, data_key_wrapped) - VALUES ($1, $2, $3, 'not-a-real-hash', $4)", - ) - .bind(id) - .bind(&email) - .bind(name) - .bind(&wrapped) - .execute(self.db()) - .await - .unwrap(); - let device = ids::uuid_v7(); - sqlx::query("INSERT INTO devices (user_id, id, name) VALUES ($1, $2, 'test')") - .bind(id) - .bind(device) - .execute(self.db()) - .await - .unwrap(); - let token = ids::random_token(); - sqlx::query( - "INSERT INTO sessions (id, user_id, device_id, token_sha256, expires_at) - VALUES ($1, $2, $3, $4, now() + interval '1 day')", - ) - .bind(ids::uuid_v7()) - .bind(id) - .bind(device) - .bind(&ids::sha256(token.as_bytes())[..]) - .execute(self.db()) - .await - .unwrap(); - User { id, email, token } - } - - /// One REST call through the full app (instance gate included). - pub async fn call( - &self, - user: Option<&User>, - method: Method, - path: &str, - body: Option, - ) -> (StatusCode, Value) { - let mut req = Request::builder() - .method(method) - .uri(format!("/v1{path}")) - .header("x-copper-instance", &self.instance_key); - if let Some(u) = user { - req = req.header("authorization", format!("Bearer {}", u.token)); - } - let req = match body { - Some(b) => req - .header("content-type", "application/json") - .body(Body::from(serde_json::to_vec(&b).unwrap())), - None => req.body(Body::empty()), - } - .unwrap(); - let resp = self.router.clone().oneshot(req).await.unwrap(); - let status = resp.status(); - let bytes = resp.into_body().collect().await.unwrap().to_bytes(); - let value = if bytes.is_empty() { - Value::Null - } else { - serde_json::from_slice(&bytes) - .unwrap_or_else(|_| Value::String(String::from_utf8_lossy(&bytes).into())) - }; - (status, value) - } - - pub async fn get(&self, user: &User, path: &str) -> (StatusCode, Value) { - self.call(Some(user), Method::GET, path, None).await - } - - pub async fn post(&self, user: &User, path: &str, body: Value) -> (StatusCode, Value) { - self.call(Some(user), Method::POST, path, Some(body)).await - } - - pub async fn patch(&self, user: &User, path: &str, body: Value) -> (StatusCode, Value) { - self.call(Some(user), Method::PATCH, path, Some(body)).await - } - - pub async fn delete(&self, user: &User, path: &str) -> (StatusCode, Value) { - self.call(Some(user), Method::DELETE, path, None).await - } - - /// Creates a shared canvas owned by `user`; returns its id. - pub async fn canvas(&self, user: &User, name: &str) -> String { - let (s, v) = self - .post(user, "/canvases", serde_json::json!({ "name": name })) - .await; - assert_eq!(s, StatusCode::CREATED, "{v}"); - v["id"].as_str().unwrap().to_owned() - } - - /// Makes `guest` an editor of `canvas` via invite + accept. - pub async fn share(&self, owner: &User, canvas: &str, guest: &User) { - let (s, inv) = self - .post( - owner, - &format!("/canvases/{canvas}/invites"), - serde_json::json!({ "email": guest.email }), - ) - .await; - assert_eq!(s, StatusCode::CREATED, "{inv}"); - let id = inv["id"].as_str().unwrap(); - let (s, v) = self - .post(guest, &format!("/invites/{id}/accept"), Value::Null) - .await; - assert_eq!(s, StatusCode::OK, "{v}"); - } - - /// The canvas document as served by `GET /state`, decoded into a fresh doc. - pub async fn state_doc(&self, user: &User, canvas: &str) -> Doc { - let (s, v) = self.get(user, &format!("/canvases/{canvas}/state")).await; - assert_eq!(s, StatusCode::OK, "{v}"); - let bytes = base64::engine::general_purpose::STANDARD - .decode(v["state"].as_str().unwrap()) - .unwrap(); - let doc = Doc::new(); - doc.transact_mut() - .apply_update(Update::decode_v1(&bytes).unwrap()) - .unwrap(); - doc - } - - pub fn ws_url(&self, canvas: &str, token: &str) -> String { - format!("ws://{}/v1/canvases/{canvas}/ws?token={token}", self.addr) - } - - /// Opens a raw WebSocket (the handshake result is returned as-is). - pub async fn ws_raw( - &self, - canvas: &str, - token: &str, - ) -> Result>, tokio_tungstenite::tungstenite::Error> - { - let mut req = self.ws_url(canvas, token).into_client_request().unwrap(); - req.headers_mut() - .insert("x-copper-instance", self.instance_key.parse().unwrap()); - tokio_tungstenite::connect_async(req) - .await - .map(|(ws, _)| ws) - } - - /// Connects a y-websocket peer and completes the initial sync. - pub async fn peer(&self, user: &User, canvas: &str) -> Peer { - let ws = self.ws_raw(canvas, &user.token).await.expect("ws connect"); - let mut p = Peer { - doc: Doc::new(), - ws, - awareness: HashMap::new(), - closed: None, - synced: false, - }; - let sv = p.doc.transact().state_vector(); - p.send(&Message::Sync(SyncMessage::SyncStep1(sv))).await; - assert!( - p.pump_until(Duration::from_secs(5), |p| p.synced).await, - "initial sync" - ); - p - } -} - -/// A minimal y-websocket client. -pub struct Peer { - pub doc: Doc, - pub ws: WebSocketStream>, - /// Latest awareness JSON per client id (`"null"` once removed). - pub awareness: HashMap, - /// Close code received from the server. - pub closed: Option, - synced: bool, -} - -impl Peer { - pub async fn send(&mut self, msg: &Message) { - self.ws - .send(WsMessage::Binary(msg.encode_v1().into())) - .await - .unwrap(); - } - - /// Applies a local change and sends the resulting update. - pub async fn edit(&mut self, f: impl FnOnce(&mut TransactionMut)) { - let before = self.doc.transact().state_vector(); - { - let mut txn = self.doc.transact_mut(); - f(&mut txn); - } - let update = self.doc.transact().encode_state_as_update_v1(&before); - self.send(&Message::Sync(SyncMessage::Update(update))).await; - } - - /// Publishes this peer's awareness state. - pub async fn set_awareness(&mut self, clock: u32, json: &str) { - let id = self.doc.client_id(); - let update = AwarenessUpdate { - clients: HashMap::from([( - id, - AwarenessUpdateEntry { - clock, - json: json.into(), - }, - )]), - }; - self.send(&Message::Awareness(update)).await; - } - - pub fn client_id(&self) -> u64 { - self.doc.client_id().get() - } - - fn handle(&mut self, bytes: &[u8]) -> Vec { - let mut replies = Vec::new(); - let mut decoder = DecoderV1::new(Cursor::new(bytes)); - for msg in MessageReader::new(&mut decoder) { - match msg.expect("server sent an undecodable message") { - Message::Sync(SyncMessage::SyncStep1(sv)) => { - let u = self.doc.transact().encode_state_as_update_v1(&sv); - replies.push(Message::Sync(SyncMessage::SyncStep2(u))); - } - Message::Sync(SyncMessage::SyncStep2(u)) => { - self.doc - .transact_mut() - .apply_update(Update::decode_v1(&u).unwrap()) - .unwrap(); - self.synced = true; - } - Message::Sync(SyncMessage::Update(u)) => { - self.doc - .transact_mut() - .apply_update(Update::decode_v1(&u).unwrap()) - .unwrap(); - } - Message::Awareness(a) => { - for (id, e) in a.clients { - self.awareness.insert(id.get(), e.json.to_string()); - } - } - Message::AwarenessQuery | Message::Auth(_) | Message::Custom(..) => {} - } - } - replies - } - - /// Reads and handles server messages until `pred` holds (true) or `timeout` passes. - pub async fn pump_until(&mut self, timeout: Duration, pred: impl Fn(&Peer) -> bool) -> bool { - let deadline = tokio::time::Instant::now() + timeout; - loop { - if pred(self) { - return true; - } - let next = tokio::time::timeout_at(deadline, self.ws.next()).await; - let Ok(Some(Ok(msg))) = next else { - return pred(self); - }; - match msg { - WsMessage::Binary(b) => { - for reply in self.handle(&b) { - self.send(&reply).await; - } - } - WsMessage::Close(frame) => { - self.closed = Some(frame.map_or(1005, |f| u16::from(f.code))); - return pred(self); - } - _ => {} - } - } - } - - /// Waits for the server to close the socket; returns the close code. - pub async fn wait_closed(&mut self, timeout: Duration) -> Option { - self.pump_until(timeout, |p| p.closed.is_some()).await; - self.closed - } - - pub async fn close(mut self) { - let _ = self - .ws - .close(Some(tokio_tungstenite::tungstenite::protocol::CloseFrame { - code: CloseCode::Normal, - reason: "bye".into(), - })) - .await; - // Drain until the server acknowledges. - let _ = tokio::time::timeout(Duration::from_secs(2), async { - while let Some(Ok(_)) = self.ws.next().await {} - }) - .await; - } - - /// Text of a sticky (a Y.Map in `shapes`) as this peer sees it. - pub fn shape_prop(&self, id: &str, key: &str) -> Option { - shape_prop(&self.doc, id, key) - } - - pub fn shape_count(&self) -> u32 { - shape_count(&self.doc) - } -} - -pub fn shape_prop(doc: &Doc, id: &str, key: &str) -> Option { - let txn = doc.transact(); - let shapes = txn.get_map("shapes")?; - let Some(Out::YMap(m)) = shapes.get(&txn, id) else { - return None; - }; - m.get(&txn, key) - .map(|o| copper_cloud_canvas::schema::out_to_json(&txn, &o)) -} - -pub fn shape_count(doc: &Doc) -> u32 { - let txn = doc.transact(); - txn.get_map("shapes").map_or(0, |m| m.len(&txn)) -} - -/// Adds a sticky as a Y.Map of props (what the TS page does). -pub fn add_sticky(txn: &mut TransactionMut, id: &str, text: &str, x: f64) { - use yrs::{Any, MapPrelim, WriteTxn as _}; - let shapes = txn.get_or_insert_map("shapes"); - let props: Vec<(&str, Any)> = vec![ - ("id", id.into()), - ("type", "sticky".into()), - ("x", x.into()), - ("y", 0.0.into()), - ("w", 200.0.into()), - ("h", 140.0.into()), - ("color", "yellow".into()), - ("text", text.into()), - ("by", "peer".into()), - ]; - shapes.insert(txn, id, MapPrelim::from_iter(props)); -} - -/// Sets one prop of an existing shape. -pub fn set_prop(txn: &mut TransactionMut, id: &str, key: &str, value: &str) { - use yrs::WriteTxn as _; - let shapes = txn.get_or_insert_map("shapes"); - if let Some(Out::YMap(m)) = shapes.get(txn, id) { - m.insert(txn, key, value); - } -} - -pub fn client_id(v: u64) -> ClientID { - ClientID::new(v) -} - -/// Polls `f` until it returns true or `timeout` passes. -pub async fn eventually(timeout: Duration, mut f: F) -> bool -where - F: FnMut() -> Fut, - Fut: std::future::Future, -{ - let deadline = tokio::time::Instant::now() + timeout; - loop { - if f().await { - return true; - } - if tokio::time::Instant::now() >= deadline { - return false; - } - tokio::time::sleep(Duration::from_millis(25)).await; - } -} - -pub fn sv_b64(doc: &Doc) -> String { - base64::engine::general_purpose::STANDARD.encode(doc.transact().state_vector().encode_v1()) -} - -pub fn empty_sv() -> StateVector { - StateVector::default() -} - -pub fn arc(t: T) -> Arc { - Arc::new(t) -} diff --git a/crates/canvas/tests/js_interop.rs b/crates/canvas/tests/js_interop.rs deleted file mode 100644 index 50f91fb..0000000 --- a/crates/canvas/tests/js_interop.rs +++ /dev/null @@ -1,236 +0,0 @@ -//! Cross-implementation check against the real canvas page code (Yjs + y-websocket + -//! `Canvas/src/ops.ts`), run with Bun. Ignored by default; run with -//! -//! ```sh -//! COPPER_CANVAS_DIR=~/src/copper-canvasweb/Canvas \ -//! cargo test -p copper-cloud-canvas --test js_interop -- --ignored -//! ``` -//! -//! It proves: a stock `y-websocket` `WebsocketProvider` syncs through the room both ways, -//! awareness relays between JS peers, ops applied by the page reach the server and vice -//! versa, and the server's `/read` equals the page's `readCanvas` on the same document — -//! checklists included (the page must have `src/canvas/checklist.ts`). - -#![allow(clippy::too_many_lines)] - -mod common; - -use std::path::PathBuf; -use std::process::Stdio; -use std::time::Duration; - -use axum::http::StatusCode; -use common::TestApp; -use serde_json::{json, Value}; -use tokio::io::{AsyncBufReadExt as _, BufReader}; - -const SCRIPT: &str = r" -import * as Y from 'yjs' -import { WebsocketProvider } from 'y-websocket' -const CANVAS = process.env.CANVAS_DIR -const { CanvasStore } = await import(`${CANVAS}/src/canvas/doc.ts`) -const { applyOps, readCanvas } = await import(`${CANVAS}/src/ops.ts`) -const { WS_BASE, CANVAS_ID, TOKEN_A, TOKEN_B, KEY } = process.env - -class WS extends WebSocket { - constructor(url, _protocols) { - super(url, { headers: { 'x-copper-instance': KEY } }) - } -} -function connect(token) { - const doc = new Y.Doc() - const provider = new WebsocketProvider(WS_BASE, `${CANVAS_ID}/ws`, doc, { - WebSocketPolyfill: WS, - params: { token }, - disableBc: true, - }) - return { doc, provider, store: new CanvasStore(doc) } -} -const waitFor = (what, pred, ms = 8000) => - new Promise((resolve, reject) => { - const t0 = Date.now() - const tick = () => { - if (pred()) resolve(true) - else if (Date.now() - t0 > ms) reject(new Error(`timeout: ${what}`)) - else setTimeout(tick, 20) - } - tick() - }) -const out = (k, v) => console.log(JSON.stringify({ k, v })) -const read = s => - readCanvas({ store: s.store, canvas: { id: CANVAS_ID, name: 'Interop', kind: 'shared' }, viewport: null, selection: [] }) - -try { - const a = connect(TOKEN_A) - const b = connect(TOKEN_B) - await waitFor('sync', () => a.provider.synced && b.provider.synced) - out('read_initial', read(a)) - const r = applyOps( - { store: a.store, viewportCenter: () => ({ x: 0, y: 0 }) }, - { - ops: [ - { op: 'add', shape: { id: 'js1', type: 'sticky', text: 'from js', x: 0, y: 600 } }, - { op: 'update', id: 'srv1', patch: { text: 'edited by js' } }, - { op: 'connect', id: 'jsarrow', from: 'js1', to: 'srv1', label: 'js arrow' }, - { op: 'add', shape: { id: 'jsframe', type: 'frame', title: 'Group', x: -100, y: 500, w: 600, h: 400 } }, - { op: 'update', id: 'srvlist', patch: { picks: { Bob: 'No' }, rows: ['Ann', 'Bob', 'Cy', 'Dee'], columns: ['Yes', 'No', 'Maybe'] } }, - { op: 'add', shape: { id: 'jslist', type: 'checklist', columns: ['Done'], rows: ['Book a table'], x: 1200, y: 600 } }, - ], - as: { name: 'JS' }, - } - ) - out('apply', r) - await waitFor('b sees js edits', () => b.store.get('js1') && b.store.get('srv1')?.text === 'edited by js') - a.provider.awareness.setLocalStateField('user', { name: 'Ann JS', color: '#123456' }) - await waitFor('awareness', () => [...b.provider.awareness.getStates().values()].some(s => s.user?.name === 'Ann JS')) - out('awareness', b.provider.awareness.getStates().size) - out('ready', true) - await waitFor( - 'server ops', - () => - a.store.get('srv2') !== undefined && - b.store.get('srv1')?.text === 'edited by js, then server' && - Object.keys(a.store.get('jslist')?.picks ?? {}).length === 1 && - Object.keys(a.store.get('srvlist')?.picks ?? {}).length === 3, - 10000 - ) - await new Promise(r => setTimeout(r, 200)) - out('read_final', read(a)) - a.provider.destroy() - b.provider.destroy() - process.exit(0) -} catch (e) { - out('error', String(e)) - process.exit(1) -} -"; - -fn shapes_of(read: &Value) -> Vec { - read["shapes"].as_array().cloned().unwrap_or_default() -} - -#[tokio::test] -#[ignore = "needs bun and COPPER_CANVAS_DIR (the canvas page sources with node_modules)"] -async fn real_y_websocket_and_page_ops_interoperate() { - let Some(canvas_dir) = std::env::var_os("COPPER_CANVAS_DIR").map(PathBuf::from) else { - eprintln!("COPPER_CANVAS_DIR not set; skipping"); - return; - }; - let canvas_dir = canvas_dir.canonicalize().unwrap(); - let work = std::env::temp_dir().join(format!("copper-canvas-interop-{}", uuid::Uuid::now_v7())); - std::fs::create_dir_all(&work).unwrap(); - std::os::unix::fs::symlink(canvas_dir.join("node_modules"), work.join("node_modules")).unwrap(); - std::fs::write(work.join("interop.mjs"), SCRIPT).unwrap(); - - let app = TestApp::new().await; - let a = app.user("ann").await; - let b = app.user("ben").await; - let id = app.canvas(&a, "Interop").await; - app.share(&a, &id, &b).await; - let ops = |body: Value| { - let app = &app; - let a = &a; - let id = &id; - async move { - let (s, r) = app.post(a, &format!("/canvases/{id}/ops"), body).await; - assert_eq!(s, StatusCode::OK, "{r}"); - assert!(r["errors"].as_array().unwrap().is_empty(), "{r}"); - r - } - }; - ops(json!({ "as": { "name": "Server agent" }, "ops": [ - { "op": "add", "shape": { "id": "srv1", "type": "sticky", "text": "from server", "x": 400, "y": 0 } }, - { "op": "add", "shape": { "id": "srvlink", "type": "link", "url": "example.com/path", "x": 800, "y": 0 } }, - { "op": "connect", "id": "srvarrow", "from": "srv1", "to": "srvlink", "fromSide": "right" }, - { "op": "add", "shape": { "id": "srvlist", "type": "checklist", "title": "Dinner", "rows": ["Ann", "Bob", "Cy"], - "picks": { "Ann": "Yes" }, "x": 0, "y": -600 } } - ]})) - .await; - - let mut child = tokio::process::Command::new("bun") - .arg("run") - .arg(work.join("interop.mjs")) - .current_dir(&work) - .env("CANVAS_DIR", &canvas_dir) - .env("WS_BASE", format!("ws://{}/v1/canvases", app.addr)) - .env("CANVAS_ID", &id) - .env("TOKEN_A", &a.token) - .env("TOKEN_B", &b.token) - .env("KEY", &app.instance_key) - .stdout(Stdio::piped()) - .stderr(Stdio::inherit()) - .kill_on_drop(true) - .spawn() - .expect("bun on PATH"); - let mut lines = BufReader::new(child.stdout.take().unwrap()).lines(); - let mut got: std::collections::HashMap = std::collections::HashMap::new(); - let deadline = tokio::time::Instant::now() + Duration::from_secs(40); - while let Ok(Ok(Some(line))) = tokio::time::timeout_at(deadline, lines.next_line()).await { - let Ok(msg) = serde_json::from_str::(&line) else { - eprintln!("bun: {line}"); - continue; - }; - let k = msg["k"].as_str().unwrap_or_default().to_owned(); - assert_ne!(k, "error", "{}", msg["v"]); - if k == "ready" { - ops(json!({ "ops": [ - { "op": "add", "shape": { "id": "srv2", "type": "text", "text": "late", "x": 0, "y": -300 } }, - { "op": "update", "id": "srv1", "patch": { "text": "edited by js, then server" } }, - { "op": "update", "id": "jslist", "patch": { "picks": { "book a table": true } } }, - { "op": "update", "id": "srvlist", "patch": { "picks": { "Dee": "maybe" } } } - ]})) - .await; - } - got.insert(k, msg["v"].clone()); - if got.contains_key("read_final") { - break; - } - } - let status = tokio::time::timeout(Duration::from_secs(10), child.wait()).await; - let _ = std::fs::remove_dir_all(&work); - assert!( - matches!(status, Ok(Ok(s)) if s.success()), - "bun exited badly: {status:?}" - ); - - // The page saw the server-made shapes exactly as the server reads them. - let initial = &got["read_initial"]; - assert_eq!(shapes_of(initial).len(), 4, "{initial}"); - assert_eq!(got["apply"]["applied"], 6, "{}", got["apply"]); - assert!(got["awareness"].as_u64().unwrap() >= 2); - - // Server and page agree on the final document, shape for shape. - let (s, server) = app.get(&a, &format!("/canvases/{id}/read?full=true")).await; - assert_eq!(s, StatusCode::OK); - let page = &got["read_final"]; - assert_eq!( - server["count"], page["count"], - "server {server}\npage {page}" - ); - let (ss, ps) = (shapes_of(&server), shapes_of(page)); - assert_eq!(ss.len(), ps.len()); - for (sv, pv) in ss.iter().zip(&ps) { - assert_eq!(sv, pv, "shape differs between server and page"); - } - let srv1 = ss.iter().find(|s| s["id"] == "srv1").unwrap(); - assert_eq!(srv1["text"], "edited by js, then server"); - let js1 = ss.iter().find(|s| s["id"] == "js1").unwrap(); - assert_eq!(js1["by"], "JS"); - assert_eq!(js1["frame"], "jsframe"); - // Checklists: picks made on both sides, read the same by both. - let srvlist = ss.iter().find(|s| s["id"] == "srvlist").unwrap(); - let picked: Vec<&Value> = srvlist["rows"] - .as_array() - .unwrap() - .iter() - .map(|r| &r["pick"]) - .collect(); - assert_eq!( - picked, - [&json!("Yes"), &json!("No"), &Value::Null, &json!("Maybe")] - ); - assert_eq!(srvlist["tally"], json!({ "Yes": 1, "No": 1, "Maybe": 1 })); - let jslist = ss.iter().find(|s| s["id"] == "jslist").unwrap(); - assert_eq!(jslist["rows"][0]["pick"], "Done"); - assert_eq!(jslist["rows"][0]["by"], "ann"); -} diff --git a/crates/canvas/tests/mentions.rs b/crates/canvas/tests/mentions.rs deleted file mode 100644 index 7bc5a14..0000000 --- a/crates/canvas/tests/mentions.rs +++ /dev/null @@ -1,432 +0,0 @@ -//! Chat @mentions (0.6.0): `POST /canvases/{id}/mentions`, `GET /mentions`, -//! `POST /mentions/read` — authorization, target filtering, idempotency, read marking, SSE -//! events, sealing, rate limit. - -#![allow(clippy::too_many_lines)] - -mod common; - -use axum::http::{Method, StatusCode}; -use common::{TestApp, User}; -use copper_cloud_canvas::mentions::{ - MAX_MENTIONS_LIMIT, MAX_MENTION_TARGETS, MENTION_POSTS_PER_MINUTE, -}; -use copper_cloud_core::events::Event; -use serde_json::{json, Value}; -use tokio::sync::broadcast::Receiver; -use uuid::Uuid; - -/// Drains the mention-related `canvas` events queued for one subscriber: `(canvas_id, kind)`. -fn mention_events(rx: &mut Receiver) -> Vec<(String, String)> { - let mut out = Vec::new(); - while let Ok(ev) = rx.try_recv() { - if let Event::Canvas { canvas_id, kind } = ev { - if kind.starts_with("mention") { - out.push((canvas_id.to_string(), kind)); - } - } - } - out -} - -fn ids(v: &Value) -> Vec { - v.as_array() - .unwrap() - .iter() - .map(|x| x.as_str().unwrap().to_owned()) - .collect() -} - -fn s(u: &User) -> String { - u.id.to_string() -} - -async fn mention( - app: &TestApp, - from: &User, - canvas: &str, - message_id: &str, - to: &[&User], - excerpt: &str, -) -> Value { - let user_ids: Vec = to.iter().map(|u| s(u)).collect(); - let (st, v) = app - .post( - from, - &format!("/canvases/{canvas}/mentions"), - json!({ "message_id": message_id, "user_ids": user_ids, "excerpt": excerpt }), - ) - .await; - assert_eq!(st, StatusCode::OK, "{v}"); - v -} - -#[tokio::test] -async fn mentions_notify_members_only_once_and_list_newest_first() { - let app = TestApp::new().await; - let ann = app.user("ann").await; - let ben = app.user("ben").await; - let cy = app.user("cy").await; - let out = app.user("outsider").await; - let id = app.canvas(&ann, "Team dinners").await; - app.share(&ann, &id, &ben).await; - app.share(&ann, &id, &cy).await; - let mut ann_ev = app.state.events.subscribe(ann.id); - let mut ben_ev = app.state.events.subscribe(ben.id); - let mut cy_ev = app.state.events.subscribe(cy.id); - let mut out_ev = app.state.events.subscribe(out.id); - - // Targets are filtered to current members other than the caller; duplicates collapse; - // unknown ids are skipped. Order follows the request. - let ghost = Uuid::now_v7().to_string(); - let path = format!("/canvases/{id}/mentions"); - let body = json!({ - "message_id": "msg-1", - "user_ids": [s(&ben), s(&out), s(&ann), s(&ben), ghost, s(&cy)], - "excerpt": "Dinner Tuesday?\n@ben @cy who is in", - }); - let (st, v) = app.post(&ann, &path, body.clone()).await; - assert_eq!(st, StatusCode::OK, "{v}"); - assert_eq!(ids(&v["notified"]), vec![s(&ben), s(&cy)]); - assert_eq!(ids(&v["skipped"]), vec![s(&out), s(&ann), ghost.clone()]); - let mentioned = vec![(id.clone(), "mention".to_owned())]; - assert_eq!(mention_events(&mut ben_ev), mentioned); - assert_eq!(mention_events(&mut cy_ev), mentioned); - assert_eq!( - mention_events(&mut out_ev), - vec![], - "non-members hear nothing" - ); - assert_eq!( - mention_events(&mut ann_ev), - vec![], - "the sender is never notified" - ); - - // A retry answers the same and notifies nobody again (one row per recipient). - let (st, again) = app.post(&ann, &path, body).await; - assert_eq!(st, StatusCode::OK, "{again}"); - assert_eq!(again, v); - assert_eq!(mention_events(&mut ben_ev), vec![]); - assert_eq!(mention_events(&mut cy_ev), vec![]); - let rows: i64 = sqlx::query_scalar( - "SELECT count(*) FROM canvas_mentions WHERE canvas_id = $1::uuid AND message_id = 'msg-1'", - ) - .bind(&id) - .fetch_one(app.db()) - .await - .unwrap(); - assert_eq!(rows, 2); - - // The recipient's list: canvas, sender, message id, cleaned excerpt, unread. - let (st, list) = app.get(&ben, "/mentions").await; - assert_eq!(st, StatusCode::OK, "{list}"); - assert_eq!(list["unread"], 1); - let items = list["mentions"].as_array().unwrap(); - assert_eq!(items.len(), 1, "{list}"); - let m = &items[0]; - assert_eq!(m["canvas"], json!({ "id": id, "name": "Team dinners" })); - assert_eq!( - m["from"], - json!({ "id": s(&ann), "email": ann.email, "display_name": "ann" }) - ); - assert_eq!(m["message_id"], "msg-1"); - assert_eq!(m["excerpt"], "Dinner Tuesday? @ben @cy who is in"); - assert_eq!(m["read_at"], Value::Null); - assert!(m["created_at"].as_str().unwrap().contains('T')); - assert!(Uuid::parse_str(m["id"].as_str().unwrap()).is_ok()); - // Nobody else's mentions leak into a list. - let (_, list) = app.get(&out, "/mentions").await; - assert_eq!(list, json!({ "mentions": [], "unread": 0 })); - let (_, list) = app.get(&ann, "/mentions").await; - assert_eq!(list["mentions"], json!([])); - - // Newest first; a member may mention the owner back. - mention(&app, &cy, &id, "msg-2", &[&ben, &ann], "second").await; - let (_, list) = app.get(&ben, "/mentions").await; - let order: Vec<&str> = list["mentions"] - .as_array() - .unwrap() - .iter() - .map(|m| m["message_id"].as_str().unwrap()) - .collect(); - assert_eq!(order, vec!["msg-2", "msg-1"]); - assert_eq!(list["unread"], 2); - let (_, list) = app.get(&ann, "/mentions").await; - assert_eq!(list["mentions"][0]["from"]["id"], s(&cy)); - assert_eq!(mention_events(&mut ann_ev), mentioned); - - // The same message id on another canvas is a different message. - let other = app.canvas(&ann, "Other").await; - app.share(&ann, &other, &ben).await; - let v = mention(&app, &ann, &other, "msg-1", &[&ben], "elsewhere").await; - assert_eq!(ids(&v["notified"]), vec![s(&ben)]); - assert_eq!( - mention_events(&mut ben_ev), - vec![ - (id.clone(), "mention".to_owned()), - (other.clone(), "mention".to_owned()) - ] - ); - - // Excerpts are sealed at rest, never plaintext. - let blobs: Vec> = - sqlx::query_scalar("SELECT excerpt_sealed FROM canvas_mentions WHERE canvas_id = $1::uuid") - .bind(&id) - .fetch_all(app.db()) - .await - .unwrap(); - assert_ne!(blobs.len(), 0); - for b in blobs { - assert!(!b.windows(6).any(|w| w == b"Dinner" || w == b"second")); - } - - // An empty or excerpt-less mention is fine; long excerpts are cut to 200 characters. - let v = mention(&app, &ann, &id, "msg-3", &[], "").await; - assert_eq!(v, json!({ "notified": [], "skipped": [] })); - mention(&app, &ann, &id, "msg-4", &[&ben], &"é".repeat(500)).await; - let (_, list) = app.get(&ben, "/mentions?limit=1").await; - assert_eq!(list["mentions"][0]["message_id"], "msg-4"); - assert_eq!( - list["mentions"][0]["excerpt"] - .as_str() - .unwrap() - .chars() - .count(), - 200 - ); -} - -#[tokio::test] -async fn mentions_are_scoped_to_members() { - let app = TestApp::new().await; - let ann = app.user("ann").await; - let ben = app.user("ben").await; - let eve = app.user("eve").await; - let id = app.canvas(&ann, "Private").await; - app.share(&ann, &id, &ben).await; - let path = format!("/canvases/{id}/mentions"); - let body = json!({ "message_id": "m", "user_ids": [s(&ben)], "excerpt": "hi" }); - - // Non-members, unknown canvases and malformed ids are 404 — no existence leak. - let (st, v) = app.post(&eve, &path, body.clone()).await; - assert_eq!(st, StatusCode::NOT_FOUND, "{v}"); - assert_eq!(v["error"], "not_found"); - let ghost = Uuid::now_v7(); - let (st, _) = app - .post(&ann, &format!("/canvases/{ghost}/mentions"), body.clone()) - .await; - assert_eq!(st, StatusCode::NOT_FOUND); - let (st, _) = app - .post(&ann, "/canvases/not-a-uuid/mentions", body.clone()) - .await; - assert_eq!(st, StatusCode::NOT_FOUND); - // No session → 401. - for (method, p) in [ - (Method::POST, path.as_str()), - (Method::GET, "/mentions"), - (Method::POST, "/mentions/read"), - ] { - let (st, v) = app.call(None, method, p, Some(json!({ "ids": [] }))).await; - assert_eq!(st, StatusCode::UNAUTHORIZED, "{p}: {v}"); - } - - // The Personal canvas has no chat. - let (st, v) = app.post(&ann, "/canvases/personal/mentions", body).await; - assert_eq!(st, StatusCode::BAD_REQUEST, "{v}"); - - // Validation. - for (bad, why) in [ - (json!({ "user_ids": [s(&ben)] }), "message_id missing"), - ( - json!({ "message_id": " ", "user_ids": [] }), - "empty message_id", - ), - ( - json!({ "message_id": "x".repeat(129), "user_ids": [] }), - "long message_id", - ), - ( - json!({ "message_id": "m", "user_ids": ["nope"] }), - "not a uuid", - ), - ( - json!({ "message_id": "m", "user_ids": (0..=MAX_MENTION_TARGETS).map(|_| Uuid::now_v7().to_string()).collect::>() }), - "too many targets", - ), - ] { - let (st, v) = app.post(&ann, &path, bad).await; - assert_eq!(st, StatusCode::BAD_REQUEST, "{why}: {v}"); - assert_eq!(v["error"], "bad_request", "{why}"); - } - - // A member who left is skipped, and their old mentions disappear from their list (but come - // back if they rejoin: the canvas is visible again). - mention(&app, &ann, &id, "before", &[&ben], "before leaving").await; - let (_, list) = app.get(&ben, "/mentions").await; - assert_eq!(list["unread"], 1); - let (st, _) = app - .delete(&ben, &format!("/canvases/{id}/members/{}", ben.id)) - .await; - assert_eq!(st, StatusCode::OK); - let v = mention(&app, &ann, &id, "after", &[&ben], "after leaving").await; - assert_eq!(v["notified"], json!([])); - assert_eq!(ids(&v["skipped"]), vec![s(&ben)]); - let (_, list) = app.get(&ben, "/mentions").await; - assert_eq!(list, json!({ "mentions": [], "unread": 0 })); - // ...and cannot be marked read from outside either. - let (st, v) = app - .post(&ben, "/mentions/read", json!({ "canvas_id": id })) - .await; - assert_eq!(st, StatusCode::OK, "{v}"); - assert_eq!(v["updated"], 0); - app.share(&ann, &id, &ben).await; - let (_, list) = app.get(&ben, "/mentions").await; - assert_eq!(list["unread"], 1); - assert_eq!(list["mentions"][0]["message_id"], "before"); - - // Deleting the canvas deletes its mentions. - let (st, _) = app.delete(&ann, &format!("/canvases/{id}")).await; - assert_eq!(st, StatusCode::OK); - let left: i64 = - sqlx::query_scalar("SELECT count(*) FROM canvas_mentions WHERE canvas_id = $1::uuid") - .bind(&id) - .fetch_one(app.db()) - .await - .unwrap(); - assert_eq!(left, 0); -} - -#[tokio::test] -async fn marking_read_clears_badges_on_every_device() { - let app = TestApp::new().await; - let ann = app.user("ann").await; - let ben = app.user("ben").await; - let cy = app.user("cy").await; - let one = app.canvas(&ann, "One").await; - let two = app.canvas(&ann, "Two").await; - for c in [&one, &two] { - app.share(&ann, c, &ben).await; - app.share(&ann, c, &cy).await; - } - mention(&app, &ann, &one, "a", &[&ben, &cy], "first").await; - mention(&app, &ann, &one, "b", &[&ben], "second").await; - mention(&app, &ann, &two, "c", &[&ben], "third").await; - let (_, list) = app.get(&ben, "/mentions").await; - assert_eq!(list["unread"], 3); - let id_of = |list: &Value, msg: &str| -> String { - list["mentions"] - .as_array() - .unwrap() - .iter() - .find(|m| m["message_id"] == msg) - .unwrap()["id"] - .as_str() - .unwrap() - .to_owned() - }; - let a_id = id_of(&list, "a"); - let mut ben_ev = app.state.events.subscribe(ben.id); - let mut cy_ev = app.state.events.subscribe(cy.id); - - // By id: only the caller's own rows; one `mention_read` per canvas touched. - let (st, v) = app - .post(&cy, "/mentions/read", json!({ "ids": [a_id] })) - .await; - assert_eq!(st, StatusCode::OK, "{v}"); - assert_eq!(v["updated"], 0, "Cy cannot mark Ben's mention"); - assert_eq!(mention_events(&mut cy_ev), vec![]); - let (st, v) = app - .post(&ben, "/mentions/read", json!({ "ids": [a_id] })) - .await; - assert_eq!(st, StatusCode::OK, "{v}"); - assert_eq!(v, json!({ "ok": true, "updated": 1 })); - assert_eq!( - mention_events(&mut ben_ev), - vec![(one.clone(), "mention_read".to_owned())] - ); - let (_, unread) = app.get(&ben, "/mentions?unread=1").await; - assert_eq!(unread["unread"], 2); - let msgs: Vec<&str> = unread["mentions"] - .as_array() - .unwrap() - .iter() - .map(|m| m["message_id"].as_str().unwrap()) - .collect(); - assert_eq!(msgs, vec!["c", "b"]); - // Cy's copy of the same message is untouched. - let (_, cy_list) = app.get(&cy, "/mentions?unread=true").await; - assert_eq!(cy_list["unread"], 1); - - // By canvas. - let (st, v) = app - .post(&ben, "/mentions/read", json!({ "canvas_id": two })) - .await; - assert_eq!(st, StatusCode::OK, "{v}"); - assert_eq!(v["updated"], 1); - assert_eq!( - mention_events(&mut ben_ev), - vec![(two.clone(), "mention_read".to_owned())] - ); - // Again: nothing changes, no event. - let (_, v) = app - .post(&ben, "/mentions/read", json!({ "canvas_id": two })) - .await; - assert_eq!(v["updated"], 0); - assert_eq!(mention_events(&mut ben_ev), vec![]); - - // The full list keeps read ones, with read_at set. - let (_, all) = app.get(&ben, "/mentions?unread=0").await; - assert_eq!(all["mentions"].as_array().unwrap().len(), 3); - assert_eq!(all["unread"], 1); - for m in all["mentions"].as_array().unwrap() { - assert_eq!(m["read_at"].is_null(), m["message_id"] == "b", "{m}"); - } - - // Bad requests. - for bad in [ - json!({}), - json!({ "ids": ["nope"] }), - json!({ "canvas_id": "personal" }), - ] { - let (st, v) = app.post(&ben, "/mentions/read", bad.clone()).await; - assert_eq!(st, StatusCode::BAD_REQUEST, "{bad}: {v}"); - } - for q in ["limit=0", "limit=-3", "unread=maybe", "limit=x"] { - let (st, v) = app.get(&ben, &format!("/mentions?{q}")).await; - assert_eq!(st, StatusCode::BAD_REQUEST, "{q}: {v}"); - } - let (st, v) = app.get(&ben, "/mentions?limit=100000").await; - assert_eq!(st, StatusCode::OK, "limit is capped, not refused: {v}"); - assert!(i64::try_from(v["mentions"].as_array().unwrap().len()).unwrap() <= MAX_MENTIONS_LIMIT); -} - -#[tokio::test] -async fn posting_mentions_is_rate_limited_per_user() { - let app = TestApp::new().await; - let ann = app.user("ann").await; - let ben = app.user("ben").await; - let id = app.canvas(&ann, "Busy").await; - app.share(&ann, &id, &ben).await; - let path = format!("/canvases/{id}/mentions"); - let mut limited = None; - for i in 0..=MENTION_POSTS_PER_MINUTE { - let (st, v) = app - .post( - &ann, - &path, - json!({ "message_id": format!("m{i}"), "user_ids": [s(&ben)], "excerpt": "hi" }), - ) - .await; - if st == StatusCode::TOO_MANY_REQUESTS { - assert_eq!(v["error"], "rate_limited"); - limited = Some(i); - break; - } - assert_eq!(st, StatusCode::OK, "{v}"); - } - assert_eq!(limited, Some(MENTION_POSTS_PER_MINUTE), "burst then 429"); - // Per user: Ben is unaffected. - let v = mention(&app, &ben, &id, "ben-1", &[&ann], "hi").await; - assert_eq!(ids(&v["notified"]), vec![s(&ann)]); -} diff --git a/crates/canvas/tests/rest.rs b/crates/canvas/tests/rest.rs deleted file mode 100644 index 72930e4..0000000 --- a/crates/canvas/tests/rest.rs +++ /dev/null @@ -1,938 +0,0 @@ -//! Canvases REST: list/create/rename/delete, Personal canvas, invites, membership scoping. - -#![allow(clippy::too_many_lines, clippy::many_single_char_names)] - -mod common; - -use axum::body::Body; -use axum::http::{Method, Request, StatusCode}; -use common::TestApp; -use http_body_util::BodyExt as _; -use serde_json::{json, Value}; -use tower::ServiceExt as _; -use yrs::{Map as _, Out, ReadTxn as _, Transact as _}; - -fn ids(list: &Value) -> Vec { - list.as_array() - .unwrap() - .iter() - .map(|c| c["id"].as_str().unwrap().to_owned()) - .collect() -} - -#[tokio::test] -async fn list_create_rename_delete() { - let app = TestApp::new().await; - let a = app.user("ann").await; - - let (s, list) = app.get(&a, "/canvases").await; - assert_eq!(s, StatusCode::OK, "{list}"); - assert_eq!(list.as_array().unwrap().len(), 1); - let personal = list[0].clone(); - assert_eq!(personal["kind"], "personal"); - assert_eq!(personal["name"], "Personal"); - assert_eq!(personal["role"], "owner"); - assert_eq!(personal["member_count"], 1); - assert_eq!(personal["owner"]["id"], a.id.to_string()); - assert_eq!(personal["owner"]["email"], a.email); - - let (s, created) = app - .post(&a, "/canvases", json!({ "name": " Roadmap " })) - .await; - assert_eq!(s, StatusCode::CREATED, "{created}"); - assert_eq!(created["name"], "Roadmap"); - assert_eq!(created["kind"], "shared"); - assert_eq!(created["role"], "owner"); - assert_eq!(created["member_count"], 1); - assert!(created["created_at"].as_str().unwrap().contains('T')); - let id = created["id"].as_str().unwrap().to_owned(); - - let (s, _) = app.post(&a, "/canvases", json!({ "name": " " })).await; - assert_eq!(s, StatusCode::BAD_REQUEST); - let (s, _) = app.post(&a, "/canvases", json!({ "nom": "x" })).await; - assert_eq!(s, StatusCode::BAD_REQUEST); - let (s, _) = app - .post(&a, "/canvases", json!({ "name": "x".repeat(201) })) - .await; - assert_eq!(s, StatusCode::BAD_REQUEST); - - let (_, list) = app.get(&a, "/canvases").await; - let listed = ids(&list); - assert_eq!(listed.len(), 2); - assert_eq!( - listed[0], - personal["id"].as_str().unwrap(), - "personal first" - ); - assert_eq!(listed[1], id); - - let (s, renamed) = app - .patch(&a, &format!("/canvases/{id}"), json!({ "name": "Plan B" })) - .await; - assert_eq!(s, StatusCode::OK, "{renamed}"); - assert_eq!(renamed["name"], "Plan B"); - let (s, one) = app.get(&a, &format!("/canvases/{id}")).await; - assert_eq!(s, StatusCode::OK); - assert_eq!(one["name"], "Plan B"); - // The document's meta.name follows the rename. - let doc = app.state_doc(&a, &id).await; - { - let txn = doc.transact(); - let meta = txn.get_map("meta").unwrap(); - assert!( - matches!(meta.get(&txn, "name"), Some(Out::Any(yrs::Any::String(s))) if &*s == "Plan B") - ); - assert!(meta.get(&txn, "createdBy").is_some()); - } - - let (s, v) = app.delete(&a, &format!("/canvases/{id}")).await; - assert_eq!(s, StatusCode::OK, "{v}"); - assert_eq!(v["ok"], true); - let (s, _) = app.get(&a, &format!("/canvases/{id}")).await; - assert_eq!(s, StatusCode::NOT_FOUND); - let (_, list) = app.get(&a, "/canvases").await; - assert_eq!( - ids(&list), - vec![personal["id"].as_str().unwrap().to_owned()] - ); - let rows: i64 = - sqlx::query_scalar("SELECT count(*) FROM canvas_updates WHERE canvas_id = $1::uuid") - .bind(&id) - .fetch_one(app.db()) - .await - .unwrap(); - assert_eq!(rows, 0, "updates cascade with the canvas"); -} - -#[tokio::test] -async fn personal_canvas_is_automatic_unique_and_unshareable() { - let app = TestApp::new().await; - let a = app.user("pat").await; - let b = app.user("bea").await; - - // Concurrent first use creates exactly one Personal canvas. - let mut tasks = Vec::new(); - for _ in 0..6 { - let state = app.state.clone(); - let uid = a.id; - tasks.push(tokio::spawn(async move { - copper_cloud_canvas::ensure_personal_canvas(&state, uid) - .await - .unwrap() - })); - } - let mut got = Vec::new(); - for t in tasks { - got.push(t.await.unwrap()); - } - got.dedup(); - assert_eq!(got.len(), 1, "{got:?}"); - let pid = got[0].to_string(); - - let (s, one) = app.get(&a, "/canvases/personal").await; - assert_eq!(s, StatusCode::OK); - assert_eq!(one["id"], pid.as_str()); - let (s, v) = app.get(&a, "/canvases/personal/state").await; - assert_eq!(s, StatusCode::OK, "{v}"); - assert!(v["state"].is_string()); - - let p = format!("/canvases/{pid}"); - let (s, _) = app - .post(&a, &format!("{p}/invites"), json!({ "email": b.email })) - .await; - assert_eq!(s, StatusCode::BAD_REQUEST); - let (s, _) = app.delete(&a, &p).await; - assert_eq!(s, StatusCode::BAD_REQUEST); - let (s, _) = app.patch(&a, &p, json!({ "name": "Mine" })).await; - assert_eq!(s, StatusCode::BAD_REQUEST); - let (s, _) = app.delete(&a, &format!("{p}/members/{}", a.id)).await; - assert_eq!(s, StatusCode::BAD_REQUEST); - let (s, members) = app.get(&a, &format!("{p}/members")).await; - assert_eq!(s, StatusCode::OK); - assert_eq!(members.as_array().unwrap().len(), 1); - - // Another user's `personal` is their own. - let (_, theirs) = app.get(&b, "/canvases/personal").await; - assert_ne!(theirs["id"], pid.as_str()); - let (s, _) = app.get(&b, &p).await; - assert_eq!(s, StatusCode::NOT_FOUND); -} - -#[tokio::test] -async fn invite_accept_decline_and_leave() { - let app = TestApp::new().await; - let a = app.user("owner").await; - let b = app.user("bob").await; - let c = app.user("cat").await; - let id = app.canvas(&a, "Team").await; - let base = format!("/canvases/{id}"); - - // Invite by email (case-insensitive), idempotent while pending. - let (s, inv) = app - .post( - &a, - &format!("{base}/invites"), - json!({ "email": b.email.to_uppercase() }), - ) - .await; - assert_eq!(s, StatusCode::CREATED, "{inv}"); - assert_eq!(inv["email"], b.email); - assert_eq!(inv["status"], "pending"); - assert_eq!(inv["canvas_name"], "Team"); - assert_eq!(inv["invited_by"]["id"], a.id.to_string()); - assert!(inv.get("token").is_none(), "tokens are never exposed"); - let (s, again) = app - .post(&a, &format!("{base}/invites"), json!({ "email": b.email })) - .await; - assert_eq!(s, StatusCode::OK); - assert_eq!(again["id"], inv["id"]); - let (s, _) = app - .post(&a, &format!("{base}/invites"), json!({ "email": "nope" })) - .await; - assert_eq!(s, StatusCode::BAD_REQUEST); - let (s, _) = app - .post(&a, &format!("{base}/invites"), json!({ "email": a.email })) - .await; - assert_eq!(s, StatusCode::BAD_REQUEST); - // An email with no account yet is still a valid invite. - let (s, _) = app - .post( - &a, - &format!("{base}/invites"), - json!({ "email": "later@example.test" }), - ) - .await; - assert_eq!(s, StatusCode::CREATED); - let (s, pending) = app.get(&a, &format!("{base}/invites")).await; - assert_eq!(s, StatusCode::OK); - assert_eq!(pending.as_array().unwrap().len(), 2); - - // Before accepting, B cannot see the canvas. - let (s, _) = app.get(&b, &base).await; - assert_eq!(s, StatusCode::NOT_FOUND); - let (s, mine) = app.get(&b, "/invites").await; - assert_eq!(s, StatusCode::OK); - assert_eq!(mine.as_array().unwrap().len(), 1); - assert_eq!(mine[0]["canvas_id"], id.as_str()); - assert_eq!(mine[0]["canvas_name"], "Team"); - assert_eq!(mine[0]["invited_by"]["email"], a.email); - // Someone else cannot accept B's invite. - let inv_id = inv["id"].as_str().unwrap(); - let (s, _) = app - .post(&c, &format!("/invites/{inv_id}/accept"), Value::Null) - .await; - assert_eq!(s, StatusCode::NOT_FOUND); - - let (s, joined) = app - .post(&b, &format!("/invites/{inv_id}/accept"), Value::Null) - .await; - assert_eq!(s, StatusCode::OK, "{joined}"); - assert_eq!(joined["id"], id.as_str()); - assert_eq!(joined["role"], "editor"); - assert_eq!(joined["member_count"], 2); - let (s, _) = app - .post(&b, &format!("/invites/{inv_id}/accept"), Value::Null) - .await; - assert_eq!(s, StatusCode::NOT_FOUND, "an invite is single-use"); - let (_, mine) = app.get(&b, "/invites").await; - assert_eq!(mine.as_array().unwrap().len(), 0); - let (_, blist) = app.get(&b, "/canvases").await; - assert!(ids(&blist).contains(&id)); - - let (s, members) = app.get(&b, &format!("{base}/members")).await; - assert_eq!(s, StatusCode::OK); - let members = members.as_array().unwrap(); - assert_eq!(members.len(), 2); - assert_eq!(members[0]["role"], "owner"); - assert_eq!(members[1]["user_id"], b.id.to_string()); - assert_eq!(members[1]["role"], "editor"); - - // Already a member → 409. - let (s, _) = app - .post(&a, &format!("{base}/invites"), json!({ "email": b.email })) - .await; - assert_eq!(s, StatusCode::CONFLICT); - // Editors may invite; C declines. - let (s, cinv) = app - .post(&b, &format!("{base}/invites"), json!({ "email": c.email })) - .await; - assert_eq!(s, StatusCode::CREATED); - let cinv_id = cinv["id"].as_str().unwrap(); - let (s, v) = app - .post(&c, &format!("/invites/{cinv_id}/decline"), Value::Null) - .await; - assert_eq!(s, StatusCode::OK, "{v}"); - let (_, cmine) = app.get(&c, "/invites").await; - assert_eq!(cmine.as_array().unwrap().len(), 0); - let (s, _) = app - .post(&c, &format!("/invites/{cinv_id}/accept"), Value::Null) - .await; - assert_eq!( - s, - StatusCode::NOT_FOUND, - "declined invites cannot be accepted" - ); - let (s, _) = app.get(&c, &base).await; - assert_eq!(s, StatusCode::NOT_FOUND); - - // Editors cannot rename, delete or remove others. - let (s, _) = app.patch(&b, &base, json!({ "name": "Mine" })).await; - assert_eq!(s, StatusCode::FORBIDDEN); - let (s, _) = app.delete(&b, &base).await; - assert_eq!(s, StatusCode::FORBIDDEN); - let (s, _) = app.delete(&b, &format!("{base}/members/{}", a.id)).await; - assert_eq!(s, StatusCode::FORBIDDEN); - // The owner cannot leave. - let (s, _) = app.delete(&a, &format!("{base}/members/{}", a.id)).await; - assert_eq!(s, StatusCode::BAD_REQUEST); - - // B leaves; then has no access. - let (s, v) = app.delete(&b, &format!("{base}/members/{}", b.id)).await; - assert_eq!(s, StatusCode::OK, "{v}"); - let (s, _) = app.get(&b, &base).await; - assert_eq!(s, StatusCode::NOT_FOUND); - let (s, _) = app.delete(&a, &format!("{base}/members/{}", b.id)).await; - assert_eq!(s, StatusCode::NOT_FOUND, "already gone"); - - // Re-invite + accept, then the owner removes B. - app.share(&a, &id, &b).await; - let (s, _) = app.delete(&a, &format!("{base}/members/{}", b.id)).await; - assert_eq!(s, StatusCode::OK); - let (s, _) = app.get(&b, &format!("{base}/state")).await; - assert_eq!(s, StatusCode::NOT_FOUND); -} - -/// Drains the `canvas` events queued for one subscriber: `(canvas_id, kind)` pairs. -fn canvas_events( - rx: &mut tokio::sync::broadcast::Receiver, -) -> Vec<(String, String)> { - let mut out = Vec::new(); - while let Ok(ev) = rx.try_recv() { - if let copper_cloud_core::events::Event::Canvas { canvas_id, kind } = ev { - out.push((canvas_id.to_string(), kind)); - } - } - out -} - -fn no_events() -> Vec<(String, String)> { - Vec::new() -} - -/// Moves an invite's last-notified time back so the re-send throttle has elapsed. -async fn age_invite(app: &TestApp, invite: &str) { - sqlx::query( - "UPDATE canvas_invites - SET created_at = created_at - interval '1 minute', - nudged_at = nudged_at - interval '1 minute' - WHERE id = $1::uuid", - ) - .bind(invite) - .execute(app.db()) - .await - .unwrap(); -} - -#[tokio::test] -async fn reinvite_nudges_the_invitee_at_most_every_30_seconds() { - assert_eq!(copper_cloud_canvas::INVITE_NUDGE_INTERVAL_SECS, 30); - let app = TestApp::new().await; - let a = app.user("owner").await; - let b = app.user("bob").await; - let e = app.user("eve").await; - let id = app.canvas(&a, "Nudges").await; - app.share(&a, &id, &e).await; - let invites = format!("/canvases/{id}/invites"); - let mut bob_events = app.state.events.subscribe(b.id); - - // A new invite: 201, notifies the invitee, names their account, not a nudge. - let (s, inv) = app.post(&a, &invites, json!({ "email": b.email })).await; - assert_eq!(s, StatusCode::CREATED, "{inv}"); - assert_eq!(inv["nudged"], false); - assert_eq!(inv["nudged_at"], Value::Null); - assert_eq!(inv["invitee"]["id"], b.id.to_string()); - assert_eq!(inv["invitee"]["email"], b.email); - assert_eq!(inv["invitee"]["display_name"], "bob"); - let inv_id = inv["id"].as_str().unwrap().to_owned(); - assert_eq!( - canvas_events(&mut bob_events), - vec![(id.clone(), "invited".to_owned())] - ); - - // Re-sent right away: still pending (200, same invite) but throttled — no event. - let (s, again) = app.post(&a, &invites, json!({ "email": b.email })).await; - assert_eq!(s, StatusCode::OK, "{again}"); - assert_eq!(again["id"], inv_id.as_str()); - assert_eq!(again["nudged"], false); - assert_eq!(again["nudged_at"], Value::Null); - assert_eq!(canvas_events(&mut bob_events), no_events()); - - // Once the interval has passed, a re-send is a nudge: nudged_at set, invitee notified. - age_invite(&app, &inv_id).await; - let (s, nudge) = app - .post(&a, &invites, json!({ "email": b.email.to_uppercase() })) - .await; - assert_eq!(s, StatusCode::OK, "{nudge}"); - assert_eq!(nudge["id"], inv_id.as_str()); - assert_eq!(nudge["nudged"], true); - let nudged_at = nudge["nudged_at"].as_str().unwrap().to_owned(); - assert_eq!( - canvas_events(&mut bob_events), - vec![(id.clone(), "invited".to_owned())] - ); - - // And it throttles again from nudged_at. - let (s, again) = app.post(&a, &invites, json!({ "email": b.email })).await; - assert_eq!(s, StatusCode::OK); - assert_eq!(again["nudged"], false); - assert_eq!(again["nudged_at"], nudged_at.as_str()); - assert_eq!(canvas_events(&mut bob_events), no_events()); - - // Any member who may invite may re-send. - age_invite(&app, &inv_id).await; - let (s, by_editor) = app.post(&e, &invites, json!({ "email": b.email })).await; - assert_eq!(s, StatusCode::OK, "{by_editor}"); - assert_eq!(by_editor["nudged"], true); - assert_eq!(canvas_events(&mut bob_events).len(), 1); - - // Both lists carry nudged_at and invitee; the inviter is unchanged. - let (_, mine) = app.get(&b, "/invites").await; - assert_eq!(mine.as_array().unwrap().len(), 1, "{mine}"); - assert!(mine[0]["nudged_at"].is_string()); - assert_eq!(mine[0]["invitee"]["id"], b.id.to_string()); - assert_eq!(mine[0]["invited_by"]["id"], a.id.to_string()); - assert!(mine[0].get("nudged").is_none(), "only POST says nudged"); - let (_, pending) = app.get(&a, &invites).await; - assert_eq!(pending[0]["id"], inv_id.as_str()); - assert!(pending[0]["nudged_at"].is_string()); - assert_eq!(pending[0]["invitee"]["email"], b.email); - - // No account with that email yet: invitee is null and a re-send cannot notify anyone. - let later = format!("later-{}@example.test", &inv_id[24..]); - let (s, ghost) = app.post(&a, &invites, json!({ "email": later })).await; - assert_eq!(s, StatusCode::CREATED, "{ghost}"); - assert_eq!(ghost["invitee"], Value::Null); - assert_eq!(ghost["nudged"], false); - let ghost_id = ghost["id"].as_str().unwrap().to_owned(); - age_invite(&app, &ghost_id).await; - let (s, ghost) = app.post(&a, &invites, json!({ "email": later })).await; - assert_eq!(s, StatusCode::OK); - assert_eq!(ghost["id"], ghost_id.as_str()); - assert_eq!(ghost["nudged"], false); - assert_eq!(ghost["nudged_at"], Value::Null); - assert_eq!(ghost["invitee"], Value::Null); - let (_, pending) = app.get(&a, &invites).await; - assert_eq!(pending.as_array().unwrap().len(), 2); - assert_eq!(pending[0]["invitee"], Value::Null, "newest first"); - - // Accepting still works on a nudged invite. - let (s, v) = app - .post(&b, &format!("/invites/{inv_id}/accept"), Value::Null) - .await; - assert_eq!(s, StatusCode::OK, "{v}"); -} - -#[tokio::test] -async fn owner_or_inviter_revokes_a_pending_invite() { - let app = TestApp::new().await; - let a = app.user("owner").await; - let b = app.user("bob").await; - let e = app.user("eve").await; - let f = app.user("fay").await; - let x = app.user("stranger").await; - let id = app.canvas(&a, "Revocable").await; - let other = app.canvas(&a, "Other").await; - app.share(&a, &id, &e).await; - app.share(&a, &id, &f).await; - let invites = format!("/canvases/{id}/invites"); - let mut bob_events = app.state.events.subscribe(b.id); - - let (s, inv) = app.post(&a, &invites, json!({ "email": b.email })).await; - assert_eq!(s, StatusCode::CREATED); - let inv_id = inv["id"].as_str().unwrap().to_owned(); - let one = format!("{invites}/{inv_id}"); - canvas_events(&mut bob_events); - - // Another member (neither owner nor inviter) may not; outsiders and wrong paths are 404. - let (s, v) = app.delete(&e, &one).await; - assert_eq!(s, StatusCode::FORBIDDEN, "{v}"); - assert_eq!(v["error"], "forbidden"); - let (s, _) = app.delete(&x, &one).await; - assert_eq!(s, StatusCode::NOT_FOUND); - let (s, _) = app.delete(&b, &one).await; - assert_eq!(s, StatusCode::NOT_FOUND, "the invitee is not a member"); - let (s, _) = app - .delete(&a, &format!("/canvases/{other}/invites/{inv_id}")) - .await; - assert_eq!(s, StatusCode::NOT_FOUND, "invite of another canvas"); - let (s, _) = app.delete(&a, &format!("{invites}/not-a-uuid")).await; - assert_eq!(s, StatusCode::NOT_FOUND); - assert_eq!(canvas_events(&mut bob_events), no_events()); - - // The owner revokes: 204, the invitee is told and no longer sees it. - let (s, v) = app.delete(&a, &one).await; - assert_eq!(s, StatusCode::NO_CONTENT, "{v}"); - assert_eq!(v, Value::Null); - assert_eq!( - canvas_events(&mut bob_events), - vec![(id.clone(), "invite_revoked".to_owned())] - ); - let status: String = - sqlx::query_scalar("SELECT status FROM canvas_invites WHERE id = $1::uuid") - .bind(&inv_id) - .fetch_one(app.db()) - .await - .unwrap(); - assert_eq!(status, "revoked"); - let (_, mine) = app.get(&b, "/invites").await; - assert_eq!(mine, json!([])); - let (_, pending) = app.get(&a, &invites).await; - assert_eq!(pending, json!([])); - let (s, _) = app - .post(&b, &format!("/invites/{inv_id}/accept"), Value::Null) - .await; - assert_eq!( - s, - StatusCode::NOT_FOUND, - "revoked invites cannot be accepted" - ); - let (s, _) = app - .post(&b, &format!("/invites/{inv_id}/decline"), Value::Null) - .await; - assert_eq!(s, StatusCode::NOT_FOUND); - - // Twice is 404. - let (s, v) = app.delete(&a, &one).await; - assert_eq!(s, StatusCode::NOT_FOUND, "{v}"); - assert_eq!(v["error"], "not_found"); - - // The slot is free again: a fresh invite is a new 201. - let (s, fresh) = app.post(&a, &invites, json!({ "email": b.email })).await; - assert_eq!(s, StatusCode::CREATED, "{fresh}"); - assert_ne!(fresh["id"], inv_id.as_str()); - - // An editor revokes their own invite (not someone else's); the owner can revoke any. - let (s, finv) = app - .post(&e, &invites, json!({ "email": "nobody-yet@example.test" })) - .await; - assert_eq!(s, StatusCode::CREATED); - let finv = format!("{invites}/{}", finv["id"].as_str().unwrap()); - let (s, _) = app.delete(&f, &finv).await; - assert_eq!(s, StatusCode::FORBIDDEN); - let (s, _) = app.delete(&e, &finv).await; - assert_eq!(s, StatusCode::NO_CONTENT, "the inviter may revoke"); - let (s, _) = app - .delete(&e, &format!("{invites}/{}", fresh["id"].as_str().unwrap())) - .await; - assert_eq!( - s, - StatusCode::FORBIDDEN, - "an editor cannot revoke the owner's invite" - ); - - // A pending invite from an editor who has since left can still be revoked by the owner. - let (s, ginv) = app - .post(&f, &invites, json!({ "email": "someone@example.test" })) - .await; - assert_eq!(s, StatusCode::CREATED); - let (s, _) = app - .delete(&f, &format!("/canvases/{id}/members/{}", f.id)) - .await; - assert_eq!(s, StatusCode::OK); - let (s, _) = app - .delete(&a, &format!("{invites}/{}", ginv["id"].as_str().unwrap())) - .await; - assert_eq!(s, StatusCode::NO_CONTENT); -} - -#[tokio::test] -async fn non_members_get_404_everywhere() { - let app = TestApp::new().await; - let a = app.user("own").await; - let x = app.user("intruder").await; - let id = app.canvas(&a, "Secret").await; - let base = format!("/canvases/{id}"); - let missing = uuid::Uuid::now_v7(); - - for path in [ - base.clone(), - format!("{base}/members"), - format!("{base}/invites"), - format!("{base}/state"), - format!("{base}/read"), - format!("/canvases/{missing}"), - "/canvases/not-a-uuid".to_owned(), - ] { - let (s, v) = app.get(&x, &path).await; - assert_eq!(s, StatusCode::NOT_FOUND, "GET {path}: {v}"); - } - let (s, _) = app.patch(&x, &base, json!({ "name": "pwned" })).await; - assert_eq!(s, StatusCode::NOT_FOUND); - let (s, _) = app.delete(&x, &base).await; - assert_eq!(s, StatusCode::NOT_FOUND); - let (s, _) = app.delete(&x, &format!("{base}/members/{}", a.id)).await; - assert_eq!(s, StatusCode::NOT_FOUND); - let (s, _) = app - .post(&x, &format!("{base}/invites"), json!({ "email": x.email })) - .await; - assert_eq!(s, StatusCode::NOT_FOUND); - let (s, _) = app - .post( - &x, - &format!("{base}/ops"), - json!({ "ops": [{ "op": "add", "shape": { "type": "sticky" } }] }), - ) - .await; - assert_eq!(s, StatusCode::NOT_FOUND); - let (s, _) = app - .post(&x, &format!("/invites/{missing}/accept"), Value::Null) - .await; - assert_eq!(s, StatusCode::NOT_FOUND); - // WebSocket upgrade is refused with 404 too. - let err = app.ws_raw(&id, &x.token).await.unwrap_err(); - assert!(err.to_string().contains("404"), "{err}"); - - // The canvas is untouched. - let (_, one) = app.get(&a, &base).await; - assert_eq!(one["name"], "Secret"); - - // No session → 401; no instance key → 401. - let (s, v) = app.call(None, Method::GET, "/canvases", None).await; - assert_eq!(s, StatusCode::UNAUTHORIZED, "{v}"); - let err = app.ws_raw(&id, "x".repeat(43).as_str()).await.unwrap_err(); - assert!(err.to_string().contains("401"), "{err}"); -} - -#[tokio::test] -async fn read_and_state_endpoints() { - let app = TestApp::new().await; - let a = app.user("reader").await; - let id = app.canvas(&a, "Notes").await; - let base = format!("/canvases/{id}"); - let long = "y".repeat(700); - let (s, r) = app - .post( - &a, - &format!("{base}/ops"), - json!({ "ops": [ - { "op": "add", "shape": { "id": "s1", "type": "sticky", "text": long, "x": 0, "y": 0 } }, - { "op": "add", "shape": { "id": "l1", "type": "link", "url": "https://example.com", "title": "Ex", "x": 400, "y": 0 } }, - { "op": "connect", "from": "s1", "to": "l1", "label": "ref" } - ]}), - ) - .await; - assert_eq!(s, StatusCode::OK, "{r}"); - assert_eq!(r["applied"], 3); - - let (s, read) = app.get(&a, &format!("{base}/read")).await; - assert_eq!(s, StatusCode::OK, "{read}"); - assert_eq!(read["canvas"]["id"], id.as_str()); - assert_eq!(read["canvas"]["name"], "Notes"); - assert_eq!(read["canvas"]["kind"], "shared"); - assert_eq!(read["selection"], json!([])); - let shapes = read["shapes"].as_array().unwrap(); - assert_eq!(shapes.len(), 3); - let s1 = shapes.iter().find(|s| s["id"] == "s1").unwrap(); - assert_eq!(s1["text"].as_str().unwrap().chars().count(), 501); - assert_eq!(s1["by"], "reader"); - let arrow = shapes.iter().find(|s| s["type"] == "arrow").unwrap(); - assert_eq!(arrow["from"], json!({ "ref": "s1" })); - assert_eq!(arrow["label"], "ref"); - let (_, full) = app.get(&a, &format!("{base}/read?full=true")).await; - let s1 = full["shapes"] - .as_array() - .unwrap() - .iter() - .find(|s| s["id"] == "s1") - .unwrap() - .clone(); - assert_eq!(s1["text"].as_str().unwrap().len(), 700); - - // Full state, then a diff against it is (nearly) empty. - let doc = app.state_doc(&a, &id).await; - assert_eq!(common::shape_count(&doc), 3); - let sv = common::sv_b64(&doc); - let (s, diff) = app - .get(&a, &format!("{base}/state?sv={}", urlencode(&sv))) - .await; - assert_eq!(s, StatusCode::OK); - let (_, fullstate) = app.get(&a, &format!("{base}/state")).await; - assert!(diff["state"].as_str().unwrap().len() < fullstate["state"].as_str().unwrap().len()); - let (s, _) = app.get(&a, &format!("{base}/state?sv=!!!")).await; - assert_eq!(s, StatusCode::BAD_REQUEST); - // A plain request to the socket path is a JSON 400, not a crash. - let (s, v) = app.get(&a, &format!("{base}/ws")).await; - assert_eq!(s, StatusCode::BAD_REQUEST); - assert_eq!(v["error"], "bad_request"); - // read filters - let (_, only) = app - .get(&a, &format!("{base}/read?types=link,arrow&ids=l1,shape:s1")) - .await; - assert_eq!(only["shapes"].as_array().unwrap().len(), 1, "{only}"); - assert_eq!(only["shapes"][0]["id"], "l1"); - assert_eq!(only["count"], 3); -} - -#[tokio::test] -async fn share_links_people_and_landing() { - let app = TestApp::new().await; - let owner = app.user("owner").await; - let editor = app.user("editor").await; - let guest = app.user("guest").await; - let disabled = app.user("disabled").await; - let id = app.canvas(&owner, "Shared").await; - app.share(&owner, &id, &editor).await; - let base = format!("/canvases/{id}"); - let (s, _) = app.post(&guest, &format!("{base}/links"), json!({})).await; - assert_eq!(s, StatusCode::NOT_FOUND); - - let (s, created) = app.post(&owner, &format!("{base}/links"), json!({})).await; - assert_eq!(s, StatusCode::CREATED, "{created}"); - let token = created["token"].as_str().unwrap().to_owned(); - assert_eq!(token.len(), 43); - assert_eq!(created["role"], "editor"); - let stored: Vec = - sqlx::query_scalar("SELECT token_sha256 FROM canvas_share_links WHERE id = $1::uuid") - .bind(created["id"].as_str().unwrap()) - .fetch_one(app.db()) - .await - .unwrap(); - assert_eq!( - stored.as_slice(), - &copper_cloud_core::ids::sha256(token.as_bytes()) - ); - assert_ne!(stored.as_slice(), token.as_bytes()); - - let (s, list) = app.get(&owner, &format!("{base}/links")).await; - assert_eq!(s, StatusCode::OK, "{list}"); - assert_eq!(list["links"].as_array().unwrap().len(), 1); - assert!(list["links"][0].get("token").is_none()); - let link_id = created["id"].as_str().unwrap(); - let (s, _) = app.get(&editor, &format!("{base}/links")).await; - assert_eq!(s, StatusCode::FORBIDDEN); - let (s, _) = app - .delete(&editor, &format!("{base}/links/{link_id}")) - .await; - assert_eq!(s, StatusCode::FORBIDDEN); - let (s, by_editor) = app - .post(&editor, &format!("{base}/links"), json!({"role":"editor"})) - .await; - assert_eq!(s, StatusCode::CREATED, "{by_editor}"); - let (s, _) = app - .post(&owner, &format!("{base}/links"), json!({"role":"owner"})) - .await; - assert_eq!(s, StatusCode::BAD_REQUEST); - - let (s, preview) = app.get(&guest, &format!("/canvas-links/{token}")).await; - assert_eq!(s, StatusCode::OK, "{preview}"); - assert_eq!(preview["member"], false); - assert_eq!(preview["owner"]["id"], owner.id.to_string()); - let (s, _) = app - .post( - &owner, - &format!("{base}/invites"), - json!({"email":guest.email}), - ) - .await; - assert_eq!(s, StatusCode::CREATED); - let (s, joined) = app - .post(&guest, &format!("/canvas-links/{token}/join"), Value::Null) - .await; - assert_eq!(s, StatusCode::OK, "{joined}"); - assert_eq!(joined["role"], "editor"); - let (s, invites) = app.get(&guest, "/invites").await; - assert_eq!(s, StatusCode::OK); - assert_eq!(invites.as_array().unwrap().len(), 0); - // Retry is idempotent, and joining as the owner never downgrades them. - let (s, _) = app - .post(&guest, &format!("/canvas-links/{token}/join"), Value::Null) - .await; - assert_eq!(s, StatusCode::OK); - let (s, owner_joined) = app - .post(&owner, &format!("/canvas-links/{token}/join"), Value::Null) - .await; - assert_eq!(s, StatusCode::OK); - assert_eq!(owner_joined["role"], "owner"); - let uses: i64 = sqlx::query_scalar("SELECT uses FROM canvas_share_links WHERE id = $1::uuid") - .bind(link_id) - .fetch_one(app.db()) - .await - .unwrap(); - assert_eq!(uses, 3); - - let (s, _) = app.delete(&owner, &format!("{base}/links/{link_id}")).await; - assert_eq!(s, StatusCode::NO_CONTENT); - let (s, err) = app.get(&guest, &format!("/canvas-links/{token}")).await; - assert_eq!(s, StatusCode::NOT_FOUND); - assert_eq!(err["error"], "link_not_found"); - let (s, _) = app - .post(&guest, &format!("/canvas-links/{token}/join"), Value::Null) - .await; - assert_eq!(s, StatusCode::NOT_FOUND); - - let (s, second) = app.post(&owner, &format!("{base}/links"), json!({})).await; - assert_eq!(s, StatusCode::CREATED); - let second_token = second["token"].as_str().unwrap(); - let (s, _) = app.delete(&owner, &format!("{base}/links")).await; - assert_eq!(s, StatusCode::NO_CONTENT); - let (s, _) = app - .get(&owner, &format!("/canvas-links/{second_token}")) - .await; - assert_eq!(s, StatusCode::NOT_FOUND); - let (s, third) = app.post(&owner, &format!("{base}/links"), json!({})).await; - assert_eq!(s, StatusCode::CREATED); - let third_token = third["token"].as_str().unwrap(); - let (s, people) = app - .get(&guest, &format!("/people?q={}", editor.email)) - .await; - assert_eq!(s, StatusCode::OK); - let (s, _) = app.call(None, Method::GET, "/people", None).await; - assert_eq!(s, StatusCode::UNAUTHORIZED); - assert_eq!(people["people"].as_array().unwrap().len(), 1); - assert_eq!(people["people"][0]["display_name"], "editor"); - sqlx::query("UPDATE users SET disabled = true WHERE id = $1") - .bind(disabled.id) - .execute(app.db()) - .await - .unwrap(); - let (s, people) = app - .get(&guest, &format!("/people?q={}", disabled.email)) - .await; - assert_eq!(s, StatusCode::OK); - assert_eq!(people["people"].as_array().unwrap().len(), 0); - - let (s, _) = app.delete(&owner, &base).await; - assert_eq!(s, StatusCode::OK); - let (s, _) = app - .get(&guest, &format!("/canvas-links/{third_token}")) - .await; - assert_eq!(s, StatusCode::NOT_FOUND); - - let req = Request::builder() - .method(Method::GET) - .uri(format!("/join/{token}")) - .header("host", "cloud.example:443") - .body(Body::empty()) - .unwrap(); - let response = app.router.clone().oneshot(req).await.unwrap(); - assert_eq!(response.status(), StatusCode::OK); - assert_eq!(response.headers()["cache-control"], "no-store"); - assert_eq!(response.headers()["referrer-policy"], "no-referrer"); - assert_eq!(response.headers()["x-content-type-options"], "nosniff"); - assert_eq!( - response.headers()["content-security-policy"], - "default-src 'none'; style-src 'unsafe-inline'" - ); - let body = response.into_body().collect().await.unwrap().to_bytes(); - let html = String::from_utf8(body.to_vec()).unwrap(); - assert!(html.contains("copper://canvas/join/")); - assert!(html.contains("cloud=cloud.example:443")); - - let req = Request::builder() - .method(Method::GET) - .uri(format!("/join/{token}")) - .header("host", "[::1]") - .body(Body::empty()) - .unwrap(); - let response = app.router.clone().oneshot(req).await.unwrap(); - let body = response.into_body().collect().await.unwrap().to_bytes(); - let html = String::from_utf8(body.to_vec()).unwrap(); - assert!(html.contains("cloud=[::1]")); - - let req = Request::builder() - .method(Method::GET) - .uri(format!("/join/{token}")) - .header("host", "") - .body(Body::empty()) - .unwrap(); - let response = app.router.clone().oneshot(req).await.unwrap(); - let body = response.into_body().collect().await.unwrap().to_bytes(); - let html = String::from_utf8(body.to_vec()).unwrap(); - assert!(!html.contains("copper://")); - assert!(!html.contains("")); -} - -fn urlencode(s: &str) -> String { - s.replace('+', "%2B") - .replace('/', "%2F") - .replace('=', "%3D") -} - -#[tokio::test] -async fn bodies_are_bounded() { - let app = TestApp::new().await; - let a = app.user("big").await; - let id = app.canvas(&a, "Big").await; - let huge = "z".repeat(copper_cloud_canvas::MAX_OPS_BODY_BYTES + 10); - let (s, v) = app - .post( - &a, - &format!("/canvases/{id}/ops"), - json!({ "ops": [{ "op": "add", "shape": { "type": "sticky", "text": huge } }] }), - ) - .await; - assert_eq!(s, StatusCode::PAYLOAD_TOO_LARGE, "{v}"); - let (s, v) = app - .post( - &a, - "/canvases", - json!({ "name": "z".repeat(copper_cloud_canvas::MAX_BODY_BYTES) }), - ) - .await; - assert_eq!(s, StatusCode::PAYLOAD_TOO_LARGE, "{v}"); - // A 2 MiB-class image fits in one ops call. - let img = format!("data:image/png;base64,{}", "A".repeat(1_500_000)); - let (s, v) = app - .post( - &a, - &format!("/canvases/{id}/ops"), - json!({ "ops": [{ "op": "add", "shape": { "type": "image", "src": img, "x": 0, "y": 0 } }] }), - ) - .await; - assert_eq!(s, StatusCode::OK); - assert_eq!(v["applied"], 1, "{v}"); - // Envelope problems are reported like the page does. - let (s, v) = app - .call( - Some(&a), - Method::POST, - &format!("/canvases/{id}/ops"), - Some(json!("not an object")), - ) - .await; - assert_eq!(s, StatusCode::OK, "{v}"); - assert_eq!(v["errors"][0]["index"], -1); - let (s, v) = app - .call( - Some(&a), - Method::POST, - &format!("/canvases/{id}/ops"), - Some(json!({ "ops": 3 })), - ) - .await; - assert_eq!(s, StatusCode::OK); - assert_eq!(v["errors"][0]["error"], "`ops` must be an array"); - // Malformed JSON is a 400. - let req = axum::http::Request::builder() - .method(Method::POST) - .uri(format!("/v1/canvases/{id}/ops")) - .header("x-copper-instance", &app.instance_key) - .header("authorization", format!("Bearer {}", a.token)) - .body(axum::body::Body::from("{nope")) - .unwrap(); - let resp = tower::ServiceExt::oneshot(app.router.clone(), req) - .await - .unwrap(); - assert_eq!(resp.status(), StatusCode::BAD_REQUEST); - // A bare array of ops works too. - let (s, v) = app - .call( - Some(&a), - Method::POST, - &format!("/canvases/{id}/ops"), - Some(json!([{ "op": "add", "shape": { "type": "text", "text": "bare" } }])), - ) - .await; - assert_eq!(s, StatusCode::OK); - assert_eq!(v["applied"], 1, "{v}"); -} diff --git a/crates/canvas/tests/ws.rs b/crates/canvas/tests/ws.rs deleted file mode 100644 index 8e818f8..0000000 --- a/crates/canvas/tests/ws.rs +++ /dev/null @@ -1,458 +0,0 @@ -//! y-websocket rooms: live sync, awareness relay, persistence across eviction, compaction, -//! REST ops fan-out, membership revocation. - -#![allow(clippy::too_many_lines, clippy::many_single_char_names)] - -mod common; - -use std::time::Duration; - -use axum::http::StatusCode; -use common::{add_sticky, eventually, set_prop, TestApp}; -use copper_cloud_core::events::Event; -use futures::SinkExt as _; -use serde_json::{json, Value}; - -const T: Duration = Duration::from_secs(5); - -fn canvas_uuid(id: &str) -> uuid::Uuid { - uuid::Uuid::parse_str(id).unwrap() -} - -#[tokio::test] -async fn two_clients_sync_both_ways_and_relay_awareness() { - let app = TestApp::new().await; - let a = app.user("alice").await; - let b = app.user("bert").await; - let id = app.canvas(&a, "Live").await; - app.share(&a, &id, &b).await; - - let mut pa = app.peer(&a, &id).await; - let mut pb = app.peer(&b, &id).await; - - // A → B - pa.edit(|txn| add_sticky(txn, "s1", "hello", 0.0)).await; - assert!( - pb.pump_until(T, |p| p.shape_prop("s1", "text") == Some(json!("hello"))) - .await, - "B sees A's sticky" - ); - // B → A - pb.edit(|txn| set_prop(txn, "s1", "text", "hello back")) - .await; - assert!( - pa.pump_until(T, |p| p.shape_prop("s1", "text") - == Some(json!("hello back"))) - .await, - "A sees B's edit" - ); - - // Awareness: A's presence reaches B; leaving clears it. - let a_client = pa.client_id(); - pa.set_awareness( - 1, - r#"{"user":{"name":"alice","color":"pink"},"cursor":{"x":1,"y":2}}"#, - ) - .await; - assert!( - pb.pump_until(T, |p| p - .awareness - .get(&a_client) - .is_some_and(|j| j.contains("alice"))) - .await, - "B sees A's awareness" - ); - // A late joiner gets the current awareness in its greeting. - let c = app.user("cleo").await; - app.share(&a, &id, &c).await; - let mut pc = app.peer(&c, &id).await; - assert!( - pc.pump_until(T, |p| p.awareness.contains_key(&a_client)) - .await - ); - assert_eq!(pc.shape_prop("s1", "text"), Some(json!("hello back"))); - - pa.close().await; - assert!( - pb.pump_until(T, |p| p - .awareness - .get(&a_client) - .is_some_and(|j| j == "null")) - .await, - "A's awareness is removed when A disconnects" - ); - assert!(copper_cloud_canvas::rooms_metrics().peers >= 2); - pb.close().await; - pc.close().await; - - // The server copy matches. - let doc = app.state_doc(&a, &id).await; - assert_eq!( - common::shape_prop(&doc, "s1", "text"), - Some(json!("hello back")) - ); -} - -#[tokio::test] -async fn personal_canvas_room_over_ws() { - let app = TestApp::new().await; - let a = app.user("solo").await; - let mut p1 = app.peer(&a, "personal").await; - p1.edit(|txn| add_sticky(txn, "me", "private", 0.0)).await; - let mut p2 = app.peer(&a, "personal").await; - assert!( - p2.pump_until(T, |p| p.shape_prop("me", "text") == Some(json!("private"))) - .await - ); - p1.close().await; - p2.close().await; -} - -#[tokio::test] -async fn persistence_survives_room_eviction() { - let app = TestApp::new().await; - let a = app.user("keeper").await; - let id = app.canvas(&a, "Durable").await; - let cid = canvas_uuid(&id); - - let mut p = app.peer(&a, &id).await; - for i in 0..3 { - let sid = format!("n{i}"); - p.edit(|txn| add_sticky(txn, &sid, &format!("note {i}"), f64::from(i) * 220.0)) - .await; - } - // Wait until the server has applied (and therefore persisted) all three. - assert!( - eventually(T, || async { - common::shape_count(&app.state_doc(&a, &id).await) == 3 - }) - .await - ); - p.close().await; - - assert!( - eventually(T, || async { copper_cloud_canvas::evict_room_now(cid) }).await, - "room evicts once the peer is gone" - ); - assert!(!copper_cloud_canvas::room_in_memory(cid)); - - // Reload from Postgres. - let doc = app.state_doc(&a, &id).await; - assert_eq!(common::shape_count(&doc), 3); - assert_eq!( - common::shape_prop(&doc, "n2", "text"), - Some(json!("note 2")) - ); - assert!(copper_cloud_canvas::room_in_memory(cid)); - - // A new peer syncs the persisted content. - let p2 = app.peer(&a, &id).await; - assert_eq!(p2.shape_count(), 3); - p2.close().await; - - // Stored blobs are sealed: no plaintext in the table. - let blobs: Vec> = - sqlx::query_scalar(r#"SELECT "update" FROM canvas_updates WHERE canvas_id = $1"#) - .bind(cid) - .fetch_all(app.db()) - .await - .unwrap(); - assert_ne!(blobs.len(), 0, "updates must be persisted"); - for blob in blobs { - assert!( - !blob.windows(6).any(|w| w == b"note 1"), - "update stored in plaintext" - ); - } -} - -#[tokio::test] -async fn compaction_after_200_updates() { - let app = TestApp::new().await; - let a = app.user("busy").await; - let id = app.canvas(&a, "Busy").await; - let cid = canvas_uuid(&id); - - let mut p = app.peer(&a, &id).await; - let n = 250; - for i in 0..n { - let sid = format!("s{i}"); - p.edit(|txn| add_sticky(txn, &sid, "x", f64::from(i))).await; - } - assert!( - eventually(Duration::from_secs(20), || async { - let (_, read) = app.get(&a, &format!("/canvases/{id}/read")).await; - read["shapes"].as_array().map_or(0, Vec::len) == n as usize - }) - .await, - "all updates applied" - ); - - let snapshot: Option = - sqlx::query_scalar("SELECT seq FROM canvas_snapshots WHERE canvas_id = $1") - .bind(cid) - .fetch_optional(app.db()) - .await - .unwrap(); - let snap_seq = snapshot.expect("a snapshot exists after > 200 updates"); - let (remaining, min_seq): (i64, Option) = - sqlx::query_as("SELECT count(*), min(seq) FROM canvas_updates WHERE canvas_id = $1") - .bind(cid) - .fetch_one(app.db()) - .await - .unwrap(); - assert!( - remaining < 200, - "older updates were deleted ({remaining} left)" - ); - assert!( - min_seq.is_none_or(|m| m > snap_seq), - "no update at or below the snapshot seq" - ); - p.close().await; - - // Snapshot + tail reload to the same content. - assert!(eventually(T, || async { copper_cloud_canvas::evict_room_now(cid) }).await); - let doc = app.state_doc(&a, &id).await; - assert_eq!(common::shape_count(&doc), n); -} - -#[tokio::test] -async fn ops_endpoint_reaches_ws_peers() { - let app = TestApp::new().await; - let a = app.user("agent-owner").await; - let id = app.canvas(&a, "Agents").await; - let base = format!("/canvases/{id}"); - let mut events = app.state.events.subscribe(a.id); - - let mut p = app.peer(&a, &id).await; - let (s, r) = app - .post( - &a, - &format!("{base}/ops"), - json!({ - "as": { "name": "Planner", "color": "purple" }, - "ops": [ - { "op": "add", "shape": { "type": "sticky", "text": "idea" } }, - { "op": "add", "shape": { "id": "t1", "type": "text", "text": "title", "x": 0, "y": -300 } }, - { "op": "bogus" } - ] - }), - ) - .await; - assert_eq!(s, StatusCode::OK, "{r}"); - assert_eq!(r["applied"], 2, "{r}"); - assert_eq!(r["errors"][0]["index"], 2); - let sticky = r["ids"][0].as_str().unwrap().to_owned(); - assert_eq!(r["ids"][1], "t1"); - assert!( - p.pump_until(T, |p| p.shape_prop(&sticky, "text") == Some(json!("idea")) - && p.shape_prop("t1", "type") == Some(json!("text"))) - .await - ); - assert_eq!(p.shape_prop(&sticky, "by"), Some(json!("Planner"))); - assert_eq!(p.shape_prop(&sticky, "color"), Some(json!("yellow"))); - - // connect / update / move / resize, then delete. - let (s, r) = app - .post( - &a, - &format!("{base}/ops"), - json!({ "ops": [ - { "op": "connect", "from": sticky, "to": "t1", "label": "explains" }, - { "op": "update", "id": sticky, "patch": { "text": "better idea", "color": "green" } }, - { "op": "move", "id": "t1", "dx": 10, "dy": 20 }, - { "op": "resize", "id": sticky, "w": 260, "h": 180 } - ]}), - ) - .await; - assert_eq!(s, StatusCode::OK, "{r}"); - assert_eq!(r["applied"], 4, "{r}"); - let arrow = r["ids"][0].as_str().unwrap().to_owned(); - assert!( - p.pump_until(T, |p| p.shape_prop(&sticky, "text") - == Some(json!("better idea")) - && p.shape_prop("t1", "y") == Some(json!(-280)) - && p.shape_prop(&sticky, "w") == Some(json!(260)) - && p.shape_prop(&arrow, "label") == Some(json!("explains"))) - .await - ); - assert_eq!(p.shape_prop(&arrow, "from"), Some(json!({ "ref": sticky }))); - assert_eq!( - p.shape_prop(&sticky, "by"), - Some(json!("Planner")), - "by is the creator" - ); - assert_eq!( - p.shape_prop(&arrow, "by"), - Some(json!("agent-owner")), - "without `as`, by = the caller's display name" - ); - - let (s, r) = app - .post( - &a, - &format!("{base}/ops"), - json!({ "ops": [{ "op": "delete", "id": "t1" }] }), - ) - .await; - assert_eq!(s, StatusCode::OK); - assert_eq!(r["applied"], 1); - assert!( - p.pump_until(T, |p| p.shape_prop("t1", "type").is_none() - && p.shape_prop(&arrow, "type").is_none()) - .await, - "delete cascades to the arrow" - ); - - // clear needs confirmation. - let (_, r) = app - .post( - &a, - &format!("{base}/ops"), - json!({ "ops": [{ "op": "clear" }] }), - ) - .await; - assert_eq!(r["applied"], 0); - assert_eq!(p.shape_count(), 1); - let (_, r) = app - .post( - &a, - &format!("{base}/ops"), - json!({ "ops": [{ "op": "clear" }], "confirm": true }), - ) - .await; - assert_eq!(r["applied"], 1); - assert!(p.pump_until(T, |p| p.shape_count() == 0).await); - - // Agent presence: `writing` with the cursor on the work, then `idle` after 1.5 s. - let (_, read) = app.get(&a, &format!("{base}/read")).await; - let agents = read["agents"].as_array().unwrap(); - assert_eq!(agents.len(), 1, "{read}"); - assert_eq!(agents[0]["id"], "agent:planner"); - assert_eq!(agents[0]["name"], "Planner"); - assert_eq!(agents[0]["color"], "purple"); - assert!(agents[0]["cursor"]["x"].is_number()); - assert!( - eventually(Duration::from_secs(4), || async { - let (_, read) = app.get(&a, &format!("{base}/read")).await; - read["agents"][0]["status"] == "idle" - }) - .await, - "agent goes idle" - ); - - // Envelope errors come back as index -1. - let (s, r) = app - .post(&a, &format!("{base}/ops"), json!({ "ops": [], "as": 5 })) - .await; - assert_eq!(s, StatusCode::OK); - assert_eq!(r["errors"][0]["index"], -1); - let too_many: Vec = (0..501).map(|_| json!({ "op": "clear" })).collect(); - let (s, r) = app - .post(&a, &format!("{base}/ops"), json!({ "ops": too_many })) - .await; - assert_eq!(s, StatusCode::OK); - assert_eq!(r["applied"], 0); - assert!(r["errors"][0]["error"] - .as_str() - .unwrap() - .contains("at most 500")); - - // Members were told the canvas changed (throttled). - let got = tokio::time::timeout(Duration::from_secs(4), async { - loop { - match events.recv().await { - Ok(Event::Canvas { canvas_id, kind }) - if canvas_id.to_string() == id && kind == "update" => - { - return true; - } - Ok(_) => {} - Err(_) => return false, - } - } - }) - .await; - assert_eq!(got, Ok(true)); - p.close().await; -} - -#[tokio::test] -async fn revoked_and_deleted_canvases_close_sockets() { - let app = TestApp::new().await; - let a = app.user("boss").await; - let b = app.user("temp").await; - let id = app.canvas(&a, "Short-lived").await; - app.share(&a, &id, &b).await; - - let mut pb = app.peer(&b, &id).await; - let mut pa = app.peer(&a, &id).await; - let (s, _) = app - .delete(&a, &format!("/canvases/{id}/members/{}", b.id)) - .await; - assert_eq!(s, StatusCode::OK); - assert_eq!( - pb.wait_closed(T).await, - Some(copper_cloud_canvas::close_codes::REVOKED) - ); - // The owner's socket stays up. - pa.edit(|txn| add_sticky(txn, "still", "here", 0.0)).await; - assert!( - eventually(T, || async { - common::shape_count(&app.state_doc(&a, &id).await) == 1 - }) - .await - ); - - let (s, _) = app.delete(&a, &format!("/canvases/{id}")).await; - assert_eq!(s, StatusCode::OK); - assert_eq!( - pa.wait_closed(T).await, - Some(copper_cloud_canvas::close_codes::DELETED) - ); - assert!(!copper_cloud_canvas::room_in_memory(canvas_uuid(&id))); -} - -#[tokio::test] -async fn invalid_frames_close_the_peer() { - let app = TestApp::new().await; - let a = app.user("fuzzer").await; - let id = app.canvas(&a, "Fuzz").await; - let mut p = app.peer(&a, &id).await; - // Sync message with an unknown sub-type. - p.ws.send(tokio_tungstenite::tungstenite::Message::Binary( - vec![0u8, 9, 1, 2].into(), - )) - .await - .unwrap(); - assert_eq!( - p.wait_closed(T).await, - Some(copper_cloud_canvas::close_codes::INVALID) - ); - // The room still works for others. - let mut q = app.peer(&a, &id).await; - q.edit(|txn| add_sticky(txn, "ok", "fine", 0.0)).await; - assert!( - eventually(T, || async { - common::shape_count(&app.state_doc(&a, &id).await) == 1 - }) - .await - ); - q.close().await; -} - -#[tokio::test] -async fn oversized_messages_are_refused() { - let app = TestApp::new().await; - let a = app.user("huge").await; - let id = app.canvas(&a, "Huge").await; - let mut p = app.peer(&a, &id).await; - let big = vec![0u8; copper_cloud_canvas::MAX_MESSAGE_BYTES + 1]; - let _ = - p.ws.send(tokio_tungstenite::tungstenite::Message::Binary(big.into())) - .await; - let code = p.wait_closed(T).await; - eprintln!("close code: {code:?}"); - assert_eq!(code, Some(1009)); -} diff --git a/crates/copper-cloud/Cargo.toml b/crates/copper-cloud/Cargo.toml index 8fc8024..926dcaa 100644 --- a/crates/copper-cloud/Cargo.toml +++ b/crates/copper-cloud/Cargo.toml @@ -26,9 +26,11 @@ workspace = true [dependencies] copper-cloud-core.workspace = true -copper-cloud-canvas.workspace = true +copper-cloud-canvas-link.workspace = true +copper-cloud-canvas-migrate.workspace = true anyhow.workspace = true axum.workspace = true +reqwest.workspace = true axum-server.workspace = true clap.workspace = true hex.workspace = true @@ -41,12 +43,14 @@ time.workspace = true tokio.workspace = true tokio-rustls.workspace = true tracing.workspace = true +url.workspace = true uuid.workspace = true zeroize.workspace = true [dev-dependencies] futures.workspace = true reqwest.workspace = true +tokio-tungstenite.workspace = true serde_json.workspace = true tokio.workspace = true uuid.workspace = true diff --git a/crates/copper-cloud/migrations/0030_canvas_link.sql b/crates/copper-cloud/migrations/0030_canvas_link.sql new file mode 100644 index 0000000..e2b81f4 --- /dev/null +++ b/crates/copper-cloud/migrations/0030_canvas_link.sql @@ -0,0 +1,58 @@ +-- Canvas link (copper-cloud ≥ 0.9): canvases live on a linked Canvas Cloud server; copper-cloud +-- proxies the canvas API there as the signed-in user (crate `copper-cloud-canvas-link`). +-- The legacy canvas tables (0100–0103) stay untouched: `copper-cloud canvas migrate` reads them, +-- and an older copper-cloud can be reinstalled against them (rollback). + +-- The link (singleton row, id = 1). The Canvas Cloud app key is envelope-encrypted like the +-- intelligence keys: `data_key_wrapped` is a random 32-byte key wrapped by the master-key KEK; +-- `app_key_sealed` is AES-256-GCM under it with AAD 'canvas_link:app_key'. +-- url base URL, no trailing '/' (https://…, or http:// for a loopback host) +-- tls_fingerprint lower-case hex SHA-256 of the pinned leaf certificate (self-signed +-- Canvas Cloud); NULL = verify against the web PKI +-- enabled false = keep the settings but serve 503 canvas_unavailable +-- linked_at first link to this URL (kept across key rotations) +-- updated_by admin email, or 'cli' +CREATE TABLE canvas_link ( + id int PRIMARY KEY DEFAULT 1 CHECK (id = 1), + url text NOT NULL CHECK (length(url) BETWEEN 8 AND 2048), + app_key_sealed bytea NOT NULL, + data_key_wrapped bytea NOT NULL, + tls_fingerprint text CHECK (tls_fingerprint ~ '^[0-9a-f]{64}$'), + enabled boolean NOT NULL DEFAULT true, + linked_at timestamptz NOT NULL DEFAULT now(), + updated_at timestamptz NOT NULL DEFAULT now(), + updated_by text +); + +-- User provisioning work that has to reach Canvas Cloud (pushed right away; kept and retried +-- with backoff while Canvas Cloud is unreachable). Dropped when the link is removed. +-- kind 'upsert_user' | 'delete_user' | 'disconnect_user' +-- subject the Copper user id (= Canvas Cloud subject and user id) +-- next_attempt_at (added column) when the item is due again +CREATE TABLE canvas_link_outbox ( + id bigserial PRIMARY KEY, + kind text NOT NULL CHECK (kind IN ('upsert_user', 'delete_user', 'disconnect_user')), + subject uuid NOT NULL, + payload jsonb, + created_at timestamptz NOT NULL DEFAULT now(), + attempts int NOT NULL DEFAULT 0, + last_error text, + next_attempt_at timestamptz NOT NULL DEFAULT now() +); + +CREATE INDEX canvas_link_outbox_due_idx ON canvas_link_outbox (next_attempt_at, id); + +-- Migration ledger of `copper-cloud canvas migrate` (idempotent re-runs, rollback evidence). +-- status 'migrated' | 'failed' +-- digest canonical document digest reported by Canvas Cloud (CONTRACT §6.3) +-- updates number of legacy updates (incl. the snapshot) sent +-- detail counts / error context +-- IF NOT EXISTS: the migrate crate's tests may have created it ahead of this migration. +CREATE TABLE IF NOT EXISTS canvas_migrations ( + canvas_id uuid PRIMARY KEY, + status text NOT NULL, + digest text, + updates int, + migrated_at timestamptz DEFAULT now(), + detail jsonb +); diff --git a/crates/copper-cloud/src/admin_api.rs b/crates/copper-cloud/src/admin_api.rs index b2193bc..d561fa1 100644 --- a/crates/copper-cloud/src/admin_api.rs +++ b/crates/copper-cloud/src/admin_api.rs @@ -16,7 +16,9 @@ use axum::http::{header, HeaderMap, HeaderValue, Method}; use axum::middleware::{from_fn, Next}; use axum::response::{IntoResponse, Response}; use axum::routing::{delete, get, patch, post}; -use axum::{Json, Router}; +use axum::{Extension, Json, Router}; +use copper_cloud_canvas_link::admin::{self as link_admin, Actor, LinkError}; +use copper_cloud_canvas_link::{provision, CanvasLink}; use copper_cloud_core::access::{self, AccessMode, NewAccessKey}; use copper_cloud_core::auth; use copper_cloud_core::config::TlsMode; @@ -59,6 +61,13 @@ pub fn router() -> Router { .route("/devices/{id}", delete(delete_device)) .route("/canvases", get(list_canvases)) .route("/canvases/{id}", delete(delete_canvas)) + .route( + "/canvas-link", + get(get_canvas_link) + .put(put_canvas_link) + .delete(delete_canvas_link), + ) + .route("/canvas-link/test", post(test_canvas_link)) .route("/pairing-codes", get(list_pairing_codes)) .route("/pairing-codes/{id}", delete(revoke_pairing_code)) .route("/audit", get(list_audit)) @@ -478,18 +487,22 @@ async fn change_password( // --------------------------------------------------------------------------------------------- // Overview + settings -async fn overview(State(state): State, _s: AdminSession) -> ApiResult> { - let (users, devices, canvases, keys): (i64, i64, i64, i64) = sqlx::query_as( +async fn overview( + State(state): State, + Extension(link): Extension, + _s: AdminSession, +) -> ApiResult> { + let (users, devices, keys): (i64, i64, i64) = sqlx::query_as( "SELECT (SELECT count(*) FROM users), (SELECT count(*) FROM devices), - (SELECT count(*) FROM canvases), (SELECT count(*) FROM access_keys WHERE revoked_at IS NULL AND (expires_at IS NULL OR expires_at > now()) AND (max_uses IS NULL OR uses < max_uses))", ) .fetch_one(&state.db) .await?; - let rooms = copper_cloud_canvas::rooms_metrics(); + // Canvases live on the linked Canvas Cloud: null when unlinked or unreachable. + let canvas = link_admin::stats(&link).await; let mode = access::access_mode(&state).await?; let fingerprint = match state.cfg.tls.mode { TlsMode::Acme => None, @@ -503,10 +516,10 @@ async fn overview(State(state): State, _s: AdminSession) -> ApiResu "counts": { "users": users, "devices": devices, - "canvases": canvases, + "canvases": canvas.map(|c| c.canvases), "access_keys": keys, - "live_rooms": rooms.rooms, - "live_peers": rooms.peers, + "live_rooms": canvas.map(|c| c.rooms.rooms), + "live_peers": canvas.map(|c| c.rooms.peers), }, "public_url": state.cfg.public_url, "tls": { "mode": state.cfg.tls.mode.to_string(), "fingerprint": fingerprint }, @@ -837,14 +850,28 @@ struct UserRow { last_seen_at: Option, disabled: bool, device_count: i64, - canvas_count: i64, + /// Canvases on the linked Canvas Cloud this user belongs to; `null` when unlinked or + /// Canvas Cloud is unreachable. + #[sqlx(skip)] + canvas_count: Option, +} + +/// Fill `canvas_count` from the linked Canvas Cloud (left `null` when unavailable). +async fn with_canvas_counts(link: &CanvasLink, rows: &mut [UserRow]) { + if rows.is_empty() { + return; + } + if let Some(counts) = link_admin::canvas_counts(link).await { + for row in rows { + row.canvas_count = Some(counts.get(&row.id).copied().unwrap_or(0)); + } + } } const USER_VIEW_SQL: &str = " SELECT u.id, u.email, u.display_name, u.created_at, u.disabled, (SELECT max(d.last_seen_at) FROM devices d WHERE d.user_id = u.id) AS last_seen_at, - (SELECT count(*) FROM devices d WHERE d.user_id = u.id) AS device_count, - (SELECT count(*) FROM canvas_members m WHERE m.user_id = u.id) AS canvas_count + (SELECT count(*) FROM devices d WHERE d.user_id = u.id) AS device_count FROM users u WHERE ($1::uuid IS NULL OR u.id = $1) AND ($2::text IS NULL OR u.email ILIKE $2 OR u.display_name ILIKE $2)"; @@ -860,12 +887,13 @@ async fn user_by_id(db: &sqlx::PgPool, id: Uuid) -> ApiResult { async fn list_users( State(state): State, + Extension(link): Extension, _s: AdminSession, QueryParams(q): QueryParams, ) -> ApiResult>> { let (limit, offset) = q.page(); let pattern = empty_to_none(q.q.as_deref()).map(like_pattern); - let items = sqlx::query_as::<_, UserRow>(&format!( + let mut items = sqlx::query_as::<_, UserRow>(&format!( "{USER_VIEW_SQL} ORDER BY u.created_at DESC, u.id DESC LIMIT $3 OFFSET $4" )) .bind(None::) @@ -881,6 +909,7 @@ async fn list_users( .bind(&pattern) .fetch_one(&state.db) .await?; + with_canvas_counts(&link, &mut items).await; Ok(Json(ListResponse { items, total, @@ -897,6 +926,7 @@ struct UserPatch { async fn update_user( State(state): State, + Extension(link): Extension, s: AdminSession, Path(raw): Path, JsonBody(req): JsonBody, @@ -949,15 +979,19 @@ async fn update_user( Some(Value::Object(detail)), ) .await?; + // Mirror the change to the linked Canvas Cloud (and close a disabled user's canvas + // sockets there); queued in this transaction so it survives Canvas Cloud being down. + provision::enqueue_user_changed(&mut tx, id, req.disabled == Some(true)).await?; tx.commit().await?; - if req.disabled == Some(true) { - copper_cloud_canvas::admin::disconnect_user(id); - } - Ok(Json(user_by_id(&state.db, id).await?)) + link.kick_outbox(); + let mut row = user_by_id(&state.db, id).await?; + with_canvas_counts(&link, std::slice::from_mut(&mut row)).await; + Ok(Json(row)) } async fn delete_user( State(state): State, + Extension(link): Extension, s: AdminSession, Path(raw): Path, ) -> ApiResult> { @@ -965,11 +999,11 @@ async fn delete_user( let email = auth::user_email(&state.db, id) .await? .ok_or(ApiError::NotFound)?; - let owned = copper_cloud_canvas::admin::owned_canvases(&state, id).await?; let mut tx = state.db.begin().await?; // Credentials and invites addressed to this person; everything keyed by user_id - // (sessions, devices, sync docs, history, pairing codes, owned canvases with their - // members/invites/updates, memberships) goes with the user row (FK cascade). + // (sessions, devices, sync docs, history, pairing codes, legacy canvas rows, + // memberships) goes with the user row (FK cascade). Their canvases on the linked Canvas + // Cloud are deleted there through the outbox below. sqlx::query("DELETE FROM access_keys WHERE lower(email) = lower($1)") .bind(&email) .execute(&mut *tx) @@ -994,9 +1028,9 @@ async fn delete_user( Some(json!({ "email": email })), ) .await?; + provision::enqueue_user_deleted(&mut tx, id).await?; tx.commit().await?; - copper_cloud_canvas::admin::canvases_deleted(&state, &owned); - copper_cloud_canvas::admin::disconnect_user(id); + link.kick_outbox(); Ok(ok()) } @@ -1135,65 +1169,43 @@ async fn delete_device( // --------------------------------------------------------------------------------------------- // Canvases -#[derive(Debug, Serialize, sqlx::FromRow)] -struct CanvasRow { - id: Uuid, - name: String, - kind: String, - owner_id: Uuid, - owner_email: String, - member_count: i64, - share_link_count: i64, - #[serde(with = "time::serde::rfc3339")] - created_at: OffsetDateTime, - #[serde(with = "time::serde::rfc3339")] - updated_at: OffsetDateTime, -} +// The canvases live on the linked Canvas Cloud (`/v1/apps/self/canvases`); the list keeps +// its shape (`items` of {id, name, kind, owner_id, owner_email, member_count, +// share_link_count, created_at, updated_at}, `total`, `limit`, `offset`) plus `linked`. +// Unlinked: an empty list with `linked: false`; Canvas Cloud down: `502 canvas_upstream`. async fn list_canvases( - State(state): State, + Extension(link): Extension, _s: AdminSession, QueryParams(q): QueryParams, -) -> ApiResult>> { +) -> Result, LinkError> { let (limit, offset) = q.page(); let kind = empty_to_none(q.kind.as_deref()); if kind.is_some_and(|k| !matches!(k, "personal" | "shared")) { - return Err(ApiError::bad_request("kind must be personal or shared")); + return Err(ApiError::bad_request("kind must be personal or shared").into()); } - let items = sqlx::query_as::<_, CanvasRow>( - "SELECT c.id, c.name, c.kind, c.owner_id, u.email AS owner_email, - (SELECT count(*) FROM canvas_members m WHERE m.canvas_id = c.id) AS member_count, - (SELECT count(*) FROM canvas_share_links l WHERE l.canvas_id = c.id) AS share_link_count, - c.created_at, c.updated_at - FROM canvases c JOIN users u ON u.id = c.owner_id - WHERE ($1::text IS NULL OR c.kind = $1) - ORDER BY c.created_at DESC, c.id DESC LIMIT $2 OFFSET $3", - ) - .bind(kind) - .bind(limit) - .bind(offset) - .fetch_all(&state.db) - .await?; - let total: i64 = - sqlx::query_scalar("SELECT count(*) FROM canvases WHERE ($1::text IS NULL OR kind = $1)") - .bind(kind) - .fetch_one(&state.db) - .await?; - Ok(Json(ListResponse { - items, - total, - limit, - offset, - })) + let (items, total, linked) = match link_admin::list_canvases(&link, kind, limit, offset).await? + { + Some((items, total)) => (items, total, true), + None => (Vec::new(), 0, false), + }; + Ok(Json(json!({ + "items": items, + "total": total, + "limit": limit, + "offset": offset, + "linked": linked, + }))) } async fn delete_canvas( State(state): State, + Extension(link): Extension, s: AdminSession, Path(raw): Path, -) -> ApiResult> { +) -> Result, LinkError> { let id = parse_id(&raw)?; - let name = copper_cloud_canvas::admin::delete_canvas(&state, id) + let name = link_admin::delete_canvas(&link, id) .await? .ok_or(ApiError::NotFound)?; audit( @@ -1207,6 +1219,80 @@ async fn delete_canvas( Ok(ok()) } +// --------------------------------------------------------------------------------------------- +// Canvas Cloud link + +fn link_actor(s: &AdminSession) -> Actor { + Actor::Admin { + id: s.admin_id, + email: s.email.clone(), + } +} + +async fn get_canvas_link( + Extension(link): Extension, + _s: AdminSession, +) -> ApiResult> { + Ok(Json(link_admin::view(&link).await?)) +} + +#[derive(Deserialize)] +struct CanvasLinkPut { + url: String, + app_key: String, + tls_fingerprint: Option, +} + +async fn put_canvas_link( + Extension(link): Extension, + s: AdminSession, + JsonBody(req): JsonBody, +) -> ApiResult> { + let app_key = zeroize::Zeroizing::new(req.app_key); + link_admin::set( + &link, + &link_actor(&s), + &req.url, + &app_key, + req.tls_fingerprint.as_deref(), + ) + .await?; + // Provision every user now (the reconcile loop also runs on the link change). + let bg = link.clone(); + tokio::spawn(async move { + if let Ok(Some(active)) = bg.enabled().await { + match provision::sync_all(&bg, &active).await { + Ok(r) => tracing::info!( + provisioned = r.provisioned, + total = r.total, + failed = r.failed.len(), + "canvas users provisioned" + ), + Err(err) => tracing::warn!( + error = format!("{err:#}"), + "canvas user provisioning failed" + ), + } + } + }); + Ok(Json(link_admin::view(&link).await?)) +} + +async fn delete_canvas_link( + Extension(link): Extension, + s: AdminSession, +) -> ApiResult> { + link_admin::remove(&link, &link_actor(&s)).await?; + Ok(ok()) +} + +async fn test_canvas_link( + Extension(link): Extension, + s: AdminSession, +) -> ApiResult> { + Ok(Json(link_admin::test(&link, &link_actor(&s)).await?)) +} + // --------------------------------------------------------------------------------------------- // Pairing codes diff --git a/crates/copper-cloud/src/cli.rs b/crates/copper-cloud/src/cli.rs index dc748c7..80c0e8d 100644 --- a/crates/copper-cloud/src/cli.rs +++ b/crates/copper-cloud/src/cli.rs @@ -68,6 +68,59 @@ pub enum Command { /// org-wide agent tool-call round budget. #[command(subcommand)] Intelligence(IntelligenceCommand), + /// Link to a Canvas Cloud server (where canvases live), check it, provision users and + /// migrate legacy canvases into it. + #[command(subcommand)] + Canvas(CanvasCommand), +} + +#[derive(Subcommand, Debug)] +pub enum CanvasCommand { + /// Link (or re-link) a Canvas Cloud: checks /v1/info and /v1/apps/self/stats with the + /// app key, stores the key encrypted, then provisions every user. A running server picks + /// the change up within seconds. + Link { + /// Canvas Cloud base URL: https://… (http:// only for a loopback host). + #[arg(long, value_name = "URL")] + url: String, + /// File holding the Canvas Cloud app key (cva_…); "-" reads stdin. Never argv. + #[arg(long, value_name = "FILE")] + key_file: PathBuf, + /// SHA-256 of a self-signed Canvas Cloud's certificate (hex, colons allowed). + #[arg(long, value_name = "HEX")] + fingerprint: Option, + /// Do not provision users now (the server's reconcile loop does it later). + #[arg(long)] + no_sync: bool, + }, + /// Remove the link (canvas routes answer 503 canvas_unavailable until linked again). + Unlink, + /// Show the link (key masked) and whether Canvas Cloud is reachable. + Status, + /// Provision every user on the linked Canvas Cloud now. + SyncUsers, + /// Copy legacy canvases (this database's canvas tables) into the linked Canvas Cloud, + /// verifying each by digest. Legacy rows are never modified. + Migrate(CanvasMigrateArgs), +} + +#[derive(Args, Debug, Default)] +pub struct CanvasMigrateArgs { + /// Read, decrypt and count everything without sending it. + #[arg(long)] + pub dry_run: bool, + /// Only this canvas (repeatable); re-sends it even if already migrated. + #[arg(long = "only", value_name = "ID")] + pub only: Vec, + /// Replace canvases that already exist on Canvas Cloud (from an earlier import). + #[arg(long)] + pub replace: bool, + /// Canvases migrated in parallel. + #[arg(long, default_value_t = 2, value_parser = clap::value_parser!(u16).range(1..=16))] + pub concurrency: u16, + /// Print the report as JSON. + #[arg(long)] + pub json: bool, } #[derive(Subcommand, Debug)] @@ -445,7 +498,199 @@ pub fn run(cli: Cli) -> anyhow::Result { runtime()?.block_on(crate::admin::intelligence(cfg, cmd))?; Ok(ExitCode::SUCCESS) } + Command::Canvas(cmd) => { + let cfg = load_config(cli.config)?; + runtime()?.block_on(canvas(cfg, cmd)) + } + } +} + +// --------------------------------------------------------------------------------------------- +// `copper-cloud canvas …` + +#[allow(clippy::needless_pass_by_value)] // used as `map_err(api_err)` +fn api_err(e: copper_cloud_core::error::ApiError) -> anyhow::Error { + anyhow::anyhow!("{e}") +} + +/// Read the app key from `path` (`-` = stdin), validated, never printed. +fn read_app_key(path: &std::path::Path) -> anyhow::Result> { + let raw = zeroize::Zeroizing::new(if path.as_os_str() == "-" { + let mut s = String::new(); + std::io::Read::read_to_string(&mut std::io::stdin().lock(), &mut s) + .context("reading the app key from stdin")?; + s + } else { + std::fs::read_to_string(path) + .with_context(|| format!("reading the app key from {}", path.display()))? + }); + copper_cloud_canvas_link::settings::clean_app_key(&raw).map_err(api_err) +} + +async fn canvas(cfg: Config, cmd: CanvasCommand) -> anyhow::Result { + use copper_cloud_canvas_link::admin::{self as link_admin, Actor}; + use copper_cloud_canvas_link::{settings, CanvasLink}; + + // Read the key before touching the database (fail fast). + let key = match &cmd { + CanvasCommand::Link { key_file, .. } => Some(read_app_key(key_file)?), + _ => None, + }; + let pool = copper_cloud_core::db::connect(&cfg).await?; + let state = copper_cloud_core::state::AppState::new(pool.clone(), cfg); + let link = CanvasLink::new(state.clone()); + let result = async { + match cmd { + CanvasCommand::Link { + url, + fingerprint, + no_sync, + .. + } => { + let key = key.unwrap_or_default(); + let active = + link_admin::set(&link, &Actor::Cli, &url, &key, fingerprint.as_deref()) + .await + .map_err(api_err)?; + let health = link.health(); + println!( + "linked {} (Canvas Cloud {}, key …{})", + active.settings.base(), + health.version.as_deref().unwrap_or("?"), + settings::last4(&active.settings.app_key) + ); + if no_sync { + println!("users not provisioned (--no-sync); the server's reconcile loop will"); + return Ok(ExitCode::SUCCESS); + } + sync_users(&link, &active).await + } + CanvasCommand::Unlink => { + match link_admin::remove(&link, &Actor::Cli) + .await + .map_err(api_err)? + { + Some(url) => println!("unlinked {url}"), + None => println!("no Canvas Cloud was linked"), + } + Ok(ExitCode::SUCCESS) + } + CanvasCommand::Status => { + let view = link_admin::view(&link).await.map_err(api_err)?; + print_status(&view); + let reachable = view["status"]["reachable"].as_bool(); + Ok(if view["linked"] == true && reachable != Some(true) { + ExitCode::FAILURE + } else { + ExitCode::SUCCESS + }) + } + CanvasCommand::SyncUsers => { + let active = link + .current() + .await + .map_err(api_err)? + .context("no Canvas Cloud is linked (copper-cloud canvas link …)")?; + sync_users(&link, &active).await + } + CanvasCommand::Migrate(args) => migrate_canvases(&state, &link, args).await, + } + } + .await; + pool.close().await; + result +} + +async fn migrate_canvases( + state: &copper_cloud_core::state::SharedState, + link: &copper_cloud_canvas_link::CanvasLink, + args: CanvasMigrateArgs, +) -> anyhow::Result { + let active = link + .current() + .await + .map_err(api_err)? + .context("no Canvas Cloud is linked (copper-cloud canvas link …)")?; + let opts = copper_cloud_canvas_migrate::MigrateOptions { + dry_run: args.dry_run, + only: args.only, + replace: args.replace, + concurrency: usize::from(args.concurrency), + json: args.json, + }; + let report = copper_cloud_canvas_migrate::run( + &state.db, + &state.crypto, + active.upstream.client(), + &active.settings.url, + &active.settings.app_key, + opts, + ) + .await?; + if args.json { + println!("{}", serde_json::to_string_pretty(&report)?); + } else { + println!("{report}"); + } + Ok(if report.ok() { + ExitCode::SUCCESS + } else { + ExitCode::FAILURE + }) +} + +async fn sync_users( + link: &copper_cloud_canvas_link::CanvasLink, + active: &copper_cloud_canvas_link::Active, +) -> anyhow::Result { + let report = copper_cloud_canvas_link::provision::sync_all(link, active).await?; + println!( + "provisioned {}/{} users on Canvas Cloud", + report.provisioned, report.total + ); + for (id, err) in &report.failed { + println!(" failed {id}: {err}"); + } + for id in &report.id_mismatch { + println!( + " {id}: Canvas Cloud linked an existing account with a different id (chat author ids will not match)" + ); + } + Ok(if report.ok() { + ExitCode::SUCCESS + } else { + ExitCode::FAILURE + }) +} + +fn print_status(v: &serde_json::Value) { + let s = |v: &serde_json::Value| v.as_str().map_or_else(|| "-".to_owned(), str::to_owned); + if v["linked"] == true { + println!("canvas link: {}", s(&v["url"])); + println!(" enabled: {}", v["enabled"]); + println!(" pinned cert: {}", s(&v["tls_fingerprint"])); + println!(" linked at: {}", s(&v["linked_at"])); + println!( + " updated: {} by {}", + s(&v["updated_at"]), + s(&v["updated_by"]) + ); + match v["status"]["reachable"].as_bool() { + Some(true) => println!( + " reachable: yes (Canvas Cloud {})", + s(&v["status"]["version"]) + ), + Some(false) => println!(" reachable: NO — {}", s(&v["status"]["error"])), + None => println!(" reachable: unknown"), + } + } else { + println!("canvas link: not linked (canvas routes answer 503 canvas_unavailable)"); } + let m = &v["migrations"]; + println!( + " migrations: {} migrated, {} failed, {} legacy canvases", + m["migrated"], m["failed"], m["legacy_canvases"] + ); } fn tls_init(cfg: &Config, force: bool, extra: &[String]) -> anyhow::Result<()> { @@ -665,6 +910,65 @@ mod tests { } } + #[test] + fn canvas_flags() { + let cli = Cli::try_parse_from([ + "copper-cloud", + "canvas", + "link", + "--url", + "http://127.0.0.1:8480", + "--key-file", + "-", + ]) + .unwrap(); + assert!(matches!( + cli.command, + Some(Command::Canvas(CanvasCommand::Link { no_sync: false, .. })) + )); + // Keys are never accepted on argv. + assert!(Cli::try_parse_from([ + "copper-cloud", + "canvas", + "link", + "--url", + "https://c.example", + "--key", + "cva_x" + ]) + .is_err()); + let cli = Cli::try_parse_from([ + "copper-cloud", + "canvas", + "migrate", + "--dry-run", + "--only", + "00000000-0000-0000-0000-000000000001", + "--only", + "00000000-0000-0000-0000-000000000002", + "--concurrency", + "4", + "--json", + ]) + .unwrap(); + let Some(Command::Canvas(CanvasCommand::Migrate(a))) = cli.command else { + panic!("parsed as {:?}", cli.command); + }; + assert!(a.dry_run && a.json && !a.replace); + assert_eq!(a.only.len(), 2); + assert_eq!(a.concurrency, 4); + for sub in ["unlink", "status", "sync-users"] { + assert!( + Cli::try_parse_from(["copper-cloud", "canvas", sub]).is_ok(), + "{sub}" + ); + } + assert!( + Cli::try_parse_from(["copper-cloud", "canvas", "migrate", "--concurrency", "0"]) + .is_err() + ); + } + #[test] fn intelligence_agent_flags() { // Agent round budget: 1..=500 on argv, cleared with `clear --agent`. diff --git a/crates/copper-cloud/src/doctor.rs b/crates/copper-cloud/src/doctor.rs index 86957c5..61201d3 100644 --- a/crates/copper-cloud/src/doctor.rs +++ b/crates/copper-cloud/src/doctor.rs @@ -203,9 +203,63 @@ async fn check_database(r: &mut Report, cfg: &Config) { ), ); } + check_canvas_link(r, cfg, &pool).await; pool.close().await; } +/// `canvas-link`: ok when linked and reachable (app key accepted), warn when not linked, +/// FAIL when linked but unreachable or refusing the key. +async fn check_canvas_link(r: &mut Report, cfg: &Config, pool: &sqlx::PgPool) { + use copper_cloud_canvas_link::{settings, upstream::Upstream}; + const CHECK: &str = "canvas-link"; + let crypto = copper_cloud_core::crypto::Crypto::from_master_key(&cfg.master_key); + let link = match settings::load(pool, &crypto).await { + Ok(Some(link)) => link, + Ok(None) => { + r.line( + Level::Warn, + CHECK, + "no Canvas Cloud linked: canvases are unavailable — copper-cloud canvas link --url … --key-file …", + ); + return; + } + Err(err) => { + r.line(Level::Warn, CHECK, format!("cannot read the link: {err}")); + return; + } + }; + if !link.enabled { + r.line( + Level::Warn, + CHECK, + format!("{} is linked but disabled", link.base()), + ); + return; + } + let upstream = match Upstream::new(&link) { + Ok(u) => u, + Err(err) => { + r.line(Level::Fail, CHECK, format!("{err:#}")); + return; + } + }; + match upstream.check(PROBE_TIMEOUT).await { + Ok(probe) => r.line( + Level::Ok, + CHECK, + format!( + "{} (Canvas Cloud {}, {} users, {} canvases, {} ms)", + link.base(), + probe.version.as_deref().unwrap_or("?"), + probe.stats.users, + probe.stats.canvases, + probe.latency.as_millis() + ), + ), + Err(err) => r.line(Level::Fail, CHECK, format!("{}: {err}", link.base())), + } +} + fn check_tls(r: &mut Report, cfg: &Config) { match cfg.tls.mode { TlsMode::Off => r.line( diff --git a/crates/copper-cloud/src/lib.rs b/crates/copper-cloud/src/lib.rs index 248a586..da9685a 100644 --- a/crates/copper-cloud/src/lib.rs +++ b/crates/copper-cloud/src/lib.rs @@ -8,19 +8,35 @@ pub mod doctor; pub mod portal; pub mod serve; -use axum::Router; +use axum::{Extension, Router}; +use copper_cloud_canvas_link::CanvasLink; use copper_cloud_core::state::SharedState; -/// The full HTTP app: core (`/healthz`, `/v1` auth/pairing/devices/sync) + canvases behind -/// the instance gate, the admin API at `/admin/api` (cookie sessions, not gated) and the -/// embedded admin portal for every other path; auth rate limiters and request -/// tracing/metrics around everything. +/// The full HTTP app: core (`/healthz`, `/v1` auth/pairing/devices/sync) + the canvas API +/// proxied to the linked Canvas Cloud behind the instance gate, the admin API at +/// `/admin/api` (cookie sessions, not gated), the `/join/{token}` share-link landing page and +/// the embedded admin portal for every other path; auth rate limiters and request +/// tracing/metrics around everything. Starts the canvas link's background tasks (event +/// bridge, user reconcile, outbox) when called inside a tokio runtime. pub fn build_app(state: SharedState) -> Router { + build_app_with_link(state).0 +} + +/// [`build_app`], also returning the canvas link handle (shutdown closes its proxied sockets). +pub fn build_app_with_link(state: SharedState) -> (Router, CanvasLink) { + let link = CanvasLink::new(state.clone()); let root = Router::new() .nest("/admin/api", admin_api::router()) - .merge(copper_cloud_canvas::landing_router()) + .merge(CanvasLink::landing_router()) .fallback(portal::serve); - copper_cloud_core::app_with(state, copper_cloud_canvas::router(), root) + let app = copper_cloud_core::app_with(state, link.router(), root) + .layer(axum::middleware::from_fn_with_state( + link.clone(), + copper_cloud_canvas_link::info_layer, + )) + .layer(Extension(link.clone())); + link.start(); + (app, link) } /// Version string: crate version plus the git revision CI stamps in at build time. diff --git a/crates/copper-cloud/src/serve.rs b/crates/copper-cloud/src/serve.rs index b4577aa..437ec22 100644 --- a/crates/copper-cloud/src/serve.rs +++ b/crates/copper-cloud/src/serve.rs @@ -5,6 +5,7 @@ use std::time::Duration; use anyhow::Context as _; use axum_server::Handle; +use copper_cloud_canvas_link::CanvasLink; use copper_cloud_core::config::Config; use copper_cloud_core::state::AppState; use copper_cloud_core::{db, observe, shutdown, tls}; @@ -75,9 +76,9 @@ pub async fn run(cfg: Config, migrate: bool) -> anyhow::Result<()> { } let handle: Handle = Handle::new(); - tokio::spawn(shutdown_on_signal(handle.clone())); + let (app, link) = crate::build_app_with_link(state); + tokio::spawn(shutdown_on_signal(handle.clone(), link)); - let app = crate::build_app(state); let result = tls::serve(app, &cfg, handle).await; shutdown::trigger(); pool.close().await; @@ -107,7 +108,7 @@ async fn connect_with_retry(cfg: &Config) -> anyhow::Result { } } -async fn shutdown_on_signal(handle: Handle) { +async fn shutdown_on_signal(handle: Handle, link: CanvasLink) { let ctrl_c = async { let _ = tokio::signal::ctrl_c().await; }; @@ -130,9 +131,9 @@ async fn shutdown_on_signal(handle: Handle) { grace_secs = GRACE.as_secs(), "shutdown signal received; draining" ); - // End SSE streams and tell canvas peers we are going away (close 1001) so the drain - // does not wait on long-lived connections. + // End SSE streams and close the proxied canvas WebSockets (1001 to Copper, a clean close + // to Canvas Cloud) so the drain does not wait on long-lived connections. shutdown::trigger(); - copper_cloud_canvas::close_all_rooms(); + link.close_all(); handle.graceful_shutdown(Some(GRACE)); } diff --git a/crates/copper-cloud/tests/admin.rs b/crates/copper-cloud/tests/admin.rs index a493a12..c1991ee 100644 --- a/crates/copper-cloud/tests/admin.rs +++ b/crates/copper-cloud/tests/admin.rs @@ -397,8 +397,8 @@ async fn people_devices_canvases_pairing_overview() { let ana = s.signup("ana@example.com", PASSWORD, &new_device()).await; let bob = s.signup("bob@example.com", PASSWORD, &new_device()).await; - // Ana: a second device, a sync doc, history, a pairing code, a shared canvas with Bob as - // member, a pending invite to someone else, an access key bound to her email. + // Ana: a second device, a sync doc, history, a pairing code, an access key bound to her + // email. let ana_d2 = new_device(); assert_eq!( s.login("ana@example.com", PASSWORD, &ana_d2).await.status(), @@ -424,46 +424,8 @@ async fn people_devices_canvases_pairing_overview() { .await .unwrap(); assert_eq!(r.status(), 200); - let canvas = json_of( - s.authed(Method::POST, "/v1/canvases", &ana) - .json(&json!({ "name": "Roadmap" })) - .send() - .await - .unwrap(), - ) - .await; - let canvas_id = canvas["id"].as_str().unwrap().to_owned(); - let inv = json_of( - s.authed( - Method::POST, - &format!("/v1/canvases/{canvas_id}/invites"), - &ana, - ) - .json(&json!({ "email": "bob@example.com" })) - .send() - .await - .unwrap(), - ) - .await; - let r = s - .authed( - Method::POST, - &format!("/v1/invites/{}/accept", inv["id"].as_str().unwrap()), - &bob, - ) - .send() - .await - .unwrap(); - assert_eq!(r.status(), 200); - s.authed( - Method::POST, - &format!("/v1/canvases/{canvas_id}/invites"), - &ana, - ) - .json(&json!({ "email": "carol@example.com" })) - .send() - .await - .unwrap(); + // (Canvases live on a linked Canvas Cloud now — see tests/canvas_link.rs; this instance + // is not linked, so every canvas number is null and the canvas routes answer 503.) s.mint_key("Ana", Some("ana@example.com"), None).await; let (_, team_key) = s.mint_key("Team", None, None).await; @@ -478,9 +440,9 @@ async fn people_devices_canvases_pairing_overview() { assert_eq!(ov["counts"]["users"], 2); assert_eq!(ov["counts"]["devices"], 3); assert_eq!(ov["counts"]["access_keys"], 2); - assert!(ov["counts"]["canvases"].as_i64().unwrap() >= 1); - assert!(ov["counts"]["live_rooms"].is_number()); - assert!(ov["counts"]["live_peers"].is_number()); + assert!(ov["counts"]["canvases"].is_null(), "no Canvas Cloud linked"); + assert!(ov["counts"]["live_rooms"].is_null()); + assert!(ov["counts"]["live_peers"].is_null()); assert_eq!(ov["access_mode"], "open"); assert_eq!(ov["public_url"], "localhost:8443"); assert_eq!(ov["tls"]["mode"], "off"); @@ -502,7 +464,7 @@ async fn people_devices_canvases_pairing_overview() { ); let ana_row = &users["items"][1]; assert_eq!(ana_row["device_count"], 2); - assert!(ana_row["canvas_count"].as_i64().unwrap() >= 1); + assert!(ana_row["canvas_count"].is_null(), "no Canvas Cloud linked"); assert!(ana_row["last_seen_at"].is_string()); assert_eq!(ana_row["disabled"], false); let q = json_of( @@ -625,7 +587,7 @@ async fn people_devices_canvases_pairing_overview() { .unwrap(); assert_eq!(r.status(), 404); - // Canvases: listed without content; pairing codes listed. + // Canvases: unlinked → an empty list saying so; pairing codes listed. let cs = json_of( s.admin(Method::GET, "canvases?kind=shared", Some(&c)) .send() @@ -633,11 +595,9 @@ async fn people_devices_canvases_pairing_overview() { .unwrap(), ) .await; - assert_eq!(cs["total"], 1); - assert_eq!(cs["items"][0]["name"], "Roadmap"); - assert_eq!(cs["items"][0]["owner_email"], "ana@example.com"); - assert_eq!(cs["items"][0]["member_count"], 2); - assert!(cs["items"][0].get("state").is_none()); + assert_eq!(cs["total"], 0); + assert_eq!(cs["linked"], false); + assert_eq!(cs["items"], json!([])); let pc = json_of( s.admin(Method::GET, "pairing-codes", Some(&c)) .send() @@ -672,17 +632,6 @@ async fn people_devices_canvases_pairing_overview() { .unwrap(); assert_eq!(n, 0, "{table}"); } - let n: i64 = sqlx::query_scalar("SELECT count(*) FROM canvas_members WHERE canvas_id = $1") - .bind(canvas_id.parse::().unwrap()) - .fetch_one(&s.state.db) - .await - .unwrap(); - assert_eq!(n, 0, "Bob's membership of Ana's canvas"); - let n: i64 = sqlx::query_scalar("SELECT count(*) FROM canvas_invites") - .fetch_one(&s.state.db) - .await - .unwrap(); - assert_eq!(n, 0, "invites of deleted canvases"); let keys: Vec = sqlx::query_scalar("SELECT label FROM access_keys") .fetch_all(&s.state.db) .await @@ -701,7 +650,8 @@ async fn people_devices_canvases_pairing_overview() { .send() .await .unwrap(); - assert_eq!(r.status(), 200); + assert_eq!(r.status(), 503); + assert_eq!(json_of(r).await["error"], "canvas_unavailable"); let r = s .admin(Method::DELETE, &format!("users/{}", ana.user_id), Some(&c)) .send() @@ -709,27 +659,18 @@ async fn people_devices_canvases_pairing_overview() { .unwrap(); assert_eq!(r.status(), 404); - // Delete Bob's Personal canvas: allowed for admins; it comes back empty on next use. - let personal = json_of( - s.admin(Method::GET, "canvases?kind=personal", Some(&c)) - .send() - .await - .unwrap(), - ) - .await; - let pid = personal["items"][0]["id"].as_str().unwrap().to_owned(); + // Deleting a canvas needs a linked Canvas Cloud. let r = s - .admin(Method::DELETE, &format!("canvases/{pid}"), Some(&c)) - .send() - .await - .unwrap(); - assert_eq!(r.status(), 200); - let r = s - .admin(Method::DELETE, &format!("canvases/{pid}"), Some(&c)) + .admin( + Method::DELETE, + &format!("canvases/{}", uuid::Uuid::now_v7()), + Some(&c), + ) .send() .await .unwrap(); - assert_eq!(r.status(), 404); + assert_eq!(r.status(), 503); + assert_eq!(json_of(r).await["error"], "canvas_unavailable"); // Pairing code revoke. let minted = json_of( @@ -767,7 +708,6 @@ async fn people_devices_canvases_pairing_overview() { "user.reset_password", "device.delete", "user.delete", - "canvas.delete", "pairing_code.revoke", ] { assert!(actions.iter().any(|x| x == a), "{a} audited: {actions:?}"); diff --git a/crates/copper-cloud/tests/api.rs b/crates/copper-cloud/tests/api.rs index e15160a..e62281d 100644 --- a/crates/copper-cloud/tests/api.rs +++ b/crates/copper-cloud/tests/api.rs @@ -72,7 +72,7 @@ async fn instance_gate() { assert_eq!(r.status(), 401); assert_eq!(r.json::().await.unwrap()["error"], "session"); - // The canvas router is mounted under the same gate + auth. + // The canvas proxy is mounted under the same gate + auth (unlinked here: 503). let r = s.http.get(s.url("/v1/canvases")).send().await.unwrap(); assert_eq!(r.status(), 401); assert_eq!(r.json::().await.unwrap()["error"], "instance_key"); @@ -84,7 +84,11 @@ async fn instance_gate() { .send() .await .unwrap(); - assert_eq!(r.status(), 200); + assert_eq!(r.status(), 503); + assert_eq!( + r.json::().await.unwrap()["error"], + "canvas_unavailable" + ); } #[tokio::test] diff --git a/crates/copper-cloud/tests/canvas_link.rs b/crates/copper-cloud/tests/canvas_link.rs new file mode 100644 index 0000000..35f609c --- /dev/null +++ b/crates/copper-cloud/tests/canvas_link.rs @@ -0,0 +1,1064 @@ +//! The canvas link end to end: the real copper-cloud app (native Postgres) in front of the +//! in-process mock Canvas Cloud — admin API, proxy rules, the WebSocket pipe, the event +//! bridge into `/v1/sync/events`, provisioning (JIT, batch, reconcile, outbox) and the +//! "never a 401 to Copper" rule. + +#![allow(clippy::too_many_lines, clippy::many_single_char_names)] + +mod common; +#[path = "../../canvas-link/tests/support/mock.rs"] +mod mock; + +use std::time::Duration; + +use common::*; +use futures::{SinkExt as _, StreamExt as _}; +use reqwest::Method; +use serde_json::{json, Value}; +use tokio_tungstenite::tungstenite::client::IntoClientRequest as _; +use tokio_tungstenite::tungstenite::protocol::frame::coding::CloseCode; +use tokio_tungstenite::tungstenite::protocol::CloseFrame; +use tokio_tungstenite::tungstenite::{self, Message}; + +async fn json_of(r: reqwest::Response) -> Value { + r.json().await.unwrap() +} + +/// Poll `f` (every 25 ms, up to 10 s). +async fn wait_for(what: &str, mut f: impl FnMut() -> bool) { + let deadline = tokio::time::Instant::now() + Duration::from_secs(10); + while !f() { + assert!( + tokio::time::Instant::now() < deadline, + "timed out waiting for {what}" + ); + tokio::time::sleep(Duration::from_millis(25)).await; + } +} + +async fn put_link(s: &TestServer, c: &str, body: &Value) -> reqwest::Response { + s.admin(Method::PUT, "canvas-link", Some(c)) + .json(body) + .send() + .await + .unwrap() +} + +/// Link `m` through the admin API and wait until the background provisioning settled. +async fn link(s: &TestServer, c: &str, m: &mock::Mock, users: &[&Session]) -> Value { + let r = put_link(s, c, &json!({ "url": m.base, "app_key": mock::APP_KEY })).await; + assert_eq!(r.status(), 200); + let view = json_of(r).await; + for u in users { + wait_for("users provisioned", || m.st.has_user(&u.user_id)).await; + } + // Let the link-time sync and the first reconcile pass finish. + tokio::time::sleep(Duration::from_millis(300)).await; + view +} + +/// Admin + mock + two users, linked. +async fn setup() -> (TestServer, String, mock::Mock, Session, Session) { + let s = start().await; + s.create_admin().await; + let c = s.admin_login().await; + let m = mock::start().await; + let ana = s.signup("ana@example.com", PASSWORD, &new_device()).await; + let bob = s.signup("bob@example.com", PASSWORD, &new_device()).await; + link(&s, &c, &m, &[&ana, &bob]).await; + (s, c, m, ana, bob) +} + +fn assert_view_shape(v: &Value) { + for key in [ + "linked", + "url", + "enabled", + "tls_fingerprint", + "linked_at", + "updated_at", + "updated_by", + "status", + "bridge", + "migrations", + ] { + assert!(v.get(key).is_some(), "missing {key}: {v}"); + } + for key in ["reachable", "version", "error", "checked_at"] { + assert!(v["status"].get(key).is_some(), "status.{key}: {v}"); + } + for key in ["connected", "last_event_at", "last_error"] { + assert!(v["bridge"].get(key).is_some(), "bridge.{key}: {v}"); + } + for key in ["migrated", "failed", "legacy_canvases"] { + assert!(v["migrations"][key].is_i64(), "migrations.{key}: {v}"); + } +} + +#[tokio::test] +async fn admin_link_lifecycle() { + let s = start().await; + s.create_admin().await; + let c = s.admin_login().await; + let m = mock::start().await; + let ana = s.signup("ana@example.com", PASSWORD, &new_device()).await; + + // Unlinked: the view says so, canvas routes answer 503, /v1/info says linked: false. + let v = json_of( + s.admin(Method::GET, "canvas-link", Some(&c)) + .send() + .await + .unwrap(), + ) + .await; + assert_view_shape(&v); + assert_eq!(v["linked"], false); + assert_eq!(v["url"], Value::Null); + assert_eq!(v["status"]["reachable"], Value::Null); + assert_eq!(v["bridge"]["connected"], false); + assert_eq!( + v["migrations"], + json!({ "migrated": 0, "failed": 0, "legacy_canvases": 0 }) + ); + let r = s + .authed(Method::GET, "/v1/canvases", &ana) + .send() + .await + .unwrap(); + assert_eq!(r.status(), 503); + assert_eq!(json_of(r).await["error"], "canvas_unavailable"); + let info = json_of(s.req(Method::GET, "/v1/info").send().await.unwrap()).await; + assert_eq!(info["canvas"], json!({ "linked": false })); + assert!(info["features"] + .as_array() + .unwrap() + .contains(&json!("canvas_link"))); + + // Validation: bad URLs, a wrong key, an unreachable server → 400 with a clear message. + for (body, needle) in [ + ( + json!({ "url": "http://10.1.2.3:8480", "app_key": mock::APP_KEY }), + "localhost", + ), + ( + json!({ "url": "ftp://x.example", "app_key": mock::APP_KEY }), + "https://", + ), + (json!({ "url": m.base, "app_key": "x" }), "app_key"), + ( + json!({ "url": m.base, "app_key": mock::APP_KEY, "tls_fingerprint": "abc" }), + "tls_fingerprint", + ), + ( + json!({ "url": m.base, "app_key": "cva_definitely_wrong_key" }), + "rejected the app key", + ), + ( + json!({ "url": "http://127.0.0.1:9", "app_key": mock::APP_KEY }), + "could not reach", + ), + ] { + let r = put_link(&s, &c, &body).await; + assert_eq!(r.status(), 400, "{body}"); + let e = json_of(r).await; + assert_eq!(e["error"], "bad_request"); + assert!( + e["message"].as_str().unwrap().contains(needle), + "{body}: {e}" + ); + } + // CSRF still applies. + let r = s + .http + .put(s.url("/admin/api/canvas-link")) + .header("Cookie", &c) + .json(&json!({ "url": m.base, "app_key": mock::APP_KEY })) + .send() + .await + .unwrap(); + assert_eq!(r.status(), 403); + + // Link: tested, stored sealed, users provisioned in bulk. + let v = link(&s, &c, &m, &[&ana]).await; + assert_view_shape(&v); + assert_eq!(v["linked"], true); + assert_eq!(v["url"], m.base.as_str()); + assert_eq!(v["enabled"], true); + assert_eq!(v["updated_by"], ADMIN_EMAIL); + assert_eq!(v["status"]["reachable"], true); + assert_eq!(v["status"]["version"], "0.0.0-mock"); + assert!(v["linked_at"].is_string()); + assert!(m.st.batch_calls.load(std::sync::atomic::Ordering::SeqCst) >= 1); + let u = m.st.user(&ana.user_id).unwrap(); + assert_eq!(u.email, "ana@example.com"); + assert_eq!( + u.user_id, ana.user_id, + "Canvas Cloud user id == Copper user id" + ); + assert_eq!(u.display_name.as_deref(), Some("Test User")); + let sealed: Vec = sqlx::query_scalar("SELECT app_key_sealed FROM canvas_link") + .fetch_one(&s.state.db) + .await + .unwrap(); + assert!(!sealed + .windows(mock::APP_KEY.len()) + .any(|w| w == mock::APP_KEY.as_bytes())); + let info = json_of(s.req(Method::GET, "/v1/info").send().await.unwrap()).await; + assert_eq!(info["canvas"], json!({ "linked": true })); + // The key never comes back. + let text = s + .admin(Method::GET, "canvas-link", Some(&c)) + .send() + .await + .unwrap() + .text() + .await + .unwrap(); + assert!(!text.contains(mock::APP_KEY)); + + // The event bridge connects. + wait_for("bridge connected", || m.st.event_clients() >= 1).await; + let mut connected = false; + for _ in 0..100 { + let v = json_of( + s.admin(Method::GET, "canvas-link", Some(&c)) + .send() + .await + .unwrap(), + ) + .await; + if v["bridge"]["connected"] == true { + connected = true; + break; + } + tokio::time::sleep(Duration::from_millis(50)).await; + } + assert!(connected, "bridge.connected"); + + // Test. + let t = json_of( + s.admin(Method::POST, "canvas-link/test", Some(&c)) + .send() + .await + .unwrap(), + ) + .await; + assert_eq!(t["ok"], true); + assert_eq!(t["version"], "0.0.0-mock"); + assert!(t["latency_ms"].is_u64()); + assert_eq!(t["error"], Value::Null); + m.st.set_app_key("cva_rotated_elsewhere_000"); + let t = json_of( + s.admin(Method::POST, "canvas-link/test", Some(&c)) + .send() + .await + .unwrap(), + ) + .await; + assert_eq!(t["ok"], false); + assert_eq!(t["latency_ms"], Value::Null); + assert!(t["error"].as_str().unwrap().contains("401"), "{t}"); + m.st.set_app_key(mock::APP_KEY); + + // Unlink (idempotent), audited once. + for _ in 0..2 { + let r = s + .admin(Method::DELETE, "canvas-link", Some(&c)) + .send() + .await + .unwrap(); + assert_eq!(r.status(), 200); + assert_eq!(json_of(r).await, json!({ "ok": true })); + } + let v = json_of( + s.admin(Method::GET, "canvas-link", Some(&c)) + .send() + .await + .unwrap(), + ) + .await; + assert_eq!(v["linked"], false); + let r = s + .authed(Method::GET, "/v1/canvases", &ana) + .send() + .await + .unwrap(); + assert_eq!(r.status(), 503); + let t = json_of( + s.admin(Method::POST, "canvas-link/test", Some(&c)) + .send() + .await + .unwrap(), + ) + .await; + assert_eq!(t["ok"], false); + + let actions: Vec = sqlx::query_scalar( + "SELECT action FROM admin_audit WHERE action LIKE 'canvas_link.%' ORDER BY id", + ) + .fetch_all(&s.state.db) + .await + .unwrap(); + assert_eq!( + actions, + [ + "canvas_link.set", + "canvas_link.test", + "canvas_link.test", + "canvas_link.remove" + ] + ); + let detail: Value = + sqlx::query_scalar("SELECT detail FROM admin_audit WHERE action = 'canvas_link.set'") + .fetch_one(&s.state.db) + .await + .unwrap(); + assert_eq!(detail["url"], m.base.as_str()); + assert!(!detail.to_string().contains(mock::APP_KEY)); +} + +#[tokio::test] +async fn proxy_forwards_requests_and_responses() { + let (s, _c, _m, ana, _bob) = setup().await; + + // Header and query rules. + let r = s + .authed( + Method::GET, + "/v1/canvases/c1/echo?token=SECRET&full=true&token=again", + &ana, + ) + .header("Cookie", "copper=1") + .header("If-None-Match", "\"v1\"") + .header("Accept", "application/json") + .header("X-Custom", "nope") + .header("X-Canvas-User", "spoofed") + .send() + .await + .unwrap(); + assert_eq!(r.status(), 200); + assert_eq!(r.headers()["cache-control"], "private, max-age=5"); + assert_eq!(r.headers()["etag"], "\"mock-etag\""); + assert!(r.headers()["content-type"] + .to_str() + .unwrap() + .starts_with("application/json")); + assert!(r.headers().get("x-mock-internal").is_none()); + assert!(r.headers().get("set-cookie").is_none()); + let e = json_of(r).await; + assert_eq!(e["path"], "/v1/canvases/c1/echo"); + assert_eq!(e["query"], "full=true"); + assert_eq!(e["user"], ana.user_id.as_str()); + let h = &e["headers"]; + assert_eq!(h["authorization"], format!("Bearer {}", mock::APP_KEY)); + assert_eq!(h["x-canvas-user"], ana.user_id.as_str()); + assert_eq!(h["if-none-match"], "\"v1\""); + assert_eq!(h["accept"], "application/json"); + assert_eq!(h["x-forwarded-for"], "127.0.0.1"); + for gone in ["cookie", "x-copper-instance", "x-custom"] { + assert!(h.get(gone).is_none(), "{gone} forwarded: {h}"); + } + + // Method + body + content type; the other mounted prefixes. + let r = s + .authed(Method::PATCH, "/v1/invites/i1/accept", &ana) + .json(&json!({ "x": 1 })) + .send() + .await + .unwrap(); + let e = json_of(r).await; + assert_eq!(e["method"], "PATCH"); + assert_eq!(e["path"], "/v1/invites/i1/accept"); + assert_eq!(e["body"], "{\"x\":1}"); + assert_eq!(e["headers"]["content-type"], "application/json"); + for path in [ + "/v1/people", + "/v1/invites", + "/v1/mentions", + "/v1/mentions/read", + "/v1/canvas-links/tok", + ] { + let r = s.authed(Method::GET, path, &ana).send().await.unwrap(); + assert_eq!(r.status(), 200, "{path}"); + assert_eq!(json_of(r).await["path"], path); + } + // Real list/create through the proxy. + let created = json_of( + s.authed(Method::POST, "/v1/canvases", &ana) + .json(&json!({ "name": "Roadmap" })) + .send() + .await + .unwrap(), + ) + .await; + assert_eq!(created["name"], "Roadmap"); + let list = json_of( + s.authed(Method::GET, "/v1/canvases", &ana) + .send() + .await + .unwrap(), + ) + .await; + assert_eq!(list["canvases"][0]["id"], created["id"]); + + // Client errors pass through verbatim (clients rely on the exact codes)… + for code in [403_u16, 404, 409, 422, 429, 500] { + let r = s + .authed(Method::GET, &format!("/v1/canvases/c1/status/{code}"), &ana) + .send() + .await + .unwrap(); + assert_eq!(r.status().as_u16(), code); + if code == 429 { + assert_eq!(r.headers()["retry-after"], "7"); + } + assert_eq!(json_of(r).await["error"], format!("mock_{code}")); + } + // … but never a 401 (Copper would sign the user out). + let r = s + .authed(Method::GET, "/v1/canvases/c1/status/401", &ana) + .send() + .await + .unwrap(); + assert_eq!(r.status(), 502); + let e = json_of(r).await; + assert_eq!(e["error"], "canvas_upstream"); + assert!(e["message"] + .as_str() + .unwrap() + .contains("Canvas Cloud rejected this Copper Cloud's app key/user")); + + // Copper auth still applies first. + let r = s.req(Method::GET, "/v1/canvases").send().await.unwrap(); + assert_eq!(r.status(), 401); + assert_eq!(json_of(r).await["error"], "session"); + + // Body limits: 64 KiB in general, 4 MiB for ops. + let r = s + .authed(Method::POST, "/v1/canvases/c1/ops", &ana) + .header("content-type", "application/json") + .body(vec![b' '; 3 << 20]) + .send() + .await + .unwrap(); + assert_eq!(r.status(), 200); + assert_eq!(json_of(r).await["received"], 3 << 20); + let r = s + .authed(Method::POST, "/v1/canvases/c1/ops", &ana) + .body(vec![b' '; 5 << 20]) + .send() + .await + .unwrap(); + assert_eq!(r.status(), 413); + let r = s + .authed(Method::POST, "/v1/canvases/c1/chat", &ana) + .body(vec![b' '; 100 << 10]) + .send() + .await + .unwrap(); + assert_eq!(r.status(), 413); +} + +#[tokio::test] +async fn upstream_401_is_never_forwarded() { + let (s, _c, m, ana, _bob) = setup().await; + let puts = || m.st.user_puts.load(std::sync::atomic::Ordering::SeqCst); + + // unknown_user → provision just in time, retry once, succeed. + m.st.users.lock().unwrap().remove(&ana.user_id); + let before = puts(); + let r = s + .authed(Method::GET, "/v1/canvases", &ana) + .send() + .await + .unwrap(); + assert_eq!(r.status(), 200); + assert_eq!(puts(), before + 1); + assert_eq!(m.st.user(&ana.user_id).unwrap().email, "ana@example.com"); + + // A second unknown_user after provisioning → 502, one provisioning attempt only. + m.st.users.lock().unwrap().remove(&ana.user_id); + m.st.sticky_unknown + .store(true, std::sync::atomic::Ordering::SeqCst); + let before = puts(); + let r = s + .authed(Method::GET, "/v1/canvases", &ana) + .send() + .await + .unwrap(); + assert_eq!(r.status(), 502); + assert_eq!(json_of(r).await["error"], "canvas_upstream"); + assert_eq!(puts(), before + 1); + m.st.sticky_unknown + .store(false, std::sync::atomic::Ordering::SeqCst); + + // Provisioning itself fails → 502. + m.st.fail_provision + .store(true, std::sync::atomic::Ordering::SeqCst); + let r = s + .authed(Method::GET, "/v1/canvases", &ana) + .send() + .await + .unwrap(); + assert_eq!(r.status(), 502); + assert_eq!(json_of(r).await["error"], "canvas_upstream"); + m.st.fail_provision + .store(false, std::sync::atomic::Ordering::SeqCst); + + // The app key is refused (revoked/rotated on Canvas Cloud) → 502, not 401. + m.st.add_user(&ana.user_id, "ana@example.com"); + m.st.set_app_key("cva_rotated_elsewhere_000"); + let r = s + .authed(Method::GET, "/v1/canvases", &ana) + .send() + .await + .unwrap(); + assert_eq!(r.status(), 502); + let e = json_of(r).await; + assert_eq!(e["error"], "canvas_upstream"); + assert!( + e["message"] + .as_str() + .unwrap() + .starts_with("Canvas Cloud rejected this Copper Cloud's app key/user ("), + "{e}" + ); + + // The same for a refused WebSocket handshake. + let err = ws_connect(&s, &ana, "c1").await.unwrap_err(); + let (status, body) = http_error(err); + assert_eq!(status, 502); + assert_eq!(body["error"], "canvas_upstream"); + m.st.set_app_key(mock::APP_KEY); + // …and an unknown user on the WebSocket is provisioned and let through. + m.st.users.lock().unwrap().remove(&ana.user_id); + let (mut ws, _) = ws_connect(&s, &ana, "c1").await.unwrap(); + ws.send(Message::Text("hi".into())).await.unwrap(); + assert_eq!(next_data(&mut ws).await, Message::Text("hi".into())); + m.st.users.lock().unwrap().remove(&ana.user_id); + m.st.sticky_unknown + .store(true, std::sync::atomic::Ordering::SeqCst); + let (status, body) = http_error(ws_connect(&s, &ana, "c2").await.unwrap_err()); + assert_eq!(status, 502); + assert_eq!(body["error"], "canvas_upstream"); + m.st.sticky_unknown + .store(false, std::sync::atomic::Ordering::SeqCst); + + // Canvas Cloud unreachable (the stored URL changed, e.g. by the CLI) → 502. + let dead = { + let l = std::net::TcpListener::bind("127.0.0.1:0").unwrap(); + format!("http://{}", l.local_addr().unwrap()) + }; + sqlx::query("UPDATE canvas_link SET url = $1, updated_at = now()") + .bind(&dead) + .execute(&s.state.db) + .await + .unwrap(); + s.link.reload().await.unwrap(); + let r = s + .authed(Method::GET, "/v1/canvases", &ana) + .send() + .await + .unwrap(); + assert_eq!(r.status(), 502); + let e = json_of(r).await; + assert_eq!(e["error"], "canvas_upstream"); + assert!( + e["message"].as_str().unwrap().contains("unreachable"), + "{e}" + ); + let (status, body) = http_error(ws_connect(&s, &ana, "c1").await.unwrap_err()); + assert_eq!(status, 502); + assert_eq!(body["error"], "canvas_upstream"); +} + +type Ws = + tokio_tungstenite::WebSocketStream>; + +/// Open `/v1/canvases/{id}/ws` the way Copper (Mac) does: instance key header, bearer +/// header and `?token=`. +async fn ws_connect( + s: &TestServer, + who: &Session, + id: &str, +) -> Result<(Ws, tungstenite::handshake::client::Response), tungstenite::Error> { + let mut req = format!("ws://{}/v1/canvases/{id}/ws?token={}", s.addr, who.token) + .into_client_request() + .unwrap(); + req.headers_mut() + .insert("x-copper-instance", s.key().parse().unwrap()); + req.headers_mut().insert( + "authorization", + format!("Bearer {}", who.token).parse().unwrap(), + ); + let config = tungstenite::protocol::WebSocketConfig::default() + .max_message_size(Some(64 << 20)) + .max_frame_size(Some(16 << 20)); + tokio_tungstenite::connect_async_with_config(req, Some(config), true).await +} + +fn http_error(err: tungstenite::Error) -> (u16, Value) { + match err { + tungstenite::Error::Http(resp) => { + let status = resp.status().as_u16(); + let body = resp.into_body().unwrap_or_default(); + (status, serde_json::from_slice(&body).unwrap_or(Value::Null)) + } + other => panic!("expected an HTTP refusal, got {other:?}"), + } +} + +/// Next text/binary/close message (pings skipped), 10 s budget. +async fn next_data(ws: &mut Ws) -> Message { + loop { + let m = tokio::time::timeout(Duration::from_secs(10), ws.next()) + .await + .expect("timed out waiting for a WebSocket message") + .expect("WebSocket ended") + .unwrap(); + if !matches!(m, Message::Ping(_) | Message::Pong(_)) { + return m; + } + } +} + +#[tokio::test] +async fn websocket_pipe() { + let (s, _c, m, ana, _bob) = setup().await; + + let (mut ws, _) = ws_connect(&s, &ana, "c1").await.unwrap(); + // The upstream handshake carried the app key + user, not Copper's token. + let seen = m.st.requests_for("/v1/canvases/c1/ws"); + let last = seen.last().unwrap(); + assert_eq!(last.query, None, "token= stripped"); + assert_eq!( + last.headers["authorization"], + format!("Bearer {}", mock::APP_KEY) + ); + assert_eq!(last.headers["x-canvas-user"], ana.user_id); + assert!(!last.headers.contains_key("x-copper-instance")); + + // Frames are piped byte-exact, both kinds, including large ones. + let bin: Vec = (0..=255_u8).cycle().take(1000).collect(); + ws.send(Message::Binary(bin.clone().into())).await.unwrap(); + assert_eq!(next_data(&mut ws).await, Message::Binary(bin.into())); + ws.send(Message::Text("hello ✓".into())).await.unwrap(); + assert_eq!(next_data(&mut ws).await, Message::Text("hello ✓".into())); + let big: Vec = (0..(12_u32 << 20)).map(|i| (i % 251) as u8).collect(); + ws.send(Message::Binary(big.clone().into())).await.unwrap(); + match next_data(&mut ws).await { + Message::Binary(b) => assert!(b.as_ref() == big.as_slice(), "12 MiB frame intact"), + other => panic!("{other:?}"), + } + assert_eq!(m.st.live_peers.load(std::sync::atomic::Ordering::SeqCst), 1); + + // Canvas Cloud closes with 4404 → the client sees 4404 with the reason. + ws.send(Message::Text("close 4404 canvas deleted".into())) + .await + .unwrap(); + match next_data(&mut ws).await { + Message::Close(Some(f)) => { + assert_eq!(u16::from(f.code), 4404); + assert_eq!(f.reason.as_str(), "canvas deleted"); + } + other => panic!("{other:?}"), + } + drop(ws); + + // The client closes with 4403 → Canvas Cloud sees 4403 with the reason. + let (mut ws, _) = ws_connect(&s, &ana, "c1").await.unwrap(); + ws.close(Some(CloseFrame { + code: CloseCode::from(4403), + reason: "bye".into(), + })) + .await + .unwrap(); + wait_for("client close forwarded", || { + m.st.ws_closes + .lock() + .unwrap() + .contains(&(4403, "bye".to_owned())) + }) + .await; + + // Handshake refusals are returned as-is, without upgrading. + let (status, body) = http_error(ws_connect(&s, &ana, "forbidden").await.unwrap_err()); + assert_eq!(status, 403); + assert_eq!(body["error"], "forbidden"); + let (status, body) = http_error(ws_connect(&s, &ana, "missing").await.unwrap_err()); + assert_eq!(status, 404); + assert_eq!(body["error"], "not_found"); + + // copper-cloud shutting down → 1001 to Copper, a close to Canvas Cloud. + let (mut ws, _) = ws_connect(&s, &ana, "c1").await.unwrap(); + ws.send(Message::Text("ping".into())).await.unwrap(); + assert_eq!(next_data(&mut ws).await, Message::Text("ping".into())); + s.link.close_all(); + match next_data(&mut ws).await { + Message::Close(Some(f)) => { + assert_eq!(u16::from(f.code), 1001); + assert_eq!(f.reason.as_str(), "server shutting down"); + } + other => panic!("{other:?}"), + } + wait_for("upstream told", || { + m.st.ws_closes + .lock() + .unwrap() + .iter() + .any(|(code, _)| *code == 1001) + }) + .await; +} + +/// Read SSE frames until one with `event: ` arrives; returns its data JSON. +async fn next_event( + stream: &mut (impl futures::Stream> + Unpin), + buf: &mut String, + name: &str, +) -> Value { + let deadline = tokio::time::Instant::now() + Duration::from_secs(10); + loop { + while let Some(end) = buf.find("\n\n") { + let frame: String = buf.drain(..end + 2).collect(); + let mut event = None; + let mut data = String::new(); + for line in frame.lines() { + if let Some(v) = line.strip_prefix("event:") { + event = Some(v.trim().to_owned()); + } else if let Some(v) = line.strip_prefix("data:") { + data.push_str(v.trim()); + } + } + if event.as_deref() == Some(name) { + return serde_json::from_str(&data).unwrap(); + } + } + let chunk = tokio::time::timeout_at(deadline, stream.next()) + .await + .unwrap_or_else(|_| panic!("timed out waiting for SSE event {name}")) + .expect("stream ended") + .unwrap(); + buf.push_str(std::str::from_utf8(&chunk).unwrap()); + } +} + +#[tokio::test] +async fn event_bridge_feeds_sync_events() { + let (s, c, m, ana, bob) = setup().await; + wait_for("bridge connected", || m.st.event_clients() >= 1).await; + + let resp = s + .authed(Method::GET, "/v1/sync/events", &ana) + .send() + .await + .unwrap(); + assert_eq!(resp.status(), 200); + let mut stream = resp.bytes_stream(); + let mut buf = String::new(); + + let canvas_id = uuid::Uuid::now_v7().to_string(); + // Bob's event does not reach Ana; hers does, in copper-cloud's own `canvas` shape. + m.st.publish_canvas(&bob.user_id, &uuid::Uuid::now_v7().to_string(), "renamed"); + let id = + m.st.publish_canvas(&ana.user_id, &canvas_id, "member_added"); + let ev = next_event(&mut stream, &mut buf, "canvas").await; + assert_eq!(ev["type"], "canvas"); + assert_eq!(ev["canvas_id"], canvas_id.as_str()); + assert_eq!(ev["kind"], "member_added"); + + let mut last_event_at = Value::Null; + for _ in 0..100 { + let v = json_of( + s.admin(Method::GET, "canvas-link", Some(&c)) + .send() + .await + .unwrap(), + ) + .await; + if v["bridge"]["last_event_at"].is_string() { + last_event_at = v["bridge"]["last_event_at"].clone(); + break; + } + tokio::time::sleep(Duration::from_millis(50)).await; + } + assert!(last_event_at.is_string()); + + // A changed link (re-saved settings) restarts the stream, resuming after the last id. + sqlx::query("UPDATE canvas_link SET updated_at = now()") + .execute(&s.state.db) + .await + .unwrap(); + s.link.reload().await.unwrap(); + let want = Some(id.to_string()); + wait_for("resumed with Last-Event-ID", || { + m.st.last_event_ids.lock().unwrap().contains(&want) + }) + .await; + // (The new stream subscribes right after recording the id.) + tokio::time::sleep(Duration::from_millis(300)).await; + let canvas_id2 = uuid::Uuid::now_v7().to_string(); + m.st.publish_canvas(&ana.user_id, &canvas_id2, "deleted"); + let ev = next_event(&mut stream, &mut buf, "canvas").await; + assert_eq!(ev["canvas_id"], canvas_id2.as_str()); + assert_eq!(ev["kind"], "deleted"); +} + +#[tokio::test] +async fn admin_canvases_overview_people_and_user_hooks() { + let (s, c, m, ana, bob) = setup().await; + let roadmap = m.st.add_canvas(&ana.user_id, "Roadmap", "shared"); + m.st.canvases.lock().unwrap()[0] + .members + .push(bob.user_id.clone()); + m.st.add_canvas(&ana.user_id, "Personal", "personal"); + + // Overview numbers come from Canvas Cloud. + let ov = json_of( + s.admin(Method::GET, "overview", Some(&c)) + .send() + .await + .unwrap(), + ) + .await; + assert_eq!(ov["counts"]["canvases"], 2); + assert_eq!(ov["counts"]["live_rooms"], 0); + assert_eq!(ov["counts"]["live_peers"], 0); + assert_eq!(ov["counts"]["users"], 2); + + // The canvas list keeps its shape (+ linked). + let cs = json_of( + s.admin(Method::GET, "canvases?kind=shared&limit=10", Some(&c)) + .send() + .await + .unwrap(), + ) + .await; + assert_eq!(cs["linked"], true); + assert_eq!(cs["total"], 1); + assert_eq!(cs["limit"], 10); + assert_eq!(cs["offset"], 0); + let row = &cs["items"][0]; + assert_eq!(row["id"], roadmap.as_str()); + assert_eq!(row["name"], "Roadmap"); + assert_eq!(row["kind"], "shared"); + assert_eq!(row["owner_id"], ana.user_id.as_str()); + assert_eq!(row["owner_email"], "ana@example.com"); + assert_eq!(row["member_count"], 2); + assert_eq!(row["share_link_count"], 0); + assert!(row["created_at"].is_string() && row["updated_at"].is_string()); + let r = s + .admin(Method::GET, "canvases?kind=bogus", Some(&c)) + .send() + .await + .unwrap(); + assert_eq!(r.status(), 400); + + // People: canvas_count from Canvas Cloud. + let users = json_of( + s.admin(Method::GET, "users", Some(&c)) + .send() + .await + .unwrap(), + ) + .await; + let count = |email: &str| { + users["items"] + .as_array() + .unwrap() + .iter() + .find(|u| u["email"] == email) + .unwrap()["canvas_count"] + .clone() + }; + assert_eq!(count("ana@example.com"), 2); + assert_eq!(count("bob@example.com"), 1); + + // Delete a canvas (audited); again → 404. + let r = s + .admin(Method::DELETE, &format!("canvases/{roadmap}"), Some(&c)) + .send() + .await + .unwrap(); + assert_eq!(r.status(), 200); + assert_eq!(m.st.canvas_count(), 1); + let r = s + .admin(Method::DELETE, &format!("canvases/{roadmap}"), Some(&c)) + .send() + .await + .unwrap(); + assert_eq!(r.status(), 404); + let n: i64 = sqlx::query_scalar( + "SELECT count(*) FROM admin_audit WHERE action = 'canvas.delete' AND target = $1", + ) + .bind(&roadmap) + .fetch_one(&s.state.db) + .await + .unwrap(); + assert_eq!(n, 1); + + // Canvas Cloud refusing us: the list says 502 canvas_upstream, the overview goes null. + m.st.set_app_key("cva_rotated_elsewhere_000"); + let r = s + .admin(Method::GET, "canvases", Some(&c)) + .send() + .await + .unwrap(); + assert_eq!(r.status(), 502); + assert_eq!(json_of(r).await["error"], "canvas_upstream"); + let r = s + .admin(Method::DELETE, &format!("canvases/{roadmap}"), Some(&c)) + .send() + .await + .unwrap(); + assert_eq!(r.status(), 502); + let ov = json_of( + s.admin(Method::GET, "overview", Some(&c)) + .send() + .await + .unwrap(), + ) + .await; + assert_eq!(ov["counts"]["canvases"], Value::Null); + assert_eq!(ov["counts"]["live_rooms"], Value::Null); + assert_eq!(ov["counts"]["live_peers"], Value::Null); + m.st.set_app_key(mock::APP_KEY); + + // Disable Bob → pushed as disabled and disconnected on Canvas Cloud. + let r = s + .admin(Method::PATCH, &format!("users/{}", bob.user_id), Some(&c)) + .json(&json!({ "disabled": true, "display_name": "Robert" })) + .send() + .await + .unwrap(); + assert_eq!(r.status(), 200); + wait_for("bob disabled upstream", || { + m.st.user(&bob.user_id) + .is_some_and(|u| u.disabled && u.display_name.as_deref() == Some("Robert")) + }) + .await; + wait_for("bob disconnected", || { + m.st.disconnects.lock().unwrap().contains(&bob.user_id) + }) + .await; + + // Delete Ana → deleted on Canvas Cloud (with her canvases). + let r = s + .admin(Method::DELETE, &format!("users/{}", ana.user_id), Some(&c)) + .send() + .await + .unwrap(); + assert_eq!(r.status(), 200); + wait_for("ana deleted upstream", || { + m.st.deleted_users.lock().unwrap().contains(&ana.user_id) + }) + .await; + assert_eq!(m.st.canvas_count(), 0); + + // Canvas Cloud down while deleting Bob: kept in the outbox, retried later. + m.st.set_app_key("cva_rotated_elsewhere_000"); + let r = s + .admin(Method::DELETE, &format!("users/{}", bob.user_id), Some(&c)) + .send() + .await + .unwrap(); + assert_eq!(r.status(), 200); + let mut row: Option<(i32, Option)> = None; + for _ in 0..200 { + row = sqlx::query_as( + "SELECT attempts, last_error FROM canvas_link_outbox + WHERE kind = 'delete_user' AND subject = $1 AND attempts > 0", + ) + .bind(bob.user_id.parse::().unwrap()) + .fetch_optional(&s.state.db) + .await + .unwrap(); + if row.is_some() { + break; + } + tokio::time::sleep(Duration::from_millis(50)).await; + } + let (attempts, last_error) = row.expect("failed delete kept in the outbox"); + assert_eq!(attempts, 1); + assert!(last_error.unwrap().contains("401")); + assert!(m.st.has_user(&bob.user_id)); + m.st.set_app_key(mock::APP_KEY); + sqlx::query("UPDATE canvas_link_outbox SET next_attempt_at = now()") + .execute(&s.state.db) + .await + .unwrap(); + s.link.kick_outbox(); + wait_for("bob deleted upstream", || { + m.st.deleted_users.lock().unwrap().contains(&bob.user_id) + }) + .await; + let mut left = 1_i64; + for _ in 0..100 { + left = sqlx::query_scalar("SELECT count(*) FROM canvas_link_outbox") + .fetch_one(&s.state.db) + .await + .unwrap(); + if left == 0 { + break; + } + tokio::time::sleep(Duration::from_millis(50)).await; + } + assert_eq!(left, 0, "outbox drained"); +} + +#[tokio::test] +async fn reconcile_pushes_changes_and_removes_deleted_users() { + let (s, _c, m, ana, bob) = setup().await; + let active = s.link.enabled().await.unwrap().unwrap(); + + // Nothing changed: nothing pushed. + let (pushed, disconnected, deleted) = + copper_cloud_canvas_link::provision::reconcile_once(&s.link, &active) + .await + .unwrap(); + assert_eq!((pushed, disconnected, deleted), (0, 0, 0)); + + // Changed outside the admin API (e.g. the CLI): email change, disable, delete. + sqlx::query("UPDATE users SET email = 'ana2@example.com' WHERE id = $1") + .bind(ana.user_id.parse::().unwrap()) + .execute(&s.state.db) + .await + .unwrap(); + let carol = s.signup("carol@example.com", PASSWORD, &new_device()).await; + sqlx::query("UPDATE users SET disabled = true WHERE id = $1") + .bind(carol.user_id.parse::().unwrap()) + .execute(&s.state.db) + .await + .unwrap(); + sqlx::query("DELETE FROM users WHERE id = $1") + .bind(bob.user_id.parse::().unwrap()) + .execute(&s.state.db) + .await + .unwrap(); + let (pushed, disconnected, deleted) = + copper_cloud_canvas_link::provision::reconcile_once(&s.link, &active) + .await + .unwrap(); + assert_eq!((pushed, disconnected, deleted), (2, 1, 1)); + assert_eq!(m.st.user(&ana.user_id).unwrap().email, "ana2@example.com"); + assert!(m.st.user(&carol.user_id).unwrap().disabled); + assert!(m.st.disconnects.lock().unwrap().contains(&carol.user_id)); + assert!(!m.st.has_user(&bob.user_id)); +} + +#[tokio::test] +async fn join_landing_page_is_public() { + let s = start().await; + let token = copper_cloud_core::ids::random_token(); + // No instance key: the landing page is outside /v1 and needs no database. + let r = s + .http + .get(s.url(&format!("/join/{token}"))) + .send() + .await + .unwrap(); + assert_eq!(r.status(), 200); + assert_eq!(r.headers()["cache-control"], "no-store"); + let html = r.text().await.unwrap(); + assert!(html.contains(&format!("copper://canvas/join/{token}"))); +} diff --git a/crates/copper-cloud/tests/common/mod.rs b/crates/copper-cloud/tests/common/mod.rs index efb8df6..d74f557 100644 --- a/crates/copper-cloud/tests/common/mod.rs +++ b/crates/copper-cloud/tests/common/mod.rs @@ -24,6 +24,8 @@ pub struct TestServer { pub addr: SocketAddr, pub base: String, pub state: SharedState, + /// The app's canvas link (settings reload, outbox kick, shutdown close). + pub link: copper_cloud_canvas_link::CanvasLink, pub http: reqwest::Client, _guard: MutexGuard<'static, ()>, } @@ -48,7 +50,7 @@ pub async fn start_with(f: impl FnOnce(&mut Config)) -> TestServer { } truncate_all(&pool).await; let state = AppState::new(pool, cfg); - let app = copper_cloud::build_app(state.clone()); + let (app, link) = copper_cloud::build_app_with_link(state.clone()); let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap(); let addr = listener.local_addr().unwrap(); tokio::spawn(async move { @@ -63,6 +65,7 @@ pub async fn start_with(f: impl FnOnce(&mut Config)) -> TestServer { addr, base: format!("http://{addr}"), state, + link, http: reqwest::Client::new(), _guard: guard, } diff --git a/crates/copper-cloud/tests/history_delete.rs b/crates/copper-cloud/tests/history_delete.rs index c675759..5271e5b 100644 --- a/crates/copper-cloud/tests/history_delete.rs +++ b/crates/copper-cloud/tests/history_delete.rs @@ -653,7 +653,8 @@ async fn info_lists_features() { .json() .await .unwrap(); - assert_eq!(info["features"], json!(["history_delete"])); + assert_eq!(info["features"], json!(["history_delete", "canvas_link"])); + assert_eq!(info["canvas"], json!({ "linked": false })); assert_eq!(info["version"], env!("CARGO_PKG_VERSION")); } diff --git a/crates/core/src/app.rs b/crates/core/src/app.rs index 4560a5a..2abc898 100644 --- a/crates/core/src/app.rs +++ b/crates/core/src/app.rs @@ -33,6 +33,8 @@ pub const ADMIN_LOGIN_PATH: &str = "/admin/api/login"; pub const FEATURES: &[&str] = &[ // `DELETE /v1/sync/history` (0.8.0). "history_delete", + // Canvases are served by a linked Canvas Cloud (0.9.0; see `canvas` in `/v1/info`). + "canvas_link", ]; /// Core `/v1` routes (auth, pairing, devices, sync, intelligence, info). Paths are relative to `/v1`. From a5e778c43e23592a722884f8ff8e4bd21931510b Mon Sep 17 00:00:00 2001 From: Felipe Arce Date: Mon, 19 Oct 2026 18:54:45 +0000 Subject: [PATCH 2/9] canvas-migrate: move legacy canvases into a linked Canvas Cloud copper-cloud canvas migrate [--dry-run] [--only ID]... [--replace] [--concurrency N] [--json] streams every legacy canvas (snapshot + updates unsealed with the Copper KEK, members, invites, share-link digests, mention excerpts) as import format v1 to the linked Canvas Cloud, with canvas ids, user ids and share tokens preserved. Each import is verified by document digest and recorded in canvas_migrations; legacy tables are never modified, so rolling back is reinstalling the previous release. --- crates/canvas-migrate/Cargo.toml | 41 + crates/canvas-migrate/src/ledger.rs | 126 +++ crates/canvas-migrate/src/legacy.rs | 492 ++++++++++ crates/canvas-migrate/src/lib.rs | 610 +++++++++++++ crates/canvas-migrate/src/remote.rs | 436 +++++++++ crates/canvas-migrate/src/report.rs | 372 ++++++++ crates/canvas-migrate/src/wire.rs | 136 +++ crates/canvas-migrate/tests/common/digest.rs | 189 ++++ crates/canvas-migrate/tests/common/mock.rs | 580 ++++++++++++ crates/canvas-migrate/tests/common/mod.rs | 519 +++++++++++ crates/canvas-migrate/tests/digest.rs | 220 +++++ crates/canvas-migrate/tests/migrate.rs | 914 +++++++++++++++++++ 12 files changed, 4635 insertions(+) create mode 100644 crates/canvas-migrate/Cargo.toml create mode 100644 crates/canvas-migrate/src/ledger.rs create mode 100644 crates/canvas-migrate/src/legacy.rs create mode 100644 crates/canvas-migrate/src/lib.rs create mode 100644 crates/canvas-migrate/src/remote.rs create mode 100644 crates/canvas-migrate/src/report.rs create mode 100644 crates/canvas-migrate/src/wire.rs create mode 100644 crates/canvas-migrate/tests/common/digest.rs create mode 100644 crates/canvas-migrate/tests/common/mock.rs create mode 100644 crates/canvas-migrate/tests/common/mod.rs create mode 100644 crates/canvas-migrate/tests/digest.rs create mode 100644 crates/canvas-migrate/tests/migrate.rs diff --git a/crates/canvas-migrate/Cargo.toml b/crates/canvas-migrate/Cargo.toml new file mode 100644 index 0000000..0af2d92 --- /dev/null +++ b/crates/canvas-migrate/Cargo.toml @@ -0,0 +1,41 @@ +[package] +name = "copper-cloud-canvas-migrate" +description = "Moves copper-cloud's legacy canvases into a linked Canvas Cloud (import format v1), verified and idempotent" +version.workspace = true +edition.workspace = true +license.workspace = true +repository.workspace = true +homepage.workspace = true +documentation.workspace = true +authors.workspace = true +rust-version.workspace = true +publish = false + +[lints] +workspace = true + +[dependencies] +copper-cloud-core.workspace = true +anyhow.workspace = true +base64.workspace = true +bytes.workspace = true +futures.workspace = true +hex.workspace = true +# TLS is the caller's business: `run` takes a pre-configured client (rustls + pinning live +# in copper-cloud-canvas-link). This crate only needs streamed request bodies and JSON. +reqwest = { workspace = true, features = ["json", "stream"] } +serde.workspace = true +serde_json.workspace = true +sqlx.workspace = true +time.workspace = true +tokio.workspace = true +tracing.workspace = true +url.workspace = true +uuid.workspace = true + +[dev-dependencies] +axum.workspace = true +sha2.workspace = true +# Test-only: the mock Canvas Cloud really applies the imported Yjs updates. copper-cloud +# itself does not depend on yrs. +yrs = "0.28" diff --git a/crates/canvas-migrate/src/ledger.rs b/crates/canvas-migrate/src/ledger.rs new file mode 100644 index 0000000..6aa8d4c --- /dev/null +++ b/crates/canvas-migrate/src/ledger.rs @@ -0,0 +1,126 @@ +//! The `canvas_migrations` ledger (created by copper-cloud migration `0030_canvas_link.sql`): +//! `(canvas_id uuid pk, status text, digest text, updates int, migrated_at timestamptz, +//! detail jsonb)`. +//! +//! Statuses written here: +//! +//! * `imported` — canvas-cloud answered `200` to the import and the counts matched; the +//! digest is the one the import returned. Written *before* the digest round trip, so a crash +//! between the import and verification leaves evidence a re-run can match (a `409` whose +//! digest equals it is then accepted). +//! * `migrated` — imported and verified (`digest` == `GET …/digest`). Re-runs skip these. +//! * `failed` — the last attempt failed (`detail.error`). A failure that did not change +//! canvas-cloud (refused import, transport error, unreadable legacy row, "exists with other +//! content") never downgrades an existing `migrated` row. + +use std::collections::HashMap; + +use anyhow::Context as _; +use serde_json::Value; +use sqlx::PgPool; +use uuid::Uuid; + +#[derive(Debug, Clone)] +pub(crate) struct Entry { + pub status: String, + pub digest: Option, +} + +impl Entry { + /// A record this tool wrote after canvas-cloud accepted the canvas. + pub fn vouches_for(&self, digest: &str) -> bool { + matches!(self.status.as_str(), "migrated" | "imported") + && self.digest.as_deref() == Some(digest) + } +} + +pub(crate) async fn load(db: &PgPool) -> anyhow::Result> { + let rows: Vec<(Uuid, String, Option)> = + sqlx::query_as("SELECT canvas_id, status, digest FROM canvas_migrations") + .fetch_all(db) + .await + .context("reading the canvas_migrations ledger")?; + Ok(rows + .into_iter() + .map(|(id, status, digest)| (id, Entry { status, digest })) + .collect()) +} + +/// How a ledger write treats an existing row. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(crate) enum Write { + /// Always replace the row (canvas-cloud changed). + Overwrite, + /// Leave a `migrated` row alone (nothing changed in canvas-cloud). + KeepMigrated, +} + +pub(crate) async fn record( + db: &PgPool, + canvas_id: Uuid, + status: &str, + digest: Option<&str>, + updates: u64, + detail: &Value, + mode: Write, +) -> anyhow::Result<()> { + let sql = match mode { + Write::Overwrite => { + "INSERT INTO canvas_migrations (canvas_id, status, digest, updates, migrated_at, detail) + VALUES ($1, $2, $3, $4, now(), $5) + ON CONFLICT (canvas_id) DO UPDATE SET + status = EXCLUDED.status, + digest = COALESCE(EXCLUDED.digest, canvas_migrations.digest), + updates = EXCLUDED.updates, + migrated_at = EXCLUDED.migrated_at, + detail = EXCLUDED.detail" + } + Write::KeepMigrated => { + "INSERT INTO canvas_migrations (canvas_id, status, digest, updates, migrated_at, detail) + VALUES ($1, $2, $3, $4, now(), $5) + ON CONFLICT (canvas_id) DO UPDATE SET + status = EXCLUDED.status, + digest = COALESCE(EXCLUDED.digest, canvas_migrations.digest), + updates = EXCLUDED.updates, + migrated_at = EXCLUDED.migrated_at, + detail = EXCLUDED.detail + WHERE canvas_migrations.status <> 'migrated'" + } + }; + sqlx::query(sql) + .bind(canvas_id) + .bind(status) + .bind(digest) + .bind(i32::try_from(updates).unwrap_or(i32::MAX)) + .bind(detail) + .execute(db) + .await + .with_context(|| format!("recording canvas {canvas_id} in canvas_migrations"))?; + Ok(()) +} + +/// Marks an existing `imported` row (or a missing one) `migrated` after a re-run proved +/// canvas-cloud holds exactly what was imported. +pub(crate) async fn confirm( + db: &PgPool, + canvas_id: Uuid, + digest: &str, + detail: &Value, +) -> anyhow::Result<()> { + sqlx::query( + "INSERT INTO canvas_migrations (canvas_id, status, digest, updates, migrated_at, detail) + VALUES ($1, 'migrated', $2, NULL, now(), $3) + ON CONFLICT (canvas_id) DO UPDATE SET + status = 'migrated', + digest = EXCLUDED.digest, + detail = COALESCE(canvas_migrations.detail, '{}'::jsonb) || EXCLUDED.detail + WHERE canvas_migrations.status <> 'migrated'", + ) + .bind(canvas_id) + .bind(digest) + .bind(detail) + .execute(db) + .await + .with_context(|| format!("confirming canvas {canvas_id} in canvas_migrations"))?; + Ok(()) +} diff --git a/crates/canvas-migrate/src/legacy.rs b/crates/canvas-migrate/src/legacy.rs new file mode 100644 index 0000000..cf62428 --- /dev/null +++ b/crates/canvas-migrate/src/legacy.rs @@ -0,0 +1,492 @@ +//! Reads copper-cloud's legacy canvas tables (migrations `0100`–`0103`) — strictly read-only. +//! +//! Decryption follows the old `copper-cloud-canvas` crate exactly: +//! +//! * doc key: `canvases.doc_key_wrapped`, unwrapped with the copper KEK +//! ([`Crypto::unwrap_key`]); +//! * snapshot (`canvas_snapshots.state`) and updates (`canvas_updates."update"`): AES-256-GCM +//! under the doc key, AAD = the canvas id's 16 raw bytes (`store.rs`); +//! * mention excerpts (`canvas_mentions.excerpt_sealed`): same doc key, AAD = +//! `b"copper-cloud/v1/mention:"` ‖ the mention row id's 16 raw bytes (`mentions.rs`). +//! +//! A canvas is read inside one `REPEATABLE READ, READ ONLY` transaction, so the snapshot and +//! the update tail are consistent even if a compaction ran concurrently (the runbook stops +//! copper-cloud first anyway). Updates are paged by `seq`, so memory stays bounded by one +//! page however large the canvas is. + +use std::sync::{Arc, Mutex, PoisonError}; + +use anyhow::Context as _; +use base64::Engine as _; +use bytes::Bytes; +use copper_cloud_core::crypto::Crypto; +use futures::Stream; +use sqlx::PgPool; +use time::OffsetDateTime; +use uuid::Uuid; + +use crate::wire::{subject, Line, VERSION}; + +/// AAD prefix of a sealed mention excerpt in copper-cloud (`mentions.rs`). +pub const EXCERPT_AAD_PREFIX: &[u8] = b"copper-cloud/v1/mention:"; + +/// Update rows fetched per page while streaming. +const UPDATE_PAGE: i64 = 16; + +/// The AAD binding a sealed update/snapshot to its canvas (`store.rs`). +pub fn doc_aad(canvas_id: Uuid) -> [u8; 16] { + *canvas_id.as_bytes() +} + +/// The AAD binding a sealed mention excerpt to its row (`mentions.rs`). +pub fn excerpt_aad(mention_id: Uuid) -> Vec { + let mut aad = Vec::with_capacity(EXCERPT_AAD_PREFIX.len() + 16); + aad.extend_from_slice(EXCERPT_AAD_PREFIX); + aad.extend_from_slice(mention_id.as_bytes()); + aad +} + +/// Optional parts of the legacy schema (older deployments may predate some migrations). +#[derive(Debug, Clone, Copy)] +#[allow(clippy::struct_excessive_bools)] // independent "does this table/column exist" flags +pub(crate) struct Schema { + pub canvases: bool, + pub nudged_at: bool, + pub links: bool, + pub mentions: bool, + pub ledger: bool, +} + +pub(crate) async fn schema(db: &PgPool) -> anyhow::Result { + let (canvases, nudged_at, links, mentions, ledger): (bool, bool, bool, bool, bool) = + sqlx::query_as( + "SELECT to_regclass('canvases') IS NOT NULL, + EXISTS (SELECT 1 FROM pg_attribute + WHERE attrelid = to_regclass('canvas_invites') + AND attname = 'nudged_at' AND NOT attisdropped), + to_regclass('canvas_share_links') IS NOT NULL, + to_regclass('canvas_mentions') IS NOT NULL, + to_regclass('canvas_migrations') IS NOT NULL", + ) + .fetch_one(db) + .await + .context("inspecting the legacy canvas schema")?; + Ok(Schema { + canvases, + nudged_at, + links, + mentions, + ledger, + }) +} + +/// Fails fast when `crypto` cannot unwrap the doc keys (not copper-cloud's `master_key`), +/// instead of failing every canvas one by one. Checks the oldest and the newest canvas. +pub(crate) async fn check_master_key(db: &PgPool, crypto: &Crypto) -> anyhow::Result<()> { + let keys: Vec<(Uuid, Vec)> = sqlx::query_as( + "(SELECT id, doc_key_wrapped FROM canvases ORDER BY created_at, id LIMIT 1) + UNION + (SELECT id, doc_key_wrapped FROM canvases ORDER BY created_at DESC, id DESC LIMIT 1)", + ) + .fetch_all(db) + .await + .context("reading canvas doc keys")?; + for (id, wrapped) in keys { + crypto.unwrap_key(&wrapped).with_context(|| { + format!("cannot unwrap the doc key of canvas {id}: is this copper-cloud's master_key?") + })?; + } + Ok(()) +} + +/// A copper-cloud account, as provisioned into canvas-cloud. +#[derive(Debug, Clone, sqlx::FromRow)] +pub(crate) struct LegacyUser { + pub id: Uuid, + pub email: String, + pub display_name: String, + pub disabled: bool, +} + +pub(crate) async fn users(db: &PgPool) -> anyhow::Result> { + sqlx::query_as("SELECT id, email, display_name, disabled FROM users ORDER BY created_at, id") + .fetch_all(db) + .await + .context("reading users") +} + +/// One legacy canvas (metadata only). +#[derive(Debug, Clone, sqlx::FromRow)] +pub(crate) struct LegacyCanvas { + pub id: Uuid, + pub name: String, + pub kind: String, + pub owner_id: Uuid, + /// Sealed bytes of snapshot + update tail (sizing the import timeout). + pub stored_bytes: i64, +} + +pub(crate) async fn canvases(db: &PgPool) -> anyhow::Result> { + sqlx::query_as( + r#"SELECT c.id, c.name, c.kind, c.owner_id, + (COALESCE((SELECT octet_length(s.state) FROM canvas_snapshots s + WHERE s.canvas_id = c.id), 0) + + COALESCE((SELECT sum(octet_length(u."update")) FROM canvas_updates u + WHERE u.canvas_id = c.id), 0))::bigint AS stored_bytes + FROM canvases c + ORDER BY c.created_at, c.id"#, + ) + .fetch_all(db) + .await + .context("reading canvases") +} + +/// What one canvas stream has produced so far (shared with the caller, which reads it after +/// the stream ends or fails). +#[derive(Debug, Default, Clone)] +pub(crate) struct Sent { + pub updates: u64, + /// Plaintext Yjs bytes (snapshot + updates). + pub bytes: u64, + pub members: u64, + pub invites: u64, + pub links: u64, + pub mentions: u64, + pub snapshot: bool, + pub warnings: Vec, + /// The first error that aborted the stream (an HTTP error then only says "body error"). + pub error: Option, + /// The `end` line was produced. + pub complete: bool, +} + +pub(crate) type SharedSent = Arc>; + +pub(crate) fn with_sent(sent: &SharedSent, f: impl FnOnce(&mut Sent) -> R) -> R { + f(&mut sent.lock().unwrap_or_else(PoisonError::into_inner)) +} + +#[derive(sqlx::FromRow)] +struct CanvasRow { + name: String, + kind: String, + owner_id: Uuid, + doc_key_wrapped: Vec, + created_at: OffsetDateTime, + updated_at: OffsetDateTime, +} + +#[derive(sqlx::FromRow)] +struct MemberRow { + user_id: Uuid, + role: String, + added_at: OffsetDateTime, +} + +#[derive(sqlx::FromRow)] +struct InviteRow { + id: Uuid, + email: String, + invited_by: Option, + token: String, + status: String, + created_at: OffsetDateTime, + accepted_at: Option, + nudged_at: Option, +} + +#[derive(sqlx::FromRow)] +struct LinkRow { + id: Uuid, + token_sha256: Vec, + role: String, + created_by: Option, + created_at: OffsetDateTime, + uses: i64, +} + +#[derive(sqlx::FromRow)] +struct MentionRow { + id: Uuid, + message_id: String, + from_user: Uuid, + to_user: Uuid, + excerpt_sealed: Vec, + created_at: OffsetDateTime, + read_at: Option, +} + +fn invites_sql(schema: Schema) -> &'static str { + if schema.nudged_at { + "SELECT id, email, invited_by, token, status, created_at, accepted_at, nudged_at + FROM canvas_invites WHERE canvas_id = $1 ORDER BY created_at, id" + } else { + "SELECT id, email, invited_by, token, status, created_at, accepted_at, + NULL::timestamptz AS nudged_at + FROM canvas_invites WHERE canvas_id = $1 ORDER BY created_at, id" + } +} + +/// Lines buffered between the database reader and the HTTP body. +const CHANNEL_LINES: usize = 4; + +/// The import v1 body of one canvas, produced lazily by a reader task behind a small bounded +/// channel (backpressure: at most a few lines in flight). Never writes to the database. The +/// first error is recorded in `sent.error` and ends the stream with `Err`, which aborts the +/// HTTP body — canvas-cloud then never sees the `end` line and commits nothing. +pub(crate) fn canvas_stream( + db: PgPool, + crypto: Crypto, + canvas_id: Uuid, + schema: Schema, + sent: SharedSent, +) -> impl Stream> + Send + 'static { + let (out, rx) = tokio::sync::mpsc::channel::>(CHANNEL_LINES); + tokio::spawn(async move { + let result = produce(&db, &crypto, canvas_id, schema, &sent, &out).await; + match result { + Ok(()) => with_sent(&sent, |s| s.complete = true), + Err(e) => { + with_sent(&sent, |s| { + s.error.get_or_insert_with(|| format!("{e:#}")); + }); + // The receiver may be gone (request already failed); nothing else to do. + let _ = out.send(Err(e)).await; + } + } + }); + futures::stream::unfold(rx, |mut rx| async move { + let item = rx.recv().await?; + Some((item, rx)) + }) +} + +/// canvas-cloud's default import line limit (CONTRACT §6.1). Configurable server-side, so +/// an oversized line is only a warning here (visible in a dry run); the server decides. +pub(crate) const LINE_LIMIT: usize = 32 * 1024 * 1024; + +fn check_line(sent: &SharedSent, what: &str, line: &[u8]) { + if line.len() > LINE_LIMIT { + with_sent(sent, |s| { + s.warnings.push(format!( + "{what} is a {} MiB import line, over canvas-cloud's default 32 MiB line limit", + line.len() / (1024 * 1024) + )); + }); + } +} + +/// Sends one line; fails when the consumer went away (the request was dropped). +async fn emit( + out: &tokio::sync::mpsc::Sender>, + bytes: Bytes, +) -> anyhow::Result<()> { + out.send(Ok(bytes)) + .await + .map_err(|_| anyhow::anyhow!("the import request ended before the body was sent")) +} + +#[allow(clippy::too_many_lines)] // one linear pass over the format, line type by line type +async fn produce( + db: &PgPool, + crypto: &Crypto, + canvas_id: Uuid, + schema: Schema, + sent: &SharedSent, + out: &tokio::sync::mpsc::Sender>, +) -> anyhow::Result<()> { + let mut tx = db.begin().await.context("opening a read transaction")?; + sqlx::query("SET TRANSACTION ISOLATION LEVEL REPEATABLE READ, READ ONLY") + .execute(&mut *tx) + .await?; + let canvas: CanvasRow = sqlx::query_as( + "SELECT name, kind, owner_id, doc_key_wrapped, created_at, updated_at + FROM canvases WHERE id = $1", + ) + .bind(canvas_id) + .fetch_optional(&mut *tx) + .await? + .context("the canvas disappeared from the legacy tables")?; + let key = crypto + .unwrap_key(&canvas.doc_key_wrapped) + .context("unwrapping the canvas doc key (is this copper-cloud's master_key?)")?; + let aad = doc_aad(canvas_id); + + let bytes = Line::Canvas { + version: VERSION, + id: canvas_id, + name: &canvas.name, + kind: &canvas.kind, + owner: subject(canvas.owner_id), + created_at: canvas.created_at, + updated_at: canvas.updated_at, + } + .to_bytes()?; + emit(out, bytes).await?; + + let members: Vec = sqlx::query_as( + "SELECT user_id, role, added_at FROM canvas_members WHERE canvas_id = $1 + ORDER BY (role = 'owner') DESC, added_at, user_id", + ) + .bind(canvas_id) + .fetch_all(&mut *tx) + .await?; + for m in &members { + let bytes = Line::Member { + subject: subject(m.user_id), + role: &m.role, + added_at: m.added_at, + } + .to_bytes()?; + emit(out, bytes).await?; + } + with_sent(sent, |s| s.members = members.len() as u64); + + let invites: Vec = sqlx::query_as(invites_sql(schema)) + .bind(canvas_id) + .fetch_all(&mut *tx) + .await?; + for i in &invites { + // `canvas_invites.token` already is the lower-case hex SHA-256 of the invite's + // random secret (rest.rs: `ids::sha256_hex(random_token())`), so it is sent as is. + let bytes = Line::Invite { + id: i.id, + email: &i.email, + invited_by: i.invited_by.map(subject), + status: &i.status, + created_at: i.created_at, + accepted_at: i.accepted_at, + nudged_at: i.nudged_at, + token_sha256_hex: &i.token, + } + .to_bytes()?; + emit(out, bytes).await?; + } + with_sent(sent, |s| s.invites = invites.len() as u64); + + if schema.links { + let links: Vec = sqlx::query_as( + "SELECT id, token_sha256, role, created_by, created_at, uses + FROM canvas_share_links WHERE canvas_id = $1 ORDER BY created_at, id", + ) + .bind(canvas_id) + .fetch_all(&mut *tx) + .await?; + for l in &links { + let bytes = Line::Link { + id: l.id, + token_sha256_hex: hex::encode(&l.token_sha256), + role: &l.role, + created_by: l.created_by.map(subject), + created_at: l.created_at, + uses: l.uses, + } + .to_bytes()?; + emit(out, bytes).await?; + } + with_sent(sent, |s| s.links = links.len() as u64); + } + + if schema.mentions { + let mentions: Vec = sqlx::query_as( + "SELECT id, message_id, from_user, to_user, excerpt_sealed, created_at, read_at + FROM canvas_mentions WHERE canvas_id = $1 ORDER BY created_at, id", + ) + .bind(canvas_id) + .fetch_all(&mut *tx) + .await?; + for m in &mentions { + // The old server served an excerpt that no longer decrypts as "" (with a + // warning); keep that behaviour rather than failing the whole canvas. + let excerpt = + if let Ok(bytes) = Crypto::open(&key, &excerpt_aad(m.id), &m.excerpt_sealed) { + String::from_utf8_lossy(&bytes).into_owned() + } else { + with_sent(sent, |s| { + s.warnings.push(format!( + "mention {} excerpt does not decrypt; sent empty", + m.id + )); + }); + String::new() + }; + let bytes = Line::Mention { + id: m.id, + message_id: &m.message_id, + from: subject(m.from_user), + to: subject(m.to_user), + excerpt: &excerpt, + created_at: m.created_at, + read_at: m.read_at, + } + .to_bytes()?; + emit(out, bytes).await?; + } + with_sent(sent, |s| s.mentions = mentions.len() as u64); + } + + let engine = base64::engine::general_purpose::STANDARD; + let snapshot: Option<(i64, Vec)> = + sqlx::query_as("SELECT seq, state FROM canvas_snapshots WHERE canvas_id = $1") + .bind(canvas_id) + .fetch_optional(&mut *tx) + .await?; + let mut since = 0_i64; + if let Some((seq, sealed)) = snapshot { + let state = Crypto::open(&key, &aad, &sealed) + .with_context(|| format!("the snapshot (seq {seq}) does not decrypt"))?; + since = seq; + with_sent(sent, |s| { + s.snapshot = true; + s.updates += 1; + s.bytes += state.len() as u64; + }); + let bytes = Line::Update { + b64: engine.encode(&state), + } + .to_bytes()?; + check_line(sent, "the snapshot", &bytes); + emit(out, bytes).await?; + } + loop { + let page: Vec<(i64, Vec)> = sqlx::query_as( + r#"SELECT seq, "update" FROM canvas_updates + WHERE canvas_id = $1 AND seq > $2 ORDER BY seq LIMIT $3"#, + ) + .bind(canvas_id) + .bind(since) + .bind(UPDATE_PAGE) + .fetch_all(&mut *tx) + .await?; + if page.is_empty() { + break; + } + for (seq, sealed) in page { + let update = Crypto::open(&key, &aad, &sealed) + .with_context(|| format!("update seq {seq} does not decrypt"))?; + since = seq; + with_sent(sent, |s| { + s.updates += 1; + s.bytes += update.len() as u64; + }); + let bytes = Line::Update { + b64: engine.encode(&update), + } + .to_bytes()?; + check_line(sent, &format!("update seq {seq}"), &bytes); + emit(out, bytes).await?; + } + } + tx.rollback().await?; + + let end = with_sent(sent, |s| Line::End { + updates: s.updates, + members: s.members, + invites: s.invites, + links: s.links, + mentions: s.mentions, + }); + let bytes = end.to_bytes()?; + emit(out, bytes).await?; + Ok(()) +} diff --git a/crates/canvas-migrate/src/lib.rs b/crates/canvas-migrate/src/lib.rs new file mode 100644 index 0000000..2ea448c --- /dev/null +++ b/crates/canvas-migrate/src/lib.rs @@ -0,0 +1,610 @@ +//! `copper-cloud canvas migrate`: moves copper-cloud's legacy canvases (tables from +//! migrations `0100`–`0103`) into a linked Canvas Cloud through its import format v1 +//! (`POST /v1/apps/self/import`, CONTRACT §6/§8), preserving canvas ids, user ids, share-link +//! tokens, invites and mentions. +//! +//! * **Read-only on the legacy side.** Legacy rows are never modified or deleted (rollback = +//! reinstall the old version); the only writes are to the `canvas_migrations` ledger. +//! * **Users first.** Every copper user is provisioned (`users:batch`, subject = id hint = the +//! copper user id) and must come back with that same id — otherwise chat author ids and +//! member lists would break, so nothing is imported. +//! * **Streamed.** Each canvas is one NDJSON request body produced lazily from the database +//! (one page of updates in memory at a time); `concurrency` canvases are in flight. +//! * **Verified.** A canvas counts as migrated only when canvas-cloud's import counts equal +//! what was sent and the import's digest equals `GET /v1/apps/self/canvases/{id}/digest`. +//! * **Idempotent.** Canvases recorded `migrated` are skipped (unless `--replace`/`--only`); +//! a canvas that already exists in canvas-cloud is accepted only when its digest matches +//! this ledger's record — anything else needs `--replace`. + +mod ledger; +mod legacy; +mod remote; +mod report; +mod wire; + +use std::collections::{HashMap, HashSet}; +use std::sync::atomic::{AtomicBool, Ordering}; +use std::time::Duration; + +use copper_cloud_core::crypto::Crypto; +use futures::{StreamExt as _, TryStreamExt as _}; +use reqwest::StatusCode; +use serde_json::json; +use sqlx::PgPool; +use uuid::Uuid; + +pub use legacy::{doc_aad, excerpt_aad, EXCERPT_AAD_PREFIX}; +pub use report::{CanvasReport, MigrateReport, Status, Totals, UserFailure, UsersReport}; +pub use wire::{CONTENT_TYPE, VERSION}; + +use crate::legacy::{LegacyCanvas, Schema, SharedSent}; +use crate::remote::{Failure, Remote}; + +/// Canvases in flight when `MigrateOptions::concurrency` is 0. +pub const DEFAULT_CONCURRENCY: usize = 2; + +/// Options of one run (`copper-cloud canvas migrate …`). +#[derive(Debug, Clone)] +pub struct MigrateOptions { + /// Read, decrypt and encode everything, send nothing (only the preflight request). + pub dry_run: bool, + /// Only these canvases (even if the ledger says `migrated`). Empty = all. + pub only: Vec, + /// Import with `mode=replace` (overwrite earlier imports by this app), ledger ignored. + pub replace: bool, + /// Canvases in flight (0 → [`DEFAULT_CONCURRENCY`]). + pub concurrency: usize, + /// The report's `Display` is JSON instead of a table. + pub json: bool, +} + +impl Default for MigrateOptions { + fn default() -> Self { + Self { + dry_run: false, + only: Vec::new(), + replace: false, + concurrency: DEFAULT_CONCURRENCY, + json: false, + } + } +} + +/// Runs the migration against the Canvas Cloud at `base_url` using `app_key` (a `cva_…` key +/// of an app with `allow_import`). `client` is pre-configured by the caller (timeouts, TLS +/// pinning); this function adds `Authorization: Bearer ` itself. +/// +/// Returns `Err` only for run-level failures (preflight, the app lacks `allow_import`, +/// database errors). Per-canvas failures and unprovisionable users are in the report — +/// check [`MigrateReport::ok`] for the exit status. +pub async fn run( + db: &PgPool, + crypto: &Crypto, + client: &reqwest::Client, + base_url: &url::Url, + app_key: &str, + opts: MigrateOptions, +) -> anyhow::Result { + let remote = Remote::new(client, base_url, app_key)?; + let mut report = MigrateReport::new(remote.base(), opts.dry_run, opts.replace, opts.json); + + let stats = remote.preflight().await?; + if stats + .get("allow_import") + .and_then(serde_json::Value::as_bool) + == Some(false) + && !opts.dry_run + { + anyhow::bail!( + "the canvas-cloud app lacks allow_import (recreate it with `canvas-cloud apps create --allow-import`)" + ); + } + + let schema = legacy::schema(db).await?; + if !schema.canvases { + tracing::info!("no legacy canvas tables: nothing to migrate"); + report.finish(); + return Ok(report); + } + if !schema.ledger && !opts.dry_run { + anyhow::bail!( + "the canvas_migrations ledger table is missing — run copper-cloud's database migrations (0030_canvas_link.sql) first" + ); + } + legacy::check_master_key(db, crypto).await?; + let ledger = if schema.ledger { + ledger::load(db).await? + } else { + HashMap::new() + }; + + // 1. Users. + let users = legacy::users(db).await?; + report.users.total = users.len(); + if !opts.dry_run { + provision(&remote, &users, &mut report).await?; + if report.error.is_some() { + report.finish(); + return Ok(report); + } + } + + // 2. Which canvases. + let (plan, unknown_only) = plan(legacy::canvases(db).await?, &ledger, &opts); + + // 3. Canvases, `concurrency` at a time, results in legacy order. + let ctx = Ctx { + db, + crypto, + remote: &remote, + schema, + ledger: &ledger, + dry_run: opts.dry_run, + replace: opts.replace, + forbidden: AtomicBool::new(false), + }; + let rows: Vec = futures::stream::iter(plan) + .map(|p| ctx.handle(p)) + .buffered(if opts.concurrency == 0 { + DEFAULT_CONCURRENCY + } else { + opts.concurrency + }) + .collect() + .await; + if ctx.forbidden.load(Ordering::SeqCst) { + anyhow::bail!( + "canvas-cloud refused the import (403): the app lacks allow_import \ + (recreate it with `canvas-cloud apps create --allow-import`); nothing was migrated by this run" + ); + } + report.canvases = rows; + report.canvases.extend(unknown_only); + report.finish(); + Ok(report) +} + +/// Provisions every copper user; any user that does not come back with its own id is a +/// run-level error (recorded in the report, nothing gets imported). +async fn provision( + remote: &Remote<'_>, + users: &[legacy::LegacyUser], + report: &mut MigrateReport, +) -> anyhow::Result<()> { + let outcomes = remote.provision(users).await?; + for (user, outcome) in users.iter().zip(outcomes) { + match outcome.error { + None => { + report.users.provisioned += 1; + report.users.created += usize::from(outcome.created); + } + Some(error) => report.users.failed.push(UserFailure { + user_id: user.id, + email: user.email.clone(), + error, + }), + } + } + if report.users.failed.is_empty() { + tracing::info!( + users = report.users.provisioned, + created = report.users.created, + "users provisioned" + ); + } else { + report.error = Some(format!( + "{} user(s) could not be provisioned with their copper user id; no canvas was migrated \ + (canvas-cloud user ids must equal copper ids for chat authors, members and mentions)", + report.users.failed.len() + )); + } + Ok(()) +} + +/// Selects the canvases of this run (legacy order) and reports `--only` ids that are not +/// legacy canvases. +fn plan( + all: Vec, + ledger: &HashMap, + opts: &MigrateOptions, +) -> (Vec, Vec) { + let only: HashSet = opts.only.iter().copied().collect(); + let mut plan: Vec = Vec::new(); + for canvas in all { + if !only.is_empty() && !only.contains(&canvas.id) { + continue; + } + let migrated = ledger + .get(&canvas.id) + .filter(|e| e.status == "migrated" && !opts.replace && !only.contains(&canvas.id)); + plan.push(match migrated { + Some(entry) => Plan::Skip(canvas, entry.digest.clone()), + None => Plan::Go(canvas), + }); + } + let known: HashSet = plan.iter().map(|p| p.canvas().id).collect(); + let mut seen = HashSet::new(); + let mut unknown = Vec::new(); + for id in &opts.only { + if !known.contains(id) && seen.insert(*id) { + let mut row = CanvasReport::new(*id, "", "", None); + row.error = + Some("not a legacy canvas (no such id in copper-cloud's canvases table)".into()); + unknown.push(row); + } + } + (plan, unknown) +} + +enum Plan { + /// Ledger says migrated (with this digest). + Skip(LegacyCanvas, Option), + Go(LegacyCanvas), +} + +impl Plan { + fn canvas(&self) -> &LegacyCanvas { + match self { + Self::Skip(c, _) | Self::Go(c) => c, + } + } +} + +struct Ctx<'a> { + db: &'a PgPool, + crypto: &'a Crypto, + remote: &'a Remote<'a>, + schema: Schema, + ledger: &'a HashMap, + dry_run: bool, + replace: bool, + /// An import answered 403: the app lacks `allow_import`; stop starting new imports. + forbidden: AtomicBool, +} + +/// Import request timeout: 2 minutes plus 2 s per MiB stored. +fn import_timeout(stored_bytes: i64) -> Duration { + let mib = u64::try_from(stored_bytes).unwrap_or(0) / (1024 * 1024); + Duration::from_secs(120 + 2 * mib) +} + +impl Ctx<'_> { + async fn handle(&self, plan: Plan) -> CanvasReport { + let canvas = plan.canvas().clone(); + let mut row = + CanvasReport::new(canvas.id, &canvas.name, &canvas.kind, Some(canvas.owner_id)); + match plan { + Plan::Skip(_, digest) => { + row.status = Status::Already; + row.digest = digest; + row.note = Some("recorded as migrated in the ledger".into()); + } + Plan::Go(_) if self.dry_run => self.dry_run(&canvas, &mut row).await, + Plan::Go(_) => { + if self.forbidden.load(Ordering::SeqCst) { + row.error = Some("not attempted: the app lacks allow_import".into()); + } else if let Err(e) = self.migrate(&canvas, &mut row).await { + // Only ledger/database errors reach here. + row.status = Status::Failed; + row.error = Some(format!("{e:#}")); + } + } + } + let level = if row.status == Status::Failed { + "failed" + } else { + "ok" + }; + tracing::info!( + canvas_id = %row.id, + status = row.status.as_str(), + updates = row.updates, + bytes = row.bytes, + error = row.error.as_deref().unwrap_or(""), + "canvas {level}" + ); + row + } + + fn stream( + &self, + canvas: &LegacyCanvas, + sent: &SharedSent, + ) -> impl futures::Stream> + Send + 'static { + legacy::canvas_stream( + self.db.clone(), + self.crypto.clone(), + canvas.id, + self.schema, + SharedSent::clone(sent), + ) + } + + async fn dry_run(&self, canvas: &LegacyCanvas, row: &mut CanvasReport) { + let sent = SharedSent::default(); + let result = self + .stream(canvas, &sent) + .try_fold(0_u64, |n, line| async move { Ok(n + line.len() as u64) }) + .await; + copy_counts(row, &sent); + match result { + Ok(_) => row.status = Status::DryRun, + Err(e) => row.error = Some(format!("{e:#}")), + } + } + + /// One canvas, for real. `Err` = ledger/database trouble only. + async fn migrate(&self, canvas: &LegacyCanvas, row: &mut CanvasReport) -> anyhow::Result<()> { + // Cheap re-run path: a canvas that already exists is judged without uploading it. + if !self.replace { + match self.remote.digest(canvas.id).await { + Ok(Some(stored)) => return self.exists(canvas, row, &stored, &stored).await, + Ok(None) => {} + Err(e) => { + tracing::warn!(canvas_id = %canvas.id, "digest pre-check failed ({e}); importing"); + } + } + } + + let sent = SharedSent::default(); + let body = reqwest::Body::wrap_stream(self.stream(canvas, &sent)); + let result = self + .remote + .import(body, self.replace, import_timeout(canvas.stored_bytes)) + .await; + copy_counts(row, &sent); + let local_error = legacy::with_sent(&sent, |s| s.error.clone()); + + let imported = match result { + Ok(imported) => imported, + Err(Failure::Http(h)) + if h.status == StatusCode::CONFLICT && h.code == "canvas_exists" => + { + let stored = match remote::conflict_digest(&h.body) { + Some(d) => Some(d), + None => self.remote.digest(canvas.id).await.ok().flatten(), + }; + let Some(conflict) = stored else { + return self.fail(canvas, row, "canvas exists in canvas-cloud but no digest is available (use --replace)".into(), None).await; + }; + let now = self + .remote + .digest(canvas.id) + .await + .ok() + .flatten() + .unwrap_or_else(|| conflict.clone()); + return self.exists(canvas, row, &conflict, &now).await; + } + Err(Failure::Http(h)) if h.status == StatusCode::FORBIDDEN => { + self.forbidden.store(true, Ordering::SeqCst); + row.error = Some(format!("{h} — the app lacks allow_import")); + return Ok(()); + } + Err(Failure::Http(h)) + if h.status == StatusCode::CONFLICT && h.code == "personal_exists" => + { + let msg = format!("{h} — the owner already has a non-empty Personal canvas in canvas-cloud (use --replace)"); + return self.fail(canvas, row, msg, None).await; + } + Err(failure) => { + // A local read/decrypt error aborts the body; report it rather than the + // resulting transport error. + let msg = match (local_error, &failure) { + (Some(local), Failure::Transport(_)) => { + format!("reading the legacy canvas: {local}") + } + (Some(local), Failure::Http(h)) => { + format!("reading the legacy canvas: {local} (canvas-cloud: {h})") + } + (None, f) => f.to_string(), + }; + return self.fail(canvas, row, msg, None).await; + } + }; + + self.verify(canvas, row, imported).await + } + + /// Checks a `200` import: counts equal what was sent, and the response digest equals the + /// stored one; then the ledger says `migrated`. + async fn verify( + &self, + canvas: &LegacyCanvas, + row: &mut CanvasReport, + imported: remote::Imported, + ) -> anyhow::Result<()> { + let replaced = imported.status.as_deref() == Some("replaced"); + let sent_counts = ( + row.updates, + row.members, + row.invites, + row.links, + row.mentions, + ); + let got_counts = ( + imported.updates_applied, + imported.members, + imported.invites, + imported.links, + imported.mentions, + ); + let digest = imported.digest.trim().to_ascii_lowercase(); + if sent_counts != got_counts { + let msg = format!( + "count mismatch: sent (updates, members, invites, links, mentions) = {sent_counts:?}, canvas-cloud applied {got_counts:?}" + ); + return self.fail_changed(canvas, row, msg, &digest).await; + } + let detail = self.detail(row, if replaced { "replaced" } else { "created" }); + ledger::record( + self.db, + canvas.id, + "imported", + Some(&digest), + row.updates, + &detail, + ledger::Write::Overwrite, + ) + .await?; + + match self.remote.digest(canvas.id).await { + Ok(Some(stored)) if stored == digest => { + ledger::record( + self.db, + canvas.id, + "migrated", + Some(&digest), + row.updates, + &detail, + ledger::Write::Overwrite, + ) + .await?; + row.status = if replaced { + Status::Replaced + } else { + Status::Migrated + }; + row.digest = Some(digest); + Ok(()) + } + Ok(Some(stored)) => { + let msg = format!("digest mismatch: the import returned {digest} but canvas-cloud now stores {stored}"); + self.fail_changed(canvas, row, msg, &digest).await + } + Ok(None) => { + let msg = + "verification failed: canvas-cloud reports the imported canvas does not exist" + .to_owned(); + self.fail_changed(canvas, row, msg, &digest).await + } + Err(e) => { + let msg = format!("verification failed: GET digest: {e} (the ledger keeps status `imported`; re-run to verify)"); + row.status = Status::Failed; + row.digest = Some(digest); + row.error = Some(msg); + Ok(()) + } + } + } + + /// The canvas already exists in canvas-cloud with digest `conflict` (and `stored` now): + /// `already` only if this ledger recorded exactly that import. + async fn exists( + &self, + canvas: &LegacyCanvas, + row: &mut CanvasReport, + conflict: &str, + stored: &str, + ) -> anyhow::Result<()> { + let entry = self.ledger.get(&canvas.id); + if conflict == stored && entry.is_some_and(|e| e.vouches_for(stored)) { + if entry.is_some_and(|e| e.status != "migrated") { + ledger::confirm( + self.db, + canvas.id, + stored, + &json!({ "confirmed_by_rerun": true }), + ) + .await?; + } + row.status = Status::Already; + row.digest = Some(stored.to_owned()); + row.note = Some("exists in canvas-cloud with the digest recorded in the ledger".into()); + return Ok(()); + } + let why = match entry { + Some(e) if e.digest.is_some() && e.status == "migrated" => { + "it changed since this ledger recorded it (edited in canvas-cloud?)" + } + Some(_) | None => "this ledger has no verified record of importing it", + }; + let msg = format!( + "exists in canvas-cloud with different content (digest {}…; {why}) — use --replace to overwrite it from the legacy data", + &stored[..stored.len().min(12)] + ); + self.fail(canvas, row, msg, None).await + } + + fn detail(&self, row: &CanvasReport, import_status: &str) -> serde_json::Value { + json!({ + "name": row.name, + "kind": row.kind, + "owner": row.owner, + "import_status": import_status, + "bytes": row.bytes, + "snapshot": row.snapshot, + "members": row.members, + "invites": row.invites, + "links": row.links, + "mentions": row.mentions, + "warnings": row.warnings, + "canvas_cloud": self.remote.base(), + "tool": concat!("copper-cloud-canvas-migrate ", env!("CARGO_PKG_VERSION")), + }) + } + + /// Failed without changing canvas-cloud: never downgrades a `migrated` ledger row. + async fn fail( + &self, + canvas: &LegacyCanvas, + row: &mut CanvasReport, + msg: String, + digest: Option<&str>, + ) -> anyhow::Result<()> { + let mut detail = self.detail(row, "none"); + detail["error"] = json!(msg); + ledger::record( + self.db, + canvas.id, + "failed", + digest, + row.updates, + &detail, + ledger::Write::KeepMigrated, + ) + .await?; + row.status = Status::Failed; + row.error = Some(msg); + Ok(()) + } + + /// Failed after canvas-cloud accepted (and stored) an import: always recorded. + async fn fail_changed( + &self, + canvas: &LegacyCanvas, + row: &mut CanvasReport, + msg: String, + digest: &str, + ) -> anyhow::Result<()> { + let mut detail = self.detail(row, "unverified"); + detail["error"] = json!(msg); + ledger::record( + self.db, + canvas.id, + "failed", + Some(digest), + row.updates, + &detail, + ledger::Write::Overwrite, + ) + .await?; + row.status = Status::Failed; + row.digest = Some(digest.to_owned()); + row.error = Some(msg); + Ok(()) + } +} + +fn copy_counts(row: &mut CanvasReport, sent: &SharedSent) { + legacy::with_sent(sent, |s| { + row.updates = s.updates; + row.bytes = s.bytes; + row.snapshot = s.snapshot; + row.members = s.members; + row.invites = s.invites; + row.links = s.links; + row.mentions = s.mentions; + row.warnings.clone_from(&s.warnings); + }); +} diff --git a/crates/canvas-migrate/src/remote.rs b/crates/canvas-migrate/src/remote.rs new file mode 100644 index 0000000..02c14ef --- /dev/null +++ b/crates/canvas-migrate/src/remote.rs @@ -0,0 +1,436 @@ +//! The few canvas-cloud app endpoints the migration uses (`/v1/apps/self/*`, app key only — +//! never `X-Canvas-User`). + +use std::time::Duration; + +use anyhow::Context as _; +use reqwest::header::{ACCEPT, AUTHORIZATION, CONTENT_TYPE}; +use reqwest::StatusCode; +use serde::{Deserialize, Serialize}; +use serde_json::Value; +use uuid::Uuid; + +use crate::legacy::LegacyUser; +use crate::wire::{subject, CONTENT_TYPE as NDJSON}; + +/// Users per `users:batch` call (canvas-cloud's limit). +pub(crate) const BATCH: usize = 1000; + +const SHORT: Duration = Duration::from_secs(30); +const BATCH_TIMEOUT: Duration = Duration::from_secs(120); +/// Computing a digest reloads the whole document server-side. +const DIGEST_TIMEOUT: Duration = Duration::from_secs(120); + +/// A non-2xx answer, decoded from canvas-cloud's `{"error","message"}` shape when possible. +#[derive(Debug, Clone)] +pub(crate) struct HttpFailure { + pub status: StatusCode, + pub code: String, + pub message: String, + pub body: Value, +} + +impl std::fmt::Display for HttpFailure { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + write!(f, "HTTP {} {}", self.status.as_u16(), self.code)?; + if !self.message.is_empty() { + write!(f, ": {}", self.message)?; + } + Ok(()) + } +} + +#[derive(Debug)] +pub(crate) enum Failure { + Http(HttpFailure), + /// The request never produced a response (connect, TLS, timeout, aborted body…). + Transport(String), +} + +impl std::fmt::Display for Failure { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + match self { + Self::Http(h) => h.fmt(f), + Self::Transport(e) => f.write_str(e), + } + } +} + +fn transport(e: &reqwest::Error) -> Failure { + // `{:#}`-style chain: reqwest's top-level message alone ("error sending request") hides + // the cause. + let mut msg = e.to_string(); + let mut source = std::error::Error::source(e); + while let Some(s) = source { + msg.push_str(": "); + msg.push_str(&s.to_string()); + source = s.source(); + } + Failure::Transport(msg) +} + +async fn failure(resp: reqwest::Response) -> HttpFailure { + let status = resp.status(); + let text = resp.text().await.unwrap_or_default(); + let body: Value = serde_json::from_str(&text).unwrap_or(Value::Null); + let code = body + .get("error") + .and_then(Value::as_str) + .unwrap_or_else(|| status.canonical_reason().unwrap_or("error")) + .to_owned(); + let message = body.get("message").and_then(Value::as_str).map_or_else( + || text.chars().take(300).collect::().trim().to_owned(), + ToOwned::to_owned, + ); + HttpFailure { + status, + code, + message, + body, + } +} + +/// `200` body of `POST /v1/apps/self/import`. +#[derive(Debug, Clone, Deserialize)] +pub(crate) struct Imported { + #[serde(default)] + pub status: Option, + pub updates_applied: u64, + pub members: u64, + pub invites: u64, + pub links: u64, + pub mentions: u64, + pub digest: String, +} + +#[derive(Serialize)] +struct BatchUser<'a> { + subject: String, + email: &'a str, + display_name: &'a str, + disabled: bool, + id: Uuid, +} + +/// Outcome of provisioning one user. +#[derive(Debug)] +pub(crate) struct Provisioned { + pub created: bool, + /// `Some(reason)` when canvas-cloud did not give this user its copper id. + pub error: Option, +} + +pub(crate) struct Remote<'a> { + client: &'a reqwest::Client, + base: String, + auth: String, +} + +impl<'a> Remote<'a> { + pub fn new( + client: &'a reqwest::Client, + base: &url::Url, + app_key: &str, + ) -> anyhow::Result { + anyhow::ensure!( + matches!(base.scheme(), "http" | "https"), + "canvas-cloud URL must be http(s), got {base}" + ); + anyhow::ensure!( + !app_key.trim().is_empty(), + "the canvas-cloud app key is empty" + ); + let mut base = base.clone(); + base.set_query(None); + base.set_fragment(None); + Ok(Self { + client, + base: base.as_str().trim_end_matches('/').to_owned(), + auth: format!("Bearer {}", app_key.trim()), + }) + } + + pub fn base(&self) -> &str { + &self.base + } + + fn url(&self, path: &str) -> String { + format!("{}{path}", self.base) + } + + fn get(&self, path: &str, timeout: Duration) -> reqwest::RequestBuilder { + self.client + .get(self.url(path)) + .header(AUTHORIZATION, &self.auth) + .header(ACCEPT, "application/json") + .timeout(timeout) + } + + /// `GET /v1/apps/self/stats` must answer 200 with JSON. + pub async fn preflight(&self) -> anyhow::Result { + let path = "/v1/apps/self/stats"; + let resp = self + .get(path, SHORT) + .send() + .await + .map_err(|e| anyhow::anyhow!("{}", transport(&e))) + .with_context(|| format!("canvas-cloud preflight: GET {} failed", self.url(path)))?; + if resp.status() != StatusCode::OK { + let f = failure(resp).await; + let hint = match f.status { + StatusCode::UNAUTHORIZED => " (the app key was rejected — revoked or wrong key?)", + StatusCode::NOT_FOUND => " (is this URL a Canvas Cloud? /v1/apps/self is missing)", + _ => "", + }; + anyhow::bail!( + "canvas-cloud preflight: GET {} answered {f}{hint}", + self.url(path) + ); + } + let stats: Value = resp + .json() + .await + .context("canvas-cloud preflight: /v1/apps/self/stats did not return JSON")?; + Ok(stats) + } + + /// Provisions `users` (subject = id hint = copper user id) through `users:batch`. + /// Returns one outcome per input user, in order. + pub async fn provision(&self, users: &[LegacyUser]) -> anyhow::Result> { + let path = "/v1/apps/self/users:batch"; + let mut out = Vec::with_capacity(users.len()); + for chunk in users.chunks(BATCH) { + let body = serde_json::json!({ + "users": chunk.iter().map(|u| BatchUser { + subject: subject(u.id), + email: &u.email, + display_name: &u.display_name, + disabled: u.disabled, + id: u.id, + }).collect::>(), + }); + let resp = self + .client + .post(self.url(path)) + .header(AUTHORIZATION, &self.auth) + .timeout(BATCH_TIMEOUT) + .json(&body) + .send() + .await + .map_err(|e| anyhow::anyhow!("{}", transport(&e))) + .with_context(|| format!("provisioning users: POST {}", self.url(path)))?; + if !resp.status().is_success() { + let f = failure(resp).await; + anyhow::bail!("provisioning users: POST {} answered {f}", self.url(path)); + } + let body: Value = resp + .json() + .await + .context("provisioning users: users:batch did not return JSON")?; + out.extend(match_results(chunk, &body)); + } + Ok(out) + } + + /// `GET /v1/apps/self/canvases/{id}/digest`: `Ok(None)` when the canvas does not exist. + pub async fn digest(&self, canvas_id: Uuid) -> Result, Failure> { + let resp = self + .get( + &format!("/v1/apps/self/canvases/{canvas_id}/digest"), + DIGEST_TIMEOUT, + ) + .send() + .await + .map_err(|e| transport(&e))?; + match resp.status() { + StatusCode::OK => { + let text = resp.text().await.map_err(|e| transport(&e))?; + parse_digest(&text).map(Some).ok_or_else(|| { + Failure::Transport(format!( + "digest endpoint returned no digest: {}", + text.chars().take(200).collect::() + )) + }) + } + StatusCode::NOT_FOUND => Ok(None), + _ => Err(Failure::Http(failure(resp).await)), + } + } + + /// `POST /v1/apps/self/import[?mode=replace]` with a streamed NDJSON body. + pub async fn import( + &self, + body: reqwest::Body, + replace: bool, + timeout: Duration, + ) -> Result { + let mut url = self.url("/v1/apps/self/import"); + if replace { + url.push_str("?mode=replace"); + } + let resp = self + .client + .post(url) + .header(AUTHORIZATION, &self.auth) + .header(CONTENT_TYPE, NDJSON) + .header(ACCEPT, "application/json") + .timeout(timeout) + .body(body) + .send() + .await + .map_err(|e| transport(&e))?; + if resp.status() != StatusCode::OK { + return Err(Failure::Http(failure(resp).await)); + } + let text = resp.text().await.map_err(|e| transport(&e))?; + serde_json::from_str(&text).map_err(|e| { + Failure::Transport(format!( + "import answered 200 with an unexpected body ({e}): {}", + text.chars().take(200).collect::() + )) + }) + } +} + +/// A digest from a digest response: `{"digest": "…"}`, a JSON string, or bare hex. +pub(crate) fn parse_digest(text: &str) -> Option { + match serde_json::from_str::(text) { + Ok(Value::Object(m)) => m.get("digest").and_then(Value::as_str).and_then(normalize), + Ok(Value::String(s)) => normalize(&s), + _ => normalize(text), + } +} + +/// The digest a `409 canvas_exists` body carries, if any. +pub(crate) fn conflict_digest(body: &Value) -> Option { + body.get("digest") + .and_then(Value::as_str) + .and_then(normalize) +} + +/// Lower-case 64-hex-digit SHA-256, or `None`. +fn normalize(raw: &str) -> Option { + let d = raw.trim().to_ascii_lowercase(); + (d.len() == 64 && d.bytes().all(|b| b.is_ascii_hexdigit())).then_some(d) +} + +/// Matches `users:batch` per-item results to the users sent. Accepts `{"results":[…]}`, +/// `{"users":[…]}` or a bare array; items carry `subject` and `user_id` (or `id`), or an +/// `error`/`message`. Items without a `subject` are matched by position. +fn match_results(chunk: &[LegacyUser], body: &Value) -> Vec { + let items: &[Value] = body + .get("results") + .or_else(|| body.get("users")) + .unwrap_or(body) + .as_array() + .map_or(&[], Vec::as_slice); + let by_subject: std::collections::HashMap<&str, &Value> = items + .iter() + .filter_map(|v| Some((v.get("subject")?.as_str()?, v))) + .collect(); + chunk + .iter() + .enumerate() + .map(|(i, u)| { + let wanted = subject(u.id); + let item = by_subject + .get(wanted.as_str()) + .copied() + .or_else(|| items.get(i).filter(|v| v.get("subject").is_none())); + let Some(item) = item else { + return Provisioned { + created: false, + error: Some("missing from the users:batch response".into()), + }; + }; + if let Some(code) = item.get("error").and_then(Value::as_str) { + let message = item.get("message").and_then(Value::as_str).unwrap_or(""); + return Provisioned { + created: false, + error: Some(format!("{code}: {message}").trim_end_matches(": ").to_owned()), + }; + } + let got = item + .get("user_id") + .or_else(|| item.get("id")) + .and_then(Value::as_str) + .and_then(|s| Uuid::parse_str(s).ok()); + let created = item.get("created").and_then(Value::as_bool).unwrap_or(false); + match got { + Some(id) if id == u.id => Provisioned { + created, + error: None, + }, + Some(id) => Provisioned { + created, + error: Some(format!( + "canvas-cloud mapped it to user {id} instead of {} (email already used by another canvas-cloud account?)", + u.id + )), + }, + None => Provisioned { + created, + error: Some("users:batch result has no user_id".into()), + }, + } + }) + .collect() +} + +#[cfg(test)] +mod tests { + use super::*; + + fn user(n: u128) -> LegacyUser { + LegacyUser { + id: Uuid::from_u128(n), + email: format!("u{n}@example.com"), + display_name: String::new(), + disabled: false, + } + } + + #[test] + fn digests_parse_leniently_but_strictly_hex() { + let d = "AB".repeat(32); + assert_eq!( + parse_digest(&format!(r#"{{"digest":"{d}"}}"#)), + Some("ab".repeat(32)) + ); + assert_eq!(parse_digest(&format!(r#""{d}""#)), Some("ab".repeat(32))); + assert_eq!(parse_digest(&format!("{d}\n")), Some("ab".repeat(32))); + assert_eq!(parse_digest(r#"{"digest":"xyz"}"#), None); + assert_eq!(parse_digest("{}"), None); + assert_eq!( + conflict_digest(&serde_json::json!({"error":"canvas_exists","digest": d})), + Some("ab".repeat(32)) + ); + assert_eq!( + conflict_digest(&serde_json::json!({"error":"canvas_exists"})), + None + ); + } + + #[test] + fn batch_results_match_by_subject_or_position() { + let users = [user(1), user(2), user(3), user(4)]; + let body = serde_json::json!({"results": [ + {"subject": subject(users[1].id), "user_id": users[1].id, "created": true}, + {"subject": subject(users[0].id), "user_id": users[0].id, "created": false}, + {"subject": subject(users[2].id), "user_id": Uuid::from_u128(99)}, + ]}); + let out = match_results(&users, &body); + assert!(out[0].error.is_none() && !out[0].created); + assert!(out[1].error.is_none() && out[1].created); + assert!(out[2].error.as_deref().unwrap().contains("instead of")); + assert!(out[3].error.as_deref().unwrap().contains("missing")); + + let positional = serde_json::json!([ + {"user_id": users[0].id}, + {"error": "email_taken", "message": "taken"}, + ]); + let out = match_results(&users[..2], &positional); + assert!(out[0].error.is_none()); + assert_eq!(out[1].error.as_deref(), Some("email_taken: taken")); + } +} diff --git a/crates/canvas-migrate/src/report.rs b/crates/canvas-migrate/src/report.rs new file mode 100644 index 0000000..436e24c --- /dev/null +++ b/crates/canvas-migrate/src/report.rs @@ -0,0 +1,372 @@ +//! What a migration run did: one row per canvas plus totals; a table (`Display`) or JSON +//! (`Serialize`). + +use std::fmt::{self, Write as _}; + +use serde::Serialize; +use time::OffsetDateTime; +use uuid::Uuid; + +/// Outcome of one canvas. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize)] +#[serde(rename_all = "snake_case")] +pub enum Status { + /// Imported (created) and verified. + Migrated, + /// Re-imported over an earlier import (`--replace`) and verified. + Replaced, + /// Nothing to do: the ledger (or canvas-cloud, matching the ledger) shows it migrated. + Already, + /// `--dry-run`: read, decrypted and encoded, not sent. + DryRun, + /// Not migrated; see `error`. Legacy rows are untouched. + Failed, +} + +impl Status { + pub fn as_str(self) -> &'static str { + match self { + Self::Migrated => "migrated", + Self::Replaced => "replaced", + Self::Already => "already", + Self::DryRun => "dry-run", + Self::Failed => "failed", + } + } +} + +impl fmt::Display for Status { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + f.write_str(self.as_str()) + } +} + +/// One canvas row. +#[derive(Debug, Clone, Serialize)] +pub struct CanvasReport { + pub id: Uuid, + pub name: String, + pub kind: String, + pub owner: Option, + pub status: Status, + /// Yjs updates sent (a snapshot counts as one, sent first). + pub updates: u64, + /// Plaintext Yjs bytes sent. + pub bytes: u64, + pub snapshot: bool, + pub members: u64, + pub invites: u64, + pub links: u64, + pub mentions: u64, + /// Verified canvas-cloud digest (CONTRACT §6.3), when known. + pub digest: Option, + /// Why a row is `already`, or other context. + pub note: Option, + pub error: Option, + pub warnings: Vec, +} + +impl CanvasReport { + pub(crate) fn new(id: Uuid, name: &str, kind: &str, owner: Option) -> Self { + Self { + id, + name: name.to_owned(), + kind: kind.to_owned(), + owner, + status: Status::Failed, + updates: 0, + bytes: 0, + snapshot: false, + members: 0, + invites: 0, + links: 0, + mentions: 0, + digest: None, + note: None, + error: None, + warnings: Vec::new(), + } + } +} + +/// A user canvas-cloud could not provision with its copper id. +#[derive(Debug, Clone, Serialize)] +pub struct UserFailure { + pub user_id: Uuid, + pub email: String, + pub error: String, +} + +/// User provisioning (skipped on a dry run). +#[derive(Debug, Clone, Default, Serialize)] +pub struct UsersReport { + pub total: usize, + pub provisioned: usize, + pub created: usize, + pub failed: Vec, +} + +/// Sums over the canvas rows. +#[derive(Debug, Clone, Default, Serialize)] +pub struct Totals { + pub canvases: usize, + pub migrated: usize, + pub replaced: usize, + pub already: usize, + pub dry_run: usize, + pub failed: usize, + pub updates: u64, + pub bytes: u64, + pub members: u64, + pub invites: u64, + pub links: u64, + pub mentions: u64, +} + +/// The result of [`crate::run`]. +/// +/// `Display` renders the per-canvas table, or pretty JSON when the run was started with +/// `MigrateOptions::json` (so a CLI can always just print it); `Serialize` is the JSON form. +#[derive(Debug, Clone, Serialize)] +#[allow(clippy::struct_excessive_bools)] // run flags + result, not a state machine +pub struct MigrateReport { + pub canvas_cloud: String, + pub dry_run: bool, + pub replace: bool, + #[serde(with = "time::serde::rfc3339")] + pub started_at: OffsetDateTime, + #[serde(with = "time::serde::rfc3339")] + pub finished_at: OffsetDateTime, + pub users: UsersReport, + pub canvases: Vec, + pub totals: Totals, + /// A failure that stopped the run before (or instead of) migrating canvases. + pub error: Option, + pub ok: bool, + #[serde(skip)] + pub(crate) json: bool, +} + +impl MigrateReport { + pub(crate) fn new(canvas_cloud: &str, dry_run: bool, replace: bool, json: bool) -> Self { + let now = OffsetDateTime::now_utc(); + Self { + canvas_cloud: canvas_cloud.to_owned(), + dry_run, + replace, + started_at: now, + finished_at: now, + users: UsersReport::default(), + canvases: Vec::new(), + totals: Totals::default(), + error: None, + ok: false, + json, + } + } + + /// `true` when every user was provisioned and no canvas failed (exit status 0). + pub fn ok(&self) -> bool { + self.error.is_none() + && self.users.failed.is_empty() + && self.canvases.iter().all(|c| c.status != Status::Failed) + } + + pub(crate) fn finish(&mut self) { + let mut t = Totals { + canvases: self.canvases.len(), + ..Totals::default() + }; + for c in &self.canvases { + match c.status { + Status::Migrated => t.migrated += 1, + Status::Replaced => t.replaced += 1, + Status::Already => t.already += 1, + Status::DryRun => t.dry_run += 1, + Status::Failed => t.failed += 1, + } + t.updates += c.updates; + t.bytes += c.bytes; + t.members += c.members; + t.invites += c.invites; + t.links += c.links; + t.mentions += c.mentions; + } + self.totals = t; + self.finished_at = OffsetDateTime::now_utc(); + self.ok = self.ok(); + } + + /// The table, regardless of the `json` option. + pub fn table(&self) -> String { + let mut out = String::new(); + let _ = self.write_table(&mut out); + out + } + + fn write_table(&self, out: &mut String) -> fmt::Result { + writeln!( + out, + "canvas migrate → {}{}{}", + self.canvas_cloud, + if self.dry_run { + " (dry run: nothing sent)" + } else { + "" + }, + if self.replace { " (replace)" } else { "" }, + )?; + if self.dry_run { + writeln!( + out, + "users: {} (not provisioned on a dry run)", + self.users.total + )?; + } else { + writeln!( + out, + "users: {} provisioned ({} new), {} failed", + self.users.provisioned, + self.users.created, + self.users.failed.len() + )?; + } + for u in &self.users.failed { + writeln!(out, " user {} <{}>: {}", u.user_id, u.email, u.error)?; + } + if !self.canvases.is_empty() { + writeln!( + out, + "{:<36} {:<24} {:<8} {:>7} {:>9} {:>7} {:<8} {:<12} ERROR", + "ID", "NAME", "KIND", "UPDATES", "BYTES", "MEMBERS", "STATUS", "DIGEST" + )?; + } + for c in &self.canvases { + let digest = c.digest.as_deref().map_or("-", |d| &d[..d.len().min(12)]); + let detail = match (&c.error, &c.note) { + (Some(e), _) => e.clone(), + (None, Some(n)) => format!("({n})"), + (None, None) => String::new(), + }; + writeln!( + out, + "{:<36} {:<24} {:<8} {:>7} {:>9} {:>7} {:<8} {:<12} {}", + c.id, + clip(&c.name, 24), + c.kind, + c.updates, + human_bytes(c.bytes), + c.members, + c.status.as_str(), + digest, + detail + )?; + for w in &c.warnings { + writeln!(out, " warning: {w}")?; + } + } + let t = &self.totals; + writeln!( + out, + "total: {} canvases — {} migrated, {} replaced, {} already, {} dry-run, {} failed; \ + {} updates, {}, {} members, {} invites, {} links, {} mentions", + t.canvases, + t.migrated, + t.replaced, + t.already, + t.dry_run, + t.failed, + t.updates, + human_bytes(t.bytes), + t.members, + t.invites, + t.links, + t.mentions + )?; + if let Some(e) = &self.error { + writeln!(out, "error: {e}")?; + } + write!(out, "{}", if self.ok() { "ok" } else { "FAILED" }) + } +} + +impl fmt::Display for MigrateReport { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + if self.json { + let text = serde_json::to_string_pretty(self).map_err(|_| fmt::Error)?; + f.write_str(&text) + } else { + f.write_str(&self.table()) + } + } +} + +fn clip(s: &str, n: usize) -> String { + let one_line: String = s + .chars() + .map(|c| if c.is_control() { ' ' } else { c }) + .collect(); + if one_line.chars().count() <= n { + one_line + } else { + let mut out: String = one_line.chars().take(n - 1).collect(); + out.push('…'); + out + } +} + +#[allow(clippy::cast_precision_loss)] // display only +fn human_bytes(n: u64) -> String { + const UNITS: [&str; 4] = ["KiB", "MiB", "GiB", "TiB"]; + if n < 1024 { + return format!("{n} B"); + } + let mut v = n as f64 / 1024.0; + let mut unit = 0; + while v >= 1024.0 && unit < UNITS.len() - 1 { + v /= 1024.0; + unit += 1; + } + format!("{v:.1} {}", UNITS[unit]) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn bytes_and_clipping() { + assert_eq!(human_bytes(10), "10 B"); + assert_eq!(human_bytes(1536), "1.5 KiB"); + assert_eq!(human_bytes(88 * 1024 * 1024), "88.0 MiB"); + assert_eq!(clip("short", 24), "short"); + assert_eq!(clip("a\nb", 24), "a b"); + assert_eq!(clip(&"x".repeat(30), 5), "xxxx…"); + } + + #[test] + fn ok_and_display() { + let mut r = MigrateReport::new("http://127.0.0.1:8480", false, false, false); + let mut c = CanvasReport::new(Uuid::nil(), "Roadmap", "shared", None); + c.status = Status::Migrated; + c.digest = Some("ab".repeat(32)); + c.updates = 3; + r.canvases.push(c); + r.finish(); + assert!(r.ok()); + let text = r.to_string(); + assert!(text.contains("Roadmap"), "{text}"); + assert!(text.contains("migrated"), "{text}"); + assert!(text.contains("abababababab "), "{text}"); + assert!(text.ends_with("ok"), "{text}"); + r.canvases[0].status = Status::Failed; + r.finish(); + assert!(!r.ok()); + assert_eq!(r.totals.failed, 1); + r.json = true; + let v: serde_json::Value = serde_json::from_str(&r.to_string()).unwrap(); + assert_eq!(v["canvases"][0]["status"], "failed"); + assert_eq!(v["ok"], false); + assert!(v.get("json").is_none()); + } +} diff --git a/crates/canvas-migrate/src/wire.rs b/crates/canvas-migrate/src/wire.rs new file mode 100644 index 0000000..d76ca1f --- /dev/null +++ b/crates/canvas-migrate/src/wire.rs @@ -0,0 +1,136 @@ +//! Import format v1 lines (`application/x-ndjson`, one JSON object per line). +//! +//! Field order follows the contract so a human reading a captured stream sees `type` first. +//! Subjects are copper-cloud user ids rendered as hyphenated lower-case UUID strings — the +//! same subjects the link provisions, so canvas-cloud user ids equal copper user ids. + +use serde::Serialize; +use time::OffsetDateTime; +use uuid::Uuid; + +/// Format version written on the `canvas` line. +pub const VERSION: u32 = 1; + +/// Content type of an import body. +pub const CONTENT_TYPE: &str = "application/x-ndjson"; + +/// One line of an import body. +#[derive(Debug, Serialize)] +#[serde(tag = "type", rename_all = "lowercase")] +pub enum Line<'a> { + Canvas { + version: u32, + id: Uuid, + name: &'a str, + kind: &'a str, + owner: String, + #[serde(with = "time::serde::rfc3339")] + created_at: OffsetDateTime, + #[serde(with = "time::serde::rfc3339")] + updated_at: OffsetDateTime, + }, + Member { + subject: String, + role: &'a str, + #[serde(with = "time::serde::rfc3339")] + added_at: OffsetDateTime, + }, + Invite { + id: Uuid, + email: &'a str, + invited_by: Option, + status: &'a str, + #[serde(with = "time::serde::rfc3339")] + created_at: OffsetDateTime, + #[serde(with = "time::serde::rfc3339::option")] + accepted_at: Option, + #[serde(with = "time::serde::rfc3339::option")] + nudged_at: Option, + token_sha256_hex: &'a str, + }, + Link { + id: Uuid, + token_sha256_hex: String, + role: &'a str, + created_by: Option, + #[serde(with = "time::serde::rfc3339")] + created_at: OffsetDateTime, + uses: i64, + }, + Mention { + id: Uuid, + message_id: &'a str, + from: String, + to: String, + excerpt: &'a str, + #[serde(with = "time::serde::rfc3339")] + created_at: OffsetDateTime, + #[serde(with = "time::serde::rfc3339::option")] + read_at: Option, + }, + Update { + b64: String, + }, + End { + updates: u64, + members: u64, + invites: u64, + links: u64, + mentions: u64, + }, +} + +impl Line<'_> { + /// The line as bytes, newline-terminated. + pub fn to_bytes(&self) -> anyhow::Result { + let mut out = serde_json::to_vec(self)?; + out.push(b'\n'); + Ok(out.into()) + } +} + +/// A user id as an import/app subject. +pub fn subject(id: Uuid) -> String { + id.hyphenated().to_string() +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn lines_have_type_first_and_rfc3339_times() { + let at = time::macros::datetime!(2026-10-19 18:00:00.25 UTC); + let line = Line::Invite { + id: Uuid::nil(), + email: "a@example.com", + invited_by: None, + status: "pending", + created_at: at, + accepted_at: None, + nudged_at: Some(at), + token_sha256_hex: "ab", + }; + let text = String::from_utf8(line.to_bytes().unwrap().to_vec()).unwrap(); + assert!(text.starts_with(r#"{"type":"invite","id":"00000000-0000-0000-0000-000000000000""#)); + assert!(text.contains(r#""invited_by":null"#)); + assert!(text.contains(r#""accepted_at":null"#)); + assert!(text.contains(r#""nudged_at":"2026-10-19T18:00:00.25Z""#)); + assert!(text.ends_with("}\n")); + let end = Line::End { + updates: 3, + members: 2, + invites: 1, + links: 0, + mentions: 4, + }; + assert_eq!( + end.to_bytes().unwrap(), + &b"{\"type\":\"end\",\"updates\":3,\"members\":2,\"invites\":1,\"links\":0,\"mentions\":4}\n"[..] + ); + assert_eq!( + subject(Uuid::from_u128(0xA1)), + "00000000-0000-0000-0000-0000000000a1" + ); + } +} diff --git a/crates/canvas-migrate/tests/common/digest.rs b/crates/canvas-migrate/tests/common/digest.rs new file mode 100644 index 0000000..9d05526 --- /dev/null +++ b/crates/canvas-migrate/tests/common/digest.rs @@ -0,0 +1,189 @@ +//! Canonical document digest — reference implementation of CONTRACT §6.3. +//! +//! Canvas Cloud's `crates/apps` must compute byte-identical digests (the migration compares +//! the import response's digest with `GET /v1/apps/self/canvases/{id}/digest`, and the ledger +//! keeps it as rollback evidence). The algorithm: +//! +//! ```text +//! digest(doc) = lower-case hex( SHA-256( canonical_json(doc) as UTF-8 bytes ) ) +//! ``` +//! +//! `canonical_json(doc)` is one JSON object with a member per **root type** of the document +//! (`txn.root_refs()`), written with no whitespace at all: +//! +//! 1. **Roots.** For every root `(name, value)`: +//! * a typed root (`get_or_insert_map/array/text/xml_*` was called on this `Doc`) renders by +//! its type; +//! * an *undefined* root (the `Doc` only received updates — Yjs updates carry no root +//! types) is inferred from its content, so a server that never touched the root and a +//! server that did agree: map entries present → **map**; else a non-empty +//! `TextRef::get_string` → **text**; else list content (`ArrayRef::len > 0`) → **array**; +//! else **empty**. +//! * roots whose rendering is empty (`{}`, `[]` or `""`) are **omitted** — a deleted-only +//! root may or may not survive snapshot compaction, and an empty root means nothing to a +//! client. +//! * members are ordered by root name (byte-wise UTF-8). +//! 2. **Values.** +//! * `Y.Map` and `Any::Map` → object; members sorted by key (byte-wise UTF-8); deleted +//! entries are not present. +//! * `Y.Array` and `Any::Array` → array, document order. +//! * `Y.Text` → its plain string (`get_string`: formatting attributes and embeds ignored). +//! * `Y.XmlElement`, `Y.XmlFragment`, `Y.XmlText` → their `get_string` (XML string). +//! * sub-document → `{"guid":""}`. +//! * `Any::Null`, `Any::Undefined` → `null`; `Any::Bool` → `true`/`false`. +//! * `Any::Number` (both `Int(i64)` and `Float(f64)`) → the value as `f64`, rendered by +//! `serde_json` (`serde_json::Number::from_f64(v).to_string()`: shortest round-trip, +//! always with a fraction or exponent — `1` → `1.0`, `0.5` → `0.5`, `1e21` → `1e21`); +//! NaN/±∞ → `null`. +//! * `Any::String` → JSON string, escaped exactly as `serde_json::to_string(&str)` does. +//! * `Any::Buffer` → JSON string of standard base64 **with** padding (RFC 4648 §4). +//! * anything else (weak links) → `null`. +//! +//! Property (unit-tested in `tests/digest.rs`): the same set of updates yields the same +//! digest regardless of update order, of snapshot compaction (`encode_state_as_update_v1` +//! of the merged state), and of whether the reader typed its roots. + +#![allow(dead_code)] + +use std::fmt::Write as _; + +use base64::Engine as _; +use sha2::{Digest as _, Sha256}; +use yrs::types::xml::XmlTextRef; +use yrs::types::{GetString as _, ToJson as _}; +use yrs::{Any, Array as _, ArrayRef, Doc, Map as _, MapRef, Out, ReadTxn, TextRef, Transact as _}; + +/// Hex SHA-256 of [`canonical_json`]. +pub fn digest(doc: &Doc) -> String { + let txn = doc.transact(); + digest_txn(&txn) +} + +/// [`digest`] inside an existing transaction. +pub fn digest_txn(txn: &T) -> String { + hex::encode(Sha256::digest(canonical_json(txn).as_bytes())) +} + +/// Canonical JSON rendering of the whole document (see the module docs). +pub fn canonical_json(txn: &T) -> String { + let mut roots: Vec<(String, String)> = txn + .root_refs() + .map(|(name, value)| (name.to_owned(), render_root(txn, &value))) + .filter(|(_, v)| !matches!(v.as_str(), "{}" | "[]" | "\"\"")) + .collect(); + roots.sort_by(|a, b| a.0.as_bytes().cmp(b.0.as_bytes())); + let mut out = String::from("{"); + for (i, (name, value)) in roots.iter().enumerate() { + if i > 0 { + out.push(','); + } + out.push_str(&json_string(name)); + out.push(':'); + out.push_str(value); + } + out.push('}'); + out +} + +fn render_root(txn: &T, value: &Out) -> String { + let Out::UndefinedRef(ptr) = value else { + return render_out(txn, value); + }; + let ptr = *ptr; + let map = MapRef::from(ptr); + if map.len(txn) > 0 { + return render_map(txn, &map); + } + let text = TextRef::from(ptr).get_string(txn); + if !text.is_empty() { + return json_string(&text); + } + let array = ArrayRef::from(ptr); + if array.len(txn) > 0 { + return render_array(txn, &array); + } + "{}".to_owned() +} + +fn render_out(txn: &T, value: &Out) -> String { + #[allow(unreachable_patterns)] // `Out::YWeakLink` exists only with yrs' `weak` feature + match value { + Out::Any(any) => render_any(any), + Out::YMap(map) => render_map(txn, map), + Out::YArray(array) => render_array(txn, array), + Out::YText(text) => json_string(&text.get_string(txn)), + Out::YXmlElement(x) => json_string(&x.get_string(txn)), + Out::YXmlFragment(x) => json_string(&x.get_string(txn)), + Out::YXmlText(x) => json_string(&XmlTextRef::get_string(x, txn)), + Out::YDoc(doc) => { + let guid = doc.guid(); + format!("{{\"guid\":{}}}", json_string(guid.as_ref())) + } + Out::UndefinedRef(_) => render_root(txn, value), + _ => "null".to_owned(), + } +} + +fn render_map(txn: &T, map: &MapRef) -> String { + let mut entries: Vec<(String, String)> = map + .iter(txn) + .map(|(k, v)| (k.to_owned(), render_out(txn, &v))) + .collect(); + object(&mut entries) +} + +fn render_array(txn: &T, array: &ArrayRef) -> String { + let items: Vec = array.iter(txn).map(|v| render_out(txn, &v)).collect(); + format!("[{}]", items.join(",")) +} + +fn render_any(any: &Any) -> String { + match any { + Any::Null | Any::Undefined => "null".to_owned(), + Any::Bool(b) => b.to_string(), + Any::Number(n) => { + #[allow(clippy::cast_precision_loss)] // the contract normalises every number to f64 + let v = match *n { + yrs::Number::Int(i) => i as f64, + yrs::Number::Float(f) => f, + }; + serde_json::Number::from_f64(v).map_or_else(|| "null".to_owned(), |n| n.to_string()) + } + Any::String(s) => json_string(s), + Any::Buffer(b) => json_string(&base64::engine::general_purpose::STANDARD.encode(b)), + Any::Array(items) => { + let items: Vec = items.iter().map(render_any).collect(); + format!("[{}]", items.join(",")) + } + Any::Map(m) => { + let mut entries: Vec<(String, String)> = + m.iter().map(|(k, v)| (k.clone(), render_any(v))).collect(); + object(&mut entries) + } + } +} + +fn object(entries: &mut [(String, String)]) -> String { + entries.sort_by(|a, b| a.0.as_bytes().cmp(b.0.as_bytes())); + let mut out = String::from("{"); + for (i, (k, v)) in entries.iter().enumerate() { + if i > 0 { + out.push(','); + } + let _ = write!(out, "{}:{v}", json_string(k)); + } + out.push('}'); + out +} + +fn json_string(s: &str) -> String { + serde_json::to_string(s).expect("a str always serialises") +} + +/// `yrs`' own `to_json` of the document — only for debugging test failures. +pub fn debug_json(doc: &Doc) -> String { + let txn = doc.transact(); + let mut out = String::new(); + doc.to_json(&txn).to_json(&mut out); + out +} diff --git a/crates/canvas-migrate/tests/common/mock.rs b/crates/canvas-migrate/tests/common/mock.rs new file mode 100644 index 0000000..4b2a887 --- /dev/null +++ b/crates/canvas-migrate/tests/common/mock.rs @@ -0,0 +1,580 @@ +//! An in-process mock Canvas Cloud implementing just the app endpoints the migration uses: +//! `GET /v1/apps/self/stats`, `POST /v1/apps/self/users:batch`, `POST /v1/apps/self/import` +//! and `GET /v1/apps/self/canvases/{id}/digest`. +//! +//! The import handler really streams and parses the NDJSON body line by line, validates the +//! order and the subjects, applies every update to a fresh `yrs::Doc`, stores one compacted +//! state (like canvas-cloud's single snapshot) and answers with the §6.3 digest. The digest +//! endpoint reloads the stored state into a fresh `Doc`, so "import digest == stored digest" +//! also exercises compaction invariance. + +#![allow(dead_code, clippy::missing_panics_doc, clippy::too_many_lines)] + +use std::collections::{HashMap, HashSet}; +use std::net::SocketAddr; +use std::sync::{Arc, Mutex}; + +use axum::body::Body; +use axum::extract::{Path, Query, State}; +use axum::http::{HeaderMap, StatusCode}; +use axum::response::{IntoResponse, Response}; +use axum::routing::{get, post}; +use axum::{Json, Router}; +use base64::Engine as _; +use futures::StreamExt as _; +use serde_json::{json, Value}; +use uuid::Uuid; +use yrs::updates::decoder::Decode as _; +use yrs::{Doc, ReadTxn as _, StateVector, Transact as _, Update}; + +use super::digest; + +pub const APP_KEY: &str = "cva_mock_app_key"; + +/// One import request as the mock parsed it. +#[derive(Debug, Clone, Default)] +pub struct Received { + pub canvas: Value, + pub members: Vec, + pub invites: Vec, + pub links: Vec, + pub mentions: Vec, + pub updates: usize, + pub end: Option, + pub mode: String, + /// Status the mock answered. + pub status: u16, + pub error: Option, +} + +/// A stored canvas. +#[derive(Debug, Clone)] +pub struct Stored { + pub state: Vec, + pub owner: String, + pub kind: String, + pub imported: bool, + pub empty: bool, + pub received: Received, +} + +#[derive(Debug)] +pub struct MockState { + /// subject → user id + pub users: HashMap, + /// Every user payload `users:batch` received. + pub user_payloads: Vec, + pub canvases: HashMap, + pub imports: Vec, + pub stats_gets: usize, + pub batch_posts: usize, + pub import_posts: usize, + pub digest_gets: usize, + // Knobs. + pub allow_import: bool, + /// `GET …/digest` lies for these canvases. + pub corrupt_digest: HashSet, + /// `users:batch` maps these subjects to another user id. + pub remap_user: HashMap, + /// `users:batch` claims success for these subjects but does not keep them. + pub forget_subjects: HashSet, +} + +impl Default for MockState { + fn default() -> Self { + Self { + users: HashMap::new(), + user_payloads: Vec::new(), + canvases: HashMap::new(), + imports: Vec::new(), + stats_gets: 0, + batch_posts: 0, + import_posts: 0, + digest_gets: 0, + allow_import: true, + corrupt_digest: HashSet::new(), + remap_user: HashMap::new(), + forget_subjects: HashSet::new(), + } + } +} + +type Shared = Arc>; + +pub struct Mock { + pub addr: SocketAddr, + pub url: url::Url, + pub state: Shared, +} + +impl Mock { + pub async fn start() -> Self { + let state: Shared = Arc::new(Mutex::new(MockState::default())); + let app = Router::new() + .route("/v1/apps/self/stats", get(stats)) + .route("/v1/apps/self/users:batch", post(users_batch)) + .route("/v1/apps/self/import", post(import)) + .route("/v1/apps/self/canvases/{id}/digest", get(canvas_digest)) + .with_state(state.clone()); + let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap(); + let addr = listener.local_addr().unwrap(); + tokio::spawn(async move { + let _ = axum::serve(listener, app).await; + }); + Self { + addr, + url: url::Url::parse(&format!("http://{addr}/")).unwrap(), + state, + } + } + + pub fn with(&self, f: impl FnOnce(&mut MockState) -> R) -> R { + f(&mut self.state.lock().unwrap()) + } + + /// Gives `owner` an existing Personal canvas (as canvas-cloud's `ensure_personal_canvas` + /// would), empty or with content. + pub fn seed_personal(&self, owner: Uuid, with_content: bool) -> Uuid { + let doc = Doc::new(); + if with_content { + let meta = doc.get_or_insert_map("meta"); + let mut txn = doc.transact_mut(); + yrs::Map::insert(&meta, &mut txn, "name", "Personal (canvas-cloud)"); + } + let state = doc + .transact() + .encode_state_as_update_v1(&StateVector::default()); + let id = Uuid::new_v4(); + self.with(|s| { + s.canvases.insert( + id, + Stored { + state, + owner: owner.to_string(), + kind: "personal".into(), + imported: false, + empty: !with_content, + received: Received::default(), + }, + ); + }); + id + } + + /// The digest the mock stores for `id`, recomputed from its compacted state. + pub fn stored_digest(&self, id: Uuid) -> Option { + self.with(|s| s.canvases.get(&id).map(|c| state_digest(&c.state))) + } + + pub fn received(&self, id: Uuid) -> Option { + self.with(|s| s.canvases.get(&id).map(|c| c.received.clone())) + } +} + +/// Digest of a stored (compacted) state, reloaded into a fresh document. +pub fn state_digest(state: &[u8]) -> String { + let doc = Doc::new(); + { + let mut txn = doc.transact_mut(); + txn.apply_update(Update::decode_v1(state).unwrap()).unwrap(); + } + digest::digest(&doc) +} + +fn err(status: StatusCode, code: &str, message: &str) -> Response { + (status, Json(json!({"error": code, "message": message}))).into_response() +} + +fn authorized(headers: &HeaderMap) -> bool { + headers + .get("authorization") + .and_then(|v| v.to_str().ok()) + .is_some_and(|v| v == format!("Bearer {APP_KEY}")) + && headers.get("x-canvas-user").is_none() +} + +async fn stats(State(state): State, headers: HeaderMap) -> Response { + if !authorized(&headers) { + return err(StatusCode::UNAUTHORIZED, "unauthorized", "bad app key"); + } + let mut s = state.lock().unwrap(); + s.stats_gets += 1; + Json(json!({ + "users": s.users.len(), + "canvases": s.canvases.len(), + "rooms": {"rooms": 0, "peers": 0}, + })) + .into_response() +} + +async fn users_batch( + State(state): State, + headers: HeaderMap, + Json(body): Json, +) -> Response { + if !authorized(&headers) { + return err(StatusCode::UNAUTHORIZED, "unauthorized", "bad app key"); + } + let mut s = state.lock().unwrap(); + s.batch_posts += 1; + let users = body["users"].as_array().cloned().unwrap_or_default(); + if users.len() > 1000 { + return err( + StatusCode::BAD_REQUEST, + "bad_request", + "at most 1000 users per batch", + ); + } + let mut results = Vec::new(); + for u in users { + s.user_payloads.push(u.clone()); + let subject = u["subject"].as_str().unwrap_or_default().to_owned(); + let hint = u["id"].as_str().and_then(|i| Uuid::parse_str(i).ok()); + let created = !s.users.contains_key(&subject); + let user_id = s + .remap_user + .get(&subject) + .copied() + .or_else(|| s.users.get(&subject).copied()) + .or(hint) + .unwrap_or_else(Uuid::now_v7); + if !s.forget_subjects.contains(&subject) { + s.users.insert(subject.clone(), user_id); + } + results.push(json!({ + "subject": subject, + "user_id": user_id, + "email": u["email"], + "display_name": u["display_name"], + "disabled": u["disabled"], + "created": created, + })); + } + Json(json!({ "results": results })).into_response() +} + +#[derive(serde::Deserialize)] +struct ImportQuery { + mode: Option, +} + +/// Why an import is refused (status, code, message). +type Refusal = (StatusCode, &'static str, String); + +struct Importer { + doc: Doc, + got: Received, + users: HashSet, + phase: u8, + failure: Option, +} + +impl Importer { + fn subject(&mut self, v: &Value) -> Result<(), Refusal> { + match v { + Value::Null => Ok(()), + Value::String(s) if self.users.contains(s) => Ok(()), + other => Err(( + StatusCode::UNPROCESSABLE_ENTITY, + "unknown_subject", + format!("subject {other} is not provisioned for this app"), + )), + } + } + + fn line(&mut self, raw: &[u8]) -> Result<(), Refusal> { + let bad = |m: String| (StatusCode::BAD_REQUEST, "bad_request", m); + let v: Value = serde_json::from_slice(raw).map_err(|e| bad(format!("bad line: {e}")))?; + let kind = v["type"].as_str().unwrap_or_default().to_owned(); + let order = match kind.as_str() { + "canvas" => 0, + "member" => 1, + "invite" => 2, + "link" => 3, + "mention" => 4, + "update" => 5, + "end" => 6, + other => return Err(bad(format!("unknown line type {other:?}"))), + }; + if self.got.end.is_some() + || (self.phase == 0 && order != 0) + || order < self.phase + || (order == 0 && self.got.canvas != Value::Null) + { + return Err(bad(format!("line {kind:?} out of order"))); + } + self.phase = order.max(1); + match kind.as_str() { + "canvas" => { + if v["version"] != 1 { + return Err(bad("unsupported version".into())); + } + Uuid::parse_str(v["id"].as_str().unwrap_or_default()) + .map_err(|_| bad("bad canvas id".into()))?; + self.subject(&v["owner"])?; + self.got.canvas = v; + } + "member" => { + self.subject(&v["subject"])?; + self.got.members.push(v); + } + "invite" => { + self.subject(&v["invited_by"])?; + self.got.invites.push(v); + } + "link" => { + self.subject(&v["created_by"])?; + let hex = v["token_sha256_hex"].as_str().unwrap_or_default(); + if hex.len() != 64 { + return Err(bad("token_sha256_hex must be 64 hex digits".into())); + } + self.got.links.push(v); + } + "mention" => { + self.subject(&v["from"])?; + self.subject(&v["to"])?; + self.got.mentions.push(v); + } + "update" => { + let bytes = base64::engine::general_purpose::STANDARD + .decode(v["b64"].as_str().unwrap_or_default()) + .map_err(|e| bad(format!("bad base64: {e}")))?; + let update = Update::decode_v1(&bytes).map_err(|e| { + ( + StatusCode::UNPROCESSABLE_ENTITY, + "bad_update", + format!("undecodable update: {e}"), + ) + })?; + self.doc.transact_mut().apply_update(update).map_err(|e| { + ( + StatusCode::UNPROCESSABLE_ENTITY, + "bad_update", + format!("update does not apply: {e}"), + ) + })?; + self.got.updates += 1; + } + _ => { + let counts = [ + ("updates", self.got.updates), + ("members", self.got.members.len()), + ("invites", self.got.invites.len()), + ("links", self.got.links.len()), + ("mentions", self.got.mentions.len()), + ]; + for (k, n) in counts { + if v[k].as_u64() != Some(n as u64) { + return Err(bad(format!("end.{k} = {} but {n} received", v[k]))); + } + } + self.got.end = Some(v); + } + } + Ok(()) + } +} + +async fn import( + State(state): State, + headers: HeaderMap, + Query(q): Query, + body: Body, +) -> Response { + let ok_key = authorized(&headers); + let (allow, users) = { + let mut s = state.lock().unwrap(); + s.import_posts += 1; + ( + s.allow_import, + s.users.keys().cloned().collect::>(), + ) + }; + let mode = q.mode.unwrap_or_else(|| "create".into()); + let mut imp = Importer { + doc: Doc::new(), + got: Received { + mode: mode.clone(), + ..Received::default() + }, + users, + phase: 0, + failure: None, + }; + if headers.get("content-type").and_then(|v| v.to_str().ok()) != Some("application/x-ndjson") { + imp.failure = Some(( + StatusCode::UNSUPPORTED_MEDIA_TYPE, + "bad_request", + "expected application/x-ndjson".into(), + )); + } + // Read (and parse) the whole body line by line even after a failure, so the client always + // gets the response rather than a reset connection. + let mut stream = body.into_data_stream(); + let mut buf: Vec = Vec::new(); + let mut broken = None; + while let Some(chunk) = stream.next().await { + let chunk = match chunk { + Ok(c) => c, + Err(e) => { + broken = Some(e.to_string()); + break; + } + }; + buf.extend_from_slice(&chunk); + while let Some(nl) = buf.iter().position(|b| *b == b'\n') { + let line: Vec = buf.drain(..=nl).collect(); + if imp.failure.is_none() && ok_key && allow { + if let Err(f) = imp.line(&line[..line.len() - 1]) { + imp.failure = Some(f); + } + } + } + } + if let Some(e) = broken { + // The client aborted the body: nothing is committed. + let mut s = state.lock().unwrap(); + imp.got.status = 400; + imp.got.error = Some(format!("body aborted: {e}")); + s.imports.push(imp.got); + return err(StatusCode::BAD_REQUEST, "bad_request", "body aborted"); + } + if !ok_key { + return err(StatusCode::UNAUTHORIZED, "unauthorized", "bad app key"); + } + if !allow { + return err( + StatusCode::FORBIDDEN, + "forbidden", + "this app lacks allow_import", + ); + } + if imp.failure.is_none() && !buf.is_empty() { + imp.failure = Some(( + StatusCode::BAD_REQUEST, + "bad_request", + "trailing bytes without newline".into(), + )); + } + if imp.failure.is_none() && imp.got.end.is_none() { + imp.failure = Some(( + StatusCode::BAD_REQUEST, + "bad_request", + "truncated: no end line".into(), + )); + } + let mut s = state.lock().unwrap(); + if let Some((status, code, message)) = imp.failure.take() { + imp.got.status = status.as_u16(); + imp.got.error = Some(format!("{code}: {message}")); + s.imports.push(imp.got); + return err(status, code, &message); + } + let id = Uuid::parse_str(imp.got.canvas["id"].as_str().unwrap()).unwrap(); + let owner = imp.got.canvas["owner"].as_str().unwrap().to_owned(); + let kind = imp.got.canvas["kind"].as_str().unwrap().to_owned(); + let replace = mode == "replace"; + + if let Some(existing) = s.canvases.get(&id) { + if !replace { + let d = state_digest(&existing.state); + imp.got.status = 409; + s.imports.push(imp.got); + return ( + StatusCode::CONFLICT, + Json(json!({"error": "canvas_exists", "message": "a canvas with this id exists", "digest": d})), + ) + .into_response(); + } + if !existing.imported { + imp.got.status = 409; + s.imports.push(imp.got); + return err( + StatusCode::CONFLICT, + "not_imported", + "only imported canvases can be replaced", + ); + } + } + if kind == "personal" { + let other = s + .canvases + .iter() + .find(|(cid, c)| **cid != id && c.kind == "personal" && c.owner == owner) + .map(|(cid, c)| (*cid, c.empty)); + if let Some((other_id, empty)) = other { + if !empty && !replace { + imp.got.status = 409; + s.imports.push(imp.got); + return err( + StatusCode::CONFLICT, + "personal_exists", + "the owner already has a non-empty Personal canvas", + ); + } + s.canvases.remove(&other_id); + } + } + let status = if s.canvases.contains_key(&id) { + "replaced" + } else { + "created" + }; + let state_bytes = imp + .doc + .transact() + .encode_state_as_update_v1(&StateVector::default()); + let d = digest::digest(&imp.doc); + imp.got.status = 200; + let got = imp.got; + let body = json!({ + "canvas_id": id, + "status": status, + "updates_applied": got.updates, + "members": got.members.len(), + "invites": got.invites.len(), + "links": got.links.len(), + "mentions": got.mentions.len(), + "digest": d, + }); + s.imports.push(got.clone()); + s.canvases.insert( + id, + Stored { + state: state_bytes, + owner, + kind, + imported: true, + empty: got.updates == 0, + received: got, + }, + ); + Json(body).into_response() +} + +async fn canvas_digest( + State(state): State, + headers: HeaderMap, + Path(id): Path, +) -> Response { + if !authorized(&headers) { + return err(StatusCode::UNAUTHORIZED, "unauthorized", "bad app key"); + } + let (stored, corrupt) = { + let mut s = state.lock().unwrap(); + s.digest_gets += 1; + ( + s.canvases.get(&id).map(|c| c.state.clone()), + s.corrupt_digest.contains(&id), + ) + }; + let Some(stored) = stored else { + return err(StatusCode::NOT_FOUND, "not_found", "no such canvas"); + }; + let d = if corrupt { + "0".repeat(64) + } else { + state_digest(&stored) + }; + Json(json!({"canvas_id": id, "digest": d})).into_response() +} diff --git a/crates/canvas-migrate/tests/common/mod.rs b/crates/canvas-migrate/tests/common/mod.rs new file mode 100644 index 0000000..7095cf7 --- /dev/null +++ b/crates/canvas-migrate/tests/common/mod.rs @@ -0,0 +1,519 @@ +//! Test harness: an isolated schema per test in the native PostgreSQL test database +//! (`CCMIG_TEST_DATABASE_URL`, default `postgres://farce@localhost:5433/ccmig_test`) with +//! copper-cloud's real migrations applied, legacy canvas rows seeded through the same sealing +//! code paths the old `copper-cloud-canvas` crate used, and the mock Canvas Cloud. + +#![allow(dead_code, clippy::missing_panics_doc)] + +pub mod digest; +pub mod mock; + +use std::path::Path; +use std::str::FromStr as _; +use std::time::Duration; + +use copper_cloud_canvas_migrate::{doc_aad, excerpt_aad, run, MigrateOptions, MigrateReport}; +use copper_cloud_core::crypto::Crypto; +use copper_cloud_core::ids; +use sqlx::postgres::{PgConnectOptions, PgPoolOptions}; +use sqlx::PgPool; +use uuid::Uuid; +use yrs::updates::decoder::Decode as _; +use yrs::{Doc, Map as _, ReadTxn as _, StateVector, Transact as _, Update}; + +pub const DEFAULT_DATABASE_URL: &str = "postgres://farce@localhost:5433/ccmig_test"; + +pub fn database_url() -> String { + std::env::var("CCMIG_TEST_DATABASE_URL").unwrap_or_else(|_| DEFAULT_DATABASE_URL.to_owned()) +} + +/// The ledger table as cc-link's `0030_canvas_link.sql` defines it (created here only if the +/// migration is not in the tree yet). +pub const LEDGER_DDL: &str = "CREATE TABLE IF NOT EXISTS canvas_migrations ( + canvas_id uuid PRIMARY KEY, + status text, + digest text, + updates int, + migrated_at timestamptz, + detail jsonb +)"; + +/// One isolated schema with copper-cloud's migrations applied. +pub struct TestDb { + pub pool: PgPool, + admin: PgPool, + schema: String, +} + +impl TestDb { + pub async fn new() -> Self { + let base = PgConnectOptions::from_str(&database_url()).expect("database url"); + let admin = PgPoolOptions::new() + .max_connections(1) + .acquire_timeout(Duration::from_secs(10)) + .connect_with(base.clone()) + .await + .expect( + "connecting to the test database (createdb -w -h localhost -p 5433 ccmig_test)", + ); + sweep_stale_schemas(&admin).await; + let now = std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .unwrap() + .as_secs(); + let schema = format!("ccmig_{now}_{}", &Uuid::new_v4().simple().to_string()[..12]); + sqlx::query(&format!("CREATE SCHEMA {schema}")) + .execute(&admin) + .await + .unwrap(); + let pool = PgPoolOptions::new() + .max_connections(8) + .acquire_timeout(Duration::from_secs(30)) + .connect_with(base.options([("search_path", schema.as_str())])) + .await + .unwrap(); + let dir = Path::new(env!("CARGO_MANIFEST_DIR")).join("../copper-cloud/migrations"); + let migrator = sqlx::migrate::Migrator::new(dir) + .await + .expect("reading copper-cloud migrations"); + migrator + .run(&pool) + .await + .expect("applying copper-cloud migrations"); + sqlx::query(LEDGER_DDL).execute(&pool).await.unwrap(); + Self { + pool, + admin, + schema, + } + } + + pub async fn cleanup(self) { + self.pool.close().await; + let _ = sqlx::query(&format!("DROP SCHEMA {} CASCADE", self.schema)) + .execute(&self.admin) + .await; + } +} + +/// Drops `ccmig__…` schemas older than two hours, left behind by panicked tests +/// (a passing test drops its own schema in [`TestDb::cleanup`]). +async fn sweep_stale_schemas(admin: &PgPool) { + let Ok(names) = sqlx::query_scalar::<_, String>( + r"SELECT nspname FROM pg_namespace WHERE nspname LIKE 'ccmig\_%'", + ) + .fetch_all(admin) + .await + else { + return; + }; + let cutoff = std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .unwrap() + .as_secs() + .saturating_sub(2 * 3600); + for name in names { + let born = name.split('_').nth(1).and_then(|t| t.parse::().ok()); + if born.is_some_and(|t| t < cutoff) { + let _ = sqlx::query(&format!("DROP SCHEMA IF EXISTS {name} CASCADE")) + .execute(admin) + .await; + } + } +} + +/// A seeded legacy canvas. +#[derive(Clone)] +pub struct Canvas { + pub id: Uuid, + pub key: [u8; 32], + /// Every plaintext update ever appended (including the ones a compaction later folded + /// into the snapshot): the ground truth of the document. + pub log: Vec>, +} + +impl Canvas { + /// The §6.3 digest of the full update log. + pub fn truth(&self) -> String { + let doc = Doc::new(); + { + let mut txn = doc.transact_mut(); + for u in &self.log { + txn.apply_update(Update::decode_v1(u).unwrap()).unwrap(); + } + } + digest::digest(&doc) + } +} + +/// Writes legacy rows exactly like copper-cloud 0.8 did (`rest.rs`, `store.rs`, +/// `mentions.rs`). +#[derive(Clone)] +pub struct Legacy { + pub db: PgPool, + pub crypto: Crypto, +} + +impl Legacy { + pub fn new(db: &PgPool) -> Self { + Self { + db: db.clone(), + crypto: Crypto::from_master_key(&[7u8; 32]), + } + } + + pub async fn user(&self, email: &str, display_name: &str, disabled: bool) -> Uuid { + let id = ids::uuid_v7(); + sqlx::query( + "INSERT INTO users (id, email, display_name, password_hash, data_key_wrapped, disabled) + VALUES ($1, $2, $3, 'argon2-not-used', $4, $5)", + ) + .bind(id) + .bind(email) + .bind(display_name) + .bind(self.crypto.wrap_key(&Crypto::new_key())) + .bind(disabled) + .execute(&self.db) + .await + .unwrap(); + id + } + + /// `rest.rs::insert_canvas`: canvas row, owner membership, initial `meta` update. + pub async fn canvas(&self, owner: Uuid, name: &str, kind: &str) -> Canvas { + let mut c = self.bare_canvas(owner, name, kind).await; + let doc = Doc::new(); + let meta = doc.get_or_insert_map("meta"); + { + let mut txn = doc.transact_mut(); + meta.insert(&mut txn, "name", name); + meta.insert(&mut txn, "createdBy", owner.to_string()); + } + let initial = doc + .transact() + .encode_state_as_update_v1(&StateVector::default()); + self.append(&mut c, &initial).await; + c + } + + /// A canvas row + owner membership but no document rows at all. + pub async fn bare_canvas(&self, owner: Uuid, name: &str, kind: &str) -> Canvas { + let id = ids::uuid_v7(); + let key = Crypto::new_key(); + sqlx::query( + "INSERT INTO canvases (id, owner_id, name, kind, doc_key_wrapped) + VALUES ($1, $2, $3, $4, $5)", + ) + .bind(id) + .bind(owner) + .bind(name) + .bind(kind) + .bind(self.crypto.wrap_key(&key)) + .execute(&self.db) + .await + .unwrap(); + sqlx::query( + "INSERT INTO canvas_members (canvas_id, user_id, role) VALUES ($1, $2, 'owner')", + ) + .bind(id) + .bind(owner) + .execute(&self.db) + .await + .unwrap(); + Canvas { + id, + key, + log: Vec::new(), + } + } + + /// `store.rs::append`. + pub async fn append(&self, c: &mut Canvas, update: &[u8]) -> i64 { + let sealed = Crypto::seal(&c.key, &doc_aad(c.id), update); + let seq: i64 = sqlx::query_scalar( + r#"INSERT INTO canvas_updates (canvas_id, "update") VALUES ($1, $2) RETURNING seq"#, + ) + .bind(c.id) + .bind(&sealed) + .fetch_one(&self.db) + .await + .unwrap(); + c.log.push(update.to_vec()); + seq + } + + /// `store.rs::compact`: the room's merged state of every update up to `upto_seq` becomes + /// the snapshot and the covered rows go. + pub async fn compact(&self, c: &Canvas, upto_seq: i64) { + let rows: Vec<(i64, Vec)> = sqlx::query_as( + r#"SELECT seq, "update" FROM canvas_updates WHERE canvas_id = $1 AND seq <= $2 ORDER BY seq"#, + ) + .bind(c.id) + .bind(upto_seq) + .fetch_all(&self.db) + .await + .unwrap(); + let doc = Doc::new(); + { + let mut txn = doc.transact_mut(); + if let Some(state) = self.snapshot_state(c).await { + txn.apply_update(Update::decode_v1(&state).unwrap()) + .unwrap(); + } + for (_, sealed) in rows { + let plain = Crypto::open(&c.key, &doc_aad(c.id), &sealed).unwrap(); + txn.apply_update(Update::decode_v1(&plain).unwrap()) + .unwrap(); + } + } + let state = doc + .transact() + .encode_state_as_update_v1(&StateVector::default()); + let sealed = Crypto::seal(&c.key, &doc_aad(c.id), &state); + sqlx::query( + "INSERT INTO canvas_snapshots (canvas_id, seq, state, updated_at) VALUES ($1, $2, $3, now()) + ON CONFLICT (canvas_id) DO UPDATE SET seq = EXCLUDED.seq, state = EXCLUDED.state, updated_at = now()", + ) + .bind(c.id) + .bind(upto_seq) + .bind(&sealed) + .execute(&self.db) + .await + .unwrap(); + sqlx::query("DELETE FROM canvas_updates WHERE canvas_id = $1 AND seq <= $2") + .bind(c.id) + .bind(upto_seq) + .execute(&self.db) + .await + .unwrap(); + } + + async fn snapshot_state(&self, c: &Canvas) -> Option> { + let sealed: Option> = + sqlx::query_scalar("SELECT state FROM canvas_snapshots WHERE canvas_id = $1") + .bind(c.id) + .fetch_optional(&self.db) + .await + .unwrap(); + sealed.map(|s| Crypto::open(&c.key, &doc_aad(c.id), &s).unwrap()) + } + + pub async fn member(&self, c: &Canvas, user: Uuid) { + sqlx::query( + "INSERT INTO canvas_members (canvas_id, user_id, role) VALUES ($1, $2, 'editor')", + ) + .bind(c.id) + .bind(user) + .execute(&self.db) + .await + .unwrap(); + } + + /// `rest.rs` invite: `token` = hex SHA-256 of a random secret. Returns `(id, token)`. + pub async fn invite( + &self, + c: &Canvas, + email: &str, + invited_by: Option, + status: &str, + nudged: bool, + ) -> (Uuid, String) { + let id = ids::uuid_v7(); + let token = ids::sha256_hex(ids::random_token().as_bytes()); + sqlx::query( + "INSERT INTO canvas_invites (id, canvas_id, email, invited_by, token, status, accepted_at, nudged_at) + VALUES ($1, $2, $3, $4, $5, $6, + CASE WHEN $6 = 'accepted' THEN now() END, + CASE WHEN $7 THEN now() END)", + ) + .bind(id) + .bind(c.id) + .bind(email) + .bind(invited_by) + .bind(&token) + .bind(status) + .bind(nudged) + .execute(&self.db) + .await + .unwrap(); + (id, token) + } + + /// `rest.rs` share link: only the SHA-256 of the token is stored. Returns + /// `(id, plaintext token)`. + pub async fn share_link(&self, c: &Canvas, created_by: Uuid, uses: i64) -> (Uuid, String) { + let id = ids::uuid_v7(); + let token = ids::random_token(); + sqlx::query( + "INSERT INTO canvas_share_links (id, canvas_id, token_sha256, role, created_by, uses) + VALUES ($1, $2, $3, 'editor', $4, $5)", + ) + .bind(id) + .bind(c.id) + .bind(ids::sha256(token.as_bytes()).to_vec()) + .bind(created_by) + .bind(uses) + .execute(&self.db) + .await + .unwrap(); + (id, token) + } + + /// `mentions.rs::post_mentions`: excerpt sealed under the doc key, AAD + /// `"copper-cloud/v1/mention:" ‖ row id`. + pub async fn mention( + &self, + c: &Canvas, + from: Uuid, + to: Uuid, + message_id: &str, + excerpt: &str, + read: bool, + ) -> Uuid { + let id = ids::uuid_v7(); + let sealed = Crypto::seal(&c.key, &excerpt_aad(id), excerpt.as_bytes()); + sqlx::query( + "INSERT INTO canvas_mentions (id, canvas_id, message_id, from_user, to_user, excerpt_sealed, read_at) + VALUES ($1, $2, $3, $4, $5, $6, CASE WHEN $7 THEN now() END)", + ) + .bind(id) + .bind(c.id) + .bind(message_id) + .bind(from) + .bind(to) + .bind(&sealed) + .bind(read) + .execute(&self.db) + .await + .unwrap(); + id + } +} + +/// A Yjs client editing a canvas: every edit returns the incremental update it produced. +pub struct Client { + pub doc: Doc, +} + +impl Client { + pub fn new(client_id: u64, base: &[Vec]) -> Self { + let doc = Doc::with_client_id(client_id); + { + let mut txn = doc.transact_mut(); + for u in base { + txn.apply_update(Update::decode_v1(u).unwrap()).unwrap(); + } + } + Self { doc } + } + + pub fn edit(&self, f: impl FnOnce(&Doc)) -> Vec { + let before = self.doc.transact().state_vector(); + f(&self.doc); + self.doc.transact().encode_state_as_update_v1(&before) + } +} + +/// Everything a migration test needs. +pub struct Env { + pub db: TestDb, + pub legacy: Legacy, + pub mock: mock::Mock, + pub client: reqwest::Client, +} + +impl Env { + pub async fn new() -> Self { + let db = TestDb::new().await; + let legacy = Legacy::new(&db.pool); + let mock = mock::Mock::start().await; + let client = reqwest::Client::builder() + .timeout(Duration::from_secs(60)) + .build() + .unwrap(); + Self { + db, + legacy, + mock, + client, + } + } + + pub async fn migrate(&self, opts: MigrateOptions) -> anyhow::Result { + self.migrate_with_key(mock::APP_KEY, opts).await + } + + pub async fn migrate_with_key( + &self, + key: &str, + opts: MigrateOptions, + ) -> anyhow::Result { + let report = run( + &self.db.pool, + &self.legacy.crypto, + &self.client, + &self.mock.url, + key, + opts, + ) + .await?; + println!("{report}"); + Ok(report) + } + + pub async fn ledger(&self) -> Vec<(Uuid, String, Option, Option)> { + sqlx::query_as( + "SELECT canvas_id, status, digest, updates FROM canvas_migrations ORDER BY canvas_id", + ) + .fetch_all(&self.db.pool) + .await + .unwrap() + } + + pub async fn ledger_row( + &self, + id: Uuid, + ) -> Option<(String, Option, serde_json::Value)> { + sqlx::query_as("SELECT status, digest, detail FROM canvas_migrations WHERE canvas_id = $1") + .bind(id) + .fetch_optional(&self.db.pool) + .await + .unwrap() + } + + /// A fingerprint of every legacy table, to prove the tool never writes them. + pub async fn legacy_fingerprint(&self) -> String { + let mut out = String::new(); + for table in [ + "users", + "canvases", + "canvas_members", + "canvas_invites", + "canvas_updates", + "canvas_snapshots", + "canvas_share_links", + "canvas_mentions", + ] { + let rows: Vec<(String,)> = + sqlx::query_as(&format!("SELECT t::text FROM {table} t ORDER BY t::text")) + .fetch_all(&self.db.pool) + .await + .unwrap(); + out.push_str(table); + for (r,) in rows { + out.push_str(&r); + out.push('\n'); + } + } + out + } + + pub async fn finish(self) { + self.db.cleanup().await; + } +} + +/// Default options for tests (`concurrency` 2). +pub fn opts() -> MigrateOptions { + MigrateOptions::default() +} diff --git a/crates/canvas-migrate/tests/digest.rs b/crates/canvas-migrate/tests/digest.rs new file mode 100644 index 0000000..b8e3d02 --- /dev/null +++ b/crates/canvas-migrate/tests/digest.rs @@ -0,0 +1,220 @@ +//! Properties of the §6.3 canonical digest (`common::digest`): independent of update order, +//! snapshot compaction and root typing; sensitive to content. + +mod common; + +use common::digest::{canonical_json, digest}; +use yrs::types::text::Text as _; +use yrs::updates::decoder::Decode as _; +use yrs::{ + any, Any, Array as _, Doc, Map as _, MapPrelim, Out, ReadTxn as _, StateVector, Transact as _, + Update, +}; + +fn doc_from(updates: &[&[u8]]) -> Doc { + let doc = Doc::new(); + { + let mut txn = doc.transact_mut(); + for u in updates { + txn.apply_update(Update::decode_v1(u).unwrap()).unwrap(); + } + } + doc +} + +fn edit(doc: &Doc, f: impl FnOnce(&Doc)) -> Vec { + let before = doc.transact().state_vector(); + f(doc); + doc.transact().encode_state_as_update_v1(&before) +} + +fn permutations(n: usize) -> Vec> { + if n == 0 { + return vec![Vec::new()]; + } + let mut out = Vec::new(); + for p in permutations(n - 1) { + for i in 0..=p.len() { + let mut q = p.clone(); + q.insert(i, n - 1); + out.push(q); + } + } + out +} + +fn compacted(doc: &Doc) -> Vec { + doc.transact() + .encode_state_as_update_v1(&StateVector::default()) +} + +/// Updates from two concurrent clients over every root kind. +fn history() -> Vec> { + let a = Doc::with_client_id(1); + let b = Doc::with_client_id(2); + let mut out = Vec::new(); + out.push(edit(&a, |d| { + let meta = d.get_or_insert_map("meta"); + let shapes = d.get_or_insert_map("shapes"); + let mut t = d.transact_mut(); + meta.insert(&mut t, "name", "Board"); + shapes.insert( + &mut t, + "s1", + MapPrelim::from([("x", Any::from(1)), ("y", Any::from(2.5))]), + ); + shapes.insert(&mut t, "gone", any!({"type": "sticky"})); + })); + out.push(edit(&b, |d| { + let chat = d.get_or_insert_array("chat"); + let notes = d.get_or_insert_text("notes"); + let mut t = d.transact_mut(); + chat.push_back(&mut t, any!({"id": "m1", "text": "hi \u{2028} "})); + notes.insert(&mut t, 0, "naïve ✓"); + })); + { + let mut t = b.transact_mut(); + t.apply_update(Update::decode_v1(&out[0]).unwrap()).unwrap(); + } + out.push(edit(&b, |d| { + let shapes = d.get_or_insert_map("shapes"); + let mut t = d.transact_mut(); + if let Some(Out::YMap(s1)) = shapes.get(&t, "s1") { + s1.insert(&mut t, "x", 99); + } + shapes.remove(&mut t, "gone"); + })); + out.push(edit(&a, |d| { + let meta = d.get_or_insert_map("meta"); + let mut t = d.transact_mut(); + meta.insert(&mut t, "thumb", Any::Buffer(vec![1_u8, 2, 3].into())); + meta.insert(&mut t, "big", Any::Number(yrs::Number::Int(7))); + })); + out +} + +#[test] +fn order_compaction_and_typing_do_not_matter() { + let h = history(); + let refs: Vec<&[u8]> = h.iter().map(Vec::as_slice).collect(); + let forward = doc_from(&refs); + let expected = digest(&forward); + + // Every permutation of the four updates. Causal orders (u1,u2 ≺ u3; u1 ≺ u4) always + // integrate completely. yrs 0.28 can leave structs pending for some non-causal orders + // (e.g. exact reverse: `has_missing_updates()` stays true) — then the document really is + // different until the missing pieces are re-delivered. The property is therefore: every + // *complete* application yields the same digest. + let mut complete = 0; + for order in permutations(4) { + let us: Vec<&[u8]> = order.iter().map(|i| refs[*i]).collect(); + let doc = doc_from(&us); + let pos = |u: usize| order.iter().position(|x| *x == u).unwrap(); + let causal = pos(0) < pos(2) && pos(1) < pos(2) && pos(0) < pos(3); + if doc.transact().has_missing_updates() { + assert!(!causal, "a causal order {order:?} left updates pending"); + // Re-delivering everything completes it. + let again = doc_from(&[us.clone(), refs.clone()].concat()); + assert_eq!(digest(&again), expected, "{order:?} re-delivered"); + } else { + complete += 1; + assert_eq!(digest(&doc), expected, "order {order:?}"); + } + } + assert!(complete >= 6, "{complete}"); + + // One compacted snapshot, and a snapshot of a prefix followed by the tail. + assert_eq!( + digest(&doc_from(&[&compacted(&forward)])), + expected, + "compacted" + ); + let prefix = compacted(&doc_from(&refs[..2])); + assert_eq!( + digest(&doc_from(&[&prefix, refs[2], refs[3]])), + expected, + "snapshot + tail" + ); + + // A reader that typed its roots (like a server that called get_or_insert_*). + let typed = Doc::new(); + typed.get_or_insert_map("meta"); + typed.get_or_insert_map("shapes"); + typed.get_or_insert_array("chat"); + typed.get_or_insert_text("notes"); + typed.get_or_insert_map("never-used"); + { + let mut t = typed.transact_mut(); + for u in &refs { + t.apply_update(Update::decode_v1(u).unwrap()).unwrap(); + } + } + assert_eq!(digest(&typed), expected, "typed roots, plus an empty one"); +} + +#[test] +fn content_changes_change_the_digest() { + let h = history(); + let refs: Vec<&[u8]> = h.iter().map(Vec::as_slice).collect(); + let doc = doc_from(&refs); + let before = digest(&doc); + let more = edit(&doc, |d| { + let meta = d.get_or_insert_map("meta"); + let mut t = d.transact_mut(); + meta.insert(&mut t, "name", "Board 2"); + }); + assert_ne!(digest(&doc), before); + assert_eq!( + digest(&doc_from(&[&compacted(&doc_from(&refs)), &more])), + digest(&doc) + ); + // Missing an update is detected. + assert_ne!(digest(&doc_from(&refs[..3])), before); +} + +#[test] +fn canonical_form_is_exact() { + let h = history(); + let refs: Vec<&[u8]> = h.iter().map(Vec::as_slice).collect(); + let doc = doc_from(&refs); + let json = canonical_json(&doc.transact()); + assert_eq!( + json, + concat!( + r#"{"chat":[{"id":"m1","text":"hi "#, + "\u{2028}", + r#" "}],"#, + r#""meta":{"big":7.0,"name":"Board","thumb":"AQID"},"#, + r#""notes":"naïve ✓","#, + r#""shapes":{"s1":{"x":99.0,"y":2.5}}}"# + ) + ); + // Valid JSON, and the digest is SHA-256 of exactly these bytes. + let parsed: serde_json::Value = serde_json::from_str(&json).unwrap(); + assert_eq!(parsed["shapes"]["s1"]["x"], 99.0); + assert_eq!(digest(&doc).len(), 64); +} + +#[test] +fn empty_documents_and_empty_roots() { + let empty = Doc::new(); + assert_eq!(canonical_json(&empty.transact()), "{}"); + // e3b0… is SHA-256("") — the digest is over "{}", not the empty string. + assert_eq!( + digest(&empty), + "44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a" + ); + // A root whose only entry was deleted renders like no root at all. + let doc = Doc::new(); + let m = doc.get_or_insert_map("shapes"); + { + let mut t = doc.transact_mut(); + m.insert(&mut t, "a", 1); + } + { + let mut t = doc.transact_mut(); + m.remove(&mut t, "a"); + } + assert_eq!(digest(&doc), digest(&empty)); + assert_eq!(digest(&doc_from(&[&compacted(&doc)])), digest(&empty)); +} diff --git a/crates/canvas-migrate/tests/migrate.rs b/crates/canvas-migrate/tests/migrate.rs new file mode 100644 index 0000000..c960ac0 --- /dev/null +++ b/crates/canvas-migrate/tests/migrate.rs @@ -0,0 +1,914 @@ +//! End-to-end migration tests: legacy rows in native PostgreSQL → `run` → the mock Canvas +//! Cloud (which really applies the Yjs updates and computes §6.3 digests). + +#![allow(clippy::too_many_lines, clippy::many_single_char_names)] // long, linear scenarios + +mod common; + +use common::mock::APP_KEY; +use common::{opts, Canvas, Client, Env}; +use copper_cloud_canvas_migrate::{MigrateOptions, Status}; +use copper_cloud_core::ids; +use serde_json::Value; +use uuid::Uuid; +use yrs::types::text::Text as _; +use yrs::updates::decoder::Decode as _; +use yrs::{any, Any, Array as _, Doc, Map as _, MapPrelim, Out, Transact as _, Update}; + +struct World { + alice: Uuid, + bob: Uuid, + carol: Uuid, + dave: Uuid, + /// Shared: snapshot + tail, members, invites, a share link, mentions. + roadmap: Canvas, + /// Alice's Personal canvas. + personal: Canvas, + /// Shared canvas with no document rows at all. + empty: Canvas, + /// Shared canvas with more updates than one read page. + busy: Canvas, + link_id: Uuid, + link_token: String, + invites: Vec<(Uuid, String, &'static str)>, + mention_ab: Uuid, +} + +fn apply(client: &Client, updates: &[&[u8]]) { + let mut txn = client.doc.transact_mut(); + for u in updates { + txn.apply_update(Update::decode_v1(u).unwrap()).unwrap(); + } +} + +async fn world(env: &Env) -> World { + let l = &env.legacy; + let alice = l.user("alice@example.com", "Alice", false).await; + let bob = l.user("bob@example.com", "Bob", false).await; + let carol = l.user("carol@example.com", "", false).await; + let dave = l.user("dave@example.com", "Dave", true).await; + + // Roadmap: two clients editing concurrently, then a compaction, then a tail. + let mut roadmap = l.canvas(alice, "Roadmap", "shared").await; + l.member(&roadmap, bob).await; + l.member(&roadmap, carol).await; + let a = Client::new(101, &roadmap.log); + let b = Client::new(202, &roadmap.log); + let u1 = a.edit(|d| { + let shapes = d.get_or_insert_map("shapes"); + let mut t = d.transact_mut(); + shapes.insert( + &mut t, + "s1", + MapPrelim::from([ + ("type", Any::from("sticky")), + ("x", Any::from(10)), + ("y", Any::from(20.5)), + ("text", Any::from("Ship it")), + ]), + ); + shapes.insert( + &mut t, + "s2", + MapPrelim::from([("type", Any::from("frame")), ("w", Any::from(480.0))]), + ); + }); + l.append(&mut roadmap, &u1).await; + let u2 = b.edit(|d| { + let chat = d.get_or_insert_array("chat"); + let agents = d.get_or_insert_map("agents"); + let mut t = d.transact_mut(); + chat.push_back( + &mut t, + any!({"id": "m1", "authorId": bob.to_string(), "text": "@Alice timeline?", "at": 1_760_000_000_000_i64}), + ); + agents.insert(&mut t, "agent-1", any!({"name": "Scout", "cursor": null, "status": "idle"})); + }); + l.append(&mut roadmap, &u2).await; + apply(&b, &[&u1]); + let u3 = b.edit(|d| { + let shapes = d.get_or_insert_map("shapes"); + let notes = d.get_or_insert_text("notes"); + let mut t = d.transact_mut(); + if let Some(Out::YMap(s1)) = shapes.get(&t, "s1") { + s1.insert(&mut t, "x", 42.25); + } + notes.insert(&mut t, 0, "Q4 — draft ✓"); + }); + let seq3 = l.append(&mut roadmap, &u3).await; + l.compact(&roadmap, seq3).await; + apply(&a, &[&u2, &u3]); + let u4 = a.edit(|d| { + let shapes = d.get_or_insert_map("shapes"); + let meta = d.get_or_insert_map("meta"); + let mut t = d.transact_mut(); + shapes.remove(&mut t, "s2"); + meta.insert(&mut t, "thumb", Any::Buffer(vec![0_u8, 1, 2, 250].into())); + }); + l.append(&mut roadmap, &u4).await; + let u5 = b.edit(|d| { + let chat = d.get_or_insert_array("chat"); + let mut t = d.transact_mut(); + chat.push_back( + &mut t, + any!({"id": "m2", "authorId": bob.to_string(), "text": "🚀 \"quoted\" \\ done"}), + ); + }); + l.append(&mut roadmap, &u5).await; + + let invites = vec![ + ( + l.invite(&roadmap, "erin@example.com", Some(alice), "pending", true) + .await, + "pending", + ), + ( + l.invite(&roadmap, "bob@example.com", Some(alice), "accepted", false) + .await, + "accepted", + ), + ( + l.invite(&roadmap, "frank@example.com", Some(bob), "revoked", false) + .await, + "revoked", + ), + ( + l.invite(&roadmap, "gina@example.com", None, "declined", false) + .await, + "declined", + ), + ] + .into_iter() + .map(|((id, token), status)| (id, token, status)) + .collect(); + let (link_id, link_token) = l.share_link(&roadmap, alice, 3).await; + let mention_ab = l + .mention( + &roadmap, + alice, + bob, + "m1-reply", + "@Bob see the timeline ✓", + false, + ) + .await; + l.mention(&roadmap, bob, carol, "m2", "@carol done", true) + .await; + + let mut personal = l.canvas(alice, "Personal", "personal").await; + let p = Client::new(303, &personal.log); + let pu = p.edit(|d| { + let shapes = d.get_or_insert_map("shapes"); + let mut t = d.transact_mut(); + shapes.insert( + &mut t, + "p1", + MapPrelim::from([("type", Any::from("text")), ("text", Any::from("mine"))]), + ); + }); + l.append(&mut personal, &pu).await; + + let empty = l.bare_canvas(bob, "Empty", "shared").await; + + let mut busy = l.canvas(carol, "Busy", "shared").await; + let c = Client::new(404, &busy.log); + for i in 0..40 { + let u = c.edit(|d| { + let shapes = d.get_or_insert_map("shapes"); + let mut t = d.transact_mut(); + shapes.insert(&mut t, format!("b{}", i % 7), any!({"n": i})); + }); + l.append(&mut busy, &u).await; + } + + World { + alice, + bob, + carol, + dave, + roadmap, + personal, + empty, + busy, + link_id, + link_token, + invites, + mention_ab, + } +} + +fn row( + report: &copper_cloud_canvas_migrate::MigrateReport, + id: Uuid, +) -> &copper_cloud_canvas_migrate::CanvasReport { + report + .canvases + .iter() + .find(|c| c.id == id) + .unwrap_or_else(|| panic!("no row for {id}")) +} + +#[tokio::test] +async fn full_run_migrates_everything_verified() { + let env = Env::new().await; + let w = world(&env).await; + let before = env.legacy_fingerprint().await; + + let report = env.migrate(opts()).await.unwrap(); + assert!(report.ok(), "{report}"); + assert_eq!(report.users.provisioned, 4); + assert_eq!(report.users.created, 4); + assert_eq!(report.totals.migrated, 4); + assert_eq!(report.totals.failed, 0); + // Legacy order (created_at). + let order: Vec = report.canvases.iter().map(|c| c.id).collect(); + assert_eq!( + order, + vec![w.roadmap.id, w.personal.id, w.empty.id, w.busy.id] + ); + + // Users: same ids, display names and disabled flags passed through. + env.mock.with(|s| { + for (u, email) in [ + (w.alice, "alice"), + (w.bob, "bob"), + (w.carol, "carol"), + (w.dave, "dave"), + ] { + assert_eq!(s.users.get(&u.to_string()), Some(&u), "{email}"); + } + let carol = s + .user_payloads + .iter() + .find(|p| p["subject"] == w.carol.to_string()) + .unwrap(); + assert_eq!(carol["display_name"], ""); + assert_eq!(carol["id"], w.carol.to_string()); + let dave = s + .user_payloads + .iter() + .find(|p| p["subject"] == w.dave.to_string()) + .unwrap(); + assert_eq!(dave["disabled"], true); + }); + + // Every canvas: digest = the ground truth of its full update log (snapshot + tail). + for c in [&w.roadmap, &w.personal, &w.busy] { + let r = row(&report, c.id); + assert_eq!(r.status, Status::Migrated, "{r:?}"); + assert_eq!(r.digest.as_deref(), Some(c.truth().as_str()), "{}", c.id); + assert_eq!( + env.mock.stored_digest(c.id).as_deref(), + Some(c.truth().as_str()) + ); + } + + // Roadmap details. + let r = row(&report, w.roadmap.id); + assert!(r.snapshot); + assert_eq!(r.updates, 3, "snapshot + 2 tail updates"); + assert_eq!((r.members, r.invites, r.links, r.mentions), (3, 4, 1, 2)); + let got = env.mock.received(w.roadmap.id).unwrap(); + assert_eq!(got.canvas["name"], "Roadmap"); + assert_eq!(got.canvas["kind"], "shared"); + assert_eq!(got.canvas["owner"], w.alice.to_string()); + assert_eq!(got.members[0]["role"], "owner"); + assert_eq!(got.members[0]["subject"], w.alice.to_string()); + let mut member_ids: Vec<&str> = got + .members + .iter() + .map(|m| m["subject"].as_str().unwrap()) + .collect(); + member_ids.sort_unstable(); + let mut expected = vec![w.alice.to_string(), w.bob.to_string(), w.carol.to_string()]; + expected.sort(); + assert_eq!(member_ids, expected); + // Invites: every status, the raw token digest, nudged_at. + for (id, token, status) in &w.invites { + let inv = got + .invites + .iter() + .find(|i| i["id"] == id.to_string()) + .unwrap(); + assert_eq!(inv["status"], *status); + assert_eq!(inv["token_sha256_hex"], token.as_str()); + assert_eq!(inv["nudged_at"].is_string(), *status == "pending"); + assert_eq!(inv["accepted_at"].is_string(), *status == "accepted"); + } + let declined = got + .invites + .iter() + .find(|i| i["status"] == "declined") + .unwrap(); + assert_eq!(declined["invited_by"], Value::Null); + // Share link: the digest of the existing token is preserved, so the token keeps working. + let link = &got.links[0]; + assert_eq!(link["id"], w.link_id.to_string()); + assert_eq!( + link["token_sha256_hex"], + ids::sha256_hex(w.link_token.as_bytes()) + ); + assert_eq!(link["uses"], 3); + assert_eq!(link["created_by"], w.alice.to_string()); + // Mention: the excerpt arrives as plaintext. + let m = got + .mentions + .iter() + .find(|m| m["id"] == w.mention_ab.to_string()) + .unwrap(); + assert_eq!(m["excerpt"], "@Bob see the timeline ✓"); + assert_eq!(m["from"], w.alice.to_string()); + assert_eq!(m["to"], w.bob.to_string()); + assert_eq!(m["message_id"], "m1-reply"); + assert_eq!(m["read_at"], Value::Null); + assert!(got + .mentions + .iter() + .any(|m| m["excerpt"] == "@carol done" && m["read_at"].is_string())); + + // Zero-update canvas. + let e = row(&report, w.empty.id); + assert_eq!(e.status, Status::Migrated); + assert_eq!((e.updates, e.bytes, e.snapshot), (0, 0, false)); + assert_eq!(env.mock.received(w.empty.id).unwrap().updates, 0); + // Paged reads: more updates than one page. + assert_eq!(row(&report, w.busy.id).updates, 41); + + // Ledger: all migrated with the verified digest and update counts. + let ledger = env.ledger().await; + assert_eq!(ledger.len(), 4); + for (id, status, digest, updates) in &ledger { + let r = row(&report, *id); + assert_eq!(status, "migrated"); + assert_eq!(digest.as_deref(), r.digest.as_deref()); + assert_eq!(*updates, Some(i32::try_from(r.updates).unwrap())); + } + let (_, _, detail) = env.ledger_row(w.roadmap.id).await.unwrap(); + assert_eq!(detail["import_status"], "created"); + assert_eq!(detail["mentions"], 2); + + // Legacy rows untouched; the table and JSON renderings work. + assert_eq!(env.legacy_fingerprint().await, before); + let table = report.table(); + assert!( + table.contains("Roadmap") && table.contains("migrated") && table.trim_end().ends_with("ok"), + "{table}" + ); + let json: Value = serde_json::to_value(&report).unwrap(); + assert_eq!(json["totals"]["migrated"], 4); + assert_eq!(json["ok"], true); + env.finish().await; +} + +#[tokio::test] +async fn reruns_are_idempotent_and_never_trust_unverified_canvases() { + let env = Env::new().await; + let w = world(&env).await; + assert!(env.migrate(opts()).await.unwrap().ok()); + let posts = env.mock.with(|s| s.import_posts); + + // Plain re-run: the ledger says migrated → nothing is sent. + let again = env.migrate(opts()).await.unwrap(); + assert!(again.ok(), "{again}"); + assert_eq!(again.totals.already, 4); + assert!(again.canvases.iter().all(|c| c.digest.is_some())); + assert_eq!(env.mock.with(|s| s.import_posts), posts); + + // --only on a migrated canvas: canvas-cloud's digest matches the ledger → already. + let only = env + .migrate(MigrateOptions { + only: vec![w.roadmap.id], + ..opts() + }) + .await + .unwrap(); + assert!(only.ok(), "{only}"); + assert_eq!(only.canvases.len(), 1); + assert_eq!(only.canvases[0].status, Status::Already); + assert_eq!( + env.mock.with(|s| s.import_posts), + posts, + "pre-check avoids the upload" + ); + + // Crash between import and verification: ledger `imported` + matching digest → confirmed. + sqlx::query("UPDATE canvas_migrations SET status = 'imported' WHERE canvas_id = $1") + .bind(w.personal.id) + .execute(&env.db.pool) + .await + .unwrap(); + let confirm = env.migrate(opts()).await.unwrap(); + assert!(confirm.ok(), "{confirm}"); + assert_eq!(row(&confirm, w.personal.id).status, Status::Already); + assert_eq!(env.ledger_row(w.personal.id).await.unwrap().0, "migrated"); + + // No ledger record but the canvas exists in canvas-cloud: not provably the same → failed, + // and the hint says how to fix it. + sqlx::query("DELETE FROM canvas_migrations WHERE canvas_id = $1") + .bind(w.busy.id) + .execute(&env.db.pool) + .await + .unwrap(); + let unknown = env.migrate(opts()).await.unwrap(); + assert!(!unknown.ok()); + let r = row(&unknown, w.busy.id); + assert_eq!(r.status, Status::Failed); + assert!(r.error.as_deref().unwrap().contains("--replace"), "{r:?}"); + assert_eq!(env.ledger_row(w.busy.id).await.unwrap().0, "failed"); + assert_eq!( + env.mock.with(|s| s.import_posts), + posts, + "still nothing uploaded" + ); + + // --replace --only fixes it. + let fixed = env + .migrate(MigrateOptions { + only: vec![w.busy.id], + replace: true, + ..opts() + }) + .await + .unwrap(); + assert!(fixed.ok(), "{fixed}"); + assert_eq!(fixed.canvases[0].status, Status::Replaced); + assert_eq!(env.ledger_row(w.busy.id).await.unwrap().0, "migrated"); + env.finish().await; +} + +#[tokio::test] +async fn only_limits_the_run_and_reports_unknown_ids() { + let env = Env::new().await; + let w = world(&env).await; + let stranger = Uuid::now_v7(); + let report = env + .migrate(MigrateOptions { + only: vec![w.empty.id, stranger, w.personal.id], + ..opts() + }) + .await + .unwrap(); + assert!(!report.ok(), "an unknown --only id fails the run"); + let ids: Vec = report.canvases.iter().map(|c| c.id).collect(); + assert_eq!(ids, vec![w.personal.id, w.empty.id, stranger]); + assert_eq!(row(&report, w.personal.id).status, Status::Migrated); + assert_eq!(row(&report, w.empty.id).status, Status::Migrated); + assert!(row(&report, stranger) + .error + .as_deref() + .unwrap() + .contains("not a legacy canvas")); + assert!(env.mock.received(w.roadmap.id).is_none()); + assert_eq!(env.ledger().await.len(), 2); + // Users are provisioned in full regardless of --only. + assert_eq!(report.users.provisioned, 4); + env.finish().await; +} + +#[tokio::test] +async fn replace_reimports_canvases_that_changed_in_copper() { + let env = Env::new().await; + let mut w = world(&env).await; + assert!(env.migrate(opts()).await.unwrap().ok()); + let old = w.roadmap.truth(); + + // A rollback period: the legacy canvas keeps changing in the old copper-cloud. + let c = Client::new(909, &w.roadmap.log); + let u = c.edit(|d| { + let shapes = d.get_or_insert_map("shapes"); + let mut t = d.transact_mut(); + shapes.insert( + &mut t, + "late", + any!({"type": "sticky", "text": "after rollback"}), + ); + }); + env.legacy.append(&mut w.roadmap, &u).await; + let new = w.roadmap.truth(); + assert_ne!(old, new); + + // A plain re-run trusts the ledger … + let plain = env.migrate(opts()).await.unwrap(); + assert_eq!(row(&plain, w.roadmap.id).status, Status::Already); + assert_eq!( + env.mock.stored_digest(w.roadmap.id).as_deref(), + Some(old.as_str()) + ); + + // … --replace re-imports every canvas from the legacy data. + let replaced = env + .migrate(MigrateOptions { + replace: true, + ..opts() + }) + .await + .unwrap(); + assert!(replaced.ok(), "{replaced}"); + assert_eq!(replaced.totals.replaced, 4); + assert!(env + .mock + .with(|s| s.imports.iter().rev().take(4).all(|i| i.mode == "replace"))); + let r = row(&replaced, w.roadmap.id); + assert_eq!(r.digest.as_deref(), Some(new.as_str())); + assert_eq!(r.updates, 4); + let (status, digest, detail) = env.ledger_row(w.roadmap.id).await.unwrap(); + assert_eq!( + (status.as_str(), digest.as_deref()), + ("migrated", Some(new.as_str())) + ); + assert_eq!(detail["import_status"], "replaced"); + env.finish().await; +} + +#[tokio::test] +async fn dry_run_reads_everything_and_sends_nothing() { + let env = Env::new().await; + let w = world(&env).await; + let report = env + .migrate(MigrateOptions { + dry_run: true, + ..opts() + }) + .await + .unwrap(); + assert!(report.ok(), "{report}"); + assert!(report.dry_run); + assert_eq!(report.totals.dry_run, 4); + assert_eq!(report.users.total, 4); + assert_eq!(report.users.provisioned, 0); + let r = row(&report, w.roadmap.id); + assert_eq!(r.status, Status::DryRun); + assert_eq!( + (r.updates, r.members, r.invites, r.links, r.mentions), + (3, 3, 4, 1, 2) + ); + assert!(r.bytes > 0 && r.digest.is_none()); + assert_eq!(report.totals.updates, 3 + 2 + 41); + env.mock.with(|s| { + assert_eq!(s.stats_gets, 1, "only the preflight"); + assert_eq!((s.batch_posts, s.import_posts, s.digest_gets), (0, 0, 0)); + }); + assert_eq!(env.ledger().await, Vec::new(), "a dry run records nothing"); + assert!(report.table().contains("dry run")); + + // Without the ledger table a dry run still works; a real run refuses clearly. + sqlx::query("DROP TABLE canvas_migrations") + .execute(&env.db.pool) + .await + .unwrap(); + assert!(env + .migrate(MigrateOptions { + dry_run: true, + ..opts() + }) + .await + .unwrap() + .ok()); + let err = env.migrate(opts()).await.unwrap_err().to_string(); + assert!(err.contains("canvas_migrations"), "{err}"); + env.finish().await; +} + +#[tokio::test] +async fn digest_mismatch_fails_and_is_recorded() { + let env = Env::new().await; + let w = world(&env).await; + env.mock.with(|s| s.corrupt_digest.insert(w.personal.id)); + let report = env.migrate(opts()).await.unwrap(); + assert!(!report.ok()); + assert_eq!(report.totals.failed, 1); + assert_eq!(report.totals.migrated, 3); + let r = row(&report, w.personal.id); + assert_eq!(r.status, Status::Failed); + assert!( + r.error.as_deref().unwrap().contains("digest mismatch"), + "{r:?}" + ); + let (status, digest, detail) = env.ledger_row(w.personal.id).await.unwrap(); + assert_eq!(status, "failed"); + assert_eq!( + digest.as_deref(), + Some(w.personal.truth().as_str()), + "the import's digest is kept as evidence" + ); + assert!(detail["error"] + .as_str() + .unwrap() + .contains("digest mismatch")); + assert!(report.table().trim_end().ends_with("FAILED")); + // A later run does not mistake the unverified import for a migrated canvas. + let again = env.migrate(opts()).await.unwrap(); + assert_eq!(row(&again, w.personal.id).status, Status::Failed); + env.finish().await; +} + +#[tokio::test] +async fn unknown_subject_fails_only_that_canvas() { + let env = Env::new().await; + let w = world(&env).await; + // canvas-cloud "loses" carol after provisioning her (e.g. deleted between the calls). + env.mock + .with(|s| s.forget_subjects.insert(w.carol.to_string())); + let report = env.migrate(opts()).await.unwrap(); + assert!(!report.ok()); + for id in [w.roadmap.id, w.busy.id] { + let r = row(&report, id); + assert_eq!(r.status, Status::Failed); + let e = r.error.as_deref().unwrap(); + assert!( + e.contains("422") && e.contains("unknown_subject") && e.contains(&w.carol.to_string()), + "{e}" + ); + assert!(env.mock.received(id).is_none(), "nothing stored"); + } + assert_eq!(row(&report, w.personal.id).status, Status::Migrated); + assert_eq!(row(&report, w.empty.id).status, Status::Migrated); + assert_eq!(env.ledger_row(w.roadmap.id).await.unwrap().0, "failed"); + env.finish().await; +} + +#[tokio::test] +async fn users_that_do_not_keep_their_id_stop_the_run() { + let env = Env::new().await; + let w = world(&env).await; + let other = Uuid::now_v7(); + env.mock + .with(|s| s.remap_user.insert(w.bob.to_string(), other)); + let report = env.migrate(opts()).await.unwrap(); + assert!(!report.ok()); + assert_eq!(report.users.failed.len(), 1); + assert_eq!(report.users.failed[0].user_id, w.bob); + assert!(report.users.failed[0].error.contains(&other.to_string())); + assert!(report + .error + .as_deref() + .unwrap() + .contains("no canvas was migrated")); + assert!(report.canvases.is_empty()); + assert_eq!(env.mock.with(|s| s.import_posts), 0); + assert_eq!(env.ledger().await, Vec::new()); + assert!(report.table().contains("bob@example.com")); + env.finish().await; +} + +#[tokio::test] +async fn preflight_and_permission_errors_are_clear() { + let env = Env::new().await; + world(&env).await; + let bad_key = env.migrate_with_key("cva_wrong", opts()).await.unwrap_err(); + let msg = format!("{bad_key:#}"); + assert!( + msg.contains("/v1/apps/self/stats") && msg.contains("401"), + "{msg}" + ); + + let nowhere = copper_cloud_canvas_migrate::run( + &env.db.pool, + &env.legacy.crypto, + &env.client, + &url::Url::parse("http://127.0.0.1:9/").unwrap(), + APP_KEY, + opts(), + ) + .await + .unwrap_err(); + assert!(format!("{nowhere:#}").contains("preflight"), "{nowhere:#}"); + + env.mock.with(|s| s.allow_import = false); + let forbidden = env.migrate(opts()).await.unwrap_err().to_string(); + assert!(forbidden.contains("allow_import"), "{forbidden}"); + assert_eq!(env.ledger().await, Vec::new()); + env.finish().await; +} + +#[tokio::test] +async fn personal_canvases_follow_the_import_rules() { + let env = Env::new().await; + let w = world(&env).await; + // Bob got an empty auto-created Personal in canvas-cloud: the legacy one replaces it. + let mut bob_personal = env.legacy.canvas(w.bob, "Personal", "personal").await; + let b = Client::new(505, &bob_personal.log); + let u = b.edit(|d| { + let shapes = d.get_or_insert_map("shapes"); + let mut t = d.transact_mut(); + shapes.insert(&mut t, "bp", any!({"type": "text"})); + }); + env.legacy.append(&mut bob_personal, &u).await; + let auto = env.mock.seed_personal(w.bob, false); + // Alice already uses a non-empty Personal in canvas-cloud: refused without --replace. + let used = env.mock.seed_personal(w.alice, true); + + let report = env.migrate(opts()).await.unwrap(); + assert_eq!(row(&report, bob_personal.id).status, Status::Migrated); + assert!(env.mock.with(|s| !s.canvases.contains_key(&auto))); + let r = row(&report, w.personal.id); + assert_eq!(r.status, Status::Failed); + assert!( + r.error.as_deref().unwrap().contains("personal_exists"), + "{r:?}" + ); + + let replaced = env + .migrate(MigrateOptions { + only: vec![w.personal.id], + replace: true, + ..opts() + }) + .await + .unwrap(); + assert!(replaced.ok(), "{replaced}"); + assert!(env.mock.with(|s| !s.canvases.contains_key(&used))); + assert_eq!( + env.mock.stored_digest(w.personal.id), + Some(w.personal.truth()) + ); + env.finish().await; +} + +#[tokio::test] +async fn unreadable_legacy_rows_fail_cleanly() { + let env = Env::new().await; + let w = world(&env).await; + // A tampered update row on Busy, and a mention excerpt sealed under the wrong row id. + sqlx::query( + r#"UPDATE canvas_updates SET "update" = set_byte("update", 20, get_byte("update", 20) # 1) + WHERE seq = (SELECT max(seq) FROM canvas_updates WHERE canvas_id = $1)"#, + ) + .bind(w.busy.id) + .execute(&env.db.pool) + .await + .unwrap(); + let wrong = copper_cloud_core::crypto::Crypto::seal( + &w.roadmap.key, + &copper_cloud_canvas_migrate::excerpt_aad(Uuid::nil()), + b"lost", + ); + sqlx::query("UPDATE canvas_mentions SET excerpt_sealed = $1 WHERE id = $2") + .bind(&wrong) + .bind(w.mention_ab) + .execute(&env.db.pool) + .await + .unwrap(); + + let report = env + .migrate(MigrateOptions { + concurrency: 4, + ..opts() + }) + .await + .unwrap(); + assert!(!report.ok()); + let busy = row(&report, w.busy.id); + assert_eq!(busy.status, Status::Failed); + assert!( + busy.error.as_deref().unwrap().contains("does not decrypt"), + "{busy:?}" + ); + assert!( + env.mock.received(w.busy.id).is_none(), + "an aborted body commits nothing" + ); + // The undecryptable excerpt is sent empty with a warning, as the old server served it. + let roadmap = row(&report, w.roadmap.id); + assert_eq!(roadmap.status, Status::Migrated); + assert_eq!(roadmap.warnings.len(), 1, "{roadmap:?}"); + let got = env.mock.received(w.roadmap.id).unwrap(); + let m = got + .mentions + .iter() + .find(|m| m["id"] == w.mention_ab.to_string()) + .unwrap(); + assert_eq!(m["excerpt"], ""); + env.finish().await; +} + +#[tokio::test] +async fn wrong_master_key_is_refused_before_anything_is_sent() { + let env = Env::new().await; + world(&env).await; + let wrong = copper_cloud_core::crypto::Crypto::from_master_key(&[9u8; 32]); + let err = copper_cloud_canvas_migrate::run( + &env.db.pool, + &wrong, + &env.client, + &env.mock.url, + APP_KEY, + opts(), + ) + .await + .unwrap_err(); + assert!(format!("{err:#}").contains("master_key"), "{err:#}"); + env.mock + .with(|s| assert_eq!((s.batch_posts, s.import_posts), (0, 0))); + // A dry run checks the same thing. + let err = copper_cloud_canvas_migrate::run( + &env.db.pool, + &wrong, + &env.client, + &env.mock.url, + APP_KEY, + MigrateOptions { + dry_run: true, + ..opts() + }, + ) + .await + .unwrap_err(); + assert!(format!("{err:#}").contains("master_key"), "{err:#}"); + env.finish().await; +} + +#[test] +fn chat_and_text_roots_survive_in_the_truth() { + // Sanity check of the fixture itself: a reader that types its roots sees the content. + let doc = Doc::new(); + let chat = doc.get_or_insert_array("chat"); + { + let mut t = doc.transact_mut(); + chat.push_back(&mut t, any!({"id": "x"})); + } + assert!(common::digest::canonical_json(&doc.transact()).contains("\"chat\":[{\"id\":\"x\"}]")); +} + +#[tokio::test] +async fn large_canvases_stream_with_bounded_concurrency() { + let env = Env::new().await; + let l = &env.legacy; + let owner = l.user("big@example.com", "Big", false).await; + // A canvas whose snapshot carries a 12 MiB image, then 300 small updates. + let mut big = l.canvas(owner, "Big", "shared").await; + let editor = Client::new(606, &big.log); + let blob: Vec = (0..12 * 1024 * 1024_u32).map(|i| (i % 251) as u8).collect(); + let image = editor.edit(|d| { + let assets = d.get_or_insert_map("assets"); + let mut t = d.transact_mut(); + assets.insert(&mut t, "img", Any::Buffer(blob.clone().into())); + }); + let seq = l.append(&mut big, &image).await; + l.compact(&big, seq).await; + for i in 0..300 { + let u = editor.edit(|d| { + let shapes = d.get_or_insert_map("shapes"); + let mut t = d.transact_mut(); + shapes.insert(&mut t, format!("s{}", i % 50), any!({"x": i, "label": "x"})); + }); + l.append(&mut big, &u).await; + } + // And 20 ordinary canvases. + let mut small = Vec::new(); + for i in 0..20 { + small.push(l.canvas(owner, &format!("Small {i}"), "shared").await.id); + } + + let started = std::time::Instant::now(); + let report = env + .migrate(MigrateOptions { + concurrency: 3, + ..opts() + }) + .await + .unwrap(); + assert!(report.ok(), "{report}"); + let r = row(&report, big.id); + assert_eq!(r.updates, 301); + assert!(r.bytes > 12 * 1024 * 1024); + assert_eq!(r.digest.as_deref(), Some(big.truth().as_str())); + assert_eq!(report.totals.migrated, 21); + let ids: Vec = report.canvases.iter().map(|c| c.id).collect(); + assert_eq!(ids[0], big.id); + assert_eq!(&ids[1..], small.as_slice(), "report keeps legacy order"); + println!( + "migrated 21 canvases (12 MiB snapshot) in {:?}", + started.elapsed() + ); + env.finish().await; +} + +#[tokio::test] +async fn dry_run_warns_about_lines_over_the_default_limit() { + let env = Env::new().await; + let l = &env.legacy; + let owner = l.user("huge@example.com", "Huge", false).await; + let mut huge = l.bare_canvas(owner, "Huge", "shared").await; + let editor = Client::new(707, &[]); + // 25 MiB of binary → ~33.3 MiB of base64 on one line. + let blob = vec![7_u8; 25 * 1024 * 1024]; + let image = editor.edit(|d| { + let assets = d.get_or_insert_map("assets"); + let mut t = d.transact_mut(); + assets.insert(&mut t, "img", Any::Buffer(blob.into())); + }); + l.append(&mut huge, &image).await; + let report = env + .migrate(MigrateOptions { + dry_run: true, + ..opts() + }) + .await + .unwrap(); + let r = row(&report, huge.id); + assert_eq!(r.status, Status::DryRun); + assert_eq!(r.warnings.len(), 1, "{r:?}"); + assert!(r.warnings[0].contains("32 MiB"), "{r:?}"); + env.finish().await; +} From 8f428c81bb4672eb023d6bd1551b288baddc9b4d Mon Sep 17 00:00:00 2001 From: Felipe Arce Date: Mon, 19 Oct 2026 18:54:46 +0000 Subject: [PATCH 3/9] portal: Canvas Cloud page, unlinked states for Canvases and Overview New console page to link, test, re-key and unlink a Canvas Cloud, with event-bridge status and the migration ledger summary. The Canvases page and the Overview tiles show an explicit not-linked / unreachable state instead of numbers. --- .../src/app/(console)/canvas-cloud/page.tsx | 9 + .../src/app/(console)/canvas-cloud/view.tsx | 857 ++++++++++++++++++ portal/src/app/(console)/canvases/view.tsx | 198 ++-- portal/src/app/(console)/overview-view.tsx | 43 + portal/src/components/app-rail.tsx | 2 + portal/src/lib/api.ts | 14 +- portal/src/lib/audit.ts | 23 + portal/src/lib/mock.ts | 180 +++- portal/src/lib/types.ts | 68 +- 9 files changed, 1320 insertions(+), 74 deletions(-) create mode 100644 portal/src/app/(console)/canvas-cloud/page.tsx create mode 100644 portal/src/app/(console)/canvas-cloud/view.tsx diff --git a/portal/src/app/(console)/canvas-cloud/page.tsx b/portal/src/app/(console)/canvas-cloud/page.tsx new file mode 100644 index 0000000..ae9bfba --- /dev/null +++ b/portal/src/app/(console)/canvas-cloud/page.tsx @@ -0,0 +1,9 @@ +import type { Metadata } from "next"; + +import { CanvasCloudView } from "./view"; + +export const metadata: Metadata = { title: "Canvas Cloud" }; + +export default function Page() { + return ; +} diff --git a/portal/src/app/(console)/canvas-cloud/view.tsx b/portal/src/app/(console)/canvas-cloud/view.tsx new file mode 100644 index 0000000..b7ef388 --- /dev/null +++ b/portal/src/app/(console)/canvas-cloud/view.tsx @@ -0,0 +1,857 @@ +"use client"; + +import { CircleAlertIcon, CircleCheckIcon, CloudOffIcon } from "lucide-react"; +import { useId, useRef, useState } from "react"; +import { toast } from "sonner"; + +import { ConfirmDialog } from "@/components/confirm-dialog"; +import { CopyButton } from "@/components/copy-button"; +import { Fingerprint } from "@/components/fingerprint"; +import { PageHeader } from "@/components/page-header"; +import { RelativeTime } from "@/components/relative-time"; +import { useSession } from "@/components/session-provider"; +import { StatusBadge, type Tone } from "@/components/status-badge"; +import { Button } from "@/components/ui/button"; +import { Input } from "@/components/ui/input"; +import { Label } from "@/components/ui/label"; +import { Skeleton } from "@/components/ui/skeleton"; +import { useNow } from "@/hooks/use-now"; +import { type Resource, useResource } from "@/hooks/use-resource"; +import { api, errorMessage } from "@/lib/api"; +import { formatCount, plural } from "@/lib/format"; +import type { CanvasLink, CanvasLinkTest } from "@/lib/types"; +import { cn } from "@/lib/utils"; + +type Link = Resource; + +const APP_KEY_COMMAND = 'canvas-cloud apps create --name "Copper Cloud" --allow-import'; +const MIGRATE_COMMAND = "copper-cloud canvas migrate"; + +export function CanvasCloudView() { + const link = useResource("canvas-link", (signal) => api.canvasLink.get(signal)); + const data = link.data; + + return ( + <> + + {link.error && !data ? ( + + ) : !data ? ( +
+ + + +
+ ) : data.linked ? ( + <> + + + + + + ) : ( + <> + + + {data.migrations.legacy_canvases > 0 && } + + )} + + ); +} + +function Section({ + id, + title, + description, + children, +}: { + id: string; + title: string; + description: React.ReactNode; + children: React.ReactNode; +}) { + return ( +
+
+

+ {title} +

+
+ {description} +
+
+
{children}
+
+ ); +} + +/** Server-side error strings arrive lowercase and unpunctuated. */ +function sentence(s: string): string { + const t = s.trim(); + if (!t) return t; + const first = t.charAt(0).toUpperCase() + t.slice(1); + return /[.!?)]$/.test(first) ? first : `${first}.`; +} + +/** + * A timestamp for something that already happened. The shared relative-time clock + * ticks every 30 s, so a fresh server timestamp can read "in a moment"; clamp it. + */ +function PastTime({ + value, + empty, + className, +}: { + value: string | null; + empty?: string; + className?: string; +}) { + const now = useNow() || Date.now(); + const clamped = value && Date.parse(value) > now ? new Date(now).toISOString() : value; + return ; +} + +const hostOf = (url: string) => url.replace(/^https?:\/\//i, "").replace(/\/+$/, ""); + +function updatedBy(by: string | null) { + if (!by) return null; + return by === "cli" ? "the server CLI" : by; +} + +function reachability(data: CanvasLink): { tone: Tone; label: string } { + if (!data.enabled) return { tone: "muted", label: "Turned off" }; + if (data.status.reachable === true) return { tone: "success", label: "Reachable" }; + if (data.status.reachable === false) return { tone: "danger", label: "Unreachable" }; + return { tone: "muted", label: "Not checked yet" }; +} + +/** What `DELETE canvas-link` leaves behind, so the page flips before the reload lands. */ +function unlinked(prev: CanvasLink): CanvasLink { + return { + ...prev, + linked: false, + url: null, + enabled: false, + tls_fingerprint: null, + linked_at: null, + status: { reachable: null, version: null, error: null, checked_at: null }, + bridge: { connected: false, last_event_at: null, last_error: null }, + }; +} + +function Row({ label, children }: { label: string; children: React.ReactNode }) { + return ( +
+
{label}
+
{children}
+
+ ); +} + +/** A shell command to run on a server, with a copy button. */ +function Command({ value }: { value: string }) { + return ( +
+ + {value} + + +
+ ); +} + +// Linked -------------------------------------------------------------------- + +function ConnectionSection({ link, data }: { link: Link; data: CanvasLink }) { + const { refreshOverview } = useSession(); + const [testing, setTesting] = useState(false); + const [result, setResult] = useState(null); + const [confirmOpen, setConfirmOpen] = useState(false); + const url = data.url ?? ""; + const host = hostOf(url); + const reach = reachability(data); + const plainHttp = /^http:\/\//i.test(url); + + async function test() { + if (testing) return; + setTesting(true); + setResult(null); + try { + setResult(await api.canvasLink.test()); + } catch (error) { + setResult({ + ok: false, + version: null, + latency_ms: null, + error: errorMessage(error), + }); + } + // The test refreshes the server's own health check; show it. + await link.reload(); + setTesting(false); + } + + return ( +
+
+
+

+ {host} +

+ {reach.label} +
+ +
+ +
+ + {url} + + +
+
+ + {data.status.version ? ( + {data.status.version} + ) : ( + Unknown + )} + + + + + + + + + {data.tls_fingerprint ? ( + "Self-signed, pinned" + ) : plainHttp ? ( + + None, plain http on this machine + + ) : ( + "Publicly trusted" + )} + + {data.tls_fingerprint && ( +
+

Pinned fingerprint (SHA-256)

+
+ +
+
+ )} +
+ + {data.status.reachable === false && data.status.error && ( +
+
+ )} + +
+ +
+ {result && ( + + )} +
+ +
+
+ + {!data.enabled && ( +

+ This link is turned off, so Copper users can’t open canvases. Saving the + app key below turns it back on. +

+ )} + {data.updated_at && ( +

+ Last changed + {updatedBy(data.updated_by) && <> by {updatedBy(data.updated_by)}}. +

+ )} + + { + await api.canvasLink.remove(); + link.mutate((prev) => (prev ? unlinked(prev) : prev)); + setResult(null); + toast.success(`Unlinked ${host}`); + void link.reload(); + void refreshOverview(); + }} + > +

+ The canvases stay on {host}. Linking it + again takes an app key from that server. +

+
+
+ ); +} + +function TestResult({ + result, + shownAbove, +}: { + result: CanvasLinkTest; + /** The same error is already in the callout above; don't print it twice. */ + shownAbove: boolean; +}) { + if (!result.ok) { + return ( +

+

+ ); + } + return ( +

+

+ ); +} + +function BridgeSection({ data }: { data: CanvasLink }) { + const { connected, last_event_at: lastEvent, last_error: lastError } = data.bridge; + return ( +
+
+
+

Event stream

+ + {connected ? "Connected" : "Disconnected"} + +
+

+ {connected + ? lastEvent + ? "Coppers hear about canvas changes as they happen." + : "Connected. No canvas events yet." + : "Copper Cloud keeps retrying. Until it reconnects, Coppers see canvas list changes on their next refresh."} +

+ {(lastEvent || (!connected && lastError)) && ( +
+ {lastEvent && ( +
+
Last event
+
+ +
+
+ )} + {!connected && lastError && ( +
+
Last error
+
+ {lastError} +
+
+ )} +
+ )} +
+
+ ); +} + +function MigrationSection({ data }: { data: CanvasLink }) { + const { migrated, failed, legacy_canvases: legacy } = data.migrations; + const done = Math.min(migrated, legacy); + const left = Math.max(0, legacy - migrated); + const pct = (n: number) => (legacy > 0 ? `${(n / legacy) * 100}%` : "0%"); + + return ( +
+ {legacy === 0 ? ( +

+ This server has no canvases from before Canvas Cloud. Nothing to migrate. +

+ ) : ( +
+
+

+ + {formatCount(done)} of{" "} + {plural(legacy, "legacy canvas", "legacy canvases")} migrated + + + 0 ? "font-medium text-destructive" : "text-muted-foreground", + )} + > + {formatCount(failed)} failed + +

+ + )} +
+ ); +} + +function KeySection({ link, data }: { link: Link; data: CanvasLink }) { + return ( +
+ Kept sealed on this server and never shown again. Replace it after rotating it + on the Canvas Cloud ( + + canvas-cloud apps rotate + + ), or to move to a new address. + + } + > + +
+ ); +} + +// Not linked ---------------------------------------------------------------- + +function NotLinkedSection() { + return ( +
+
+
+
+
+

No Canvas Cloud linked

+

+ Copper users can’t open canvases until you link one. Nothing is deleted + in the meantime. +

+
+
+
+ ); +} + +// Link / replace form --------------------------------------------------------- + +const LOOPBACK = /^(localhost|127(?:\.\d{1,3}){3}|\[::1\])$/i; + +type FieldKey = "url" | "key" | "fingerprint"; +type Errors = Partial>; + +function checkUrl(raw: string): { value?: string; error?: string } { + const typed = raw.trim(); + if (!typed) return { error: "Enter the Canvas Cloud’s address." }; + const withScheme = /^[a-z][a-z\d+.-]*:\/\//i.test(typed) ? typed : `https://${typed}`; + let url: URL; + try { + url = new URL(withScheme); + } catch { + return { error: "Use a full address, like https://canvas.example.com." }; + } + if (url.protocol !== "https:" && url.protocol !== "http:") { + return { error: "Use an https:// address." }; + } + if (url.protocol === "http:" && !LOOPBACK.test(url.hostname)) { + return { error: "Plain http:// only works for localhost. Use https://." }; + } + return { value: withScheme.replace(/\/+$/, "") }; +} + +function checkKey(raw: string): { value?: string; error?: string } { + const key = raw.trim(); + if (!key) return { error: "Paste the app key from the Canvas Cloud." }; + if (/\s/.test(key)) return { error: "App keys have no spaces." }; + if (key.length < 8) return { error: "That’s too short to be an app key." }; + return { value: key }; +} + +function checkFingerprint(raw: string): { value?: string; error?: string } { + const hex = raw.replace(/[\s:]/g, "").toLowerCase(); + if (!hex) return { value: "" }; + if (!/^[0-9a-f]{64}$/.test(hex)) { + return { error: "Use the SHA-256 fingerprint: 64 hex characters, colons optional." }; + } + return { value: hex }; +} + +interface FieldProps { + id: string; + label: string; + optional?: boolean; + value: string; + onChange: (v: string) => void; + placeholder?: string; + hint?: React.ReactNode; + error?: string; + secret?: boolean; + mono?: boolean; + type?: "text" | "url"; + inputRef: React.Ref; +} + +function Field({ + id, + label, + optional, + value, + onChange, + placeholder, + hint, + error, + secret, + mono, + type = "text", + inputRef, +}: FieldProps) { + return ( +
+ + onChange(e.target.value)} + placeholder={placeholder} + autoComplete="off" + autoCapitalize="off" + spellCheck={false} + className={cn((secret || mono) && "font-mono placeholder:font-sans")} + aria-invalid={error ? true : undefined} + aria-describedby={error || hint ? `${id}-hint` : undefined} + /> + {(error || hint) && ( +

+ {error ?? hint} +

+ )} +
+ ); +} + +function StoredKey() { + return ( +
+ Stored key + + •••••••••••• + +
+ ); +} + +function LinkForm({ + mode, + link, + data, +}: { + mode: "link" | "replace"; + link: Link; + data: CanvasLink; +}) { + const { refreshOverview } = useSession(); + const replacing = mode === "replace"; + const [url, setUrl] = useState(replacing ? (data.url ?? "") : ""); + const [key, setKey] = useState(""); + const [fingerprint, setFingerprint] = useState( + replacing ? (data.tls_fingerprint ?? "") : "", + ); + const [errors, setErrors] = useState({}); + const [formError, setFormError] = useState(); + const [pending, setPending] = useState(false); + const ids = { url: useId(), key: useId(), fingerprint: useId() }; + const refs = { + url: useRef(null), + key: useRef(null), + fingerprint: useRef(null), + }; + + async function submit(event: React.FormEvent) { + event.preventDefault(); + if (pending) return; + const u = checkUrl(url); + const k = checkKey(key); + const f = checkFingerprint(fingerprint); + const next: Errors = { url: u.error, key: k.error, fingerprint: f.error }; + setErrors(next); + setFormError(undefined); + const first = (["url", "key", "fingerprint"] as const).find((name) => next[name]); + if (first) { + refs[first].current?.focus(); + return; + } + + setPending(true); + try { + const saved = await api.canvasLink.set({ + url: u.value!, + app_key: k.value!, + ...(f.value ? { tls_fingerprint: f.value } : {}), + }); + setKey(""); + link.mutate(() => saved); + toast.success( + replacing ? "Canvas Cloud link saved" : `Linked ${hostOf(saved.url ?? u.value!)}`, + ); + void refreshOverview(); + } catch (error) { + setFormError(errorMessage(error)); + } finally { + setPending(false); + } + } + + const edit = (name: FieldKey, set: (v: string) => void) => (v: string) => { + set(v); + if (errors[name]) setErrors((e) => ({ ...e, [name]: undefined })); + }; + + return ( +
+ {replacing && } + + + + {formError && ( +
+
+ )} +
+ +

+ {replacing + ? "Nothing changes unless the server accepts the new address and key." + : "Copper Cloud checks the address and key before saving."} +

+
+ + ); +} + +function LoadError({ error, onRetry }: { error: unknown; onRetry: () => void }) { + return ( +
+
+ ); +} diff --git a/portal/src/app/(console)/canvases/view.tsx b/portal/src/app/(console)/canvases/view.tsx index cca40f7..28f224b 100644 --- a/portal/src/app/(console)/canvases/view.tsx +++ b/portal/src/app/(console)/canvases/view.tsx @@ -1,6 +1,7 @@ "use client"; -import { FrameIcon, LockIcon } from "lucide-react"; +import { CircleAlertIcon, CloudOffIcon, FrameIcon, LockIcon } from "lucide-react"; +import Link from "next/link"; import { useMemo, useState } from "react"; import { toast } from "sonner"; @@ -17,19 +18,36 @@ import { RelativeTime } from "@/components/relative-time"; import { RowActions } from "@/components/row-actions"; import { Segmented } from "@/components/segmented"; import { useSession } from "@/components/session-provider"; +import { Button } from "@/components/ui/button"; import { DropdownMenuItem } from "@/components/ui/dropdown-menu"; import { useList } from "@/hooks/use-list"; -import { api } from "@/lib/api"; +import { api, ApiError, errorMessage } from "@/lib/api"; import { formatCount, matches, plural } from "@/lib/format"; import type { Canvas, CanvasKind } from "@/lib/types"; import { cn } from "@/lib/utils"; type KindFilter = "all" | CanvasKind; -export function CanvasesView() { - const list = useList("canvases", (offset, limit, signal) => - api.canvases.list({ offset, limit }, signal), +/** Where the table would be, for states with no table to show. */ +function CanvasCloudState({ children }: { children: React.ReactNode }) { + return ( +
+
{children}
+
); +} + +export function CanvasesView() { + // `linked: false` = no Canvas Cloud is linked, so there is nothing to list. + const [linked, setLinked] = useState(); + const list = useList("canvases", async (offset, limit, signal) => { + const page = await api.canvases.list({ offset, limit }, signal); + setLinked(page.linked !== false); + return page; + }); + const errorCode = list.error instanceof ApiError ? list.error.code : undefined; + const notLinked = linked === false || errorCode === "canvas_unavailable"; + const upstreamDown = !notLinked && errorCode === "canvas_upstream"; const { refreshOverview } = useSession(); const [filter, setFilter] = useState(""); const [kind, setKind] = useState("all"); @@ -143,68 +161,118 @@ export function CanvasesView() { description="Names, owners and members only. Canvas content never reaches the admin console." /> - - - - - c.id} - loading={list.loading} - error={list.error} - onRetry={list.reload} - filtered={!!filter || kind !== "all"} - empty={ - - Each person gets a Personal canvas the first time they open Canvas in Copper. - Shared canvases appear when someone creates one and invites others. + {notLinked ? ( + + } nativeButton={false}> + Link a Canvas Cloud + + } + > + Canvases live on a Canvas Cloud server. Until one is linked, Copper users + can’t open canvases. Nothing has been deleted. - } - noMatches={ - - Try another name or owner.{" "} - - - } - footer={ - + ) : upstreamDown ? ( + +
+
+
+ ) : ( + <> + + + + + c.id} + loading={list.loading} + error={list.error} + onRetry={list.reload} + filtered={!!filter || kind !== "all"} + empty={ + + Each person gets a Personal canvas the first time they open Canvas in + Copper. Shared canvases appear when someone creates one and invites + others. + + } + noMatches={ + + Try another name or owner.{" "} + + + } + footer={ + + } /> - } - /> + + )} + api.canvasLink.get(signal), + ); + if (!missing) return undefined; + if (link.data) { + if (!link.data.linked) return "Not linked"; + if (link.data.status.reachable === false) return "Unreachable"; + return "Unavailable"; + } + return link.error ? "Unavailable" : undefined; +} + function StatStrip({ counts }: { counts: Counts | undefined }) { const liveNow = (counts?.live_peers ?? 0) > 0; + const canvasGap = useCanvasGap(counts); return (
{STATS.map((s) => { @@ -187,6 +210,26 @@ function StatStrip({ counts }: { counts: Counts | undefined }) { {value === undefined ? ( + ) : value === null ? ( + + + {/* The reason shows once, on the Canvases tile; the live tiles say it + to screen readers only. */} + + No count. Canvas Cloud: + {canvasGap ?? "Unavailable"} + + ) : ( {formatCount(value)} diff --git a/portal/src/components/app-rail.tsx b/portal/src/components/app-rail.tsx index 0372ae4..5c539d1 100644 --- a/portal/src/components/app-rail.tsx +++ b/portal/src/components/app-rail.tsx @@ -1,5 +1,6 @@ "use client"; +import { CloudIcon } from "lucide-react"; // Canvas Cloud nav item import { FrameIcon, KeyRoundIcon, @@ -30,6 +31,7 @@ const NAV = [ { href: "/people/", label: "People", icon: UsersIcon }, { href: "/devices/", label: "Devices", icon: LaptopIcon }, { href: "/canvases/", label: "Canvases", icon: FrameIcon }, + { href: "/canvas-cloud/", label: "Canvas Cloud", icon: CloudIcon }, { href: "/intelligence/", label: "AI keys", icon: SparklesIcon }, { href: "/settings/", label: "Settings", icon: SettingsIcon }, ] as const; diff --git a/portal/src/lib/api.ts b/portal/src/lib/api.ts index ba503c8..a1bd42e 100644 --- a/portal/src/lib/api.ts +++ b/portal/src/lib/api.ts @@ -13,7 +13,6 @@ import type { AccessKeyQuery, ApiErrorBody, AuditEntry, - Canvas, CanvasQuery, CreateAccessKey, CreatedAccessKey, @@ -37,6 +36,8 @@ import type { UserPatch, UserQuery, } from "./types"; +// Canvas Cloud link +import type { CanvasLink, CanvasLinkTest, CanvasLinkUpdate, CanvasPage } from "./types"; export const API_BASE = (process.env.NEXT_PUBLIC_API_BASE ?? "").replace(/\/+$/, ""); export const MOCK_MODE = process.env.NEXT_PUBLIC_MOCK === "1"; @@ -280,11 +281,20 @@ export const api = { // Canvases (metadata only; content is never exposed) canvases: { list: (q?: CanvasQuery, signal?: AbortSignal) => - request>("canvases", { query: { ...page(q), kind: q?.kind }, signal }), + request("canvases", { query: { ...page(q), kind: q?.kind }, signal }), remove: (canvasId: string) => request(`canvases/${id(canvasId)}`, { method: "DELETE" }), }, + // Canvas Cloud link (the server canvases are served from; the key is write-only) + canvasLink: { + get: (signal?: AbortSignal) => request("canvas-link", { signal }), + set: (input: CanvasLinkUpdate) => + request("canvas-link", { method: "PUT", body: input }), + remove: () => request("canvas-link", { method: "DELETE" }), + test: () => request("canvas-link/test", { method: "POST" }), + }, + // Pairing codes (active only) pairingCodes: { list: (q?: PairingCodeQuery, signal?: AbortSignal) => diff --git a/portal/src/lib/audit.ts b/portal/src/lib/audit.ts index 7b1a6a6..2ac8341 100644 --- a/portal/src/lib/audit.ts +++ b/portal/src/lib/audit.ts @@ -28,6 +28,8 @@ export interface AuditLine { } const str = (v: unknown) => (typeof v === "string" && v ? v : undefined); +/** `https://canvas.example.com/` → `canvas.example.com` (Canvas Cloud audit rows). */ +const linkHost = (url: string) => url.replace(/^https?:\/\//, "").replace(/\/+$/, ""); /** Turn an `admin_audit` row into a readable sentence fragment. */ export function describeAudit(entry: AuditEntry, self: boolean): AuditLine { @@ -93,6 +95,27 @@ export function describeAudit(entry: AuditEntry, self: boolean): AuditLine { return { icon: "device", verb: "removed a device" }; case "canvas.delete": return { icon: "canvas", verb: "deleted canvas", object: str(d.name) }; + // Canvas Cloud link ------------------------------------------------------ + case "canvas_link.set": + return { + icon: "canvas", + verb: str(d.url) ? "linked the Canvas Cloud at" : "linked a Canvas Cloud", + object: str(d.url) ? linkHost(str(d.url)!) : undefined, + actor: d.via === "cli" ? "The server CLI" : undefined, + }; + case "canvas_link.remove": + return { + icon: "canvas", + verb: str(d.url) ? "unlinked the Canvas Cloud at" : "unlinked the Canvas Cloud", + object: str(d.url) ? linkHost(str(d.url)!) : undefined, + actor: d.via === "cli" ? "The server CLI" : undefined, + }; + case "canvas_link.test": + return { + icon: "canvas", + verb: "tested the Canvas Cloud link", + suffix: d.ok === true ? "(reachable)" : d.ok === false ? "(failed)" : undefined, + }; case "intelligence.update": { const parts: string[] = []; if (d.jev === null) parts.push("removed the Jev key"); diff --git a/portal/src/lib/mock.ts b/portal/src/lib/mock.ts index 5076c4e..7338f18 100644 --- a/portal/src/lib/mock.ts +++ b/portal/src/lib/mock.ts @@ -20,6 +20,8 @@ import type { Settings, User, } from "./types"; +// Canvas Cloud link +import type { CanvasLink } from "./types"; const NOW = Date.now(); const MIN = 60_000; @@ -218,6 +220,74 @@ let canvases: CanvasRow[] = [ }), ]; +// Canvas Cloud link --------------------------------------------------------- +// Canvases are served by a linked Canvas Cloud. Open any page with +// `?mock-canvas=unlinked|unreachable|error` to start in that state instead. + +type CanvasScenario = "linked" | "unlinked" | "unreachable" | "error"; +const CANVAS_SCENARIO: CanvasScenario = (() => { + if (typeof location === "undefined") return "linked"; + const v = new URLSearchParams(location.search).get("mock-canvas"); + return v === "unlinked" || v === "unreachable" || v === "error" ? v : "linked"; +})(); +const CANVAS_URL = "https://canvas.hollis.studio"; +const CANVAS_VERSION = "0.1.0 (a41c9e2)"; +const LEGACY_CANVASES = 16; +const CANVAS_DOWN_ERROR = "connect timeout after 5 s (canvas.hollis.studio:443)"; + +const unlinkedCanvasLink = (migrations: CanvasLink["migrations"]): CanvasLink => ({ + linked: false, + url: null, + enabled: false, + tls_fingerprint: null, + linked_at: null, + updated_at: null, + updated_by: null, + status: { reachable: null, version: null, error: null, checked_at: null }, + bridge: { connected: false, last_event_at: null, last_error: null }, + migrations, +}); + +/** True while the mock Canvas Cloud refuses connections. */ +let canvasHostDown = CANVAS_SCENARIO === "unreachable"; +let canvasLink: CanvasLink = + CANVAS_SCENARIO === "unlinked" + ? unlinkedCanvasLink({ migrated: 0, failed: 0, legacy_canvases: LEGACY_CANVASES }) + : { + linked: true, + url: CANVAS_URL, + enabled: true, + tls_fingerprint: null, + linked_at: ago(6 * DAY + 3 * HOUR), + updated_at: ago(6 * DAY + 3 * HOUR), + updated_by: admin.email, + status: canvasHostDown + ? { + reachable: false, + version: null, + error: CANVAS_DOWN_ERROR, + checked_at: ago(2 * MIN), + } + : { + reachable: true, + version: CANVAS_VERSION, + error: null, + checked_at: ago(40_000), + }, + bridge: canvasHostDown + ? { + connected: false, + last_event_at: ago(47 * MIN), + last_error: "connection refused, retrying in 30 s", + } + : { connected: true, last_event_at: ago(3 * MIN), last_error: null }, + migrations: canvasHostDown + ? { migrated: 12, failed: 1, legacy_canvases: LEGACY_CANVASES } + : { migrated: 12, failed: 0, legacy_canvases: LEGACY_CANVASES }, + }; +/** Canvases (and room counts) are only visible through a reachable link. */ +const canvasesServed = () => canvasLink.linked && canvasLink.enabled && !canvasHostDown; + type KeyRow = Omit; const keySpecs: { @@ -512,10 +582,10 @@ function overview(): Overview { counts: { users: users.length, devices: devices.length, - canvases: canvases.length, + canvases: canvasesServed() ? canvases.length : null, access_keys: keys.filter((k) => keyStatus(k) === "active").length, - live_rooms: 2, - live_peers: 3, + live_rooms: canvasesServed() ? 2 : null, + live_peers: canvasesServed() ? 3 : null, }, public_url: HOST, tls: { mode: "self-signed", fingerprint: FINGERPRINT }, @@ -820,13 +890,24 @@ function route( } if (seg[0] === "canvases") { + if (canvasLink.linked && canvasHostDown) { + throw new HttpError( + 502, + "canvas_upstream", + `the Canvas Cloud didn't answer: ${CANVAS_DOWN_ERROR}`, + ); + } + if (method === "GET" && seg.length === 1 && !canvasLink.linked) { + return { ...paginate([], params), linked: false }; + } + if (!canvasLink.linked) throw notFound(); if (method === "GET" && seg.length === 1) { const kind = params.get("kind"); const rows = [...canvases] .sort((a, b) => Date.parse(b.updated_at) - Date.parse(a.updated_at)) .filter((c) => !kind || c.kind === kind) .map(toCanvas); - return paginate(rows, params); + return { ...paginate(rows, params), linked: true }; } if (method === "DELETE" && seg.length === 2) { const row = canvases.find((c) => c.id === seg[1]); @@ -837,6 +918,97 @@ function route( } } + // Canvas Cloud link ------------------------------------------------------- + if (seg[0] === "canvas-link") { + if (CANVAS_SCENARIO === "error" && method === "GET") { + throw new HttpError(500, "internal", "couldn't unseal the stored Canvas Cloud key"); + } + if (method === "GET" && seg.length === 1) return structuredClone(canvasLink); + if (method === "PUT" && seg.length === 1) { + const raw = String(body.url ?? "").trim(); + const key = String(body.app_key ?? "").trim(); + const fp = typeof body.tls_fingerprint === "string" ? body.tls_fingerprint : ""; + let url: URL; + try { + url = new URL(raw); + } catch { + throw bad("url must be an absolute https:// URL"); + } + if (url.protocol !== "https:" && url.protocol !== "http:") + throw bad("url must be an absolute https:// URL"); + if ( + url.protocol === "http:" && + !["localhost", "127.0.0.1", "[::1]"].includes(url.hostname) + ) { + throw bad("http:// is only allowed for loopback hosts; use https://"); + } + if (!key) throw bad("app_key is required"); + if (fp && !/^[0-9a-f]{64}$/.test(fp)) + throw bad("tls_fingerprint must be a SHA-256 hex digest (64 characters)"); + if (url.hostname.endsWith(".invalid") || url.hostname.startsWith("down.")) + throw bad(`can't reach ${url.host}: connection refused`); + if (!key.startsWith("cva_")) + throw bad(`${url.host} rejected the app key (401 unknown_app)`); + canvasHostDown = false; + canvasLink = { + linked: true, + url: url.origin, + enabled: true, + tls_fingerprint: fp || null, + linked_at: canvasLink.linked ? canvasLink.linked_at : nowIso, + updated_at: nowIso, + updated_by: admin.email, + status: { + reachable: true, + version: CANVAS_VERSION, + error: null, + checked_at: nowIso, + }, + bridge: { + connected: true, + last_event_at: canvasLink.bridge.last_event_at, + last_error: null, + }, + migrations: { ...canvasLink.migrations }, + }; + record( + "canvas_link.set", + "canvas_link", + { url: url.origin, via: "admin_api" }, + nowIso, + ); + return structuredClone(canvasLink); + } + if (method === "DELETE" && seg.length === 1) { + if (!canvasLink.linked) throw notFound(); + const url = canvasLink.url; + canvasHostDown = false; + canvasLink = unlinkedCanvasLink({ ...canvasLink.migrations }); + record("canvas_link.remove", "canvas_link", { url, via: "admin_api" }, nowIso); + return { ok: true }; + } + if (method === "POST" && seg[1] === "test") { + if (!canvasLink.linked) throw bad("no Canvas Cloud is linked"); + const ok = !canvasHostDown; + const latency = ok ? 38 + Math.round(Math.random() * 60) : null; + canvasLink.status = ok + ? { reachable: true, version: CANVAS_VERSION, error: null, checked_at: nowIso } + : { + reachable: false, + version: null, + error: CANVAS_DOWN_ERROR, + checked_at: nowIso, + }; + record("canvas_link.test", "canvas_link", { ok, latency_ms: latency }, nowIso); + return { + ok, + version: ok ? CANVAS_VERSION : null, + latency_ms: latency, + error: ok ? null : CANVAS_DOWN_ERROR, + }; + } + } + if (seg[0] === "pairing-codes") { if (method === "GET" && seg.length === 1) { const userId = params.get("user_id"); diff --git a/portal/src/lib/types.ts b/portal/src/lib/types.ts index ff345cc..917b87a 100644 --- a/portal/src/lib/types.ts +++ b/portal/src/lib/types.ts @@ -54,11 +54,13 @@ export interface PasswordChanged extends Ok { export interface Counts { users: number; devices: number; - canvases: number; + /** From the linked Canvas Cloud; `null` while none is linked or it is unreachable. */ + canvases: number | null; /** Active keys only (not revoked, expired or exhausted). */ access_keys: number; - live_rooms: number; - live_peers: number; + /** Canvas Cloud rooms; `null` while none is linked or it is unreachable. */ + live_rooms: number | null; + live_peers: number | null; } export interface Overview { @@ -227,6 +229,66 @@ export interface CanvasQuery extends PageQuery { kind?: CanvasKind; } +// Canvas Cloud link (canvases live on a linked Canvas Cloud server) ------- + +/** `GET canvases`: the usual page, plus whether a Canvas Cloud is linked at all. */ +export interface CanvasPage extends Page { + /** `false` = no Canvas Cloud linked; the page is then always empty. */ + linked: boolean; +} + +export interface CanvasLinkStatus { + /** `null` until the server has checked once. */ + reachable: boolean | null; + version: string | null; + error: string | null; + checked_at: string | null; +} + +export interface CanvasLinkBridge { + connected: boolean; + last_event_at: string | null; + last_error: string | null; +} + +export interface CanvasLinkMigrations { + migrated: number; + failed: number; + /** Canvases still stored on this server from before Canvas Cloud. */ + legacy_canvases: number; +} + +/** `GET canvas-link`. The app key is write-only and never comes back. */ +export interface CanvasLink { + linked: boolean; + url: string | null; + enabled: boolean; + /** SHA-256 hex of a pinned self-signed certificate, else `null`. */ + tls_fingerprint: string | null; + linked_at: string | null; + updated_at: string | null; + /** Admin email, or `"cli"` for `copper-cloud canvas link`. */ + updated_by: string | null; + status: CanvasLinkStatus; + bridge: CanvasLinkBridge; + migrations: CanvasLinkMigrations; +} + +/** `PUT canvas-link`: always send the key; the server validates before saving. */ +export interface CanvasLinkUpdate { + url: string; + app_key: string; + tls_fingerprint?: string; +} + +/** `POST canvas-link/test`. */ +export interface CanvasLinkTest { + ok: boolean; + version: string | null; + latency_ms: number | null; + error: string | null; +} + // Pairing codes ----------------------------------------------------------- export interface PairingCode { From dd05d5ffb3a9a56bf1407e1cd8cc4f17360c0351 Mon Sep 17 00:00:00 2001 From: Felipe Arce Date: Mon, 19 Oct 2026 19:41:18 +0000 Subject: [PATCH 4/9] Canvas link: reconcile never deletes; sync-users --prune; migrate sends null names The 10-minute reconcile used to delete Canvas Cloud app users whose Copper account was gone, with their canvases. Restoring an older copper-cloud backup would have wiped newer users' canvases on Canvas Cloud. Reconcile now only counts and logs orphans; deleting them is an explicit operator action: copper-cloud canvas sync-users --prune. canvas migrate provisions empty display names as null, like the link does. The migration test database follows the repo's naming (COPPER_CLOUD_TEST_MIGRATE_DATABASE_URL, copper_cloud_test_migrate). --- .github/workflows/ci.yml | 4 +- crates/canvas-link/src/provision.rs | 74 ++++++++++++++++------- crates/canvas-migrate/src/remote.rs | 5 +- crates/canvas-migrate/tests/common/mod.rs | 11 ++-- crates/canvas-migrate/tests/migrate.rs | 2 +- crates/copper-cloud/src/cli.rs | 28 ++++++++- crates/copper-cloud/tests/canvas_link.rs | 14 ++++- 7 files changed, 98 insertions(+), 40 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index e51e1a3..2824649 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -63,7 +63,7 @@ jobs: sudo sed -i '1i host all all 127.0.0.1/32 trust\nhost all all ::1/128 trust' "$HBA" sudo -u postgres psql -qc 'SELECT pg_reload_conf()' createdb -h localhost copper_cloud_test_core - createdb -h localhost ccmig_test + createdb -h localhost copper_cloud_test_migrate psql -h localhost -d copper_cloud_test_core -tAc 'SELECT version()' # Build the portal first so the portal tests run against the real export. - uses: oven-sh/setup-bun@v2 @@ -77,8 +77,6 @@ jobs: echo "no portal/package.json; the binary serves the 'portal not built' page" fi - name: cargo test - env: - CCMIG_TEST_DATABASE_URL: postgres://localhost:5432/ccmig_test run: cargo test --workspace --locked -j4 -- --test-threads=4 - name: CLI smoke test run: | diff --git a/crates/canvas-link/src/provision.rs b/crates/canvas-link/src/provision.rs index 8148c02..692e6c1 100644 --- a/crates/canvas-link/src/provision.rs +++ b/crates/canvas-link/src/provision.rs @@ -6,7 +6,11 @@ //! * [`provision_user`] — one `PUT …/users/{subject}` (JIT after `401 unknown_user`). //! * [`reconcile_loop`] — every 10 min (and whenever the link changes): push users whose //! email / display name / disabled flag changed (a hash per user is kept in memory), -//! disconnect newly disabled users, delete app users whose Copper account is gone. +//! disconnect newly disabled users, and *count* (log) app users whose Copper account is +//! gone. Reconcile never deletes anything: an old copper-cloud backup restored over a +//! newer database would otherwise wipe the newer users' canvases on Canvas Cloud. Removing +//! those orphans is an explicit operator action ([`prune_orphans`], +//! `copper-cloud canvas sync-users --prune`). //! * The outbox (`canvas_link_outbox`): admin edits/deletes enqueue work in their own //! transaction ([`enqueue_user_changed`], [`enqueue_user_deleted`]); [`outbox_loop`] pushes //! it and retries failures with backoff, so nothing is lost while Canvas Cloud is down. @@ -303,7 +307,41 @@ pub async fn remote_users( Ok(out) } -/// One reconcile pass. Returns `(pushed, disconnected, deleted)`. +/// Canvas Cloud subjects of this app that have no Copper user any more. +async fn orphan_ids(link: &CanvasLink, active: &Active) -> anyhow::Result> { + let users = copper_users(&link.state().db).await?; + let ours: HashSet = users.iter().map(|u| u.id.to_string()).collect(); + let remote = remote_users(&active.upstream, crate::upstream::REQUEST_TIMEOUT).await?; + Ok(remote + .keys() + .filter(|s| !ours.contains(*s)) + .filter_map(|s| s.parse::().ok()) + .collect()) +} + +/// Delete Canvas Cloud app users whose Copper account is gone (their canvases there go with +/// them). Operator-initiated only (`copper-cloud canvas sync-users --prune`). Returns the ids +/// deleted and the ids that failed. +pub async fn prune_orphans( + link: &CanvasLink, + active: &Active, +) -> anyhow::Result<(Vec, Vec<(Uuid, String)>)> { + let mut deleted = Vec::new(); + let mut failed = Vec::new(); + for id in orphan_ids(link, active).await? { + match delete_remote_user(&active.upstream, id).await { + Ok(()) => { + link.synced().remove(&id); + deleted.push(id); + } + Err(e) => failed.push((id, e.to_string())), + } + } + Ok((deleted, failed)) +} + +/// One reconcile pass. Returns `(pushed, disconnected, orphaned)`; orphans (Canvas Cloud app +/// users with no Copper account) are only counted and logged, never deleted here. pub async fn reconcile_once( link: &CanvasLink, active: &Active, @@ -333,25 +371,17 @@ pub async fn reconcile_once( disconnected += 1; } } - // App users whose Copper account is gone (e.g. `copper-cloud admin delete-user`). - let ours: HashSet = users.iter().map(|u| u.id.to_string()).collect(); - let remote = remote_users(&active.upstream, crate::upstream::REQUEST_TIMEOUT).await?; - let mut deleted = 0; - for subject in remote.keys().filter(|s| !ours.contains(*s)) { - let Ok(id) = subject.parse::() else { - continue; - }; - match delete_remote_user(&active.upstream, id).await { - Ok(()) => { - deleted += 1; - link.synced().remove(&id); - } - Err(e) => { - tracing::warn!(user_id = %id, error = %e, "removing a deleted user from Canvas Cloud failed"); - } - } + // App users whose Copper account is gone (e.g. deleted with the shell CLI, or a restore of + // an older database): report only. + let orphaned = orphan_ids(link, active).await?.len(); + if orphaned > 0 { + tracing::warn!( + orphaned, + "Canvas Cloud has users of this Copper Cloud that no longer exist here; \ + run `copper-cloud canvas sync-users --prune` to delete them and their canvases" + ); } - Ok((report.provisioned, disconnected, deleted)) + Ok((report.provisioned, disconnected, orphaned)) } /// Background reconcile: right away, then every [`RECONCILE_EVERY`] and on link changes. @@ -365,8 +395,8 @@ pub async fn reconcile_loop(link: CanvasLink) { if let Ok(Some(active)) = link.enabled().await { match reconcile_once(&link, &active).await { Ok((0, 0, 0)) => {} - Ok((pushed, disconnected, deleted)) => { - tracing::info!(pushed, disconnected, deleted, "canvas users reconciled"); + Ok((pushed, disconnected, orphaned)) => { + tracing::info!(pushed, disconnected, orphaned, "canvas users reconciled"); } Err(err) => { tracing::warn!(error = format!("{err:#}"), "canvas user reconcile failed"); diff --git a/crates/canvas-migrate/src/remote.rs b/crates/canvas-migrate/src/remote.rs index 02c14ef..430fd13 100644 --- a/crates/canvas-migrate/src/remote.rs +++ b/crates/canvas-migrate/src/remote.rs @@ -107,7 +107,8 @@ pub(crate) struct Imported { struct BatchUser<'a> { subject: String, email: &'a str, - display_name: &'a str, + /// `null` when empty, exactly like the link's provisioning (no needless updates). + display_name: Option<&'a str>, disabled: bool, id: Uuid, } @@ -204,7 +205,7 @@ impl<'a> Remote<'a> { "users": chunk.iter().map(|u| BatchUser { subject: subject(u.id), email: &u.email, - display_name: &u.display_name, + display_name: Some(u.display_name.as_str()).filter(|n| !n.is_empty()), disabled: u.disabled, id: u.id, }).collect::>(), diff --git a/crates/canvas-migrate/tests/common/mod.rs b/crates/canvas-migrate/tests/common/mod.rs index 7095cf7..ebcda39 100644 --- a/crates/canvas-migrate/tests/common/mod.rs +++ b/crates/canvas-migrate/tests/common/mod.rs @@ -1,5 +1,5 @@ //! Test harness: an isolated schema per test in the native PostgreSQL test database -//! (`CCMIG_TEST_DATABASE_URL`, default `postgres://farce@localhost:5433/ccmig_test`) with +//! (`COPPER_CLOUD_TEST_MIGRATE_DATABASE_URL`, default `postgres://localhost:5432/copper_cloud_test_migrate`) with //! copper-cloud's real migrations applied, legacy canvas rows seeded through the same sealing //! code paths the old `copper-cloud-canvas` crate used, and the mock Canvas Cloud. @@ -21,10 +21,11 @@ use uuid::Uuid; use yrs::updates::decoder::Decode as _; use yrs::{Doc, Map as _, ReadTxn as _, StateVector, Transact as _, Update}; -pub const DEFAULT_DATABASE_URL: &str = "postgres://farce@localhost:5433/ccmig_test"; +pub const DEFAULT_DATABASE_URL: &str = "postgres://localhost:5432/copper_cloud_test_migrate"; pub fn database_url() -> String { - std::env::var("CCMIG_TEST_DATABASE_URL").unwrap_or_else(|_| DEFAULT_DATABASE_URL.to_owned()) + std::env::var("COPPER_CLOUD_TEST_MIGRATE_DATABASE_URL") + .unwrap_or_else(|_| DEFAULT_DATABASE_URL.to_owned()) } /// The ledger table as cc-link's `0030_canvas_link.sql` defines it (created here only if the @@ -53,9 +54,7 @@ impl TestDb { .acquire_timeout(Duration::from_secs(10)) .connect_with(base.clone()) .await - .expect( - "connecting to the test database (createdb -w -h localhost -p 5433 ccmig_test)", - ); + .expect("connecting to the test database (createdb -w copper_cloud_test_migrate)"); sweep_stale_schemas(&admin).await; let now = std::time::SystemTime::now() .duration_since(std::time::UNIX_EPOCH) diff --git a/crates/canvas-migrate/tests/migrate.rs b/crates/canvas-migrate/tests/migrate.rs index c960ac0..24ca16d 100644 --- a/crates/canvas-migrate/tests/migrate.rs +++ b/crates/canvas-migrate/tests/migrate.rs @@ -242,7 +242,7 @@ async fn full_run_migrates_everything_verified() { .iter() .find(|p| p["subject"] == w.carol.to_string()) .unwrap(); - assert_eq!(carol["display_name"], ""); + assert!(carol["display_name"].is_null(), "empty names go as null"); assert_eq!(carol["id"], w.carol.to_string()); let dave = s .user_payloads diff --git a/crates/copper-cloud/src/cli.rs b/crates/copper-cloud/src/cli.rs index 80c0e8d..66931b7 100644 --- a/crates/copper-cloud/src/cli.rs +++ b/crates/copper-cloud/src/cli.rs @@ -98,7 +98,12 @@ pub enum CanvasCommand { /// Show the link (key masked) and whether Canvas Cloud is reachable. Status, /// Provision every user on the linked Canvas Cloud now. - SyncUsers, + SyncUsers { + /// Also delete Canvas Cloud users of this link whose Copper account no longer exists + /// (and their canvases there). The background reconcile only reports them. + #[arg(long)] + prune: bool, + }, /// Copy legacy canvases (this database's canvas tables) into the linked Canvas Cloud, /// verifying each by digest. Legacy rows are never modified. Migrate(CanvasMigrateArgs), @@ -585,13 +590,30 @@ async fn canvas(cfg: Config, cmd: CanvasCommand) -> anyhow::Result { ExitCode::SUCCESS }) } - CanvasCommand::SyncUsers => { + CanvasCommand::SyncUsers { prune } => { let active = link .current() .await .map_err(api_err)? .context("no Canvas Cloud is linked (copper-cloud canvas link …)")?; - sync_users(&link, &active).await + let code = sync_users(&link, &active).await?; + if !prune { + return Ok(code); + } + let (deleted, failed) = + copper_cloud_canvas_link::provision::prune_orphans(&link, &active).await?; + println!( + "pruned {} Canvas Cloud user(s) with no Copper account", + deleted.len() + ); + for (id, err) in &failed { + println!(" failed {id}: {err}"); + } + Ok(if failed.is_empty() { + code + } else { + ExitCode::FAILURE + }) } CanvasCommand::Migrate(args) => migrate_canvases(&state, &link, args).await, } diff --git a/crates/copper-cloud/tests/canvas_link.rs b/crates/copper-cloud/tests/canvas_link.rs index 35f609c..f525c31 100644 --- a/crates/copper-cloud/tests/canvas_link.rs +++ b/crates/copper-cloud/tests/canvas_link.rs @@ -1007,7 +1007,7 @@ async fn admin_canvases_overview_people_and_user_hooks() { } #[tokio::test] -async fn reconcile_pushes_changes_and_removes_deleted_users() { +async fn reconcile_pushes_changes_and_only_reports_deleted_users() { let (s, _c, m, ana, bob) = setup().await; let active = s.link.enabled().await.unwrap().unwrap(); @@ -1035,14 +1035,22 @@ async fn reconcile_pushes_changes_and_removes_deleted_users() { .execute(&s.state.db) .await .unwrap(); - let (pushed, disconnected, deleted) = + let (pushed, disconnected, orphaned) = copper_cloud_canvas_link::provision::reconcile_once(&s.link, &active) .await .unwrap(); - assert_eq!((pushed, disconnected, deleted), (2, 1, 1)); + assert_eq!((pushed, disconnected, orphaned), (2, 1, 1)); assert_eq!(m.st.user(&ana.user_id).unwrap().email, "ana2@example.com"); assert!(m.st.user(&carol.user_id).unwrap().disabled); assert!(m.st.disconnects.lock().unwrap().contains(&carol.user_id)); + // Reconcile never deletes: bob (gone from Copper) is still on Canvas Cloud... + assert!(m.st.has_user(&bob.user_id)); + // ...until an operator prunes explicitly. + let (deleted, failed) = copper_cloud_canvas_link::provision::prune_orphans(&s.link, &active) + .await + .unwrap(); + assert!(failed.is_empty(), "{failed:?}"); + assert_eq!(deleted.len(), 1); assert!(!m.st.has_user(&bob.user_id)); } From b7183246a952c822828cdd51b16a3b6d44ce1999 Mon Sep 17 00:00:00 2001 From: Felipe Arce Date: Mon, 19 Oct 2026 19:41:18 +0000 Subject: [PATCH 5/9] docs: canvas link, migration guide and runbook; canvas docs point to Canvas Cloud New docs/canvas-link.md (architecture, proxy and error mapping, WebSocket pipe, event bridge, provisioning, TLS, admin API, CLI, security) and docs/canvas-migration.md (what is preserved, runbook, re-runs, ledger, troubleshooting, rollback). canvas.md and canvas-protocol.md become pointers to Canvas Cloud with their anchors kept; architecture, api, admin-api, operations, security, install, README and deploy/aws updated. --- README.md | 47 +++- deploy/aws/README.md | 32 +++ docs/admin-api.md | 93 ++++++- docs/api.md | 43 ++- docs/architecture.md | 53 +++- docs/canvas-link.md | 559 +++++++++++++++++++++++++++++++++++++++ docs/canvas-migration.md | 387 +++++++++++++++++++++++++++ docs/canvas-protocol.md | 228 ++-------------- docs/canvas.md | 400 +++++----------------------- docs/install.md | 15 ++ docs/operations.md | 66 ++++- docs/security.md | 39 ++- 12 files changed, 1363 insertions(+), 599 deletions(-) create mode 100644 docs/canvas-link.md create mode 100644 docs/canvas-migration.md diff --git a/README.md b/README.md index 9b2a94e..ef9ce09 100644 --- a/README.md +++ b/README.md @@ -1,13 +1,19 @@ # Copper Cloud -Self-hosted sync + collaborative canvas server for the [Copper](https://github.com/copper-browser/Copper) +Self-hosted sync server for the [Copper](https://github.com/copper-browser/Copper) browser. One Rust daemon (`copper-cloud`), one Postgres. Install it on any Linux VM with one command, or deploy to AWS with Terraform (`deploy/aws`). What it syncs between your Coppers: spaces + pinned tabs, settings (safe subset), bookmarks, -history (append-only; people can delete theirs), each device's open tabs, and shared canvases -(live Yjs rooms). It can also hand every signed-in Copper the instance's AI keys (Jev + LLM -router), set once by the admin, so nobody pastes keys by hand. +history (append-only; people can delete theirs) and each device's open tabs. It can also hand +every signed-in Copper the instance's AI keys (Jev + LLM router), set once by the admin, so +nobody pastes keys by hand. + +Shared canvases (live Yjs rooms, chat, mentions, invites, share links) are served by a linked +[Canvas Cloud](https://github.com/copper-browser/canvas-cloud) (since 0.9.0): copper-cloud +proxies the unchanged canvas API to it as the signed-in user, so Coppers need no update. See +[docs/canvas-link.md](docs/canvas-link.md); upgrading a 0.8.x instance with canvases: +[docs/canvas-migration.md](docs/canvas-migration.md). ## Install on a VM (Ubuntu 22.04/24.04, Debian 12, Amazon Linux 2023, Fedora) @@ -46,6 +52,17 @@ link code for everyone) with `COPPER_CLOUD_ACCESS_MODE=open`. Re-running the ins in place and keeps keys, certificate, admins and data. Details: [docs/install.md](docs/install.md), [docs/operations.md](docs/operations.md#admin-portal). +For canvases, install a [Canvas Cloud](https://github.com/copper-browser/canvas-cloud) (the +small setup: same host, `127.0.0.1:8480`, TLS off), create an app key there with +`canvas-cloud apps create --name "Copper Cloud" --allow-import`, and link it: + +```sh +sudo copper-cloud canvas link --url http://127.0.0.1:8480 --key-file /root/canvas-app.key +``` + +Until then canvas requests answer `503 canvas_unavailable` and `doctor` warns. Details: +[docs/canvas-link.md](docs/canvas-link.md#linking). + AWS (EC2 + RDS + Elastic IP; link code and admin password in SSM): see [deploy/aws/README.md](deploy/aws/README.md). @@ -82,7 +99,7 @@ on `127.0.0.1:9464`. Every config key can be overridden with `COPPER_CLOUD_ | `copper-cloud [serve]` | run the server (applies migrations first; `--no-migrate` to skip) | | `copper-cloud migrate` | apply database migrations | | `copper-cloud link-code` | print the link code | -| `copper-cloud doctor` | check config, database, migrations, TLS, listener | +| `copper-cloud doctor` | check config, database, migrations, TLS, listener, Canvas Cloud link | | `copper-cloud healthcheck --wait 60` | wait for `/healthz` (pinned TLS) | | `copper-cloud tls-init [--force] [--name N]` | create the self-signed certificate | | `copper-cloud init-config --write PATH` | write a config with fresh keys | @@ -91,6 +108,9 @@ on `127.0.0.1:9464`. Every config key can be overridden with `COPPER_CLOUD_ | `copper-cloud admin access-mode\|set-access-mode open\|directory\|create-access-key` | who may pass the gate | | `copper-cloud admin history delete --user EMAIL\|UUID [--since T] [--until T] [--device ID] [--dry-run]` | delete (or count) a user's synced history wholesale, by time window/device only (never by content) | | `copper-cloud intelligence set\|show\|clear\|enable\|disable` | cloud-wide Jev + LLM router keys every signed-in Copper receives (keys from files/stdin, shown masked), plus the org-wide agent round budget (`--agent-max-turns`, `clear --agent`) | +| `copper-cloud canvas link --url URL --key-file F [--fingerprint HEX] [--no-sync]` | link a Canvas Cloud (app key from a file/stdin, never argv) and provision every user | +| `copper-cloud canvas unlink\|status\|sync-users [--prune]` | remove the link / show it and test it / push every user now (`--prune`: also delete Canvas Cloud users that no longer exist here) | +| `copper-cloud canvas migrate [--dry-run] [--only ID]… [--replace] [--concurrency N] [--json]` | copy 0.8.x canvases into the linked Canvas Cloud, verified by digest ([docs/canvas-migration.md](docs/canvas-migration.md)) | | `copper-cloud version` | version | All commands take `--config PATH` (default `/etc/copper-cloud/copper-cloud.toml`, env @@ -101,7 +121,12 @@ All commands take `--config PATH` (default `/etc/copper-cloud/copper-cloud.toml` - [Architecture](docs/architecture.md) — components, data model, request path - [HTTP API](docs/api.md) — every endpoint with requests and responses - [Admin API](docs/admin-api.md) — the admin portal's `/admin/api/*` contract -- [Canvas](docs/canvas.md) — canvases REST + WebSocket rooms +- [Canvas link](docs/canvas-link.md) — how canvases are served by a linked Canvas Cloud: + proxy, events, provisioning, linking, security +- [Canvas migration](docs/canvas-migration.md) — moving 0.8.x canvases to Canvas Cloud + (runbook, rollback) +- [Canvas](docs/canvas.md) · [Canvas protocol](docs/canvas-protocol.md) — pointers to the + canvas API, now documented in Canvas Cloud - [Security](docs/security.md) — threat model, access keys, pairing, admin sessions, encryption at rest, TLS pinning - [Install](docs/install.md) — VM one-liner, manual install, configuration reference @@ -114,14 +139,16 @@ All commands take `--config PATH` (default `/etc/copper-cloud/copper-cloud.toml` ```sh cargo fmt --all --check cargo clippy --workspace --all-targets -- -D warnings -createdb copper_cloud_test_core && createdb copper_cloud_test_canvas # once +createdb copper_cloud_test_core # once +createdb copper_cloud_test_migrate # once, for crates/canvas-migrate cargo test --workspace ``` Tests use a real native Postgres (`COPPER_CLOUD_TEST_DATABASE_URL`, default -`postgres://localhost:5432/copper_cloud_test_core`; the canvas crate's tests use -`COPPER_CLOUD_TEST_CANVAS_DATABASE_URL`, default `…/copper_cloud_test_canvas`); never a -container. Releases: push a +`postgres://localhost:5432/copper_cloud_test_core`; the migration tests use +`COPPER_CLOUD_TEST_MIGRATE_DATABASE_URL`, default +`postgres://localhost:5432/copper_cloud_test_migrate`); never a +container. The canvas-link tests run against an in-process mock Canvas Cloud. Releases: push a `vX.Y.Z` tag matching the crate version; CI builds `copper-cloud-X.Y.Z-linux-{x86_64,aarch64}.tar.gz` (+ `.sha256`) and publishes the GitHub release. diff --git a/deploy/aws/README.md b/deploy/aws/README.md index 576aa67..b786d1f 100644 --- a/deploy/aws/README.md +++ b/deploy/aws/README.md @@ -186,6 +186,38 @@ and existing accounts keep working. One thing still changes: without a domain, e generates a **new self-signed certificate**. That changes the link code's `fp=`, so linked Coppers have to paste the new link code (`./link-code.sh `). +## Canvas Cloud + +Since copper-cloud 0.9.0, canvases are served by a linked +[Canvas Cloud](https://github.com/copper-browser/canvas-cloud) ([docs/canvas-link.md](../../docs/canvas-link.md)). +This Terraform does **not** create one. A new deployment runs without it: canvas requests +answer `503 canvas_unavailable` and `doctor` warns. + +- Don't install Canvas Cloud on this VM. The VM is replaced on every `--release`, `--domain` + or `--key-name` change, and its root volume goes with it. Run Canvas Cloud on a durable + host with its own database and backups, and link it from an SSM session: + + ```bash + sudo copper-cloud canvas link --url https://canvas.example.com --key-file /root/canvas-app.key \ + [--fingerprint ] + ``` + +- The link lives in RDS (`canvas_link`, app key sealed under the master key). It survives + VM replacement, so you only link once. +- The app SG only restricts inbound traffic. The VM must be able to reach the Canvas Cloud's + port outbound, and the Canvas Cloud must accept connections from the EIP. +- **Upgrading a deployment that has canvases from 0.8.x** to 0.9.0: + 1. Stop copper-cloud on the old VM (`sudo systemctl stop copper-cloud` in an SSM session) + so the snapshot holds every canvas edit, then take an RDS snapshot: + `aws rds create-db-snapshot --db-instance-identifier copper-cloud- --db-snapshot-identifier copper-cloud--pre-0-9`. + 2. Run `./up.sh --release v0.9.0`. The new VM comes up with canvases unavailable. + 3. In an SSM session, follow steps 2–7 of the runbook in + [docs/canvas-migration.md](../../docs/canvas-migration.md#runbook): stop copper-cloud, + then `canvas link`, `canvas migrate --dry-run`, `canvas migrate`, start, and verify. + Rollback: [docs/canvas-migration.md › Rollback](../../docs/canvas-migration.md#rollback) + on the VM, or `./up.sh --release v0.8.1` after removing migration `0030` as + described there (or restore the snapshot). + ## Costs (us-east-1, on-demand, approximate) | Item | $/month | diff --git a/docs/admin-api.md b/docs/admin-api.md index e5fb2c6..d06e72b 100644 --- a/docs/admin-api.md +++ b/docs/admin-api.md @@ -90,7 +90,8 @@ sorted **newest first**. Response envelope: ### Audit Every mutation (login, logout, password, settings, AI keys, key create/revoke, user update / -delete / reset-password, device delete, canvas delete, pairing-code revoke) writes an +delete / reset-password, device delete, canvas delete, Canvas Cloud link set / remove / test, +pairing-code revoke) writes an `admin_audit` row, readable with `GET audit`. Users fetching the AI keys are recorded too (`intelligence.read`). @@ -171,6 +172,9 @@ redirect — check the code). `new` shorter than 10 chars → `400`. ``` * `counts.access_keys` = **active** keys (not revoked, not expired, not exhausted). +* `counts.canvases`, `counts.live_rooms`, `counts.live_peers` come from the linked Canvas + Cloud (`GET /v1/apps/self/stats`, 5 s budget) and are `null` when no Canvas Cloud is linked + or it does not answer. * `tls.mode`: `"self-signed" | "acme" | "off"`; `fingerprint` is `null` for ACME (publicly trusted) or when no certificate file is readable. @@ -377,8 +381,9 @@ User object: ``` `last_seen_at` = latest device activity (`null` if never). `canvas_count` = canvases the user -is a member of (including their Personal canvas, which is created the first time their Copper -lists canvases). There is no "create user" call: people sign up from Copper with the access +is a member of on the linked Canvas Cloud (including their Personal canvas, which is created +the first time their Copper lists canvases), read from `GET /v1/apps/self/users` and cached +30 s; `null` when no Canvas Cloud is linked or it does not answer. There is no "create user" call: people sign up from Copper with the access key you gave them (directory mode) or the instance link code (open mode). ### `GET /admin/api/users?limit=&offset=&q=` @@ -394,15 +399,18 @@ key you gave them (directory mode) or the instance link code (open mode). ``` Disabling revokes all of the user's sessions immediately, disconnects their live canvas -peers and blocks sign-in (`401 account_disabled` in Copper). `display_name` must not be empty. +peers (queued to the linked Canvas Cloud: `disconnect_user`, plus `upsert_user` with +`disabled: true` — [provisioning](canvas-link.md#user-provisioning)) and blocks sign-in (`401 account_disabled` in Copper). `display_name` must not be empty. Unknown id (or not a UUID) → `404`. ### `DELETE /admin/api/users/{id}` Permanently deletes the user and everything they own: sessions, devices, sync docs, history, -pairing codes, access keys bound to their email, their canvases (Personal and shared ones -they own — including other members' access to them) and their memberships/invites. Live -canvas rooms of deleted canvases are closed. +pairing codes, access keys bound to their email, their legacy (pre-0.9) canvas rows and +legacy invites addressed to their email. On the linked Canvas Cloud (queued in the same +transaction, `delete_user`) their app user is deleted together with the canvases they own +(Personal and shared — including other members' access to them) and their memberships; +Canvas Cloud closes those rooms. ```json { "ok": true } @@ -474,14 +482,27 @@ Canvas object (content is never exposed): `kind`: `"personal" | "shared"`. `share_link_count` counts active (not revoked) canvas share links; link tokens and canvas content are never exposed to the admin API. +Canvases live on the linked Canvas Cloud; these endpoints read and delete them there +(`/v1/apps/self/canvases`). + ### `GET /admin/api/canvases?limit=&offset=&kind=` -`kind` optional filter. Pagination envelope. +`kind` optional filter (`personal` | `shared`, else `400`). Pagination envelope plus +`linked`: + +```json +{ "items": [ /* canvas objects */ ], "total": 15, "limit": 100, "offset": 0, "linked": true } +``` + +No Canvas Cloud linked → `200` with `items: []`, `total: 0`, `linked: false`. Linked but +unreachable → `502 {"error":"canvas_upstream",…}`. ### `DELETE /admin/api/canvases/{id}` -Deletes the canvas, its history, members and invites; connected peers are disconnected. A -deleted Personal canvas is recreated empty the next time its owner opens it. +Deletes the canvas on Canvas Cloud, with its history, members and invites; connected peers +are disconnected (`4404`). A deleted Personal canvas is recreated empty the next time its +owner opens it. Unknown id → `404`; no link → `503 canvas_unavailable`; unreachable → +`502 canvas_upstream`. ```json { "ok": true } @@ -489,6 +510,58 @@ deleted Personal canvas is recreated empty the next time its owner opens it. --- +## Canvas Cloud link + +The linked Canvas Cloud that serves canvases ([canvas-link.md](canvas-link.md#admin-api) has +the full reference with examples). + +### `GET /admin/api/canvas-link` + +```json +{ + "linked": true, + "url": "http://127.0.0.1:8480", + "enabled": true, + "tls_fingerprint": null, + "linked_at": "2026-10-19T18:02:11Z", + "updated_at": "2026-10-19T18:02:11Z", + "updated_by": "cli", + "status": { "reachable": true, "version": "0.1.0", "error": null, "checked_at": "2026-10-19T18:30:00Z" }, + "bridge": { "connected": true, "last_event_at": "2026-10-19T18:29:41Z", "last_error": null }, + "migrations": { "migrated": 36, "failed": 0, "legacy_canvases": 36 } +} +``` + +Re-probes when the last check is older than 15 s. Not linked: `linked: false`, `enabled: +false`, the other fields `null`, `bridge.connected: false`. The app key is never returned. + +### `PUT /admin/api/canvas-link` + +```json +// request +{ "url": "https://canvas.example.com", "app_key": "cva_…", "tls_fingerprint": null } +// 200 → the GET body +``` + +Probes `GET /v1/info` and `GET /v1/apps/self/stats` with the key first; nothing is stored if +the input is invalid or the probe fails (`400 bad_request` with the reason). Users are +provisioned in the background. + +### `DELETE /admin/api/canvas-link` + +`{ "ok": true }`. Canvas routes answer `503 canvas_unavailable` from then on; nothing on +Canvas Cloud is deleted. + +### `POST /admin/api/canvas-link/test` + +```json +{ "ok": true, "version": "0.1.0", "latency_ms": 4, "error": null } +``` + +Always `200`; `ok: false` with `error` when not linked or unreachable. + +--- + ## Pairing codes Single-use codes a signed-in Copper mints (`POST /v1/auth/pairing`) to link and sign in diff --git a/docs/api.md b/docs/api.md index 0d489d3..651eccb 100644 --- a/docs/api.md +++ b/docs/api.md @@ -54,6 +54,11 @@ Always `{"error": "", "message": ""}`: | 413 | `payload_too_large` | body or payload over the configured limit | | 429 | `rate_limited` | auth rate limit, or a per-user limit (mentions, history delete) (header `Retry-After: 60`) | | 500 | `internal` | server bug / database failure (details only in server logs) | +| 502 | `canvas_upstream` | canvas routes only: the linked Canvas Cloud is unreachable, timed out, or rejected the app key ([canvas-link.md](canvas-link.md#status-codes)) | +| 503 | `canvas_unavailable` | canvas routes only: no Canvas Cloud is linked | + +Canvas routes otherwise return Canvas Cloud's status and body verbatim (except `401`, which +is never forwarded). ### Rate limit @@ -95,8 +100,9 @@ Key only (no session). Lets a client validate a link code before showing account "version": "0.8.1", "signup": true, "access_mode": "directory", - "features": ["history_delete"], - "limits": { "max_blob_bytes": 8000000, "max_history_batch": 2000, "max_history_entry_bytes": 16384 } + "features": ["history_delete", "canvas_link"], + "limits": { "max_blob_bytes": 8000000, "max_history_batch": 2000, "max_history_entry_bytes": 16384 }, + "canvas": { "linked": true } } ``` @@ -111,6 +117,11 @@ only ever grows, and an absent field (older servers) means none of them: | Feature | Since | Means | |---|---|---| | `history_delete` | 0.8.0 | [`DELETE /v1/sync/history`](#delete-v1synchistory) and the `history_deleted` event | +| `canvas_link` | 0.9.0 | canvases are served by a linked Canvas Cloud through this server ([canvas-link.md](canvas-link.md)); the canvas API itself is unchanged | + +`canvas` (0.9.0+): `linked` is `true` while a Canvas Cloud is linked and enabled. When it is +`false`, every canvas route answers `503 canvas_unavailable`. It says nothing about +reachability (that shows up as `502 canvas_upstream`). --- @@ -511,8 +522,8 @@ data: {"skipped":12} - A `: keepalive` comment every 15 s. The stream ends on server shutdown, when the session is revoked (checked every 5 min), or on network loss — reconnect with backoff and re-pull. - Events carry no payloads; fetch the doc/history after an event. `canvas` kinds are listed in - [canvas.md](canvas.md#change-notifications) (0.6.0 adds `mention` and `mention_read`: refetch - `GET /v1/mentions`). + [canvas-link.md](canvas-link.md#event-bridge) (0.6.0 adds `mention` and `mention_read`: refetch + `GET /v1/mentions`). Since 0.9.0 they are bridged from the linked Canvas Cloud; frames are unchanged. --- @@ -564,8 +575,14 @@ LiteLLM or Jev key. Same gate and session as every other `/v1` user endpoint (ga ## Canvases 🔒 -Mounted under the same `/v1` gate and session auth; documented in -[canvas.md](canvas.md): +Mounted under the same `/v1` gate and session auth. Since 0.9.0 they are served by the linked +Canvas Cloud: copper-cloud forwards each request as the signed-in user and returns the answer +unchanged ([canvas-link.md](canvas-link.md#the-proxy)); the routes are documented in Canvas +Cloud's [docs/canvas.md](https://github.com/copper-browser/canvas-cloud/blob/main/docs/canvas.md) +(pointers: [canvas.md](canvas.md)). Request bodies: 4 MiB for `/ops`, 64 KiB otherwise. No +Canvas Cloud linked → `503 canvas_unavailable`; unreachable, timed out, or the app key was +rejected → `502 canvas_upstream`; a Canvas Cloud `401` is never passed on (Copper would sign +out). Routes: `GET/POST /v1/canvases`, `GET/PATCH/DELETE /v1/canvases/{id}`, `GET /v1/canvases/{id}/members`, `DELETE /v1/canvases/{id}/members/{user_id}`, `GET/POST /v1/canvases/{id}/invites`, @@ -576,12 +593,15 @@ Mounted under the same `/v1` gate and session auth; documented in `GET /v1/people`, `GET /v1/canvases/{id}/ws` (WebSocket, Yjs sync protocol), `GET /v1/canvases/{id}/state`, `GET /v1/canvases/{id}/read`, `POST /v1/canvases/{id}/ops`, and chat mentions (0.6.0): `POST /v1/canvases/{id}/mentions`, -`GET /v1/mentions`, `POST /v1/mentions/read` ([canvas.md](canvas.md#chat-and-mentions)). +`GET /v1/mentions`, `POST /v1/mentions/read` ([Canvas Cloud › Chat and mentions](https://github.com/copper-browser/canvas-cloud/blob/main/docs/canvas.md#chat-and-mentions)). ### People `GET /v1/people` is available to any signed-in user on the instance. It returns active accounts -(excluding the caller), sorted by display name: +(excluding the caller), sorted by display name. Through the link these are the Canvas Cloud's +enabled users: on a Canvas Cloud used only by this copper-cloud, the instance's Copper +accounts; a Canvas Cloud with other accounts (its own sign-ups, other linked apps) lists those +too: ```json {"people":[{"id":"uuid","display_name":"Ada","email":"ada@example.com"}]} @@ -608,8 +628,11 @@ member; unknown or revoked tokens return `404 {"error":"link_not_found"}`. Joini a pending email invite, increments `uses`, and returns the normal canvas list item. Link tokens are 32 random bytes encoded base64url without padding and stored only as SHA-256 digests. -`GET /join/{token}` is an ungated, no-store HTML handoff page for web links; it does not look up -the token or reveal canvas data. +`GET /join/{token}` is an ungated, no-store HTML handoff page for web links, served by +copper-cloud itself (not proxied); it does not look up the token or reveal canvas data. + +The admin endpoints that manage the link (`GET/PUT/DELETE /admin/api/canvas-link`, +`POST /admin/api/canvas-link/test`) are in [admin-api.md](admin-api.md#canvas-cloud-link). --- diff --git a/docs/architecture.md b/docs/architecture.md index dec5a8f..5eec62c 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -6,12 +6,22 @@ │ Sync engine: PUT docs, POST history, │ X-Copper- │ observe (span + metrics) → instance gate → │ │ SSE /v1/sync/events → pull │ Instance + │ auth rate limit → routes │ │ Canvas host: WebSocket relay │ Bearer token │ /v1/auth /v1/devices /v1/sync /v1/intelligence │ -└────────────────────────────────────────┘ │ /v1/canvases /v1/invites /v1/mentions (canvas) │ - │ Postgres (local or RDS) │ - │ /metrics on 127.0.0.1:9464, JSON logs → journald │ +└────────────────────────────────────────┘ │ /v1/canvases /v1/invites /v1/mentions … ──┐ │ + │ (canvas link: proxy, ws pipe, events) │ │ + │ Postgres (local or RDS) │ │ + │ /metrics on 127.0.0.1:9464, JSON logs │ │ + └─────────────────────────────────────────────┼────┘ + app key + X-Canvas-User, SSE events │ + ▼ + ┌──────────── Canvas Cloud (separate server) ──────┐ + │ canvases REST + Yjs rooms, chat, mentions, │ + │ invites, share links; its own Postgres │ └──────────────────────────────────────────────────┘ ``` +Canvases are served by a linked Canvas Cloud ([copper-browser/canvas-cloud](https://github.com/copper-browser/canvas-cloud)); +copper-cloud proxies the canvas API to it as the signed-in user — see [canvas-link.md](canvas-link.md). + ## Crates | Crate | Role | @@ -19,7 +29,10 @@ | `crates/copper-cloud` (bin + lib) | CLI (clap), `serve` bootstrap, graceful shutdown, `doctor`, `admin`, admin API (`/admin/api`), embedded admin portal (rust-embed of `portal/out`), migrations dir | | `crates/core` (`copper-cloud-core`) | config, db pool + embedded migrations, crypto, access mode + access keys + gate, auth extractor + routes, pairing codes, sync docs/history/SSE, cloud-wide intelligence keys, events, TLS, link codes, observability, app composition | | `portal/` | Next.js static export (the admin portal), built with bun into `portal/out` before cargo in CI/release | -| `crates/canvas` (`copper-cloud-canvas`) | canvases REST + Yjs rooms (see [canvas.md](canvas.md)) | +| `crates/canvas-link` (`copper-cloud-canvas-link`) | the canvas link: settings (sealed app key), REST proxy + WebSocket pipe to Canvas Cloud, event bridge, user provisioning (batch, JIT, reconcile, outbox), `/join/{token}` landing page, `GET /v1/info` `canvas` field, admin/CLI/doctor helpers (see [canvas-link.md](canvas-link.md)) | +| `crates/canvas-migrate` (`copper-cloud-canvas-migrate`) | `copper-cloud canvas migrate`: reads the legacy canvas tables, streams them to Canvas Cloud's import, verifies by digest (see [canvas-migration.md](canvas-migration.md)) | + +The canvas engine (`crates/canvas`, `yrs`) was removed in 0.9.0; it lives on in Canvas Cloud. Stack: tokio, axum 0.8, axum-server (rustls, `ring` provider), rustls-acme, rcgen, sqlx 0.8 (Postgres, runtime queries — no DB needed at compile time), argon2, aes-gcm, hkdf, sha2, @@ -27,16 +40,22 @@ governor, tracing (+ JSON), metrics + Prometheus exporter. `unsafe` is forbidden ## Request path -`copper_cloud::build_app` = `copper_cloud_core::app_with(state, canvas::router(), root)`: +`copper_cloud::build_app` = `copper_cloud_core::app_with(state, link.router(), root)` wrapped +in the link's `info_layer`, where `link = CanvasLink::new(state)`; `link.start()` spawns the +event bridge, the user reconcile loop and the outbox worker: ``` Router ├── GET /healthz (not gated) -├── /v1 = core::router() ⊕ canvas::router() (fallback → 404 JSON) +├── /v1 = core::router() ⊕ link.router() (fallback → 404 JSON) +│ link.router(): /canvases, /canvases/{*rest}, /people, /invites, /invites/{*rest}, +│ /canvas-links/{*rest}, /mentions, /mentions/{*rest} (any method) → Canvas Cloud │ POST /v1/auth/pair is the only /v1 route without the gate header ├── /admin/api = admin_api::router() (cookie sessions + CSRF header; fallback → 404 JSON) +├── GET /join/{token} (share-link landing page; not gated) └── fallback → portal::serve (embedded portal/out; /v1x, /metrics… → 404 JSON) layers (outermost first): + info_layer adds "canvas": {"linked": bool} to successful GET /v1/info responses observe::track span {method, path, route, ip, user_id, status, latency_ms, error, error_message (400s)}; http_* metrics instance_gate /v1 only: X-Copper-Instance = instance key (open mode, constant-time) @@ -46,7 +65,7 @@ layers (outermost first): its own bucket. GCRA keyed by client IP (IPv6 /64) ``` -Handlers authenticate with the `AuthUser` extractor (shared with the canvas crate): one SQL +Handlers authenticate with the `AuthUser` extractor (the canvas proxy uses it too): one SQL round trip that verifies `sha256(token)`, checks expiry and `users.disabled`, and — at most once a minute per session — slides `sessions.expires_at` and touches `devices.last_seen_at` (data-modifying CTEs in the same statement). The result is cached in request extensions. @@ -71,11 +90,19 @@ it. The query string is never logged (it can carry `?token=`). | `admins` | `id`; `lower(email)` unique | portal admin accounts (Argon2id), `last_login_at` | | `admin_sessions` | `id`; `token_sha256` unique | `cc_admin` cookie sessions, fixed 7-day `expires_at` | | `admin_audit` | `id` bigserial | every admin mutation: `admin_id`, `action`, `target`, `detail` jsonb, `at` | - -Canvas tables (`0100+` migrations) are owned by the canvas crate. All migrations live in +| `canvas_link` | `id` = 1 (singleton) | the linked Canvas Cloud: `url`, sealed `app_key_sealed` + `data_key_wrapped`, `tls_fingerprint` (pin or NULL), `enabled`, `linked_at`, `updated_at/by` | +| `canvas_link_outbox` | `id` bigserial | queued provisioning: `kind` (`upsert_user` / `delete_user` / `disconnect_user`), `subject` (Copper user id), `attempts`, `next_attempt_at`, `last_error` | +| `canvas_migrations` | `canvas_id` | migration ledger: `status` (`imported` / `migrated` / `failed`), `digest`, `updates`, `migrated_at`, `detail` | + +The three canvas-link tables come from `0030_canvas_link.sql`. The legacy canvas tables +(`0100`–`0103`: `canvases`, `canvas_members`, `canvas_invites`, `canvas_updates`, +`canvas_snapshots`, `canvas_share_links`, `canvas_mentions`) are still created — sqlx needs the +files on every existing deployment — but 0.9.0 only reads them, in `canvas migrate`. All migrations live in `crates/copper-cloud/migrations` and are embedded once as `copper_cloud_core::db::MIGRATOR`; `serve` applies pending migrations at startup (advisory-locked), `migrate` does it explicitly. -Every user-owned row cascades from `users`, so `admin delete-user` removes everything. +Every user-owned row cascades from `users`, so `admin delete-user` removes everything here; the +user's Canvas Cloud account and canvases are removed through the link (admin API: outbox at +once; shell CLI deletes: `copper-cloud canvas sync-users --prune`, never automatically). ## Sync semantics @@ -92,7 +119,7 @@ Every user-owned row cascades from `users`, so `admin delete-user` removes every reused, so cursors survive deletes. - **Events**: an in-process per-user `tokio::sync::broadcast` fan-out (`Events`). Doc writes publish `doc`, history appends publish `history`, history deletes publish `history_deleted`, - the canvas crate publishes `canvas`. SSE streams turn them into `event:` frames with a 15 s + the canvas link's event bridge publishes `canvas` (from Canvas Cloud's app event stream). SSE streams turn them into `event:` frames with a 15 s keepalive comment; a lagging stream gets `resync`. Single-process by design (one daemon per instance). @@ -118,8 +145,8 @@ HTTP/2 and HTTP/1.1 are negotiated via ALPN. `serve`: load config → tracing → metrics recorder → connect Postgres (retries up to 60 s) → migrate → state → spawn metrics listener + hourly expired-session purge → serve. SIGTERM / -SIGINT: trigger the process-wide shutdown signal (ends SSE streams; canvas rooms close their -peers with 1001), stop accepting, drain in-flight requests for up to 10 s, close the pool. +SIGINT: trigger the process-wide shutdown signal (ends SSE streams; proxied canvas WebSockets +are closed with 1001 on both sides), stop accepting, drain in-flight requests for up to 10 s, close the pool. ## Performance notes diff --git a/docs/canvas-link.md b/docs/canvas-link.md new file mode 100644 index 0000000..8b9c5d9 --- /dev/null +++ b/docs/canvas-link.md @@ -0,0 +1,559 @@ +# Canvas link + +Since 0.9.0 copper-cloud does not store or serve canvases itself. Canvases live on a +**Canvas Cloud** — a separate, self-hostable collaborative canvas server +([copper-browser/canvas-cloud](https://github.com/copper-browser/canvas-cloud), MIT) — and +copper-cloud reaches it through a thin **canvas link** (crate `copper-cloud-canvas-link`, +`crates/canvas-link`): once an admin links a Canvas Cloud (URL + app key), copper-cloud +reverse-proxies the unchanged canvas API there, acting as the signed-in Copper user, and +re-publishes Canvas Cloud's canvas events on each user's `/v1/sync/events` stream. + +Copper (macOS and Linux) needs no update: the same routes, JSON, status codes, +WebSocket protocol, close codes, SSE events and user ids as copper-cloud 0.8. + +Moving an existing deployment's canvases over is a one-time step: +[canvas-migration.md](canvas-migration.md). + +## Why + +The canvas server (REST, Yjs rooms, ops/read, chat mentions, invites, share links, people) +used to be the `copper-cloud-canvas` crate inside this binary. It now is its own product with +its own users and auth (password accounts, personal access tokens, OIDC), linked apps, +an MCP surface for agents and an import format, so it can be used — and +hosted — without Copper. copper-cloud keeps what is Copper-specific: the instance gate, +Copper accounts and sessions, sync, intelligence keys, the admin portal, and the +`copper://` share-link landing page. + +## Architecture + +``` +┌── Copper ──────────────────┐ ┌── copper-cloud ───────────┐ ┌── Canvas Cloud ────────────┐ +│ canvas list, invites, │ │ instance gate → session │ │ /v1/canvases… /v1/invites… │ +│ share links, mentions │ TLS (pinned) │ canvas link: │ HTTPS/WSS, or │ /v1/mentions… /v1/people │ +│ canvas room (y-websocket) │──────────────▶│ REST proxy │──────────────▶│ /v1/canvas-links/… │ +│ │ X-Copper- │ WebSocket pipe │ loopback http │ /v1/apps/self/* (users, │ +│ GET /v1/sync/events │ Instance + │ event bridge │ Bearer cva_… │ events, canvases, stats, │ +│ ← event: canvas │ Bearer token │ provisioning │ X-Canvas-User │ import, digest) │ +└────────────────────────────┘ │ Postgres: users, sync, │◀── SSE ───────│ own users, sessions, PATs, │ + │ canvas_link, ledger, │ /apps/self/ │ OIDC, MCP, Postgres │ + │ legacy canvas tables │ events └────────────────────────────┘ + └───────────────────────────┘ +``` + +- Copper only ever talks to copper-cloud, on the same host, port and pinned certificate as + before. It never learns the Canvas Cloud's address. +- copper-cloud authenticates to Canvas Cloud as one **linked app** (`cva_…` key) and names + the Copper user it acts for in `X-Canvas-User`. Every Copper user is a Canvas Cloud app user + whose subject **and** user id are the Copper user id (see [Provisioning](#user-provisioning)). +- Canvas content (Yjs documents, chat, mention excerpts) is stored and encrypted by Canvas + Cloud, in Canvas Cloud's database. copper-cloud stores the link (one row), a provisioning + outbox, the migration ledger, and — untouched — the legacy canvas tables of 0.8.x. + +The small recommended setup runs Canvas Cloud on the same host as copper-cloud, listening on +loopback with TLS off (`listen = "127.0.0.1:8480"`, `tls.mode = "off"`), and links it as +`http://127.0.0.1:8480`. + +## What Copper clients see + +Nothing changes while a Canvas Cloud is linked and reachable: + +| Surface | Behaviour | +|---|---| +| Routes | `/v1/canvases…` (incl. `/v1/canvases/{id}/ws`), `/v1/invites…`, `/v1/canvas-links/…`, `/v1/mentions…`, `/v1/people` — any method, same paths and query strings | +| JSON | byte-for-byte what Canvas Cloud returns — the canvas API moved verbatim (top-level arrays, `201` vs `200` on invites, `404 link_not_found`, …) | +| Ids | canvas ids, user ids (members, chat `authorId`, mention ids) and share-link tokens are the same as on 0.8 (after [migration](canvas-migration.md)) | +| Live room | the same y-websocket protocol and close codes (`4403` removed, `4404` deleted, `1001` shutdown, …) | +| Notifications | `event: canvas` frames on `GET /v1/sync/events` with `{"type":"canvas","canvas_id","kind"}`, same kinds ([below](#event-bridge)) | +| `GET /v1/info` | same `name`/`version`/limits; `features` gains `"canvas_link"`, and a new `"canvas": {"linked": bool}` object | +| `/join/{token}` | the same `copper://canvas/join/{token}?cloud={host}` landing page, served by copper-cloud | + +Clients gate canvas features on the server **version** (share links/people ≥ 0.3.0, invite +reminders ≥ 0.5.0, mentions ≥ 0.6.0), not on `features`, so nothing has to be negotiated. + +When no Canvas Cloud is linked, or it cannot be reached, every canvas route fails with a +non-2xx JSON error (`503 canvas_unavailable`, `502 canvas_upstream`) — never an empty `200` +list (Copper would drop the canvases from its list and close their tabs), never a +`404 not_found` (Copper may cache that as "feature unsupported"), and never a `401` (Copper +signs the user out). Copper retries `GET`s a couple of times, keeps what it had, and shows the +error message; sync, history and SSE keep working. + +## The proxy + +`CanvasLink::router()` is mounted inside `/v1`, where the canvas router used to be, so the +instance gate, request tracing/metrics and rate limits run first. Routes (any method): +`/canvases`, `/canvases/{*rest}`, `/people`, `/invites`, `/invites/{*rest}`, +`/canvas-links/{*rest}`, `/mentions`, `/mentions/{*rest}`. + +For each request (`crates/canvas-link/src/proxy.rs`): + +1. **Authenticate the Copper user** with the core `AuthUser` extractor (`Authorization: + Bearer ` or `?token=`). No or bad session → copper-cloud's own `401 session`, as + for every `/v1` route. +2. **Load the link** (decrypted settings cached 5 s; changes made through the admin API apply + at once, CLI changes within 5 s). None, or `enabled = false` → + `503 canvas_unavailable`. +3. **Forward** to `{canvas cloud url}/v1{same path}{same query minus every token= parameter}`. + +### Headers + +| Direction | Forwarded | Never forwarded | Added | +|---|---|---|---| +| request | `Content-Type`, `Accept`, `If-Match`, `If-None-Match`, `If-Modified-Since`, `If-Unmodified-Since`, `If-Range` | `Authorization` (the Copper session), `Cookie`, `X-Copper-Instance`, hop-by-hop headers, everything else | `Authorization: Bearer `, `X-Canvas-User: `, `X-Forwarded-For: ` (as resolved by copper-cloud, honouring `trust_proxy`) | +| response | status, body (streamed), `Content-Type`, `Cache-Control`, `ETag`, `Last-Modified`, `Location`, `Retry-After`, `Content-Disposition` | everything else | — | + +The `?token=` query parameter (Copper's session token for WebSocket tooling) is consumed by +copper-cloud and removed; all other parameters (`q`, `limit`, `sv`, `unread`, `full`, `ids`, +`types`, …) pass unchanged. Redirects from Canvas Cloud are not followed. + +### Bodies, limits and timeouts + +| What | Limit | +|---|---| +| `POST /v1/canvases/{id}/ops` request body | 4 MiB (`413 payload_too_large` above; checked on `Content-Length` and while reading) | +| every other proxied request body | 64 KiB (`413 payload_too_large`) | +| request body read deadline | 60 s (`400`) | +| connect (TCP + TLS) to Canvas Cloud | 5 s | +| whole request incl. the response body | 30 s; `/ops` 120 s | +| WebSocket handshake to Canvas Cloud (TCP + TLS + upgrade) | 15 s | +| WebSocket message / frame, both sides | 64 MiB / 16 MiB | + +Response bodies are streamed through, not buffered. Content limits (op counts, name length, +mention rate limits, …) are Canvas Cloud's and reach Copper as Canvas Cloud's own errors. + +### Status codes + +| Situation | Copper gets | +|---|---| +| Canvas Cloud answers anything but `401` (`2xx`, `400`, `403`, `404`, `409`, `413`, `422`, `429`, `5xx`, …) | that status and body, verbatim | +| Canvas Cloud answers `401 {"error":"unknown_user"}` | copper-cloud provisions the user (`PUT /v1/apps/self/users/{id}`) and retries **once**; the retry's answer is handled like the first | +| provisioning after `unknown_user` fails | `502 {"error":"canvas_upstream","message":"could not provision this user on Canvas Cloud (…)"}` | +| any other `401`, or `unknown_user` again after the retry | `502 {"error":"canvas_upstream","message":"Canvas Cloud rejected this Copper Cloud's app key/user (: )"}` (logged at `error`: check the link's app key) | +| Canvas Cloud unreachable (DNS, connect, TLS, pin mismatch) | `502 {"error":"canvas_upstream","message":"Canvas Cloud is unreachable (…)"}` | +| connect/request timeout | `502 {"error":"canvas_upstream","message":"Canvas Cloud did not answer in time"}` | +| connection broken mid-request | `502 {"error":"canvas_upstream","message":"Canvas Cloud request failed (…)"}` | +| no Canvas Cloud linked (or the link is disabled) | `503 {"error":"canvas_unavailable","message":"no Canvas Cloud is linked to this Copper Cloud"}` | +| request body over the limit | `413 {"error":"payload_too_large"}` (copper-cloud's own) | +| no / expired Copper session | `401 {"error":"session"}` (copper-cloud's own, as before) | + +**Why a `401` from Canvas Cloud is never forwarded.** Both Copper clients treat +`401 {"error":"session"}` — in practice any `401` on these routes — as "your session ended" +and sign the user out. A `401` from Canvas Cloud means the *app* was rejected (revoked or +rotated app key, unknown subject), which is an operator problem, not the user's: signing +every Copper out over it would be wrong and would not fix anything. So it becomes a `502`, +which Copper treats as a transient server error. + +A Canvas Cloud that lacks a route answers `404 not_found`, which is passed through; keep the +Canvas Cloud at least as new as the canvas API your Coppers expect. + +### WebSocket pipe + +`GET /v1/canvases/{id}/ws` — HTTP/1.1 `Upgrade` or HTTP/2 extended `CONNECT` from Copper, as +before (`crates/canvas-link/src/ws.rs`): + +1. copper-cloud first opens the **upstream** WebSocket to the same path on Canvas Cloud + (`ws://` or `wss://`, query minus `token=`, app key + `X-Canvas-User` + `X-Forwarded-For`, + HTTP/1.1, 15 s budget). +2. If Canvas Cloud refuses the handshake with an HTTP status (`403`, `404`, …), Copper gets + that status and body **without** an upgrade — exactly what the old server did for + non-members. A refused `401` is handled like REST: `unknown_user` → provision and retry + once; anything else → `502 canvas_upstream`. A transport failure or timeout → + `502 canvas_upstream` (`"Canvas Cloud WebSocket is unreachable (…)"`, + `"Canvas Cloud did not answer the WebSocket handshake in time"`). +3. Only then is Copper's connection upgraded, and binary and text frames are piped both ways + unchanged (Yjs sync, awareness, auth messages — no re-framing). + +- **Ping/pong** is per side: copper-cloud pings Copper every 30 s and answers Canvas Cloud's + pings; control frames are not forwarded. There is **no idle timeout** — a room can stay open + indefinitely. +- **Close frames are forwarded with the same code and reason** in both directions (`4401`, + `4403`, `4404`, `1001`, `1013`, …); `1005`/`1006`/`1015` (local-only codes) are forwarded as a + close without a code. +- copper-cloud's own closes: Canvas Cloud connection lost, or a frame to Canvas Cloud not + accepted within 60 s → Copper gets `1011` "canvas server connection lost"; Copper gone → + Canvas Cloud gets `1001` "client went away"; copper-cloud shutting down (SIGTERM) → Copper + gets `1001` "server shutting down" and Canvas Cloud a `1001` too. +- Copper reconnects as it always did (1–30 s backoff); `4403`/`4404` stop the Linux client's + reconnect loop as before. + +## Event bridge + +While a Canvas Cloud is linked, one background task (`crates/canvas-link/src/bridge.rs`) +holds an SSE stream on `GET {url}/v1/apps/self/events` with the app key: + +- Each `event: canvas` with `data: {"subject","user_id","canvas_id","kind"}` is published to + the Copper user whose id is `subject` (falling back to `user_id`) as the core + `Event::Canvas {canvas_id, kind}` — Copper's `GET /v1/sync/events` then carries the same + frame as before the split: + + ``` + event: canvas + data: {"type":"canvas","canvas_id":"…","kind":"member_added"} + ``` + +- `Last-Event-ID` resume: the bridge sends the last event id it saw when it reconnects (reset + when the link points at a new URL). `event: resync` (events were missed) is logged only — + Copper refreshes its lists on its own `open`/`resync`. +- Reconnects with exponential backoff, 1 s → 30 s; treats 45 s without any bytes (Canvas + Cloud sends a keep-alive every 15 s) as a dead stream; follows link changes within 5 s. + +The `kind`s are Canvas Cloud's, unchanged from 0.8: + +| `kind` | Sent to | When | +|---|---|---| +| `created` | owner | canvas created | +| `renamed` | members | rename | +| `deleted` | former members | delete | +| `invited` | the invitee (if they have an account) | new invite, or a re-send reminder (at most every 30 s per invite) | +| `invite_revoked` | the invitee (if they have an account) | owner or inviter revoked a pending invite | +| `invite_declined` | the inviter | invite declined | +| `member_added` | members | invite accepted / share link joined | +| `member_removed` | members + the removed user | member removed / left | +| `update` | members | document changed (at most once per 2 s per canvas, plus one trailing) | +| `mention` | each newly mentioned member | someone @mentioned you in the canvas chat | +| `mention_read` | you | your mentions on that canvas were marked read | + +Only users of this app receive events through the bridge, which is every Copper user. + +`GET /v1/people` lists **all** enabled Canvas Cloud users, not only this app's: a Canvas Cloud +shared with other apps or with its own sign-ups shows those people to Coppers too (and they +can be invited and mentioned). + +## User provisioning + +Every Copper account exists on Canvas Cloud as an app user (`crates/canvas-link/src/provision.rs`): + +- **subject = id hint = the Copper `users.id`** (UUID string). Canvas Cloud creates new app + users with the hinted id, so Canvas Cloud user ids equal Copper user ids: member lists, + chat `authorId`s inside documents and mention ids stay valid, and migrated canvases need no + id rewriting. +- What is pushed: `email`, `display_name` (`null` when empty), `disabled`, `id`. + +```http +PUT /v1/apps/self/users/0192f1d0-7a3e-7c41-9a55-2f7c1d2e9b10 +{"email":"ana@example.com","display_name":"Ana","disabled":false,"id":"0192f1d0-7a3e-7c41-9a55-2f7c1d2e9b10"} + +POST /v1/apps/self/users:batch +{"users":[{"subject":"0192f1d0-…","email":"ana@example.com","display_name":"Ana","disabled":false,"id":"0192f1d0-…"}]} +``` + +When users are pushed: + +| Trigger | What happens | +|---|---| +| link (`canvas link`, portal, `PUT /admin/api/canvas-link`) | every user, `users:batch` in chunks of 1000 (the CLI waits for it unless `--no-sync`; the admin API runs it in the background) | +| `copper-cloud canvas sync-users [--prune]` | every user, now; with `--prune` also deletes Canvas Cloud users of this link whose Copper account no longer exists (and the canvases they own there) | +| Canvas Cloud answers `401 unknown_user` to a proxied request | just-in-time `PUT` of that user, then the request is retried once (a user that no longer exists in Copper is deleted on Canvas Cloud instead) | +| reconcile loop — at start, on every link change, then every 10 min | users whose email / display name / disabled flag changed since this process last pushed them (kept in memory, so after a restart everyone is pushed once); newly disabled users are also disconnected (`POST /v1/apps/self/users/{id}/disconnect`); app users whose Copper account no longer exists are only **counted and logged** (`canvas users reconciled … orphaned=N`) — reconcile never deletes, so restoring an older copper-cloud backup cannot wipe newer users' canvases; delete them deliberately with `canvas sync-users --prune` | +| admin API `PATCH /admin/api/users/{id}` | outbox `upsert_user`, plus `disconnect_user` when it disables the user | +| admin API `DELETE /admin/api/users/{id}` | outbox `delete_user` — Canvas Cloud deletes the app user and the canvases they own | + +**Outbox** (`canvas_link_outbox`): admin-API changes are queued in the same transaction as the +change (only while a link exists) and pushed right after commit; failures are retried with +backoff (30 s × 2^attempts, at most 1 h apart) and dropped with a warning after 50 attempts. +The worker also polls every 30 s. Unlinking empties the outbox (it targeted that server). +Changes made with `copper-cloud admin …` on the shell do not use the outbox; the server's +reconcile loop picks up edits and disables within 10 minutes. A user deleted on the shell stays +on Canvas Cloud (with their canvases) until you run `copper-cloud canvas sync-users --prune`. + +**Id mismatches.** If Canvas Cloud already has an account with a Copper user's email and the +app trusts email (`apps create` default), Canvas Cloud links that existing account — with its +own id — to the subject. The proxy still works for that person, but ids inside documents no +longer match theirs. `canvas link` / `canvas sync-users` report each such user and exit `1`; +`canvas migrate` refuses to import anything until every user maps to its own id. Link a Canvas +Cloud whose accounts do not overlap with Copper's, or create the app with `--no-trust-email` +(the user then fails with `409 email_taken` instead). + +## TLS to Canvas Cloud + +| URL | Verification | +|---|---| +| `https://…` (default) | the web PKI: the bundled Mozilla root set (`webpki-roots`) — **not** the operating system's trust store, so a private CA is not trusted; use a pin | +| `https://…` + fingerprint pin | exactly one leaf certificate: its SHA-256 (DER) must equal the pin; names, issuer and validity dates are not checked (the TLS handshake signature still is) — for a self-signed Canvas Cloud | +| `http://…` | allowed only for a loopback host (`localhost`, `127.0.0.0/8`, `[::1]`): a Canvas Cloud on the same machine | + +REST, the event stream and the WebSocket all use the same policy, over HTTP/1.1 (rustls, +`ring`). The URL may carry a path prefix (`https://host[:port][/prefix]`), never credentials, +a query or a fragment. A pin is 64 hex characters, any case, `:` separators allowed. To read +the fingerprint of a running Canvas Cloud: + +```sh +timeout 10 openssl s_client -connect canvas.example.com:443 -servername canvas.example.com /dev/null \ + | openssl x509 -outform DER | sha256sum +``` + +Re-link with the new pin whenever the Canvas Cloud certificate changes; until then every +canvas request is `502 canvas_upstream` (`"Canvas Cloud certificate does not match the pinned +fingerprint"`). + +## Linking + +### On the Canvas Cloud + +A Copper operator sets, on the Canvas Cloud (`/etc/canvas-cloud/canvas-cloud.toml`; see the +[Canvas Cloud README](https://github.com/copper-browser/canvas-cloud)): + +```toml +# Copper canvases contain copper:// links (link shapes). Canvas Cloud accepts only +# http, https and mailto by default; without this, ops that set a copper:// URL are rejected. +[ops] +extra_url_schemes = ["copper"] +``` + +(environment: `CANVAS_CLOUD_OPS__EXTRA_URL_SCHEMES=copper`), and creates the app copper-cloud +signs in as: + +```sh +sudo canvas-cloud apps create --name "Copper Cloud" --allow-import # prints cva_… once +``` + +`--allow-import` lets the app create canvases with `POST /v1/apps/self/import`, which +`copper-cloud canvas migrate` needs; it is harmless to keep. Add `--no-trust-email` if the +Canvas Cloud also has its own (non-Copper) accounts — see [id mismatches](#user-provisioning). +`canvas-cloud apps list`, `apps rotate ` (new key, old one stops working) and +`apps revoke ` manage it later. + +For the same-host setup, also: + +```toml +listen = "127.0.0.1:8480" + +[tls] +mode = "off" +``` + +### CLI + +```sh +umask 077 +sudo canvas-cloud apps create --name "Copper Cloud" --allow-import > /root/canvas-app.key # on the Canvas Cloud host; or use - to read stdin +sudo copper-cloud canvas link --url http://127.0.0.1:8480 --key-file /root/canvas-app.key +# self-signed Canvas Cloud on another host: +sudo copper-cloud canvas link --url https://canvas.example.com:8443 --key-file /root/canvas-app.key \ + --fingerprint 9F:86:D0:81:… +shred -u /root/canvas-app.key +``` + +``` +linked http://127.0.0.1:8480 (Canvas Cloud 0.1.0, key …x7Qa) +provisioned 12/12 users on Canvas Cloud +``` + +| Command | What it does | +|---|---| +| `copper-cloud canvas link --url URL --key-file PATH [--fingerprint HEX] [--no-sync]` | checks `GET /v1/info` and `GET /v1/apps/self/stats` with the key (5 s each); only if both pass stores the link (enabled) and provisions every user (skipped with `--no-sync`). Re-running it replaces the link (key rotation, new URL, new pin). The key is read from the file (`-` = stdin), never from argv; 8–4096 printable ASCII characters. Exit `1` if a user could not be provisioned or got a different id (the link stays saved) | +| `copper-cloud canvas unlink` | removes the link and the outbox ([Unlinking](#unlinking)) | +| `copper-cloud canvas status` | the link (the key is never shown), a fresh reachability check and the migration ledger summary; exit `1` when linked but unreachable | +| `copper-cloud canvas sync-users [--prune]` | provisions every user now (works with a disabled link too); `--prune` then deletes Canvas Cloud users of this link that no longer exist in Copper, with their canvases there | +| `copper-cloud canvas migrate [--dry-run] [--only ID]… [--replace] [--concurrency N] [--json]` | copies legacy canvases into the linked Canvas Cloud — [canvas-migration.md](canvas-migration.md) | + +The CLI talks to the database directly (run it as root, like the other admin commands); a +running server picks up a CLI change within 5 seconds. These commands need copper-cloud's +database migrations applied (`copper-cloud migrate`; `serve` and `install.sh` do it). + +### Portal + +**Canvas Cloud** in the admin portal: link status (reachable / unreachable / turned off, +version, last check, linked since, publicly trusted or pinned certificate), **Test +connection**, **Unlink**, the event bridge ("Live updates": connected, last event, last +error), the migration summary (migrated / failed / legacy canvases, with the command to run), +and a form to link or replace the app key (write-only: the stored key is never shown). It +shows `canvas-cloud apps create --name "Copper Cloud" --allow-import` for creating the key. +The **Canvases** page and the overview show an "unlinked" state when no Canvas Cloud is +linked. + +### Admin API + +Same conventions as the rest of [admin-api.md](admin-api.md) (admin session cookie, +`X-Requested-With: copper-cloud-portal` on non-GET, `{"error","message"}` errors). Every mutation is audited. + +#### `GET /admin/api/canvas-link` + +Re-probes Canvas Cloud when the last check is older than 15 s. + +```json +{ + "linked": true, + "url": "http://127.0.0.1:8480", + "enabled": true, + "tls_fingerprint": null, + "linked_at": "2026-10-19T18:02:11Z", + "updated_at": "2026-10-19T18:02:11Z", + "updated_by": "cli", + "status": { "reachable": true, "version": "0.1.0", "error": null, "checked_at": "2026-10-19T18:30:00Z" }, + "bridge": { "connected": true, "last_event_at": "2026-10-19T18:29:41Z", "last_error": null }, + "migrations": { "migrated": 36, "failed": 0, "legacy_canvases": 36 } +} +``` + +Not linked: `linked: false`, `enabled: false`, every other field `null`, `status` all `null`, +`bridge` `{"connected": false, "last_event_at": null, "last_error": null}`, and `migrations` +as above. `updated_by` is the admin's email or `"cli"`. `tls_fingerprint` is the lower-case +pin or `null`. `migrations` counts ledger rows `migrated` / `failed` and the rows of the legacy +`canvases` table. The app key is never returned. + +#### `PUT /admin/api/canvas-link` + +```json +{ "url": "https://canvas.example.com", "app_key": "cva_…", "tls_fingerprint": null } +``` + +Validates the input, probes `GET /v1/info` + `GET /v1/apps/self/stats` with the key (5 s +each), and only then stores the link (replacing any previous one; `linked_at` is kept when the +URL is unchanged) and starts provisioning every user in the background. `200` → the +`GET` body. `400 bad_request` with a human message when the URL, key or fingerprint is invalid, +or the probe fails (`"could not reach Canvas Cloud at …: …"`, `"Canvas Cloud at … rejected the +app key (…)"`, `"Canvas Cloud at … did not pass the check (…)"`); nothing is stored then. +Audit: `canvas_link.set` (detail: `url`, `tls_fingerprint`, `key_last4`, `version`). + +#### `DELETE /admin/api/canvas-link` + +Removes the link and the outbox. `200 {"ok": true}` (also when nothing was linked). Audit: +`canvas_link.remove` (only when something was linked). + +#### `POST /admin/api/canvas-link/test` + +```json +{ "ok": true, "version": "0.1.0", "latency_ms": 4, "error": null } +{ "ok": false, "version": null, "latency_ms": null, "error": "Canvas Cloud is unreachable (Connection refused (os error 111))" } +``` + +Always `200`; not linked → `ok: false`, `error: "no Canvas Cloud is linked"`. Audit: +`canvas_link.test` (when linked). + +#### Endpoints backed by the link + +| Endpoint | Source | When unlinked / unreachable | +|---|---|---| +| `GET /admin/api/overview` `counts.canvases`, `counts.live_rooms`, `counts.live_peers` | `GET /v1/apps/self/stats` | `null` | +| `GET /admin/api/users` `canvas_count` | `GET /v1/apps/self/users` (cached 30 s) | `null` | +| `GET /admin/api/canvases` | `GET /v1/apps/self/canvases` (same item shape as before, plus top-level `linked`) | unlinked: `200` empty list with `"linked": false`; unreachable: `502 canvas_upstream` | +| `DELETE /admin/api/canvases/{id}` | `DELETE /v1/apps/self/canvases/{id}` (Canvas Cloud closes the room with `4404` and notifies members) | `503 canvas_unavailable` / `502 canvas_upstream`; unknown id `404` | +| `PATCH` / `DELETE /admin/api/users/{id}` | outbox → Canvas Cloud ([provisioning](#user-provisioning)) | queued, retried | + +## Status and doctor + +```sh +sudo copper-cloud canvas status +``` + +``` +canvas link: http://127.0.0.1:8480 + enabled: true + pinned cert: - + linked at: 2026-10-19T18:02:11Z + updated: 2026-10-19T18:02:11Z by cli + reachable: yes (Canvas Cloud 0.1.0) + migrations: 36 migrated, 0 failed, 36 legacy canvases +``` + +`copper-cloud doctor` has a `canvas-link` check: + +``` + ok canvas-link http://127.0.0.1:8480 (Canvas Cloud 0.1.0, 12 users, 36 canvases, 4 ms) + warn canvas-link no Canvas Cloud linked: canvases are unavailable — copper-cloud canvas link --url … --key-file … + warn canvas-link http://127.0.0.1:8480 is linked but disabled + FAIL canvas-link http://127.0.0.1:8480: Canvas Cloud is unreachable (Connection refused (os error 111)) + FAIL canvas-link https://canvas.example.com: Canvas Cloud answered 401 : +``` + +`ok` = linked and both probes passed (the app key is accepted); `warn` = not linked (the +instance runs, canvases are `503`); `FAIL` = linked but unreachable or the key is refused +(non-zero exit). The event bridge lives in the server process, so its state is in the portal +(**Live updates**) and `GET /admin/api/canvas-link` `bridge`, not in the CLI or doctor. + +## Unlinking + +`copper-cloud canvas unlink`, portal **Unlink**, or `DELETE /admin/api/canvas-link`: + +- The `canvas_link` row and every queued outbox item are deleted. **Nothing on Canvas Cloud + is deleted** — users, canvases and the app stay; revoke the app there + (`canvas-cloud apps revoke `) if it should stop working. +- From then on (CLI: within 5 s) every canvas route answers `503 canvas_unavailable`, the + event bridge and the reconcile loop stop, and the overview/people canvas counts are `null`. +- Canvas WebSockets that are already open stay connected to Canvas Cloud until they close; + restart copper-cloud (`systemctl restart copper-cloud`, which closes them with `1001`) to + cut them at once. +- Copper users keep their local copies and see canvas requests fail with the message above; + nothing is removed from their canvas list (a failed fetch is never adopted). +- Linking the **same** Canvas Cloud again restores everything as it was. Linking a + **different** one shows users that server's canvases — for Copper a successful list without + their old canvases means those canvases are gone, so their tabs close. Migrate first. + +## Security model + +- **Copper users authenticate to copper-cloud only** (instance gate + session), exactly as + before. Canvas Cloud trusts copper-cloud's `X-Canvas-User` assertion for this app's users; + it never sees a Copper session token (the `Authorization` header and `?token=` are stripped). +- **What the app key can do** on Canvas Cloud: act as any user of this app (every Copper + user) on the whole canvas API; create, update, disconnect and delete this app's users (with + `trust_email`, provisioning an email that already has a Canvas Cloud account links that + account to the app); list, delete and digest the canvases those users own; read every + canvas event of those users; import canvases (only with `allow_import`). It cannot act as users that are not linked + to this app, or use Canvas Cloud's admin CLI (but those users' names and emails are visible + through `GET /v1/people`, as to any Canvas Cloud user). Treat it like the + `master_key`: whoever holds it can read and change every Copper user's canvases. +- **Where it is stored**: `canvas_link.app_key_sealed`, AES-256-GCM under a random per-row + data key (`data_key_wrapped`, wrapped by the KEK derived from `master_key`) with AAD + `canvas_link:app_key` — the same envelope as the intelligence keys. A database dump without + the `master_key` does not reveal it. It is never returned by the admin API, printed by the + CLI (only its last four characters), written to logs or audit rows (`key_last4` only), or + passed on argv; the outgoing header is marked sensitive. +- **Who can change the link**: portal admins (session + CSRF) and root on the host (the CLI + reads the 0600 config). Both are audited (`canvas_link.set` / `remove` / `test`). +- **Transport**: HTTPS verified against the web PKI or a pinned leaf; plain HTTP only to a + loopback address. Run a same-host Canvas Cloud with `listen` on `127.0.0.1` so it is not + reachable from the network without TLS. +- **Data at rest**: canvas documents, chat and mention excerpts are encrypted by Canvas Cloud + under its own `master_key`, in its own database. copper-cloud keeps the legacy canvas tables + (sealed under copper-cloud's `master_key`) untouched for rollback; see + [canvas-migration.md](canvas-migration.md#after-the-migration). +- **Logs** never contain the app key, session tokens, document contents or update bytes. + +## Metrics and logs + +| Metric | Type | Meaning | +|---|---|---| +| `canvas_link_ws_pipes` | gauge | proxied canvas WebSockets open now | +| `canvas_link_events_total` | counter | canvas events bridged into `/v1/sync/events` | + +Proxied requests appear in the normal `request` log lines and `http_*` metrics (route +`/v1/canvases/{*rest}` etc.); `502`/`503` carry `span.error` `canvas_upstream` / +`canvas_unavailable` and the message. Room, peer and document metrics are Canvas Cloud's. +Useful log filters: + +```sh +journalctl -u copper-cloud -o cat | jq -c 'select(.span.error=="canvas_upstream" or .span.error=="canvas_unavailable") | .span | {path, status, error_message}' +journalctl -u copper-cloud -o cat | jq -c 'select(.message | test("canvas"; "i")) | {level, message, error}' +``` + +Notable messages: `canvas event bridge connected` / `disconnected`, `canvas users +reconciled`, `canvas link outbox processed`, `provisioned user on Canvas Cloud`, `Canvas Cloud +answered 401 to the app; check the canvas link's app key`, `Canvas Cloud linked this Copper user +to an existing account with a different id; chat author ids will not match`. + +## Code map + +| Path | What | +|---|---| +| `crates/canvas-link/src/lib.rs` | `CanvasLink` handle: router, settings cache, background tasks, `503`/`502` bodies | +| `crates/canvas-link/src/proxy.rs` | REST proxy, header/query rules, `401` handling | +| `crates/canvas-link/src/ws.rs` | WebSocket pipe | +| `crates/canvas-link/src/bridge.rs` | SSE client + event bridge | +| `crates/canvas-link/src/provision.rs` | batch/JIT provisioning, reconcile loop, outbox | +| `crates/canvas-link/src/settings.rs` | the sealed `canvas_link` row, URL/key/pin validation | +| `crates/canvas-link/src/upstream.rs`, `tls.rs` | HTTP/WS client, timeouts, TLS pinning | +| `crates/canvas-link/src/admin.rs` | admin API / CLI / doctor helpers, audit | +| `crates/canvas-link/src/landing.rs` | `GET /join/{token}` landing page | +| `crates/canvas-link/src/info.rs` | adds `canvas` to `GET /v1/info` | +| `crates/canvas-migrate/` | `copper-cloud canvas migrate` ([canvas-migration.md](canvas-migration.md)) | +| `crates/copper-cloud/migrations/0030_canvas_link.sql` | `canvas_link`, `canvas_link_outbox`, `canvas_migrations` | + +## Testing + +`cargo test -p copper-cloud --test canvas_link` runs the link against an in-process mock Canvas +Cloud (axum) on the native Postgres test database (`COPPER_CLOUD_TEST_DATABASE_URL`, default +`postgres://localhost:5432/copper_cloud_test_core`): admin link lifecycle, request/response +forwarding and header rules, upstream failures and `401` mapping, the WebSocket pipe incl. +close codes, the event bridge into `/v1/sync/events`, admin canvases/overview/people and user +hooks, reconcile, and the public landing page. `cargo test -p copper-cloud-canvas-link` covers +the client, TLS pinning and SSE parsing without a database. diff --git a/docs/canvas-migration.md b/docs/canvas-migration.md new file mode 100644 index 0000000..187d217 --- /dev/null +++ b/docs/canvas-migration.md @@ -0,0 +1,387 @@ +# Migrating canvases to Canvas Cloud + +copper-cloud ≤ 0.8.x stored canvases in its own database (the `canvases`, `canvas_members`, +`canvas_invites`, `canvas_updates`, `canvas_snapshots`, `canvas_share_links` and +`canvas_mentions` tables, migrations `0100`–`0103`). From 0.9.0 canvases are served by a linked +[Canvas Cloud](https://github.com/copper-browser/canvas-cloud) ([canvas-link.md](canvas-link.md)). +This page is the one-time move of an existing deployment's canvases, and the runbook for it. + +A fresh 0.9.0 install has empty legacy tables: link a Canvas Cloud and skip the migration. + +## In short + +`copper-cloud canvas migrate` (crate `copper-cloud-canvas-migrate`, `crates/canvas-migrate`): + +1. checks the link (`GET /v1/apps/self/stats`, 30 s) and that the app may import; +2. provisions every Copper user on Canvas Cloud with `users:batch` (subject = id = Copper user + id); if any user does not come back with **its own id**, it stops and imports nothing; +3. for each legacy canvas — two at a time by default, each read in its own `REPEATABLE READ, + READ ONLY` transaction — decrypts it with copper-cloud's `master_key` and streams it as one + import-v1 NDJSON body to `POST /v1/apps/self/import`; +4. verifies each import: the counts Canvas Cloud applied must equal the counts sent, and the + digest of the document Canvas Cloud now stores (`GET /v1/apps/self/canvases/{id}/digest`) + must equal the digest the import returned; +5. records each result in the `canvas_migrations` ledger and prints a report. + +It **never modifies or deletes the legacy tables**; its only database writes are ledger rows. +Rollback is reinstalling the previous release ([Rollback](#rollback)). + +## What is preserved + +| Data | After the migration | +|---|---| +| canvas id, name, kind (Personal / shared), owner, `created_at`, `updated_at` | identical — Copper keeps its tabs and links (`copper://canvas/`) | +| the document (shapes, order, meta, agents, chat, …) | identical content: the digest of the stored document equals the digest of the legacy updates applied in order | +| members and roles (`owner` / `editor`), `added_at` | identical | +| user ids | identical: Canvas Cloud app users are created with the Copper user id, so member lists, chat `authorId`s inside documents and mention senders/recipients still match | +| invites — `pending`, `accepted`, `declined`, `revoked` — with inviter, `created_at`, `accepted_at`, `nudged_at` (reminder throttle) | identical; pending invites can still be accepted or declined, and invite ids are unchanged | +| share links: id, role, creator, `created_at`, `uses` | identical, and the stored token digests are copied, so **every existing share-link URL keeps working** | +| mentions: id, chat message id, from, to, excerpt, `created_at`, `read_at` | identical (excerpts are decrypted and re-sealed under Canvas Cloud's key); unread mentions stay unread | + +What is **not** preserved: + +- **Update history.** The legacy snapshot (if any) and every update after it are replayed into + one document, and Canvas Cloud stores it as **one compacted snapshot** with no update rows. + The final state is the same; the individual updates, their order and their per-update + timestamps (`canvas_updates.created_at`) are gone. Copper does not use them. +- Live room state (who is connected, awareness/cursors): rooms reopen on Canvas Cloud. +- Encryption keys: each canvas gets a new random document key under Canvas Cloud's + `master_key`. + +## Prerequisites + +1. **Canvas Cloud installed and running**, with `/v1/apps/self/import` and + `/v1/apps/self/canvases/{id}/digest` (the release that ships the linked-apps API). The + recommended small setup is **the same host, on loopback, without TLS**; any reachable host + works ([TLS to Canvas Cloud](canvas-link.md#tls-to-canvas-cloud)). + On AWS (`deploy/aws`) the copper-cloud VM is replaced on every release, so run Canvas Cloud + somewhere durable instead ([deploy/aws/README.md](../deploy/aws/README.md#canvas-cloud)). +2. On that Canvas Cloud: `[ops] extra_url_schemes = ["copper"]` (Copper canvases contain + `copper://` link shapes), and an **app key created with `--allow-import`** + ([Linking › On the Canvas Cloud](canvas-link.md#on-the-canvas-cloud)). + If that Canvas Cloud already has its own accounts with the same emails as Copper users, + create the app with `--no-trust-email` — otherwise those users map to existing ids and the + migration refuses to run. +3. **A backup** of the copper-cloud database (`pg_dump`, or an RDS snapshot) and of + `/etc/copper-cloud/copper-cloud.toml` ([operations.md](operations.md#backups)). The + migration does not change legacy rows, but the upgrade applies migration `0030`. +4. A maintenance window: copper-cloud is **stopped** from just before the upgrade until the + migration finished. Sync, history and sign-in are unavailable for that time; Coppers keep + working offline and catch up. For scale: the Exowatt dev1 instance had 36 canvases, + 64 memberships, 889 updates (≈ 88 MB stored), 22 invites, 5 share links and 5 mentions. +5. The release tarball or installer for copper-cloud 0.9.0. + +## Runbook + +All commands as root on the copper-cloud host unless noted. Every step can be re-run. + +### 0. Same-host Canvas Cloud (if not installed yet) + +The Canvas Cloud installer listens on `0.0.0.0:443` by default, which copper-cloud already +uses. Install on another port, then move it to loopback with TLS off: + +```sh +curl -fsSL https://raw.githubusercontent.com/copper-browser/canvas-cloud/main/install.sh \ + | sudo CANVAS_CLOUD_PORT=8480 sh +``` + +Edit `/etc/canvas-cloud/canvas-cloud.toml` — set `listen` and add or change the two tables: + +```toml +listen = "127.0.0.1:8480" + +[tls] +mode = "off" + +[ops] +extra_url_schemes = ["copper"] +``` + +```sh +sudo ufw delete allow 8480/tcp 2>/dev/null || true # the installer opened it (ufw/firewalld) +sudo systemctl restart canvas-cloud +timeout 10 curl -fsS http://127.0.0.1:8480/v1/info +umask 077 +sudo canvas-cloud apps create --name "Copper Cloud" --allow-import > /root/canvas-app.key +``` + +`apps create` prints only the key on stdout (the "created app " line goes to stderr; note +the id for `apps rotate` / `apps revoke`). + +The installer creates a separate `canvas_cloud` database on the same native PostgreSQL. + +### 1. Stop copper-cloud and back up + +```sh +sudo systemctl stop copper-cloud +sudo -u postgres pg_dump -Fc copper_cloud > /root/copper_cloud-pre-0.9-$(date +%F).dump +# RDS: aws rds create-db-snapshot --db-instance-identifier copper-cloud- \ +# --db-snapshot-identifier copper-cloud--pre-0-9 +sudo -u postgres psql -w -d copper_cloud -Atc 'SELECT count(*) FROM canvases' # legacy canvases +``` + +### 2. Install the new release + +```sh +tar xzf copper-cloud-0.9.0-linux-$(uname -m).tar.gz && sudo ./install.sh +# or: curl -fsSL https://raw.githubusercontent.com/copper-browser/copper-cloud/main/install.sh \ +# | sudo COPPER_CLOUD_VERSION=0.9.0 sh +``` + +The installer applies the database migrations (adds `canvas_link`, `canvas_link_outbox`, +`canvas_migrations`) and **restarts the service and waits for it to be healthy**. Nothing is +linked yet, so it serves sync normally and answers every canvas route with +`503 canvas_unavailable` — no canvas can change anywhere. Stop it again before linking: + +```sh +sudo systemctl stop copper-cloud +``` + +Why: a running copper-cloud picks up a new link within 5 seconds and starts proxying. Linked +before the canvases are migrated, Coppers would get successful canvas lists **without** their +canvases (Copper then drops them and closes their tabs), and people could create new canvases +that the migration does not know about. + +The `canvas` commands do not apply database migrations themselves; if you upgraded without +`install.sh`, run `sudo copper-cloud migrate` first. + +### 3. Link + +```sh +sudo copper-cloud canvas link --url http://127.0.0.1:8480 --key-file /root/canvas-app.key +``` + +``` +linked http://127.0.0.1:8480 (Canvas Cloud 0.1.0, key …x7Qa) +provisioned 12/12 users on Canvas Cloud +``` + +A non-zero exit after `linked …` means some users failed or were mapped to a different id +(listed); the link is saved. Fix the cause and run `sudo copper-cloud canvas sync-users` until +it exits 0. Keep `/root/canvas-app.key` until step 7. + +### 4. Dry run + +```sh +sudo copper-cloud canvas migrate --dry-run +``` + +Reads, decrypts and encodes every legacy canvas without sending anything (only the preflight +request reaches Canvas Cloud, and no users are provisioned). It catches a wrong `master_key`, +undecryptable rows and oversized lines before anything is written. Expect every canvas +`dry-run` and the last line `ok`. + +### 5. Migrate + +```sh +sudo copper-cloud canvas migrate +# or with the JSON report kept for the record: +sudo copper-cloud canvas migrate --json > /root/canvas-migrate-$(date +%F).json +``` + +Exit `0` = every user provisioned and no canvas failed. Otherwise read the report +([Reading the report](#reading-the-report), [Troubleshooting](#troubleshooting)), fix, and run +it again: migrated canvases are skipped, failed ones are retried. + +### 6. Start and check + +```sh +sudo systemctl start copper-cloud +sudo copper-cloud healthcheck --wait 60 +sudo copper-cloud canvas status # reachable: yes; N migrated, 0 failed, N legacy canvases +sudo copper-cloud doctor # canvas-link: ok +``` + +In the admin portal: **Canvas Cloud** shows the link reachable and live updates connected; +**Canvases** lists the same canvases as before. + +### 7. Verify with a client and spot-check documents + +- Open Copper: the canvas list, an open canvas tab, chat, the mentions inbox, pending + invites, and a share link (`copper://canvas/join/…`) all behave as before; an edit appears + live on a second Mac. +- Compare a few documents with what they held before. With your own account (any member of + the canvas), `GET /v1/canvases/{id}/read?full=true` through copper-cloud returns the same + JSON as the 0.8 server did. To capture the "before" side, run the first two commands in + step 1 before stopping the old server, and the rest after step 6: + + ```sh + H="X-Copper-Instance: "; U=https://cloud.example.com + T=$(timeout 15 curl -fsSk -H "$H" "$U/v1/auth/login" \ + -d '{"email":"you@example.com","password":"…","device":{"name":"migration check"}}' | jq -r .token) + timeout 30 curl -fsSk -H "$H" -H "Authorization: Bearer $T" "$U/v1/canvases/$ID/read?full=true" | jq -S . > before.json + # after step 6: + timeout 30 curl -fsSk -H "$H" -H "Authorization: Bearer $T" "$U/v1/canvases/$ID/read?full=true" | jq -S . > after.json + diff before.json after.json && echo identical + timeout 15 curl -fsSk -H "$H" -H "Authorization: Bearer $T" -X POST "$U/v1/auth/logout" -d '{}' + ``` + + Or ask Canvas Cloud directly, as that user (same answer as through the proxy): + + ```sh + timeout 30 curl -fsS -H "Authorization: Bearer $(cat /root/canvas-app.key)" -H "X-Canvas-User: " \ + "http://127.0.0.1:8480/v1/canvases/$ID/read?full=true" | jq -S . + ``` + +- Then `shred -u /root/canvas-app.key`. + +## Re-runs, `--only` and `--replace` + +| Command | What it does | +|---|---| +| `canvas migrate` | every legacy canvas not yet `migrated` in the ledger; ledger-`migrated` canvases are reported `already` without contacting Canvas Cloud | +| `canvas migrate --only ID [--only ID]…` | just those canvases, **even if** the ledger says migrated: a canvas that exists on Canvas Cloud with the digest the ledger recorded is reported `already` (and an `imported` row is confirmed as `migrated`); different content fails with "use --replace". An id that is not a legacy canvas is reported failed | +| `canvas migrate --replace` | every legacy canvas, ledger ignored, imported with `mode=replace`: a canvas that already exists on Canvas Cloud is deleted and re-imported atomically from the legacy data (its room is closed first). Only canvases that an import by this app created can be replaced. **Edits made on Canvas Cloud since are lost** | +| `canvas migrate --only ID --replace` | overwrite one canvas from the legacy data | +| `--concurrency N` | canvases in flight at once, 1–16 (default 2) | +| `--dry-run` | read + decrypt + count, send nothing, write nothing | +| `--json` | the report as JSON instead of the table | + +Without `--replace`, before uploading a canvas the tool asks Canvas Cloud for its digest; a +canvas that already exists is never uploaded again. It counts as done (`already`) only when +the ledger recorded exactly that digest — so a re-run after a crash or a partial run is safe +and cheap. Anything else needs a decision (`--replace`) and is reported failed. + +A Personal canvas whose owner already has an **empty** Personal canvas on Canvas Cloud (created +automatically when their Copper listed canvases) replaces it. A **non-empty** one fails with +`personal_exists`; `--replace` overwrites it. + +## The ledger + +`canvas_migrations` in copper-cloud's database, one row per legacy canvas attempted: + +| Column | | +|---|---| +| `canvas_id` (pk) | the legacy (= Canvas Cloud) canvas id | +| `status` | `imported` — accepted by Canvas Cloud, digest not yet verified (written before the check); `migrated` — counts and digest verified; `failed` — see `detail.error` | +| `digest` | lower-case hex SHA-256 of the canonical document (what Canvas Cloud stores) | +| `updates` | updates sent (the snapshot counts as one) | +| `migrated_at` | time of the last write | +| `detail` | JSON: `name`, `kind`, `owner`, `import_status` (`created` / `replaced` / `none` / `unverified`), `bytes`, `snapshot`, `members`, `invites`, `links`, `mentions`, `warnings`, `canvas_cloud` (URL), `tool` (version), `error` | + +A failure never downgrades a `migrated` row unless Canvas Cloud stored something that did not +verify. `canvas status`, the portal and `GET /admin/api/canvas-link` show +`migrated` / `failed` / legacy counts. Query it directly: + +```sh +sudo -u postgres psql -w -d copper_cloud -c \ + "SELECT canvas_id, status, updates, detail->>'name' AS name, detail->>'error' AS error + FROM canvas_migrations ORDER BY migrated_at" +``` + +The ledger is kept after unlinking and is the record of what was moved where. + +## Digest verification + +The digest is the lower-case hex SHA-256 of a canonical JSON rendering of the whole Yjs +document (every root type, map keys sorted, arrays in order, numbers as `f64`, binary as +base64; an empty document hashes `{}`). It depends only on the content, not on how the +updates were encoded, ordered or compacted — so the import's compacted snapshot has the same +digest as the legacy updates. Canvas Cloud computes it at import time (returned in the import +response) and again on `GET /v1/apps/self/canvases/{id}/digest`; the tool requires both to +match, and the counts Canvas Cloud applied (`updates_applied`, `members`, `invites`, `links`, +`mentions`) to equal what it sent. + +## Reading the report + +``` +canvas migrate → http://127.0.0.1:8480 +users: 12 provisioned (12 new), 0 failed +ID NAME KIND UPDATES BYTES MEMBERS STATUS DIGEST ERROR +0192f1d5-3c1a-7d2e-8f00-5b6a7c8d9e0f Personal personal 41 1.2 MiB 1 migrated 4f1c9a03be27 +0192f1d5-41aa-7c3d-9e11-2a3b4c5d6e7f Roadmap shared 212 12.4 MiB 3 migrated 9b07e2d1c4aa +0192f1d6-0b2c-7e4f-8a22-1c2d3e4f5a6b Notes shared 0 0 B 0 already 77de01a9f3c2 (recorded as migrated in the ledger) +total: 3 canvases — 2 migrated, 0 replaced, 1 already, 0 dry-run, 0 failed; 253 updates, 13.6 MiB, 4 members, 4 invites, 2 links, 1 mentions +ok +``` + +- **users**: on a real run, `provisioned (new)` / `failed`, with one `user : ` + line per failure; a dry run only counts them. +- one row per canvas, in legacy creation order. `UPDATES` includes the snapshot; `BYTES` is + the decrypted size of what was sent; `DIGEST` shows 12 characters; `ERROR` is the error, or a + note in parentheses. Canvases skipped from the ledger show zero counts. `warning:` lines follow a row (e.g. an undecryptable mention excerpt + sent empty, or an update over Canvas Cloud's default 32 MiB line limit). +- `STATUS`: `migrated` (created and verified), `replaced` (overwritten with `--replace` and + verified), `already` (nothing to do), `dry-run`, `failed`. +- `error: …` = a run-level failure; the last line is `ok` or `FAILED` (exit `0` / `1`). + +`--json` prints the same as one object: `canvas_cloud`, `dry_run`, `replace`, `started_at`, +`finished_at`, `users {total, provisioned, created, failed[{user_id, email, error}]}`, +`canvases[{id, name, kind, owner, status, updates, bytes, snapshot, members, invites, links, +mentions, digest, note, error, warnings}]`, `totals {canvases, migrated, replaced, already, +dry_run, failed, updates, bytes, members, invites, links, mentions}`, `error`, `ok`. + +## Troubleshooting + +Run-level errors stop the run (nothing more is imported); per-canvas errors fail that canvas +only. + +| Message | Cause / fix | +|---|---| +| `no Canvas Cloud is linked (copper-cloud canvas link …)` | link first (step 3); a disabled link is fine for migrating | +| `canvas-cloud preflight: GET …/v1/apps/self/stats failed` + a transport error | Canvas Cloud not running / wrong URL / TLS: `timeout 10 curl -fsS /v1/info`, `systemctl status canvas-cloud` | +| `… answered 401 … (the app key was rejected — revoked or wrong key?)` | the app was revoked or rotated: create/rotate it on Canvas Cloud and run `canvas link` again with the new key | +| `… answered 404 … (is this URL a Canvas Cloud? /v1/apps/self is missing)` | the URL is not a Canvas Cloud, or one without the linked-apps API | +| `the canvas-cloud app lacks allow_import (recreate it with canvas-cloud apps create --allow-import)` / `canvas-cloud refused the import (403) …; nothing was migrated by this run` / `not attempted: the app lacks allow_import` | create a new app with `--allow-import`, re-link with its key, re-run | +| `the canvas_migrations ledger table is missing — run copper-cloud's database migrations (0030_canvas_link.sql) first` | `sudo copper-cloud migrate` | +| `cannot unwrap the doc key of canvas …: is this copper-cloud's master_key?` | the config's `master_key` (or `COPPER_CLOUD_MASTER_KEY`) is not the one the canvases were written with: restore the original config | +| `N user(s) could not be provisioned with their copper user id; no canvas was migrated …` with `canvas-cloud mapped it to user … instead of … (email already used by another canvas-cloud account?)` | that Canvas Cloud already has an account with the user's email and the app trusts email: use a Canvas Cloud without overlapping accounts, or an app created with `--no-trust-email`, then re-link | +| `user …: missing from the users:batch response` / `: ` | Canvas Cloud rejected that user (e.g. invalid email); fix the account in Copper or on Canvas Cloud | +| `provisioning users: POST …/users:batch answered …` | Canvas Cloud refused the batch; see its logs | +| `exists in canvas-cloud with different content (digest …; it changed since this ledger recorded it (edited in canvas-cloud?)) — use --replace …` | someone edited it on Canvas Cloud after the migration: leave it (the Canvas Cloud copy is newer) or `--only ID --replace` to restore the legacy content | +| `exists in canvas-cloud with different content (…; this ledger has no verified record of importing it) — use --replace …` | a canvas with that id exists on Canvas Cloud from another import (another copper-cloud database, an earlier attempt without a ledger): `--only ID --replace` after checking | +| `canvas exists in canvas-cloud but no digest is available (use --replace)` | the digest endpoint failed for an existing canvas; retry, or `--only ID --replace` | +| `409 personal_exists … — the owner already has a non-empty Personal canvas in canvas-cloud (use --replace)` | the owner used canvases on Canvas Cloud before the migration (e.g. copper-cloud ran linked): `--only ID --replace` overwrites their Canvas Cloud Personal canvas with the legacy one | +| `422 unknown_subject …` | a member/inviter/mention user is not provisioned (deleted between steps?): re-run | +| `422 bad_update` / `422 incomplete_history` | a legacy update does not decode, or updates are missing: the legacy document is damaged. Check it with the dry run and `GET …/read` on the old release | +| `413 …` | the canvas is bigger than Canvas Cloud's `limits.import_max_bytes` (default 512 MiB) or a line over `limits.import_max_line_bytes` (32 MiB): raise them on Canvas Cloud, restart it, re-run | +| `reading the legacy canvas: update seq N does not decrypt` / `the snapshot (seq N) does not decrypt` | a damaged or foreign row in the legacy tables; nothing was imported for that canvas | +| `count mismatch: sent (updates, members, invites, links, mentions) = …, canvas-cloud applied …` | Canvas Cloud stored something different from what was sent: report it; `--only ID --replace` to retry | +| `digest mismatch: the import returned … but canvas-cloud now stores …` | the canvas changed right after the import (someone was editing on Canvas Cloud), or a Canvas Cloud bug: make sure copper-cloud is stopped, `--only ID --replace` | +| `verification failed: GET digest: … (the ledger keeps status imported; re-run to verify)` | the digest check could not be made; re-run (the canvas is accepted once its digest matches) | +| `verification failed: canvas-cloud reports the imported canvas does not exist` | deleted on Canvas Cloud right after the import: re-run | +| `not a legacy canvas (no such id in copper-cloud's canvases table)` | wrong `--only` id | +| `Canvas Cloud did not answer in time` / timeouts | big canvases: the import timeout is 120 s + 2 s per MiB stored; lower `--concurrency`, check Canvas Cloud's load | + +## After the migration + +- The legacy tables stay as they were, read by nobody but `canvas migrate`. They are the + rollback copy; keep them (and the pre-upgrade backup) until you no longer need to roll back. + They are not exposed through any route and are not changed by 0.9.0, except that deleting a + Copper user still removes their legacy rows (foreign-key cascade) and, through the admin + API, legacy invites addressed to their email. +- **Backups now include the Canvas Cloud database** (and its config with its `master_key`): + the canvases live there ([operations.md](operations.md#backups)). + +## Rollback + +The legacy tables are untouched, so the previous release still has every canvas as it was at +the moment of the migration. + +```sh +sudo systemctl stop copper-cloud +# 0.8.x refuses to start with a migration it does not know (sqlx: "migration 30 was previously +# applied but is missing"): remove 0030's link tables and its record; keep the ledger. +sudo -u postgres psql -w -d copper_cloud -v ON_ERROR_STOP=1 -c \ + "BEGIN; DROP TABLE canvas_link_outbox; DROP TABLE canvas_link; DELETE FROM _sqlx_migrations WHERE version = 30; COMMIT;" +tar xzf copper-cloud-0.8.1-linux-$(uname -m).tar.gz && sudo ./install.sh # reinstall the previous release +``` + +(Or restore the pre-upgrade dump / RDS snapshot — that also discards every sync and account +change made since.) `canvas_migrations` is created with `IF NOT EXISTS`, so upgrading again later +re-applies `0030` cleanly; link again then. + +What happens to work done after the cutover: + +- **Canvas edits, chat, new canvases, invites, share links and mentions made on Canvas Cloud + after the migration are not in the legacy tables** and are not copied back. After a rollback, + Coppers see each canvas as it was at migration time; canvases created after the cutover are + missing from the list, so Copper removes them and closes their tabs. They still exist on + Canvas Cloud (export them from there if needed). A Copper that still holds a newer version of + an open document may push it back into the legacy room when it reconnects — do not rely on it. +- Sync data, accounts and devices are unaffected (same tables in both releases). +- Users deleted after the upgrade have lost their legacy canvases too (cascade). +- Upgrading again after a rollback: the ledger still says `migrated` and Canvas Cloud still has + the earlier imports, so a plain `canvas migrate` reports them `already` even if they were + edited on the old release in between. Use `canvas migrate --replace` to bring the legacy + content over again (this overwrites what Canvas Cloud has). diff --git a/docs/canvas-protocol.md b/docs/canvas-protocol.md index a52edd6..c6d9339 100644 --- a/docs/canvas-protocol.md +++ b/docs/canvas-protocol.md @@ -1,238 +1,54 @@ # Canvas protocol -The shared contract between the canvas page (TS, `Canvas/src`), the Copper host (Swift), and -the server (`crates/canvas`, Rust/`yrs`). REST routes and the room lifecycle are in -[canvas.md](canvas.md). +The canvas wire protocol, document schema, ops and read formats moved with the canvas server +to **Canvas Cloud**: [docs/canvas-protocol.md](https://github.com/copper-browser/canvas-cloud/blob/main/docs/canvas-protocol.md) +in [copper-browser/canvas-cloud](https://github.com/copper-browser/canvas-cloud). Nothing about +the protocol changed: copper-cloud 0.9.0 proxies it unchanged ([canvas-link.md](canvas-link.md)). ## Wire protocol -`GET /v1/canvases/{id}/ws` upgrades to a WebSocket that speaks **y-websocket** — the same -framing and message flow as the reference `y-websocket` server, implemented with -`yrs::sync` (the y-sync protocol: `DefaultProtocol`'s `Message`, `SyncMessage`, -`Awareness`). +Moved: [Canvas Cloud › Wire protocol](https://github.com/copper-browser/canvas-cloud/blob/main/docs/canvas-protocol.md#wire-protocol). +copper-cloud's side is a byte-exact pipe: [canvas-link.md › WebSocket pipe](canvas-link.md#websocket-pipe). ### Connecting -``` -GET /v1/canvases//ws?token= (or Authorization: Bearer …) -X-Copper-Instance: -Upgrade: websocket -``` - -- `401` without a valid session/instance key, `404` if you are not a member, `400` if the - request is not an upgrade. -- Any `?token=` / header combination the core `AuthUser` extractor accepts works. -- With `y-websocket`'s `WebsocketProvider`: `serverUrl = "wss://HOST:PORT/v1/canvases"`, - `roomname = "/ws"`, `params: { token }`. The socket must send the instance header (the - Copper host sets it on its `URLSessionWebSocketTask` and relays frames to the page). +Moved: [Canvas Cloud › Connecting](https://github.com/copper-browser/canvas-cloud/blob/main/docs/canvas-protocol.md#connecting). +Copper still connects to copper-cloud's `GET /v1/canvases/{id}/ws` with its gate credential +and session; copper-cloud opens the Canvas Cloud socket first and only then upgrades. ### Framing -Every WebSocket **binary** message carries one or more y-protocol messages back to back; each -starts with a lib0 var-uint message type. Text frames are ignored. - -| Type | Name | Payload | -|---|---|---| -| `0` | sync | var-uint sub-type, then a var-length buffer: `0` SyncStep1 (state vector), `1` SyncStep2 (update), `2` Update (update) | -| `1` | awareness | var-length buffer: `count`, then per client `clientID` (var-uint), `clock` (var-uint), `state` (var-string JSON, `"null"` = gone) | -| `2` | auth | ignored (auth happens at upgrade) | -| `3` | awareness query | server answers with every known awareness state | - -Updates are lib0 **v1** encoded (`Y.encodeStateAsUpdate`, `Y.applyUpdate`) — not v2. +Moved: [Canvas Cloud › Framing](https://github.com/copper-browser/canvas-cloud/blob/main/docs/canvas-protocol.md#framing). +Frames are not re-framed by the proxy (64 MiB message / 16 MiB frame caps). ### Flow -``` -client server - │── HTTP upgrade (token, instance key) ──▶│ membership check, room loaded - │◀── sync: SyncStep1(server SV) ─────────│ greeting - │◀── awareness: current states ──────────│ (only if anyone is present) - │── sync: SyncStep1(client SV) ─────────▶│ - │◀── sync: SyncStep2(what client lacks) ─│ - │── sync: SyncStep2(what server lacks) ─▶│ applied → persisted → broadcast as Update - │── sync: Update … ─────────────────────▶│ applied → persisted → broadcast as Update - │◀── sync: Update (others / REST ops) ───│ - │── awareness … ────────────────────────▶│ relayed to the other peers (not persisted) - │◀── ping (every 30 s) ──────────────────│ any frame or pong within 90 s keeps it alive -``` - -The server never echoes a peer's own update back to it. Updates from REST (`/ops`, rename) -arrive as ordinary `Update` messages. +Moved: [Canvas Cloud › Flow](https://github.com/copper-browser/canvas-cloud/blob/main/docs/canvas-protocol.md#flow). ### Close codes -| Code | Meaning | Client should | -|---|---|---| -| `1000` | normal | — | -| `1001` | server shutting down | reconnect with backoff | -| `1007` | undecodable message or rejected update | fix the client; reconnecting resyncs | -| `1009` | message over 8 MiB | split the change | -| `1011` | server error (e.g. database write failed) | reconnect; the handshake resyncs | -| `1013` | too slow (fell 256 frames behind) | reconnect; the handshake resyncs | -| `4403` | you were removed from the canvas | stop; refresh the canvas list | -| `4404` | the canvas was deleted | stop; refresh the canvas list | - -Client note: the initial `SyncStep2` carries the whole document and can be larger than -1 MiB once images are pasted — raise the client's maximum message size accordingly -(`URLSessionWebSocketTask.maximumMessageSize` defaults to 1 MiB). +Moved: [Canvas Cloud › Close codes](https://github.com/copper-browser/canvas-cloud/blob/main/docs/canvas-protocol.md#close-codes). +The proxy forwards close codes and reasons unchanged and adds `1011` (Canvas Cloud connection +lost) and `1001` (copper-cloud shutting down). ## Document schema -One `Y.Doc` per canvas: - -| Root | Type | Contents | -|---|---|---| -| `shapes` | `Y.Map>` | one nested map per shape, so each property is last-writer-wins on its own | -| `agents` | `Y.Map` | plain objects, replaced whole: `{name, color, cursor: {x,y} \| null, status: "idle"\|"thinking"\|"writing", updatedAt}` | -| `meta` | `Y.Map` | `name`, `createdBy` (user id) | -| `comments` | — | optional; not touched by the server | -| `chat` | `Y.Array` | shared-canvas chat (0.6.0), written only by Copper: plain objects `{id, authorId, authorName, text (≤ 4000), mentions: [userId], at (ms epoch), editedAt?, deleted?}`, oldest first; the client caps it (about 1000, dropping the oldest). The server persists and relays it like any other root and shows the tail in [read](#read) | - -Shape props (all shapes): `id`, `type`, `x`, `y`, `w`, `h` (integers when written by the page -or server), `color`, `by` (display name of the creator), `createdAt`, `updatedAt` (ms epoch), -`z` (stacking, higher on top). - -| `type` | Extra props | Default size | -|---|---|---| -| `sticky` | `text` (**`Y.Text`**, markdown), `fontSize?` | 200×200 | -| `text` | `text` (**`Y.Text`**), `fontSize?`, `align?` (`left`/`center`/`right`) | 240×36 | -| `frame` | `title`, `image?: {src, naturalW, naturalH}` | 480×320 | -| `arrow` | `from`, `to`: `{x,y}` or `{ref: shapeId, side?: top\|right\|bottom\|left}`; `label?` | no box of its own (stored 0,0,0,0; drawn from its ends) | -| `image` | `src` (`data:image/…` ≤ 2 MiB, or `https:`), `naturalW`, `naturalH` | natural size, ≤ 480 wide | -| `link` | `url`, `title`, `favicon?` | 300×84 | -| `checklist` | `title`, `columns: string[]` (1–4, default `["Yes","No"]`), `rows: [{id, label}]` (≤ 60), and one `pick:: {col, by, byId, at}` per picked row | 324×(rows) — see below | - -Colours: `yellow`, `pink`, `blue`, `green`, `purple`, `gray`, `white`, or `#rgb[a]` / -`#rrggbb[aa]`. Defaults: sticky yellow, image/link white, checklist green, the rest gray. - -**Checklist (RSVP).** Each row has at most one pick, a column (radio across the columns; one -column is a plain to-do list). A pick is stored under its row's own key — `pick:` → -`{col, by, byId, at}` — so people picking different rows at the same moment never overwrite -each other; `columns` and `rows` are whole values (last writer wins). `col` is the column's -label: a pick naming no current column or row reads as no pick, renaming a column only in case -keeps its picks, and replacing `rows` drops the picks of rows that went. Row ids are 1–32 of -`A–Z a–z 0–9 _ -` (fresh ones are `r` + 8 base-36). Columns are cut to 40 characters, row -labels to 200. A fresh card is `324` wide for two columns (`28 + 168 + 64 × columns`; one -column: `280`) and `49 + 28 (if > 1 column) + 34 × max(rows, 1) + 14` tall; the page re-measures -the height as it lays the card out. Clients older than the checklist (Copper ≤ 1.0.20261006.28, -copper-cloud ≤ 0.5.0) do not show or read it, but keep it intact. - -Readers are tolerant: a plain-string `text` is read like a `Y.Text`; missing sizes take the -type default; unknown colours fall back to the default. Writers turn a plain-string body into -a `Y.Text` on the next edit and splice text minimally so concurrent typing merges. +Moved: [Canvas Cloud › Document schema](https://github.com/copper-browser/canvas-cloud/blob/main/docs/canvas-protocol.md#document-schema). ## Ops -`POST /v1/canvases/{id}/ops` on the server and `copperCanvas.apply` on the page run the same -ops with the same validation and results. A batch runs in one transaction, in order; a bad -op is skipped and reported, the rest still apply. - -```json -{ "ops": [ … ], "as": { "id": "agent:planner", "name": "Planner", "color": "#7c3aed" }, - "confirm": true, "near": { "x": 0, "y": 0 } } -``` - -`as` may also be a plain name (`"as": "Planner"` → id `agent:Planner`); object form defaults -`name` to `Agent` and `id` to `agent:`. A bare array of ops is accepted. - -| Op | Shape | -|---|---| -| `add` | `{op:"add", shape:{type, id?, x?, y?, w?, h?, color?, …}}` — omitted `x`/`y`: the nearest free spot (24 px gap, 32 px grid, walking outward from `near`, ties right → down → left → up). `id` (optional): 1–64 of `A–Z a–z 0–9 _ - : .`; server-generated ids are UUID v7. `z` defaults to the current max + 1. A link without a `title` gets its host name. Sizes clamp to the type's minimum. | -| `update` | `{op:"update", id, patch:{…}}` (`props` also accepted) — merges props; `image: null` removes a frame image. `id`, `type`, `by`, `createdAt`, `updatedAt` are ignored. | -| `move` | `{op:"move", id, dx, dy}` — a frame carries every shape lying wholly inside it; an arrow shifts its free ends. | -| `resize` | `{op:"resize", id, w, h}` — not for arrows; clamps to the minimum size. | -| `delete` | `{op:"delete", id}` — arrows ending on the shape are deleted too. | -| `connect` | `{op:"connect", from, to, label?, color?, fromSide?, toSide?, id?}` — adds an arrow between two (non-arrow) shapes. | -| `clear` | `{op:"clear", confirm:true}` — removes every shape. **Server:** requires `confirm: true` on the op or the request. | - -A checklist takes on `add` and `update`: `title`, `columns` (1–4 distinct names), `rows` -(labels `"Ann"` or `{label, id?}`, ≤ 60 — on `update` a row keeps its id, and so its pick, when -it names an existing id or has the same label) and `picks` (`{"Ann": "Yes", "r3": null}` — a row -by id or label, any case; a column in any case, `true` for the first, `null`/`false`/`""` to -clear). `rows`/`columns` replace the lists; new lists also size the card unless `w`/`h` are -given. Picks are attributed to `as` (else the caller): `by` its name, `byId` its id. A bad pick, -row or column fails the whole op. - -Aliases: `text` (or `label`) on a frame, link or checklist is its `title`; `text`/`title` on an arrow is its -`label`; `title` on a sticky/text is its `text`. Ids may be written `shape:`; endpoints -may be `"id"`, `{ref}`, `{id}` or `{x,y}`. - -Validation (first failing prop is reported): unknown props for the type, non-finite numbers, -non-positive `w`/`h`, bad colours, `text` over 20 000 characters (titles/labels are cut to -500), `fontSize` outside 6–240, `src` not `data:image/…` (≤ 2 MiB) or `https:`, `url` not -`http(s)`/`mailto`/`copper` (bare hosts get `https://`, URLs are normalised), refs to missing -shapes or to arrows, an arrow joining a shape to itself. +Moved: [Canvas Cloud › Ops](https://github.com/copper-browser/canvas-cloud/blob/main/docs/canvas-protocol.md#ops). +Note for Copper operators: `copper://` URLs in link shapes need `ops.extra_url_schemes = +["copper"]` on the Canvas Cloud ([canvas-link.md › On the Canvas Cloud](canvas-link.md#on-the-canvas-cloud)). ### Result -```json -{ - "applied": 3, - "ids": ["0192…", "0192…", null, "0192…"], - "errors": [{ "index": 2, "op": "update", "error": "no shape nope" }] -} -``` - -`ids[i]` is the id op `i` created or touched — `null` when it failed, and for `clear`. -Request-level problems use `index: -1` (`"at most 500 ops per call (got 600)"`). +Moved: [Canvas Cloud › Result](https://github.com/copper-browser/canvas-cloud/blob/main/docs/canvas-protocol.md#result). ### Example -```json -{ "as": { "name": "Planner" }, - "ops": [ - { "op": "add", "shape": { "id": "goal", "type": "sticky", "text": "Ship v1", "color": "green" } }, - { "op": "add", "shape": { "id": "risk", "type": "sticky", "text": "Load test" } }, - { "op": "connect", "from": "goal", "to": "risk", "label": "blocked by" }, - { "op": "add", "shape": { "type": "frame", "title": "Q4", "x": -400, "y": -300, "w": 1200, "h": 700 } }, - { "op": "move", "id": "risk", "dx": 40, "dy": 0 } - ] } -``` +Moved: [Canvas Cloud › Example](https://github.com/copper-browser/canvas-cloud/blob/main/docs/canvas-protocol.md#example). ## Read -`GET /v1/canvases/{id}/read?full=true&ids=a,b&types=sticky,text` and `copperCanvas.read` -return the same shape (the server has no viewport or selection, so `viewport` is omitted and -`selection` is `[]`): - -```json -{ - "canvas": { "id": "0192…", "name": "Roadmap", "kind": "shared" }, - "shapes": [ - { "id": "goal", "type": "sticky", "x": -100, "y": -100, "w": 200, "h": 200, - "color": "green", "z": 1, "by": "Planner", "text": "Ship v1", "frame": "q4" }, - { "id": "a1", "type": "arrow", "x": 100, "y": 0, "w": 124, "h": 0, "color": "gray", "z": 3, - "by": "Planner", "label": "blocked by", "from": { "ref": "goal" }, "to": { "ref": "risk" } }, - { "id": "pic", "type": "image", "x": 0, "y": 400, "w": 480, "h": 240, "color": "white", "z": 4, - "src": "data:image/png (812 KB)", "naturalW": 960, "naturalH": 480 }, - { "id": "doc", "type": "link", "x": 500, "y": 0, "w": 300, "h": 84, "color": "white", "z": 5, - "url": "https://example.com/", "title": "example.com" } - ], - "agents": [ - { "id": "agent:planner", "name": "Planner", "color": "hsl(212 62% 48%)", - "cursor": { "x": 162, "y": 0 }, "status": "idle", "updatedAt": 1790000000000 } - ], - "selection": [], - "count": 4, - "chat": [ - { "id": "msg_01J…", "authorId": "0192…", "authorName": "Ann", "text": "@Ben dinner Tuesday?", - "mentions": ["0192…"], "at": 1790000000000 } - ] -} -``` - -- Shapes are sorted top-to-bottom, then left-to-right (`y`, `x`, `id`); coordinates rounded. -- Per type: sticky/text → `text` (+ `fontSize`, `align`); frame → `title` (+ `image`); arrow - → `label`, `from`, `to` and a box computed from its drawn path; image → `src`, - `naturalW`, `naturalH`; link → `url`, `title`; checklist → `title`, `columns`, - `rows: [{id, label, pick, by?, at?}]` (`pick` null when none) and `tally: {column: count}`. -- `text`, `title`, `label` are cut to 500 characters plus `…` unless `full=true`. `data:` - sources are described (`data:image/png (812 KB)`), never echoed (with `full`, only if - ≤ 4 KiB). -- `frame` is the innermost frame wholly containing the shape. -- `agents`: entries written within the last 2 minutes, busiest first, then by name. -- `count` is every readable shape, before `ids`/`types` filtering. -- `chat` (server read, 0.6.0): the last 50 messages of the `chat` array, oldest first, in array - order. Deleted messages (`deleted: true`) and entries without a string `id` are left out; - `text` is cut like other text; `editedAt` only when set. `[]` when the canvas has no chat. - `ids` / `types` do not filter it. +Moved: [Canvas Cloud › Read](https://github.com/copper-browser/canvas-cloud/blob/main/docs/canvas-protocol.md#read). diff --git a/docs/canvas.md b/docs/canvas.md index 58e1a18..2c3827b 100644 --- a/docs/canvas.md +++ b/docs/canvas.md @@ -1,427 +1,149 @@ # Canvases -The `copper-cloud-canvas` crate (`crates/canvas`) serves collaborative whiteboards: a REST -API for canvases, members and invites, and one live Yjs room per open canvas that speaks the -standard `y-websocket` protocol. The wire protocol, document schema and ops/read JSON are in -[canvas-protocol.md](canvas-protocol.md). - -Everything here is mounted under `/v1`, behind the instance-key gate -(`X-Copper-Instance`) and session auth (`Authorization: Bearer ` or `?token=`), like -the rest of the [API](api.md). Errors use the common `{"error","message"}` shape. +Since 0.9.0 canvases are served by a linked **Canvas Cloud** +([copper-browser/canvas-cloud](https://github.com/copper-browser/canvas-cloud)), not by +copper-cloud. The canvas API moved there verbatim and is documented in Canvas Cloud's +[docs/canvas.md](https://github.com/copper-browser/canvas-cloud/blob/main/docs/canvas.md). + +What remains in copper-cloud is the **canvas link** — a reverse proxy that serves the same +routes under `/v1` to signed-in Coppers, acting as the user, plus the event bridge into +`GET /v1/sync/events`, user provisioning and the `/join/{token}` landing page: +[canvas-link.md](canvas-link.md). Moving an existing deployment's canvases: +[canvas-migration.md](canvas-migration.md). + +Every route below is still reachable at `/v1/…` on copper-cloud with the same request and +response; while no Canvas Cloud is linked (or it is unreachable) they answer +`503 canvas_unavailable` (`502 canvas_upstream`) — see +[canvas-link.md › Status codes](canvas-link.md#status-codes). ## Model -| Table | What | -|---|---| -| `canvases` | `id`, `owner_id`, `name`, `kind` (`personal`/`shared`), `doc_key_wrapped`, timestamps | -| `canvas_members` | `(canvas_id, user_id)`, `role` (`owner`/`editor`), `added_at` — the owner has a row too | -| `canvas_invites` | email invites: `email` (lower-cased), `invited_by`, `status` (`pending`/`accepted`/`declined`/`revoked`), `nudged_at` (last re-send reminder), `token` (hash, reserved) | -| `canvas_share_links` | opaque share-link token digest, `role` (`editor`), `created_by`, `uses`; tokens are never persisted in plaintext | -| `canvas_updates` | append-only Yjs update log: `seq bigserial`, `canvas_id`, `"update"` (sealed) | -| `canvas_snapshots` | one compacted state per canvas: `seq` (last update folded in), `state` (sealed) | -| `canvas_mentions` | chat @mentions: `canvas_id`, `message_id` (the client's chat message id), `from_user`, `to_user`, `excerpt_sealed`, `created_at`, `read_at`; one row per `(canvas_id, message_id, to_user)` | - -Migrations: `crates/copper-cloud/migrations/0100_canvases.sql`, -`0101_canvas_share_links.sql`, `0102_canvas_invite_nudge.sql` and `0103_canvas_mentions.sql` -(the canvas crate owns `0100`–`0199`). - -- **Personal canvas**: every user has exactly one (`kind = personal`, name `Personal`), - created on first use (`GET /canvases`, or any `/canvases/personal/...` route) — a unique - partial index on `owner_id WHERE kind = 'personal'` makes concurrent first use safe. It can - never be renamed, deleted, shared or left (`400`). -- **Roles**: `owner` (rename, delete, remove members) and `editor` (read/write the document, - invite others, leave). Both can edit the board. -- **Access is membership**: every lookup joins `canvas_members` on the caller. A canvas you - are not a member of — or that does not exist — is `404` on every route (no existence leak). - A member who lacks the role for an action gets `403`. - -`{id}` in every route below is a canvas UUID **or the literal `personal`**. +Moved: [Canvas Cloud › Model](https://github.com/copper-browser/canvas-cloud/blob/main/docs/canvas.md#model). +Canvas user ids are Copper user ids ([provisioning](canvas-link.md#user-provisioning)). ## REST -### `GET /canvases` - -Your canvases, Personal first, then most recently updated. +Moved: [Canvas Cloud › REST](https://github.com/copper-browser/canvas-cloud/blob/main/docs/canvas.md#rest); +proxied unchanged ([the proxy](canvas-link.md#the-proxy)). -```json -[ - { - "id": "0192…", "name": "Personal", "kind": "personal", "role": "owner", - "owner": { "id": "0192…", "email": "ann@example.com", "display_name": "Ann" }, - "member_count": 1, - "created_at": "2026-10-01T12:00:00Z", "updated_at": "2026-10-01T12:05:00Z" - } -] -``` +### `GET /canvases` -`updated_at` follows document activity (bumped at most every 2 s while a room is busy). +Moved to [Canvas Cloud › REST](https://github.com/copper-browser/canvas-cloud/blob/main/docs/canvas.md#rest). ### `POST /canvases {name}` → `201` canvas -Creates a `shared` canvas owned by you (owner member row included). `name` is trimmed, -control characters dropped, 1–200 characters. The document starts with `meta.name` and -`meta.createdBy`. +Moved to [Canvas Cloud › REST](https://github.com/copper-browser/canvas-cloud/blob/main/docs/canvas.md#rest). ### `GET /canvases/{id}` → canvas · `PATCH /canvases/{id} {name}` → canvas · `DELETE /canvases/{id}` → `{"ok":true}` -Rename and delete are owner-only (`403` for editors) and `400` on the Personal canvas. -Rename also updates the document's `meta.name` (synced to connected peers). Delete removes -the canvas, members, invites, updates and snapshot (cascade) and closes its room: connected -sockets get close code `4404`. +Moved to [Canvas Cloud › REST](https://github.com/copper-browser/canvas-cloud/blob/main/docs/canvas.md#rest). ### `GET /canvases/{id}/members` -```json -[{ "user_id": "…", "email": "…", "display_name": "…", "role": "owner", "added_at": "…" }] -``` - -Owner first, then by join time. Any member may list. +Moved to [Canvas Cloud › REST](https://github.com/copper-browser/canvas-cloud/blob/main/docs/canvas.md#rest). ### `DELETE /canvases/{id}/members/{user_id}` → `{"ok":true}` -The owner removes anyone else; any member removes themselves (leave). The owner cannot leave -(`400` — delete the canvas instead). Removed users' open sockets close with `4403`. +Moved to [Canvas Cloud › REST](https://github.com/copper-browser/canvas-cloud/blob/main/docs/canvas.md#rest). ### `POST /canvases/{id}/invites {email}` → `201` new invite · `200` already pending (re-send) -Owner or editor. The email is trimmed and lower-cased; inviting yourself is `400`, an -existing member `409`, the Personal canvas `400`. An email with no account yet is fine: the -invite waits until someone signs up with it. Even when the email already belongs to a user, -they must accept — nobody is added to a canvas without consent. - -- **New invite → `201`.** The invitee (if they have an account) gets a `canvas` event with - kind `invited`. -- **Already pending → `200`** with the same invite, and the call is a **re-send**: if the - invitee has an account and the last reminder (`nudged_at`, else `created_at`) is at least - 30 s old, `nudged_at` is set to now and the invitee gets another `invited` event — clients - use it to surface the invite again. Re-sends inside the 30 s window, or to an email with no - account yet, change nothing. - -The response is the invite view plus `nudged`: `true` only when this call reminded the -invitee (a `200` re-send that was not throttled); `false` on `201` and when throttled or -there is no account yet. - -```json -{ - "id": "…", "canvas_id": "…", "canvas_name": "Roadmap", "email": "bob@example.com", - "status": "pending", - "invited_by": { "id": "…", "email": "ann@example.com", "display_name": "Ann" }, - "invitee": { "id": "…", "email": "bob@example.com", "display_name": "Bob" }, - "created_at": "…", - "nudged_at": "…", - "nudged": true -} -``` - -Invite views (here and in both lists below) carry: - -- `invitee` — the account with that email, or `null` while nobody has signed up with it - (e.g. "waiting for them to make an account"); -- `nudged_at` — RFC 3339 time of the last re-send reminder, `null` until the first one. - -Both fields (and `nudged`) are new in 0.5.0; older clients ignore them. +Moved to [Canvas Cloud › REST](https://github.com/copper-browser/canvas-cloud/blob/main/docs/canvas.md#rest). ### `GET /canvases/{id}/invites` -Pending invites of a canvas (any member), newest first. +Moved to [Canvas Cloud › REST](https://github.com/copper-browser/canvas-cloud/blob/main/docs/canvas.md#rest). ### `DELETE /canvases/{id}/invites/{invite_id}` → `204` -Revoke a pending invite (0.5.0). Allowed for the canvas owner and for the member who sent -it; any other member gets `403 {"error":"forbidden"}`. An unknown invite, one of another -canvas, or one that is no longer pending (accepted, declined, already revoked) is -`404 {"error":"not_found"}`; non-members get `404` as everywhere. The invite's status becomes -`revoked`: it disappears from both lists, can no longer be accepted or declined (`404`), and -the email can be invited again (a new `201`). The invitee (if they have an account) gets a -`canvas` event with kind `invite_revoked`. +Moved to [Canvas Cloud › REST](https://github.com/copper-browser/canvas-cloud/blob/main/docs/canvas.md#rest). ### `GET /invites` -Pending invites addressed to **your** email (case-insensitive), with canvas name and inviter, -newest first. +Moved to [Canvas Cloud › REST](https://github.com/copper-browser/canvas-cloud/blob/main/docs/canvas.md#rest). ### `POST /invites/{id}/accept` → canvas (your view, `role: editor`) · `POST /invites/{id}/decline` → `{"ok":true}` -Only the invitee can accept or decline; anything else (wrong user, already used, declined, -revoked, unknown) is `404`. Invites are single-use. +Moved to [Canvas Cloud › REST](https://github.com/copper-browser/canvas-cloud/blob/main/docs/canvas.md#rest). ### Canvas share links -`POST /canvases/{id}/links` is available to any member. The body may be `{}` or -`{"role":"editor"}`; `owner` and other roles are rejected with `400`. It returns `201` and the -one-time-visible token: - -```json -{"id":"uuid","token":"<43 chars>","canvas_id":"uuid","role":"editor","created_at":"…"} -``` - -Owners can list links with `GET /canvases/{id}/links`: - -```json -{"links":[{"id":"uuid","role":"editor","created_by":"uuid","created_at":"…","uses":3}]} -``` - -Tokens are omitted from lists. Owners revoke one (`DELETE .../links/{link_id}`) or every link -(`DELETE .../links`); both return `204`. A missing link on these owner-scoped routes is `404`. - -A signed-in user can preview `GET /canvas-links/{token}` without being a member: -`{"canvas_id","name","owner":{"id","display_name"},"role","member"}`. Unknown, -revoked, or deleted-canvas tokens return `404` with `error: "link_not_found"`. - -`POST /canvas-links/{token}/join` is safe to retry. It adds an editor membership when needed, -never downgrades an existing owner, accepts/removes a pending email invite for the caller, -increments the link's `uses`, publishes the normal `member_added` canvas event, and returns the -same canvas object as one item from `GET /canvases`. - -Share links are unavailable for Personal canvases. They never expire on their own; deleting a -canvas cascades its links. +Moved: [Canvas Cloud › Canvas share links](https://github.com/copper-browser/canvas-cloud/blob/main/docs/canvas.md#canvas-share-links). +Tokens of links created on 0.8.x keep working after [migration](canvas-migration.md#what-is-preserved). ### `GET /canvases/{id}/state[?sv=]` -`{"state": ""}` — the whole document as one Yjs update -(standard base64, padded). With `sv` (a base64 lib0-v1 state vector) only what that state -vector is missing is returned. Apply with `Y.applyUpdate(doc, bytes)`. +Moved to [Canvas Cloud › REST](https://github.com/copper-browser/canvas-cloud/blob/main/docs/canvas.md#rest). ### `GET /canvases/{id}/read[?full=true&ids=a,b&types=sticky,text]` -The agent-friendly read format (see [canvas-protocol.md](canvas-protocol.md#read)), identical -to the page's `copperCanvas.read`. Text is cut to 500 characters unless `full=true`. Since -0.6.0 it also carries `chat`: the last 50 messages of the canvas chat, oldest first (see -[Chat and mentions](#chat-and-mentions)); `ids` / `types` narrow the shapes, not the chat. +Moved to [Canvas Cloud › REST](https://github.com/copper-browser/canvas-cloud/blob/main/docs/canvas.md#rest); +the format is in [Canvas Cloud › Read](https://github.com/copper-browser/canvas-cloud/blob/main/docs/canvas-protocol.md#read). ### `GET /people` -Any signed-in user may list active accounts on this instance (excluding themselves): -`{"people":[{"id","display_name","email"}]}`. `q` is a case-insensitive substring filter -on display name or email; results are sorted by display name. `limit` defaults to 20 and is -capped at 50. This is an intentional team-server directory surface; it is not restricted to -canvas members. +Moved to [Canvas Cloud › REST](https://github.com/copper-browser/canvas-cloud/blob/main/docs/canvas.md#rest). +Through copper-cloud it lists the instance's other Copper accounts (every Copper user is +provisioned). ### `POST /canvases/{id}/ops` -```json -{ "ops": [ { "op": "add", "shape": { "type": "sticky", "text": "Hello" } } ], - "as": { "name": "Planner", "color": "purple" }, - "near": { "x": 0, "y": 0 }, - "confirm": false } -``` - -Applies [canvas ops](canvas-protocol.md#ops) to the live document in one transaction, then -persists and broadcasts the change exactly like a WebSocket update — connected clients see -it immediately. Returns `{"applied": n, "ids": [...], "errors": [...]}` (`ids[i]` belongs to -op `i`, `null` when it failed). Lets remote agents draw without a client. - -- `as` (optional): who is drawing. Shapes created get `by = as.name`; without `as`, `by` is - your display name (or email). With `as`, the agent's entry in the `agents` map is set to - `writing` with its cursor on the last thing touched, then back to `idle` 1.5 s later. -- `near`: where auto-placement looks for free space (default `0,0`). -- `confirm: true` is required for `{op:"clear"}` (or put `confirm: true` on the op). -- At most 500 ops per call; the body may be a bare array of ops too. Envelope problems - (`ops` not an array, bad `as`, too many ops) are reported like the page does: `200` with - `errors: [{"index": -1, …}]`. Malformed JSON is `400`. +Moved to [Canvas Cloud › REST](https://github.com/copper-browser/canvas-cloud/blob/main/docs/canvas.md#rest) +and [Canvas Cloud › Ops](https://github.com/copper-browser/canvas-cloud/blob/main/docs/canvas-protocol.md#ops). +The proxy accepts request bodies up to 4 MiB here and allows 120 s. ### Chat and mentions -Shared canvases have a chat (0.6.0). The messages live **in the canvas document** — a -top-level `Y.Array` named `chat` of plain objects -`{id, authorId, authorName, text, mentions: [userId], at, editedAt?, deleted?}` (see -[canvas-protocol.md](canvas-protocol.md#document-schema)) — written by Copper and synced, -persisted, compacted and reloaded by the room like every other part of the document. The -server never writes it; `GET /canvases/{id}/read` shows the most recent 50. What the server -adds is **mention notifications**: after Copper inserts a message that @mentions people, it -reports them, and the server records one row per recipient and tells them. +Moved: [Canvas Cloud › Chat and mentions](https://github.com/copper-browser/canvas-cloud/blob/main/docs/canvas.md#chat-and-mentions). #### `POST /canvases/{id}/mentions {message_id, user_ids, excerpt}` → `200` -```json -{ "message_id": "msg_01J…", "user_ids": ["0192…ben", "0192…cy"], "excerpt": "@Ben @Cy dinner Tuesday?" } -``` - -Any member. `user_ids` is filtered to **current members of the canvas other than the -caller**; each such recipient gets a stored mention and a `canvas` event with kind `mention`. -Idempotent per `(canvas, message_id, recipient)`: repeating the call (a retry) answers the -same and notifies nobody again; the first excerpt wins. - -```json -{ "notified": ["0192…ben", "0192…cy"], "skipped": ["0192…left-the-canvas"] } -``` - -- `notified`: the requested users that are mentioned by this message (members other than you), - including on a retry; `skipped`: everyone else (yourself, non-members, unknown ids), in - request order, duplicates removed. -- `message_id`: 1–128 characters, no control characters (`400`). `user_ids`: UUIDs, at most - 100 (`400`); may be empty. `excerpt`: optional; control characters and runs of whitespace - become one space, then it is cut to 200 characters. It is sealed at rest under the canvas - doc key. -- Non-members and unknown canvases: `404` (as everywhere). The Personal canvas has no chat: - `400`. More than 60 calls per minute per user (bursts of 60, then one per second): `429 - {"error":"rate_limited"}`. +Moved to [Canvas Cloud › Chat and mentions](https://github.com/copper-browser/canvas-cloud/blob/main/docs/canvas.md#chat-and-mentions). #### `GET /mentions[?unread=1&limit=n]` -Your mentions on canvases you are still a member of, newest first. `unread=1` (or `true`) -returns only unread ones; `limit` defaults to 50 and is capped at 100 (`< 1` is `400`). -`unread` (in the body) counts every unread mention you can see, regardless of `limit`. - -```json -{ - "mentions": [ - { - "id": "0192…", - "canvas": { "id": "0192…", "name": "Team dinners" }, - "from": { "id": "0192…", "display_name": "Ann", "email": "ann@example.com" }, - "message_id": "msg_01J…", - "excerpt": "@Ben @Cy dinner Tuesday?", - "created_at": "2026-10-06T18:00:00Z", - "read_at": null - } - ], - "unread": 1 -} -``` - -Leaving a canvas hides its mentions (they come back if you rejoin); deleting a canvas or -either account deletes them. +Moved to [Canvas Cloud › Chat and mentions](https://github.com/copper-browser/canvas-cloud/blob/main/docs/canvas.md#chat-and-mentions). #### `POST /mentions/read {ids} | {canvas_id}` → `{"ok": true, "updated": n}` -Marks your own unread mentions read — by mention `id` (at most 500) and/or every mention on -`canvas_id`. Others' mentions and canvases you are not a member of are silently untouched -(`updated` counts what changed). For each canvas where something changed you get a `canvas` -event with kind `mention_read`, so your other devices clear their badges. Neither key → `400`. - -Clients gate this on `GET /v1/info` `version >= 0.6.0`; on older servers the chat still works -(it is part of the document) but there are no mention notifications. +Moved to [Canvas Cloud › Chat and mentions](https://github.com/copper-browser/canvas-cloud/blob/main/docs/canvas.md#chat-and-mentions). ### `GET /canvases/{id}/ws` — live room -WebSocket upgrade (HTTP/1.1 `GET`, or HTTP/2 extended `CONNECT`). Auth: the `Authorization` -header, or `?token=` for tools that cannot set headers; the `X-Copper-Instance` header is -still required. Non-members get `404` before the upgrade. Protocol details: -[canvas-protocol.md](canvas-protocol.md#wire-protocol). - -Example URL: `wss://HOST:PORT/v1/canvases/personal/ws?token=…`. A stock -`y-websocket` `WebsocketProvider(serverUrl = "wss://HOST:PORT/v1/canvases", room = "/ws", -doc, { params: { token } })` works as-is (its socket must also send the instance header). +Moved to Canvas Cloud; copper-cloud pipes the WebSocket byte for byte, close codes included: +[canvas-link.md › WebSocket pipe](canvas-link.md#websocket-pipe). ### Change notifications -Members get `event: canvas` on `GET /v1/sync/events` -(`{"type":"canvas","canvas_id":"…","kind":"…"}`) — refetch the list or the canvas: - -| `kind` | Sent to | When | -|---|---|---| -| `created` | owner | canvas created | -| `renamed` | members | rename | -| `deleted` | former members | delete | -| `invited` | the invitee (if they have an account) | new invite, or a re-send reminder (at most every 30 s per invite) | -| `invite_revoked` | the invitee (if they have an account) | owner or inviter revoked a pending invite | -| `invite_declined` | the inviter | invite declined | -| `member_added` | members | invite accepted | -| `member_removed` | members + the removed user | member removed / left | -| `update` | members | document changed (at most once per 2 s per canvas, plus one trailing) | -| `mention` | each newly mentioned member | someone @mentioned you in the canvas chat (0.6.0) | -| `mention_read` | you | your mentions on that canvas were marked read (e.g. on another device) (0.6.0) | +Still delivered as `event: canvas` on copper-cloud's `GET /v1/sync/events`, bridged from +Canvas Cloud: [canvas-link.md › Event bridge](canvas-link.md#event-bridge) (kinds table there). ## Rooms -One in-memory room per open canvas: a `yrs::Doc` behind a `tokio::sync::RwLock`, an -awareness table, and one `tokio::sync::broadcast` channel (256 frames) fanned out to peers. - -- **Load**: decrypt the snapshot, then every update with `seq > snapshot.seq`, in order. -- **Write path** (WebSocket update, REST ops, rename): take the write lock → apply in one - transaction → seal and `INSERT` the update the transaction produced → broadcast it to the - other peers. The lock is held across the insert, so persisted order = applied order = - broadcast order, and **nothing is broadcast before it is durable**. If the insert fails - the in-memory document is discarded (reloaded from Postgres on next use) and all peers are - disconnected (`1011`); y-websocket clients reconnect and their sync handshake re-sends - whatever the server is missing, so no edit is lost. -- **Compaction**: every 200 persisted updates, `encode_state_as_update_v1` of the room doc is - sealed into `canvas_snapshots` and the folded updates are deleted — one transaction. -- **Eviction**: a room with no peers and no REST use for 60 s is dropped from memory (it is - only removed when nothing else holds it, so an in-flight write can never be orphaned). -- **Awareness** (cursors/presence) is relayed between peers and kept in memory only, never - persisted. When a socket closes, the awareness states it published are removed and peers - are told. -- **Backpressure**: each peer has a read loop and a write task. Replies go through a bounded - per-peer queue (32); a peer that falls more than 256 broadcast frames behind, or does not - accept a frame within 15 s, is disconnected (`1013`) and resyncs on reconnect. The server - pings every 30 s and drops peers silent for 90 s. -- **Shutdown**: on the process shutdown signal every socket is closed with `1001`. +Moved: [Canvas Cloud › Rooms](https://github.com/copper-browser/canvas-cloud/blob/main/docs/canvas.md#rooms). ### Limits -| What | Limit | -|---|---| -| WebSocket message / frame | 8 MiB (the page stores images as `data:` URLs up to 2 MiB) | -| `POST /ops` body | 4 MiB; 500 ops per call | -| Other canvas request bodies | 64 KiB | -| Canvas name | 200 characters | -| Awareness clients per socket | 16 | -| Sticky/text body | 20 000 characters; titles/labels cut to 500 | -| Mentions | 100 `user_ids` per call; excerpt cut to 200 characters; `message_id` ≤ 128; 60 calls per minute per user; `GET /mentions` ≤ 100; `POST /mentions/read` ≤ 500 ids | +Canvas limits are Canvas Cloud's ([Limits](https://github.com/copper-browser/canvas-cloud/blob/main/docs/canvas.md#limits)); +the proxy's own body limits and timeouts: [canvas-link.md › Bodies, limits and timeouts](canvas-link.md#bodies-limits-and-timeouts). ### Metrics -| Name | Type | Meaning | -|---|---|---| -| `canvas_rooms` | gauge | rooms in memory | -| `canvas_peers` | gauge | connected WebSocket peers | -| `canvas_updates_persisted_total` | counter | Yjs updates written to `canvas_updates` | -| `canvas_update_bytes_total` | counter | plaintext bytes of those updates | -| `canvas_compactions_total` | counter | snapshot compactions | -| `canvas_ws_bytes_in_total` / `canvas_ws_bytes_out_total` | counter | WebSocket payload bytes | -| `canvas_mentions_total` | counter | chat mentions stored (new recipients notified) | -| `canvas_mentions_rate_limited_total` | counter | `POST /canvases/{id}/mentions` calls refused with `429` | - -`copper_cloud_canvas::rooms_metrics()` returns the live `{rooms, peers}` counts. +Room metrics are Canvas Cloud's ([Metrics](https://github.com/copper-browser/canvas-cloud/blob/main/docs/canvas.md#metrics)); +copper-cloud exports `canvas_link_ws_pipes` and `canvas_link_events_total` +([canvas-link.md › Metrics and logs](canvas-link.md#metrics-and-logs)). ## Web share-link landing page -`GET /join/{token}` is outside `/v1` and the instance-key gate. It returns a tiny self-contained -HTML page with an `Open this canvas in Copper` handoff when the request `Host` is a valid hostname -or IP with an optional port. It never looks up the token or includes canvas data. Invalid Host -headers receive a safe fallback message instead of a custom-scheme URL. The response is -`Cache-Control: no-store`, `Referrer-Policy: no-referrer`, `X-Content-Type-Options: nosniff`, -and CSP `default-src 'none'; style-src 'unsafe-inline'`. +Still served by copper-cloud, unchanged: `GET /join/{token}` (no gate, no database, no +Canvas Cloud call) hands off to `copper://canvas/join/{token}?cloud={host}` +(`crates/canvas-link/src/landing.rs`). ## Security model -- **Authentication** is the core session (`AuthUser`); the instance-key gate runs first. -- **Authorization** is membership, checked on every REST call and at WebSocket upgrade. - Removing a member or deleting a canvas disconnects affected sockets immediately. (A session - revoked by logout does not cut an already-open socket; it fails on reconnect.) -- **Encryption at rest**: each canvas has a random 32-byte `doc_key`, wrapped by the KEK - derived from `master_key` (`canvases.doc_key_wrapped`). Every update and snapshot is - AES-256-GCM `nonce(12) || ciphertext || tag` under that key with AAD = the canvas id's 16 - raw bytes, so rows copied to another canvas fail to decrypt. Key rotation is out of scope - (see [security.md](security.md)). Chat lives in the document, so it is sealed the same way; - mention excerpts are sealed under the same doc key with AAD - `"copper-cloud/v1/mention:" ‖ the mention id's 16 raw bytes`. Mention ids, message ids, - sender/recipient and timestamps are not encrypted. -- **Invite tokens**: a random secret is generated per invite but only its SHA-256 is stored - (reserved for future shareable links); v1 is email-invite only and never returns it. -- **Logging**: canvas and user ids, op counts and error kinds only — never tokens, document - contents, awareness state or update bytes. -- **Input**: y-protocol frames are decoded with `yrs`; undecodable frames or rejected updates - close that socket (`1007`) without touching the room. Ops are validated like the page - validates them (types, colours, URLs, sizes); bad ops are skipped and reported. +Canvas Cloud's: [Security model](https://github.com/copper-browser/canvas-cloud/blob/main/docs/canvas.md#security-model). +The link's (app key, sealing, TLS): [canvas-link.md › Security model](canvas-link.md#security-model). ## Testing -`cargo test -p copper-cloud-canvas` runs unit tests plus integration tests against the -native Postgres database `copper_cloud_test_canvas` (`postgres://localhost:5432/…`, current -OS user; override with `COPPER_CLOUD_TEST_CANVAS_DATABASE_URL`): REST lifecycle, Personal -canvas, invites (re-send reminders, revocation), 404 scoping, two-client sync + awareness, -persistence across eviction, compaction, ops fan-out, revocation, the `chat` array across -eviction / restart / compaction (`tests/chat.rs`), and mentions — member filtering, -idempotency, read marking, events, sealing, rate limit (`tests/mentions.rs`). - -A cross-implementation test drives the real page code (Yjs, `y-websocket`, -`Canvas/src/ops.ts`) with Bun and checks that the server's `/read` equals the page's -`readCanvas` on the same document: - -```sh -COPPER_CANVAS_DIR=~/src/copper-canvasweb/Canvas \ - cargo test -p copper-cloud-canvas --test js_interop -- --ignored -``` +The canvas engine tests moved to Canvas Cloud; the link's tests: [canvas-link.md › Testing](canvas-link.md#testing). diff --git a/docs/install.md b/docs/install.md index 4e6b1ce..36a3bd0 100644 --- a/docs/install.md +++ b/docs/install.md @@ -150,3 +150,18 @@ in the proxy. Link codes then carry an `fp` only if `tls.cert_path` points at th certificate; with a publicly trusted proxy certificate clients use normal validation. The proxy must pass `X-Copper-Instance`, allow long-lived responses (SSE: disable buffering) and WebSocket upgrades on `/v1/canvases/*/ws`. + +## Canvases (Canvas Cloud) + +Since 0.9.0 canvases are served by a linked [Canvas Cloud](https://github.com/copper-browser/canvas-cloud). +A fresh install runs without one (canvas routes answer `503 canvas_unavailable`; `doctor` +warns). Install Canvas Cloud — on the same host listening on `127.0.0.1` with TLS off is the +small setup — create an app key with `canvas-cloud apps create --name "Copper Cloud" +--allow-import`, then: + +```sh +sudo copper-cloud canvas link --url http://127.0.0.1:8480 --key-file /root/canvas-app.key +``` + +Steps and Canvas Cloud settings: [canvas-link.md › Linking](canvas-link.md#linking) and +[canvas-migration.md › Same-host Canvas Cloud](canvas-migration.md#0-same-host-canvas-cloud-if-not-installed-yet). diff --git a/docs/operations.md b/docs/operations.md index 69d3878..82e5224 100644 --- a/docs/operations.md +++ b/docs/operations.md @@ -15,14 +15,20 @@ copper-cloud doctor (0.8.1) ok tls self-signed, fp=…, 3640 days left, names: 203.0.113.10, localhost, 127.0.0.1 ok listener 0.0.0.0:443 answering /healthz over HTTPS (HTTP/1.1 200 OK), fingerprint matches ok metrics http://127.0.0.1:9464/metrics + ok canvas-link http://127.0.0.1:8480 (Canvas Cloud 0.1.0, 12 users, 36 canvases, 4 ms) doctor: all checks passed ``` Exit status is non-zero if any check FAILs (warnings do not fail). Checks: config parses and validates (and is not group/world-readable), database reachable, no pending migrations, certificate present/parseable/not expiring within 30 days/matches its key (or ACME cache), -the listener answers `/healthz` presenting the pinned certificate, metrics bind. Doctor never -prints the link code or keys. +the listener answers `/healthz` presenting the pinned certificate, metrics bind, and the +linked Canvas Cloud answers with the app key (`canvas-link`: `warn` when none is linked, +`FAIL` when it is unreachable or refuses the key — [canvas-link.md](canvas-link.md#status-and-doctor)). +Doctor never prints the link code or keys. + +- `copper-cloud canvas status` — the Canvas Cloud link, a fresh reachability check and the + migration summary (exit 1 when linked but unreachable). - `copper-cloud healthcheck --wait 60` — exit 0 once `/healthz` answers (scripts, installers). - `GET /healthz` → `ok` (no key needed; suitable for load-balancer checks). @@ -100,18 +106,36 @@ curl -s http://127.0.0.1:9464/metrics | `db_pool_size`, `db_pool_idle` | gauge | — (refreshed on scrape) | | `process_uptime_seconds` | gauge | — | -Canvas metrics (rooms, peers, bytes) are documented in [canvas.md](canvas.md). To scrape +| `canvas_link_ws_pipes` | gauge | — (proxied canvas WebSockets open now) | +| `canvas_link_events_total` | counter | — (canvas events bridged from Canvas Cloud into `/v1/sync/events`) | + +Room, peer and document metrics are Canvas Cloud's (its own `/metrics`). To scrape remotely, bind to a private interface (`metrics_bind = "10.0.0.5:9464"`) and firewall it, or scrape through an SSH tunnel / node agent. Starter alerts: `rate(http_requests_total{status=~"5.."}[5m]) > 0`, p99 of `http_request_duration_seconds` > 1 s, `up == 0`, `db_pool_idle == 0` sustained. +### Monitoring the canvas link + +- `copper-cloud doctor` (`canvas-link` line) or `copper-cloud canvas status` from cron / + a node agent: both exit non-zero when the linked Canvas Cloud is unreachable or refuses the + key. +- `http_requests_total{status="502"}` on canvas routes (`/v1/canvases…`) = Canvas Cloud + unreachable or rejecting the app key; `status="503"` = no link. The log lines carry + `span.error` `canvas_upstream` / `canvas_unavailable`. +- The portal's **Canvas Cloud** page (and `GET /admin/api/canvas-link` `bridge`) shows whether + the event bridge is connected; while it is not, canvas changes still work but Coppers are + not notified live. +- Monitor the Canvas Cloud itself too (its `doctor`, `/healthz`, metrics). + ## Backups What to back up: -1. **The database** (`copper_cloud`): all users, sessions, sync docs, history, canvases. +1. **The database** (`copper_cloud`): all users, sessions, sync docs, history, the canvas + link (app key sealed under the `master_key`), the migration ledger and — from before 0.9.0 — + the legacy canvas tables. ```sh sudo -u postgres pg_dump -Fc copper_cloud > copper_cloud-$(date +%F).dump # restore into an empty database: @@ -123,6 +147,15 @@ What to back up: from the DB dumps (password manager / secrets store). 3. **`/etc/copper-cloud/tls/`** (self-signed) — keeping the certificate keeps the pinned fingerprint valid. If lost, `tls-init` creates a new one and every Copper must re-link. +4. **The Canvas Cloud** (since 0.9.0 the canvases live there): its database (`canvas_cloud` + by default) and `/etc/canvas-cloud/canvas-cloud.toml` with its `master_key` — see the + [Canvas Cloud README](https://github.com/copper-browser/canvas-cloud). Back both servers up + at the same time: canvas user ids are Copper user ids, so a copper-cloud restore that + brings back deleted users, or a Canvas Cloud restore older than recent sign-ups, is + reconciled by the link (users are re-pushed every 10 min). App users without a Copper + account are never deleted automatically — the reconcile loop only logs them — so an older + copper-cloud restore cannot wipe newer users' canvases; review, then delete them with + `copper-cloud canvas sync-users --prune` if that is what you want. A restore onto a new VM: install with the same `COPPER_CLOUD_INSTANCE_KEY` and `COPPER_CLOUD_MASTER_KEY` (or copy the config), copy `tls/`, restore the dump, `migrate`, @@ -139,7 +172,26 @@ curl -fsSL https://raw.githubusercontent.com/copper-browser/copper-cloud/main/in The installer swaps the binary atomically, runs `migrate`, restarts and waits for health. Migrations are forward-only; take a DB backup first. Downgrading across a migration is not -supported. Clients reconnect automatically (SSE and canvas WebSockets drop during restart). +supported (for the 0.9.0 canvas move there is a documented rollback: +[canvas-migration.md](canvas-migration.md#rollback)). Clients reconnect automatically (SSE and +canvas WebSockets drop during restart). + +**Upgrading from 0.8.x to 0.9.0** moves canvases to a Canvas Cloud: follow the runbook in +[canvas-migration.md](canvas-migration.md) instead of a plain upgrade. After a plain upgrade +the server runs with canvases unavailable (`503`) until a Canvas Cloud is linked. + +### Canvas Cloud link + +```sh +sudo copper-cloud canvas link --url http://127.0.0.1:8480 --key-file /root/canvas-app.key [--fingerprint HEX] +sudo copper-cloud canvas status +sudo copper-cloud canvas sync-users # push every user now +sudo copper-cloud canvas unlink # canvases → 503 until linked again +``` + +Portal: **Canvas Cloud**. Rotating the app key: `canvas-cloud apps rotate ` on the +Canvas Cloud, then `canvas link` again with the same URL and the new key (`linked_at` is kept). +Details: [canvas-link.md](canvas-link.md). ## Administration @@ -266,3 +318,7 @@ sudo copper-cloud admin create-access-key --label "Ada" [--email ada@example.com | service restarts in a loop | `journalctl -u copper-cloud -n 50`: DB unreachable, bad config, port in use | | ACME never gets a certificate | DNS must point at the VM and :443 must be reachable from the internet; look for `acme` log lines; `tls.acme_staging = true` while testing | | "pending migrations" in doctor | `sudo copper-cloud migrate` (serve also migrates at start) | +| canvases fail with `503 canvas_unavailable` | no Canvas Cloud is linked: `sudo copper-cloud canvas status`, then `canvas link` ([canvas-link.md](canvas-link.md#linking)) | +| canvases fail with `502 canvas_upstream` | `sudo copper-cloud doctor` / `canvas status`: Canvas Cloud down (`systemctl status canvas-cloud`), wrong URL, certificate changed (re-link with the new `--fingerprint`), or the app key was revoked/rotated (re-link with a new key); `journalctl -u copper-cloud -o cat \| jq -c 'select(.span.error=="canvas_upstream") \| .span.error_message'` | +| canvas changes do not show up live on other Macs | the event bridge is disconnected: portal › Canvas Cloud › Live updates; `journalctl -u copper-cloud \| grep 'canvas event bridge'` | +| a person's canvas list is empty after linking | they were provisioned against a Canvas Cloud without their canvases: run the migration ([canvas-migration.md](canvas-migration.md)) | diff --git a/docs/security.md b/docs/security.md index 8950e46..9550911 100644 --- a/docs/security.md +++ b/docs/security.md @@ -110,7 +110,8 @@ rustls only (ring provider), TLS 1.2+ with rustls' safe defaults; HTTP/2 or HTTP SHA-256(token), so a DB leak does not yield usable sessions. Sliding 90-day expiry; expired rows are purged hourly. Password change revokes all other sessions; `admin reset-password`, `admin disable-user` and the portal's disable / reset-password revoke all (disable also - disconnects the user's live canvas peers). + disconnects the user's live canvas peers on the linked Canvas Cloud: at once from the portal, + within 10 minutes — the reconcile loop — from the CLI). - Rate limiting: `limits.auth_per_minute` (10) per client IP on `/v1/auth/*` (signup, login, logout, password, pair; `GET /v1/auth/me` and pairing-code management are exempt because a 256-bit bearer token cannot be brute-forced and clients use them as status probes). IPv6 @@ -127,7 +128,9 @@ Every query that touches user data is filtered by the authenticated `user_id` (d devices, sessions); that includes history deletes, which also refuse unknown query parameters and malformed bodies so a typo cannot widen a delete to all of the caller's history. `tabs:` docs are writable only by the owning device. Canvas -access is checked per membership by the canvas crate. Share-link preview and join routes require +access is checked per membership by the linked Canvas Cloud, for the Copper user that +copper-cloud names in `X-Canvas-User` after verifying the session (see +[Canvas link](#canvas-link)). Share-link preview and join routes require an active session; disabled accounts have no valid sessions and cannot use links. Link joins are serialized on the link row, use the membership primary key for idempotency, and never replace an existing owner role. @@ -142,7 +145,26 @@ excluded. A random 32-byte secret is generated for every canvas share link, but only its SHA-256 digest is stored. The plaintext is returned only from link creation, never from list/preview responses and never in logs. Owners can revoke one or all links; deleting a canvas cascades its links. Email -invite tokens follow the same digest-only storage rule. +invite tokens follow the same digest-only storage rule. Since 0.9.0 both are stored by Canvas +Cloud (the migration copies the digests, so existing tokens keep working). + +### Canvas link + +Since 0.9.0 canvases are stored and served by a linked Canvas Cloud; copper-cloud proxies to +it with one app key ([canvas-link.md › Security model](canvas-link.md#security-model)): + +- Coppers authenticate only to copper-cloud. Their session token, `X-Copper-Instance` and + cookies are never forwarded; copper-cloud adds `Authorization: Bearer ` and + `X-Canvas-User: `, which Canvas Cloud trusts for this app's users. +- **Whoever holds the app key can act as every Copper user on Canvas Cloud** (read, edit and + delete their canvases) and create, change or delete this app's users. It is stored in + `canvas_link.app_key_sealed` under the KEK envelope below (AAD `canvas_link:app_key`), never + returned by the admin API, shown only as its last four characters, read by the CLI from a + file or stdin (never argv), and kept out of logs and audit details. +- TLS to Canvas Cloud is verified against the web PKI (bundled Mozilla roots) or pinned to one + leaf certificate SHA-256; plain `http://` is accepted only for a loopback address. +- A Canvas Cloud `401` is turned into `502 canvas_upstream`, so an app-key problem never signs + Coppers out. ### Cloud-wide intelligence keys @@ -173,13 +195,18 @@ next fetch; `updated_at` changes). ``` master_key (config, 32 B) ──HKDF-SHA256(salt "copper-cloud", info "copper-cloud/v1/key-encryption-key")──▶ KEK KEK ──AES-256-GCM(aad "copper-cloud/v1/wrapped-key")──▶ users.data_key_wrapped, canvases.doc_key_wrapped, - intelligence_settings.data_key_wrapped + intelligence_settings.data_key_wrapped, + canvas_link.data_key_wrapped data_key ──AES-256-GCM(nonce 12 B random, aad ":")──▶ sync_docs.payload, history.payload intelligence data_key ──AES-256-GCM(aad "intelligence:jev" | "intelligence:router")──▶ jev_key_sealed, router_key_sealed -canvas doc_key ──AES-256-GCM(aad canvas id)──▶ canvas_updates."update", canvas_snapshots.state (incl. chat) -canvas doc_key ──AES-256-GCM(aad "copper-cloud/v1/mention:" ‖ mention id)──▶ canvas_mentions.excerpt_sealed +canvas_link data_key ──AES-256-GCM(aad "canvas_link:app_key")──▶ canvas_link.app_key_sealed +legacy (≤ 0.8) canvas doc_key ──AES-256-GCM(aad canvas id)──▶ canvas_updates."update", canvas_snapshots.state (incl. chat) +legacy (≤ 0.8) canvas doc_key ──AES-256-GCM(aad "copper-cloud/v1/mention:" ‖ mention id)──▶ canvas_mentions.excerpt_sealed ``` +The legacy canvas rows are only read (by `canvas migrate`); new canvas content is encrypted by +Canvas Cloud under its own `master_key`. + - Sealed blob layout: `nonce(12) ‖ ciphertext ‖ tag(16)`. - AAD binds each blob to its user and domain: copying a row to another user or domain makes it fail authentication. From a0b626266cc12f39003bc8f18742bb22c4da823c Mon Sep 17 00:00:00 2001 From: Felipe Arce Date: Mon, 19 Oct 2026 21:42:01 +0000 Subject: [PATCH 6/9] canvas link: provision new accounts via a users trigger (any path, any process; outbox polled every 5 s); pass Canvas Cloud resync on to every Copper stream; migration docs: pre-0.11 dump, digest-500/unverifiable rows, rollback drops the trigger; people scope --- crates/canvas-link/src/bridge.rs | 11 ++++-- crates/canvas-link/src/provision.rs | 9 +++-- crates/canvas-link/tests/support/mock.rs | 8 +++++ .../migrations/0030_canvas_link.sql | 15 ++++++++ crates/copper-cloud/tests/canvas_link.rs | 34 +++++++++++++++++++ crates/core/src/events.rs | 5 +++ docs/api.md | 6 ++-- docs/canvas-link.md | 14 +++++--- docs/canvas-migration.md | 14 +++++--- 9 files changed, 98 insertions(+), 18 deletions(-) diff --git a/crates/canvas-link/src/bridge.rs b/crates/canvas-link/src/bridge.rs index 32c51c7..88b27aa 100644 --- a/crates/canvas-link/src/bridge.rs +++ b/crates/canvas-link/src/bridge.rs @@ -2,8 +2,10 @@ //! `GET {url}/v1/apps/self/events` (`Last-Event-ID` resume, exponential backoff 1 s → 30 s). //! Each `canvas` event is re-published to the Copper user whose id is the event's `subject` //! as `Event::Canvas {canvas_id, kind}`, so Copper's `/v1/sync/events` stream carries the same -//! `event: canvas` frames as before the split. `resync` is logged (clients refetch on their -//! own); keep-alive comments only feed the idle watchdog. +//! `event: canvas` frames as before the split. A `resync` (Canvas Cloud could not resume from +//! our `Last-Event-ID`, e.g. after it restarted, so canvas events may have been missed) is +//! passed on to every connected Copper user as `event: resync`, which Copper clients answer by +//! refetching; keep-alive comments only feed the idle watchdog. use std::time::Duration; @@ -284,7 +286,10 @@ async fn session( link.update_bridge(|b| b.last_event_at = Some(OffsetDateTime::now_utc())); } "resync" => { - tracing::info!("canvas event bridge: resync (events were missed)"); + tracing::info!("canvas event bridge: resync (events may have been missed); telling clients to refetch"); + state.events.publish_all(&Event::Resync { + reason: "canvas".into(), + }); link.update_bridge(|b| b.last_event_at = Some(OffsetDateTime::now_utc())); } _ => {} diff --git a/crates/canvas-link/src/provision.rs b/crates/canvas-link/src/provision.rs index 692e6c1..7efc9dc 100644 --- a/crates/canvas-link/src/provision.rs +++ b/crates/canvas-link/src/provision.rs @@ -12,8 +12,11 @@ //! those orphans is an explicit operator action ([`prune_orphans`], //! `copper-cloud canvas sync-users --prune`). //! * The outbox (`canvas_link_outbox`): admin edits/deletes enqueue work in their own -//! transaction ([`enqueue_user_changed`], [`enqueue_user_deleted`]); [`outbox_loop`] pushes -//! it and retries failures with backoff, so nothing is lost while Canvas Cloud is down. +//! transaction ([`enqueue_user_changed`], [`enqueue_user_deleted`]), and every new account +//! is queued by the `canvas_link_user_created` trigger on `users` (migration 0030), whatever +//! inserted it (signup, SSO, `admin create-user`, another process); [`outbox_loop`] pushes +//! it (kicked, else every [`OUTBOX_POLL`]) and retries failures with backoff, so nothing is +//! lost while Canvas Cloud is down. use std::collections::{HashMap, HashSet}; use std::hash::{Hash as _, Hasher as _}; @@ -31,7 +34,7 @@ use crate::{Active, CanvasLink}; /// Reconcile period. pub const RECONCILE_EVERY: Duration = Duration::from_secs(600); /// Idle outbox poll (work is also kicked right after it is enqueued). -pub const OUTBOX_POLL: Duration = Duration::from_secs(30); +pub const OUTBOX_POLL: Duration = Duration::from_secs(5); /// Users per `users:batch` call (Canvas Cloud's limit). pub const BATCH: usize = 1000; /// Outbox items are dropped (with a warning) after this many failed attempts. diff --git a/crates/canvas-link/tests/support/mock.rs b/crates/canvas-link/tests/support/mock.rs index 6fc10f2..c7a836d 100644 --- a/crates/canvas-link/tests/support/mock.rs +++ b/crates/canvas-link/tests/support/mock.rs @@ -171,6 +171,14 @@ impl MockState { id } + /// Send `event: resync` (as Canvas Cloud does when it cannot resume a stream). + pub fn publish_resync(&self) { + let id = self.event_seq.load(Ordering::SeqCst); + let _ = self + .events + .send((id, "resync".into(), r#"{"reason":"gap"}"#.into())); + } + /// Number of connected event-stream clients. pub fn event_clients(&self) -> usize { self.events.receiver_count() diff --git a/crates/copper-cloud/migrations/0030_canvas_link.sql b/crates/copper-cloud/migrations/0030_canvas_link.sql index e2b81f4..f0f869e 100644 --- a/crates/copper-cloud/migrations/0030_canvas_link.sql +++ b/crates/copper-cloud/migrations/0030_canvas_link.sql @@ -42,6 +42,21 @@ CREATE TABLE canvas_link_outbox ( CREATE INDEX canvas_link_outbox_due_idx ON canvas_link_outbox (next_attempt_at, id); +-- Every new Copper account (password signup, SSO sign-in, `admin create-user`, the admin +-- portal, …) is queued for provisioning on Canvas Cloud while a link exists, in the inserting +-- transaction, whichever code path or process inserted it. The outbox worker picks it up +-- within seconds. Rollback drops this trigger and function before the tables. +CREATE FUNCTION canvas_link_user_created() RETURNS trigger LANGUAGE plpgsql AS $$ +BEGIN + INSERT INTO canvas_link_outbox (kind, subject) + SELECT 'upsert_user', NEW.id WHERE EXISTS (SELECT 1 FROM canvas_link WHERE id = 1); + RETURN NULL; +END +$$; + +CREATE TRIGGER canvas_link_user_created AFTER INSERT ON users + FOR EACH ROW EXECUTE FUNCTION canvas_link_user_created(); + -- Migration ledger of `copper-cloud canvas migrate` (idempotent re-runs, rollback evidence). -- status 'migrated' | 'failed' -- digest canonical document digest reported by Canvas Cloud (CONTRACT §6.3) diff --git a/crates/copper-cloud/tests/canvas_link.rs b/crates/copper-cloud/tests/canvas_link.rs index f525c31..50946a8 100644 --- a/crates/copper-cloud/tests/canvas_link.rs +++ b/crates/copper-cloud/tests/canvas_link.rs @@ -802,6 +802,40 @@ async fn event_bridge_feeds_sync_events() { let ev = next_event(&mut stream, &mut buf, "canvas").await; assert_eq!(ev["canvas_id"], canvas_id2.as_str()); assert_eq!(ev["kind"], "deleted"); + + // Canvas Cloud could not resume (it restarted): every connected user is told to refetch. + let resp = s + .authed(Method::GET, "/v1/sync/events", &bob) + .send() + .await + .unwrap(); + let mut bob_stream = resp.bytes_stream(); + let mut bob_buf = String::new(); + next_event(&mut bob_stream, &mut bob_buf, "ready").await; + m.st.publish_resync(); + for (stream, buf) in [(&mut stream, &mut buf), (&mut bob_stream, &mut bob_buf)] { + let ev = next_event(stream, buf, "resync").await; + assert_eq!(ev, json!({ "type": "resync", "reason": "canvas" })); + } +} + +/// Every new account is provisioned on Canvas Cloud within seconds, whichever path created +/// it — here a direct insert, as `admin create-user` (another process) does. +#[tokio::test] +async fn new_users_are_provisioned_right_away() { + let (s, _c, m, _ana, _bob) = setup().await; + let mut conn = s.state.db.acquire().await.unwrap(); + let email = format!("late-{}@example.com", uuid::Uuid::now_v7().simple()); + let user = + copper_cloud_core::auth::insert_user(&mut conn, &s.state.crypto, &email, "Late", None) + .await + .unwrap(); + drop(conn); + let id = user.id.to_string(); + wait_for("new user provisioned", || { + m.st.user(&id).is_some_and(|u| u.email == email) + }) + .await; } #[tokio::test] diff --git a/crates/core/src/events.rs b/crates/core/src/events.rs index aa9a0be..89ad31e 100644 --- a/crates/core/src/events.rs +++ b/crates/core/src/events.rs @@ -52,6 +52,10 @@ pub enum Event { /// Addons changed on this instance (admin upload / enable / audience / settings / delete); /// refetch `GET /v1/addons` with the last revision. Sent to every connected user. Addons, + /// Events may have been missed (a linked Canvas Cloud restarted, so its event stream + /// could not resume): refetch what you cache, as after a lagged stream. Sent to every + /// connected user as `event: resync`. + Resync { reason: String }, } impl Event { @@ -63,6 +67,7 @@ impl Event { Self::HistoryDeleted { .. } => "history_deleted", Self::Canvas { .. } => "canvas", Self::Addons => "addons", + Self::Resync { .. } => "resync", } } } diff --git a/docs/api.md b/docs/api.md index 6e94b44..fc54e52 100644 --- a/docs/api.md +++ b/docs/api.md @@ -672,8 +672,10 @@ data: {"skipped":12} on/off, audience, settings, priority, version, delete): refetch [`GET /v1/addons`](#addons) with `If-None-Match` (cheap when nothing changed for you). Changes made with the server CLI send no event. -- `resync`: this stream fell behind and dropped events; do a full pull - (`GET /v1/sync/docs` + history from your cursor). +- `resync`: this stream fell behind and dropped events (`{"skipped":n}`), or a linked Canvas + Cloud's events may have been missed (`{"type":"resync","reason":"canvas"}`, 0.11.0, sent to + every connected user); do a full pull (`GET /v1/sync/docs` + history from your cursor) and + refetch canvases. - A `: keepalive` comment every 15 s. The stream ends on server shutdown, when the session is revoked (checked every 5 min), or on network loss — reconnect with backoff and re-pull. - Events carry no payloads; fetch the doc/history after an event. `canvas` kinds are listed in diff --git a/docs/canvas-link.md b/docs/canvas-link.md index 4254d51..829b860 100644 --- a/docs/canvas-link.md +++ b/docs/canvas-link.md @@ -191,8 +191,11 @@ holds an SSE stream on `GET {url}/v1/apps/self/events` with the app key: ``` - `Last-Event-ID` resume: the bridge sends the last event id it saw when it reconnects (reset - when the link points at a new URL). `event: resync` (events were missed) is logged only — - Copper refreshes its lists on its own `open`/`resync`. + when the link points at a new URL). When Canvas Cloud cannot resume from it (it restarted, + or the bridge was away longer than its 4096-event buffer) it sends `event: resync`; the + bridge then sends `event: resync` (`data: {"type":"resync","reason":"canvas"}`) to every + connected Copper user, who refetch as after a lagged stream, so no canvas change is missed + silently. - Reconnects with exponential backoff, 1 s → 30 s; treats 45 s without any bytes (Canvas Cloud sends a keep-alive every 15 s) as a dead stream; follows link changes within 5 s. @@ -214,9 +217,9 @@ The `kind`s are Canvas Cloud's, unchanged from 0.8: Only users of this app receive events through the bridge, which is every Copper user. -`GET /v1/people` lists **all** enabled Canvas Cloud users, not only this app's: a Canvas Cloud -shared with other apps or with its own sign-ups shows those people to Coppers too (and they -can be invited and mentioned). +`GET /v1/people` is answered by Canvas Cloud for the signed-in user as this app's caller: it +lists every other user linked to this app (every provisioned Copper account), never Canvas +Cloud's own sign-ups or another app's users (Canvas Cloud's `docs/canvas.md`, `GET /people`). ## User provisioning @@ -242,6 +245,7 @@ When users are pushed: |---|---| | link (`canvas link`, portal, `PUT /admin/api/canvas-link`) | every user, `users:batch` in chunks of 1000 (the CLI waits for it unless `--no-sync`; the admin API runs it in the background) | | `copper-cloud canvas sync-users [--prune]` | every user, now; with `--prune` also deletes Canvas Cloud users of this link whose Copper account no longer exists (and the canvases they own there) | +| a new account (signup, SSO sign-in, `admin create-user`, admin portal) | queued in the creating transaction by the `canvas_link_user_created` trigger on `users` (migration 0030), so even another process's insert counts; pushed within seconds (the outbox is polled every 5 s) and retried while Canvas Cloud is down | | Canvas Cloud answers `401 unknown_user` to a proxied request | just-in-time `PUT` of that user, then the request is retried once (a user that no longer exists in Copper is deleted on Canvas Cloud instead) | | reconcile loop — at start, on every link change, then every 10 min | users whose email / display name / disabled flag changed since this process last pushed them (kept in memory, so after a restart everyone is pushed once); newly disabled users are also disconnected (`POST /v1/apps/self/users/{id}/disconnect`); app users whose Copper account no longer exists are only **counted and logged** (`canvas users reconciled … orphaned=N`) — reconcile never deletes, so restoring an older copper-cloud backup cannot wipe newer users' canvases; delete them deliberately with `canvas sync-users --prune` | | admin API `PATCH /admin/api/users/{id}` | outbox `upsert_user`, plus `disconnect_user` when it disables the user | diff --git a/docs/canvas-migration.md b/docs/canvas-migration.md index 4cc1864..3cace3a 100644 --- a/docs/canvas-migration.md +++ b/docs/canvas-migration.md @@ -114,9 +114,9 @@ The installer creates a separate `canvas_cloud` database on the same native Post ```sh sudo systemctl stop copper-cloud -sudo -u postgres pg_dump -Fc copper_cloud > /root/copper_cloud-pre-0.9-$(date +%F).dump +sudo -u postgres pg_dump -Fc copper_cloud > /root/copper_cloud-pre-0.11-$(date +%F).dump # RDS: aws rds create-db-snapshot --db-instance-identifier copper-cloud- \ -# --db-snapshot-identifier copper-cloud--pre-0-9 +# --db-snapshot-identifier copper-cloud--pre-0-11 sudo -u postgres psql -w -d copper_cloud -Atc 'SELECT count(*) FROM canvases' # legacy canvases ``` @@ -333,6 +333,8 @@ only. | `canvas exists in canvas-cloud but no digest is available (use --replace)` | the digest endpoint failed for an existing canvas; retry, or `--only ID --replace` | | `409 personal_exists … — the owner already has a non-empty Personal canvas in canvas-cloud (use --replace)` | the owner used canvases on Canvas Cloud before the migration (e.g. copper-cloud ran linked): `--only ID --replace` overwrites their Canvas Cloud Personal canvas with the legacy one | | `422 unknown_subject …` | a member/inviter/mention user is not provisioned (deleted between steps?): re-run | +| `verification failed: GET digest: HTTP 500` | Canvas Cloud stored the canvas but cannot read it back. Canvas Cloud before 0.1.0 could store an unreadable snapshot of a text edited inside an emoji (a yrs bug); 0.1.0 verifies snapshots. Upgrade Canvas Cloud, then `--only ID --replace` | +| `422 unverifiable_snapshot` | Canvas Cloud could not encode this canvas in a form that reads back identically (yrs bug with a text split inside an emoji); nothing was stored. The canvas stays on the legacy tables; report it with the canvas id | | `422 bad_update` / `422 incomplete_history` | a legacy update does not decode, or updates are missing: the legacy document is damaged. Check it with the dry run and `GET …/read` on the old release | | `413 …` | the canvas is bigger than Canvas Cloud's `limits.import_max_bytes` (default 512 MiB) or a line over `limits.import_max_line_bytes` (32 MiB): raise them on Canvas Cloud, restart it, re-run | | `reading the legacy canvas: update seq N does not decrypt` / `the snapshot (seq N) does not decrypt` | a damaged or foreign row in the legacy tables; nothing was imported for that canvas | @@ -356,14 +358,16 @@ only. ## Rollback The legacy tables are untouched, so the previous release still has every canvas as it was at -the moment of the migration. +the moment of the migration. Make sure the new copper-cloud is fully stopped first +(`systemctl is-active copper-cloud` says `inactive`); the SQL below works as the +`postgres` superuser or as the database owner. ```sh sudo systemctl stop copper-cloud # Older releases refuse to start with a migration it does not know (sqlx: "migration 30 was previously -# applied but is missing"): remove 0030's link tables and its record; keep the ledger. +# applied but is missing"): remove 0030's trigger, link tables and record; keep the ledger. sudo -u postgres psql -w -d copper_cloud -v ON_ERROR_STOP=1 -c \ - "BEGIN; DROP TABLE canvas_link_outbox; DROP TABLE canvas_link; DELETE FROM _sqlx_migrations WHERE version = 30; COMMIT;" + "BEGIN; DROP TRIGGER canvas_link_user_created ON users; DROP FUNCTION canvas_link_user_created(); DROP TABLE canvas_link_outbox; DROP TABLE canvas_link; DELETE FROM _sqlx_migrations WHERE version = 30; COMMIT;" tar xzf copper-cloud-0.10.0-linux-$(uname -m).tar.gz && sudo ./install.sh # reinstall the previous release ``` From c0f2e2b768649d2ca8eaaa0b554cf4ccbacd6544 Mon Sep 17 00:00:00 2001 From: Felipe Arce Date: Mon, 19 Oct 2026 21:51:12 +0000 Subject: [PATCH 7/9] canvas migrate: ledger keeps a fingerprint of the legacy rows; a re-run fails (not 'already') a canvas edited on the old release since its import --- crates/canvas-migrate/src/ledger.rs | 24 ++++++++++++---- crates/canvas-migrate/src/legacy.rs | 38 ++++++++++++++++++++------ crates/canvas-migrate/src/lib.rs | 37 +++++++++++++++++++------ crates/canvas-migrate/tests/migrate.rs | 29 ++++++++++++++++++-- docs/canvas-migration.md | 11 +++++--- 5 files changed, 111 insertions(+), 28 deletions(-) diff --git a/crates/canvas-migrate/src/ledger.rs b/crates/canvas-migrate/src/ledger.rs index 6aa8d4c..94af10c 100644 --- a/crates/canvas-migrate/src/ledger.rs +++ b/crates/canvas-migrate/src/ledger.rs @@ -24,6 +24,8 @@ use uuid::Uuid; pub(crate) struct Entry { pub status: String, pub digest: Option, + /// [`crate::legacy::LegacyCanvas::mark`] when it was imported (`None` in older ledgers). + pub mark: Option, } impl Entry { @@ -35,14 +37,24 @@ impl Entry { } pub(crate) async fn load(db: &PgPool) -> anyhow::Result> { - let rows: Vec<(Uuid, String, Option)> = - sqlx::query_as("SELECT canvas_id, status, digest FROM canvas_migrations") - .fetch_all(db) - .await - .context("reading the canvas_migrations ledger")?; + let rows: Vec<(Uuid, String, Option, Option)> = sqlx::query_as( + "SELECT canvas_id, status, digest, detail->>'legacy_mark' FROM canvas_migrations", + ) + .fetch_all(db) + .await + .context("reading the canvas_migrations ledger")?; Ok(rows .into_iter() - .map(|(id, status, digest)| (id, Entry { status, digest })) + .map(|(id, status, digest, mark)| { + ( + id, + Entry { + status, + digest, + mark, + }, + ) + }) .collect()) } diff --git a/crates/canvas-migrate/src/legacy.rs b/crates/canvas-migrate/src/legacy.rs index cf62428..fe40040 100644 --- a/crates/canvas-migrate/src/legacy.rs +++ b/crates/canvas-migrate/src/legacy.rs @@ -124,21 +124,43 @@ pub(crate) struct LegacyCanvas { pub owner_id: Uuid, /// Sealed bytes of snapshot + update tail (sizing the import timeout). pub stored_bytes: i64, + /// Fingerprint of the legacy rows (name, kind, snapshot seq, update count and last seq, + /// member / invite / link / mention counts). Kept in the ledger at import, so a re-run + /// notices a canvas that changed on the legacy tables since (an edit after a rollback). + pub mark: String, } -pub(crate) async fn canvases(db: &PgPool) -> anyhow::Result> { - sqlx::query_as( +pub(crate) async fn canvases(db: &PgPool, schema: &Schema) -> anyhow::Result> { + let links = if schema.links { + "(SELECT count(*) FROM canvas_share_links l WHERE l.canvas_id = c.id)" + } else { + "0" + }; + let mentions = if schema.mentions { + "(SELECT count(*) FROM canvas_mentions n WHERE n.canvas_id = c.id)" + } else { + "0" + }; + let sql = format!( r#"SELECT c.id, c.name, c.kind, c.owner_id, (COALESCE((SELECT octet_length(s.state) FROM canvas_snapshots s WHERE s.canvas_id = c.id), 0) + COALESCE((SELECT sum(octet_length(u."update")) FROM canvas_updates u - WHERE u.canvas_id = c.id), 0))::bigint AS stored_bytes + WHERE u.canvas_id = c.id), 0))::bigint AS stored_bytes, + md5(concat_ws('|', c.name, c.kind, + (SELECT s.seq FROM canvas_snapshots s WHERE s.canvas_id = c.id), + (SELECT count(*) || ':' || COALESCE(max(u.seq), 0) FROM canvas_updates u + WHERE u.canvas_id = c.id), + (SELECT count(*) FROM canvas_members m WHERE m.canvas_id = c.id), + (SELECT count(*) FROM canvas_invites i WHERE i.canvas_id = c.id), + {links}, {mentions})) AS mark FROM canvases c - ORDER BY c.created_at, c.id"#, - ) - .fetch_all(db) - .await - .context("reading canvases") + ORDER BY c.created_at, c.id"# + ); + sqlx::query_as(&sql) + .fetch_all(db) + .await + .context("reading canvases") } /// What one canvas stream has produced so far (shared with the caller, which reads it after diff --git a/crates/canvas-migrate/src/lib.rs b/crates/canvas-migrate/src/lib.rs index 2ea448c..f06f9ae 100644 --- a/crates/canvas-migrate/src/lib.rs +++ b/crates/canvas-migrate/src/lib.rs @@ -130,7 +130,7 @@ pub async fn run( } // 2. Which canvases. - let (plan, unknown_only) = plan(legacy::canvases(db).await?, &ledger, &opts); + let (plan, unknown_only) = plan(legacy::canvases(db, &schema).await?, &ledger, &opts); // 3. Canvases, `concurrency` at a time, results in legacy order. let ctx = Ctx { @@ -214,9 +214,14 @@ fn plan( if !only.is_empty() && !only.contains(&canvas.id) { continue; } - let migrated = ledger - .get(&canvas.id) - .filter(|e| e.status == "migrated" && !opts.replace && !only.contains(&canvas.id)); + // A canvas that changed on the legacy tables since its import goes through the full + // path, which reports it (see `Ctx::exists`). + let migrated = ledger.get(&canvas.id).filter(|e| { + e.status == "migrated" + && !opts.replace + && !only.contains(&canvas.id) + && e.mark.as_ref().is_none_or(|m| *m == canvas.mark) + }); plan.push(match migrated { Some(entry) => Plan::Skip(canvas, entry.digest.clone()), None => Plan::Go(canvas), @@ -436,7 +441,7 @@ impl Ctx<'_> { ); return self.fail_changed(canvas, row, msg, &digest).await; } - let detail = self.detail(row, if replaced { "replaced" } else { "created" }); + let detail = self.detail(canvas, row, if replaced { "replaced" } else { "created" }); ledger::record( self.db, canvas.id, @@ -498,6 +503,16 @@ impl Ctx<'_> { stored: &str, ) -> anyhow::Result<()> { let entry = self.ledger.get(&canvas.id); + let legacy_changed = entry + .and_then(|e| e.mark.as_ref()) + .is_some_and(|m| *m != canvas.mark); + if conflict == stored && entry.is_some_and(|e| e.vouches_for(stored)) && legacy_changed { + let msg = "changed in the legacy tables since it was migrated (edited in the old \ + copper-cloud after a rollback?) — use --only ID --replace to re-import \ + it, overwriting the Canvas Cloud copy" + .to_owned(); + return self.fail(canvas, row, msg, None).await; + } if conflict == stored && entry.is_some_and(|e| e.vouches_for(stored)) { if entry.is_some_and(|e| e.status != "migrated") { ledger::confirm( @@ -526,8 +541,14 @@ impl Ctx<'_> { self.fail(canvas, row, msg, None).await } - fn detail(&self, row: &CanvasReport, import_status: &str) -> serde_json::Value { + fn detail( + &self, + canvas: &LegacyCanvas, + row: &CanvasReport, + import_status: &str, + ) -> serde_json::Value { json!({ + "legacy_mark": canvas.mark, "name": row.name, "kind": row.kind, "owner": row.owner, @@ -552,7 +573,7 @@ impl Ctx<'_> { msg: String, digest: Option<&str>, ) -> anyhow::Result<()> { - let mut detail = self.detail(row, "none"); + let mut detail = self.detail(canvas, row, "none"); detail["error"] = json!(msg); ledger::record( self.db, @@ -577,7 +598,7 @@ impl Ctx<'_> { msg: String, digest: &str, ) -> anyhow::Result<()> { - let mut detail = self.detail(row, "unverified"); + let mut detail = self.detail(canvas, row, "unverified"); detail["error"] = json!(msg); ledger::record( self.db, diff --git a/crates/canvas-migrate/tests/migrate.rs b/crates/canvas-migrate/tests/migrate.rs index 24ca16d..e7f12f0 100644 --- a/crates/canvas-migrate/tests/migrate.rs +++ b/crates/canvas-migrate/tests/migrate.rs @@ -487,13 +487,38 @@ async fn replace_reimports_canvases_that_changed_in_copper() { let new = w.roadmap.truth(); assert_ne!(old, new); - // A plain re-run trusts the ledger … + // A plain re-run notices the legacy change and says how to bring it over, changing + // nothing (and the other canvases stay `already`) … let plain = env.migrate(opts()).await.unwrap(); - assert_eq!(row(&plain, w.roadmap.id).status, Status::Already); + assert!(!plain.ok(), "{plain}"); + let r = row(&plain, w.roadmap.id); + assert_eq!(r.status, Status::Failed); + assert!( + r.error + .as_deref() + .unwrap() + .contains("changed in the legacy tables"), + "{r:?}" + ); + assert_eq!(plain.totals.already, 3, "{plain}"); assert_eq!( env.mock.stored_digest(w.roadmap.id).as_deref(), Some(old.as_str()) ); + // … so does `--only` it without `--replace`, and the ledger keeps the migrated record. + let only = env + .migrate(MigrateOptions { + only: vec![w.roadmap.id], + ..opts() + }) + .await + .unwrap(); + assert_eq!(row(&only, w.roadmap.id).status, Status::Failed, "{only}"); + let (status, digest, _) = env.ledger_row(w.roadmap.id).await.unwrap(); + assert_eq!( + (status.as_str(), digest.as_deref()), + ("migrated", Some(old.as_str())) + ); // … --replace re-imports every canvas from the legacy data. let replaced = env diff --git a/docs/canvas-migration.md b/docs/canvas-migration.md index 3cace3a..84bccb4 100644 --- a/docs/canvas-migration.md +++ b/docs/canvas-migration.md @@ -229,7 +229,7 @@ In the admin portal: **Canvas Cloud** shows the link reachable and live updates | Command | What it does | |---|---| -| `canvas migrate` | every legacy canvas not yet `migrated` in the ledger; ledger-`migrated` canvases are reported `already` without contacting Canvas Cloud | +| `canvas migrate` | every legacy canvas not yet `migrated` in the ledger; ledger-`migrated` canvases are reported `already` without contacting Canvas Cloud, unless their legacy rows changed since the import (then failed, see [rollback](#rollback)) | | `canvas migrate --only ID [--only ID]…` | just those canvases, **even if** the ledger says migrated: a canvas that exists on Canvas Cloud with the digest the ledger recorded is reported `already` (and an `imported` row is confirmed as `migrated`); different content fails with "use --replace". An id that is not a legacy canvas is reported failed | | `canvas migrate --replace` | every legacy canvas, ledger ignored, imported with `mode=replace`: a canvas that already exists on Canvas Cloud is deleted and re-imported atomically from the legacy data (its room is closed first). Only canvases that an import by this app created can be replaced. **Edits made on Canvas Cloud since are lost** | | `canvas migrate --only ID --replace` | overwrite one canvas from the legacy data | @@ -329,6 +329,7 @@ only. | `user …: missing from the users:batch response` / `: ` | Canvas Cloud rejected that user (e.g. invalid email); fix the account in Copper or on Canvas Cloud | | `provisioning users: POST …/users:batch answered …` | Canvas Cloud refused the batch; see its logs | | `exists in canvas-cloud with different content (digest …; it changed since this ledger recorded it (edited in canvas-cloud?)) — use --replace …` | someone edited it on Canvas Cloud after the migration: leave it (the Canvas Cloud copy is newer) or `--only ID --replace` to restore the legacy content | +| `changed in the legacy tables since it was migrated (edited in the old copper-cloud after a rollback?) — use --only ID --replace …` | the canvas was edited on the old release after its import (a rollback period): `--only ID --replace` brings the legacy content over, overwriting the Canvas Cloud copy | | `exists in canvas-cloud with different content (…; this ledger has no verified record of importing it) — use --replace …` | a canvas with that id exists on Canvas Cloud from another import (another copper-cloud database, an earlier attempt without a ledger): `--only ID --replace` after checking | | `canvas exists in canvas-cloud but no digest is available (use --replace)` | the digest endpoint failed for an existing canvas; retry, or `--only ID --replace` | | `409 personal_exists … — the owner already has a non-empty Personal canvas in canvas-cloud (use --replace)` | the owner used canvases on Canvas Cloud before the migration (e.g. copper-cloud ran linked): `--only ID --replace` overwrites their Canvas Cloud Personal canvas with the legacy one | @@ -386,6 +387,8 @@ What happens to work done after the cutover: - Sync data, accounts and devices are unaffected (same tables in both releases). - Users deleted after the upgrade have lost their legacy canvases too (cascade). - Upgrading again after a rollback: the ledger still says `migrated` and Canvas Cloud still has - the earlier imports, so a plain `canvas migrate` reports them `already` even if they were - edited on the old release in between. Use `canvas migrate --replace` to bring the legacy - content over again (this overwrites what Canvas Cloud has). + the earlier imports. The ledger also keeps a fingerprint of each canvas's legacy rows at + import, so `canvas migrate` reports canvases unchanged since as `already` and every canvas + edited on the old release in between as **failed** ("changed in the legacy tables since it + was migrated"), changing nothing. Bring those over with `canvas migrate --only ID --replace` + (this overwrites what Canvas Cloud has for that canvas), or all at once with `--replace`. From 8c4d3f07480309920e5f02beff2097104726b20b Mon Sep 17 00:00:00 2001 From: Felipe Arce Date: Mon, 19 Oct 2026 21:57:01 +0000 Subject: [PATCH 8/9] migration runbook: install Canvas Cloud on loopback/TLS-off with the 0.1.0 installer inputs; RDS DATABASE_URL; set ops.extra_url_schemes in place --- docs/canvas-migration.md | 29 +++++++++++++++-------------- 1 file changed, 15 insertions(+), 14 deletions(-) diff --git a/docs/canvas-migration.md b/docs/canvas-migration.md index 84bccb4..9b104e5 100644 --- a/docs/canvas-migration.md +++ b/docs/canvas-migration.md @@ -78,27 +78,29 @@ All commands as root on the copper-cloud host unless noted. Every step can be re ### 0. Same-host Canvas Cloud (if not installed yet) The Canvas Cloud installer listens on `0.0.0.0:443` by default, which copper-cloud already -uses. Install on another port, then move it to loopback with TLS off: +uses. Install it on loopback with TLS off (Canvas Cloud 0.1.0's `install.sh`; on loopback it +leaves the firewall alone): ```sh curl -fsSL https://raw.githubusercontent.com/copper-browser/canvas-cloud/main/install.sh \ - | sudo CANVAS_CLOUD_PORT=8480 sh + | sudo CANVAS_CLOUD_LISTEN=127.0.0.1:8480 CANVAS_CLOUD_TLS_MODE=off \ + CANVAS_CLOUD_PUBLIC_URL=http://127.0.0.1:8480 sh ``` -Edit `/etc/canvas-cloud/canvas-cloud.toml` — set `listen` and add or change the two tables: +By default the installer creates a separate `canvas_cloud` database on a native PostgreSQL +on the same host. When copper-cloud uses a managed database (RDS), create a role and database +for Canvas Cloud there and pass it instead: `DATABASE_URL=postgres://canvas_cloud:…@:5432/canvas_cloud?sslmode=require` +in the same environment (keep it off shell history, e.g. read it from a file). -```toml -listen = "127.0.0.1:8480" +Then, in `/etc/canvas-cloud/canvas-cloud.toml`, let link shapes keep their `copper://` URLs +(the generated file has the `[ops]` table with an empty list): -[tls] -mode = "off" - -[ops] -extra_url_schemes = ["copper"] +```sh +sudo sed -i 's/^extra_url_schemes = \[\]$/extra_url_schemes = ["copper"]/' /etc/canvas-cloud/canvas-cloud.toml +grep '^extra_url_schemes' /etc/canvas-cloud/canvas-cloud.toml # extra_url_schemes = ["copper"] ``` ```sh -sudo ufw delete allow 8480/tcp 2>/dev/null || true # the installer opened it (ufw/firewalld) sudo systemctl restart canvas-cloud timeout 10 curl -fsS http://127.0.0.1:8480/v1/info umask 077 @@ -106,9 +108,8 @@ sudo canvas-cloud apps create --name "Copper Cloud" --allow-import > /root/canva ``` `apps create` prints only the key on stdout (the "created app " line goes to stderr; note -the id for `apps rotate` / `apps revoke`). - -The installer creates a separate `canvas_cloud` database on the same native PostgreSQL. +the id for `apps rotate` / `apps revoke`). Back up `/etc/canvas-cloud/canvas-cloud.toml` (it +holds Canvas Cloud's `master_key`) alongside copper-cloud's config. ### 1. Stop copper-cloud and back up From f8a44c3693a185eb024bee91049db4c51e4b0dfd Mon Sep 17 00:00:00 2001 From: Felipe Arce Date: Mon, 19 Oct 2026 22:26:15 +0000 Subject: [PATCH 9/9] canvas link: after a Canvas Cloud resync also send every user event: canvas {kind: resync} (Copper for Mac refreshes its canvas list and invites only on canvas events) --- crates/canvas-link/src/bridge.rs | 9 ++++++++- crates/copper-cloud/tests/canvas_link.rs | 4 ++++ crates/core/src/events.rs | 19 ++++++++++++++++++- docs/api.md | 3 ++- docs/canvas-link.md | 7 +++++-- 5 files changed, 37 insertions(+), 5 deletions(-) diff --git a/crates/canvas-link/src/bridge.rs b/crates/canvas-link/src/bridge.rs index 88b27aa..805d416 100644 --- a/crates/canvas-link/src/bridge.rs +++ b/crates/canvas-link/src/bridge.rs @@ -5,7 +5,9 @@ //! `event: canvas` frames as before the split. A `resync` (Canvas Cloud could not resume from //! our `Last-Event-ID`, e.g. after it restarted, so canvas events may have been missed) is //! passed on to every connected Copper user as `event: resync`, which Copper clients answer by -//! refetching; keep-alive comments only feed the idle watchdog. +//! refetching, followed by `event: canvas` `{"type":"canvas","kind":"resync"}` (no +//! `canvas_id`) for clients that refresh their canvas list only on `canvas` events (Copper for +//! Mac); keep-alive comments only feed the idle watchdog. use std::time::Duration; @@ -290,6 +292,11 @@ async fn session( state.events.publish_all(&Event::Resync { reason: "canvas".into(), }); + // Copper for Mac refreshes its canvas list and invites on `canvas` events + // only, not on `resync`. + state + .events + .publish_all(&Event::CanvasResync { kind: "resync" }); link.update_bridge(|b| b.last_event_at = Some(OffsetDateTime::now_utc())); } _ => {} diff --git a/crates/copper-cloud/tests/canvas_link.rs b/crates/copper-cloud/tests/canvas_link.rs index 50946a8..9540544 100644 --- a/crates/copper-cloud/tests/canvas_link.rs +++ b/crates/copper-cloud/tests/canvas_link.rs @@ -816,6 +816,10 @@ async fn event_bridge_feeds_sync_events() { for (stream, buf) in [(&mut stream, &mut buf), (&mut bob_stream, &mut bob_buf)] { let ev = next_event(stream, buf, "resync").await; assert_eq!(ev, json!({ "type": "resync", "reason": "canvas" })); + // …and a canvas event without a canvas, for clients that refresh their canvas list + // only on `canvas` (Copper for Mac). + let ev = next_event(stream, buf, "canvas").await; + assert_eq!(ev, json!({ "type": "canvas", "kind": "resync" })); } } diff --git a/crates/core/src/events.rs b/crates/core/src/events.rs index 89ad31e..e4a8d5d 100644 --- a/crates/core/src/events.rs +++ b/crates/core/src/events.rs @@ -56,6 +56,13 @@ pub enum Event { /// could not resume): refetch what you cache, as after a lagged stream. Sent to every /// connected user as `event: resync`. Resync { reason: String }, + /// Canvases, invites or mentions may have changed while canvas events were lost (sent + /// right after `Resync` when a linked Canvas Cloud's event stream could not resume): + /// refetch them. SSE `event: canvas`, data `{"type":"canvas","kind":"resync"}` — no + /// `canvas_id` — so clients that refresh their canvas list only on `canvas` events + /// (Copper for Mac) catch up as well. + #[serde(rename = "canvas")] + CanvasResync { kind: &'static str }, } impl Event { @@ -65,7 +72,7 @@ impl Event { Self::Doc { .. } => "doc", Self::History { .. } => "history", Self::HistoryDeleted { .. } => "history_deleted", - Self::Canvas { .. } => "canvas", + Self::Canvas { .. } | Self::CanvasResync { .. } => "canvas", Self::Addons => "addons", Self::Resync { .. } => "resync", } @@ -173,6 +180,16 @@ mod tests { ); } + #[test] + fn canvas_resync_is_a_canvas_event_without_a_canvas() { + let ev = Event::CanvasResync { kind: "resync" }; + assert_eq!(ev.name(), "canvas"); + assert_eq!( + serde_json::to_string(&ev).unwrap(), + r#"{"type":"canvas","kind":"resync"}"# + ); + } + #[test] fn publish_without_subscribers_cleans_up() { let ev = Events::new(); diff --git a/docs/api.md b/docs/api.md index fc54e52..b9220b2 100644 --- a/docs/api.md +++ b/docs/api.md @@ -674,7 +674,8 @@ data: {"skipped":12} Changes made with the server CLI send no event. - `resync`: this stream fell behind and dropped events (`{"skipped":n}`), or a linked Canvas Cloud's events may have been missed (`{"type":"resync","reason":"canvas"}`, 0.11.0, sent to - every connected user); do a full pull (`GET /v1/sync/docs` + history from your cursor) and + every connected user, then followed by `event: canvas` `{"type":"canvas","kind":"resync"}` + without a `canvas_id`); do a full pull (`GET /v1/sync/docs` + history from your cursor) and refetch canvases. - A `: keepalive` comment every 15 s. The stream ends on server shutdown, when the session is revoked (checked every 5 min), or on network loss — reconnect with backoff and re-pull. diff --git a/docs/canvas-link.md b/docs/canvas-link.md index 829b860..9f2baf5 100644 --- a/docs/canvas-link.md +++ b/docs/canvas-link.md @@ -194,8 +194,10 @@ holds an SSE stream on `GET {url}/v1/apps/self/events` with the app key: when the link points at a new URL). When Canvas Cloud cannot resume from it (it restarted, or the bridge was away longer than its 4096-event buffer) it sends `event: resync`; the bridge then sends `event: resync` (`data: {"type":"resync","reason":"canvas"}`) to every - connected Copper user, who refetch as after a lagged stream, so no canvas change is missed - silently. + connected Copper user, who refetch as after a lagged stream, followed by + `event: canvas` (`data: {"type":"canvas","kind":"resync"}`, no `canvas_id`) for clients + that refresh their canvas list and invites only on `canvas` events (Copper for Mac), so no + canvas change is missed silently. - Reconnects with exponential backoff, 1 s → 30 s; treats 45 s without any bytes (Canvas Cloud sends a keep-alive every 15 s) as a dead stream; follows link changes within 5 s. @@ -214,6 +216,7 @@ The `kind`s are Canvas Cloud's, unchanged from 0.8: | `update` | members | document changed (at most once per 2 s per canvas, plus one trailing) | | `mention` | each newly mentioned member | someone @mentioned you in the canvas chat | | `mention_read` | you | your mentions on that canvas were marked read | +| `resync` (copper-cloud, 0.11.0; no `canvas_id`) | every connected user | the bridge could not resume Canvas Cloud's event stream: refetch canvases, invites and mentions | Only users of this app receive events through the bridge, which is every Copper user.