Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
93dd92e
Add live information layer console
Aug 16, 2026
d71b4b1
Add Linux server deployment support
Aug 16, 2026
dc9913b
Fix Ubuntu shell validation
Aug 16, 2026
7541b6f
Wait for final PostGIS startup
Aug 16, 2026
342d9e5
Run Linux API with host identity
Aug 16, 2026
4d6c930
Probe configured TerraSys bind address
Aug 16, 2026
d41289f
Use resilient CDN for browser glyphs
Aug 16, 2026
aeb1e8f
Download browser glyphs concurrently
Aug 16, 2026
c9bb17d
Use resilient public OSM sources on Linux
Aug 16, 2026
cca9d6b
Precreate Docker bind targets on Linux
Aug 16, 2026
c71cea5
Normalize API image permissions for Linux UID
Aug 16, 2026
e22ba86
Bound expected references in China extract
Aug 16, 2026
bc28bb1
Cache Planetiler sources with resumable downloads
Aug 16, 2026
c31348e
Bound provider reference omissions
Aug 16, 2026
d877be4
Resume Natural Earth CDN downloads
Aug 16, 2026
9af968b
Download OSM Carto sources on Linux
Aug 16, 2026
60e297f
Fallback to digest-pinned Carto registry
Aug 16, 2026
5cf6e1f
Prefer official GHCR for Carto image
Aug 16, 2026
30cbded
Reuse local pinned images before pulling
Aug 16, 2026
e8039f6
Allow Apache to read Carto tile caches
Aug 16, 2026
903de9f
Support pinned Nominatim registry fallback
Aug 16, 2026
6669881
Expose elevation sync on Linux
Aug 16, 2026
e9f750a
Raise Valhalla initial build memory
Aug 16, 2026
4cd8117
Support digest-only Nominatim image loads
Aug 16, 2026
71b617d
Catalog digest-only image recovery test
Aug 16, 2026
61615ad
Normalize Kiwix asset permissions on Linux
Aug 16, 2026
a16c429
Refresh deployed world overview manifest
Aug 16, 2026
78d4fd2
Normalize public bind mounts on Linux
Aug 16, 2026
ee0f00f
Run Carto Apache as renderer
Aug 16, 2026
c11d13d
Flatten Linux permission validation inputs
Aug 16, 2026
87e6ba2
Invalidate map pack cache on boundary changes
Aug 16, 2026
2273860
Organize cross-platform deployment documentation
Aug 31, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .gitattributes
Original file line number Diff line number Diff line change
@@ -1 +1,3 @@
*.sh text eol=lf
scripts/linux/curl.exe text eol=lf
services/systemd/* text eol=lf
24 changes: 20 additions & 4 deletions .github/workflows/test-suite.yml
Original file line number Diff line number Diff line change
Expand Up @@ -15,12 +15,28 @@ concurrency:

jobs:
repository-contracts:
name: Static repository contracts
runs-on: windows-latest
name: Static repository contracts (${{ matrix.os }})
runs-on: ${{ matrix.os }}
timeout-minutes: 10
strategy:
fail-fast: false
matrix:
os:
- windows-latest
- ubuntu-latest
steps:
- name: Check out repository
uses: actions/checkout@v7
- name: Run static test profile
shell: powershell
run: .\tests\run-suite.ps1 -Profile static
shell: pwsh
run: ./tests/run-suite.ps1 -Profile static
- name: Validate Linux entry points
if: runner.os == 'Linux'
shell: bash
run: |
for script in terrasys.sh scripts/install-linux-service.sh scripts/linux/curl.exe; do
bash -n "$script"
done
test -x terrasys.sh
test -x scripts/install-linux-service.sh
test -x scripts/linux/curl.exe
13 changes: 11 additions & 2 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,25 +2,33 @@

> English | [简体中文](CHANGELOG.zh-CN.md)
>
> History updated: `2026-08-07T13:09:44+08:00`
> History updated: `2026-08-31`

This changelog documents user-visible development milestones. It follows the human-readable grouping used by mature open-source projects and records ISO 8601 timestamps.

The repository had no tags or GitHub Releases before this file was introduced. The `M0.x` labels below are retrospective documentation labels. Commit hashes and timestamps are the source of truth; no historical releases are implied.

## Unreleased — resilient global browsing and regional expansion

- **Development snapshot:** `2026-08-07T13:09:44+08:00`
- **Development snapshot:** `2026-08-31`

### Added

- A unified bilingual deployment index with maintained Windows workstation and Linux server procedures, equivalent command mapping, safe upgrade order, network-boundary guidance, and host-specific backup scheduling.
- A bilingual project-structure guide separating stable entry points, version-controlled implementation, private host state, generated products, and recovery material.
- A complete Ubuntu server deployment path with a unified `terrasys.sh` command surface, Docker Engine/PowerShell 7 compatibility, systemd boot service, daily backup timer, configurable trusted-interface binding, and Linux CI coverage.
- An independent additional-information-layer module with normalized GeoJSON adapters, TTL caching, viewport cropping, and graceful degradation for USGS, NASA EONET, GDACS, Open-Meteo, ADSB.lol, NOAA NDBC, and the Norwegian open AIS stream.
- The former contour shortcut is now the P0/P1 information-layer entry; contours remain in the ordinary layer panel, with ADS-B/AIS refresh, provenance popups, and a quick jump to the AIS coverage area.
- A decision catalog records keyed-source application paths and published call limits while the runtime remains `no-key-only`.
- A standalone additional-information-source console now manages no-key enablement, refresh cadence, bounded connection tests, runtime health, coverage, and licenses, with settings persisted in the maintenance directory.
- A Natural Earth 110m/50m/10m vector PMTiles overview with land, water, boundaries, places, roads, railways, and rivers through zoom 7.
- Region-aware elevation synchronization backed by AWS Open Data Terrain Tiles, with retained HGT manifests and maintenance progress.
- ETag-backed map-pack and boundary responses, deferred catalog loading, cache warming, and a browser performance smoke test.
- A machine-readable test-case catalog and unified `static`, `browser`, `full`, and `recovery` profiles, with static contracts running automatically on GitHub pull requests.

### Fixed

- Treat an unconfigured Windows WSL `bash.exe` shim as unavailable during static checks instead of failing the entire Windows validation; usable Bash installations and Linux CI still parse every shell entry point.
- Kept the verified existing geocoder and routing scope usable while newly installed regions are still being indexed.
- Kept a regional PMTiles vector fallback visible until OSM Carto has caught up, preventing blank maps after installing a region.
- Deduplicated overlapping regional OSM objects before Carto, search, and routing builds.
Expand All @@ -32,6 +40,7 @@ The repository had no tags or GitHub Releases before this file was introduced. T

### Improved

- Grouped platform deployment documentation under `docs/deployment/` while retaining every root Windows `.cmd` launcher and `terrasys.sh` as stable compatibility entry points; documentation contracts now recurse into organized subdirectories.
- Accelerated terrain tile generation by loading each required HGT grid once per row group instead of once per pixel.
- Minified the generated global catalog without changing its 547 browse regions or 554 buildable datasets, and enabled nginx compression and explicit config revalidation.
- Expanded world-map regression coverage for Carto lag, vector fallback, global movement, country selection, and persistent overview rendering.
Expand Down
13 changes: 11 additions & 2 deletions CHANGELOG.zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,25 +2,33 @@

> [English](CHANGELOG.md) | 简体中文
>
> 历史更新时间:`2026-08-07T13:09:44+08:00`
> 历史更新时间:`2026-08-31`

本文件记录面向使用者的开发里程碑,采用成熟开源项目常见的分类式结构,并保留 ISO 8601 时间戳。

在本文件加入前,仓库没有 Git tag 或 GitHub Release。下列 `M0.x` 是追溯整理的文档编号;提交哈希与时间戳才是权威依据,这些编号不代表补建了历史 Release。

## 未发布 — 稳定全球浏览与区域扩展

- **开发快照:** `2026-08-07T13:09:44+08:00`
- **开发快照:** `2026-08-31`

### 新增

- 统一双语部署索引与持续维护的 Windows 工作站、Linux 服务器流程,包含等价命令对照、安全升级顺序、网络边界和平台专用备份计划。
- 双语项目结构指南,明确稳定入口、版本控制实现、私人本机状态、生成产品与恢复材料的边界。
- 完整 Ubuntu 服务器部署路径:统一 `terrasys.sh` 命令面、Docker Engine/PowerShell 7 兼容、systemd 开机服务、每日备份计时器、可信接口监听配置和 Linux CI 覆盖。
- 独立“附加信息图层”模块,以统一 GeoJSON 适配器、TTL 缓存、视口裁剪和安全降级接入 USGS、NASA EONET、GDACS、Open-Meteo、ADSB.lol、NOAA NDBC 与挪威沿岸开放 AIS。
- 将原等高线快捷按钮改为 P0/P1 信息图层入口;等高线仍保留在普通图层面板,并新增 ADS-B/AIS 实时刷新、来源弹窗和 AIS 覆盖区快速定位。
- 新增需要密钥的数据源与公开调用限制决策目录,本轮运行代码保持 `no-key-only`。
- 新增独立“附加信息源”管理页,集中提供免密源启停、刷新频率、连接测试、运行状态、覆盖与许可说明,并将配置持久化到维护目录。
- Natural Earth 110m/50m/10m 多级矢量 PMTiles,在 z0-7 提供陆地、水体、边界、地名、道路、铁路和河流。
- 基于 AWS Open Data Terrain Tiles 的按区域高程同步,长期保留 HGT 清单并显示维护进度。
- 带 ETag 的地图包/边界缓存响应、延迟目录加载、缓存预热和浏览器性能冒烟测试。
- 机器可读测试用例目录和 `static`、`browser`、`full`、`recovery` 四层统一测试入口,并在 GitHub PR 上自动执行静态契约。

### 修复

- Windows 静态检查遇到未配置发行版的 WSL `bash.exe` 占位程序时按不可用跳过,不再让整套 Windows 校验失败;可用 Bash 和 Linux CI 仍会解析全部 Shell 入口。
- 新安装区域仍在构建索引时,已验证的原搜索与路线范围继续可用。
- OSM Carto 尚未同步新区域时保留区域 PMTiles 矢量回退,避免安装后出现空白地图。
- 在 Carto、搜索和路线构建前去除区域重叠边界产生的重复 OSM 对象。
Expand All @@ -32,6 +40,7 @@

### 改进

- 把平台部署文档统一归档到 `docs/deployment/`,同时保留根目录全部 Windows `.cmd` 启动器和 `terrasys.sh` 作为稳定兼容入口;文档契约现会递归检查整理后的子目录。
- 每个行组只加载一次所需 HGT,替代逐像素重复加载,加快地形瓦片生成。
- 在不改变 547 个浏览区域和 554 个可构建数据集的前提下压缩全球目录,并启用 nginx 压缩和配置重新验证。
- 扩展全球地图回归测试,覆盖 Carto 延迟、矢量回退、全球平移、国家选择和持续概览显示。
Expand Down
20 changes: 18 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@

> English | [简体中文](README.zh-CN.md)
>
> Documentation snapshot: `2026-08-09` · Product identity: `TerraSys`
> Documentation snapshot: `2026-08-31` · Product identity: `TerraSys`

TerraSys is a local-first personal geographic information system for owning, exploring, and recovering offline map data. It combines an OpenStreetMap-style local renderer, portable regional vector maps, private places and tracks, address search, routing, terrain, weather, nautical references, a Chinese encyclopedia and travel guide, resource lifecycle management, backups, and disconnected recovery.

Expand Down Expand Up @@ -71,15 +71,27 @@ The active Compose profile contains eight services: `web`, `api`, `postgis`, `ma

## Start and verify

Windows:

```powershell
D:\TerraSys\start-terrasys.cmd
D:\TerraSys\health-check.cmd
D:\TerraSys\smoke-test.cmd
```

Linux:

```bash
./terrasys.sh start
./terrasys.sh health
./terrasys.sh smoke
```

For a fresh host, start with the [deployment guide](docs/deployment/README.md), then follow either the [Windows workstation](docs/deployment/WINDOWS.md) or [Linux server](docs/deployment/LINUX.md) procedure. The two methods share data formats and lifecycle scripts but use platform-specific startup and backup scheduling.

Ongoing development uses the layered [test suite](tests/README.md): `static` for every commit, `browser` for map and resource UI coverage, `full` for API and personal-data lifecycles, and `recovery` for the isolated disconnected recovery drill.

`start-terrasys.cmd` creates local secrets when required, starts Docker Desktop, applies ordered PostGIS migrations, builds the API image, starts the core stack, enables prepared advanced services, and starts the allowlisted maintenance worker.
The start command creates local secrets when required, ensures Docker is available, applies ordered PostGIS migrations, builds the API image, starts the core stack, enables prepared advanced services, and starts the allowlisted maintenance worker.

Prepare or rebuild advanced offline capabilities:

Expand Down Expand Up @@ -111,6 +123,8 @@ The resource console exposes Available, Local, and Updates views. Regular update

## Storage and ownership

The complete tracked/generated/private-data boundary is documented in [Project structure](docs/PROJECT_STRUCTURE.md).

| Path | Role |
| --- | --- |
| `web/` | Browser application, local assets, and resource console |
Expand All @@ -134,6 +148,8 @@ The [documentation index](docs/README.md) provides every guide in English and Si

| Subject | English | 简体中文 |
| --- | --- | --- |
| Deployment | [English](docs/deployment/README.md) | [中文](docs/deployment/README.zh-CN.md) |
| Project structure | [English](docs/PROJECT_STRUCTURE.md) | [中文](docs/PROJECT_STRUCTURE.zh-CN.md) |
| Architecture | [English](docs/ARCHITECTURE.md) | [中文](docs/ARCHITECTURE.zh-CN.md) |
| Configuration | [English](docs/CONFIGURATION.md) | [中文](docs/CONFIGURATION.zh-CN.md) |
| Data pipeline | [English](docs/DATA_PIPELINE.md) | [中文](docs/DATA_PIPELINE.zh-CN.md) |
Expand Down
20 changes: 18 additions & 2 deletions README.zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@

> [English](README.md) | 简体中文
>
> 文档快照:`2026-08-09` · 产品标识:`TerraSys`
> 文档快照:`2026-08-31` · 产品标识:`TerraSys`

TerraSys 是一套本地优先的个人地理信息系统,用于拥有、浏览和恢复离线地图数据。系统组合了 OpenStreetMap 风格本地渲染、可移植区域矢量地图、个人点位与轨迹、地址搜索、路线规划、地形、天气、航海参考、中文百科与旅行指南、资源生命周期管理、备份和断网恢复。

Expand Down Expand Up @@ -71,15 +71,27 @@ flowchart LR

## 启动与验证

Windows:

```powershell
D:\TerraSys\start-terrasys.cmd
D:\TerraSys\health-check.cmd
D:\TerraSys\smoke-test.cmd
```

Linux:

```bash
./terrasys.sh start
./terrasys.sh health
./terrasys.sh smoke
```

新主机先阅读[部署指南](docs/deployment/README.zh-CN.md),再按 [Windows 工作站](docs/deployment/WINDOWS.zh-CN.md)或 [Linux 服务器](docs/deployment/LINUX.zh-CN.md)流程操作。两种方式共享数据格式和生命周期脚本,但使用各自平台的启动与备份计划。

后续开发使用分层的[测试用例集](tests/README.zh-CN.md):`static` 适合每次提交,`browser` 覆盖地图与资源界面,`full` 增加 API 和个人数据生命周期,`recovery` 增加隔离断网恢复演练。

`start-terrasys.cmd` 会在需要时创建本地密钥、启动 Docker Desktop、应用有序 PostGIS 迁移、构建 API 镜像、启动核心服务和已准备的高级服务,并启动白名单维护任务工作器。
启动命令会在需要时创建本地密钥、确认 Docker 可用、应用有序 PostGIS 迁移、构建 API 镜像、启动核心服务和已准备的高级服务,并启动白名单维护任务工作器。

准备或重建高级离线能力:

Expand Down Expand Up @@ -111,6 +123,8 @@ D:\TerraSys\stop-terrasys.cmd

## 存储与所有权

版本控制、生成产品与私人数据的完整边界见[项目结构](docs/PROJECT_STRUCTURE.zh-CN.md)。

| 路径 | 作用 |
| --- | --- |
| `web/` | 浏览器应用、本地资源和资源管理页面 |
Expand All @@ -134,6 +148,8 @@ D:\TerraSys\stop-terrasys.cmd

| 主题 | English | 简体中文 |
| --- | --- | --- |
| 部署 | [English](docs/deployment/README.md) | [中文](docs/deployment/README.zh-CN.md) |
| 项目结构 | [English](docs/PROJECT_STRUCTURE.md) | [中文](docs/PROJECT_STRUCTURE.zh-CN.md) |
| 架构 | [English](docs/ARCHITECTURE.md) | [中文](docs/ARCHITECTURE.zh-CN.md) |
| 配置 | [English](docs/CONFIGURATION.md) | [中文](docs/CONFIGURATION.zh-CN.md) |
| 数据流水线 | [English](docs/DATA_PIPELINE.md) | [中文](docs/DATA_PIPELINE.zh-CN.md) |
Expand Down
57 changes: 57 additions & 0 deletions docs/ADDITIONAL_INFORMATION_LAYERS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
# Additional Information Layer Module

> English | [简体中文](ADDITIONAL_INFORMATION_LAYERS.zh-CN.md) · Updated 2026-08-10

Additional information layers are a plug-in boundary separate from base maps, personal data, and offline resource packs. The module owns short-lived provider differences, caching, refresh policy, attribution, and map styling so changing third-party APIs do not spread through the main map application.

## Boundaries

- `services/api/app/live_layers.py`: fixed upstream adapters, common GeoJSON normalization, TTL cache, viewport cropping, graceful degradation, and the open AIS TCP receiver/NMEA decoder.
- `services/api/app/main.py`: `/live/*` routes only validate bounded parameters and invoke the module.
- `web/src/live-layers.js`: catalog, MapLibre sources/layers, visibility, refresh scheduling, popups, and legend entries.
- `web/src/app.js`: creates one module instance, controls the popover, remounts after style reload, and reports map movement.
- `web/information-layers.html`: a standalone console, parallel to map-resource management, for enablement, refresh cadence, connection tests, runtime health, coverage, and license review. Keyed candidates remain a read-only comparison catalog.
- The former contour shortcut now opens this module; contours remain available in the ordinary layer panel.

## Adapter contract

Every adapter returns a GeoJSON `FeatureCollection`. Collection metadata includes `source`, `status`, `fetchedAt`, and `count`. Every feature includes `kind`, `title`, `subtitle`, `observedAt`, `sourceLabel`, `sourceUrl`, `license`, and `detail`. A provider failure returns an empty collection with `status: unavailable` rather than failing the main map.

## Current no-key catalog

| Endpoint | Layer | Cache/refresh baseline |
| --- | --- | --- |
| `/live/earthquakes` | USGS seven-day earthquakes | 60 seconds |
| `/live/wildfires` | NASA EONET wildfire events | 10 minutes |
| `/live/disasters` | GDACS flood/volcano/drought/wildfire notices | 5 minutes |
| `/live/air-quality` | Open-Meteo/CAMS viewport samples | 15 minutes |
| `/live/floods` | Open-Meteo/GloFAS discharge samples | 6 hours |
| `/live/aircraft` | ADSB.lol aircraft | 5–8 seconds |
| `/live/vessels` | Norwegian Coastal Administration open AIS | continuous receiver; 8-second browser refresh |
| `/live/ocean-buoys` | NOAA NDBC latest observations | 10 minutes |
| `/live/cyclones` | GDACS tropical cyclones | 5 minutes |

`/live/catalog` exposes the machine-readable directory with `keyPolicy: no-key-only`. `/live/status` reports recent in-process checks, while `PUT /live/settings/{layer_id}` persists only `enabled` and `refreshSeconds`. Settings default to `/data/maintenance/live-layer-settings.json` and can be relocated with `LIVE_LAYER_SETTINGS_PATH`.

## Standalone management console

- The map popover remains the quick-toggle surface and links to `/information-layers.html` for management.
- Enablement and refresh cadence are loaded when the map starts. Disabling a source does not remove its adapter or metadata.
- Per-source tests use bounded representative viewports and report object count, latency, and recent failure without changing the map viewport.
- Test-all runs at most three providers concurrently.
- Keyed candidates show authentication and published call limits but are never invoked and no credentials are collected.

## AIS configuration

The default no-registration stream is `153.44.253.27:5631`. The same receiver can point at user-owned AIS hardware with `AIS_TCP_ENABLED`, `AIS_TCP_HOST`, and `AIS_TCP_PORT`. `LIVE_LAYER_SETTINGS_PATH` controls the management-setting file. Positions expire after 15 minutes. Coverage and public-disclosure limits are those of the Norwegian open AIS service.

## Adding an adapter

1. Add a fixed-URL, cached adapter in `live_layers.py` and normalize the common fields.
2. Declare provenance, time, license, coverage, latency, and safety limits.
3. Add a bounded API route and catalog entry.
4. Add one `LIVE_LAYER_DEFINITIONS` entry and its style in `pointLayer()`; keep provider details out of `app.js`.
5. Add unit tests, a stable UI fixture, and source/license documentation.
6. For credentials, read server-side environment variables only and implement 429 backoff. Update the [keyed-source decision catalog](KEYED_DATA_SOURCES.md) and obtain a decision before enabling it.

All layers default off. Air quality and discharge use a 3×3 model sample, ADS-B is limited to a 250-nautical-mile query radius, and disaster/aviation/maritime overlays are situational references rather than safety systems.
Loading