diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 09a9718..523912a 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,12 +1,14 @@ # Contributing to the DFO docs -The DFO docs serve analysts who know the DAC and the CRS, write some Python or R, and need data they can trust in their own work. AI assistants read the same pages. This guide covers where pages live, how to write one, and how a source repository publishes its own pages into the corpus. +Issues and pull requests are welcome. The ONE Data team maintains this repository. Open an issue to discuss a large change, such as a new section or a new source repository, before you write it. Contributors follow the [Code of Conduct](https://github.com/DevelopmentFinanceObservatory/.github/blob/main/CODE_OF_CONDUCT.md). Report security issues through the "Report a vulnerability" button under the repository's Security tab, as the [security policy](https://github.com/DevelopmentFinanceObservatory/.github/blob/main/SECURITY.md) describes. + +The DFO docs serve analysts who know the DAC and the CRS, write some Python or R, and need data they can trust in their own work. AI assistants read the same pages. ## Where pages live The site publishes hub pages from Markdown files under `content/hub/` in this repository. Source pages live in each source repository's docs folder, and the build fetches them through a row in `sources.yaml`. -``` +```text dfo-docs/ ├── content/hub/ hub pages; a page's URL is its path here │ ├── index.md Home, published at /docs @@ -21,7 +23,7 @@ dfo-docs/ A hub page publishes at its path under `content/hub/` without the `.md` suffix, so `content/hub/data/glossary.md` publishes at `/docs/data/glossary`. Its place in the nav comes from `hub.yaml`, so a page can move in the nav without changing its URL. Only pages named in `hub.yaml` publish. -A source repository with several pages groups them by mode: `tutorials/`, `how-to/` and `about/`, as `tossd-reader` does. +A source repository with several pages groups them by mode in `tutorials/`, `how-to/` and `about/`, as `tossd-reader` does. ## Writing a page @@ -48,11 +50,11 @@ Use DFO values in examples, including `country/NGA`, CRS purpose 12220, finance End every page with a `## Next` section of one or two links, each with a sentence saying what the reader gets there. Leave out boilerplate headings such as Overview, Summary or Conclusion. -Use only three callouts: `!!! note`, `!!! tip` and `!!! warning`. Leave them untitled and indent the body four spaces. The build renders them as site alerts. A title becomes a bold lead sentence, so put the point in the body. Use callouts for optional material and keep required information in the page body. +Use only three callouts, `!!! note`, `!!! tip` and `!!! warning`. Leave them untitled and indent the body four spaces. The build renders them as site alerts. A title becomes a bold lead sentence, so put the point in the body. Use callouts for optional material and keep required information in the page body. ```markdown !!! warning - Studio links are always snapshots. +Studio links are always snapshots. ``` The build passes collapsible `???` blocks through unchanged, where they render as raw text. Leave them out. @@ -96,7 +98,7 @@ Only published pages are valid link targets. The link check fails for a page abs ## Publishing a source repository's docs -A source repository keeps its pages in a docs folder and declares them in a nav sidecar. The build looks for the sidecar in three places, in this order: `docs/mkdocs.yml`, a root `mkdocs.yml`, and a root `zensical.toml`. The docs root is the sidecar's own `docs_dir`, resolved from the folder the sidecar sits in. +A source repository keeps its pages in a docs folder and declares them in a nav sidecar. The build looks for the sidecar at `docs/mkdocs.yml`, then a root `mkdocs.yml`, then a root `zensical.toml`. The docs root is the sidecar's own `docs_dir`, resolved from the folder the sidecar sits in. Use a root `mkdocs.yml` when user-facing pages sit in a subfolder separate from a repository's internal docs. `apis` publishes `docs/guide/` this way: @@ -119,30 +121,40 @@ nav: Then register the repository in `sources.yaml`: ```yaml - - name: dfo-apis - repo: DevelopmentFinanceObservatory/apis - ref: dev - docs_dir: docs/guide - path: api - title: API +- name: dfo-apis + repo: DevelopmentFinanceObservatory/apis + ref: dev + docs_dir: docs/guide + path: api + title: API ``` -In `sources.yaml`, `docs_dir` sets which folder the build checks out. Match it to the sidecar's `docs_dir`. Set `path` to `extra.dfo.path`. The build fails if they differ. Add a `mount` entry in `hub.yaml` that names the source row. To register a repository before publication, set `enabled: false` with a `reason` and leave it out of `hub.yaml`. A mount for a disabled source fails the build. Every registered repository must be public. The README covers `exclude`, `kind: external`, `kind: skill`, and each build failure. +In `sources.yaml`, `docs_dir` sets which folder the build checks out. Match it to the sidecar's `docs_dir`. Set `path` to `extra.dfo.path`. The build fails if they differ. Add a `mount` entry in `hub.yaml` that names the source row. To register a repository before publication, set `enabled: false` with a `reason` and leave it out of `hub.yaml`. A mount for a disabled source fails the build. Every registered repository must be public. [docs/development.md](docs/development.md) covers `exclude`, `kind: external`, `kind: skill`, and each build failure. Before opening a pull request in the source repository, check its nav against its pages from a `docs` checkout. Then check links from the source repository's docs folder: -``` +```bash uv run python scripts/check_nav.py /path/to/source-repo lychee --offline /path/to/source-repo/docs/guide/ ``` ## Checking a change here -``` +```bash uv run ruff check . +uv run ruff format --check . +uv run ty check src/dfo_docs uv run pytest uv run python scripts/build.py --out /tmp/dfo-docs-build lychee --config lychee.toml /tmp/dfo-docs-build/corpus/ ``` -The build clones every enabled source over HTTPS without credentials. +The build clones every enabled source over HTTPS without credentials, so none of this needs ONE's cloud access. Run `uv run pre-commit install` once, and pre-commit then runs the formatters and hygiene checks on each commit. + +## Pull requests + +Branch from `dev` and open the pull request into `dev`. A push to `dev` rebuilds the corpus the site reads, and a pull request runs the build without publishing. A draft pull request runs no CI, so mark it ready when you want the checks. CI runs ruff, ty, the tests on Python 3.13 and 3.14, the build, and the link check. A page change needs a description in its front matter and working links. Code changes need tests and must pass these checks. + +Commit messages follow Conventional Commits, such as `docs(docsync): describe the first run without state`. + +Maintainer internals are in [docs/development.md](docs/development.md). diff --git a/README.md b/README.md index 1028863..da191ca 100644 --- a/README.md +++ b/README.md @@ -1,123 +1,46 @@ -# dfo-docs +# DFO docs -dfo-docs fetches documentation from the DFO's source repositories, normalises it, and publishes a markdown corpus with a manifest that the DFO site renders. Each source repository keeps its own `docs/` folder and owns what is in it. +This repository holds the documentation of the [Development Finance Observatory](https://github.com/DevelopmentFinanceObservatory) (DFO), an initiative of [ONE Data](https://data.one.org) built on [Google Data Commons](https://datacommons.org). It also holds the build that gathers pages from the DFO's other repositories into one corpus for the DFO website. -To write a page or publish a repository's docs, read [CONTRIBUTING.md](CONTRIBUTING.md). +The published docs are at . Public launch is expected in late 2026. -## Adding a repository +This repository is under active development. Interfaces, schemas, and data may change without notice. -Add a row to `sources.yaml` with `name`, `repo`, `ref`, `docs_dir`, and the `path` the documentation should publish at. A row with `kind: external` points at a package that publishes its own documentation elsewhere, and carries a `url` instead of a `docs_dir`. +## What's inside -A row with `kind: skill` publishes a repository of agent skills, such as `skills`. It needs no sidecar. The build walks `docs_dir` and publishes each directory that holds a `SKILL.md` as a skill page at `/`, with each file in its `references/` as a page at `//references/`. Directories whose names start with `_` stay out. The row must set `docs_dir`, `path`, and `title`, and it may not set `exclude`. It may set `internal_dir`, a second directory the build walks the same way. Skills under `docs_dir` are offered for install. Skills under `internal_dir` are published at the same `/` and are not offered. A directory name may appear under only one of the two. Until the repository has the `internal_dir` directory, the build warns and reads `docs_dir` alone. Its `hub.yaml` mount becomes one group titled by `title`. A `categories.yaml` in `docs_dir` lists skills under `sources` or under a domain in `domains`, and each category becomes a group inside the mount, titled in sentence case: Sources, and Domains with a group such as Sovereign debt inside it. Categories and the skills in them follow the file's order. Skills the file leaves out come first, in directory order, and without the file that is every skill. Each skill page opens with a header the build writes from the front matter: the skill's name and the first sentence of its `description`. For a skill under `docs_dir`, whatever its category, the build writes a zip to `corpus//.zip`, holding `/SKILL.md` and `/references/*.md` as committed, records the zip on the page's manifest entry as `bundle_path`, and adds a line to the header that links to the hub page on using skills in your own agent and to the zip. A skill under `internal_dir` gets no zip, no `bundle_path`, and no such line. - -If the repository needs to sit under a grouping heading, or nest alongside another package, add it to `hub.yaml`'s nav as a `mount` (or `external`) entry. A grouping heading such as `bblocks` or `ODA` lives only in `hub.yaml`, since no source repository declares it. - -Every source repository is public, and the build and the registry-health workflow clone and query them without credentials. A private repository cannot be registered. - -To register a repository without building it yet, set `enabled: false` and a `reason`. A disabled row still loads and is reported by name. Turning it on means enabling the row and adding its `hub.yaml` mount, since a mount whose source is disabled fails the build. - -To take most of a repository but hold some pages back, list glob patterns under `exclude`. They match the docs-root-relative path, so `reference/*` keeps a whole section out and `changelog.md` keeps one page out. Use it for pages that cannot render outside their own site, such as an mkdocstrings reference section. A pattern matching no nav entry fails the build, so a pattern left behind by an upstream rename is reported rather than quietly letting those pages in. - -## Running the build locally - -``` -uv sync -uv run python scripts/build.py +```text +content/hub/ hub pages, written here: Get started, Build, Data, Skills, About +roadmap/ the DFO roadmap, one YAML file per track +hub.yaml the site nav: hub pages, grouping headings, and source mounts +sources.yaml the registry of repositories whose docs the build fetches +docsync.yaml the sources published as Google Docs for editors who do not use GitHub +src/dfo_docs/ the build and the docsync tool +scripts/ build.py and check_nav.py +tests/ the test suite ``` -The build clones every enabled repository over HTTPS without credentials, normalises what it finds, and writes `corpus/` and `manifest.json`. - -`corpus/` and `manifest.json` are generated output that lives on the `corpus` branch. A push to `dev` runs the same build in CI and commits the result there, and that branch is what the DFO site pins and reads. A pull request into `dev` or `main` runs the build and publishes nothing. - -## Editing in Google Docs +Every other page lives in the `docs/` folder of its own repository, such as [apis](https://github.com/DevelopmentFinanceObservatory/apis), [ai](https://github.com/DevelopmentFinanceObservatory/ai), and [skills](https://github.com/DevelopmentFinanceObservatory/skills), or in a package repository under [ONEcampaign](https://github.com/ONEcampaign), and `sources.yaml` points the build at it. To change one of those pages, open a pull request in the repository that holds it. The [org profile](https://github.com/DevelopmentFinanceObservatory) links every DFO repository. -`dfo-docsync` publishes the pages of the sources listed in `docsync.yaml` as Google Docs in the "DFO docs" folder of the ONE Data shared drive, so editors who do not use GitHub can revise them. It tracks the pages in a Sheet and turns the edits made in the Docs into pull requests, one for each source repository. The script splices the words an editor changed into the source file one word at a time, so untouched text keeps its exact form, and keeps the structure, code blocks, and tabs as they are. Editors read the guide in the Sheet's "How to edit" tab, which `init` writes from `src/dfo_docs/docsync/how_to_edit.md`. +## Getting started -``` -uv run dfo-docsync init -uv run dfo-docsync push -uv run dfo-docsync pull -``` - -- `init` creates the tracking Sheet in the folder. Run it once. -- `push` creates a Doc for each page, or replaces it with the page's current source, and updates the Sheet. It refuses to run when the state store is empty but Docs exist under the folder (`StatelessDocsError`). Restore the state, or pass `--adopt` to adopt the Docs that match their source. -- `pull` reads every Doc whose Status is "Ready for review" and opens a new pull request for each source repository, on its own branch `docsync/edits-`. docsync never pushes to that branch or edits the pull request again, so reviewers may edit it or commit to it. A page is in at most one open pull request. `pull` skips a page while its pull request is open, and fails a Ready page that an open docsync pull request carries but its state does not name, with the message "merge or close that PR, or revert the file on its branch, before sending this page". Edits made in the Doc while the pull request is open go in the next one. -- Both `push` and `pull` first settle each page's pull request. A merged pull request settles once the merge is in the branch the page's source is read from, and until then the page waits. The page then shows "Merged", or "Editing" when the Doc holds later edits, and the next `push` refreshes the Doc from GitHub. A pull request that is closed, or merged without the page, returns the page to "Editing" with the reason in the "Not applied" column. A page set to "Discard my edits" stays on it and is discarded. A "Ready for review" page whose Doc holds nothing new to send becomes "Merged" when its pull request has just merged and "Editing" otherwise, with "Nothing to send" in "Not applied". -- To drop one page from a multi-page pull request, revert that file on the pull request branch before merging. The page returns to "Editing". -- "Discard my edits" is a Status an editor sets. The next `push` resets the page's Doc from GitHub, even with open comments, and sets the page to "Editing". It refuses while the Doc has open suggestions, and waits while the page's pull request is open. When that pull request merged into a branch the page is not read from, the discard goes ahead. The page's state records the discard until a `push` reads another Status, so when the Sheet write after a discard fails, the next `push` only sets the page to "Editing" and keeps what the editor typed since. -- A Doc that was trashed or deleted is recreated on the next `push`, unless the page's pull request is open. - -`--page ` limits `push` or `pull` to one page and can be repeated. A page id is the manifest's, such as `hub/build/share`. `pull --dry-run ` writes the edited files and pull request bodies under `` and changes nothing in git, the Sheet, or the state. `docsync.yaml` sets `pr_author`, the author `gh` lists open docsync pull requests by; it is `app/dfo-docsync` for the pull requests CI opens. For a local run that opens pull requests under your own login, pass `--pr-author @me` to `push` and `pull`. - -Hub pages (`hub/...`) run only from a checkout of this repository's `dev` that is at `origin/dev`. - -A maintainer runs a sync from a `dev` checkout at `origin/dev`, signed in to Google with a one.org account. `.github/workflows/docsync.yml` (one command on dispatch, with no schedule) is not used: a Markdown upload made by its service account loses code blocks, and every page with one then fails to align. Its job runs only on `dev`, in the `docsync` environment, which is where the `dfo-docsync` App key and the Google workload identity settings live (see the workflow's header). The token lasts about an hour. Without it, the command uses Application Default Credentials. - -``` -git switch dev && git pull -gcloud auth login --enable-gdrive-access -export DOCSYNC_ACCESS_TOKEN=$(gcloud auth print-access-token) -uv run dfo-docsync pull --pr-author @me -uv run dfo-docsync push --pr-author @me -``` +Everything here runs without access to ONE's Google Cloud project, because the build clones public repositories anonymously. You need [uv](https://docs.astral.sh/uv/) and Python 3.13. -`pull` opens a pull request for each source repository with pages set to Ready for review. `push` refreshes each Doc whose page changed in its source repository, the pages of merged docsync pull requests included, and adds a Doc for each new page. Run the sequence again after docsync pull requests merge. A merge to this repository's `dev` rebuilds the corpus the site reads at once. A merge in another source repository reaches it at the 06:00 UTC build, or sooner with `gh workflow run build.yml --repo DevelopmentFinanceObservatory/docs --ref dev`. - -When the `docsync-state` branch is empty but Docs already exist under the folder, `push` stops with `StatelessDocsError`. Run it once as `uv run dfo-docsync push --adopt --pr-author @me` from a `dev` checkout at `origin/dev`. `--adopt` takes over each Doc that carries a page id and whose settled export aligns with the source, and records the current `dev` commit as its baseline, so it needs no earlier commit. A Doc that differs from the source holds an editor's unsent work, so its page fails and the Doc stays. Resolve those by hand, then rerun `push --adopt`. Later runs need no flag. - -Each page's state (Doc, source commit, baseline export, pull request) is a `state/.json` file on the orphan branch `docsync-state`, with one commit for each run. These guards hold a page back, and the run reports the reason and carries on: - -- `push` leaves a Doc alone when it has edits that have not reached GitHub ("edited in Docs; set Status to Ready for review to send"), open comments, or open suggestions, because replacing a Doc discards or detaches them. "Discard my edits" overrides the first two. It checks the Doc's revision again just before replacing it and holds the Doc if the revision changed. -- `pull` skips a Doc with open suggestions. A skipped or failed page does not stop the other pages of its repository from going in the pull request. -- `pull` leaves out an edit the source cannot take, such as a changed code block or code block language, a new table row, a folded description, or a deleted label line, and lists it in the pull request body. When that leaves nothing to send for the page (the edits are all reported, or already in GitHub), the reasons go to the Sheet's "Not applied" column, the page returns to "Editing", and the next `push` refreshes the Doc from GitHub. The script never writes the Notes column. -- A line break an editor adds inside a paragraph starts a new paragraph in the source. A new list item goes after the last block of the item above it, its note included, at that item's marker, and a new list opens after a top-level paragraph, so adding items and links to a "Next" list applies. A bullet or number added to or removed from a paragraph applies only its text, and the report says so. -- Author notes follow the Doc. An Author note whose label an editor removed or changed becomes page text and so appears on the site. A paragraph that gains the label becomes an Author note and leaves the site. The pull request reports each change ("an Author note became page text; check it belongs on the site", "a paragraph became an Author note and leaves the site"), and the reviewer decides. When a note or the Description is changed, added or deleted and page text next to it also changes, with no unchanged paragraph between them, the edit applies and the pull request flags it once ("an Author note and the page text beside it changed together; check no text moved between them", and the same with "the Description"), unless one of the two flags above already covers it. A report of an applied edit, an anchor change or an Author note flag, appears in the pull request that carries it and not in the next one. A report of an edit left out appears each time. An Author note emptied down to its label is deleted, and a new paragraph that starts with the label becomes a note. -- A label belongs at the start of its own Doc paragraph. A label typed mid-paragraph is refused. A note joined onto the paragraph before or after it, or Enter pressed in the middle of a note, is refused, or applied with that flag when the page then matches the Doc. A refused region is left unchanged, and the report quotes the editor's text of it. Text crosses a note's edge when a note's words join another paragraph or split off it; a paragraph typed with Enter next to a note is a new paragraph and applies. A paragraph typed with Enter next to a note in a list item is refused and held when the source gives it no list item to join. A refusal also covers a label that is unbolded, a Description or Tab label removed or changed with its text kept, a Note, Tip or Warning label deleted together with a change to its text, a tab line deleted together with a change to the text beside it, a note added or labelled inside a plain `>` quote, a new note that holds `-->`, a heading or list item given the note label, an added paragraph the source has no place for, and an edit beside repeated text where docsync cannot tell which copy changed. Changing Note to Tip or Warning, deleting a callout's label with its text unchanged, and deleting a whole tab line or Description line are reported as not applied and not refused, so the page records its export as applied and the next `push` refreshes the Doc. A refusal is reported as the message, then the Doc's text in a fence, byte for byte. A note inside a list item whose label is removed becomes text in that item, flagged. A line typed above the Description is page text and the Description stays front matter. -- A pull that sends nothing for a page and has a refusal leaves the Doc alone. The page returns to "Editing", the Not applied column holds each refusal's message and quoted Doc text and says to rework the edit or set "Discard my edits", and the page's state records no new applied export, so `push` finds the Doc edited and holds it. Every other report kind records the export as applied, and the next `push` refreshes the Doc from GitHub. When the pull sends a pull request for the page, the refusals go in its body, and the refused text leaves the Doc when that pull request merges and `push` refreshes it. The Not applied cell holds at most 45,000 characters and the pull request body 60,000. Refusal quotes are cut to fit. A cut quote ends with the line "(Shortened here. The full text was in the Doc when this pull ran.)". -- `pull` checks that the merged file reads in the source dialect, as `push` reads it. When it does not, nothing is sent for the page, the run exits 1, and the page is held like a refusal: the Not applied column says "the edits would leave a file docsync cannot read: ; nothing sent" and the Doc keeps the editor's text. -- When upstream and an editor changed different words of one paragraph, both changes stay. When they changed the same words, the pull request applies the editor's text and shows the upstream text beside it. - -The exit code is 1 when a page fails. It is 2 when the run stops: no credentials, folder, or Sheet; a git, `gh`, or Google error outside a page; a Page id listed twice, or a missing Page id or Status column, in the Sheet; or a state save that fails. A failed save writes the changed page states to `docsync-unsaved-states-.json` in the working directory. In CI the workflow uploads it as the `docsync-unsaved-states` artifact. Run `uv run dfo-docsync restore ` to save those states over the stored ones before the next run. `--state-dir ` restores into a local state directory. `restore` refuses when the state has moved since the failed run, because a later run may have saved newer states. Pass `--page ` (repeatable) to restore chosen pages, or `--force` to overwrite every page in the file. - -To add a source, append its name to `sources` in `docsync.yaml`. It must be an enabled `docs` or `skill` row in `sources.yaml` that `hub.yaml` mounts. For another repository, install the `dfo-docsync` GitHub App on it and add its name to the `repositories:` input of the "Mint a token for dfo-docsync" step in `.github/workflows/docsync.yml`. Then run `push`. [CONTRIBUTING.md](CONTRIBUTING.md) covers writing the pages. - -To try a change away from the real folder, make a scratch folder in the shared drive, share it with your account, and point each command at it. A Doc pushed under one folder belongs to that folder alone. - -``` -uv run dfo-docsync init --folder -uv run dfo-docsync push --folder --state-dir ./scratch-state -uv run dfo-docsync pull --folder --state-dir ./scratch-state --dry-run ./scratch-out -``` - -Without `--state-dir`, `push` and `pull` read and commit to the `docsync-state` branch on `origin`, so pass it whenever you experiment. - -## Working on this repo - -``` -uv run ruff check . -uv run ruff format . -uv run ty check src/dfo_docs -uv run pytest +```bash +uv sync +uv run python scripts/build.py --out /tmp/dfo-docs-build ``` -Run `uv run pre-commit install` once. Pre-commit then blocks a commit that fails these checks. - -## When the build fails - -The build checks every enabled repository before it exits, so the log names every failure in one run. - -**A source repository cannot be cloned.** The repository was renamed, made private, or deleted. Fix the `repo` field in `sources.yaml`, or disable the row. - -**An empty docs directory.** The source repository renamed its `docs_dir`. Update `docs_dir` in `sources.yaml` to match. +The build writes `corpus/` and `manifest.json` under the output directory. The Google Docs sync (`dfo-docsync`) needs ONE credentials and is for maintainers. -**A leaked `::: ` directive.** A repository is publishing mkdocstrings-generated reference through the normal build path. Exclude the pages that carry it, or move the whole repository to the self-published tier with `kind: external`. +## Documentation -**A sidecar declaring no nav.** The source would publish nothing. In a `zensical.toml` this is usually a `nav` array written after a `[project.]` header, which TOML scopes into that table instead of into `[project]`. Move the array above every sub-table header. +- [CONTRIBUTING.md](CONTRIBUTING.md) explains how to edit a hub page, page modes, style, the link rules, and how a repository publishes its docs. +- [docs/development.md](docs/development.md) holds maintainer notes: the source registry, build failures, and the Google Docs sync. -**A skill directory without a `SKILL.md`, or with front matter that does not parse.** The build names the directory or file. Add the `SKILL.md`, fix its front matter so it carries a `name` and a `description`, or move the directory out of the row's `docs_dir` and `internal_dir` if it is not a skill. +## Contributing -**A skill directory of the same name under both `docs_dir` and `internal_dir`.** The build names both. A skill's directory name is its page address, so keep one. +Issues and pull requests are welcome. See [CONTRIBUTING.md](CONTRIBUTING.md). -**A `categories.yaml` that does not parse, or lists a skill twice.** The build names the file. Give `sources` a list of skill directories and `domains` a mapping from each domain to such a list, and list each skill once. +## Licence -**An exclude pattern matching nothing.** A page an `exclude` row named was renamed or removed upstream. Update the pattern in `sources.yaml`, or drop it if the page is gone for good. +MIT. See [LICENSE](LICENSE). diff --git a/docs/development.md b/docs/development.md new file mode 100644 index 0000000..b65c54e --- /dev/null +++ b/docs/development.md @@ -0,0 +1,128 @@ +# Maintainer notes + +Detail for the ONE Data maintainers of this repository, covering the source registry, the build and its branches, the Google Docs sync, and build failures. Contributors writing pages need only [CONTRIBUTING.md](../CONTRIBUTING.md). + +## Adding a repository + +Add a row to `sources.yaml` with `name`, `repo`, `ref`, `docs_dir`, and the `path` the documentation should publish at. A row with `kind: external` points at a package that publishes its own documentation elsewhere, and carries a `url` instead of a `docs_dir`. + +A row with `kind: skill` publishes a repository of agent skills, such as `skills`. It needs no sidecar. The build walks `docs_dir` and publishes each directory that holds a `SKILL.md` as a skill page at `/`, with each file in its `references/` as a page at `//references/`. Directories whose names start with `_` stay out. The row must set `docs_dir`, `path`, and `title`, and it may not set `exclude`. It may set `internal_dir`, a second directory the build walks the same way. Skills under `docs_dir` are offered for install. Skills under `internal_dir` are published at the same `/` and are not offered. A directory name may appear under only one of the two. Until the repository has the `internal_dir` directory, the build warns and reads `docs_dir` alone. Its `hub.yaml` mount becomes one group titled by `title`. A `categories.yaml` in `docs_dir` lists skills under `sources` or under a domain in `domains`, and each category becomes a group inside the mount, titled in sentence case: Sources, and Domains with a group such as Sovereign debt inside it. Categories and the skills in them follow the file's order. Skills the file leaves out come first, in directory order, and without the file that is every skill. Each skill page opens with a header the build writes from the front matter: the skill's name and the first sentence of its `description`. For a skill under `docs_dir`, whatever its category, the build writes a zip to `corpus//.zip`, holding `/SKILL.md` and `/references/*.md` as committed, records the zip on the page's manifest entry as `bundle_path`, and adds a line to the header that links to the hub page on using skills in your own agent and to the zip. A skill under `internal_dir` gets no zip, no `bundle_path`, and no such line. + +If the repository needs to sit under a grouping heading, or nest alongside another package, add it to `hub.yaml`'s nav as a `mount` (or `external`) entry. A grouping heading such as `bblocks` or `ODA` lives only in `hub.yaml`, since no source repository declares it. + +Every source repository is public, and the build and the registry-health workflow clone and query them without credentials. A private repository cannot be registered. + +To register a repository without building it yet, set `enabled: false` and a `reason`. A disabled row still loads and is reported by name. Turning it on means enabling the row and adding its `hub.yaml` mount, since a mount whose source is disabled fails the build. + +To take most of a repository but hold some pages back, list glob patterns under `exclude`. They match the docs-root-relative path, so `reference/*` keeps a whole section out and `changelog.md` keeps one page out. Use it for pages that cannot render outside their own site, such as an mkdocstrings reference section. A pattern matching no nav entry fails the build, so a pattern left behind by an upstream rename is reported. + +## Building and publishing + +```bash +uv sync +uv run python scripts/build.py +``` + +The build clones every enabled repository over HTTPS without credentials, normalises what it finds, and writes `corpus/` and `manifest.json`. + +`corpus/` and `manifest.json` are generated output that lives on the `corpus` branch. A push to `dev` runs the same build in CI and commits the result there, and that branch is what the DFO site pins and reads. A pull request into `dev` or `main` runs the build and publishes nothing. + +The build also runs in CI every day at 06:00 UTC. + +## Editing in Google Docs + +`dfo-docsync` publishes the pages of the sources listed in `docsync.yaml` as Google Docs in the "DFO docs" folder of the ONE Data shared drive, so editors who do not use GitHub can revise them. It tracks the pages in a Sheet and turns the edits made in the Docs into pull requests, one for each source repository. The script splices the words an editor changed into the source file one word at a time, so untouched text keeps its exact form, and keeps the structure, code blocks, and tabs as they are. Editors read the guide in the Sheet's "How to edit" tab, which `init` writes from `src/dfo_docs/docsync/how_to_edit.md`. + +```bash +uv run dfo-docsync init +uv run dfo-docsync push +uv run dfo-docsync pull +``` + +- `init` creates the tracking Sheet in the folder. Run it once. +- `push` creates a Doc for each page, or replaces it with the page's current source, and updates the Sheet. It refuses to run when the state store is empty but Docs exist under the folder (`StatelessDocsError`). Restore the state, or pass `--adopt` to adopt the Docs that match their source. +- `pull` reads every Doc whose Status is "Ready for review" and opens a new pull request for each source repository, on its own branch `docsync/edits-`. docsync never pushes to that branch or edits the pull request again, so reviewers may edit it or commit to it. A page is in at most one open pull request. `pull` skips a page while its pull request is open, and fails a Ready page that an open docsync pull request carries but its state does not name, with the message "merge or close that PR, or revert the file on its branch, before sending this page". Edits made in the Doc while the pull request is open go in the next one. +- Both `push` and `pull` first settle each page's pull request. A merged pull request settles once the merge is in the branch the page's source is read from, and until then the page waits. The page then shows "Merged", or "Editing" when the Doc holds later edits, and the next `push` refreshes the Doc from GitHub. A pull request that is closed, or merged without the page, returns the page to "Editing" with the reason in the "Not applied" column. A page set to "Discard my edits" stays on it and is discarded. A "Ready for review" page whose Doc holds nothing new to send becomes "Merged" when its pull request has just merged and "Editing" otherwise, with "Nothing to send" in "Not applied". +- To drop one page from a multi-page pull request, revert that file on the pull request branch before merging. The page returns to "Editing". +- "Discard my edits" is a Status an editor sets. The next `push` resets the page's Doc from GitHub, even with open comments, and sets the page to "Editing". It refuses while the Doc has open suggestions, and waits while the page's pull request is open. When that pull request merged into a branch the page is not read from, the discard goes ahead. The page's state records the discard until a `push` reads another Status, so when the Sheet write after a discard fails, the next `push` only sets the page to "Editing" and keeps what the editor typed since. +- A Doc that was trashed or deleted is recreated on the next `push`, unless the page's pull request is open. + +`--page ` limits `push` or `pull` to one page and can be repeated. A page id is the manifest's, such as `hub/build/share`. `pull --dry-run ` writes the edited files and pull request bodies under `` and changes nothing in git, the Sheet, or the state. `docsync.yaml` sets `pr_author`, the author `gh` lists open docsync pull requests by; it is `app/dfo-docsync` for the pull requests CI opens. For a local run that opens pull requests under your own login, pass `--pr-author @me` to `push` and `pull`. + +Hub pages (`hub/...`) run only from a checkout of this repository's `dev` that is at `origin/dev`. + +A maintainer runs a sync from a `dev` checkout at `origin/dev`, signed in to Google with a one.org account. `.github/workflows/docsync.yml` runs one command on dispatch, with no schedule. Maintainers do not use it, because a Markdown upload made by its service account loses code blocks, and every page with one then fails to align. Its job runs only on `dev`, in the `docsync` environment, which holds the `dfo-docsync` App key and the Google workload identity settings (see the workflow's header). To run locally, pass a Google access token, which lasts about an hour. Without one, the command uses Application Default Credentials. + +```bash +git switch dev && git pull +gcloud auth login --enable-gdrive-access +export DOCSYNC_ACCESS_TOKEN=$(gcloud auth print-access-token) +uv run dfo-docsync pull --pr-author @me +uv run dfo-docsync push --pr-author @me +``` + +`pull` opens a pull request for each source repository with pages set to Ready for review. `push` refreshes each Doc whose page changed in its source repository, the pages of merged docsync pull requests included, and adds a Doc for each new page. Run the sequence again after docsync pull requests merge. A merge to this repository's `dev` rebuilds the corpus the site reads at once. A merge in another source repository reaches it at the 06:00 UTC build, or sooner with `gh workflow run build.yml --repo DevelopmentFinanceObservatory/docs --ref dev`. + +When the `docsync-state` branch is empty but Docs already exist under the folder, `push` stops with `StatelessDocsError`. Run it once as `uv run dfo-docsync push --adopt --pr-author @me` from a `dev` checkout at `origin/dev`. `--adopt` takes over each Doc that carries a page id and whose settled export aligns with the source, and records the current `dev` commit as its baseline, so it needs no earlier commit. A Doc that differs from the source holds an editor's unsent work, so its page fails and the Doc stays. Resolve those by hand, then rerun `push --adopt`. Later runs need no flag. + +Each page's state (Doc, source commit, baseline export, pull request) is a `state/.json` file on the orphan branch `docsync-state`, with one commit for each run. These guards hold a page back, and the run reports the reason and carries on: + +- `push` leaves a Doc alone when it has edits that have not reached GitHub ("edited in Docs; set Status to Ready for review to send"), open comments, or open suggestions, because replacing a Doc discards or detaches them. "Discard my edits" overrides the first two. It checks the Doc's revision again just before replacing it and holds the Doc if the revision changed. +- `pull` skips a Doc with open suggestions. A skipped or failed page does not stop the other pages of its repository from going in the pull request. +- `pull` leaves out an edit the source cannot take, such as a changed code block or code block language, a new table row, a folded description, or a deleted label line, and lists it in the pull request body. When that leaves nothing to send for the page (the edits are all reported, or already in GitHub), the reasons go to the Sheet's "Not applied" column, the page returns to "Editing", and the next `push` refreshes the Doc from GitHub. The script never writes the Notes column. +- A line break an editor adds inside a paragraph starts a new paragraph in the source. A new list item goes after the last block of the item above it, its note included, at that item's marker, and a new list opens after a top-level paragraph, so adding items and links to a "Next" list applies. A bullet or number added to or removed from a paragraph applies only its text, and the report says so. +- Author notes follow the Doc. An Author note whose label an editor removed or changed becomes page text and so appears on the site. A paragraph that gains the label becomes an Author note and leaves the site. The pull request reports each change ("an Author note became page text; check it belongs on the site", "a paragraph became an Author note and leaves the site"), and the reviewer decides. When a note or the Description is changed, added or deleted and page text next to it also changes, with no unchanged paragraph between them, the edit applies and the pull request flags it once ("an Author note and the page text beside it changed together; check no text moved between them", and the same with "the Description"), unless one of the two flags above already covers it. A report of an applied edit, an anchor change or an Author note flag, appears in the pull request that carries it and not in the next one. A report of an edit left out appears each time. An Author note emptied down to its label is deleted, and a new paragraph that starts with the label becomes a note. +- A label belongs at the start of its own Doc paragraph. A label typed mid-paragraph is refused. A note joined onto the paragraph before or after it, or Enter pressed in the middle of a note, is refused, or applied with that flag when the page then matches the Doc. A refused region is left unchanged, and the report quotes the editor's text of it. Text crosses a note's edge when a note's words join another paragraph or split off it; a paragraph typed with Enter next to a note is a new paragraph and applies. A paragraph typed with Enter next to a note in a list item is refused and held when the source gives it no list item to join. A refusal also covers a label that is unbolded, a Description or Tab label removed or changed with its text kept, a Note, Tip or Warning label deleted together with a change to its text, a tab line deleted together with a change to the text beside it, a note added or labelled inside a plain `>` quote, a new note that holds `-->`, a heading or list item given the note label, an added paragraph the source has no place for, and an edit beside repeated text where docsync cannot tell which copy changed. Changing Note to Tip or Warning, deleting a callout's label with its text unchanged, and deleting a whole tab line or Description line are reported as not applied and not refused, so the page records its export as applied and the next `push` refreshes the Doc. A refusal is reported as the message, then the Doc's text in a fence, byte for byte. A note inside a list item whose label is removed becomes text in that item, flagged. A line typed above the Description is page text and the Description stays front matter. +- A pull that sends nothing for a page and has a refusal leaves the Doc alone. The page returns to "Editing", the Not applied column holds each refusal's message and quoted Doc text and says to rework the edit or set "Discard my edits", and the page's state records no new applied export, so `push` finds the Doc edited and holds it. Every other report kind records the export as applied, and the next `push` refreshes the Doc from GitHub. When the pull sends a pull request for the page, the refusals go in its body, and the refused text leaves the Doc when that pull request merges and `push` refreshes it. The Not applied cell holds at most 45,000 characters and the pull request body 60,000. Refusal quotes are cut to fit. A cut quote ends with the line "(Shortened here. The full text was in the Doc when this pull ran.)". +- `pull` checks that the merged file reads in the source dialect, as `push` reads it. When it does not, nothing is sent for the page, the run exits 1, and the page is held like a refusal: the Not applied column says "the edits would leave a file docsync cannot read: ; nothing sent" and the Doc keeps the editor's text. +- When upstream and an editor changed different words of one paragraph, both changes stay. When they changed the same words, the pull request applies the editor's text and shows the upstream text beside it. + +The exit code is 1 when a page fails. It is 2 when the run stops: no credentials, folder, or Sheet; a git, `gh`, or Google error outside a page; a Page id listed twice, or a missing Page id or Status column, in the Sheet; or a state save that fails. A failed save writes the changed page states to `docsync-unsaved-states-.json` in the working directory. In CI the workflow uploads it as the `docsync-unsaved-states` artifact. Run `uv run dfo-docsync restore ` to save those states over the stored ones before the next run. `--state-dir ` restores into a local state directory. `restore` refuses when the state has moved since the failed run, because a later run may have saved newer states. Pass `--page ` (repeatable) to restore chosen pages, or `--force` to overwrite every page in the file. + +To add a source, append its name to `sources` in `docsync.yaml`. It must be an enabled `docs` or `skill` row in `sources.yaml` that `hub.yaml` mounts. For another repository, install the `dfo-docsync` GitHub App on it and add its name to the `repositories:` input of the "Mint a token for dfo-docsync" step in `.github/workflows/docsync.yml`. Then run `push`. [CONTRIBUTING.md](../CONTRIBUTING.md) covers writing the pages. + +To try a change away from the real folder, make a scratch folder in the shared drive, share it with your account, and point each command at it. A Doc pushed under one folder belongs to that folder alone. + +```bash +uv run dfo-docsync init --folder +uv run dfo-docsync push --folder --state-dir ./scratch-state +uv run dfo-docsync pull --folder --state-dir ./scratch-state --dry-run ./scratch-out +``` + +Without `--state-dir`, `push` and `pull` read and commit to the `docsync-state` branch on `origin`, so pass it whenever you experiment. + +## When the build fails + +The build checks every enabled repository before it exits, so the log names every failure in one run. + +### A source repository cannot be cloned + +The repository was renamed, made private, or deleted. Fix the `repo` field in `sources.yaml`, or disable the row. + +### An empty docs directory + +The source repository renamed its `docs_dir`. Update `docs_dir` in `sources.yaml` to match. + +### A leaked `::: ` directive + +A repository is publishing mkdocstrings-generated reference through the normal build path. Exclude the pages that carry it, or move the whole repository to the self-published tier with `kind: external`. + +### A sidecar declaring no nav + +The source would publish nothing. In a `zensical.toml` this is usually a `nav` array written after a `[project.
]` header, which TOML scopes into that table instead of into `[project]`. Move the array above every sub-table header. + +### A skill directory without a `SKILL.md`, or with front matter that does not parse + +The build names the directory or file. Add the `SKILL.md`, fix its front matter so it carries a `name` and a `description`, or move the directory out of the row's `docs_dir` and `internal_dir` if it is not a skill. + +### A skill directory of the same name under both `docs_dir` and `internal_dir` + +The build names both. A skill's directory name is its page address, so keep one. + +### A `categories.yaml` that does not parse, or lists a skill twice + +The build names the file. Give `sources` a list of skill directories and `domains` a mapping from each domain to such a list, and list each skill once. + +### An exclude pattern matching nothing + +A page an `exclude` row named was renamed or removed upstream. Update the pattern in `sources.yaml`, or drop it if the page is gone for good.