Repository navigation
Adopt the engineering standard and fix the Substack sync paging - #32
Merged
Merged
Conversation
The 2 October 2026 scheduled sync failed with HTTP Error 403 Forbidden from the archive endpoint. fetch_json now retries those statuses, plus 429 and 5xx, three times with waits of 10, 30 and 90 seconds, then exits with a message that names the URL.
Update the dev dependency with npm so package.json and package-lock.json match. This is the bump from Dependabot pull request 27.
- .claude/engineering-standards.md: the engineering standard shared by every repository, copied from Democracy-by-Design, where tools/sync-standards.sh keeps the copies identical. - CLAUDE.md: imports the standard on its third line; nothing else changed. - .github/dependabot.yml: monthly, minor and patch updates grouped into one pull request per ecosystem, at most two open. Each merged pull request is a production deploy, so fewer, larger updates cost fewer Netlify credits. - Workflows with no permissions block now declare contents: read. None of them writes to the repository. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014MHzNDc4396EfY5Y96DGT8
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014MHzNDc4396EfY5Y96DGT8
…ps-2026-10-18' into ccr-eb68c78d-h22iov
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014MHzNDc4396EfY5Y96DGT8
Builds on PR #28's retries (merged into this branch). Closes the remaining items of #29 and #30: - list_posts() stopped at the first page shorter than limit=50. Substack returns short pages that are not the last (23, then 26, then 3 overlapping, measured 2026-10-19), so only 23 of 49 posts were ever listed. It now stops on an empty page, caps at 40 pages and keeps each post once by id, falling back to slug. - Browser User-Agent, with the project contact in From. - URLError, timeouts and resets retry like HTTP errors and end on one line. - A final fetch failure exits 2, apart from --check's 1. - --dry-run fetches and reports which files would be added or changed, and writes nothing. - The indexes are built from the fetched posts together with every file already in archive/, so a partial listing cannot drop one. - Workflow: a dry_run input (default true) that skips the rebuild and the PR, a step printing Substack's status code for the runner, a 10-minute timeout, and runs-on pinned to ubuntu-24.04. Not done: the RSS fallback in #30 (optional), because the API answered 200 here and RSS carries only recent posts. The schedule (#31) is unchanged; it is Varna's decision. The 403 did not reproduce from this sandbox with either User-Agent, so whether it is fixed on GitHub's runners is known only after a run there. Tests: 33 pass (10 new in tests/test_sync_substack.py, no network). build_companions.py --check passes. A live --dry-run listed 49 posts, 36 to add and 3 to change, and git status showed nothing written. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014MHzNDc4396EfY5Y96DGT8
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
.claude/engineering-standards.md), imported fromCLAUDE.md, plus the missing repository files.d9b6d8c. If this PR is merged, Fix: retry Substack sync failures and bump auto-changelog to 2.7.0 #28 can be closed.limit=50pages came back as 23, 26 and 3 posts, overlapping, then 0. The sync stopped at the first short page and so listed 23 of 49 posts. It now pages until an empty page, with a cap of 40 pages, and keeps each post once by id, falling back to slug.Fromheader;--checkstill exits 1;--dry-run;archive/.dry_runinput (default true), a first step that prints the status code Substack gives the runner, a 10-minute timeout and anubuntu-24.04runner.Verified
build_companions.py --checkpasses.Generated by Claude Code