Skip to content

docs: one paragraph per line in km's Markdown, no hard wrap - #44

Merged
pasrom merged 1 commit into
mainfrom
docs/unwrap-markdown
Nov 7, 2026
Merged

pasrom merged 1 commit into
mainfrom
docs/unwrap-markdown

Conversation

@pasrom

@pasrom pasrom commented Nov 7, 2026 •

Copy link
Copy Markdown
Owner

Why

km's Markdown was hard-wrapped at about 100 characters, so every edit of a sentence reflowed its whole paragraph and the diff showed lines that did not change. This makes it one paragraph per line, as the brains' own Markdown is written.

Regenerated on main after #42, #43 and #45 merged (all three edit Markdown): the branch was reset to main and unwrap.py and prove.py below were rerun on the same file list, so this is one commit on top of main. #45 added no Markdown file (its new templates are a Python script and a workflow), so the list is the same 19 files. The templates under src/km/templates/ are written verbatim into brains, where a hard wrap would not matter to the renderer; they are included anyway, so that km's own Markdown and what it writes follow one style.

What changes

Every tracked *.md except the test fixtures (tests/fixtures/) and the eval cases' prompts and graders (evals/*/, inputs of the plugin evals): 19 files, of which 15 change. A hard-wrapped line is joined onto the line before it when that one is paragraph or list-item text without a hard break, and the line itself starts no block. Never touched: front matter (the skill's description: >- block stays as it is), fenced code, tables, HTML lines (a comment, a block tag, or a line that is one tag or placeholder, such as <FOLDER_ROWS> and <PLACEMENT_RULES>; a text line that only opens with a placeholder, such as <TEAM>'s shared ... or a wrapped <url> --file ..., is paragraph text and joins), headings, block quotes, link definitions, rules, and separate list items; a list item's wrapped continuation joins into its item.

Proof: whitespace only

prove.py HEAD~1 <files> (below) checks per file: (1) text: with every whitespace run normalised to one space, old and new are identical; (2) structure: the counts of headings, fence lines, table rows, list items, blank lines, HTML lines, quotes and hard breaks are unchanged, so no join swallowed a block; (3) render: pandoc (gfm) renders the same HTML, whitespace normalised. Output:

identical  CLAUDE.md  lines 24 -> 17  text same  structure same  html same
identical  README.md  lines 164 -> 110  text same  structure same  html same
identical  evals/README.md  lines 10 -> 10  text same  structure same  html same
identical  skills/km/SKILL.md  lines 151 -> 125  text same  structure same  html same
identical  skills/km/reference/brains.md  lines 186 -> 130  text same  structure same  html same
identical  skills/km/reference/github-reads.md  lines 58 -> 30  text same  structure same  html same
identical  skills/km/reference/pins-and-cites.md  lines 55 -> 21  text same  structure same  html same
identical  skills/km/reference/queue.md  lines 46 -> 17  text same  structure same  html same
identical  skills/km/reference/review-and-pr.md  lines 140 -> 65  text same  structure same  html same
identical  skills/km/reference/setup.md  lines 115 -> 52  text same  structure same  html same
identical  skills/km/reference/validate-and-gate.md  lines 148 -> 98  text same  structure same  html same
identical  src/km/templates/CLAUDE.template.md  lines 37 -> 35  text same  structure same  html same
identical  src/km/templates/CONVENTIONS.template.md  lines 129 -> 120  text same  structure same  html same
identical  src/km/templates/_index.inbox.template.md  lines 13 -> 13  text same  structure same  html same
identical  src/km/templates/team/CLAUDE.team.template.md  lines 50 -> 39  text same  structure same  html same
identical  src/km/templates/team/CONVENTIONS.team.template.md  lines 163 -> 113  text same  structure same  html same
identical  src/km/templates/team/README.team.template.md  lines 10 -> 8  text same  structure same  html same
identical  src/km/templates/team/_index.folder.template.md  lines 16 -> 16  text same  structure same  html same
identical  src/km/templates/team/_index.root.template.md  lines 15 -> 15  text same  structure same  html same
19 file(s), 0 different

Also: git diff --word-diff=porcelain HEAD~1 lists no changed word.

prove.py
"""Prove a re-wrap changed whitespace only, per file, old (a git revision) against the work tree.

    python3 prove.py REV FILE...

1. text: every whitespace run normalised to one space, old and new must be identical;
2. structure: line kinds that a join could swallow are counted (headings, fence lines, table rows,
   list items, blank lines, HTML lines (a comment, a block tag or a line that is one tag or
   placeholder), block quotes, hard breaks), old and new equal;
3. render: pandoc (gfm) renders old and new to HTML, compared with whitespace runs normalised.
"""
from __future__ import annotations

import re
import shutil
import subprocess
import sys

KINDS = {
    "headings": r"^\s*#",
    "fences": r"^\s*(```|~~~)",
    "table rows": r"^\s*\|",
    "list items": r"^\s*([-*+]|\d+[.)])\s+",
    "blank lines": r"^\s*$",
    # as unwrap.py: a comment, a CommonMark block tag, or a line that is one tag or placeholder
    "html": r"(?i)^\s*(<[!?]|</?(address|article|aside|blockquote|body|center|details|dialog|dd|div|dl|dt"
            r"|figcaption|figure|footer|form|h[1-6]|head|header|hr|html|li|main|nav|ol|p|pre|script"
            r"|section|style|summary|table|tbody|td|textarea|tfoot|th|thead|tr|ul)(\s|/?>|$)|<[^<>]+>\s*$)",
    "quotes": r"^\s*>",
    "hard breaks": r"(  |\\)$",
}


def flat(text: str) -> str:
    return " ".join(text.split())


def render(text: str) -> str:
    out = subprocess.run(["pandoc", "-f", "gfm", "-t", "html"], input=text, capture_output=True, text=True, check=True)
    return flat(out.stdout)


def main() -> int:
    rev, files = sys.argv[1], sys.argv[2:]
    pandoc = shutil.which("pandoc")
    bad = 0
    for f in files:
        old = subprocess.run(["git", "show", f"{rev}:{f}"], capture_output=True, text=True, check=True).stdout
        new = open(f, encoding="utf-8").read()
        same_text = flat(old) == flat(new)
        counts = {k: (sum(bool(re.search(p, ln)) for ln in old.split("\n")), sum(bool(re.search(p, ln)) for ln in new.split("\n")))
                  for k, p in KINDS.items()}
        same_kinds = all(a == b for a, b in counts.values())
        same_html = render(old) == render(new) if pandoc else None
        ok = same_text and same_kinds and same_html is not False
        bad += not ok
        lines = (len(old.split("\n")), len(new.split("\n")))
        print(f"{'identical' if ok else 'DIFFERENT'}  {f}  lines {lines[0]} -> {lines[1]}"
              f"  text {'same' if same_text else 'differs'}"
              f"  structure {'same' if same_kinds else {k: v for k, v in counts.items() if v[0] != v[1]}}"
              f"  html {'same' if same_html else 'not checked' if same_html is None else 'differs'}")
    print(f"{len(files)} file(s), {bad} different")
    return 1 if bad else 0


if __name__ == "__main__":
    sys.exit(main())
unwrap.py, the script that made the change (idempotent: a second run joins nothing)
"""Re-wrap Markdown to one paragraph per line: join a hard-wrapped line onto the line before it.

    python3 unwrap.py FILE...        rewrite in place, print what was joined per file

Never touched: front matter, fenced code, indented code, tables, HTML lines (a comment, a block
tag, or a line that is one tag or placeholder such as <FOLDER_ROWS>), headings, block quotes, link
definitions, horizontal rules and setext underlines. A line joins the line before it only when that one is paragraph or list-item text
without a hard break (two trailing spaces or a backslash) and the line itself starts no block:
not a list item, so separate items stay separate, while a list item's wrapped continuation joins
into its item.
"""
from __future__ import annotations

import re
import sys

LIST = re.compile(r"^\s*([-*+]|\d+[.)])\s+")
FENCE = re.compile(r"^\s*(```|~~~)")
RULE = re.compile(r"^\s*([-*_=])(\s*\1){2,}\s*$")
LINKDEF = re.compile(r"^\s*\[[^\]]+\]:\s")
# An HTML line: a comment or declaration, a CommonMark block tag, or a line that is a single tag or
# placeholder. A paragraph line that merely opens with a placeholder (<TEAM>'s ...) is text.
HTML = re.compile(r"^\s*(<[!?]|</?(address|article|aside|blockquote|body|center|details|dialog|dd|div|dl|dt"
                  r"|figcaption|figure|footer|form|h[1-6]|head|header|hr|html|li|main|nav|ol|p|pre|script"
                  r"|section|style|summary|table|tbody|td|textarea|tfoot|th|thead|tr|ul)(\s|/?>|$)"
                  r"|<[^<>]+>\s*$)", re.IGNORECASE)


def starts_block(line: str) -> bool:
    s = line.lstrip()
    return bool(LIST.match(line) or FENCE.match(line) or RULE.match(line) or LINKDEF.match(line)
                or HTML.match(line) or s.startswith(("#", "|", ">")))


def unwrap(text: str) -> tuple[str, int]:
    lines = text.split("\n")
    out: list[str] = []
    joins = 0
    i = 0
    if lines and lines[0] == "---":                       # front matter
        end = next((j for j in range(1, len(lines)) if lines[j] == "---"), None)
        if end is not None:
            out, i = lines[:end + 1], end + 1
    fence = None
    joinable = False          # whether the last line in `out` is paragraph text a line may join
    in_list = False           # whether the current block belongs to a list
    prev_blank = True
    html_comment = False
    while i < len(lines):
        line = lines[i]
        i += 1
        if fence:
            out.append(line)
            if line.strip().startswith(fence):
                fence = None
            continue
        if html_comment:
            out.append(line)
            html_comment = "-->" not in line
            continue
        if m := FENCE.match(line):
            fence = m.group(1)
            out.append(line)
            joinable = False
            prev_blank = False
            continue
        if not line.strip():
            out.append(line)
            joinable, prev_blank = False, True
            continue
        if line.lstrip().startswith("<!--") and "-->" not in line:
            html_comment = True
        if LIST.match(line):
            in_list = True
        elif prev_blank and not line.startswith((" ", "\t")):
            in_list = False
        indented_code = prev_blank and not in_list and re.match(r"^( {4}|\t)", line)
        if joinable and not starts_block(line):
            out[-1] = out[-1].rstrip() + " " + line.strip()
            joins += 1
        else:
            out.append(line)
        last = out[-1]
        joinable = not (indented_code or starts_block(last) and not LIST.match(last)
                        or last.endswith("  ") or last.endswith("\\"))
        if indented_code:
            joinable = False
        prev_blank = False
    return "\n".join(out), joins


def main() -> int:
    for path in sys.argv[1:]:
        old = open(path, encoding="utf-8", newline="").read()
        new, joins = unwrap(old)
        if new != old:
            open(path, "w", encoding="utf-8", newline="").write(new)
        print(f"{path}: {joins} line(s) joined")
    return 0


if __name__ == "__main__":
    sys.exit(main())

Evals

The change touches skills/km/, and CLAUDE.md asks for the plugin evals on such a change. They were not run: the proof above shows the skill text is the same word for word, so they would measure the model's variance, not the change, at a cost. That is your call; the cases to run for a core change are all of them (evals/README.md).

Reviews

/simplify: no code in the change, nothing to simplify. /code-review on the regenerated commit, findings and what was done:

  • Fixed: three paragraphs stayed hard-wrapped (skills/km/reference/setup.md "Keeping the copy current", the cites: item in the team CONVENTIONS template, the first paragraph of the team README template), because the first unwrap.py treated every line starting with < as HTML. It now treats a line as HTML only when it is a comment, a CommonMark block tag, or one tag or placeholder alone; prove.py's HTML count uses the same rule (the pandoc check stays the independent one). A scan for two adjacent text lines outside fences finds none left.
  • Left out, not in a whitespace change: the line-based checks in tests/skill_inventory.py (SKILL.md at most 500 lines, a ## Contents list in a reference file over 100 lines) bind far less once a paragraph is one line: review-and-pr.md, setup.md and validate-and-gate.md now fall below 100 lines with the same text (they keep their Contents lists). CORE_BYTES is unaffected: SKILL.md went from 11,408 to 11,384 bytes (the joins swap a newline for a space and drop list-continuation indent), and the bytes-per-token mix is the same text. A follow-up could move the line limits to bytes.
  • Noted: the longest lines are now 1,875 (skills/km/reference/queue.md), 1,772 (validate-and-gate.md) and 1,667 characters (README.md), below the 2,000 characters at which Claude Code's Read tool cuts a line. A follow-up could add a line-length check to the skill inventory.
  • Evals: see above.

All smoke suites pass locally on the regenerated commit: gate 116, index 30, pins 37, cites 87, doctor 57, sync 63, list 40, skill 2, hook 23, mounted_pins 2, team 141, encoding 4, each with 0 failed.

@pasrom
pasrom changed the base branch from test/retire-skill-baseline to main November 7, 2026 20:38
Every tracked Markdown file except the eval cases' prompts and graders
(test inputs) and the test fixtures: hard-wrapped lines are joined into
their paragraph or list item. Front matter, fenced code, tables, HTML and
placeholder lines, headings, block quotes and separate list items are
untouched. Whitespace only: with whitespace runs normalised, every file
is identical; the counts of headings, fences, table rows, list items,
blank lines and hard breaks are unchanged; pandoc renders the same HTML.
@pasrom
pasrom force-pushed the docs/unwrap-markdown branch from 30c4297 to 1be2013 Compare November 7, 2026 21:51
@pasrom
pasrom merged commit 608efc0 into main Nov 7, 2026
5 checks passed
@pasrom
pasrom deleted the docs/unwrap-markdown branch November 7, 2026 21:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant