Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
84 commits
Select commit Hold shift + click to select a range
8b5b4b2
fix(graphify): canonicalize docstruct merge before publish
abmmhasan Sep 24, 2026
84ec675
test(graphify): require prepublish round-trip canonicalization
abmmhasan Sep 24, 2026
df89d68
docs(graphify): explain canonical docstruct publication
abmmhasan Sep 24, 2026
13c8298
docs(graphify): describe round-trip-safe hybrid flow
abmmhasan Sep 24, 2026
4b80412
docs(graphify): document canonical staged handoff
abmmhasan Sep 24, 2026
cbb2b0f
test(docs): lock round-trip-safe Graphify workflow
abmmhasan Sep 24, 2026
803df28
test(graphify): prove canonical round-trip idempotence
abmmhasan Sep 24, 2026
4930f94
ci(graphify): validate canonical round trips on min and latest
abmmhasan Sep 24, 2026
e2a40cc
fix(graphify): keep round-trip copy portable
abmmhasan Sep 24, 2026
76974e6
feat(documents): add containerized Pandoc conversion
abmmhasan Sep 24, 2026
45c6f9b
fix(documents): order Docker mounts before Tools image
abmmhasan Sep 24, 2026
751616b
feat(cli): expose host document conversion
abmmhasan Sep 24, 2026
d4eed24
test(documents): cover host Pandoc conversion
abmmhasan Sep 24, 2026
4e76291
ci(documents): run Pandoc conversion contract
abmmhasan Sep 24, 2026
bc08579
test(cli): expose document conversion command
abmmhasan Sep 24, 2026
b856a15
docs(documents): add Pandoc conversion guide
abmmhasan Sep 24, 2026
c4d2278
docs(documents): link conversion guide
abmmhasan Sep 24, 2026
3c36ae8
docs(cli): document host conversion command
abmmhasan Sep 24, 2026
fe89173
docs(quickstart): mention document conversion
abmmhasan Sep 24, 2026
16f45df
docs(readme): add containerized document conversion
abmmhasan Sep 24, 2026
2a70ad7
fix(documents): protect against output symlink overwrite
abmmhasan Sep 24, 2026
35402aa
test(docs): lock document conversion surface
abmmhasan Sep 24, 2026
4761a5f
fix(documents): reserve compact Pandoc output options
abmmhasan Sep 24, 2026
cdcd807
test(documents): cover symlink and compact output safety
abmmhasan Sep 24, 2026
662de33
test(windows): lock document conversion path safety
abmmhasan Sep 24, 2026
40ddfd6
ci(documents): smoke real Tools Pandoc conversion
abmmhasan Sep 24, 2026
db2a929
refactor(convert): namespace docs and image conversion
abmmhasan Sep 24, 2026
e2f49c6
refactor(convert): expose docs and image subcommands
abmmhasan Sep 24, 2026
ce2bbca
refactor(convert): remove superseded documents helper
abmmhasan Sep 24, 2026
dbc4951
hardening(convert): isolate conversion containers
abmmhasan Sep 24, 2026
c7d6fa0
test(convert): move document conversion under docs namespace
abmmhasan Sep 24, 2026
dca7946
test(convert): cover ImageMagick image conversion
abmmhasan Sep 24, 2026
c7c21f5
ci(convert): validate docs and image conversion
abmmhasan Sep 24, 2026
c1d896e
test(windows): follow unified conversion helper
abmmhasan Sep 24, 2026
fef62f1
test(cli): expose convert namespace
abmmhasan Sep 24, 2026
6bd6e85
docs(convert): namespace docs and image conversion
abmmhasan Sep 24, 2026
296dbde
docs(convert): introduce docs and image namespaces
abmmhasan Sep 24, 2026
9145933
docs(convert): document docs and image subcommands
abmmhasan Sep 24, 2026
7d2a41e
docs(convert): cover image conversion alongside docs
abmmhasan Sep 24, 2026
ceffd27
test(docs): lock docs and image conversion namespaces
abmmhasan Sep 24, 2026
17ad2fd
docs(convert): rename guide for docs and images
abmmhasan Sep 24, 2026
a086ccd
docs(convert): remove document-only guide path
abmmhasan Sep 24, 2026
5a34ff3
docs(convert): correct ImageMagick option examples
abmmhasan Sep 24, 2026
0d6ac13
docs(convert): point to unified conversion guide
abmmhasan Sep 24, 2026
1b703d6
test(docs): follow unified conversion guide
abmmhasan Sep 24, 2026
ad0137a
docs(convert): generalize conversion heading
abmmhasan Sep 24, 2026
75d5495
docs(convert): generalize quickstart conversion wording
abmmhasan Sep 24, 2026
0d2a271
refine(convert): limit forced first-frame outputs
abmmhasan Sep 24, 2026
365263f
test(convert): isolate animated WebP assertion
abmmhasan Sep 24, 2026
8ec8c74
test(cli): name unified conversion namespace
abmmhasan Sep 24, 2026
dd5c10e
test(windows): name unified conversion contract
abmmhasan Sep 24, 2026
7896884
test(docs): name unified conversion guide
abmmhasan Sep 24, 2026
cb24b86
test(convert): avoid format-list broken pipe
abmmhasan Sep 24, 2026
8cca76e
fix(tools): preserve stdin and Tools runtime env
abmmhasan Sep 24, 2026
fd9a10a
feat(tools): expose curated Toolset utility namespace
abmmhasan Sep 24, 2026
5dc9cbe
test(tools): cover catalog and temporary runner routing
abmmhasan Sep 24, 2026
e8024e3
test(tools): lock stdin and runtime env propagation
abmmhasan Sep 24, 2026
9f7a610
docs(tools): expose curated Tools catalog and runner
abmmhasan Sep 24, 2026
78a8d24
docs(tools): document temporary utility runner
abmmhasan Sep 24, 2026
cd0ea37
docs(tools): add workspace Toolset examples
abmmhasan Sep 24, 2026
2c8cf3f
docs(tools): make bundled utility runner discoverable
abmmhasan Sep 24, 2026
965512b
test(cli): expose Tools utility runner in help
abmmhasan Sep 24, 2026
f44e866
test(docs): lock Tools utility catalog and runner
abmmhasan Sep 24, 2026
edeb251
test(tools): exercise piped temporary utility runner
abmmhasan Sep 24, 2026
c2efa2e
test(cli): follow expanded Tools help wording
abmmhasan Sep 24, 2026
1fedda2
fix(tools): avoid env-inspection pipe truncation
abmmhasan Sep 24, 2026
d5df4dc
test(tools): report temporary runner failures clearly
abmmhasan Sep 24, 2026
7d76b2e
fix(tools): tolerate optional runtime env gaps
abmmhasan Sep 24, 2026
1755df9
docs(tools): state privileged runner boundary
abmmhasan Sep 24, 2026
2d1452e
docs(tools): document privileged runner boundary
abmmhasan Sep 24, 2026
3ed5e40
test(docs): lock Tools privilege boundary
abmmhasan Sep 24, 2026
10229a6
feat(convert): add FFmpeg audio and video conversion
abmmhasan Sep 24, 2026
d32985f
test(convert): cover FFmpeg audio and video conversion
abmmhasan Sep 24, 2026
73ca564
docs(cli): expose audio and video conversion
abmmhasan Sep 24, 2026
02e4cee
feat(tools): add media utilities to curated catalog
abmmhasan Sep 24, 2026
e46d86b
ci(convert): run audio and video conversion contract
abmmhasan Sep 24, 2026
1bd98bc
docs(media): document audio video conversion and tools
abmmhasan Sep 24, 2026
c854f01
docs(media): add audio video conversion guide
abmmhasan Sep 24, 2026
0a52e48
docs(media): document FFmpeg conversion surface
abmmhasan Sep 24, 2026
b1272ca
docs(media): add audio video quickstart examples
abmmhasan Sep 24, 2026
7673bb0
test(cli): expose audio and video conversion help
abmmhasan Sep 24, 2026
7bd2ed3
test(docs): lock media conversion and tools guidance
abmmhasan Sep 24, 2026
fee0886
test(tools): lock media utility catalog
abmmhasan Sep 24, 2026
ea73aca
docs(quickstart): separate image and media literal blocks
abmmhasan Sep 24, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 23 additions & 1 deletion .github/workflows/check.yml
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,11 @@ jobs:
- name: Container execution substrate contract
run: tests/container-exec-contract.sh


- name: Conversion contracts
run: |
bash tests/document-convert-contract.sh
bash tests/image-convert-contract.sh
bash tests/media-convert-contract.sh

- name: Environment contract
run: tests/env-contract.sh
Expand Down Expand Up @@ -117,6 +121,9 @@ jobs:
- name: Smoke direct provider on minimum Graphify
run: bash tests/graphify-direct-provider-contract.sh

- name: Validate document merge round trip on minimum Graphify
run: bash tests/graphify-roundtrip-contract.sh

- name: Validate latest Graphify contract
run: |
python -m pip install --upgrade "graphifyy[openai]"
Expand All @@ -139,6 +146,9 @@ jobs:
- name: Smoke direct provider on latest Graphify
run: bash tests/graphify-direct-provider-contract.sh

- name: Validate document merge round trip on latest Graphify
run: bash tests/graphify-roundtrip-contract.sh


compose:
name: Compose contract
Expand Down Expand Up @@ -178,6 +188,18 @@ jobs:
- name: Validate published compatibility images
run: tests/published-images.sh

- name: Smoke host document conversion
run: |
set -euo pipefail
tmp="$(mktemp -d)"
trap 'rm -rf "$tmp"' EXIT
mkdir -p "$tmp/input" "$tmp/output"
printf '# LocalDevStack conversion\n\nPandoc-backed host document.\n' >"$tmp/input/source.md"
./lds convert docs "$tmp/input/source.md" "$tmp/output/source.html" --standalone
grep -Fq '<h1' "$tmp/output/source.html"
formats="$(./lds convert docs --list-input-formats)"
grep -qx markdown <<<"$formats"

windows-bridge:
name: Windows bridge
runs-on: windows-latest
Expand Down
57 changes: 56 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -224,6 +224,61 @@ lds es ...

See `docs/guides/databases-and-clients.rst` for the profile/client map.

## Tools and Toolset utilities

The Tools image also provides developer utilities through an explicit workspace-aware
runner:

```bash
lds tools list
lds tools gitx status
lds tools gitx worklog HEAD~20..HEAD
lds tools sqlitex --db app.db tables
cat app.log | lds tools chromacat --log
lds tools netx route show
lds tools ffprobe media.mkv
lds tools mediainfo media.mkv
lds tools soxi recording.wav
lds tools mkvinfo media.mkv
lds tools jq --version
lds tools shellcheck script.sh
lds tools ui
```

`lds tools <tool>` starts a temporary Tools container with the current host directory at
`/workspace`, shares the running server-tools network/volumes, preserves stdin/TTY, and
inherits the active LDS AI/Git runtime settings. Because `server-tools` owns the Docker
socket and trusted control-plane/secret mounts, this runner is a privileged workstation
context—not a sandbox—and should be used only with trusted commands from the Tools image.
Use `lds tools run <tool> ...` when a tool name collides with an LDS `tools` subcommand.
The older `tools sh/exec/file` forms continue to target the long-running control-plane
container.

## File conversion

Pandoc is available from the Tools image without installing it on the host or starting
the LocalDevStack services:

```bash
lds convert docs README.md README.html
lds convert docs docs/guide.rst guide.docx --toc
lds convert docs report.docx report.md --wrap=none
lds convert docs book.md book.epub --toc
lds convert docs --list-input-formats
lds convert docs --list-output-formats
lds convert image photo.jpg photo.webp -- -quality 82 -strip
lds convert image animation.gif animation.webp
lds convert image --formats
lds convert audio recording.wav recording.mp3
lds convert audio recording.wav recording.ogg -- -c:a libopus -b:a 128k
lds convert video recording.mov recording.mp4
lds convert video recording.mkv recording.webm -- -c:v libvpx-vp9 -crf 32 -b:v 0
```

Documents use Pandoc, images use ImageMagick, and audio/video use FFmpeg. The input mount is read-only and only the output mount is writable. The
short-lived converter receives no Docker socket or LocalDevStack networks. Existing
outputs require `--force`. PDF generation additionally depends on a PDF engine; the base
Tools image ships Pandoc itself, not a TeX/PDF rendering stack.
## Optional local AI

Enable the `ai` profile through `lds setup profile`.
Expand Down Expand Up @@ -286,7 +341,7 @@ lds graphify ./your-project --mode deep

Nginx owns the loopback-only native route `127.0.0.1:11434 -> nginx:11434 -> llm:11434`. Provider containers do not publish host ports.

`lds graphify` keeps the host Graphify CLI on `http://llm.localhost:11434/v1`, with no Graphify proxy service or Python adapter. When the Tools image supports `docstruct`, `.md/.rst/.yaml/.yml/.json/.toml/.ini/.cfg` files are extracted mechanically, optionally reviewed in bounded AI chunks, validated as a Graphify fragment, and merged into a reserved document layer; Graphify continues to own code ASTs and unsupported semantic formats. `LDS_GRAPHIFY_DOCSTRUCT=auto|on|off` and `LDS_GRAPHIFY_DOC_REVIEW=auto|on|off` control the handoff. Explicit `--code-only` remains code-only.
`lds graphify` keeps the host Graphify CLI on `http://llm.localhost:11434/v1`, with no Graphify proxy service or Python adapter. When the Tools image supports `docstruct`, `.md/.rst/.yaml/.yml/.json/.toml/.ini/.cfg` files are extracted mechanically, optionally reviewed in bounded AI chunks, validated as a Graphify fragment, and merged into a staged reserved document layer. Before publication, LocalDevStack runs that staged graph through an isolated LLM-free `graphify cluster-only --no-label --no-viz` round trip and publishes only Graphify's canonical output; Graphify continues to own code ASTs and unsupported semantic formats. Document review retries one malformed structured response once before deterministic fallback. `LDS_GRAPHIFY_DOCSTRUCT=auto|on|off` and `LDS_GRAPHIFY_DOC_REVIEW=auto|on|off` control the handoff. Explicit `--code-only` remains code-only.

The built-in Compose layout keeps both provider definitions in `docker/compose/companion.yaml`, but runtime-generated profile selectors enable exactly one. NVIDIA/ROCm hardware augmentation is generated ephemerally under `docker/.runtime/`; FastFlow's `/dev/accel/accel0` + memlock contract lives in its tracked service definition.

Expand Down
44 changes: 37 additions & 7 deletions bin/tool-runner
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,38 @@ server_tools_image() {
server_tools_env_value() {
local key="${1:?}"
"$(bin_path docker)" inspect -f '{{range .Config.Env}}{{println .}}{{end}}' "$SERVER_TOOLS_CONTAINER" 2>/dev/null \
| awk -F= -v k="$key" '$1==k { sub(/^[^=]*=/, ""); print; exit }'
| awk -F= -v k="$key" '
$1 == k && !found {
sub(/^[^=]*=/, "")
print
found=1
}
END { if (!found) exit 1 }
'
}

stdin_has_data() {
[[ -p /dev/stdin || -f /dev/stdin ]]
}

append_server_tools_env() {
local key value
local -n target="${1:?}"

for key in \
TZ USERNAME GIT_USER_NAME GIT_USER_EMAIL GIT_CREDENTIAL_MODE \
LDS_AI_ENABLED LDS_AI_RUNTIME LDS_AI_MODEL LDS_AI_THINK \
LDS_AI_CONNECT_TIMEOUT LDS_AI_PREFLIGHT_TIMEOUT LDS_AI_TIMEOUT \
LDS_AI_AVAILABILITY_TTL LDS_AI_MAX_CONTEXT_BYTES \
LDS_AI_MAX_REQUEST_BYTES LDS_AI_MAX_RESPONSE_BYTES
do
value="$(server_tools_env_value "$key" || true)"
if [[ -n "$value" ]]; then
target+=(-e "$key=$value")
fi
done

return 0
}

resolve_workspace() {
Expand Down Expand Up @@ -100,15 +131,13 @@ main() {
server_tools_running || die "$SERVER_TOOLS_CONTAINER is not running"
server_tools_has "$cmd" || die "command '$cmd' not found in $SERVER_TOOLS_CONTAINER"

local image workspace tz username home_dir
local image workspace home_dir
image="$(server_tools_image)" || die "unable to inspect $SERVER_TOOLS_CONTAINER image"
[[ -n "$image" ]] || die "unable to resolve $SERVER_TOOLS_CONTAINER image"

workspace="$(resolve_workspace)"
[[ -n "$workspace" ]] || die "unable to resolve workspace"

tz="$(server_tools_env_value TZ || true)"
username="$(server_tools_env_value USERNAME || true)"
home_dir="/home/root"

local -a flags envs
Expand All @@ -120,14 +149,15 @@ main() {
-e TERM="${TERM:-xterm-256color}"
)

[[ -n "$tz" ]] && envs+=(-e TZ="$tz")
[[ -n "$username" ]] && envs+=(-e USERNAME="$username")
append_server_tools_env envs
[[ -n "${COLORTERM:-}" ]] && envs+=(-e COLORTERM="$COLORTERM")
[[ -n "${NO_COLOR:-}" ]] && envs+=(-e NO_COLOR="$NO_COLOR")
[[ -n "${CLICOLOR_FORCE:-}" ]] && envs+=(-e CLICOLOR_FORCE="$CLICOLOR_FORCE")
[[ -n "${FORCE_COLOR:-}" ]] && envs+=(-e FORCE_COLOR="$FORCE_COLOR")

[[ -t 0 ]] && flags+=(-i)
if [[ -t 0 ]] || stdin_has_data; then
flags+=(-i)
fi
[[ -t 1 ]] && flags+=(-t)

if [[ -n "${MSYSTEM:-}${CYGWIN:-}" ]]; then
Expand Down
131 changes: 131 additions & 0 deletions docs/guides/conversion.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,131 @@
Conversion
==========

LocalDevStack exposes Pandoc from the Tools image as a host-file conversion command.
Pandoc does not need to be installed on the workstation and the main LocalDevStack
services do not need to be running.

Basic Usage
-----------

Convert one host file to another::

lds convert docs README.md README.html
lds convert docs docs/guide.rst guide.docx
lds convert docs report.docx report.md
lds convert docs book.md book.epub --toc

The first path is mounted read-only. Only the output directory is mounted writable.
The short-lived conversion container receives no Docker socket, project volumes, or
LocalDevStack networks.

Pandoc Options
--------------

Arguments after the output path are passed to Pandoc without shell flattening::

lds convert docs README.md README.html --toc --standalone
lds convert docs report.docx report.md --wrap=none
lds convert docs book.md book.epub --metadata title="Developer Guide"

An optional ``--`` separator is accepted::

lds convert docs README.md README.html -- --toc --standalone

``-o`` / ``--output`` is intentionally rejected because LocalDevStack owns the output
path through the second positional argument.

Relative Assets
---------------

Pandoc runs with the input directory as its working directory and with
``--resource-path=/lds-input``. Relative images and other resources next to the input
document therefore remain available during conversion.

Auxiliary file options such as a reference document should use files below the input
directory and refer to them with relative paths.

Overwrite Safety
----------------

Existing output files are not replaced unless ``--force`` is supplied::

lds convert docs --force README.md README.html

The input and output may not resolve to the same file.

Format Discovery
----------------

The supported readers and writers come from the Pandoc version currently shipped by the
Tools image::

lds convert docs --list-input-formats
lds convert docs --list-output-formats
lds convert docs --version

Pandoc supports many text/document formats, but not every format can be converted to
every other format. PDF output additionally requires a compatible PDF engine; the base
Tools image currently provides Pandoc itself, not a TeX/PDF rendering stack.

Windows and Git Bash
--------------------

The converter normalizes Windows/Git Bash host paths before creating Docker mounts and
disables MSYS argument rewriting for the Docker invocation. Paths containing spaces are
preserved as individual argv values.

Image Conversion
----------------

Raster/image conversion uses ImageMagick from the same Tools image::

lds convert image photo.jpg photo.png
lds convert image photo.png photo.webp -- -quality 82 -strip
lds convert image animation.gif animation.webp
lds convert image animation.gif preview.jpg

Use ``lds convert image --formats`` to inspect the delegates/formats available in the current image. Static outputs such as JPEG and PNG use the first frame of animated inputs by default; animation-capable GIF/WebP outputs preserve frames when supported.

Audio and Video Conversion
--------------------------

Audio and video conversion use FFmpeg from the Tools image::

lds convert audio recording.wav recording.mp3
lds convert audio recording.wav recording.ogg -- -c:a libopus -b:a 128k
lds convert video recording.mov recording.mp4
lds convert video recording.mkv recording.webm -- -c:v libvpx-vp9 -crf 32 -b:v 0

The first-class media converter owns one input, overwrite policy, and one output path.
FFmpeg output options are passed after the input as exact argv. LDS reserves ``-i``,
``-y``, and ``-n`` because it owns input/output and ``--force`` behavior.

Inspect FFmpeg capabilities with::

lds convert audio --formats
lds convert audio --codecs
lds convert audio --encoders
lds convert video --version

For multi-input, concat, capture, or complex filtergraph workflows use the raw Tools
surface instead::

lds tools ffmpeg ...
lds tools ffprobe media.mkv

Specialist Media Tools
----------------------

The Tools image also exposes media utilities directly::

lds tools sox ...
lds tools soxi recording.wav
lds tools mkvmerge ...
lds tools mkvinfo media.mkv
lds tools mkvextract ...
lds tools mkvpropedit ...
lds tools mediainfo media.mkv

``xvidcore`` is installed as an explicit FFmpeg codec runtime dependency; it is a
library rather than a standalone command.
11 changes: 8 additions & 3 deletions docs/guides/local-ai.rst
Original file line number Diff line number Diff line change
Expand Up @@ -295,8 +295,12 @@ When the active docker-tools image exposes the docstruct Graphify handoff,
5. optional semantic review runs in bounded chunks against the active local model;
6. docker-tools emits a Graphify-compatible fragment;
7. Graphify's public ``merge-chunks`` validates that fragment;
8. docker-tools atomically replaces only the reserved ``docstruct_`` semantic layer;
9. ``graphify label`` reclusters and relabels the final combined graph.
8. docker-tools replaces only the reserved ``docstruct_`` semantic layer in a staged graph;
9. LocalDevStack copies that staged graph into an isolated temporary workspace and runs
``graphify cluster-only --no-label --no-viz``; only Graphify's canonical round-trip
output is eligible for publication;
10. the canonical graph is atomically published to ``graphify-out/graph.json``, then
``graphify label`` reclusters and relabels the final combined graph.

This removes Markdown/RST/config parsing and recognized Python pip requirement manifests
from the fragile raw LLM extraction path while preserving Graphify's existing support for
Expand All @@ -321,7 +325,8 @@ Controls
``LDS_GRAPHIFY_DOC_REVIEW``:

- ``auto`` (default): review bounded document chunks on the built-in local provider;
if review fails, keep the deterministic structure and continue;
docker-tools retries one malformed structured response once, then LocalDevStack keeps
the deterministic structure and continues if review still fails;
- ``on``: require semantic review to succeed;
- ``off``: use deterministic document structure only.

Expand Down
1 change: 1 addition & 0 deletions docs/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ LocalDevStack is designed for trusted developer workstations, not production dep

guides/domain-setup
guides/databases-and-clients
guides/conversion
guides/tls-and-certificates
guides/local-ai
guides/operations-and-support
Expand Down
Loading
Loading