Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -263,6 +263,8 @@ not open the PR against the fork unless the user explicitly asks you to.
| [docs/architecture/overview.md](docs/architecture/overview.md) | System architecture, component interactions, data flow |
| [docs/architecture/app-design.md](docs/architecture/app-design.md) | Data models, API design, package dependency graph |
| [docs/architecture/prompt-evaluations.md](docs/architecture/prompt-evaluations.md) | Static prompt quality, datasets, commands, and maintenance rules |
| [docs/architecture/taxonome.md](docs/architecture/taxonome.md) | Standalone Taxonome engine, catalog, CLI, ATIF/JSON API, evals and replay |
| [docs/architecture/trajectory-taxonomy-format.md](docs/architecture/trajectory-taxonomy-format.md) | Normative taxonomy authoring, shared assessments, typed ATIF evidence, JSONPath, hashing, and interoperability contract |
| [docs/architecture/data-organization-projects.md](docs/architecture/data-organization-projects.md) | Projects (a single container) to isolate/group data within a cluster; composes with data-tags and auth-rbac |
| [docs/architecture/auth-rbac.md](docs/architecture/auth-rbac.md) | Explicit-login IdP auth, Redis user-access cache, Portal handshake; deferred RBAC/internal-token roadmap |
| [docs/architecture/vscode-web-worker.md](docs/architecture/vscode-web-worker.md) | XState chat machine, GitHub auth flow, ARIA snapshots |
Expand Down
17 changes: 14 additions & 3 deletions NOTICE

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

83 changes: 83 additions & 0 deletions apps/taxonome/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,83 @@
# Taxonome

Standalone experimental YAML taxonomy CLI for plain ATIF and existing Scope run
archives. The catalog evaluates Skipped Invocation, Wrong Discovery, and
Content Misused through Ax/Copilot SDK, plus Discovered, Malformed Invocation,
and Timeout through deterministic TypeScript. Explicit local or
installed-package plugins can add custom deterministic handlers without
modifying Taxonome. No Scope backing services are needed.

The generic runtime is `packages/taxonomy-engine`; the shipped
agent-behavior definitions and Timeout handler are isolated in
`packages/ax-taxonomy`. The CLI explicitly composes both packages.

See [the architecture and usage guide](../../docs/architecture/taxonome.md)
for YAML and `extra.scope` contracts, archive safety, plugin authoring,
build/run/eval/optimization/replay commands, the typed API, evidence policy,
and scope limitations. A runnable custom project lives in
[`examples/taxonome-custom-handler`](../../examples/taxonome-custom-handler/README.md).
The language-neutral authoring/result/reference contract is the
[Trajectory Taxonomy Format](../../docs/architecture/trajectory-taxonomy-format.md).

`run`, `eval`, and `optimize` default to concise human-readable output even
when redirected. Pass `--json` for their complete machine-readable shapes.
`readout` always emits JSON and can optionally write Markdown.

`taxonome run` auto-detects single-run and multi-run Scope archives. Single-run
stdout remains unchanged. Multi-run archives require `--output-dir`, write one
atomic `<request-id>--<run-id>.json` record per root, and use
`--concurrency <1..16>` (default `4`). Compatible successful records resume
automatically without being rewritten; errors, corrupt/stale records, and
configuration or evidence-fingerprint mismatches are retried. Pass `--force`
to recompute every root. Archive scanning streams and discards workspace
snapshots, nested code archives, HAR, logs, and raw chat while retaining only
`run.yaml`, ATIF trajectories, and tool-call evidence.

Archive progress is visible before evaluation starts. TTYs use one throttled
scanner line, then switch to one stable row per archive root/run. Each row
shows final/run state, selected-element `DONE/RUN/WAIT` counts, and the active
element with `Prompt` or `Code` distinction. Rows remain in archive order;
narrow/short terminals keep active and failed runs visible and summarize
hidden rows. Non-TTY and `--json` modes emit bounded, plain append-only scan,
run, and element lifecycle lines on stderr. JSON stdout remains one document.

`--concurrency` bounds active runs, not elements. A worker evaluates one run's
selected elements sequentially in taxonomy order, reusing only its own healthy
prompt transport. An authored element error is recorded and later elements
continue when safe; poisoned transports are closed and replaced before the
next prompt element or run.

`taxonome readout` requires `--taxonomy` plus repeatable `--inputs` selections
covering the exact plain ATIF files or local Scope archives used by results.
Result arguments accept explicit files, directories, and quoted globs.
Taxonome matches normalized input identity/fingerprint, revalidates every
role/path/type/constraint, and resolves source attribution from ATIF.
Included error records add a failed-runs section and cause a nonzero exit while
preserving the partial JSON/Markdown report.

Prompt commands use `COPILOT_GITHUB_TOKEN`, `GH_TOKEN`, or `GITHUB_TOKEN` when
set; otherwise they securely resolve the active GitHub CLI account. No token is
printed or persisted.

Element preflight rules are authored under optional `preconditions` and
implemented as pure modules in
`packages/taxonomy-engine/src/preconditions/`, with one file per flag plus the
ordered evaluator.

From `packages/ax-taxonomy`, `pnpm eval` delegates directly to native
`taxonome eval` with quoted input/ground-truth globs. Add
`-- --element <stable-id>` to filter applicability to one element. Native eval
streams input/element progress, continues after reviewed failures, and keeps
`pnpm eval -- --json` as one clean machine document.

`--concurrency <1..16>` defaults to four global workers. Human TTYs receive a
fixed numeric in-place dashboard; non-TTY streams receive append-only
completion events. After a TTY dashboard, only a compact evaluation-level
Result line is printed; non-TTY human output retains detailed input rows.
Prompt workers own independent lazy SDK transports, while deterministic-only
workers never initialize the SDK.

The [recorded findings](../../docs/architecture/taxonome.md#recorded-spike-findings-2026-10-21)
link the actual SDK baselines, complete Ax artifacts, fresh final comparisons,
the historical one-off 256/500-call ledger, and deterministic JSON/Markdown
preview. The current CLI does not expose that ledger/budget feature.
24 changes: 24 additions & 0 deletions apps/taxonome/package.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
{
"name": "taxonome",
"version": "0.1.0",
"private": true,
"type": "module",
"bin": {
"taxonome": "dist/index.js"
},
"scripts": {
"build": "tsc",
"lint": "tsc --noEmit"
},
"dependencies": {
"ax-taxonomy": "workspace:*",
"commander": "14.0.3",
"taxonomy-engine": "workspace:*"
},
"devDependencies": {
"@types/node": "^22.7.0",
"@types/tar-stream": "^3.1.4",
"tar-stream": "3.1.7",
"typescript": "^5.6.2"
}
}
56 changes: 56 additions & 0 deletions apps/taxonome/src/cli-runtime.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
// Copyright (c) Microsoft Corporation.
// Licensed under the MIT License.

import { describe, expect, it, vi } from "vitest";
import {
createHandlerRegistry,
loadTaxonomy,
parseScopeAtif,
type PromptExecutionResult,
type PromptExecutor,
} from "taxonomy-engine";
import { createLazyPromptExecutor } from "./cli.js";

describe("Taxonome lazy prompt runtime wrapper", () => {
it("forwards AbortSignal through the exact lazy executor layer", async () => {
const taxonomy = await loadTaxonomy(
new URL(
"../../../packages/taxonomy-engine/fixtures/test-taxonomy.yaml",
import.meta.url,
).pathname,
{ handlers: createHandlerRegistry() },
);
const element = taxonomy.elementById.get("test-observation")!;
const input = {
taxonomy,
atif: parseScopeAtif({
schema_version: "ATIF-v1.7",
session_id: "lazy-signal",
agent: { name: "fixture", version: "1.0.0" },
steps: [{ step_id: 1, source: "user", message: "Test." }],
}),
};
const controller = new AbortController();
const underlying: PromptExecutor = {
evaluate: vi.fn(async (_element, _input, signal) => {
expect(signal).toBe(controller.signal);
return {
status: "succeeded",
candidate: { state: "completed", observed: false },
attempts: [{
attempt: 1,
kind: "initial",
outcome: "succeeded",
}],
} satisfies PromptExecutionResult;
}),
};
const start = vi.fn(async () => underlying);
const lazy = createLazyPromptExecutor(start);

await lazy.evaluate(element, input, controller.signal);
await lazy.evaluate(element, input, controller.signal);
expect(start).toHaveBeenCalledOnce();
expect(underlying.evaluate).toHaveBeenCalledTimes(2);
});
});
Loading
Loading