MinerU literature library

This repository manages a citation-safe literature library built from numeric pre-commit workspaces and transactionally installed formal papers.

Core model

• Metadata v2.0 is a frozen bibliographic record and the only citation source.
• Catalog v3.2 is a complete paper-content dossier for people and downstream models. It includes trusted source abstract, Chinese summary, question, methods, data, findings, mechanisms, limitations, terminology, figures, evidence, writing value, screening, and the semantic paper_name.
• paper_number is permanent. paper_name is human-readable and generated by the Catalog task. They are linked only by paper_number.
• data/paper_raw/<paper_number>/ is the only pre-commit workspace. Formalize writes an installation plan; commit renames only inside hidden staging.

Read project status, contract, architecture, testing, script index, agent rules, mirrored agent rules, dependencies, writing acceptance, writer plan, and change governance (Chinese).

Ingest

Manual PDF path:

stage_raw_pdfs_to_paper_raw.py --move --apply
convert_paper_raw_gpu.py --all --apply
resolve_paper_raw_metadata.py --all-unmatched --apply --allow-network
freeze_paper_raw_metadata.py
prepare_paper_raw_catalog_task.py
# Catalog Skill writes complete Catalog v3.2
validate_paper_raw_catalog.py --apply
formalize_paper_raw.py --all-ready --apply
commit_paper_raw_to_papers.py --all-ready --apply

data/raw/ is a queue. The resolver consumes converted Markdown after MinerU; normal staging is always --move --apply. Normal conversion uses --only-convertible and does not require Metadata matching first.

Network Metadata path stages DOI-backed citation Metadata first, fetches and attaches the PDF, performs strict match/freeze, converts, then joins the same Catalog/formalize/commit path. Local conversion may run before Metadata freeze.

All production commands support isolated roots where applicable. Tests and agents must pass temporary paper_raw, papers, ledger, index, transaction, and report paths; they must not use real network or MinerU.

Writing

Writing selects by Catalog/paper_number, copies formal assets into write/jobs/<job_id>/article/<paper_number>/, and generates BibTeX/citation keys only from copied Metadata. It never infers citations from Catalog, Markdown, a directory name, or paper_name.

Catalog folders and classification

Catalog folders are the browsing index over formal papers. Chinese keyword categories are sourced from enabled schema-v4 DOI keyword notebooks. The currently configured registry contains:

data/catalog/
├── all/<paper_name>              # every active formal paper
├── _pending/<paper_name>         # papers lacking current classification
├── 大气边界层/<paper_name>
├── 风沙物理学/<paper_name>
├── 风雪物理学/<paper_name>
├── 风沙动力学/<paper_name>
├── 风雪动力学/<paper_name>
└── .state/                       # assignments, tasks, results, registry

• paper_name is the link name and target directory name; paper_number identifies assignments, tasks, results, and receipts.
• DOI notebook keyword_zh is the sole source of keyword_id, Catalog categories, directory names, and classification identity.
• search_queries contains all enabled Chinese and English provider queries; Discovery submits both languages to OpenAlex/Crossref and keeps independent provider progress for every query.
• enabled=true notebooks must have at least one active Chinese and one active English query. New or incomplete definitions are created as disabled drafts and become active only through the store-level readiness check.
• English search queries never create Catalog categories, directories, or classification tasks. Adding, removing, or disabling a search query does not change keyword_id, definition_hash, or an existing classification decision.
• assignment is the classification fact source; catalog folder links are a recoverable projection.
• Classification apply uses a transaction journal for crash recovery.
• Doctor diagnoses folder integrity, classification completeness, and notebook/registry drift. Writer refuses keyword-filtered reads when writer_category_safe is false.
• Classification lifecycle: sync_catalog_categories.py → claim tasks → classify (via skill) → import results → doctor verifies.

audit_discovery_keyword_index_sources.py is strict-v4 and read-only. Legacy v3 notebooks are unsupported inputs; the one-time v3→v4 migration finalized on 2026-07-25 and its tooling has been removed.

The current production set contains five enabled Chinese schema-v4 notebooks. Their migration was inventory-checked and applied from a fixed reviewed plan; the confirmed source mapping remains operator state under .local/discovery_keyword_v3/, while the fixed plan and committed journal remain excluded runtime transaction state. None is source or snapshot content. The read-only workflow is:

python scripts/audit_discovery_keyword_index_sources.py
python scripts/sync_catalog_categories.py
python scripts/discover_papers_concurrent.py --from-enabled-notebooks --dry-run

The discovery dry-run prints every notebook's active queries, provider lanes, generation/signature state, refresh and backfill page limits, worker count, and page budget. It performs no provider I/O, cursor advance, page-journal write, paper allocation, notebook mutation, or Catalog mutation. Real network search starts only on an explicit user command after these checks.

Discovery startup also requires the active formal publication closure at data/papers/.formal_publication_state.json. Audit and repair it with:

python scripts/repair_formal_publications.py --audit-only --json-report reports/formal_publication_repair.json
python scripts/repair_discovery_workspaces.py --audit-only --limit 100

The formal repair writer only applies identity-only fixes after a temporary current-schema validation; closure or freeze drift emits a rollback_recommit_required plan. Raw incomplete metadata_staged entries may be explicitly demoted to reserved without recycling their numbers.

python scripts/sync_catalog_categories.py --apply
python scripts/doctor_catalog_folders.py --json

Discovery relevance is notebook-local and fail-closed. Enabled notebooks bind one taxonomy-resolved profile before provider I/O, and each batch freezes the complete active mapping once. Profile replacement closes only safe pre-staging old-profile observations. Recovery-required candidates and unknown lifecycle values block the plan; historical terminal candidates and durable DOI facts are never rewritten. Apply uses deterministic expected-after bytes, a zero-write global preflight, a durable crash-resumable journal, and notebook-last commit. The five-profile example is unresolved source material, not directly applicable. A/B/C evaluation fetches one shared frozen corpus and replays it offline; human Precision@50 is never fabricated.

Rollback

Rollback is an external-journal transaction that restores the exact frozen assets to a numeric raw workspace before removing the quarantined formal copy. Its public entry point automatically resumes the sole valid active rollback journal and fails closed on commit conflicts or ambiguous journals. Only Metadata v2.0 and Catalog v3.2 are supported; older layouts must be regenerated outside the active pipeline because this repository has no runtime migration path.

Verification and packaging

C:\Users\Admin\.conda\envs\mineru\python.exe scripts/agent_acceptance.py --full
C:\Users\Admin\.conda\envs\mineru\python.exe scripts/pack_repo.py

Default audit and source snapshots are runtime-zero. Audit includes dirty lightweight source and synthetic fixtures only; it never samples real paper_raw, papers, transactions, ledger, output, logs, credentials, writing jobs, or runtime catalog folders (assignments, tasks, results, links).

Commit recovery requires explicit trusted transaction, raw-workspace, and formal-paper roots. PDF retrieval is direct-first with content-aware proxy fallback and independently configured route timeouts; persisted diagnostics strip URL queries recursively. Discovery exports and source snapshots are accepted only after complete identity and byte-level integrity verification.

Discovery staging single track

Discovery batches share one staging context and one journal index. Both numeric raw workspaces and active formal papers participate in DOI/provider-identity deduplication; formal papers win only after generation, ledger, path, Metadata, paper-number, and immutable manifest-hash validation. Repair backlog probing is capped and cursor-rotated at batch boundaries, whereas matched workspaces are revalidated immediately. A bounded producer queue feeds one staging consumer in groups of at most 16 without weakening either per-paper ledger durability checkpoint. Retryable outcomes remain scheduled in the current runtime. Provider page publication uses a short Journal-index lock and consumer shutdown is governed by a configurable no-progress watchdog rather than a fixed 60-second total limit.

Network candidates follow one production path: pending_queue → stage_network_metadata_records → DiscoveryStageTransaction → WorkspaceRegistry. The ledger owns numbering only; Evidence reads facts; Readiness applies the manifest workflow profile (unknown profiles fail closed); Registry alone scans DOI/identity facts and atomically publishes copy-on-write refreshes. The allocator is discovery-agnostic, and the pending queue makes no workspace reuse, duplicate, or allocation decision.

An incomplete reserved workspace remains unsettled; an incomplete metadata_staged closure is repair_required before allocation. Identity facts use identity_key + paper_number, so one paper retains multiple provider identities through freeze. Warm staging reuses one locked ledger view, keeps reservation and final-state durable checkpoints, and directly publishes the validated record with no successful-stage post-refresh.